[jira] [Resolved] (COUCHDB-1004) list_to_existing_atom is too restrictive as used by couch_rep
[ https://issues.apache.org/jira/browse/COUCHDB-1004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Filipe Manana resolved COUCHDB-1004. Resolution: Fixed Fix Version/s: (was: 1.2) Applied Adam's suggestion (use couch_util:to_existing_atom/1) to 1.0.x and 1.1.x. Trunk does not suffer from this issue. list_to_existing_atom is too restrictive as used by couch_rep - Key: COUCHDB-1004 URL: https://issues.apache.org/jira/browse/COUCHDB-1004 Project: CouchDB Issue Type: Bug Components: Replication Environment: erlang Reporter: Bob Dionne Priority: Minor Fix For: 1.0.3, 1.1 Attachments: COUCHDB-1004.patch We'd like to additional information to db_info in BigCouch, such as the Q and N constants for a given database. This causes replication to fail when replicating from BigCouch to CouchDB due to the use of list_to_existing_atom in couch_rep:dbinfo(... The claim is that list_to_atom pollutes the atoms table, however superficial testing indicates this is not the case, list_to_atom when called repeatedly seems to work fine. If this is true then consider reverting list_to_existing_atom back to list_to_atom. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1132) Track used space of database and view index files
[
https://issues.apache.org/jira/browse/COUCHDB-1132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13022078#comment-13022078
]
Filipe Manana commented on COUCHDB-1132:
I made a few tests with much larger databases, and here follow the results.
* 75Mb database, 12 531 documents
Before compaction:
$ curl http://localhost:5985/testdb1
{db_name:testdb1,doc_count:12531,doc_del_count:0,update_seq:12531,purge_seq:0,
compact_running:false,disk_size:77545585,data_size:35483560,
instance_start_time:1303288992990482,disk_format_version:5,committed_update_seq:12531}
After compaction:
$ curl http://localhost:5985/testdb1
{db_name:testdb1,doc_count:12531,doc_del_count:0,update_seq:12531,purge_seq:0,
compact_running:false,disk_size:41271409,data_size:35453857,instance_start_time:1303288992990482,
disk_format_version:5,committed_update_seq:12531}
data size is about 86% of the file size
* 1.8Gb database, 262 531 documents
Before compaction:
$ curl http://localhost:5985/testdb1
{db_name:testdb1,doc_count:262531,doc_del_count:0,update_seq:262531,purge_seq:0,
compact_running:false,disk_size:1962610801,data_size:744835248,instance_start_time:1303289719133306,
disk_format_version:5,committed_update_seq:262531}
After compaction:
$ curl http://localhost:5985/testdb1
{db_name:testdb1,doc_count:262531,doc_del_count:0,update_seq:262531,purge_seq:0,
compact_running:false,disk_size:1139642481,data_size:744292081,instance_start_time:1303289719133306,
disk_format_version:5,committed_update_seq:262531}
data size is about 65% of the file size
After changing compaction checkpoint frequency from 10 000 to 10 000 000 000
and compacting the database again:
$ curl http://localhost:5985/testdb1
{db_name:testdb1,doc_count:262531,doc_del_count:0,update_seq:262531,purge_seq:0,
compact_running:false,disk_size:1139601521,data_size:744292168,instance_start_time:1303296830183399,
disk_format_version:5,committed_update_seq:262531}
data size is still about 65% of the file size
After changing compaction batch size from 1 000 to 100 000
$ curl http://localhost:5985/testdb1
{db_name:testdb1,doc_count:262531,doc_del_count:0,update_seq:262531,purge_seq:0,
compact_running:false,disk_size:776962161,data_size:744307523,instance_start_time:1303297206958149,
disk_format_version:5,committed_update_seq:262531}
data size is now about 96% of the file size
* 16Gb database, 3 341 491 documents
(No data dize before compaction since it was a database created with trunk
CouchDB)
After compaction:
$ curl http://localhost:5985/large_1_20
{db_name:large_1_20,doc_count:3341491,doc_del_count:0,update_seq:3341491,purge_seq:0,
compact_running:false,disk_size:16318431354,data_size:15069943338,instance_start_time:1303296570043058,
disk_format_version:5,committed_update_seq:3341491}
data size is about 92% of the file size
This makes me think we should make the compaction checkpoint frequency and
batch size configurable in the .ini (specially the batch size), since this can
reduce significantly the final file size as well as make the compaction a bit
faster.
Anyone -1 on doing this?
For view indexes, the batch size is controlled by the size of the work queues,
but I believe Adam and/or Paul were thinking about making this configurable.
Track used space of database and view index files
-
Key: COUCHDB-1132
URL: https://issues.apache.org/jira/browse/COUCHDB-1132
Project: CouchDB
Issue Type: New Feature
Components: Database Core
Reporter: Filipe Manana
Fix For: 1.2
Currently users have no reliable way to know if a database or view index
compaction is needed.
Both me, Adam and Robert Dionne have been working on a feature to compute and
expose the current data size (in bytes) of databases and view indexes. These
computations are exposed as a single field in the database info and view
index info URIs.
Comparing this new value with the disk_size value (the total space in bytes
used by the database or view index file) would allow users to decide whether
or not it's worth to trigger a compaction.
Adam and Robert's work can be found at:
https://github.com/cloudant/bigcouch/compare/7d1adfa...a9410e6
Mine can be found at:
https://github.com/fdmanana/couchdb/compare/file_space
After chatting with Adam on IRC, the main difference seems to be that they're
work accounts only for user data (document bodies + attachments), while mine
also accounts for the btree values (including all meta information, keys, rev
trees, etc) and the data added by couch_file (4 bytes length prefix, md5s,
block boundary markers).
An example:
$ curl http://localhost:5984/btree_db/_design/test/_info
[jira] [Commented] (COUCHDB-1132) Track used space of database and view index files
[
https://issues.apache.org/jira/browse/COUCHDB-1132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13022099#comment-13022099
]
Jan Lehnardt commented on COUCHDB-1132:
---
I'm all for making the compactor smarter :)
Great work Filipe!
I wish we could accurately make this equation work file_size - data_size =
post_compaction_file_size, but it seems overly complicated to try, it would
just be a nice API behaviour, that isn't required for any of this. So yeah.
Track used space of database and view index files
-
Key: COUCHDB-1132
URL: https://issues.apache.org/jira/browse/COUCHDB-1132
Project: CouchDB
Issue Type: New Feature
Components: Database Core
Reporter: Filipe Manana
Fix For: 1.2
Currently users have no reliable way to know if a database or view index
compaction is needed.
Both me, Adam and Robert Dionne have been working on a feature to compute and
expose the current data size (in bytes) of databases and view indexes. These
computations are exposed as a single field in the database info and view
index info URIs.
Comparing this new value with the disk_size value (the total space in bytes
used by the database or view index file) would allow users to decide whether
or not it's worth to trigger a compaction.
Adam and Robert's work can be found at:
https://github.com/cloudant/bigcouch/compare/7d1adfa...a9410e6
Mine can be found at:
https://github.com/fdmanana/couchdb/compare/file_space
After chatting with Adam on IRC, the main difference seems to be that they're
work accounts only for user data (document bodies + attachments), while mine
also accounts for the btree values (including all meta information, keys, rev
trees, etc) and the data added by couch_file (4 bytes length prefix, md5s,
block boundary markers).
An example:
$ curl http://localhost:5984/btree_db/_design/test/_info
{name:test,view_index:{signature:aba9f066ed7f042f63d245ce0c7d870e,language:javascript,disk_size:274556,data_size:270455,updater_running:false,compact_running:false,waiting_commit:false,waiting_clients:0,update_seq:1004,purge_seq:0}}
$ curl http://localhost:5984/btree_db
{db_name:btree_db,doc_count:1004,doc_del_count:0,update_seq:1004,purge_seq:0,compact_running:false,disk_size:6197361,data_size:6186460,instance_start_time:1303231080936421,disk_format_version:5,committed_update_seq:1004}
This example was executed just after compacting the test database and view
index. The new filed data_size has a value very close to the final file
size.
The only thing that my branch doesn't include in the data_size computation,
for databases, are the size of the last header, the size of the _security
object and purged revs list - in practice these are very small and
insignificant that adding extra code to account them doesn't seem worth it.
I'm sure we can merge the best from both branches.
Adam, Robert, thoughts?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1132) Track used space of database and view index files
[
https://issues.apache.org/jira/browse/COUCHDB-1132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13022142#comment-13022142
]
Adam Kocoloski commented on COUCHDB-1132:
-
@janl Did you mean data_size = post_compaction_file_size? What you wrote
doesn't make sense to me. And yes, I think it would be too complicated to try
to do that.
@fdmanana The view compactor uses a static batch size of 1. The work
queues are only involved during indexing. I put a patch somewhere to place a
configurable minimum bound on the size of the batch written to disk during
indexing, which does help reduce the file size.
Regarding the config entry, I've started to think that every new config entry
we add represents a problem we couldn't solve for the end user. If we need to
have an entry, maybe we should use units that make more sense for the user,
e.g. a threshold in bytes for the compactor process above which it flushes to
disk. I'd be particularly in favor of such a threshold for the view compactor,
since the the map values are loaded into memory simultaneously (as opposed to
the document bodies, which are written to the new file one at a time regardless
of batch size). Different view compactions can use wildly different amounts of
memory depending on the average value size.
Track used space of database and view index files
-
Key: COUCHDB-1132
URL: https://issues.apache.org/jira/browse/COUCHDB-1132
Project: CouchDB
Issue Type: New Feature
Components: Database Core
Reporter: Filipe Manana
Fix For: 1.2
Currently users have no reliable way to know if a database or view index
compaction is needed.
Both me, Adam and Robert Dionne have been working on a feature to compute and
expose the current data size (in bytes) of databases and view indexes. These
computations are exposed as a single field in the database info and view
index info URIs.
Comparing this new value with the disk_size value (the total space in bytes
used by the database or view index file) would allow users to decide whether
or not it's worth to trigger a compaction.
Adam and Robert's work can be found at:
https://github.com/cloudant/bigcouch/compare/7d1adfa...a9410e6
Mine can be found at:
https://github.com/fdmanana/couchdb/compare/file_space
After chatting with Adam on IRC, the main difference seems to be that they're
work accounts only for user data (document bodies + attachments), while mine
also accounts for the btree values (including all meta information, keys, rev
trees, etc) and the data added by couch_file (4 bytes length prefix, md5s,
block boundary markers).
An example:
$ curl http://localhost:5984/btree_db/_design/test/_info
{name:test,view_index:{signature:aba9f066ed7f042f63d245ce0c7d870e,language:javascript,disk_size:274556,data_size:270455,updater_running:false,compact_running:false,waiting_commit:false,waiting_clients:0,update_seq:1004,purge_seq:0}}
$ curl http://localhost:5984/btree_db
{db_name:btree_db,doc_count:1004,doc_del_count:0,update_seq:1004,purge_seq:0,compact_running:false,disk_size:6197361,data_size:6186460,instance_start_time:1303231080936421,disk_format_version:5,committed_update_seq:1004}
This example was executed just after compacting the test database and view
index. The new filed data_size has a value very close to the final file
size.
The only thing that my branch doesn't include in the data_size computation,
for databases, are the size of the last header, the size of the _security
object and purged revs list - in practice these are very small and
insignificant that adding extra code to account them doesn't seem worth it.
I'm sure we can merge the best from both branches.
Adam, Robert, thoughts?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1132) Track used space of database and view index files
[
https://issues.apache.org/jira/browse/COUCHDB-1132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13022158#comment-13022158
]
Jan Lehnardt commented on COUCHDB-1132:
---
@Adam I meant that it'd be nice if a user could look at disk_size and data_size
and do the math on what a compaction could do. the more accurate the better,
but I'm happy to settle for too complicated.
I like your take on configuration options.
Track used space of database and view index files
-
Key: COUCHDB-1132
URL: https://issues.apache.org/jira/browse/COUCHDB-1132
Project: CouchDB
Issue Type: New Feature
Components: Database Core
Reporter: Filipe Manana
Fix For: 1.2
Currently users have no reliable way to know if a database or view index
compaction is needed.
Both me, Adam and Robert Dionne have been working on a feature to compute and
expose the current data size (in bytes) of databases and view indexes. These
computations are exposed as a single field in the database info and view
index info URIs.
Comparing this new value with the disk_size value (the total space in bytes
used by the database or view index file) would allow users to decide whether
or not it's worth to trigger a compaction.
Adam and Robert's work can be found at:
https://github.com/cloudant/bigcouch/compare/7d1adfa...a9410e6
Mine can be found at:
https://github.com/fdmanana/couchdb/compare/file_space
After chatting with Adam on IRC, the main difference seems to be that they're
work accounts only for user data (document bodies + attachments), while mine
also accounts for the btree values (including all meta information, keys, rev
trees, etc) and the data added by couch_file (4 bytes length prefix, md5s,
block boundary markers).
An example:
$ curl http://localhost:5984/btree_db/_design/test/_info
{name:test,view_index:{signature:aba9f066ed7f042f63d245ce0c7d870e,language:javascript,disk_size:274556,data_size:270455,updater_running:false,compact_running:false,waiting_commit:false,waiting_clients:0,update_seq:1004,purge_seq:0}}
$ curl http://localhost:5984/btree_db
{db_name:btree_db,doc_count:1004,doc_del_count:0,update_seq:1004,purge_seq:0,compact_running:false,disk_size:6197361,data_size:6186460,instance_start_time:1303231080936421,disk_format_version:5,committed_update_seq:1004}
This example was executed just after compacting the test database and view
index. The new filed data_size has a value very close to the final file
size.
The only thing that my branch doesn't include in the data_size computation,
for databases, are the size of the last header, the size of the _security
object and purged revs list - in practice these are very small and
insignificant that adding extra code to account them doesn't seem worth it.
I'm sure we can merge the best from both branches.
Adam, Robert, thoughts?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1132) Track used space of database and view index files
[
https://issues.apache.org/jira/browse/COUCHDB-1132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13022162#comment-13022162
]
Paul Joseph Davis commented on COUCHDB-1132:
@Jan
That won't be doable until we make the b+tree balance itself during writes. But
once/if we get around to that your request would happen more or less
automatically with a few tweaks to the compactor.
Track used space of database and view index files
-
Key: COUCHDB-1132
URL: https://issues.apache.org/jira/browse/COUCHDB-1132
Project: CouchDB
Issue Type: New Feature
Components: Database Core
Reporter: Filipe Manana
Fix For: 1.2
Currently users have no reliable way to know if a database or view index
compaction is needed.
Both me, Adam and Robert Dionne have been working on a feature to compute and
expose the current data size (in bytes) of databases and view indexes. These
computations are exposed as a single field in the database info and view
index info URIs.
Comparing this new value with the disk_size value (the total space in bytes
used by the database or view index file) would allow users to decide whether
or not it's worth to trigger a compaction.
Adam and Robert's work can be found at:
https://github.com/cloudant/bigcouch/compare/7d1adfa...a9410e6
Mine can be found at:
https://github.com/fdmanana/couchdb/compare/file_space
After chatting with Adam on IRC, the main difference seems to be that they're
work accounts only for user data (document bodies + attachments), while mine
also accounts for the btree values (including all meta information, keys, rev
trees, etc) and the data added by couch_file (4 bytes length prefix, md5s,
block boundary markers).
An example:
$ curl http://localhost:5984/btree_db/_design/test/_info
{name:test,view_index:{signature:aba9f066ed7f042f63d245ce0c7d870e,language:javascript,disk_size:274556,data_size:270455,updater_running:false,compact_running:false,waiting_commit:false,waiting_clients:0,update_seq:1004,purge_seq:0}}
$ curl http://localhost:5984/btree_db
{db_name:btree_db,doc_count:1004,doc_del_count:0,update_seq:1004,purge_seq:0,compact_running:false,disk_size:6197361,data_size:6186460,instance_start_time:1303231080936421,disk_format_version:5,committed_update_seq:1004}
This example was executed just after compacting the test database and view
index. The new filed data_size has a value very close to the final file
size.
The only thing that my branch doesn't include in the data_size computation,
for databases, are the size of the last header, the size of the _security
object and purged revs list - in practice these are very small and
insignificant that adding extra code to account them doesn't seem worth it.
I'm sure we can merge the best from both branches.
Adam, Robert, thoughts?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1132) Track used space of database and view index files
[
https://issues.apache.org/jira/browse/COUCHDB-1132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13022164#comment-13022164
]
Paul Joseph Davis commented on COUCHDB-1132:
@Adam,
I agree with your comment about using memory usage as the threshold so that
people have a better understanding of what it is they're setting.
Track used space of database and view index files
-
Key: COUCHDB-1132
URL: https://issues.apache.org/jira/browse/COUCHDB-1132
Project: CouchDB
Issue Type: New Feature
Components: Database Core
Reporter: Filipe Manana
Fix For: 1.2
Currently users have no reliable way to know if a database or view index
compaction is needed.
Both me, Adam and Robert Dionne have been working on a feature to compute and
expose the current data size (in bytes) of databases and view indexes. These
computations are exposed as a single field in the database info and view
index info URIs.
Comparing this new value with the disk_size value (the total space in bytes
used by the database or view index file) would allow users to decide whether
or not it's worth to trigger a compaction.
Adam and Robert's work can be found at:
https://github.com/cloudant/bigcouch/compare/7d1adfa...a9410e6
Mine can be found at:
https://github.com/fdmanana/couchdb/compare/file_space
After chatting with Adam on IRC, the main difference seems to be that they're
work accounts only for user data (document bodies + attachments), while mine
also accounts for the btree values (including all meta information, keys, rev
trees, etc) and the data added by couch_file (4 bytes length prefix, md5s,
block boundary markers).
An example:
$ curl http://localhost:5984/btree_db/_design/test/_info
{name:test,view_index:{signature:aba9f066ed7f042f63d245ce0c7d870e,language:javascript,disk_size:274556,data_size:270455,updater_running:false,compact_running:false,waiting_commit:false,waiting_clients:0,update_seq:1004,purge_seq:0}}
$ curl http://localhost:5984/btree_db
{db_name:btree_db,doc_count:1004,doc_del_count:0,update_seq:1004,purge_seq:0,compact_running:false,disk_size:6197361,data_size:6186460,instance_start_time:1303231080936421,disk_format_version:5,committed_update_seq:1004}
This example was executed just after compacting the test database and view
index. The new filed data_size has a value very close to the final file
size.
The only thing that my branch doesn't include in the data_size computation,
for databases, are the size of the last header, the size of the _security
object and purged revs list - in practice these are very small and
insignificant that adding extra code to account them doesn't seem worth it.
I'm sure we can merge the best from both branches.
Adam, Robert, thoughts?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1131) Port Futon2 local replication target work to Futon
[ https://issues.apache.org/jira/browse/COUCHDB-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13022173#comment-13022173 ] Benjamin Young commented on COUCHDB-1131: - These have been added to Futon 2, fwiw: https://github.com/BigBlueHat/couchdb/commits/sammy/ Port Futon2 local replication target work to Futon -- Key: COUCHDB-1131 URL: https://issues.apache.org/jira/browse/COUCHDB-1131 Project: CouchDB Issue Type: New Feature Components: Futon Reporter: Sam Bisbee Priority: Minor Attachments: 0001-Adding-jQuery-UI-1.8.11-JS-and-CSS-for-just-the-auto.patch, 0002-Replacing-replication-s-local-target-dropdown-with-a.patch Futon2 currently has a patch that makes the replication target into a text input, using jQuery UI's autocomplete widget to still provide the all databases functionality. If the database doesn't exist, then the user is prompted to see if they want it created with a JS confirm() prompt. If they do, then the create_target replication option is set to true. This will be a port of this Futon2 pull request: https://github.com/BigBlueHat/couchdb/pull/1 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1132) Track used space of database and view index files
[
https://issues.apache.org/jira/browse/COUCHDB-1132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13022257#comment-13022257
]
Filipe Manana commented on COUCHDB-1132:
Ok, sounds good to me enough. And actually just did a small patch for it:
http://friendpaste.com/3ACjKssyNXMhFju9irTdJg
If objections, I'll create a ticket for it to avoid this one blocked.
Track used space of database and view index files
-
Key: COUCHDB-1132
URL: https://issues.apache.org/jira/browse/COUCHDB-1132
Project: CouchDB
Issue Type: New Feature
Components: Database Core
Reporter: Filipe Manana
Fix For: 1.2
Currently users have no reliable way to know if a database or view index
compaction is needed.
Both me, Adam and Robert Dionne have been working on a feature to compute and
expose the current data size (in bytes) of databases and view indexes. These
computations are exposed as a single field in the database info and view
index info URIs.
Comparing this new value with the disk_size value (the total space in bytes
used by the database or view index file) would allow users to decide whether
or not it's worth to trigger a compaction.
Adam and Robert's work can be found at:
https://github.com/cloudant/bigcouch/compare/7d1adfa...a9410e6
Mine can be found at:
https://github.com/fdmanana/couchdb/compare/file_space
After chatting with Adam on IRC, the main difference seems to be that they're
work accounts only for user data (document bodies + attachments), while mine
also accounts for the btree values (including all meta information, keys, rev
trees, etc) and the data added by couch_file (4 bytes length prefix, md5s,
block boundary markers).
An example:
$ curl http://localhost:5984/btree_db/_design/test/_info
{name:test,view_index:{signature:aba9f066ed7f042f63d245ce0c7d870e,language:javascript,disk_size:274556,data_size:270455,updater_running:false,compact_running:false,waiting_commit:false,waiting_clients:0,update_seq:1004,purge_seq:0}}
$ curl http://localhost:5984/btree_db
{db_name:btree_db,doc_count:1004,doc_del_count:0,update_seq:1004,purge_seq:0,compact_running:false,disk_size:6197361,data_size:6186460,instance_start_time:1303231080936421,disk_format_version:5,committed_update_seq:1004}
This example was executed just after compacting the test database and view
index. The new filed data_size has a value very close to the final file
size.
The only thing that my branch doesn't include in the data_size computation,
for databases, are the size of the last header, the size of the _security
object and purged revs list - in practice these are very small and
insignificant that adding extra code to account them doesn't seem worth it.
I'm sure we can merge the best from both branches.
Adam, Robert, thoughts?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (COUCHDB-1132) Track used space of database and view index files
[
https://issues.apache.org/jira/browse/COUCHDB-1132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Filipe Manana resolved COUCHDB-1132.
Resolution: Fixed
Applied to trunk.
Thanks everyone, specially Adam and Robert Dionne
Track used space of database and view index files
-
Key: COUCHDB-1132
URL: https://issues.apache.org/jira/browse/COUCHDB-1132
Project: CouchDB
Issue Type: New Feature
Components: Database Core
Reporter: Filipe Manana
Fix For: 1.2
Currently users have no reliable way to know if a database or view index
compaction is needed.
Both me, Adam and Robert Dionne have been working on a feature to compute and
expose the current data size (in bytes) of databases and view indexes. These
computations are exposed as a single field in the database info and view
index info URIs.
Comparing this new value with the disk_size value (the total space in bytes
used by the database or view index file) would allow users to decide whether
or not it's worth to trigger a compaction.
Adam and Robert's work can be found at:
https://github.com/cloudant/bigcouch/compare/7d1adfa...a9410e6
Mine can be found at:
https://github.com/fdmanana/couchdb/compare/file_space
After chatting with Adam on IRC, the main difference seems to be that they're
work accounts only for user data (document bodies + attachments), while mine
also accounts for the btree values (including all meta information, keys, rev
trees, etc) and the data added by couch_file (4 bytes length prefix, md5s,
block boundary markers).
An example:
$ curl http://localhost:5984/btree_db/_design/test/_info
{name:test,view_index:{signature:aba9f066ed7f042f63d245ce0c7d870e,language:javascript,disk_size:274556,data_size:270455,updater_running:false,compact_running:false,waiting_commit:false,waiting_clients:0,update_seq:1004,purge_seq:0}}
$ curl http://localhost:5984/btree_db
{db_name:btree_db,doc_count:1004,doc_del_count:0,update_seq:1004,purge_seq:0,compact_running:false,disk_size:6197361,data_size:6186460,instance_start_time:1303231080936421,disk_format_version:5,committed_update_seq:1004}
This example was executed just after compacting the test database and view
index. The new filed data_size has a value very close to the final file
size.
The only thing that my branch doesn't include in the data_size computation,
for databases, are the size of the last header, the size of the _security
object and purged revs list - in practice these are very small and
insignificant that adding extra code to account them doesn't seem worth it.
I'm sure we can merge the best from both branches.
Adam, Robert, thoughts?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1103) Make vhost definitions not order-dependent, remove variable matching
[
https://issues.apache.org/jira/browse/COUCHDB-1103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13022309#comment-13022309
]
Jan Lehnardt commented on COUCHDB-1103:
---
@Benoit, in line
+DefaultPort = mochiweb_socket_server:get(couch_httpd, port),
did you mean s/mochiweb_socket_server/couch_config/?
If yes, here's a patch against 1.1.x that passes all etap tests for me reliably.
diff --git a/src/couchdb/couch_httpd_vhost.erl
b/src/couchdb/couch_httpd_vhost.erl
index 3dba291..b64996a 100644
--- a/src/couchdb/couch_httpd_vhost.erl
+++ b/src/couchdb/couch_httpd_vhost.erl
@@ -329,10 +329,10 @@ bind_path(_, _) -
%% create vhost list from ini
make_vhosts() -
-lists:foldl(fun({Vhost, Path}, Acc) -
-[{parse_vhost(Vhost), split_path(Path)}|Acc]
-end, [], couch_config:get(vhosts)).
-
+Vhosts = lists:foldl(fun({Vhost, Path}, Acc) -
+[{parse_vhost(Vhost), split_path(Path)}|Acc]
+ end, [], couch_config:get(vhosts)),
+lists:reverse(lists:usort(Vhosts)).
parse_vhost(Vhost) -
case urlsplit_netloc(Vhost, []) of
@@ -350,15 +350,16 @@ parse_vhost(Vhost) -
split_host_port(HostAsString) -
+DefaultPort = couch_config:get(couch_httpd, port),
case string:rchr(HostAsString, $:) of
0 -
-{split_host(HostAsString), 80};
+{split_host(HostAsString), DefaultPort};
N -
HostPart = string:substr(HostAsString, 1, N-1),
case (catch erlang:list_to_integer(HostAsString, N+1,
length(HostAsString))) of
{'EXIT', _} -
-{split_host(HostAsString), 80};
+{split_host(HostAsString), DefaultPort};
Port -
{split_host(HostPart), Port}
end
Make vhost definitions not order-dependent, remove variable matching
Key: COUCHDB-1103
URL: https://issues.apache.org/jira/browse/COUCHDB-1103
Project: CouchDB
Issue Type: Bug
Components: HTTP Interface
Affects Versions: 1.1, 1.2
Reporter: Jan Lehnardt
Assignee: Benoit Chesneau
Priority: Critical
Fix For: 1.1
Attachments:
0001-fix-COUCHDB-1103.-Order-results-and-make-sure-we-use.patch
As per
http://mail-archives.apache.org/mod_mbox/couchdb-dev/201103.mbox/%3cAANLkTin6zRiiFtKZoFa60DRVyOcF1dXzMYtNdKuX=z...@mail.gmail.com%3e
Recording it here to not lose track.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (COUCHDB-1131) Port Futon2 local replication target work to Futon
[ https://issues.apache.org/jira/browse/COUCHDB-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Bisbee updated COUCHDB-1131: Attachment: 0003-NOTICE-and-LICENSE-updates-for-inclusion-of-jQuery-U.patch Modifications to NOTICE and LICENSE files for including the jQuery UI project, per discussion with Paul. Port Futon2 local replication target work to Futon -- Key: COUCHDB-1131 URL: https://issues.apache.org/jira/browse/COUCHDB-1131 Project: CouchDB Issue Type: New Feature Components: Futon Reporter: Sam Bisbee Priority: Minor Attachments: 0001-Adding-jQuery-UI-1.8.11-JS-and-CSS-for-just-the-auto.patch, 0002-Replacing-replication-s-local-target-dropdown-with-a.patch, 0003-NOTICE-and-LICENSE-updates-for-inclusion-of-jQuery-U.patch Futon2 currently has a patch that makes the replication target into a text input, using jQuery UI's autocomplete widget to still provide the all databases functionality. If the database doesn't exist, then the user is prompted to see if they want it created with a JS confirm() prompt. If they do, then the create_target replication option is set to true. This will be a port of this Futon2 pull request: https://github.com/BigBlueHat/couchdb/pull/1 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1131) Port Futon2 local replication target work to Futon
[ https://issues.apache.org/jira/browse/COUCHDB-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13022367#comment-13022367 ] Dale Harvey commented on COUCHDB-1131: -- +1 on this, currently its a nightmare to replicate to databases that dont exist Port Futon2 local replication target work to Futon -- Key: COUCHDB-1131 URL: https://issues.apache.org/jira/browse/COUCHDB-1131 Project: CouchDB Issue Type: New Feature Components: Futon Reporter: Sam Bisbee Priority: Minor Attachments: 0001-Adding-jQuery-UI-1.8.11-JS-and-CSS-for-just-the-auto.patch, 0002-Replacing-replication-s-local-target-dropdown-with-a.patch, 0003-NOTICE-and-LICENSE-updates-for-inclusion-of-jQuery-U.patch Futon2 currently has a patch that makes the replication target into a text input, using jQuery UI's autocomplete widget to still provide the all databases functionality. If the database doesn't exist, then the user is prompted to see if they want it created with a JS confirm() prompt. If they do, then the create_target replication option is set to true. This will be a port of this Futon2 pull request: https://github.com/BigBlueHat/couchdb/pull/1 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1131) Port Futon2 local replication target work to Futon
[ https://issues.apache.org/jira/browse/COUCHDB-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13022371#comment-13022371 ] Benjamin Young commented on COUCHDB-1131: - +1 This is definitely an improvement. I've also pulled it into Futon2 (via the pull request above), fwiw. Port Futon2 local replication target work to Futon -- Key: COUCHDB-1131 URL: https://issues.apache.org/jira/browse/COUCHDB-1131 Project: CouchDB Issue Type: New Feature Components: Futon Reporter: Sam Bisbee Priority: Minor Attachments: 0001-Adding-jQuery-UI-1.8.11-JS-and-CSS-for-just-the-auto.patch, 0002-Replacing-replication-s-local-target-dropdown-with-a.patch, 0003-NOTICE-and-LICENSE-updates-for-inclusion-of-jQuery-U.patch Futon2 currently has a patch that makes the replication target into a text input, using jQuery UI's autocomplete widget to still provide the all databases functionality. If the database doesn't exist, then the user is prompted to see if they want it created with a JS confirm() prompt. If they do, then the create_target replication option is set to true. This will be a port of this Futon2 pull request: https://github.com/BigBlueHat/couchdb/pull/1 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (COUCHDB-1133) Fix opacity on favicon
Fix opacity on favicon -- Key: COUCHDB-1133 URL: https://issues.apache.org/jira/browse/COUCHDB-1133 Project: CouchDB Issue Type: Improvement Components: Futon Reporter: Dale Harvey Priority: Trivial Fix the opacity on the favicon so it doesnt look awkward on non white browser chrome -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (COUCHDB-1131) Port Futon2 local replication target work to Futon
[ https://issues.apache.org/jira/browse/COUCHDB-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Joseph Davis resolved COUCHDB-1131. Resolution: Fixed Committed to trunk and backported to 1.1.0 Port Futon2 local replication target work to Futon -- Key: COUCHDB-1131 URL: https://issues.apache.org/jira/browse/COUCHDB-1131 Project: CouchDB Issue Type: New Feature Components: Futon Reporter: Sam Bisbee Priority: Minor Attachments: 0001-Adding-jQuery-UI-1.8.11-JS-and-CSS-for-just-the-auto.patch, 0002-Replacing-replication-s-local-target-dropdown-with-a.patch, 0003-NOTICE-and-LICENSE-updates-for-inclusion-of-jQuery-U.patch Futon2 currently has a patch that makes the replication target into a text input, using jQuery UI's autocomplete widget to still provide the all databases functionality. If the database doesn't exist, then the user is prompted to see if they want it created with a JS confirm() prompt. If they do, then the create_target replication option is set to true. This will be a port of this Futon2 pull request: https://github.com/BigBlueHat/couchdb/pull/1 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (COUCHDB-984) Animated spinner icon has glitch
[ https://issues.apache.org/jira/browse/COUCHDB-984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jan Lehnardt updated COUCHDB-984: - Priority: Blocker (was: Minor) Fix Version/s: 1.1 Animated spinner icon has glitch Key: COUCHDB-984 URL: https://issues.apache.org/jira/browse/COUCHDB-984 Project: CouchDB Issue Type: Bug Components: Futon Environment: any Reporter: Nathan Vander Wilt Assignee: Paul Joseph Davis Priority: Blocker Fix For: 1.0.2, 1.1 Attachments: 16x16-Spinner.gif, spinner33-main_page.gif, spinner55-compromise.gif, spinner6b-upload_dialog.gif Original Estimate: 0.5h Remaining Estimate: 0.5h Futon's progress spinner icon found in /share/couchdb/www/image/spinner.gif (used when uploading files and perhaps elsewhere) suffers from the glitch described at http://www.panic.com/blog/2010/10/spinner-rage/, where the fifth frame of the animation flashes more darkly than the others. The Panic post on this issue provides a fixed version of the spinner, but it is a Photoshop file: http://panic.com/blog/wp-content/files/16x16%20Spinner.psd.zip This simply needs to be re-exported by someone with a copy of Adobe's software. (I have a LazyTwitter out on this, otherwise next week I can pester the designers at work for a favor.) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1133) Fix opacity on favicon
[ https://issues.apache.org/jira/browse/COUCHDB-1133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13022403#comment-13022403 ] mikeal commented on COUCHDB-1133: - i did this a while back for Futon2. You can see the latest versions in ben's branch: https://github.com/BigBlueHat/couchdb/tree/sammy/share/www favicon.ico and favicon.png there is also new markup in futon that favors the .png Fix opacity on favicon -- Key: COUCHDB-1133 URL: https://issues.apache.org/jira/browse/COUCHDB-1133 Project: CouchDB Issue Type: Improvement Components: Futon Reporter: Dale Harvey Priority: Trivial Fix the opacity on the favicon so it doesnt look awkward on non white browser chrome -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (COUCHDB-1053) missing_named_view for reduce fxns that compile but throw runtime errors
[
https://issues.apache.org/jira/browse/COUCHDB-1053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paul Joseph Davis updated COUCHDB-1053:
---
Affects Version/s: (was: 1.0)
Fix Version/s: (was: 1.0.2)
1.2
missing_named_view for reduce fxns that compile but throw runtime errors
Key: COUCHDB-1053
URL: https://issues.apache.org/jira/browse/COUCHDB-1053
Project: CouchDB
Issue Type: Bug
Components: JavaScript View Server
Environment: OSX, prebuilt
Reporter: Ryan Richt
Priority: Trivial
Fix For: 1.2
Original Estimate: 1h
Remaining Estimate: 1h
Attempting to query a map-only view with reduce=false yields the same error
message as (1) a view that is undefined and (2) a view that has a defined,
compile-time valid, but run-time invalid reduce function. So if your reduce
function throws a TypeError, you get this cryptic error (cryptic b/c so many
problems yield the same error) instead of an error to that effect. Errors do
show up in the log, and this is what the view gives:
{error:not_found,reason:missing_named_view}
Might be fixed in 1.0.2 but I cant use the built-in updater anymore after
corporate firewall changes!
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (COUCHDB-1134) Bug in database size calculation
Bug in database size calculation
Key: COUCHDB-1134
URL: https://issues.apache.org/jira/browse/COUCHDB-1134
Project: CouchDB
Issue Type: Bug
Affects Versions: 1.2
Reporter: Paul Joseph Davis
Noticed these backtraces while poking at something else. Haven't even peeped
the code so I've got no idea if its trivial. These are also old databases.
[error] [0.120.0] Uncaught error in HTTP request: {error,badarith}
[info] [0.120.0] Stacktrace: [{couch_db,db_data_size,3},
{couch_db,get_db_info,1},
{couch_httpd_db,db_req,2},
{couch_httpd_db,do_db_req,2},
{couch_httpd,handle_request_int,5},
{mochiweb_http,headers,5},
{proc_lib,init_p_do_apply,3}]
[info] [0.120.0] 127.0.0.1 - - GET /foo 500
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1134) Bug in database size calculation
[
https://issues.apache.org/jira/browse/COUCHDB-1134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13022546#comment-13022546
]
Filipe Manana commented on COUCHDB-1134:
Good finding Paul. This missing clause should fix it:
diff --git a/src/couchdb/couch_db.erl b/src/couchdb/couch_db.erl
index ed72bb6..7952366 100644
--- a/src/couchdb/couch_db.erl
+++ b/src/couchdb/couch_db.erl
@@ -281,6 +281,8 @@ db_data_size(_, nil, _) -
db_data_size(_, _, {_Count, _DelCount}) -
% pre 1.2 format, upgraded on compaction
null;
+db_data_size(_, _, {_Count, _DelCount, nil}) -
+null;
db_data_size(SeqBtreeSize, IdBtreeSize, {_Count, _DelCount, DocAndAttsSize}) -
SeqBtreeSize + IdBtreeSize + DocAndAttsSize.
Bug in database size calculation
Key: COUCHDB-1134
URL: https://issues.apache.org/jira/browse/COUCHDB-1134
Project: CouchDB
Issue Type: Bug
Affects Versions: 1.2
Reporter: Paul Joseph Davis
Noticed these backtraces while poking at something else. Haven't even peeped
the code so I've got no idea if its trivial. These are also old databases.
[error] [0.120.0] Uncaught error in HTTP request: {error,badarith}
[info] [0.120.0] Stacktrace: [{couch_db,db_data_size,3},
{couch_db,get_db_info,1},
{couch_httpd_db,db_req,2},
{couch_httpd_db,do_db_req,2},
{couch_httpd,handle_request_int,5},
{mochiweb_http,headers,5},
{proc_lib,init_p_do_apply,3}]
[info] [0.120.0] 127.0.0.1 - - GET /foo 500
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (COUCHDB-984) Animated spinner icon has glitch
[ https://issues.apache.org/jira/browse/COUCHDB-984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Joseph Davis resolved COUCHDB-984. --- Resolution: Fixed Fix Version/s: 1.2 Applied to trunk in 1095557. Backported to 1.1.x and 1.0.x Animated spinner icon has glitch Key: COUCHDB-984 URL: https://issues.apache.org/jira/browse/COUCHDB-984 Project: CouchDB Issue Type: Bug Components: Futon Environment: any Reporter: Nathan Vander Wilt Assignee: Paul Joseph Davis Priority: Blocker Fix For: 1.1, 1.2, 1.0.2 Attachments: 16x16-Spinner.gif, spinner33-main_page.gif, spinner55-compromise.gif, spinner6b-upload_dialog.gif Original Estimate: 0.5h Remaining Estimate: 0.5h Futon's progress spinner icon found in /share/couchdb/www/image/spinner.gif (used when uploading files and perhaps elsewhere) suffers from the glitch described at http://www.panic.com/blog/2010/10/spinner-rage/, where the fifth frame of the animation flashes more darkly than the others. The Panic post on this issue provides a fixed version of the spinner, but it is a Photoshop file: http://panic.com/blog/wp-content/files/16x16%20Spinner.psd.zip This simply needs to be re-exported by someone with a copy of Adobe's software. (I have a LazyTwitter out on this, otherwise next week I can pester the designers at work for a favor.) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
RFC: Releasing 1.0.3
As per the release procedure I'm asking developers for comments on releasing 1.0.3. Also per release procedure I am compelled to specifically ask people to check the NEWS and CHANGES files in the 1.0.x branch for changes since the last release. That is all.
