[jira] Updated: (SOLR-207) snappuller inefficient finding latest snapshot
[
https://issues.apache.org/jira/browse/SOLR-207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yonik Seeley updated SOLR-207:
--
Attachment: find_maxdepth.patch
uses -maxdepth 1 to avoid recursion.
Bill - does this look OK?
snappuller inefficient finding latest snapshot
--
Key: SOLR-207
URL: https://issues.apache.org/jira/browse/SOLR-207
Project: Solr
Issue Type: Bug
Components: replication
Reporter: Yonik Seeley
Attachments: find_maxdepth.patch
snapinstaller (and snappuller) do the following to find the latest snapshot:
name=`find ${data_dir} -name snapshot.* -print|grep -v wip|sort -r|head -1`
This recurses into all of the snapshot directories, doing much more disk-io
than is necessary.
I think it is the cause of bloated kernel memory usage we have seen on some
of our Linux boxes, caused
by kernel dentry and inode caches. Those caches compete with buffer cache
(caching the actual data of the index)
and can thus decrease performance.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-207) snappuller inefficient finding latest snapshot
[
https://issues.apache.org/jira/browse/SOLR-207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yonik Seeley updated SOLR-207:
--
Attachment: find_maxdepth.patch
Updated patch:
- switches back to ls,
- tries to determine if maxdepth is supported for the cleanup scripts that
need to find -mtime
- in snappuller, make the master find the latest snapshot instead of sending
the complete ls across the network.
This has not yet been tested.
snappuller inefficient finding latest snapshot
--
Key: SOLR-207
URL: https://issues.apache.org/jira/browse/SOLR-207
Project: Solr
Issue Type: Bug
Components: replication
Reporter: Yonik Seeley
Attachments: find_maxdepth.patch, find_maxdepth.patch
snapinstaller (and snappuller) do the following to find the latest snapshot:
name=`find ${data_dir} -name snapshot.* -print|grep -v wip|sort -r|head -1`
This recurses into all of the snapshot directories, doing much more disk-io
than is necessary.
I think it is the cause of bloated kernel memory usage we have seen on some
of our Linux boxes, caused
by kernel dentry and inode caches. Those caches compete with buffer cache
(caching the actual data of the index)
and can thus decrease performance.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-207) snappuller inefficient finding latest snapshot
[
https://issues.apache.org/jira/browse/SOLR-207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yonik Seeley updated SOLR-207:
--
Attachment: find_maxdepth.patch
re-attaching with ASF perms (in the older JIRA version, the grant license
option was first, and now it is last... hence I keep clicking the incorrect one)
snappuller inefficient finding latest snapshot
--
Key: SOLR-207
URL: https://issues.apache.org/jira/browse/SOLR-207
Project: Solr
Issue Type: Bug
Components: replication
Reporter: Yonik Seeley
Attachments: find_maxdepth.patch, find_maxdepth.patch
snapinstaller (and snappuller) do the following to find the latest snapshot:
name=`find ${data_dir} -name snapshot.* -print|grep -v wip|sort -r|head -1`
This recurses into all of the snapshot directories, doing much more disk-io
than is necessary.
I think it is the cause of bloated kernel memory usage we have seen on some
of our Linux boxes, caused
by kernel dentry and inode caches. Those caches compete with buffer cache
(caching the actual data of the index)
and can thus decrease performance.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-207) snappuller inefficient finding latest snapshot
[
https://issues.apache.org/jira/browse/SOLR-207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yonik Seeley updated SOLR-207:
--
Attachment: (was: find_maxdepth.patch)
snappuller inefficient finding latest snapshot
--
Key: SOLR-207
URL: https://issues.apache.org/jira/browse/SOLR-207
Project: Solr
Issue Type: Bug
Components: replication
Reporter: Yonik Seeley
Attachments: find_maxdepth.patch, find_maxdepth.patch
snapinstaller (and snappuller) do the following to find the latest snapshot:
name=`find ${data_dir} -name snapshot.* -print|grep -v wip|sort -r|head -1`
This recurses into all of the snapshot directories, doing much more disk-io
than is necessary.
I think it is the cause of bloated kernel memory usage we have seen on some
of our Linux boxes, caused
by kernel dentry and inode caches. Those caches compete with buffer cache
(caching the actual data of the index)
and can thus decrease performance.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
