What's your farm topology like? Is crawling happening from a specific application server, multiple application servers or is it crawling from an 'all-in-one' single SharePoint server that also serves user web requests? This could potentially be due to the fact that the server is taking a bit of time for crawling and indexing each individual document, which then impacts the rate it can crawl at.
What's CPU, RAM, Disk and Network activity like on the server during the crawling? Apart from disabling Antivirus, you can also try disabling the Windows Search indexer, right click the disk drive in Windows Explorer and select properties, untick the "Allow files on this drive to have contents indexed in addition to file properties" although this probably won't solve your specific issue... Sezai. On Wed, Jul 18, 2012 at 12:22 PM, Dylan Tusler < dylan.tus...@sunshinecoast.qld.gov.au> wrote: > ** > I just tried copying one large file, and got about 600Mbps transfer speed. > > I also tried copying about 11Gb of miscellaneous files (about 3700 of > them) which got a speed of about 400Mbps. > > At the rate our crawl was running, that would have taken more than 2 hours. > > Dylan Tusler > > Team Lead Data, Development & Integration > ICTS Branch > Sunshine Coast Regional Council > P 07 5420 8002 > E > *dylan.tus...@sunshinecoast.qld.gov.au*<dylan.tus...@sunshinecoast.qld.gov.au> > A Locked Bag 72, Sunshine Coast Mail Centre QLD 4560 > W *www.sunshinecoast.qld.gov.au* <http://www.sunshinecoast.qld.gov.au/> > > Please consider the sanity of others before replying to replies to replies > to this email. Sometimes it just makes more sense to pick up the phone. > > > ------------------------------ > *From:* ozmoss-boun...@ozmoss.com [mailto:ozmoss-boun...@ozmoss.com] *On > Behalf Of *Daniel Brown > *Sent:* Wednesday, 18 July 2012 14:00 > > *To:* 'ozMOSS' > *Subject:* RE: Slow crawl help > > Hi Dylan,**** > > ** ** > > How does it work with copying a large from share to server/vice versa? Is > the speed issue replicated?**** > > ** ** > > -DB**** > > ** ** > > *From:* ozmoss-boun...@ozmoss.com [mailto:ozmoss-boun...@ozmoss.com] *On > Behalf Of *Dylan Tusler > *Sent:* Wednesday, 18 July 2012 1:21 PM > *To:* 'ozMOSS' > *Subject:* RE: Slow crawl help**** > > ** ** > > After exhaustive examinations by the data centres guys here, we can't see > why the crawl is so slow.**** > > **** > > The bottleneck appears to be network performance. During an entire crawl > (about 80 hours) network traffic to and from Sharepoint server sits at > about 10Mbps. Both the server and the location of the crawled files have > NICs capable of scaling traffic up to 10Gbps. We observe traffic overnight > as server backup is performed of around 400Mbps (which is occurring on top > of the crawl.)**** > > **** > > There is no disk latency issue on our SharePoint infrastructure.**** > > **** > > Stumped.**** > > **** > > Time to call Microsoft?**** > > Dylan Tusler > Team Lead Data, Development & Integration > ICTS Branch > Sunshine Coast Regional Council > P 07 5420 8002 > E dylan.tus...@sunshinecoast.qld.gov.au > A Locked Bag 72, Sunshine Coast Mail Centre QLD 4560 > W www.sunshinecoast.qld.gov.au**** > > Please consider the sanity of others before replying to replies to replies > to this email. Sometimes it just makes more sense to pick up the phone.*** > * > > **** > > ** ** > ------------------------------ > > *From:* ozmoss-boun...@ozmoss.com [mailto:ozmoss-boun...@ozmoss.com] *On > Behalf Of *Dylan Tusler > *Sent:* Friday, 13 July 2012 13:31 > *To:* 'ozMOSS' > *Subject:* RE: Slow crawl help**** > > I know we're crawling a DFS, with the actual content on a variety of > servers. The majority of it is on a single SAN though.**** > > **** > > I'm trying to see if there are any obvious slowdowns, but having trouble > locating the logging data. When I go to > http://[ourserver]:[centraladminport]/_admin/search/logviewer.aspx?appid={26cbdb93-9472-4690-8e5f-409e72d802a9} > I get a list of crawled items, but it is in a daft 50-to-a-page format that > is difficult to do any analysis with.**** > > **** > > Is there another place I can obtain this info that is more queryable?**** > > **** > > Dylan Tusler > Team Lead Data, Development & Integration > ICTS Branch > Sunshine Coast Regional Council > P 07 5420 8002 > E dylan.tus...@sunshinecoast.qld.gov.au > A Locked Bag 72, Sunshine Coast Mail Centre QLD 4560 > W www.sunshinecoast.qld.gov.au**** > > Please consider the sanity of others before replying to replies to replies > to this email. Sometimes it just makes more sense to pick up the phone.*** > * > > **** > > ** ** > ------------------------------ > > *From:* ozmoss-boun...@ozmoss.com [mailto:ozmoss-boun...@ozmoss.com] *On > Behalf Of *Chris Walsh > *Sent:* Friday, 13 July 2012 13:22 > *To:* ozMOSS > *Subject:* RE: Slow crawl help**** > > Are they all on the same subnet? Are there network QoS policies in place > that are affecting the search server from crawling that share? What > storage infrastructure is hosting the 40GB of files?**** > > ** ** > > Seems rather odd though, we’re crawling 400+GB in under 8 hours for a full > crawl so something is amiss with your config.**** > > ** ** > > *From:* ozmoss-boun...@ozmoss.com [mailto:ozmoss-boun...@ozmoss.com] *On > Behalf Of *Dylan Tusler > *Sent:* Friday, 13 July 2012 1:16 PM > *To:* 'ozMOSS' > *Subject:* RE: Slow crawl help**** > > ** ** > > Thanks Sezai.**** > > **** > > No crawler impact rules here.**** > > **** > > We've only crawled this share once, so it has always taken this long, I > guess. I also suspect latency/bandwidth issues, as we seem to have network > traffic moving at about 100Mbps onto that server.**** > > **** > > (I've checked a bunch of other metrics out in relation to RAM, CPU, and > even the OSS performance monitor stuff, and it all seems normal.)**** > > Dylan Tusler > Team Lead Data, Development & Integration > ICTS Branch > Sunshine Coast Regional Council > P 07 5420 8002 > E dylan.tus...@sunshinecoast.qld.gov.au > A Locked Bag 72, Sunshine Coast Mail Centre QLD 4560 > W www.sunshinecoast.qld.gov.au**** > > Please consider the sanity of others before replying to replies to replies > to this email. Sometimes it just makes more sense to pick up the phone.*** > * > > **** > > ** ** > ------------------------------ > > *From:* ozmoss-boun...@ozmoss.com [mailto:ozmoss-boun...@ozmoss.com] *On > Behalf Of *Sezai Komur > *Sent:* Friday, 13 July 2012 12:17 > *To:* ozMOSS > *Subject:* Re: Slow crawl help**** > > Has indexing that specific file share ever completed any faster? **** > > It could be the network link (latency & bandwidth?) between the SharePoint > server doing the crawling and the file share?**** > > **** > > Another thing to check is if there is a *Crawler Impact Rule* configured, > review this in the Search Service Application, ensure there aren't any > configured for that specific file share. **** > > **** > > You can configure a max limit to the number of simultaneous requests, or > set a specific time in seconds for the crawler to wait between requesting > each document. This is handy to prevent a content source from being > saturated by your crawler. It does take the crawler longer to complete > crawling if these are configured.**** > > **** > > I reckon it's a network latency & bandwidth issue :)**** > > **** > > Sezai.**** > > > **** > > On Fri, Jul 13, 2012 at 9:55 AM, Wes MacDonald <wmacdon...@like10.com> > wrote:**** > > Hi,**** > > **** > > The first thing I would suspect is Antivirus software which can really > slow things up.**** > > **** > > Wes**** > > **** > > *From:* ozmoss-boun...@ozmoss.com [mailto:ozmoss-boun...@ozmoss.com] *On > Behalf Of *Dylan Tusler > *Sent:* July-12-12 7:16 PM > *To:* 'ozMOSS' > *Subject:* Slow crawl help**** > > **** > > Sharepoint 2010**** > > **** > > Someone here recently switched on a file share crawl on a network share, > and it is running really slowly.**** > > **** > > (Last incremental crawl took about 140 hours to crawl about 40GB of files.) > **** > > **** > > I've looked around a bit, but having trouble working out how to determine > what the bottleneck in the process might be.**** > > **** > > Another incremental crawl is running right now, and has been going for > over 40 hours.**** > > **** > > Any suggestions what I should be looking at?**** > > **** > > Cheers,**** > > **** > > Dylan Tusler > Team Lead Data, Development & Integration > ICTS Branch > Sunshine Coast Regional Council > P 07 5420 8002 > E dylan.tus...@sunshinecoast.qld.gov.au > A Locked Bag 72, Sunshine Coast Mail Centre QLD 4560 > W www.sunshinecoast.qld.gov.au**** > > Please consider the sanity of others before replying to replies to replies > to this email. Sometimes it just makes more sense to pick up the phone.*** > * > > > > __ __ > To find out more about the Sunshine Coast Council, visit your local office > at Caloundra, Maroochydore, Nambour or Tewantin or visit us online at > www.sunshinecoast.qld.gov.au. If correspondence includes personal > information, please refer to Council's Privacy > Policy****<http://www.sunshinecoast.qld.gov.au/> > > This email and any attachments are confidential and only for the use of > the addressee. If you have received this email in error you are requested > to notify the sender by return email or contact council on 07 5475 7272, > and are prohibited from forwarding, printing, copying or using it in > anyway, in whole or part. Please note that some council staff utilise > mobile devices, which may result in information being transmitted overseas > prior to delivery of any communication to the device. In sending an email > to Council you are agreeing that the content of your email may be > transmitted overseas. > Any views expressed in this email are the author's, except where the email > makes it clear otherwise. The unauthorised publication of an email and any > attachments generated for the official functions of council is strictly > prohibited. Please note that council is subject to the Right to Information > Act 2009 (Qld) and Information Privacy Act 2009 > (Qld).****<http://www.sunshinecoast.qld.gov.au/> > > ** ** > > > _______________________________________________ > ozmoss mailing list > ozmoss@ozmoss.com > http://prdlxvm0001.codify.net/mailman/listinfo/ozmoss<http://www.sunshinecoast.qld.gov.au/> > **** > > _______________________________________________ > ozmoss mailing list > ozmoss@ozmoss.com > http://prdlxvm0001.codify.net/mailman/listinfo/ozmoss > >
_______________________________________________ ozmoss mailing list ozmoss@ozmoss.com http://prdlxvm0001.codify.net/mailman/listinfo/ozmoss
