At present, we're trying this out on a single server. However, the CPU and RAM are nowhere near maxed out during a crawl. I've been through everything I can think of, and all I can see is the low network traffic. But not sure if it is cause or a symptom.
I'll try turning off indexing on the server drive. Can't see anywhere else to go at the moment. Dylan Tusler Team Lead Data, Development & Integration ICTS Branch Sunshine Coast Regional Council P 07 5420 8002 E dylan.tus...@sunshinecoast.qld.gov.au<mailto:dylan.tus...@sunshinecoast.qld.gov.au> A Locked Bag 72, Sunshine Coast Mail Centre QLD 4560 W www.sunshinecoast.qld.gov.au<http://www.sunshinecoast.qld.gov.au/> Please consider the sanity of others before replying to replies to replies to this email. Sometimes it just makes more sense to pick up the phone. ________________________________ From: ozmoss-boun...@ozmoss.com [mailto:ozmoss-boun...@ozmoss.com] On Behalf Of Sezai Komur Sent: Wednesday, 18 July 2012 14:57 To: ozMOSS Subject: Re: Slow crawl help What's your farm topology like? Is crawling happening from a specific application server, multiple application servers or is it crawling from an 'all-in-one' single SharePoint server that also serves user web requests? This could potentially be due to the fact that the server is taking a bit of time for crawling and indexing each individual document, which then impacts the rate it can crawl at. What's CPU, RAM, Disk and Network activity like on the server during the crawling? Apart from disabling Antivirus, you can also try disabling the Windows Search indexer, right click the disk drive in Windows Explorer and select properties, untick the "Allow files on this drive to have contents indexed in addition to file properties" although this probably won't solve your specific issue... Sezai. On Wed, Jul 18, 2012 at 12:22 PM, Dylan Tusler <dylan.tus...@sunshinecoast.qld.gov.au<mailto:dylan.tus...@sunshinecoast.qld.gov.au>> wrote: I just tried copying one large file, and got about 600Mbps transfer speed. I also tried copying about 11Gb of miscellaneous files (about 3700 of them) which got a speed of about 400Mbps. At the rate our crawl was running, that would have taken more than 2 hours. Dylan Tusler Team Lead Data, Development & Integration ICTS Branch Sunshine Coast Regional Council P 07 5420 8002 E dylan.tus...@sunshinecoast.qld.gov.au<mailto:dylan.tus...@sunshinecoast.qld.gov.au> A Locked Bag 72, Sunshine Coast Mail Centre QLD 4560 W www.sunshinecoast.qld.gov.au<http://www.sunshinecoast.qld.gov.au/> Please consider the sanity of others before replying to replies to replies to this email. Sometimes it just makes more sense to pick up the phone. ________________________________ From: ozmoss-boun...@ozmoss.com<mailto:ozmoss-boun...@ozmoss.com> [mailto:ozmoss-boun...@ozmoss.com<mailto:ozmoss-boun...@ozmoss.com>] On Behalf Of Daniel Brown Sent: Wednesday, 18 July 2012 14:00 To: 'ozMOSS' Subject: RE: Slow crawl help Hi Dylan, How does it work with copying a large from share to server/vice versa? Is the speed issue replicated? -DB From: ozmoss-boun...@ozmoss.com<mailto:ozmoss-boun...@ozmoss.com> [mailto:ozmoss-boun...@ozmoss.com<mailto:ozmoss-boun...@ozmoss.com>] On Behalf Of Dylan Tusler Sent: Wednesday, 18 July 2012 1:21 PM To: 'ozMOSS' Subject: RE: Slow crawl help After exhaustive examinations by the data centres guys here, we can't see why the crawl is so slow. The bottleneck appears to be network performance. During an entire crawl (about 80 hours) network traffic to and from Sharepoint server sits at about 10Mbps. Both the server and the location of the crawled files have NICs capable of scaling traffic up to 10Gbps. We observe traffic overnight as server backup is performed of around 400Mbps (which is occurring on top of the crawl.) There is no disk latency issue on our SharePoint infrastructure. Stumped. Time to call Microsoft? Dylan Tusler Team Lead Data, Development & Integration ICTS Branch Sunshine Coast Regional Council P 07 5420 8002 E dylan.tus...@sunshinecoast.qld.gov.au<mailto:dylan.tus...@sunshinecoast.qld.gov.au> A Locked Bag 72, Sunshine Coast Mail Centre QLD 4560 W www.sunshinecoast.qld.gov.au<http://www.sunshinecoast.qld.gov.au/> Please consider the sanity of others before replying to replies to replies to this email. Sometimes it just makes more sense to pick up the phone. ________________________________ From: ozmoss-boun...@ozmoss.com<mailto:ozmoss-boun...@ozmoss.com> [mailto:ozmoss-boun...@ozmoss.com]<mailto:[mailto:ozmoss-boun...@ozmoss.com]> On Behalf Of Dylan Tusler Sent: Friday, 13 July 2012 13:31 To: 'ozMOSS' Subject: RE: Slow crawl help I know we're crawling a DFS, with the actual content on a variety of servers. The majority of it is on a single SAN though. I'm trying to see if there are any obvious slowdowns, but having trouble locating the logging data. When I go to http://[ourserver]:[centraladminport]/_admin/search/logviewer.aspx?appid={26cbdb93-9472-4690-8e5f-409e72d802a9} I get a list of crawled items, but it is in a daft 50-to-a-page format that is difficult to do any analysis with. Is there another place I can obtain this info that is more queryable? Dylan Tusler Team Lead Data, Development & Integration ICTS Branch Sunshine Coast Regional Council P 07 5420 8002 E dylan.tus...@sunshinecoast.qld.gov.au<mailto:dylan.tus...@sunshinecoast.qld.gov.au> A Locked Bag 72, Sunshine Coast Mail Centre QLD 4560 W www.sunshinecoast.qld.gov.au<http://www.sunshinecoast.qld.gov.au/> Please consider the sanity of others before replying to replies to replies to this email. Sometimes it just makes more sense to pick up the phone. ________________________________ From: ozmoss-boun...@ozmoss.com<mailto:ozmoss-boun...@ozmoss.com> [mailto:ozmoss-boun...@ozmoss.com]<mailto:[mailto:ozmoss-boun...@ozmoss.com]> On Behalf Of Chris Walsh Sent: Friday, 13 July 2012 13:22 To: ozMOSS Subject: RE: Slow crawl help Are they all on the same subnet? Are there network QoS policies in place that are affecting the search server from crawling that share? What storage infrastructure is hosting the 40GB of files? Seems rather odd though, we're crawling 400+GB in under 8 hours for a full crawl so something is amiss with your config. From: ozmoss-boun...@ozmoss.com<mailto:ozmoss-boun...@ozmoss.com> [mailto:ozmoss-boun...@ozmoss.com]<mailto:[mailto:ozmoss-boun...@ozmoss.com]> On Behalf Of Dylan Tusler Sent: Friday, 13 July 2012 1:16 PM To: 'ozMOSS' Subject: RE: Slow crawl help Thanks Sezai. No crawler impact rules here. We've only crawled this share once, so it has always taken this long, I guess. I also suspect latency/bandwidth issues, as we seem to have network traffic moving at about 100Mbps onto that server. (I've checked a bunch of other metrics out in relation to RAM, CPU, and even the OSS performance monitor stuff, and it all seems normal.) Dylan Tusler Team Lead Data, Development & Integration ICTS Branch Sunshine Coast Regional Council P 07 5420 8002 E dylan.tus...@sunshinecoast.qld.gov.au<mailto:dylan.tus...@sunshinecoast.qld.gov.au> A Locked Bag 72, Sunshine Coast Mail Centre QLD 4560 W www.sunshinecoast.qld.gov.au<http://www.sunshinecoast.qld.gov.au/> Please consider the sanity of others before replying to replies to replies to this email. Sometimes it just makes more sense to pick up the phone. ________________________________ From: ozmoss-boun...@ozmoss.com<mailto:ozmoss-boun...@ozmoss.com> [mailto:ozmoss-boun...@ozmoss.com]<mailto:[mailto:ozmoss-boun...@ozmoss.com]> On Behalf Of Sezai Komur Sent: Friday, 13 July 2012 12:17 To: ozMOSS Subject: Re: Slow crawl help Has indexing that specific file share ever completed any faster? It could be the network link (latency & bandwidth?) between the SharePoint server doing the crawling and the file share? Another thing to check is if there is a Crawler Impact Rule configured, review this in the Search Service Application, ensure there aren't any configured for that specific file share. You can configure a max limit to the number of simultaneous requests, or set a specific time in seconds for the crawler to wait between requesting each document. This is handy to prevent a content source from being saturated by your crawler. It does take the crawler longer to complete crawling if these are configured. I reckon it's a network latency & bandwidth issue :) Sezai. On Fri, Jul 13, 2012 at 9:55 AM, Wes MacDonald <wmacdon...@like10.com<mailto:wmacdon...@like10.com>> wrote: Hi, The first thing I would suspect is Antivirus software which can really slow things up. Wes From: ozmoss-boun...@ozmoss.com<mailto:ozmoss-boun...@ozmoss.com> [mailto:ozmoss-boun...@ozmoss.com<mailto:ozmoss-boun...@ozmoss.com>] On Behalf Of Dylan Tusler Sent: July-12-12 7:16 PM To: 'ozMOSS' Subject: Slow crawl help Sharepoint 2010 Someone here recently switched on a file share crawl on a network share, and it is running really slowly. (Last incremental crawl took about 140 hours to crawl about 40GB of files.) I've looked around a bit, but having trouble working out how to determine what the bottleneck in the process might be. Another incremental crawl is running right now, and has been going for over 40 hours. Any suggestions what I should be looking at? Cheers, Dylan Tusler Team Lead Data, Development & Integration ICTS Branch Sunshine Coast Regional Council P 07 5420 8002 E dylan.tus...@sunshinecoast.qld.gov.au<mailto:dylan.tus...@sunshinecoast.qld.gov.au> A Locked Bag 72, Sunshine Coast Mail Centre QLD 4560 W www.sunshinecoast.qld.gov.au<http://www.sunshinecoast.qld.gov.au/> Please consider the sanity of others before replying to replies to replies to this email. Sometimes it just makes more sense to pick up the phone. __ __ To find out more about the Sunshine Coast Council, visit your local office at Caloundra, Maroochydore, Nambour or Tewantin or visit us online at www.sunshinecoast.qld.gov.au. If correspondence includes personal information, please refer to Council's Privacy Policy<http://www.sunshinecoast.qld.gov.au/> This email and any attachments are confidential and only for the use of the addressee. If you have received this email in error you are requested to notify the sender by return email or contact council on 07 5475 7272, and are prohibited from forwarding, printing, copying or using it in anyway, in whole or part. Please note that some council staff utilise mobile devices, which may result in information being transmitted overseas prior to delivery of any communication to the device. In sending an email to Council you are agreeing that the content of your email may be transmitted overseas. Any views expressed in this email are the author's, except where the email makes it clear otherwise. The unauthorised publication of an email and any attachments generated for the official functions of council is strictly prohibited. Please note that council is subject to the Right to Information Act 2009 (Qld) and Information Privacy Act 2009 (Qld).<http://www.sunshinecoast.qld.gov.au/> _______________________________________________ ozmoss mailing list ozmoss@ozmoss.com http://prdlxvm0001.codify.net/mailman/listinfo/ozmoss<http://www.sunshinecoast.qld.gov.au/> _______________________________________________ ozmoss mailing list ozmoss@ozmoss.com<mailto:ozmoss@ozmoss.com> http://prdlxvm0001.codify.net/mailman/listinfo/ozmoss ------------------------------------------------------------- To find out more about the Sunshine Coast Regional Council, visit your local office at Caloundra, Maroochydore, Nambour or Tewantin or visit us online at www.sunshinecoast.qld.gov.au. If correspondence includes personal information, please refer to Council's Privacy Policy at http://www.sunshinecoast.qld.gov.au . This email and any attachments are confidential and only for the use of the addressee. If you have received this email in error you are requested to notify the sender by return email or contact council on 07 5475 7272, and are prohibited from forwarding, printing, copying or using it in anyway, in whole or part. Please note that some council staff utilise mobile devices, which may result in information being transmitted overseas prior to delivery of any communication to the device. In sending an email to Council you are agreeing that the content of your email may be transmitted overseas. Any views expressed in this email are the author's, except where the email makes it clear otherwise. The unauthorised publication of an email and any attachments generated for the official functions of council is strictly prohibited. Please note that council is subject to the Right to Information Act 2009 (Qld) and Information Privacy Act 2009 (Qld).
_______________________________________________ ozmoss mailing list ozmoss@ozmoss.com http://prdlxvm0001.codify.net/mailman/listinfo/ozmoss
