RE: Using curl comparing with using WebService::Solr

[Francis Yakin]

I also commit too many I guess, since we have 1000 folders, so each loop will 
executed the load and commit.
So 1000 loops with 1000 commits. I think it will be help if I only commit once 
after the 1000 loops completed.

Any inputs?

Thhanks

Francis


-----Original Message-----
From: Francis Yakin [mailto:fya...@liquid.com]
Sent: Thursday, July 09, 2009 11:13 PM
To: 'solr-user@lucene.apache.org'; 'noble.p...@gmail.com'
Subject: RE: Using curl comparing with using WebService::Solr

Yes, the xml files are in complete add format.

This is my code:

#!/usr/bin/perl


   if (($#ARGV + 1) <= 0 ) {
        print "Usage: perl prod.pl <dir> \n\n";
        exit(1);
   }


###### -- CHANGE accordingly
   $timeout = 300;
   $topdir = "/opt/Test/xml-file/";
   #$topdir = "/opt/Test/";
   $dir = $topdir . $ARGV[0];
   $commit_dir = "/opt/commit";

#####


   $curl="/usr/bin/curl";
   print "Loading xml files in $dir in progress \n";
   opendir(BIN, $dir) or die "Can't open $dir: $!";
   $commitCmd = '(cd ' . $commit_dir . '; ' . '/usr/bin/curl 
http://localhost:7001/solr/update --data-binary @commit.txt -H 
\'Content-type:text/plain; charset=utf-8\')';



   while( defined ($file = readdir BIN) ) {

   next if $file =~ /^\./;
   $insertCmd = '(cd ' . $dir   . '; ' . '/usr/bin/curl 
http://localhost:7001/solr/update --data-binary @' .  $file . ' -H 
\'Content-type:text/plain; charset=utf-8\')';

   system($insertCmd);


  }

  system($commitCmd);
closedir(BIN);

Thanks

Francis

-----Original Message-----
From: noble.p...@gmail.com [mailto:noble.p...@gmail.com] On Behalf Of Noble 
Paul ??????? ??????
Sent: Thursday, July 09, 2009 10:40 PM
To: solr-user@lucene.apache.org
Subject: Re: Using curl comparing with using WebService::Solr

are these xml files in the solr add xml format?

When you post using curl, I guess it opens as many http connections as
there are files. if you can write a small program to post all these
files in one request, you should be able to get better perf.

the following can be the pseudo-code

open connection
write "<root>"
for each file
  write filecontent
write "</root>"
close connection




On Fri, Jul 10, 2009 at 10:23 AM, Francis Yakin<fya...@liquid.com> wrote:
>
> I have about 1000 folders, each folder consist 2581 xml files. Total of xml 
> files is ~ 2.6 millions
>
> I developed perl script, inside my script it's executed this cmd:
>
>  curl http://localhost:7001/solr/update --data-binary "@0039000.xml" -H 
> 'Content-type:text/plain; charset=utf-8'
>
> It tooks me about 4 1/2 hrs to load and commit.
>
> I would like to know the advantages using curl to posting/add/update the xml 
> files to solr comparing with using WebService::Solr module?
>
> Is using WebService::Solr faster?
>
> The XML files are local on the Solr Master box, so I posting it locally( not 
> using wan or lan).
>
> Any input will be much appreciated.
>
> Thanks
>
> Francis
>
>
>



--
-----------------------------------------------------
Noble Paul | Principal Engineer| AOL | http://aol.com