i'm having the same problem. i emailed the list and
posted on the message board but still no response.
i've read about others having the same problem. so you
aren't alone here.
--- Ernesto Vargas [EMAIL PROTECTED] wrote:
I am having problems running the indexer for more
that 1 or 2 hours. It
i'm indexing my state government's web sites. however,
there are some sites that indexer just stalls on. See
the last line below:
...
Indexer[22838]: [1]
http://www.doer.state.mn.us/lr-mlea/artcl-30.htm
Indexer[22838]: [1]
http://www.doer.state.mn.us/lr-mlea/artcl-31.htm
Indexer[22838]: [1]
Caffeinate The World wrote:
since we can tell indexer which tag to index, will we
be able to do this for categories as well?
It is not possible now. We've wrote it to our TODO.
__
If you want to unsubscribe send "unsubscribe udmsearch"
to [EMAIL PROTECTED]
On Fri, 03 Nov 2000, Alexander Barkov wrote:
Which database do you use?
Sorry - I'm using mysql and crc-multi mode.
David Robley wrote:
It's Friday afternoon so something has to go wrong with mnogosearch :-)
Version 3.1.8 on Linux and the problem is that indexer -a keeps
Jacob Friis Larsen wrote:
If I run indexer with -k (skip locking (affects for MySQL and PostgreSQL));
Will that affect my search.cgi speed ?
No.
My guess is that indexer should lock only if there are more than one indexer
running.
Yes. If you are using one indexer, you may specify -k
It's Friday afternoon so something has to go wrong with mnogosearch :-)
Version 3.1.8 on Linux and the problem is that indexer -a keeps starting
new processes, so that once all urls have been indexed, it keeps running
through the site and adding words from urls that have already been indexed.
Hi Alexander,
the same happens when I use the command you have sent me.
Namely the indexer just quits without doing anything.
The ouput is:
sincity:ebastuz /usr/local/udmsearch-3.0.23/sbin/indexer -v6 -am
/usr/local/udmsearch-3.0.23/etc/indexer.conf
Indexer[9964]: indexer
Can you post your indexer.conf ?
Emre Bastuz wrote:
Hi Alexander,
the same happens when I use the command you have sent me.
Namely the indexer just quits without doing anything.
The ouput is:
sincity:ebastuz /usr/local/udmsearch-3.0.23/sbin/indexer -v6 -am
Try the following:
1. Open sql.c in your editor
2. Add a line "#define DEBUG_SQL 1"
3. recompile and reinstall indexer
4. Run it and check it's STDERR output. You'll see the query being
executed and how much time it takes.
5. Either try to execute the same query in "mysql" tool or just
send a
I just installed version 3.0.23 a few days ago on
my Solaris system, and I have to say it is running very strangely. Indexer had a
catastrophic failure a few days ago which shutdown mysql entirely (with a "too
many connections, error 1040' error), which necessitated my running iamchk on
all
You have to add nont linked pages manualy.
Using either Server command or -i key for indexer.
"Alain Tésio" wrote:
There\'s also other pages placed in /Archive,
which are NOT linked from 1st, main site in
any way (so can\'t be reached by following links).
- What I need
Hello, guys
I am totally amazed with Indexer and his configuration.
From 10 days I am playing with it and can't get it working.
Help me if you are able.
(I also tryed to post messages to WWW-board, but got page ("no spamming /
you cannot post") when
using Opera 4, and nothing (and no post) using
There\'s also other pages placed in /Archive,
which are NOT linked from 1st, main site in
any way (so can\'t be reached by following links).
- What I need
Every allowed file (.htm .html .txt) in *ALL*
directories (including NOT linked from the main
site) to be indexed.
Hello,
I started indexer from UdmSearch 3.0.2[01] few weeks ago to index few
domains and now I give (with PHP front-end) some queries. And I found some
oddities:
When I give keyword say "hany" I receive results like this:
1. Mega ?Loman: intranet: M1 [1]
Mega ?Loman - intranet: M1
On Thu, 10 Aug 2000, Alexander Barkov wrote:
Hello!
Peter Hanecak wrote:
Hello again,
Indexer[5553]: [1] Deleting URL
I tried to slightly trace the problem of indexer freezing in "Deleting
URL" and I come to this:
Indexer[22445]: [1] Deleting URL
calls ...
Hi,
On Thu, 10 Aug 2000, Alexander Barkov wrote:
How many records in your "robots" table?
Now i have 276 records. I'm running with patch I did send you
(udmsearch-3.0.20-indexer-locking-fix.patch).
The problem lies in that lock I'm sure - I put debug message right before
and right after the
"David J. M. Karlsen" wrote:
indexer process had grown to 97M memusage since 2AM until now... is
this normal? Should I put on any tracing info for you guys?
using postgres v7 and developversion of indexer (3.1.3).
Hello!
There is a bug in PgSQL code. We'll try to fix this soon.
--
Hello again,
Indexer[5553]: [1] Deleting URL
I tried to slightly trace the problem of indexer freezing in "Deleting
URL" and I come to this:
Indexer[22445]: [1] Deleting URL
calls ...
UdmLoadRobots()
which calls ...
UdmFreeRobots()
which stops at trying to obtain lock
Hello again,
Indexer[5553]: [1] Deleting URL
I tried to slightly trace the problem of indexer freezing in "Deleting
URL" and I come to this:
Indexer[22445]: [1] Deleting URL
calls ...
UdmLoadRobots()
obtains lock and then calls ...
UdmFreeRobots()
which stops at trying
Hello again,
Indexer[5553]: [1] Deleting URL
I tried to slightly trace the problem of indexer freezing in "Deleting
URL" and I come to this:
Indexer[22445]: [1] Deleting URL
calls ...
UdmLoadRobots()
obtains lock and then calls ...
UdmFreeRobots()
which stops at trying
At 15:23 4.8.00, David J. M. Karlsen wrote:
indexer process had grown to 97M memusage since 2AM until now... is this
normal? Should I put on any tracing info for you guys?
Do you have many Server entries? With over 5000 Server entries indexer used
over 50MB RAM on my machine.
Tomaz
On Wed, 2 Aug 2000, Peter Green wrote:
Linux/Alpha, UdmSearch 3.0.19. It seems to happen mostly (always?) on the
robots.txt file for my second server (members.gospelcom.net above), but not
the first one...
I got same feeling: mostly robots.txt and not the first, but second one.
Hany
--
When I started indexer with '-v 5' option and in just one process, the
last messages has been these:
Indexer[5553]: [1] http://www.banky.sk/robots.txt
I have the same problem. Everytime my indexer finds a robots.txt, it
hangs. Although, I have robots set to no in my indexer.conf.
also sprach lang:
When I started indexer with '-v 5' option and in just one process, the
last messages has been these:
Indexer[5553]: [1] http://www.banky.sk/robots.txt
I have the same problem. Everytime my indexer finds a robots.txt, it
hangs. Although, I have robots set to no in
Hello!
Igor is currently learning this problem.
Daniel Hanks wrote:
Hello there,
I am currently using udmsearch-3.1.3-pre4 running on Linux against
Oracle 8.1.6. Oracle is running on Sparc hardware running Solaris.
The indexer inserted a huge list of URLs just fine, but when it comes
Hello there,
I am currently using udmsearch-3.1.3-pre4 running on Linux against
Oracle 8.1.6. Oracle is running on Sparc hardware running Solaris.
The indexer inserted a huge list of URLs just fine, but when it comes time
to actually start indexing things it will run for several urls, and then
Hello,
I am running the latest development release v3.1.2 and when I run the
indexer I get the following error:
Error #1054: Unknown column 'crc32' in 'field list'
Any idea?
--
/ Gary DeMontigny/ TeleSoft Systems
/ www.telesoft.mb.ca / [EMAIL PROTECTED]
__
If you want to
On Thu, Jun 22, 2000 at 05:32:51PM -0700, J C Lawrence wrote:
On Thu, 22 Jun 2000 21:38:34 +0200
Willem Brown [EMAIL PROTECTED] wrote:
Hi, I expire those pages manualy before starting the indexer. The
command is like this.
indexer -a -u %/~lists/%/index.html \
-u
Find being executed queries in MySQL log and try to run
it manually. It seems that it is a database crash and probably
you have to repair it.
Paul Stewart wrote:
When I run indexer I have had no problems until my MySQL database gets quite
large..
Now when I try to run indexer, I get
: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]On
Behalf Of Alexander Barkov
Sent: Tuesday, July 04, 2000 8:13 AM
To: Paul Stewart
Cc: [EMAIL PROTECTED]
Subject: Re: UdmSearch: Indexer Mysql Problem
Find being executed queries in MySQL log and try to run
it manually. It seems that it is a database
--- Paul Stewart [EMAIL PROTECTED] wrote:
I am not a MySQL expert by any means but can usually find my way
around...
How would you suggest I do this logging... is there a command line
that will
debug to a log file in MySQL?
Start the server with the option "--log-update"
Alain
Hi,
My process is not that refined yet. Whenever I do this it will go an re-check all
the link in the index pages. It doesn't seem to re-index them unless they are
older that "Period" which in my case is still the default.
Although it's not very efficient, it does ensure that I get all of the
UdmSearch 3.0.18, Linux 2.2.16, Oracle 8.0.5 client libs, indexing in
multi mode. indexer -S segfaults:
(gdb) run -S
Starting program: /usr/local/udmsearch/sbin/indexer -S
UdmSearch statistics
StatusExpired Total
-
200 0
What are the ways to speed indexing?
My site (100K pages) is now taking the better part of a day to
index, despite the fact that most content hasn't changed on each
re-index.
I've built the pthreads based indexer and have been running it with
various numbers of threads ranging from 1 to 150
On Thu, 22 Jun 2000, J C Lawrence wrote:
On Thu, 22 Jun 2000 15:46:20 -0300 (ADT)
The Hermit Hacker [EMAIL PROTECTED] wrote:
Try setting your Period higher and using the -n option to restrict
the number of pages it does in an invocation ...
for instance, set Period to 1week, and -n
-BEGIN PGP SIGNED MESSAGE-
What about using a different config file for those sections that need to
be index more often? Leave the defauft config with longer period.
Tin Le
-
http://tin.le.org
Internet Security and Firewall Consulting
Tin Le - [EMAIL PROTECTED]
On Thu, 22 Jun
Or, unless I'm missing something, there's an -f option to the indexer that
you can use to tell the indexer to read a specific list of URLs from a
file. So if you need to reindex www.foo.com/apple.html and
www.bar.com/orange.html every time the indexer starts up, you could just:
echo
that one I'm interested in as well ... I've been suffering with things
so
far, as I didn't/don't think there is currently a way of treating
'subpages' seperately if the toplevel page is already being indexed ...
Someway of doing:
Period 604800
Server http://www.postgresql.org/%
I'm looking at making some changes to the indexer so that it would
pay attention to (say) specific HTML comments and NOT index the data
between them.
Why?
Many pages contain data which is generic to the site or at least
that sub-section of it, and therefore is unworthy of being indexed.
- Original Message -
From: J C Lawrence [EMAIL PROTECTED]
To: [EMAIL PROTECTED]
Sent: Friday, May 05, 2000 12:36 AM
Subject: UdmSearch: Indexer indexing only defined sections of a page?
I'm looking at making some changes to the indexer so that it would
pay attention to (say
hi!
While using postgres (6.5.3 or 7.0beta5) as backend,
'indexer' (udmsearch 3.0.10) eats memory like popcorn...
after 2 hours running 'top' shows about 190MB used by
indexer.
If I switch back (recompile) to mysql, everything works
perfectly.
Is that a kind of bug, and if, is there a
On Wed, 29 Dec 1999, ---%serek--- wrote:
why? why C is not enough?
perl and php are script languages to develop portable and rather small
applications...
Because indexer isn't of much complexity anyway. If you remove all
supplemental modules implemented in CPAN (md5, parser, http
crawls the web, grabbing data
Yes, it is WWW::Robot module from CPAN.
indexer = component which indexes the grabbed data
These two 'can' be in one program, such as udmsearch 'indexer', but does
not have to be. Division of labor will allow you to scale, i.e. run
multiple crawlers on multiple
On Wed, 29 Dec 1999, Juan Ignacio PÊrez SacristÂn wrote:
php may be executed as an standalone program, like perl
Only if it is compiled as cgi program. If it is compiled in as Apache
module, you just don't have any PHP executable to execute :) Only httpd.
mike
__
If you want
Hi!
It is not implemented yet.
But I think we'll add this feature in next version.
Charlie Hornberger wrote:
-u flag does not mean "Add this URL in the database".
It means that indexer will work with those URLs that are
already in database and do match SQL LIKE wildcards given
in
I am using the EDU-SEARCH indexer.conf, they only change I made was to change
the MySQLUser, MySQLPass and the Server entries.
If I index my site with "FollowOutside no" I only get 118 entries in the dict
table when I index 10,417 html documents.
If I index my site with "FollowOutside yes" I
46 matches
Mail list logo
