Re: which HTML parser is better?

2005-02-04 Thread Ian Soboroff

Oops.  It's in the Google cache and also the Internet Archive Wayback
machine.  I'll drop the original author a note to let him know that
his links are stale.

http://web.archive.org/web/20040208014740/http://www.ltg.ed.ac.uk/~richard/ftp-area/html-parser/

Ian

"Karl Koch" <[EMAIL PROTECTED]> writes:

> The link does not work.
>
>> 
>> One which we've been using can be found at:
>> http://www.ltg.ed.ac.uk/~richard/ftp-area/html-parser/
>> 
>> We absolutely need to be able to recover gracefully from malformed
>> HTML and/or SGML.  Most of the nicer SAX/DOM/TLA parsers out there
>> failed this criterion when we started our effort.  The above one is
>> kind of SAX-y but doesn't fall over at the sight of a real web page
>> ;-)



-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



Re: which HTML parser is better?

2005-02-04 Thread Karl Koch
The link does not work.

> 
> One which we've been using can be found at:
> http://www.ltg.ed.ac.uk/~richard/ftp-area/html-parser/
> 
> We absolutely need to be able to recover gracefully from malformed
> HTML and/or SGML.  Most of the nicer SAX/DOM/TLA parsers out there
> failed this criterion when we started our effort.  The above one is
> kind of SAX-y but doesn't fall over at the sight of a real web page
> ;-)
> 
> Ian
> 
> 
> -
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
> 

-- 
DSL Komplett von GMX +++ Supergünstig und stressfrei einsteigen!
AKTION "Kein Einrichtungspreis" nutzen: http://www.gmx.net/de/go/dsl

-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



Re: which HTML parser is better?

2005-02-03 Thread Ian Soboroff

One which we've been using can be found at:
http://www.ltg.ed.ac.uk/~richard/ftp-area/html-parser/

We absolutely need to be able to recover gracefully from malformed
HTML and/or SGML.  Most of the nicer SAX/DOM/TLA parsers out there
failed this criterion when we started our effort.  The above one is
kind of SAX-y but doesn't fall over at the sight of a real web page
;-)

Ian


-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



Re: which HTML parser is better?

2005-02-03 Thread aurora
For all parser suggestion I think there is one important attribute. Some  
parsers returns data provide that the input HTML is sensible. Some parsers  
is designed to be most flexible as tolerant as it can be. If the input is  
clean and controlled the former class is sufficient. Even some regular  
expression may be sufficient. (I that's the original poster wants). If you  
are building a web crawler you need something really tolerant.

Once I have prototyped a nice and fast parser. Later I have to abandon it  
because it failed to parse about 15% documents (problem handling nested  
quotes like onclick="alert('hi')").

No one has yet mentioned using ParserDelegator and ParserCallback that  
are part of HTMLEditorKit in Swing.  I have been successfully using  
these classes to parse out the text of an HTML file.  You just need to  
extend HTMLEditorKit.ParserCallback and override the various methods  
that are called when different tags are encountered.

On Feb 1, 2005, at 3:14 AM, Jingkang Zhang wrote:
Three HTML parsers(Lucene web application
demo,CyberNeko HTML Parser,JTidy) are mentioned in
Lucene FAQ
1.3.27.Which is the best?Can it filter tags that are
auto-created by MS-word 'Save As HTML files' function?

-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]


Re: which HTML parser is better? - Thread closed

2005-02-03 Thread Karl Koch
Thank you, I will do that.

> Karl Koch wrote:
> 
> >I appologise in advance, if some of my writing here has been said before.
> >The last three answers to my question have been suggesting pattern
> matching
> >solutions and Swing. Pattern matching was introduced in Java 1.4 and
> Swing
> >is something I cannot use since I work with Java 1.1 on a PDA.
> >  
> >
> I see,
> 
> In this case you can read line by line your HTML file and then write 
> something like this:
> 
> String line;
> int startPos, endPos;
> StringBuffer text = new StringBuffer();
> while((line = reader.readLine()) != null   ){
> startPos = line.indexOf(">");
> endPos = line.indexOf("<");
> if(startPos >0 && endPos > startPos)
>   text.append(line.substring(startPos, endPos));
> }
> 
> This is just a sample code that should work if you have just one tag per 
> line in the HTML file.
> This can be a start point for you.
> 
>   Hope it helps,
> 
>  Best,
> 
>  Sergiu
> 
> >I am wondering if somebody knows a piece of simple sourcecode with low
> >requirement which is running under this tense specification.
> >
> >Thank you all,
> >Karl
> >
> >  
> >
> >>No one has yet mentioned using ParserDelegator and ParserCallback that 
> >>are part of HTMLEditorKit in Swing.  I have been successfully using 
> >>these classes to parse out the text of an HTML file.  You just need to 
> >>extend HTMLEditorKit.ParserCallback and override the various methods 
> >>that are called when different tags are encountered.
> >>
> >>
> >>On Feb 1, 2005, at 3:14 AM, Jingkang Zhang wrote:
> >>
> >>
> >>
> >>>Three HTML parsers(Lucene web application
> >>>demo,CyberNeko HTML Parser,JTidy) are mentioned in
> >>>Lucene FAQ
> >>>1.3.27.Which is the best?Can it filter tags that are
> >>>auto-created by MS-word 'Save As HTML files' function?
> >>>  
> >>>
> >>-- 
> >>Bill Tschumy
> >>Otherwise -- Austin, TX
> >>http://www.otherwise.com
> >>
> >>
> >>-
> >>To unsubscribe, e-mail: [EMAIL PROTECTED]
> >>For additional commands, e-mail: [EMAIL PROTECTED]
> >>
> >>
> >>
> >
> >  
> >
> 
> 
> -
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
> 

-- 
10 GB Mailbox, 100 FreeSMS http://www.gmx.net/de/go/topmail
+++ GMX - die erste Adresse für Mail, Message, More +++

-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



Re: which HTML parser is better?

2005-02-03 Thread Dawid Weiss
Karl,
Two things, try to experiment with both:
1) I would try to write a lexical scanner that strips HTML tags, much 
like the regular expression does. Java lexical scanner packages produce 
nice pure Java classes that seldom use any advanced API, so they should 
work on Java 1.1. They are simple state machines with states encoded in 
integers -- this should work like a charm, be fast and small.

2) Write a parser yourself. Having a regular expression it isn't that 
difficult to do... :)

D.
Karl Koch wrote:
I appologise in advance, if some of my writing here has been said before.
The last three answers to my question have been suggesting pattern matching
solutions and Swing. Pattern matching was introduced in Java 1.4 and Swing
is something I cannot use since I work with Java 1.1 on a PDA.
I am wondering if somebody knows a piece of simple sourcecode with low
requirement which is running under this tense specification.
Thank you all,
Karl

No one has yet mentioned using ParserDelegator and ParserCallback that 
are part of HTMLEditorKit in Swing.  I have been successfully using 
these classes to parse out the text of an HTML file.  You just need to 
extend HTMLEditorKit.ParserCallback and override the various methods 
that are called when different tags are encountered.

On Feb 1, 2005, at 3:14 AM, Jingkang Zhang wrote:

Three HTML parsers(Lucene web application
demo,CyberNeko HTML Parser,JTidy) are mentioned in
Lucene FAQ
1.3.27.Which is the best?Can it filter tags that are
auto-created by MS-word 'Save As HTML files' function?
--
Bill Tschumy
Otherwise -- Austin, TX
http://www.otherwise.com
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]


Re: which HTML parser is better?

2005-02-03 Thread sergiu gordea
Karl Koch wrote:
I appologise in advance, if some of my writing here has been said before.
The last three answers to my question have been suggesting pattern matching
solutions and Swing. Pattern matching was introduced in Java 1.4 and Swing
is something I cannot use since I work with Java 1.1 on a PDA.
 

I see,
In this case you can read line by line your HTML file and then write 
something like this:

String line;
int startPos, endPos;
StringBuffer text = new StringBuffer();
while((line = reader.readLine()) != null   ){
   startPos = line.indexOf(">");
   endPos = line.indexOf("<");
   if(startPos >0 && endPos > startPos)
 text.append(line.substring(startPos, endPos));
}
This is just a sample code that should work if you have just one tag per 
line in the HTML file.
This can be a start point for you.

 Hope it helps,
Best,
Sergiu
I am wondering if somebody knows a piece of simple sourcecode with low
requirement which is running under this tense specification.
Thank you all,
Karl
 

No one has yet mentioned using ParserDelegator and ParserCallback that 
are part of HTMLEditorKit in Swing.  I have been successfully using 
these classes to parse out the text of an HTML file.  You just need to 
extend HTMLEditorKit.ParserCallback and override the various methods 
that are called when different tags are encountered.

On Feb 1, 2005, at 3:14 AM, Jingkang Zhang wrote:
   

Three HTML parsers(Lucene web application
demo,CyberNeko HTML Parser,JTidy) are mentioned in
Lucene FAQ
1.3.27.Which is the best?Can it filter tags that are
auto-created by MS-word 'Save As HTML files' function?
 

--
Bill Tschumy
Otherwise -- Austin, TX
http://www.otherwise.com
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
   

 


-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]


Re: which HTML parser is better?

2005-02-03 Thread Karl Koch
I am using Java 1.1 with a Sharp Zaurus PDA. I have very limited memory
constraints. I do not think CPU performance is a big issues though. But I
have other parts in my application which use quite a lot of memory and
soemthing run short. I therefore do not look into solutions which build up
tag trees etc. More like a solution who reads a stream of HTML and
transforms it into a stream of text.

I see your point of using an external program. I am however not entirely
sure if this is available. Also it would be much simpler to have a 3-5 kB
solution in Java, perhaps encapsulated in a class which does the job without
the need for advanced libraries which need 100-200 KB on my internal
storage. 

I hope I could clarify my situation now.

Cheers,
Karl 

> Karl Koch wrote:
> 
> >Hello Sergiu,
> >
> >thank you for your help so far. I appreciate it.
> >
> >I am working with Java 1.1 which does not include regular expressions.
> >  
> >
> Why are you using Java 1.1? Are you so limited in resources?
> What operating system do you use?
> I asume that you just need to index the html files, and you need a 
> html2txt conversion.
> If  an external converter si a solution for you, you can use
> Runtime.executeCommnand(...) to run the converter that will extract the 
> information from your HTMLs
> and generate a .txt file. Then you can use a reader to index the txt.
> 
> As I told you before, the best solution depends on your constraints 
> (time, effort, hardware, performance) and requirements :)
> 
>   Best,
> 
>   Sergiu
> 
> >Your turn ;-)
> >Karl 
> >
> >  
> >
> >>Karl Koch wrote:
> >>
> >>
> >>
> >>>I am in control of the html, which means it is well formated HTML. I
> use
> >>>only HTML files which I have transformed from XML. No external HTML
> (e.g.
> >>>the web).
> >>>
> >>>Are there any very-short solutions for that?
> >>> 
> >>>
> >>>  
> >>>
> >>if you are using only correct formated HTML pages and you are in control
> >>of these pages.
> >>you can use a regular exprestion to remove the tags.
> >>
> >>something like
> >>replaceAll("<*>","");
> >>
> >>This is the ideea behind the operation. If you will search on google you
> >>will find a more robust
> >>regular expression.
> >>
> >>Using a simple regular expression will be a very cheap solution, that 
> >>can cause you a lot of problems in the future.
> >> 
> >> It's up to you to use it 
> >>
> >> Best,
> >> 
> >> Sergiu
> >>
> >>
> >>
> >>>Karl
> >>>
> >>> 
> >>>
> >>>  
> >>>
> Karl Koch wrote:
> 
>    
> 
> 
> 
> >Hi,
> >
> >yes, but the library your are using is quite big. I was thinking that
> a
> > 
> >
> >  
> >
> 5kB
>    
> 
> 
> 
> >code could actually do that. That sourceforge project is doing much
> >  
> >
> >>more
> >>
> >>
> >than that but I do not need it.
> >
> >
> > 
> >
> >  
> >
> you need just the htmlparser.jar 200k.
> ... you know ... the functionality is strongly correclated with the
> 
> 
> >>size.
> >>
> >>
>  You can use 3 lines of code with a good regular expresion to
> eliminate
> the html tags,
> but this won't give you any guarantie that the text from the bad 
> fromated html files will be
> correctly extracted...
> 
>  Best,
> 
>  Sergiu
> 
>    
> 
> 
> 
> >Karl
> >
> >
> >
> > 
> >
> >  
> >
> >>Hi Karl,
> >>
> >>I already submitted a peace of code that removes the html tags.
> >>Search for my previous answer in this thread.
> >>
> >>Best,
> >>
> >> Sergiu
> >>
> >>Karl Koch wrote:
> >>
> >>  
> >>
> >>   
> >>
> >>
> >>
> >>>Hello,
> >>>
> >>>I have  been following this thread and have another question. 
> >>>
> >>>Is there a piece of sourcecode (which is preferably very short and
> >>> 
> >>>
> >>>  
> >>>
> simple
>    
> 
> 
> 
> >>>(KISS)) which allows to remove all HTML tags from HTML content?
> HTML
> >>> 
> >>>
> >>>  
> >>>
> 3.2
>    
> 
> 
> 
> >>>would be enough...also no frames, CSS, etc. 
> >>>
> >>>I do not need to have the HTML strucutre tree or any other
> structure
> >>> 
> >>>
> >>>  
> >>>
> but
>    
> 
> 
> 
> >>>need a facility to clean up HTML into its normal underlying content
> >>>
> >>>
> >>> 
> >>>
> >>>  
> >>>
> >>before
> >>  
> >>
> >>   
> >>
> >>
> >>
> >>>indexing that content as a whole.
> >>>
> >>>Karl
> >>>
> >>>
> >>>
> >>>
> >>>
> >

Re: which HTML parser is better?

2005-02-03 Thread sergiu gordea
Karl Koch wrote:
Unfortunaltiy I am faithful ;-). Just for practical reason I want to do that
in a single class or even method called by another part in my Java
application. It should also run on Java 1.1 and it should be small and
simple. As I said before, I am in control of the HTML and it will be well
formated, because I generate it from XML using XSLT.
 

Why don't you get the data directly from  XML files?
You can use a SAX parser, ... but I think it will require java 1.3 or at 
least 1.2.2

Best,
 Sergiu
Karl
 

If you are not married to Java:
http://search.cpan.org/~kilinrax/HTML-Strip-1.04/Strip.pm
Otis
--- sergiu gordea <[EMAIL PROTECTED]> wrote:
   

Karl Koch wrote:
 

I am in control of the html, which means it is well formated HTML. I
   

use
 

only HTML files which I have transformed from XML. No external HTML
   

(e.g.
 

the web).
Are there any very-short solutions for that?
   

if you are using only correct formated HTML pages and you are in
control 
of these pages.
you can use a regular exprestion to remove the tags.

something like
replaceAll("<*>","");
This is the ideea behind the operation. If you will search on google
you 
will find a more robust
regular expression.

Using a simple regular expression will be a very cheap solution, that
can cause you a lot of problems in the future.
It's up to you to use it 
Best,
Sergiu
 

Karl

   

Karl Koch wrote:
  

 

Hi,
yes, but the library your are using is quite big. I was thinking
   

that a
 



   

5kB
  

 

code could actually do that. That sourceforge project is doing
   

much more
 

than that but I do not need it.


   

you need just the htmlparser.jar 200k.
... you know ... the functionality is strongly correclated with the
 

size.
 

You can use 3 lines of code with a good regular expresion to
 

eliminate 
 

the html tags,
but this won't give you any guarantie that the text from the bad 
fromated html files will be
correctly extracted...

Best,
Sergiu
  

 

Karl



   

Hi Karl,
I already submitted a peace of code that removes the html tags.
Search for my previous answer in this thread.
Best,
Sergiu
Karl Koch wrote:
 

  

 

Hello,
I have  been following this thread and have another question. 

Is there a piece of sourcecode (which is preferably very short
   

and
 



   

simple
  

 

(KISS)) which allows to remove all HTML tags from HTML content?
   

HTML
 



   

3.2
  

 

would be enough...also no frames, CSS, etc. 

I do not need to have the HTML strucutre tree or any other
   

structure
 



   

but
  

 

need a facility to clean up HTML into its normal underlying
   

content
 

   



   

before
 

  

 

indexing that content as a whole.
Karl

   



   

I think that depends on what you want to do.  The Lucene demo
 

parser
 

 

  

 

does
 

  

 

simple mapping of HTML files into Lucene Documents; it does not
 

give
 

  

 

you
  

 

 

  

 

a
 

  

 

parse tree for the HTML doc.  CyberNeko is an extension of
 

Xerces
 

  

 

(uses
  

 


 

  

 

the
   



   

same API; will likely become part of Xerces), and so maps an
 

HTML
 

 

  

 

document
 

  

 

into a full DOM that you can manipulate easily for a wide range
 

of
 

purposes.  I haven't used JTidy at an API level and so don't
 

know it
 

  

 

as
  

 


 

  

 

well --
   



   

based on its UI, it appears to be focused primarily on HTML
 

validation
 

 

  

 

and
 

  

 

error detection/correction.
I use CyberNeko for a range of operations on HTML documents
 

that go
 

 

  

 

beyond
 

  

 

indexing them in Lucene, and really like it.  It has been
 

robust for
 

  

 

me
  

 

 

  

 

so
 

  

 

far.
Chuck
  

 

-Original Message-
From: Jingkang Zhang [mailto:[EMAIL PROTECTED]
Sent: Tuesday, February 01, 2005 1:15 AM
To: lucene-user@jakarta.apache.org
Subject: which HTML parser is better?
Three HTML parsers(Lucene web application
demo,Cybe

Re: which HTML parser is better?

2005-02-03 Thread Karl Koch
I appologise in advance, if some of my writing here has been said before.
The last three answers to my question have been suggesting pattern matching
solutions and Swing. Pattern matching was introduced in Java 1.4 and Swing
is something I cannot use since I work with Java 1.1 on a PDA.

I am wondering if somebody knows a piece of simple sourcecode with low
requirement which is running under this tense specification.

Thank you all,
Karl

> No one has yet mentioned using ParserDelegator and ParserCallback that 
> are part of HTMLEditorKit in Swing.  I have been successfully using 
> these classes to parse out the text of an HTML file.  You just need to 
> extend HTMLEditorKit.ParserCallback and override the various methods 
> that are called when different tags are encountered.
> 
> 
> On Feb 1, 2005, at 3:14 AM, Jingkang Zhang wrote:
> 
> > Three HTML parsers(Lucene web application
> > demo,CyberNeko HTML Parser,JTidy) are mentioned in
> > Lucene FAQ
> > 1.3.27.Which is the best?Can it filter tags that are
> > auto-created by MS-word 'Save As HTML files' function?
> -- 
> Bill Tschumy
> Otherwise -- Austin, TX
> http://www.otherwise.com
> 
> 
> -
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
> 

-- 
Sparen beginnt mit GMX DSL: http://www.gmx.net/de/go/dsl

-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



Re: which HTML parser is better?

2005-02-03 Thread sergiu gordea
Karl Koch wrote:
Hello Sergiu,
thank you for your help so far. I appreciate it.
I am working with Java 1.1 which does not include regular expressions.
 

Why are you using Java 1.1? Are you so limited in resources?
What operating system do you use?
I asume that you just need to index the html files, and you need a 
html2txt conversion.
If  an external converter si a solution for you, you can use
Runtime.executeCommnand(...) to run the converter that will extract the 
information from your HTMLs
and generate a .txt file. Then you can use a reader to index the txt.

As I told you before, the best solution depends on your constraints 
(time, effort, hardware, performance) and requirements :)

 Best,
 Sergiu
Your turn ;-)
Karl 

 

Karl Koch wrote:
   

I am in control of the html, which means it is well formated HTML. I use
only HTML files which I have transformed from XML. No external HTML (e.g.
the web).
Are there any very-short solutions for that?
 

if you are using only correct formated HTML pages and you are in control 
of these pages.
you can use a regular exprestion to remove the tags.

something like
replaceAll("<*>","");
This is the ideea behind the operation. If you will search on google you 
will find a more robust
regular expression.

Using a simple regular expression will be a very cheap solution, that 
can cause you a lot of problems in the future.

It's up to you to use it 
Best,
Sergiu
   

Karl

 

Karl Koch wrote:
  

   

Hi,
yes, but the library your are using is quite big. I was thinking that a


 

5kB
  

   

code could actually do that. That sourceforge project is doing much
 

more
   

than that but I do not need it.


 

you need just the htmlparser.jar 200k.
... you know ... the functionality is strongly correclated with the
   

size.
   

You can use 3 lines of code with a good regular expresion to eliminate
the html tags,
but this won't give you any guarantie that the text from the bad 
fromated html files will be
correctly extracted...

Best,
Sergiu
  

   

Karl



 

Hi Karl,
I already submitted a peace of code that removes the html tags.
Search for my previous answer in this thread.
Best,
Sergiu
Karl Koch wrote:
 

  

   

Hello,
I have  been following this thread and have another question. 

Is there a piece of sourcecode (which is preferably very short and


 

simple
  

   

(KISS)) which allows to remove all HTML tags from HTML content? HTML


 

3.2
  

   

would be enough...also no frames, CSS, etc. 

I do not need to have the HTML strucutre tree or any other structure


 

but
  

   

need a facility to clean up HTML into its normal underlying content
   



 

before
 

  

   

indexing that content as a whole.
Karl

   



 

I think that depends on what you want to do.  The Lucene demo parser
 

  

   

does
 

  

   

simple mapping of HTML files into Lucene Documents; it does not give
  

   

you
  

   

 

  

   

a
 

  

   

parse tree for the HTML doc.  CyberNeko is an extension of Xerces
  

   

(uses
  

   


 

  

   

the
   



 

same API; will likely become part of Xerces), and so maps an HTML
 

  

   

document
 

  

   

into a full DOM that you can manipulate easily for a wide range of
purposes.  I haven't used JTidy at an API level and so don't know it
  

   

as
  

   


 

  

   

well --
   



 

based on its UI, it appears to be focused primarily on HTML
   

validation
   

 

  

   

and
 

  

   

error detection/correction.
I use CyberNeko for a range of operations on HTML documents that go
 

  

   

beyond
 

  

   

indexing them in Lucene, and really like it.  It has been robust for
  

   

me
  

   

 

  

   

so
 

  

   

far.
Chuck
  

   

-Original Message-
From: Jingkang Zhang [mailto:[EMAIL PROTECTED]
Sent: Tuesday, February 01, 2005 1:15 AM
To: lucene-user@jakarta.apache.org
Subject: which HTML parser is better?
Three HTML parsers(Lucene web application
demo,CyberNeko HTML Parser,JTidy) are mentioned in
Lucene FAQ
1.3.27.Which is the best?Can it filter tags that are
auto-created by MS-word 'Save As HTML files' function?
_
Do You Yahoo!?
150ÍòÇúMP3·è¿ñËÑ£¬´øÄú´³ÈëÒôÀÖµîÌÃ
http://music.yisou.com/
ÃÀÅ®Ã÷ÐÇÓ¦Óо¡ÓУ¬ËѱéÃÀͼ¡¢ÑÞͼºÍ¿áͼ
http://image.yisou.com
1G¾ÍÊÇ1000Õ×£¬ÑÅ»¢µçÓÊ×ÔÖúÀ©ÈÝ£¡



Re: which HTML parser is better?

2005-02-03 Thread Karl Koch
Unfortunaltiy I am faithful ;-). Just for practical reason I want to do that
in a single class or even method called by another part in my Java
application. It should also run on Java 1.1 and it should be small and
simple. As I said before, I am in control of the HTML and it will be well
formated, because I generate it from XML using XSLT.

Karl

> If you are not married to Java:
> http://search.cpan.org/~kilinrax/HTML-Strip-1.04/Strip.pm
> 
> Otis
> 
> --- sergiu gordea <[EMAIL PROTECTED]> wrote:
> 
> > Karl Koch wrote:
> > 
> > >I am in control of the html, which means it is well formated HTML. I
> > use
> > >only HTML files which I have transformed from XML. No external HTML
> > (e.g.
> > >the web).
> > >
> > >Are there any very-short solutions for that?
> > >  
> > >
> > if you are using only correct formated HTML pages and you are in
> > control 
> > of these pages.
> > you can use a regular exprestion to remove the tags.
> > 
> > something like
> > replaceAll("<*>","");
> > 
> > This is the ideea behind the operation. If you will search on google
> > you 
> > will find a more robust
> > regular expression.
> > 
> > Using a simple regular expression will be a very cheap solution, that
> > 
> > can cause you a lot of problems in the future.
> >  
> >  It's up to you to use it 
> > 
> >  Best,
> >  
> >  Sergiu
> > 
> > >Karl
> > >
> > >  
> > >
> > >>Karl Koch wrote:
> > >>
> > >>
> > >>
> > >>>Hi,
> > >>>
> > >>>yes, but the library your are using is quite big. I was thinking
> > that a
> > >>>  
> > >>>
> > >>5kB
> > >>
> > >>
> > >>>code could actually do that. That sourceforge project is doing
> > much more
> > >>>than that but I do not need it.
> > >>> 
> > >>>
> > >>>  
> > >>>
> > >>you need just the htmlparser.jar 200k.
> > >>... you know ... the functionality is strongly correclated with the
> > size.
> > >>
> > >>  You can use 3 lines of code with a good regular expresion to
> > eliminate 
> > >>the html tags,
> > >>but this won't give you any guarantie that the text from the bad 
> > >>fromated html files will be
> > >>correctly extracted...
> > >>
> > >>  Best,
> > >>
> > >>  Sergiu
> > >>
> > >>
> > >>
> > >>>Karl
> > >>>
> > >>> 
> > >>>
> > >>>  
> > >>>
> >  Hi Karl,
> > 
> > I already submitted a peace of code that removes the html tags.
> > Search for my previous answer in this thread.
> > 
> >  Best,
> > 
> >   Sergiu
> > 
> > Karl Koch wrote:
> > 
> >    
> > 
> > 
> > 
> > >Hello,
> > >
> > >I have  been following this thread and have another question. 
> > >
> > >Is there a piece of sourcecode (which is preferably very short
> > and
> > >  
> > >
> > >>simple
> > >>
> > >>
> > >(KISS)) which allows to remove all HTML tags from HTML content?
> > HTML
> > >  
> > >
> > >>3.2
> > >>
> > >>
> > >would be enough...also no frames, CSS, etc. 
> > >
> > >I do not need to have the HTML strucutre tree or any other
> > structure
> > >  
> > >
> > >>but
> > >>
> > >>
> > >need a facility to clean up HTML into its normal underlying
> > content
> > > 
> > >
> > >  
> > >
> > before
> >    
> > 
> > 
> > 
> > >indexing that content as a whole.
> > >
> > >Karl
> > >
> > >
> > >
> > >
> > > 
> > >
> > >  
> > >
> > >>I think that depends on what you want to do.  The Lucene demo
> > parser
> > >>   
> > >>
> > >>
> > >>
> > does
> >    
> > 
> > 
> > 
> > >>simple mapping of HTML files into Lucene Documents; it does not
> > give
> > >>
> > >>
> > >>you
> > >>
> > >>
> > >>   
> > >>
> > >>
> > >>
> > a
> >    
> > 
> > 
> > 
> > >>parse tree for the HTML doc.  CyberNeko is an extension of
> > Xerces
> > >>
> > >>
> > >>(uses
> > >>
> > >>
> > >>  
> > >>
> > >>   
> > >>
> > >>
> > >>
> > >the
> > >
> > >
> > > 
> > >
> > >  
> > >
> > >>same API; will likely become part of Xerces), and so maps an
> > HTML
> > >>   
> > >>
> > >>
> > >>
> > document
> >    
> > 
> > 
> > 
> > >>into a full DOM that you can manipulate easily for a wide range
> > of
> > >>purposes.  I haven't used JTidy at an API level and so don't
> > know it
> > >>
> > >>
> > >>as
> > >>
> > >>
> > >>  
> > >>
> > >>   
> > >>
> > >>
> > >>
> > >well --
> > >
> > >
> > > 
> > >
> > >  
> > >
> > >>based on its UI, it appears to be focused primarily on HTML
> > validation
> > >>   
> > >

Re: which HTML parser is better?

2005-02-03 Thread Karl Koch
Hello Sergiu,

thank you for your help so far. I appreciate it.

I am working with Java 1.1 which does not include regular expressions.

Your turn ;-)
Karl 

> Karl Koch wrote:
> 
> >I am in control of the html, which means it is well formated HTML. I use
> >only HTML files which I have transformed from XML. No external HTML (e.g.
> >the web).
> >
> >Are there any very-short solutions for that?
> >  
> >
> if you are using only correct formated HTML pages and you are in control 
> of these pages.
> you can use a regular exprestion to remove the tags.
> 
> something like
> replaceAll("<*>","");
> 
> This is the ideea behind the operation. If you will search on google you 
> will find a more robust
> regular expression.
> 
> Using a simple regular expression will be a very cheap solution, that 
> can cause you a lot of problems in the future.
>  
>  It's up to you to use it 
> 
>  Best,
>  
>  Sergiu
> 
> >Karl
> >
> >  
> >
> >>Karl Koch wrote:
> >>
> >>
> >>
> >>>Hi,
> >>>
> >>>yes, but the library your are using is quite big. I was thinking that a
> >>>  
> >>>
> >>5kB
> >>
> >>
> >>>code could actually do that. That sourceforge project is doing much
> more
> >>>than that but I do not need it.
> >>> 
> >>>
> >>>  
> >>>
> >>you need just the htmlparser.jar 200k.
> >>... you know ... the functionality is strongly correclated with the
> size.
> >>
> >>  You can use 3 lines of code with a good regular expresion to eliminate
> >>the html tags,
> >>but this won't give you any guarantie that the text from the bad 
> >>fromated html files will be
> >>correctly extracted...
> >>
> >>  Best,
> >>
> >>  Sergiu
> >>
> >>
> >>
> >>>Karl
> >>>
> >>> 
> >>>
> >>>  
> >>>
>  Hi Karl,
> 
> I already submitted a peace of code that removes the html tags.
> Search for my previous answer in this thread.
> 
>  Best,
> 
>   Sergiu
> 
> Karl Koch wrote:
> 
>    
> 
> 
> 
> >Hello,
> >
> >I have  been following this thread and have another question. 
> >
> >Is there a piece of sourcecode (which is preferably very short and
> >  
> >
> >>simple
> >>
> >>
> >(KISS)) which allows to remove all HTML tags from HTML content? HTML
> >  
> >
> >>3.2
> >>
> >>
> >would be enough...also no frames, CSS, etc. 
> >
> >I do not need to have the HTML strucutre tree or any other structure
> >  
> >
> >>but
> >>
> >>
> >need a facility to clean up HTML into its normal underlying content
> > 
> >
> >  
> >
> before
>    
> 
> 
> 
> >indexing that content as a whole.
> >
> >Karl
> >
> >
> >
> >
> > 
> >
> >  
> >
> >>I think that depends on what you want to do.  The Lucene demo parser
> >>   
> >>
> >>
> >>
> does
>    
> 
> 
> 
> >>simple mapping of HTML files into Lucene Documents; it does not give
> >>
> >>
> >>you
> >>
> >>
> >>   
> >>
> >>
> >>
> a
>    
> 
> 
> 
> >>parse tree for the HTML doc.  CyberNeko is an extension of Xerces
> >>
> >>
> >>(uses
> >>
> >>
> >>  
> >>
> >>   
> >>
> >>
> >>
> >the
> >
> >
> > 
> >
> >  
> >
> >>same API; will likely become part of Xerces), and so maps an HTML
> >>   
> >>
> >>
> >>
> document
>    
> 
> 
> 
> >>into a full DOM that you can manipulate easily for a wide range of
> >>purposes.  I haven't used JTidy at an API level and so don't know it
> >>
> >>
> >>as
> >>
> >>
> >>  
> >>
> >>   
> >>
> >>
> >>
> >well --
> >
> >
> > 
> >
> >  
> >
> >>based on its UI, it appears to be focused primarily on HTML
> validation
> >>   
> >>
> >>
> >>
> and
>    
> 
> 
> 
> >>error detection/correction.
> >>
> >>I use CyberNeko for a range of operations on HTML documents that go
> >>   
> >>
> >>
> >>
> beyond
>    
> 
> 
> 
> >>indexing them in Lucene, and really like it.  It has been robust for
> >>
> >>
> >>me
> >>
> >>
> >>   
> >>
> >>
> >>
> so
>    
> 
> 
> 
> >>far.
> >>
> >>Chuck
> >>
> >>
> >>
> >>>-Original Message-
> >>>From: Jingkang Zhang [mailto:[EMAIL PROTECTED]
> >>>Sent: Tuesday, February 01, 2005 1:15 AM
> >>>To: lucene-user@jakarta.apache.org
> >>>Subject: which HTML parser is b

Re: which HTML parser is better?

2005-02-02 Thread sergiu gordea
Kauler, Leto S wrote:
Another very cheap, but robust solution in the case you use linux is to 
make lynx to parse your pages.

lynx page.html > page.txt.
This will strip out all html and  script, style, csimport tags. And you 
will have a .txt file ready for indexing.

 Best,
 Sergiu
We index the content from HTML files and because we only want the "good"
text and do not care about the structure, well-formedness, etc we went
with regular expressions similar to what Luke Shannon offered.
Only real difference being that we firstly remove entire blocks of
(script|style|csimport) and similar since the contents of those are not
useful for keyword searching, and afterward just remove every leftover
HTML tags.  I have been meaning to add an expression to extract things
like alt attribute text from  though.
--Leto

 

-Original Message-
From: Karl Koch [mailto:[EMAIL PROTECTED] 

I have  been following this thread and have another question. 

Is there a piece of sourcecode (which is preferably very 
short and simple
(KISS)) which allows to remove all HTML tags from HTML 
content? HTML 3.2 would be enough...also no frames, CSS, etc. 

I do not need to have the HTML strucutre tree or any other 
structure but need a facility to clean up HTML into its 
normal underlying content before indexing that content as a whole.

Karl
   

 > -Original Message-
 > From: Jingkang Zhang [mailto:[EMAIL PROTECTED]
 > Sent: Tuesday, February 01, 2005 1:15 AM
 > To: lucene-user@jakarta.apache.org
 > Subject: which HTML parser is better?
 > 
 > Three HTML parsers(Lucene web application
 > demo,CyberNeko HTML Parser,JTidy) are mentioned in
 > Lucene FAQ
 > 1.3.27.Which is the best?Can it filter tags that are
 > auto-created by MS-word 'Save As HTML files' function?
 > 
 

CONFIDENTIALITY NOTICE AND DISCLAIMER
Information in this transmission is intended only for the person(s) to whom it 
is addressed and may contain privileged and/or confidential information. If you 
are not the intended recipient, any disclosure, copying or dissemination of the 
information is unauthorised and you should delete/destroy all copies and notify 
the sender. No liability is accepted for any unauthorised use of the 
information contained in this transmission.
This disclaimer has been automatically added.
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
 


-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]


Re: which HTML parser is better?

2005-02-02 Thread Bill Tschumy
No one has yet mentioned using ParserDelegator and ParserCallback that 
are part of HTMLEditorKit in Swing.  I have been successfully using 
these classes to parse out the text of an HTML file.  You just need to 
extend HTMLEditorKit.ParserCallback and override the various methods 
that are called when different tags are encountered.

On Feb 1, 2005, at 3:14 AM, Jingkang Zhang wrote:
Three HTML parsers(Lucene web application
demo,CyberNeko HTML Parser,JTidy) are mentioned in
Lucene FAQ
1.3.27.Which is the best?Can it filter tags that are
auto-created by MS-word 'Save As HTML files' function?
--
Bill Tschumy
Otherwise -- Austin, TX
http://www.otherwise.com
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]


RE: which HTML parser is better?

2005-02-02 Thread Kauler, Leto S
We index the content from HTML files and because we only want the "good"
text and do not care about the structure, well-formedness, etc we went
with regular expressions similar to what Luke Shannon offered.

Only real difference being that we firstly remove entire blocks of
(script|style|csimport) and similar since the contents of those are not
useful for keyword searching, and afterward just remove every leftover
HTML tags.  I have been meaning to add an expression to extract things
like alt attribute text from  though.

--Leto



> -Original Message-
> From: Karl Koch [mailto:[EMAIL PROTECTED] 
> 
> I have  been following this thread and have another question. 
> 
> Is there a piece of sourcecode (which is preferably very 
> short and simple
> (KISS)) which allows to remove all HTML tags from HTML 
> content? HTML 3.2 would be enough...also no frames, CSS, etc. 
> 
> I do not need to have the HTML strucutre tree or any other 
> structure but need a facility to clean up HTML into its 
> normal underlying content before indexing that content as a whole.
> 
> Karl
> 
> > 
> >   > -Original Message-
> >   > From: Jingkang Zhang [mailto:[EMAIL PROTECTED]
> >   > Sent: Tuesday, February 01, 2005 1:15 AM
> >   > To: lucene-user@jakarta.apache.org
> >   > Subject: which HTML parser is better?
> >   > 
> >   > Three HTML parsers(Lucene web application
> >   > demo,CyberNeko HTML Parser,JTidy) are mentioned in
> >   > Lucene FAQ
> >   > 1.3.27.Which is the best?Can it filter tags that are
> >   > auto-created by MS-word 'Save As HTML files' function?
> >   > 

CONFIDENTIALITY NOTICE AND DISCLAIMER

Information in this transmission is intended only for the person(s) to whom it 
is addressed and may contain privileged and/or confidential information. If you 
are not the intended recipient, any disclosure, copying or dissemination of the 
information is unauthorised and you should delete/destroy all copies and notify 
the sender. No liability is accepted for any unauthorised use of the 
information contained in this transmission.

This disclaimer has been automatically added.

-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



Re: which HTML parser is better?

2005-02-02 Thread Luke Shannon
In our application I use regular expressions to strip all tags in one
situation and specific ones in another situation. Here is sample code for
both:

This strips all html 4.0 tags except , , , , , ,
:

html_source =
Pattern.compile("",
Pattern.CASE_INSENSITIVE).matcher(html_source).replaceAll("");

When I want to strip anything in a tag I use the following pattern with the
code above:

String strPattern1 = "<\\s?(.|\n)*?\\s?>";

HTH

Luke



- Original Message - 
From: "sergiu gordea" <[EMAIL PROTECTED]>
To: "Lucene Users List" 
Sent: Wednesday, February 02, 2005 1:23 PM
Subject: Re: which HTML parser is better?


> Karl Koch wrote:
>
> >I am in control of the html, which means it is well formated HTML. I use
> >only HTML files which I have transformed from XML. No external HTML (e.g.
> >the web).
> >
> >Are there any very-short solutions for that?
> >
> >
> if you are using only correct formated HTML pages and you are in control
> of these pages.
> you can use a regular exprestion to remove the tags.
>
> something like
> replaceAll("<*>","");
>
> This is the ideea behind the operation. If you will search on google you
> will find a more robust
> regular expression.
>
> Using a simple regular expression will be a very cheap solution, that
> can cause you a lot of problems in the future.
>
>  It's up to you to use it 
>
>  Best,
>
>  Sergiu
>
> >Karl
> >
> >
> >
> >>Karl Koch wrote:
> >>
> >>
> >>
> >>>Hi,
> >>>
> >>>yes, but the library your are using is quite big. I was thinking that a
> >>>
> >>>
> >>5kB
> >>
> >>
> >>>code could actually do that. That sourceforge project is doing much
more
> >>>than that but I do not need it.
> >>>
> >>>
> >>>
> >>>
> >>you need just the htmlparser.jar 200k.
> >>... you know ... the functionality is strongly correclated with the
size.
> >>
> >>  You can use 3 lines of code with a good regular expresion to eliminate
> >>the html tags,
> >>but this won't give you any guarantie that the text from the bad
> >>fromated html files will be
> >>correctly extracted...
> >>
> >>  Best,
> >>
> >>  Sergiu
> >>
> >>
> >>
> >>>Karl
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>> Hi Karl,
> >>>>
> >>>>I already submitted a peace of code that removes the html tags.
> >>>>Search for my previous answer in this thread.
> >>>>
> >>>> Best,
> >>>>
> >>>>  Sergiu
> >>>>
> >>>>Karl Koch wrote:
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>>Hello,
> >>>>>
> >>>>>I have  been following this thread and have another question.
> >>>>>
> >>>>>Is there a piece of sourcecode (which is preferably very short and
> >>>>>
> >>>>>
> >>simple
> >>
> >>
> >>>>>(KISS)) which allows to remove all HTML tags from HTML content? HTML
> >>>>>
> >>>>>
> >>3.2
> >>
> >>
> >>>>>would be enough...also no frames, CSS, etc.
> >>>>>
> >>>>>I do not need to have the HTML strucutre tree or any other structure
> >>>>>
> >>>>>
> >>but
> >>
> >>
> >>>>>need a facility to clean up HTML into its normal underlying content
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>before
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>>indexing that content as a whole.
> >>>>>
> >>>>>Karl
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>I think that depends on what you want to do.  The Lucene demo parser
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>does
> >>>>
> >>

Re: which HTML parser is better?

2005-02-02 Thread Otis Gospodnetic
If you are not married to Java:
http://search.cpan.org/~kilinrax/HTML-Strip-1.04/Strip.pm

Otis

--- sergiu gordea <[EMAIL PROTECTED]> wrote:

> Karl Koch wrote:
> 
> >I am in control of the html, which means it is well formated HTML. I
> use
> >only HTML files which I have transformed from XML. No external HTML
> (e.g.
> >the web).
> >
> >Are there any very-short solutions for that?
> >  
> >
> if you are using only correct formated HTML pages and you are in
> control 
> of these pages.
> you can use a regular exprestion to remove the tags.
> 
> something like
> replaceAll("<*>","");
> 
> This is the ideea behind the operation. If you will search on google
> you 
> will find a more robust
> regular expression.
> 
> Using a simple regular expression will be a very cheap solution, that
> 
> can cause you a lot of problems in the future.
>  
>  It's up to you to use it 
> 
>  Best,
>  
>  Sergiu
> 
> >Karl
> >
> >  
> >
> >>Karl Koch wrote:
> >>
> >>
> >>
> >>>Hi,
> >>>
> >>>yes, but the library your are using is quite big. I was thinking
> that a
> >>>  
> >>>
> >>5kB
> >>
> >>
> >>>code could actually do that. That sourceforge project is doing
> much more
> >>>than that but I do not need it.
> >>> 
> >>>
> >>>  
> >>>
> >>you need just the htmlparser.jar 200k.
> >>... you know ... the functionality is strongly correclated with the
> size.
> >>
> >>  You can use 3 lines of code with a good regular expresion to
> eliminate 
> >>the html tags,
> >>but this won't give you any guarantie that the text from the bad 
> >>fromated html files will be
> >>correctly extracted...
> >>
> >>  Best,
> >>
> >>  Sergiu
> >>
> >>
> >>
> >>>Karl
> >>>
> >>> 
> >>>
> >>>  
> >>>
>  Hi Karl,
> 
> I already submitted a peace of code that removes the html tags.
> Search for my previous answer in this thread.
> 
>  Best,
> 
>   Sergiu
> 
> Karl Koch wrote:
> 
>    
> 
> 
> 
> >Hello,
> >
> >I have  been following this thread and have another question. 
> >
> >Is there a piece of sourcecode (which is preferably very short
> and
> >  
> >
> >>simple
> >>
> >>
> >(KISS)) which allows to remove all HTML tags from HTML content?
> HTML
> >  
> >
> >>3.2
> >>
> >>
> >would be enough...also no frames, CSS, etc. 
> >
> >I do not need to have the HTML strucutre tree or any other
> structure
> >  
> >
> >>but
> >>
> >>
> >need a facility to clean up HTML into its normal underlying
> content
> > 
> >
> >  
> >
> before
>    
> 
> 
> 
> >indexing that content as a whole.
> >
> >Karl
> >
> >
> >
> >
> > 
> >
> >  
> >
> >>I think that depends on what you want to do.  The Lucene demo
> parser
> >>   
> >>
> >>
> >>
> does
>    
> 
> 
> 
> >>simple mapping of HTML files into Lucene Documents; it does not
> give
> >>
> >>
> >>you
> >>
> >>
> >>   
> >>
> >>
> >>
> a
>    
> 
> 
> 
> >>parse tree for the HTML doc.  CyberNeko is an extension of
> Xerces
> >>
> >>
> >>(uses
> >>
> >>
> >>  
> >>
> >>   
> >>
> >>
> >>
> >the
> >
> >
> > 
> >
> >  
> >
> >>same API; will likely become part of Xerces), and so maps an
> HTML
> >>   
> >>
> >>
> >>
> document
>    
> 
> 
> 
> >>into a full DOM that you can manipulate easily for a wide range
> of
> >>purposes.  I haven't used JTidy at an API level and so don't
> know it
> >>
> >>
> >>as
> >>
> >>
> >>  
> >>
> >>   
> >>
> >>
> >>
> >well --
> >
> >
> > 
> >
> >  
> >
> >>based on its UI, it appears to be focused primarily on HTML
> validation
> >>   
> >>
> >>
> >>
> and
>    
> 
> 
> 
> >>error detection/correction.
> >>
> >>I use CyberNeko for a range of operations on HTML documents
> that go
> >>   
> >>
> >>
> >>
> beyond
>    
> 
> 
> 
> >>indexing them in Lucene, and really like it.  It has been
> robust for
> >>
> >>
> >>me
> >>
> >>
> >>   
> >>
> >>
> >>
> so
>    
> 
> 
> 
> >>far.
> >>
> >>Chuck
> >>
> >>
> >>
> >>>-Original Message-
> >>>From: Jingkang Zhang [mailto:[EMAIL PROTECTED]
> >>>Sent: Tuesday, February 01, 2005 1:15 AM
> >>>To: lucene-user@jakarta.apache.org
> >>>Subject

Re: which HTML parser is better?

2005-02-02 Thread sergiu gordea
Karl Koch wrote:
I am in control of the html, which means it is well formated HTML. I use
only HTML files which I have transformed from XML. No external HTML (e.g.
the web).
Are there any very-short solutions for that?
 

if you are using only correct formated HTML pages and you are in control 
of these pages.
you can use a regular exprestion to remove the tags.

something like
replaceAll("<*>","");
This is the ideea behind the operation. If you will search on google you 
will find a more robust
regular expression.

Using a simple regular expression will be a very cheap solution, that 
can cause you a lot of problems in the future.

It's up to you to use it 
Best,
Sergiu
Karl
 

Karl Koch wrote:
   

Hi,
yes, but the library your are using is quite big. I was thinking that a
 

5kB
   

code could actually do that. That sourceforge project is doing much more
than that but I do not need it.
 

you need just the htmlparser.jar 200k.
... you know ... the functionality is strongly correclated with the size.
 You can use 3 lines of code with a good regular expresion to eliminate 
the html tags,
but this won't give you any guarantie that the text from the bad 
fromated html files will be
correctly extracted...

 Best,
 Sergiu
   

Karl

 

Hi Karl,
I already submitted a peace of code that removes the html tags.
Search for my previous answer in this thread.
Best,
 Sergiu
Karl Koch wrote:
  

   

Hello,
I have  been following this thread and have another question. 

Is there a piece of sourcecode (which is preferably very short and
 

simple
   

(KISS)) which allows to remove all HTML tags from HTML content? HTML
 

3.2
   

would be enough...also no frames, CSS, etc. 

I do not need to have the HTML strucutre tree or any other structure
 

but
   

need a facility to clean up HTML into its normal underlying content


 

before
  

   

indexing that content as a whole.
Karl



 

I think that depends on what you want to do.  The Lucene demo parser
  

   

does
  

   

simple mapping of HTML files into Lucene Documents; it does not give
   

you
   

  

   

a
  

   

parse tree for the HTML doc.  CyberNeko is an extension of Xerces
   

(uses
   

 

  

   

the


 

same API; will likely become part of Xerces), and so maps an HTML
  

   

document
  

   

into a full DOM that you can manipulate easily for a wide range of
purposes.  I haven't used JTidy at an API level and so don't know it
   

as
   

 

  

   

well --


 

based on its UI, it appears to be focused primarily on HTML validation
  

   

and
  

   

error detection/correction.
I use CyberNeko for a range of operations on HTML documents that go
  

   

beyond
  

   

indexing them in Lucene, and really like it.  It has been robust for
   

me
   

  

   

so
  

   

far.
Chuck
   

-Original Message-
From: Jingkang Zhang [mailto:[EMAIL PROTECTED]
Sent: Tuesday, February 01, 2005 1:15 AM
To: lucene-user@jakarta.apache.org
Subject: which HTML parser is better?
Three HTML parsers(Lucene web application
demo,CyberNeko HTML Parser,JTidy) are mentioned in
Lucene FAQ
1.3.27.Which is the best?Can it filter tags that are
auto-created by MS-word 'Save As HTML files' function?
_
Do You Yahoo!?
150ÍòÇúMP3·è¿ñËÑ£¬´øÄú´³ÈëÒôÀÖµîÌÃ
http://music.yisou.com/
ÃÀÅ®Ã÷ÐÇÓ¦Óо¡ÓУ¬ËѱéÃÀͼ¡¢ÑÞͼºÍ¿áͼ
http://image.yisou.com
1G¾ÍÊÇ1000Õ×£¬ÑÅ»¢µçÓÊ×ÔÖúÀ©ÈÝ£¡
 

http://cn.rd.yahoo.com/mail_cn/tag/1g/*http://cn.mail.yahoo.com/event/ma
 

il_1g/
 

  

   

-
  

   

To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail:
 

[EMAIL PROTECTED]
   

-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
 

  

   



 

-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
  

   


 

-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
   

 


-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]


Re: which HTML parser is better?

2005-02-02 Thread Karl Koch
I am in control of the html, which means it is well formated HTML. I use
only HTML files which I have transformed from XML. No external HTML (e.g.
the web).

Are there any very-short solutions for that?

Karl

> Karl Koch wrote:
> 
> >Hi,
> >
> >yes, but the library your are using is quite big. I was thinking that a
> 5kB
> >code could actually do that. That sourceforge project is doing much more
> >than that but I do not need it.
> >  
> >
> you need just the htmlparser.jar 200k.
> ... you know ... the functionality is strongly correclated with the size.
> 
>   You can use 3 lines of code with a good regular expresion to eliminate 
> the html tags,
> but this won't give you any guarantie that the text from the bad 
> fromated html files will be
> correctly extracted...
> 
>   Best,
> 
>   Sergiu
> 
> >Karl
> >
> >  
> >
> >>  Hi Karl,
> >>
> >> I already submitted a peace of code that removes the html tags.
> >> Search for my previous answer in this thread.
> >>
> >>  Best,
> >>
> >>   Sergiu
> >>
> >>Karl Koch wrote:
> >>
> >>
> >>
> >>>Hello,
> >>>
> >>>I have  been following this thread and have another question. 
> >>>
> >>>Is there a piece of sourcecode (which is preferably very short and
> simple
> >>>(KISS)) which allows to remove all HTML tags from HTML content? HTML
> 3.2
> >>>would be enough...also no frames, CSS, etc. 
> >>>
> >>>I do not need to have the HTML strucutre tree or any other structure
> but
> >>>need a facility to clean up HTML into its normal underlying content
> >>>  
> >>>
> >>before
> >>
> >>
> >>>indexing that content as a whole.
> >>>
> >>>Karl
> >>>
> >>>
> >>> 
> >>>
> >>>  
> >>>
> I think that depends on what you want to do.  The Lucene demo parser
> 
> 
> >>does
> >>
> >>
> simple mapping of HTML files into Lucene Documents; it does not give
> you
> 
> 
> >>a
> >>
> >>
> parse tree for the HTML doc.  CyberNeko is an extension of Xerces
> (uses
>    
> 
> 
> 
> >>>the
> >>> 
> >>>
> >>>  
> >>>
> same API; will likely become part of Xerces), and so maps an HTML
> 
> 
> >>document
> >>
> >>
> into a full DOM that you can manipulate easily for a wide range of
> purposes.  I haven't used JTidy at an API level and so don't know it
> as
>    
> 
> 
> 
> >>>well --
> >>> 
> >>>
> >>>  
> >>>
> based on its UI, it appears to be focused primarily on HTML validation
> 
> 
> >>and
> >>
> >>
> error detection/correction.
> 
> I use CyberNeko for a range of operations on HTML documents that go
> 
> 
> >>beyond
> >>
> >>
> indexing them in Lucene, and really like it.  It has been robust for
> me
> 
> 
> >>so
> >>
> >>
> far.
> 
> Chuck
> 
>  > -Original Message-
>  > From: Jingkang Zhang [mailto:[EMAIL PROTECTED]
>  > Sent: Tuesday, February 01, 2005 1:15 AM
>  > To: lucene-user@jakarta.apache.org
>  > Subject: which HTML parser is better?
>  > 
>  > Three HTML parsers(Lucene web application
>  > demo,CyberNeko HTML Parser,JTidy) are mentioned in
>  > Lucene FAQ
>  > 1.3.27.Which is the best?Can it filter tags that are
>  > auto-created by MS-word 'Save As HTML files' function?
>  > 
>  > _
>  > Do You Yahoo!?
>  > 150ÍòÇúMP3·è¿ñËÑ£¬´øÄú´³ÈëÒôÀÖµîÌÃ
>  > http://music.yisou.com/
>  > ÃÀÅ®Ã÷ÐÇÓ¦Óо¡ÓУ¬ËѱéÃÀͼ¡¢ÑÞͼºÍ¿áͼ
>  > http://image.yisou.com
>  > 1G¾ÍÊÇ1000Õ×£¬ÑÅ»¢µçÓÊ×ÔÖúÀ©ÈÝ£¡
>  >
>
http://cn.rd.yahoo.com/mail_cn/tag/1g/*http://cn.mail.yahoo.com/event/ma
>  > il_1g/
>  > 
>  >
> 
> 
> >>-
> >>
> >>
>  > To unsubscribe, e-mail: [EMAIL PROTECTED]
>  > For additional commands, e-mail:
> [EMAIL PROTECTED]
> 
> 
> -
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
> 
>    
> 
> 
> 
> >>> 
> >>>
> >>>  
> >>>
> >>-
> >>To unsubscribe, e-mail: [EMAIL PROTECTED]
> >>For additional commands, e-mail: [EMAIL PROTECTED]
> >>
> >>
> >>
> >
> >  
> >
> 
> 
> -
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
> 

-- 
GMX im TV ... Die Gedanken sind frei ... Schon gesehen?
Jetzt Spot online ansehen: http://www.gmx.net/de/go/tv-spot

-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



Re: which HTML parser is better?

2005-02-02 Thread sergiu gordea
Karl Koch wrote:
Hi,
yes, but the library your are using is quite big. I was thinking that a 5kB
code could actually do that. That sourceforge project is doing much more
than that but I do not need it.
 

you need just the htmlparser.jar 200k.
... you know ... the functionality is strongly correclated with the size.
 You can use 3 lines of code with a good regular expresion to eliminate 
the html tags,
but this won't give you any guarantie that the text from the bad 
fromated html files will be
correctly extracted...

 Best,
 Sergiu
Karl
 

 Hi Karl,
I already submitted a peace of code that removes the html tags.
Search for my previous answer in this thread.
 Best,
  Sergiu
Karl Koch wrote:
   

Hello,
I have  been following this thread and have another question. 

Is there a piece of sourcecode (which is preferably very short and simple
(KISS)) which allows to remove all HTML tags from HTML content? HTML 3.2
would be enough...also no frames, CSS, etc. 

I do not need to have the HTML strucutre tree or any other structure but
need a facility to clean up HTML into its normal underlying content
 

before
   

indexing that content as a whole.
Karl

 

I think that depends on what you want to do.  The Lucene demo parser
   

does
   

simple mapping of HTML files into Lucene Documents; it does not give you
   

a
   

parse tree for the HTML doc.  CyberNeko is an extension of Xerces (uses
  

   

the
 

same API; will likely become part of Xerces), and so maps an HTML
   

document
   

into a full DOM that you can manipulate easily for a wide range of
purposes.  I haven't used JTidy at an API level and so don't know it as
  

   

well --
 

based on its UI, it appears to be focused primarily on HTML validation
   

and
   

error detection/correction.
I use CyberNeko for a range of operations on HTML documents that go
   

beyond
   

indexing them in Lucene, and really like it.  It has been robust for me
   

so
   

far.
Chuck
> -Original Message-
> From: Jingkang Zhang [mailto:[EMAIL PROTECTED]
> Sent: Tuesday, February 01, 2005 1:15 AM
> To: lucene-user@jakarta.apache.org
> Subject: which HTML parser is better?
> 
> Three HTML parsers(Lucene web application
> demo,CyberNeko HTML Parser,JTidy) are mentioned in
> Lucene FAQ
> 1.3.27.Which is the best?Can it filter tags that are
> auto-created by MS-word 'Save As HTML files' function?
> 
> _
> Do You Yahoo!?
> 150ÍòÇúMP3·è¿ñËÑ£¬´øÄú´³ÈëÒôÀÖµîÌÃ
> http://music.yisou.com/
> ÃÀÅ®Ã÷ÐÇÓ¦Óо¡ÓУ¬ËѱéÃÀͼ¡¢ÑÞͼºÍ¿áͼ
> http://image.yisou.com
> 1G¾ÍÊÇ1000Õ×£¬ÑÅ»¢µçÓÊ×ÔÖúÀ©ÈÝ£¡
>
http://cn.rd.yahoo.com/mail_cn/tag/1g/*http://cn.mail.yahoo.com/event/ma
> il_1g/
> 
>
   

-
   

> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
  

   


 

-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
   

 


-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]


Re: which HTML parser is better?

2005-02-02 Thread Karl Koch
Hi,

yes, but the library your are using is quite big. I was thinking that a 5kB
code could actually do that. That sourceforge project is doing much more
than that but I do not need it.

Karl

>   Hi Karl,
> 
>  I already submitted a peace of code that removes the html tags.
>  Search for my previous answer in this thread.
> 
>   Best,
> 
>Sergiu
> 
> Karl Koch wrote:
> 
> >Hello,
> >
> >I have  been following this thread and have another question. 
> >
> >Is there a piece of sourcecode (which is preferably very short and simple
> >(KISS)) which allows to remove all HTML tags from HTML content? HTML 3.2
> >would be enough...also no frames, CSS, etc. 
> >
> >I do not need to have the HTML strucutre tree or any other structure but
> >need a facility to clean up HTML into its normal underlying content
> before
> >indexing that content as a whole.
> >
> >Karl
> >
> >
> >  
> >
> >>I think that depends on what you want to do.  The Lucene demo parser
> does
> >>simple mapping of HTML files into Lucene Documents; it does not give you
> a
> >>parse tree for the HTML doc.  CyberNeko is an extension of Xerces (uses
> >>
> >>
> >the
> >  
> >
> >>same API; will likely become part of Xerces), and so maps an HTML
> document
> >>into a full DOM that you can manipulate easily for a wide range of
> >>purposes.  I haven't used JTidy at an API level and so don't know it as
> >>
> >>
> >well --
> >  
> >
> >>based on its UI, it appears to be focused primarily on HTML validation
> and
> >>error detection/correction.
> >>
> >>I use CyberNeko for a range of operations on HTML documents that go
> beyond
> >>indexing them in Lucene, and really like it.  It has been robust for me
> so
> >>far.
> >>
> >>Chuck
> >>
> >>  > -Original Message-
> >>  > From: Jingkang Zhang [mailto:[EMAIL PROTECTED]
> >>  > Sent: Tuesday, February 01, 2005 1:15 AM
> >>  > To: lucene-user@jakarta.apache.org
> >>  > Subject: which HTML parser is better?
> >>  > 
> >>  > Three HTML parsers(Lucene web application
> >>  > demo,CyberNeko HTML Parser,JTidy) are mentioned in
> >>  > Lucene FAQ
> >>  > 1.3.27.Which is the best?Can it filter tags that are
> >>  > auto-created by MS-word 'Save As HTML files' function?
> >>  > 
> >>  > _
> >>  > Do You Yahoo!?
> >>  > 150ÍòÇúMP3·è¿ñËÑ£¬´øÄú´³ÈëÒôÀÖµîÌÃ
> >>  > http://music.yisou.com/
> >>  > ÃÀÅ®Ã÷ÐÇÓ¦Óо¡ÓУ¬ËѱéÃÀͼ¡¢ÑÞͼºÍ¿áͼ
> >>  > http://image.yisou.com
> >>  > 1G¾ÍÊÇ1000Õ×£¬ÑÅ»¢µçÓÊ×ÔÖúÀ©ÈÝ£¡
> >>  >
> >>http://cn.rd.yahoo.com/mail_cn/tag/1g/*http://cn.mail.yahoo.com/event/ma
> >>  > il_1g/
> >>  > 
> >>  >
> -
> >>  > To unsubscribe, e-mail: [EMAIL PROTECTED]
> >>  > For additional commands, e-mail: [EMAIL PROTECTED]
> >>
> >>
> >>-
> >>To unsubscribe, e-mail: [EMAIL PROTECTED]
> >>For additional commands, e-mail: [EMAIL PROTECTED]
> >>
> >>
> >>
> >
> >  
> >
> 
> 
> -
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
> 

-- 
10 GB Mailbox, 100 FreeSMS http://www.gmx.net/de/go/topmail
+++ GMX - die erste Adresse für Mail, Message, More +++

-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



Re: which HTML parser is better?

2005-02-02 Thread Erik Hatcher
On Feb 2, 2005, at 6:17 AM, Karl Koch wrote:
Hello,
I have  been following this thread and have another question.
Is there a piece of sourcecode (which is preferably very short and 
simple
(KISS)) which allows to remove all HTML tags from HTML content? HTML 
3.2
would be enough...also no frames, CSS, etc.

I do not need to have the HTML strucutre tree or any other structure 
but
need a facility to clean up HTML into its normal underlying content 
before
indexing that content as a whole.

The code in the Lucene Sandbox for parsing HTML with JTidy (under 
contributions/ant) for the  task does what you ask.

Erik
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]


Re: which HTML parser is better?

2005-02-02 Thread sergiu gordea
 Hi Karl,
I already submitted a peace of code that removes the html tags.
Search for my previous answer in this thread.
 Best,
  Sergiu
Karl Koch wrote:
Hello,
I have  been following this thread and have another question. 

Is there a piece of sourcecode (which is preferably very short and simple
(KISS)) which allows to remove all HTML tags from HTML content? HTML 3.2
would be enough...also no frames, CSS, etc. 

I do not need to have the HTML strucutre tree or any other structure but
need a facility to clean up HTML into its normal underlying content before
indexing that content as a whole.
Karl
 

I think that depends on what you want to do.  The Lucene demo parser does
simple mapping of HTML files into Lucene Documents; it does not give you a
parse tree for the HTML doc.  CyberNeko is an extension of Xerces (uses
   

the
 

same API; will likely become part of Xerces), and so maps an HTML document
into a full DOM that you can manipulate easily for a wide range of
purposes.  I haven't used JTidy at an API level and so don't know it as
   

well --
 

based on its UI, it appears to be focused primarily on HTML validation and
error detection/correction.
I use CyberNeko for a range of operations on HTML documents that go beyond
indexing them in Lucene, and really like it.  It has been robust for me so
far.
Chuck
 > -Original Message-
 > From: Jingkang Zhang [mailto:[EMAIL PROTECTED]
 > Sent: Tuesday, February 01, 2005 1:15 AM
 > To: lucene-user@jakarta.apache.org
 > Subject: which HTML parser is better?
 > 
 > Three HTML parsers(Lucene web application
 > demo,CyberNeko HTML Parser,JTidy) are mentioned in
 > Lucene FAQ
 > 1.3.27.Which is the best?Can it filter tags that are
 > auto-created by MS-word 'Save As HTML files' function?
 > 
 > _
 > Do You Yahoo!?
 > 150ÍòÇúMP3·è¿ñËÑ£¬´øÄú´³ÈëÒôÀÖµîÌÃ
 > http://music.yisou.com/
 > ÃÀÅ®Ã÷ÐÇÓ¦Óо¡ÓУ¬ËѱéÃÀͼ¡¢ÑÞͼºÍ¿áͼ
 > http://image.yisou.com
 > 1G¾ÍÊÇ1000Õ×£¬ÑÅ»¢µçÓÊ×ÔÖúÀ©ÈÝ£¡
 >
http://cn.rd.yahoo.com/mail_cn/tag/1g/*http://cn.mail.yahoo.com/event/ma
 > il_1g/
 > 
 > -
 > To unsubscribe, e-mail: [EMAIL PROTECTED]
 > For additional commands, e-mail: [EMAIL PROTECTED]

-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
   

 


-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]


RE: which HTML parser is better?

2005-02-02 Thread Karl Koch
Hello,

I have  been following this thread and have another question. 

Is there a piece of sourcecode (which is preferably very short and simple
(KISS)) which allows to remove all HTML tags from HTML content? HTML 3.2
would be enough...also no frames, CSS, etc. 

I do not need to have the HTML strucutre tree or any other structure but
need a facility to clean up HTML into its normal underlying content before
indexing that content as a whole.

Karl


> I think that depends on what you want to do.  The Lucene demo parser does
> simple mapping of HTML files into Lucene Documents; it does not give you a
> parse tree for the HTML doc.  CyberNeko is an extension of Xerces (uses
the
> same API; will likely become part of Xerces), and so maps an HTML document
> into a full DOM that you can manipulate easily for a wide range of
> purposes.  I haven't used JTidy at an API level and so don't know it as
well --
> based on its UI, it appears to be focused primarily on HTML validation and
> error detection/correction.
> 
> I use CyberNeko for a range of operations on HTML documents that go beyond
> indexing them in Lucene, and really like it.  It has been robust for me so
> far.
> 
> Chuck
> 
>   > -Original Message-
>   > From: Jingkang Zhang [mailto:[EMAIL PROTECTED]
>   > Sent: Tuesday, February 01, 2005 1:15 AM
>   > To: lucene-user@jakarta.apache.org
>   > Subject: which HTML parser is better?
>   > 
>   > Three HTML parsers(Lucene web application
>   > demo,CyberNeko HTML Parser,JTidy) are mentioned in
>   > Lucene FAQ
>   > 1.3.27.Which is the best?Can it filter tags that are
>   > auto-created by MS-word 'Save As HTML files' function?
>   > 
>   > _
>   > Do You Yahoo!?
>   > 150ÍòÇúMP3·è¿ñËÑ£¬´øÄú´³ÈëÒôÀÖµîÌÃ
>   > http://music.yisou.com/
>   > ÃÀÅ®Ã÷ÐÇÓ¦Óо¡ÓУ¬ËѱéÃÀͼ¡¢ÑÞͼºÍ¿áͼ
>   > http://image.yisou.com
>   > 1G¾ÍÊÇ1000Õ×£¬ÑÅ»¢µçÓÊ×ÔÖúÀ©ÈÝ£¡
>   >
> http://cn.rd.yahoo.com/mail_cn/tag/1g/*http://cn.mail.yahoo.com/event/ma
>   > il_1g/
>   > 
>   > -
>   > To unsubscribe, e-mail: [EMAIL PROTECTED]
>   > For additional commands, e-mail: [EMAIL PROTECTED]
> 
> 
> -
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
> 

-- 
GMX im TV ... Die Gedanken sind frei ... Schon gesehen?
Jetzt Spot online ansehen: http://www.gmx.net/de/go/tv-spot

-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



RE: which HTML parser is better?

2005-02-01 Thread Chuck Williams
I think that depends on what you want to do.  The Lucene demo parser does 
simple mapping of HTML files into Lucene Documents; it does not give you a 
parse tree for the HTML doc.  CyberNeko is an extension of Xerces (uses the 
same API; will likely become part of Xerces), and so maps an HTML document into 
a full DOM that you can manipulate easily for a wide range of purposes.  I 
haven't used JTidy at an API level and so don't know it as well -- based on its 
UI, it appears to be focused primarily on HTML validation and error 
detection/correction.

I use CyberNeko for a range of operations on HTML documents that go beyond 
indexing them in Lucene, and really like it.  It has been robust for me so far.

Chuck

  > -Original Message-
  > From: Jingkang Zhang [mailto:[EMAIL PROTECTED]
  > Sent: Tuesday, February 01, 2005 1:15 AM
  > To: lucene-user@jakarta.apache.org
  > Subject: which HTML parser is better?
  > 
  > Three HTML parsers(Lucene web application
  > demo,CyberNeko HTML Parser,JTidy) are mentioned in
  > Lucene FAQ
  > 1.3.27.Which is the best?Can it filter tags that are
  > auto-created by MS-word 'Save As HTML files' function?
  > 
  > _
  > Do You Yahoo!?
  > 150万曲MP3疯狂搜,带您闯入音乐殿堂
  > http://music.yisou.com/
  > 美女明星应有尽有,搜遍美图、艳图和酷图
  > http://image.yisou.com
  > 1G就是1000兆,雅虎电邮自助扩容!
  > http://cn.rd.yahoo.com/mail_cn/tag/1g/*http://cn.mail.yahoo.com/event/ma
  > il_1g/
  > 
  > -
  > To unsubscribe, e-mail: [EMAIL PROTECTED]
  > For additional commands, e-mail: [EMAIL PROTECTED]


-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



Re: which HTML parser is better?

2005-02-01 Thread Michael Giles
When I tested parsers a year or so ago for intensive use in Furl, the
best (tolerant of bad HTML) and fastest (tested on a 1.5M HTML page)
parser by far was TagSoup ( http://www.tagsoup.info ). It is actively
maintained and improved and I have never had any problems with it.

-Mike

Jingkang Zhang wrote:

>Three HTML parsers(Lucene web application
>demo,CyberNeko HTML Parser,JTidy) are mentioned in
>Lucene FAQ
>1.3.27.Which is the best?Can it filter tags that are
>auto-created by MS-word 'Save As HTML files' function?
>
>_
>Do You Yahoo!?
>150万曲MP3疯狂搜,带您闯入音乐殿堂
>http://music.yisou.com/
>美女明星应有尽有,搜遍美图、艳图和酷图
>http://image.yisou.com
>1G就是1000兆,雅虎电邮自助扩容!
>http://cn.rd.yahoo.com/mail_cn/tag/1g/*http://cn.mail.yahoo.com/event/mail_1g/
>
>-
>To unsubscribe, e-mail: [EMAIL PROTECTED]
>For additional commands, e-mail: [EMAIL PROTECTED]
>
>  
>

-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



Re: which HTML parser is better?

2005-02-01 Thread sergiu gordea
Jingkang Zhang wrote:

>Three HTML parsers(Lucene web application
>demo,CyberNeko HTML Parser,JTidy) are mentioned in
>Lucene FAQ
>1.3.27.Which is the best?Can it filter tags that are
>auto-created by MS-word 'Save As HTML files' function?
>  
>

maybe you can try this library...

http://htmlparser.sourceforge.net/

I use the following code to get the text from HTML files,
it was not intensively tested, but it works.

import org.htmlparser.Node;
import org.htmlparser.Parser;
import org.htmlparser.util.NodeIterator;
import org.htmlparser.util.Translate;

Parser parser = new Parser(source.getAbsolutePath());
NodeIterator iter = parser.elements();
while (iter.hasMoreNodes()) {
Node element = (Node) iter.nextNode();
//System.out.println("1:" + element.getText());
String text = Translate.decode(element.toPlainTextString());
if (Utils.notEmptyString(text))
writer.write(text);
}

Sergiu

>_
>Do You Yahoo!?
>150万曲MP3疯狂搜,带您闯入音乐殿堂
>http://music.yisou.com/
>美女明星应有尽有,搜遍美图、艳图和酷图
>http://image.yisou.com
>1G就是1000兆,雅虎电邮自助扩容!
>http://cn.rd.yahoo.com/mail_cn/tag/1g/*http://cn.mail.yahoo.com/event/mail_1g/
>
>-
>To unsubscribe, e-mail: [EMAIL PROTECTED]
>For additional commands, e-mail: [EMAIL PROTECTED]
>
>  
>


-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]