Revision: 19720
http://sourceforge.net/p/gate/code/19720
Author: ian_roberts
Date: 2016-11-06 20:36:04 +0000 (Sun, 06 Nov 2016)
Log Message:
-----------
Improved handling of orth files in the normaliser. The file pointed to by the
orthURL parameter can now be a "definition" type file similar to a gazetteer
.def file, which lists other files to load, allowing you to split the orth list
up into separate chunks. Also added an explicit encoding parameter for the
orth files (default UTF-8) rather than relying on the platform default
encoding. And parsing of orth files is now done with a proper CSV parser
library so it can handle things like commas within values (if you double-quote
the entry).
Note that the encoding parameter only applies to the orth files - jaspell uses
the platform default encoding internally and there's no way to change that
(other than to fork jaspell, which might not be such a bad idea actually)
Modified Paths:
--------------
gate/trunk/plugins/Twitter/LICENSE.txt
gate/trunk/plugins/Twitter/creole.xml
gate/trunk/plugins/Twitter/resources/normaliser/orth.en.csv
gate/trunk/plugins/Twitter/src/gate/twitter/Normaliser.java
Added Paths:
-----------
gate/trunk/plugins/Twitter/lib/opencsv-3.8.jar
Modified: gate/trunk/plugins/Twitter/LICENSE.txt
===================================================================
--- gate/trunk/plugins/Twitter/LICENSE.txt 2016-11-06 10:01:40 UTC (rev
19719)
+++ gate/trunk/plugins/Twitter/LICENSE.txt 2016-11-06 20:36:04 UTC (rev
19720)
@@ -1,4 +1,6 @@
-The GATE components within the Twitter plugin are licensed under the LGPL
(version 3).
+The GATE components within the Twitter plugin are licensed under the LGPL
+(version 3). The jaspell library uses a BSD licence and the opencsv library is
+released under the Apache licence version 2.
Note, however, that the TwitIE application is licensed under the GPL (version
3)
due to it containing components from the Standford_CoreNLP plugin.
Modified: gate/trunk/plugins/Twitter/creole.xml
===================================================================
--- gate/trunk/plugins/Twitter/creole.xml 2016-11-06 10:01:40 UTC (rev
19719)
+++ gate/trunk/plugins/Twitter/creole.xml 2016-11-06 20:36:04 UTC (rev
19720)
@@ -2,5 +2,6 @@
<CREOLE-DIRECTORY>
<JAR SCAN="true">twitter.jar</JAR>
<JAR>lib/jaspell.jar</JAR>
+ <JAR>lib/opencsv-3.8.jar</JAR>
<REQUIRES>../Format_Twitter</REQUIRES>
</CREOLE-DIRECTORY>
Added: gate/trunk/plugins/Twitter/lib/opencsv-3.8.jar
===================================================================
(Binary files differ)
Index: gate/trunk/plugins/Twitter/lib/opencsv-3.8.jar
===================================================================
--- gate/trunk/plugins/Twitter/lib/opencsv-3.8.jar 2016-11-06 10:01:40 UTC
(rev 19719)
+++ gate/trunk/plugins/Twitter/lib/opencsv-3.8.jar 2016-11-06 20:36:04 UTC
(rev 19720)
Property changes on: gate/trunk/plugins/Twitter/lib/opencsv-3.8.jar
___________________________________________________________________
Added: svn:mime-type
## -0,0 +1 ##
+application/zip
\ No newline at end of property
Modified: gate/trunk/plugins/Twitter/resources/normaliser/orth.en.csv
===================================================================
--- gate/trunk/plugins/Twitter/resources/normaliser/orth.en.csv 2016-11-06
10:01:40 UTC (rev 19719)
+++ gate/trunk/plugins/Twitter/resources/normaliser/orth.en.csv 2016-11-06
20:36:04 UTC (rev 19720)
@@ -1,355 +1,2 @@
-10q,thank you
-10x,thanks
-1337,elite
-2day,today
-2gva,together
-2moro,tomorrow
-2morrow,tomorrow
-2moz,tomorrow
-2night,tonight
-2nite,tonight
-<3,love
-4eva,forever
-5-0,police
-5o,police
-5-O,police
-abt,about
-adad,another day another dollar
-adih,another day in hell
-adip,another day in paradise
-aeap,as early as possible
-afaicr,as far as i can recall
-afaics,as far as i can see
-afaict,as far as i can tell
-afaik,as far as i know
-afair,as far as i remember
-afap,as far as possible
-afta,have to
-agen,again
-aint,ain't
-anyways,anyway
-apr,April
-atm,now
-aug,August
-awol,away
-b4,before
-bbiab,be back in a bit
-bbl,be back later
-bbq,barbecue
-bbs,be back soon
-bby,baby
-b/c,because
-bc,because
-bcuz,because
-b-day,birthday
-bday,birthday
-bff,best friends forever
-bfn,bye for now
-birfday,birthday
-blate,blatently
-blates,blatently
-bles,bless
-blu,blue
-bofh,bastard operator from hell
-boi,boy
-brb,be right back
-bringin,bringing
-bro,brother
-bsod,blue screen of death
-bt,but
-btdt,been there done that
-btw,by the way
-cmiiw,correct me if i'm wrong
-cnt,can't
-cont,continued
-convo,conversation
-cryin,crying
-cum,come
-cus,curse
-cu,see you
-cussin,cursing
-cusz,because
-cuz,because
-da,the
-dat,that
-dec,December
-deez,these
-def,definitely
-dey,they
-dftt,don't feed the trolls
-dgaf,don't give a fuck
-diaf,die in a fire
-dint,didn't
-dis,this
-dl,download
-dnd,do not disturb
-dnt,don't
-doa,dead on arrival
-doin,doing
-dossin,dossing
-dun,don't
-ep,episode
-eps,episodes
-es,he's
-eta,estimated time of arrival
-every1,everyone
-everytim,every time
-evry1,everyone
-evry,every
-ez,easy
-f9,fine
-fakin,faking
-fallin,falling
-fb,Facebook
-fck,fuck
-fckn,fucking
-feb,February
-ffs,for fuck's sake
-fkn,fucking
-fmcdh,from my cold dead hands
-fml,fuck my life.
-foad,fuck off and die
-foaf,friend of a friend
-fo,for
-fo,for
-frm,from
-ftfy,fixed that for you
-ftl,for the loss
-ftw,for the win
-fucken,fucking
-fu,fuck you
-fwiw,for what it's worth
-fyi,for your information
-g2g,got to go
-gbtw,get back to work
-gd,good
-gen,again
-getn,getting
-gf,great/good fight
-gfu,good for you
-gfy,go fuck yourself
-gg,good game
-ggs,good games
-gj,good job
-gl,good luck
-gmta,great minds think alike
-goin,going
-govt,government
-gr8,great
-gtfo,get the fuck out
-gt,get
-gtg,got to go
-gud,good
-hella,very
-hf,have fun
-hpe,hope
-hse,house
-hses,houses
-hth,hope this helps
-ianal,i am not a lawyer
-ibtl,in before the lock
-idk,i don't know
-idk,I don't know
-iht,i had to
-iirc,if i remember correctly
-iiuc,if i understand correctly
-ily,i love you
-i'ma,i'll
-ima,I'll
-imao,in my arrogant opinion
-imho,in my humble opinion
-imma,I'll
-imnsho,in my not so humble opinion
-imo,in my opinion
-incl,including
-inna,in a
-innit,isn't it
-int,isn't
-iono,i don't know
-iow,in other words
-irc,internet relay chat
-irl,in real life
-itym,i think you mean
-ive,I've
-iwsn,i want sex now
-iykwim,if you know what i mean
-jan,January
-jas,just a sec
-jftr,just for the record
-j/k,joking
-jk,joking
-jul,July
-jun,June
-jus,just
-kkk,ok
-kk,ok
-kno,know
-kos,kill on sight
-kthx,thanks
-l8r,later
-lfg,looking for group
-lfm,looking for more
-lik,like
-lmao,haha
-lmfao,haha
-lmirl,let's meet in real life
-lmk,let me know
-lst,last
-ltns,long time no see
-lurv,love
-lurve,love
-luv,love
-luvz,loves
-lv,love
-lyk,like
-lylab,love you like a brother
-lylas,love you like a sister
-m8,mate
-mar,March
-mfw,my face when
-mor,more
-motd,message of the day
-mtfbwy,may the force be with you
-myob,mind your own business
-n't,n't
-n1,nice one
-nd,and
-ne1,anyone
-ngl,not gonna lie
-nifoc,naked in front of computer
-nite,night
-nm,never mind
-nothin,nothing
-nov,November
-np,no problem
-nsfw,naughty
-nsoh,no sense of humor
-ntn,nothing
-nuff,enough
-nutn,nothing
-nvm,nevermind
-nw,now
-nyt,night
-nyts,nights
-obvblates,assuredly
-obv,obviously
-obvs,obviously
-oct,October
-ofn,old freaking news
-oic,oh i see
-omw,on my way
-onoz,oh no
-op,original poster
-os,operating system
-otb,off to bed
-ot,off topic
-otoh,on the other hand
-otp,on the phone
-pic,picture
-pics,pictures
-pita,pain in the ass
-plez,please
-plmk,please let me know
-pls,please
-plz,please
-pmsl,pissing myself laughing
-pov,point of view
-ppl,people
-ptkfgs,punch the keys for god's sake
-pwned,owned
-pw,password
-pz,peace
-rl,real life
-rms,ride me sideways
-rofl,haha
-rsn,real soon now
-rtfb,read the fucking binary
-rtfs,read the fucking source
-sep,September
-sfw,safe for work
-sho,sure
-sis,sister
-smh,shaking my head
-sohf,sense of humor failure
-sq,square
-stfu,shut the fuck up
-sth,something
-str8,straight
-sumn,something
-takin,taking
-tbf,time between failures
-tbh,to be honest
-tellin,telling
-tgif,thank god it's friday
-tg,that's great
-tha,the
-thn,then
-thnx,thanks
-thro,through
-thru,through
-tht,that
-thx,thanks
-tia,thanks in advance
-timwtk,this inquiring mind wants to know
-tins,there is no spoon
-tl;dr,too long; didn't read
-tmi,too much information
-tnx,thanks
-tos,terms of service
-totes,totally
-tryna,trying to
-ttbomk,to the best of my knowledge
-ttfn,ta ta for now
-ttyl,talk to you later
-ttyn,talk to you never
-ttys,talk to you soon
-twtr,twitter
-twt,tweet
-tymz,times
-ty,thankyou
-ty,thank you
-tyt,take your time
-tyvm,thank you very much
-ugo,you got owned
-urs,yours
-ur,you're
-utfse,use the fucking search engine
-vids,videos
-v.,very
-w8,wait
-wanna,want to
-wb,welcome back
-w/e,weekend
-w/e,whatever
-whts,what's
-wif,with
-wil,will
-wit,with
-w/o,without
-w/o,without
-wrt,with respect to
-wtb,want to buy
-wtb,want to buy
-wtf,what the fuck
-wtg,way to go
-wth,what the hell
-wts,want to sell
-wtt,want to trade
-wug,what you got?
-w/,with
-yagni,you ain't gonna need it.
-yagtoh,you are going to own him.
-ya,your
-yea,yeah
-yec,young earth creationism
-ygm,you've got mail.
-yhbt,you have been trolled.
-yhuu,you
-ykw,you know what?
-ymmv,your mileage may vary.
-yoyo,you're on your own
-ytmnd,you're the man now dog
-yuhuu,yes
-yuhu,yes
-yu,you
-yw,you're welcome.
+abbreviations.en.csv
+slang.en.csv
Modified: gate/trunk/plugins/Twitter/src/gate/twitter/Normaliser.java
===================================================================
--- gate/trunk/plugins/Twitter/src/gate/twitter/Normaliser.java 2016-11-06
10:01:40 UTC (rev 19719)
+++ gate/trunk/plugins/Twitter/src/gate/twitter/Normaliser.java 2016-11-06
20:36:04 UTC (rev 19720)
@@ -27,6 +27,8 @@
import gate.util.Files;
import gate.util.GateRuntimeException;
+import com.opencsv.CSVReader;
+
import pt.tumba.spell.*;
@CreoleResource(name = "Tweet Normaliser", comment = "Normalise texts in
tweets (convert into standard English spelling mistakes, colloquialisms, typing
variations and so on)", helpURL =
"http://gate.ac.uk/userguide/sec:social:twitter:prs";)
@@ -43,45 +45,48 @@
if(this.dictURL == null)
throw new ResourceInstantiationException("dict file not set");
+ String dictFilePath = null;
checker = new SpellChecker();
- BufferedReader dict_reader;
- BufferedReader orth_reader;
wordlist = new HashSet<String>();
orthmappings = new HashMap<String, String>();
- // start up dict reader & checker
+ // initialize spell checker - jaspell *requires* a File, it can't read from
+ // an arbitrary URL, so write a temp file if necessary
try {
- String filepath = Files.fileFromURL(dictURL).getAbsolutePath();
- dict_reader = new BufferedReader(new FileReader(filepath));
- checker.initialize(filepath);
+ dictFilePath = Files.fileFromURL(dictURL).getAbsolutePath();
+ } catch(IllegalArgumentException e) {
+ // dictURL is not a file: URL
+ try {
+ File tmpDictFile = Files.writeTempFile(dictURL.openStream());
+ dictFilePath = tmpDictFile.getAbsolutePath();
+ } catch(Exception e2) {
+ throw new ResourceInstantiationException("Could not write dictionary
to temp file", e2);
+ }
+ }
+ try {
+ checker.initialize(dictFilePath);
} catch(Exception e) {
- throw new ResourceInstantiationException(e);
+ throw new ResourceInstantiationException("Error initializing
spellchecker", e);
}
- // populate the wordlist
- String entry;
- try {
- while((entry = dict_reader.readLine()) != null) {
+ // read the wordlist
+ try(InputStream in = dictURL.openStream();
+ InputStreamReader inReader = new InputStreamReader(in); //, encoding);
// jaspell always uses default encoding, so we should match it
+ BufferedReader dictReader = new BufferedReader(inReader)) {
+ String entry;
+ while((entry = dictReader.readLine()) != null) {
String[] tokens = entry.split(" : ");
wordlist.add(tokens[0]);
}
- dict_reader.close();
- } catch(IOException e) {
+ } catch(Exception e) {
throw new ResourceInstantiationException(e);
}
// populate the common norm. lookup list
try {
- String filepath = Files.fileFromURL(orthURL).getAbsolutePath();
- orth_reader = new BufferedReader(new FileReader(filepath));
-
- while((entry = orth_reader.readLine()) != null) {
- String[] tokens = entry.split(",");
- orthmappings.put(tokens[0], tokens[1]);
- }
- orth_reader.close();
- } catch(Exception e) {
- throw new ResourceInstantiationException(e);
+ readOrthMappings(orthURL);
+ } catch(IOException e) {
+ throw new ResourceInstantiationException("Error loading orth mappings",
e);
}
dist = new LevenshteinDistance();
@@ -89,6 +94,35 @@
return this;
}
+ /**
+ * Read an orth mapping file. This file may be a "definition" file giving a
+ * list of other files to read, in which case each of those files will be
+ * read recursively.
+ */
+ protected void readOrthMappings(URL url) throws IOException {
+ try(InputStream in = url.openStream();
+ InputStreamReader reader = new InputStreamReader(in, encoding);
+ CSVReader csvReader = new CSVReader(reader)) {
+ String[] line = csvReader.readNext();
+ if(line == null) return;
+
+ if(line.length < 2) {
+ // definition file pointing to other lists
+ do {
+ String relpath = line[0].trim();
+ if(!"".equals(relpath)) {
+ readOrthMappings(new URL(url, relpath));
+ }
+ } while((line = csvReader.readNext()) != null);
+ } else {
+ // single list
+ do {
+ orthmappings.put(line[0], line[1]);
+ } while((line = csvReader.readNext()) != null);
+ }
+ }
+ }
+
@Override
public void execute() throws ExecutionException {
@@ -338,7 +372,10 @@
return this.dictURL;
}
- @CreoleParameter(comment = "Path to common normalisation terms list (for
orthographic mappings, e.g. 'b4' to 'before')", defaultValue =
"resources/normaliser/orth.en.csv")
+ @CreoleParameter(comment = "Path to common normalisation terms list (for
orthographic mappings, e.g. 'b4' to 'before'). "
+ + "This can either be a single two-column CSV file where the first
column is the term to be mapped and the second "
+ + "column is the target, or a single column file listing relative paths
to other lists (which is useful if you want "
+ + "to include more than one list of terms)", defaultValue =
"resources/normaliser/orth.en.csv")
public void setOrthURL(URL orthURL) {
this.orthURL = orthURL;
}
@@ -347,6 +384,15 @@
return this.orthURL;
}
+ @CreoleParameter(comment = "Character encoding used to read the orth files",
defaultValue = "UTF-8")
+ public void setEncoding(String encoding) {
+ this.encoding = encoding;
+ }
+
+ public String getEncoding() {
+ return this.encoding;
+ }
+
@RunTime
@Optional
@CreoleParameter(comment = "Maximum distance to consider (this determines
OOV/IV threshold).\nBased on Levenshtein edit dist (with a case change
downweighted to 0.5) and double-metaphone.", defaultValue = "2.0")
@@ -380,6 +426,8 @@
private URL orthURL;
+ private String encoding;
+
private double maxDistance;
}
This was sent by the SourceForge.net collaborative development platform, the
world's largest Open Source development site.
------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi
_______________________________________________
GATE-cvs mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/gate-cvs
