Revision: 19729
http://sourceforge.net/p/gate/code/19729
Author: dgmaynard
Date: 2016-11-09 10:56:04 +0000 (Wed, 09 Nov 2016)
Log Message:
-----------
now with an additional dictionary of common misspellings, which improves
performance; also some improvements to the slang dictionary
Modified Paths:
--------------
gate/trunk/plugins/Twitter/resources/normaliser/orth.en.csv
gate/trunk/plugins/Twitter/resources/normaliser/slang.en.csv
Added Paths:
-----------
gate/trunk/plugins/Twitter/resources/normaliser/misspellings.csv
Added: gate/trunk/plugins/Twitter/resources/normaliser/misspellings.csv
===================================================================
--- gate/trunk/plugins/Twitter/resources/normaliser/misspellings.csv
(rev 0)
+++ gate/trunk/plugins/Twitter/resources/normaliser/misspellings.csv
2016-11-09 10:56:04 UTC (rev 19729)
@@ -0,0 +1,242 @@
+absense,absence
+absance,absence
+acceptible,acceptable
+accidentaly,accidentally
+accomodate,accommodate
+acommodate,accommodate
+acheive,achieve
+acknowlege,acknowledge
+aknowledge,acknowledge
+acquaintence,acquaintance
+aquaintance,acquaintance
+aquire,acquire
+adquire,acquire
+aquit,acquit
+acrage,acreage
+acerage,acreage
+adress,address
+adultary,adultery
+adviseable,advisable
+advizable,advisable
+agression,aggression
+agressive,aggressive
+allegaince,allegiance
+allegience,allegiance
+alegiance,allegiance
+allmost,almost
+amatuer,amateur
+amature,amateur
+anually,annually
+annualy,annually
+apparant,apparent
+aparent,apparent
+artic,arctic
+arguement,argument
+athiest,atheist
+aweful,awful
+awfull,awful
+becuase,because
+becomeing,becoming
+beginning,begining
+beleive,believe
+bouy,buoy
+bouyant,buoyant
+buisness,business
+calender,calendar
+camoflage,camouflage
+camoflague,camouflage
+Carribean,Caribbean
+catagory,category
+cauhgt,caught
+caugt,caught
+cemetary,cemetery
+cematery,cemetery
+changable,changeable
+cheif,chief
+collaegue,colleague
+collegue,colleague
+collectable,collectible
+colum,column
+comming,coming
+commited,committed
+comitted,committed
+conceed,concede
+congradulate,congratulate
+consciencious,conscientious
+concious,conscious
+consious,conscious
+concensus,consensus
+contraversy,controversy
+cooly,coolly
+dacquiri,daiquiri
+daquiri,daiquiri
+decieve,deceive
+definate,definite
+definit,definite
+definitly,definitely
+definately,definitely
+defiantly,definitely
+desparate,desperate
+diffrence,difference
+dilema,dilemma
+disapoint,disappoint
+disasterous,disastrous
+drunkeness,drunkenness
+dumbell,dumbbell
+embarass,embarrass
+equiptment,equipment
+excede,exceed
+exilerate,exhilarate
+existance,existence
+experiance,experience
+extreem,extreme
+facinating,fascinating
+firey,fiery
+flourescent,fluorescent
+foriegn,foreign
+freind,friend
+fullfil,fulfil
+guage,gauge
+gratefull,grateful
+greatful,grateful
+garantee,guarantee
+garentee,guarantee
+garanty,guarantee
+guidence,guidance
+harrass,harass
+heighth,height
+heigth,height
+heirarchy,hierarchy
+humerous,humorous
+hygene,hygiene
+hygine,hygiene
+hiygeine,hygiene
+higeine,hygiene
+hygeine,hygiene
+hipocrit,hypocrite
+ignorence,ignorance
+immitate,imitate
+imediately,immediately
+independant,independent
+indispensible,indispensable
+innoculate,inoculate
+inteligence,intelligence
+intelligance,intelligence
+jewelery,jewellery
+kernal,kernel
+liesure,leisure
+liason,liaison
+libary,library
+liberry,library
+lisence,licence
+lightening,lightning
+maintainance,maintenance
+maintnance,maintenance
+medeval,medieval
+medevil,medieval
+mideval,medieval
+momento,memento
+millenium,millennium
+milennium,millennium
+miniture,miniature
+miniscule,minuscule
+mischievious,mischievous
+mischevous,mischievous
+mischevious,mischievous
+mispell,misspell
+misspel,misspell
+neccessary,necessary
+necessery,necessary
+neice,niece
+nieghbour,neighbour
+noticable,noticeable
+occassion,occasion
+occasionaly,occasionally
+occassionally,occasionally
+occurrance,occurrence
+occurence,occurrence
+occured,occurred
+ommision,omission
+omision,omission
+orignal,original
+outragous,outrageous
+parliment,parliament
+passtime,pastime
+pasttime,pastime
+percieve,perceive
+perseverence,perseverance
+personell,personnel
+personel,personal
+plagerize,plagiarize
+playright,playwright
+playwrite,playwright
+posession,possession
+possesion,possession
+potatos,potatoes
+preceed,precede
+presance,presence
+privelege,privilege
+priviledge,privilege
+professer,professor
+promiss,promise
+pronounciation,pronunciation
+prufe,proof
+publically,publicly
+quarentine,quarantine
+questionaire,questionnaire
+questionnair,questionnaire
+readible,readable
+realy,really
+recieve,receive
+reciept,receipt
+recomend,recommend
+reccommend,recommend
+refered,referred
+referance,reference
+refrence,reference
+relevent,relevant
+ revelant,relevant
+religous,religious
+religius,religious
+repitition,repetition
+restarant,restaurant
+restaraunt,restaurant
+rime,rhyme
+ryme,rhyme
+rythm,rhythm
+rythem,rhythm
+secratary,secretary
+secretery,secretary
+sieze,seize
+seperate,separate
+sargent,sergeant
+similer,similar
+skilfull,skilful
+speach,speech
+succesful,successful
+successfull,successful
+sucessful,successful
+supercede,supersede
+suprise,surprise
+surprize,surprise
+tomatos,tomatoes
+tommorrow,tomorrow
+twelth,twelfth
+tyrany,tyranny
+underate,underrate
+untill,until
+upholstry,upholstery
+usible,usable
+vaccuum,vacuum
+vaccum,vacuum
+vacume,vacuum
+vehical,vehicle
+visious,vicious
+wierd,weird
+wellfare,welfare
+welfair,welfare
+wether,whether
+wilfull,wilful
+withold,withhold
+writting,writing
+writeing,writing
\ No newline at end of file
Modified: gate/trunk/plugins/Twitter/resources/normaliser/orth.en.csv
===================================================================
--- gate/trunk/plugins/Twitter/resources/normaliser/orth.en.csv 2016-11-09
02:23:09 UTC (rev 19728)
+++ gate/trunk/plugins/Twitter/resources/normaliser/orth.en.csv 2016-11-09
10:56:04 UTC (rev 19729)
@@ -1,2 +1,3 @@
abbreviations.en.csv
slang.en.csv
+misspellings.csv
\ No newline at end of file
Modified: gate/trunk/plugins/Twitter/resources/normaliser/slang.en.csv
===================================================================
--- gate/trunk/plugins/Twitter/resources/normaliser/slang.en.csv
2016-11-09 02:23:09 UTC (rev 19728)
+++ gate/trunk/plugins/Twitter/resources/normaliser/slang.en.csv
2016-11-09 10:56:04 UTC (rev 19729)
@@ -127,6 +127,7 @@
n't,n't
nd,and
ne1,anyone
+nigga,nigger
nite,night
nothin,nothing
nov,November
@@ -163,6 +164,7 @@
tha,the
thn,then
thnx,thanks
+tho,though
thro,through
thru,through
tht,that
This was sent by the SourceForge.net collaborative development platform, the
world's largest Open Source development site.
------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi
_______________________________________________
GATE-cvs mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/gate-cvs
