Re: UTF-8 Encoding in Jsp | RESOLVED
I concur, thanks for posting your findings. Also if I may ask: please don't change the subject of your mails. Those of us who view this list as a newsgroup get all messed up! Andoni. - Original Message - From: Shapira, Yoav [EMAIL PROTECTED] Newsgroups: gmane.comp.jakarta.tomcat.user Sent: Thursday, December 02, 2004 2:08 PM Subject: RE: UTF-8 Encoding in Jsp | RESOLVED Hi, Thanks for posting your findings ;) Yoav Shapira http://www.yoavshapira.com -Original Message- From: Arnab Chakravarty [mailto:[EMAIL PROTECTED] Sent: Thursday, December 02, 2004 9:03 AM To: Tomcat Users List Subject: RE: UTF-8 Encoding in Jsp | RESOLVED Hi all, First of all thanks to all the people who helped in the first place (I am grateful). The problem was resolved and was due to some problem with the home grown framework we were using with the application. Tomcat had nothing to do with the problem and content type is the only thing required to make it work. As far as the database persistence was concerned, oracle did no mistake in storing the data but when our framework was persisting the values, it somehow corrupted the data somewhere in the middle of submitting the page with non-english characters and writing to the database. We found this problem by simply writing a simple jsp page without using the framework and rendered some non-english characters successfully. Thanks again, Arnab -Original Message- From: Arnab Chakravarty [mailto:[EMAIL PROTECTED] Sent: Wednesday, December 01, 2004 4:08 PM To: Tomcat Users List Subject: RE: UTF-8 Encoding in Jsp Hi, Thanks for the reply but it did not work. May be I didn't explain the problem correctly. I am running an application that supports all the languages but only in some specific places of the application and I have made those places UTF-8 complaint. Further, they are being saved to Database (Oracle 9). When we are reading the data back from the database, junk characters are displayed on the screen. Yes, the database is set to support UTF-8 Encoding and this is working with the old version of tomcat 3.3 and not with current upgraded version of tomcat 5.0 There are also places in the application where drop downs contain some different language support and we can see those charsets (Japanese, Chinese etc) appearing. Only, when I try to display on the screen through the jsp file, I am encountering this problem of junk characters begin displayed. Hope I have set more context around the problem. Please help me resolve this issue. Thanks, Arnab -Original Message- From: Mariano [mailto:[EMAIL PROTECTED] Sent: Wednesday, December 01, 2004 12:54 PM To: 'Tomcat Users List' Subject: RE: UTF-8 Encoding in Jsp You should use too: head META http-equiv=Content-Type content=text/html; charset=UTF-8 /head and this scriptlet: request.setCharacterEncoding(UTF-8); at the beginning. I hope this help you -Mensaje original- De: Arnab Chakravarty [mailto:[EMAIL PROTECTED] Enviado el: martes, 30 de noviembre de 2004 15:28 Para: Tomcat Users List Asunto: UTF-8 Encoding in Jsp Hi all, I need to make my all jsp files compatible with UTF-8 Encoding and even though I am using the directives: %@ page pageEncoding=UTF-8% %@ page contentType = text/html;charset=UTF-8% in the jsp files, cannot make it work. Using tomcat version 5. Is there any config changes I need to make for the UTF-8 Encoding to work. Please help. Thanks in advance, Arnab - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] This e-mail, including any attachments, is a confidential business communication, and may contain information that is confidential, proprietary and/or privileged. This e-mail is intended only for the individual(s) to whom it is addressed, and may not be saved, copied, printed, disclosed or used by anyone else. If you are not the(an) intended recipient, please immediately delete this e-mail from your computer system and notify the sender. Thank you. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: UTF-8 Encoding in Jsp | RESOLVED
Hi all, First of all thanks to all the people who helped in the first place (I am grateful). The problem was resolved and was due to some problem with the home grown framework we were using with the application. Tomcat had nothing to do with the problem and content type is the only thing required to make it work. As far as the database persistence was concerned, oracle did no mistake in storing the data but when our framework was persisting the values, it somehow corrupted the data somewhere in the middle of submitting the page with non-english characters and writing to the database. We found this problem by simply writing a simple jsp page without using the framework and rendered some non-english characters successfully. Thanks again, Arnab -Original Message- From: Arnab Chakravarty [mailto:[EMAIL PROTECTED] Sent: Wednesday, December 01, 2004 4:08 PM To: Tomcat Users List Subject: RE: UTF-8 Encoding in Jsp Hi, Thanks for the reply but it did not work. May be I didn't explain the problem correctly. I am running an application that supports all the languages but only in some specific places of the application and I have made those places UTF-8 complaint. Further, they are being saved to Database (Oracle 9). When we are reading the data back from the database, junk characters are displayed on the screen. Yes, the database is set to support UTF-8 Encoding and this is working with the old version of tomcat 3.3 and not with current upgraded version of tomcat 5.0 There are also places in the application where drop downs contain some different language support and we can see those charsets (Japanese, Chinese etc) appearing. Only, when I try to display on the screen through the jsp file, I am encountering this problem of junk characters begin displayed. Hope I have set more context around the problem. Please help me resolve this issue. Thanks, Arnab -Original Message- From: Mariano [mailto:[EMAIL PROTECTED] Sent: Wednesday, December 01, 2004 12:54 PM To: 'Tomcat Users List' Subject: RE: UTF-8 Encoding in Jsp You should use too: head META http-equiv=Content-Type content=text/html; charset=UTF-8 /head and this scriptlet: request.setCharacterEncoding(UTF-8); at the beginning. I hope this help you -Mensaje original- De: Arnab Chakravarty [mailto:[EMAIL PROTECTED] Enviado el: martes, 30 de noviembre de 2004 15:28 Para: Tomcat Users List Asunto: UTF-8 Encoding in Jsp Hi all, I need to make my all jsp files compatible with UTF-8 Encoding and even though I am using the directives: %@ page pageEncoding=UTF-8% %@ page contentType = text/html;charset=UTF-8% in the jsp files, cannot make it work. Using tomcat version 5. Is there any config changes I need to make for the UTF-8 Encoding to work. Please help. Thanks in advance, Arnab - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: UTF-8 Encoding in Jsp | RESOLVED
Hi, Thanks for posting your findings ;) Yoav Shapira http://www.yoavshapira.com -Original Message- From: Arnab Chakravarty [mailto:[EMAIL PROTECTED] Sent: Thursday, December 02, 2004 9:03 AM To: Tomcat Users List Subject: RE: UTF-8 Encoding in Jsp | RESOLVED Hi all, First of all thanks to all the people who helped in the first place (I am grateful). The problem was resolved and was due to some problem with the home grown framework we were using with the application. Tomcat had nothing to do with the problem and content type is the only thing required to make it work. As far as the database persistence was concerned, oracle did no mistake in storing the data but when our framework was persisting the values, it somehow corrupted the data somewhere in the middle of submitting the page with non-english characters and writing to the database. We found this problem by simply writing a simple jsp page without using the framework and rendered some non-english characters successfully. Thanks again, Arnab -Original Message- From: Arnab Chakravarty [mailto:[EMAIL PROTECTED] Sent: Wednesday, December 01, 2004 4:08 PM To: Tomcat Users List Subject: RE: UTF-8 Encoding in Jsp Hi, Thanks for the reply but it did not work. May be I didn't explain the problem correctly. I am running an application that supports all the languages but only in some specific places of the application and I have made those places UTF-8 complaint. Further, they are being saved to Database (Oracle 9). When we are reading the data back from the database, junk characters are displayed on the screen. Yes, the database is set to support UTF-8 Encoding and this is working with the old version of tomcat 3.3 and not with current upgraded version of tomcat 5.0 There are also places in the application where drop downs contain some different language support and we can see those charsets (Japanese, Chinese etc) appearing. Only, when I try to display on the screen through the jsp file, I am encountering this problem of junk characters begin displayed. Hope I have set more context around the problem. Please help me resolve this issue. Thanks, Arnab -Original Message- From: Mariano [mailto:[EMAIL PROTECTED] Sent: Wednesday, December 01, 2004 12:54 PM To: 'Tomcat Users List' Subject: RE: UTF-8 Encoding in Jsp You should use too: head META http-equiv=Content-Type content=text/html; charset=UTF-8 /head and this scriptlet: request.setCharacterEncoding(UTF-8); at the beginning. I hope this help you -Mensaje original- De: Arnab Chakravarty [mailto:[EMAIL PROTECTED] Enviado el: martes, 30 de noviembre de 2004 15:28 Para: Tomcat Users List Asunto: UTF-8 Encoding in Jsp Hi all, I need to make my all jsp files compatible with UTF-8 Encoding and even though I am using the directives: %@ page pageEncoding=UTF-8% %@ page contentType = text/html;charset=UTF-8% in the jsp files, cannot make it work. Using tomcat version 5. Is there any config changes I need to make for the UTF-8 Encoding to work. Please help. Thanks in advance, Arnab - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] This e-mail, including any attachments, is a confidential business communication, and may contain information that is confidential, proprietary and/or privileged. This e-mail is intended only for the individual(s) to whom it is addressed, and may not be saved, copied, printed, disclosed or used by anyone else. If you are not the(an) intended recipient, please immediately delete this e-mail from your computer system and notify the sender. Thank you. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: UTF-8 Encoding in Jsp
Hi, Thanks for the reply but it did not work. May be I didn't explain the problem correctly. I am running an application that supports all the languages but only in some specific places of the application and I have made those places UTF-8 complaint. Further, they are being saved to Database (Oracle 9). When we are reading the data back from the database, junk characters are displayed on the screen. Yes, the database is set to support UTF-8 Encoding and this is working with the old version of tomcat 3.3 and not with current upgraded version of tomcat 5.0 There are also places in the application where drop downs contain some different language support and we can see those charsets (Japanese, Chinese etc) appearing. Only, when I try to display on the screen through the jsp file, I am encountering this problem of junk characters begin displayed. Hope I have set more context around the problem. Please help me resolve this issue. Thanks, Arnab -Original Message- From: Mariano [mailto:[EMAIL PROTECTED] Sent: Wednesday, December 01, 2004 12:54 PM To: 'Tomcat Users List' Subject: RE: UTF-8 Encoding in Jsp You should use too: head META http-equiv=Content-Type content=text/html; charset=UTF-8 /head and this scriptlet: request.setCharacterEncoding(UTF-8); at the beginning. I hope this help you -Mensaje original- De: Arnab Chakravarty [mailto:[EMAIL PROTECTED] Enviado el: martes, 30 de noviembre de 2004 15:28 Para: Tomcat Users List Asunto: UTF-8 Encoding in Jsp Hi all, I need to make my all jsp files compatible with UTF-8 Encoding and even though I am using the directives: %@ page pageEncoding=UTF-8% %@ page contentType = text/html;charset=UTF-8% in the jsp files, cannot make it work. Using tomcat version 5. Is there any config changes I need to make for the UTF-8 Encoding to work. Please help. Thanks in advance, Arnab - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: UTF-8 Encoding in Jsp
Hi, These encoding issues are always a nightmare ;) There are some relevant areas of the Servlet spec you may want to look at wrt encoding, notably (Internationalization and Request data encoding). In terms of UTF-8 not coming back correctly from your database you need to ensure that when they were _added_ that the character encoding was UTF-8. You should also verify yuor database is in UTF-8 mode. If both these statements are true, then you need to read Internationalization in the Servlet spec which says If the servlet does not specify a character encoding before the getWriter method of the ServletResponse interface is called or the response is committed, the default ISO-8859-1 is used. In other words, you need to call setLocale or setCharacterEncoding before the response is committed. I am not entirely sure whether that is actually what that JSP page directive is doing, maybe it is. Perhaps in your JSP you can output %= request.getCharacterEncoding() % to make sure your UTF-8 has been set. If it is null, it has not been set. If it _is_ UTF-8 then the character data is either not actually UTF-8 coming from the database either because a) your database driver connection URL is not operating in UTF-8 mode, b) the data when put into the database was not UTF-8 or c) the database is not running UTF-8. In terms of sending data to the database as UTF-8 check your driver parameters (normally on the URL string) and also database setting. You also need to take note of this section of the Servlet spec. We had to write a servlet filter to change our inbound form posts to the correct encoding for our database Cp1252. Request data encoding extract The default encoding of a request the container uses to create the request reader and parse POST data must be ISO-8859-1 if none has been specified by the client request. However, in order to indicate to the developer in this case the failure of the client to send a character encoding, the container returns null from the getCharacterEncoding method. If the client hasn't set character encoding and the request data is encoded with a different encoding than the default as described above, breakage can occur. To remedy this situation, a new method setCharacterEncoding(String enc) has been added to the ServletRequest interface. Developers can override the character encoding supplied by the container by calling this method. It must be called prior to parsing any post data or reading any input from the request. Hope this info gets you thinking, Allistair. -Original Message- From: Arnab Chakravarty [mailto:[EMAIL PROTECTED] Sent: 01 December 2004 10:38 To: Tomcat Users List Subject: RE: UTF-8 Encoding in Jsp Hi, Thanks for the reply but it did not work. May be I didn't explain the problem correctly. I am running an application that supports all the languages but only in some specific places of the application and I have made those places UTF-8 complaint. Further, they are being saved to Database (Oracle 9). When we are reading the data back from the database, junk characters are displayed on the screen. Yes, the database is set to support UTF-8 Encoding and this is working with the old version of tomcat 3.3 and not with current upgraded version of tomcat 5.0 There are also places in the application where drop downs contain some different language support and we can see those charsets (Japanese, Chinese etc) appearing. Only, when I try to display on the screen through the jsp file, I am encountering this problem of junk characters begin displayed. Hope I have set more context around the problem. Please help me resolve this issue. Thanks, Arnab -Original Message- From: Mariano [mailto:[EMAIL PROTECTED] Sent: Wednesday, December 01, 2004 12:54 PM To: 'Tomcat Users List' Subject: RE: UTF-8 Encoding in Jsp You should use too: head META http-equiv=Content-Type content=text/html; charset=UTF-8 /head and this scriptlet: request.setCharacterEncoding(UTF-8); at the beginning. I hope this help you -Mensaje original- De: Arnab Chakravarty [mailto:[EMAIL PROTECTED] Enviado el: martes, 30 de noviembre de 2004 15:28 Para: Tomcat Users List Asunto: UTF-8 Encoding in Jsp Hi all, I need to make my all jsp files compatible with UTF-8 Encoding and even though I am using the directives: %@ page pageEncoding=UTF-8% %@ page contentType = text/html;charset=UTF-8% in the jsp files, cannot make it work. Using tomcat version 5. Is there any config changes I need to make for the UTF-8 Encoding to work. Please help. Thanks in advance, Arnab - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED
Re: UTF-8 Encoding in Jsp
I would recommend that you make the Entire Site UTF-8. The parts that are in English will still work no problem but I would really not try mixing the encoding for requests. The junk characters you are getting back are also not actually junk. You can work out what encoding is being used by interpreting these string and knowing what the intended string is. Also the fact that you are not just getting lots of ? characters means that it is not Oracle that is having the problem. I will read the other reply when I get a chance and see if I have any further contributions but for now I really strenuously suggest making ALL the pages/servlets UTF-8. Regards, Andoni. - Original Message - From: Arnab Chakravarty [EMAIL PROTECTED] Newsgroups: gmane.comp.jakarta.tomcat.user Sent: Wednesday, December 01, 2004 10:37 AM Subject: RE: UTF-8 Encoding in Jsp Hi, Thanks for the reply but it did not work. May be I didn't explain the problem correctly. I am running an application that supports all the languages but only in some specific places of the application and I have made those places UTF-8 complaint. Further, they are being saved to Database (Oracle 9). When we are reading the data back from the database, junk characters are displayed on the screen. Yes, the database is set to support UTF-8 Encoding and this is working with the old version of tomcat 3.3 and not with current upgraded version of tomcat 5.0 There are also places in the application where drop downs contain some different language support and we can see those charsets (Japanese, Chinese etc) appearing. Only, when I try to display on the screen through the jsp file, I am encountering this problem of junk characters begin displayed. Hope I have set more context around the problem. Please help me resolve this issue. Thanks, Arnab -Original Message- From: Mariano [mailto:[EMAIL PROTECTED] Sent: Wednesday, December 01, 2004 12:54 PM To: 'Tomcat Users List' Subject: RE: UTF-8 Encoding in Jsp You should use too: head META http-equiv=Content-Type content=text/html; charset=UTF-8 /head and this scriptlet: request.setCharacterEncoding(UTF-8); at the beginning. I hope this help you -Mensaje original- De: Arnab Chakravarty [mailto:[EMAIL PROTECTED] Enviado el: martes, 30 de noviembre de 2004 15:28 Para: Tomcat Users List Asunto: UTF-8 Encoding in Jsp Hi all, I need to make my all jsp files compatible with UTF-8 Encoding and even though I am using the directives: %@ page pageEncoding=UTF-8% %@ page contentType = text/html;charset=UTF-8% in the jsp files, cannot make it work. Using tomcat version 5. Is there any config changes I need to make for the UTF-8 Encoding to work. Please help. Thanks in advance, Arnab - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: UTF-8 Encoding in Jsp
Hello, First and foremost I would say: be absolutely sure that it is the JSP's fault. I hope you are not getting some data from a database and trying to show it? Be sure that your editor is saving the JSP in UTF-8 format. Add the flag: -Dfile.encoding=UTF-8 to the CATALINA_OPTS environment variable in your catalina.bat (or equivalent) startup file. and use: req.setCharacterEncoding(UTF-8); to set the encoding on the request. This may help: http://marc.theaimsgroup.com/?l=tomcat-userm=105524550416364w=2 Though you can ignore the method wich is used to set the encoding as the above line does the same job in servlets. Andoni. - Original Message - From: Arnab Chakravarty [EMAIL PROTECTED] Newsgroups: gmane.comp.jakarta.tomcat.user Sent: Tuesday, November 30, 2004 2:28 PM Subject: UTF-8 Encoding in Jsp Hi all, I need to make my all jsp files compatible with UTF-8 Encoding and even though I am using the directives: %@ page pageEncoding=UTF-8% %@ page contentType = text/html;charset=UTF-8% in the jsp files, cannot make it work. Using tomcat version 5. Is there any config changes I need to make for the UTF-8 Encoding to work. Please help. Thanks in advance, Arnab - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: UTF-8 Encoding in Jsp
You should use too: head META http-equiv=Content-Type content=text/html; charset=UTF-8 /head and this scriptlet: request.setCharacterEncoding(UTF-8); at the beginning. I hope this help you -Mensaje original- De: Arnab Chakravarty [mailto:[EMAIL PROTECTED] Enviado el: martes, 30 de noviembre de 2004 15:28 Para: Tomcat Users List Asunto: UTF-8 Encoding in Jsp Hi all, I need to make my all jsp files compatible with UTF-8 Encoding and even though I am using the directives: %@ page pageEncoding=UTF-8% %@ page contentType = text/html;charset=UTF-8% in the jsp files, cannot make it work. Using tomcat version 5. Is there any config changes I need to make for the UTF-8 Encoding to work. Please help. Thanks in advance, Arnab - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: UTF-8 Encoding Issue Since 5.0.27 ( gun in my mouth )
Thanks for you info Mark.
It does appear that a part of my issue stems from my .properties files
being in UTF-8.
So I have to ask the question, why has this changed since if I run the same
code in 5.0.24, I have no issue, and 5.0.28 has a problem. It sounds like
a substantial problem that UTF-8 resource bundles aren't supported any more.
Besides this simple example, I'm still seeing problems with a servlet
returning XML in UTF-8. Again, no issue in 5.0.24, only after 5.0.25.
I will put together a sample and post it shortly.
Thanks again for the help,
Rick
-Original Message-
From: Mark Thomas [mailto:[EMAIL PROTECTED]
Posted At: Wednesday, September 01, 2004 4:14 PM
Posted To: Tomcat Dev
Conversation: UTF-8 Encoding Issue Since 5.0.27 ( gun in my mouth )
Subject: RE: UTF-8 Encoding Issue Since 5.0.27 ( gun in my mouth )
OK. I have a simple test case and all seems to be well. See the end of this
message for the contents of my test files.
My environment:
Win XP SP2 - brave I know but all has been OK so far ;) JDK 1.4.2_05 Tomcat
5.0 branch, HEAD (latest) from CVS (very close to 5.0.28)
Points to note:
1. All my test files are ASCII files.
2. I had all sorts of problems with non-ASCII properties files. I didn't get
to the bottom of it but I think Windows was adding junk to the start of the
file if it was UTF-8 encoded. Maybe having the first line as a comment would
fix this but I haven't tested this.
3. There were times where Eclipse and Windows were reporting the exact same
file as having different encodings. There is something odd here but I didn't
look at this any further.
4. I had property file issues with 4.1.HEAD as well as 5.0.HEAD.
5. The downside of using ASCII files is that entering the UTF-8 characters
by hand is a real pain. A simple conversion app should fix this though.
6. Apart from the property file issue, everything seems fine.
Test files follow.
Hope this helps,
Mark
PS I noticed that you cross-posted to the dev list. Please don't do this.
Any message cross-posted is less likely rather than more likely to get a
response.
=== utf8.jsp
%@ page language=java import=java.lang.*,java.util.*
contentType=text/html; charset=UTF-8 % !DOCTYPE HTML PUBLIC -//W3C//DTD
HTML 4.01 Transitional//EN html
head
titleUTF-8 Encoding issue/title
/head
body
pText from JSP page (which is ASCII encoded)./p
form action=utf8.jsp method=post
pEnglishinput type=radio value=en name=language /p
pJapaneseinput type=radio value=ja name=language /p
input type=submit value=Post form data /
/form
pText from resources bundle:/p
%
String language = request.getParameter(language);
if (language == null) {
language=en;
}
Locale locale = null;
if (language.equalsIgnoreCase(en)) {
locale = Locale.ENGLISH;
} else {
locale = Locale.JAPAN;
}
ResourceBundle bundle =
ResourceBundle.getBundle(foo.bar.LocalStrings,
locale);
out.println(p + bundle.getString(test) + /p);
%
p%=request.getParameter(language) %/p
/body
/html
= LocalStrings_en.properties =
test=Test string from resources bundle
= LocalStrings_ja.properties =
test=\u30d5\u30a1\u30a4\u30eb\u30ed
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
RE: UTF-8 Encoding Issue Since 5.0.27 ( gun in my mouth )
The change (which is required by the spec) is that if the character set has not been set before a call to getWriter() then it will default to ISO-8859-1. There was some discussion on the tomcat-dev list about this (see http://marc.theaimsgroup.com/?l=tomcat-devm=109104739719572w=2) I'll try and put together a very simple JSP test case and get back to you. Mark -Original Message- From: Rick [mailto:[EMAIL PROTECTED] Sent: Wednesday, September 01, 2004 3:44 AM To: 'Tomcat Users List'; [EMAIL PROTECTED] Subject: UTF-8 Encoding Issue Since 5.0.27 ( gun in my mouth ) Since 5.0.27, pretty much all of my UTF-8 i8 code seems to be messed up. The problem seems to have been caused by whatever fix was created for issue -- ServletResponse.setContentType sets response encoding after getWriter was called (Bugtraq 5062838) (luehe) -- Now it seems almost impossible to properly set the encoding type of some of my JSPs and all of my Servlets that return UTF-8 XML data. As an example, my login page allows the user to switch to Japanese text. Text data is read with a ResourceBundle, which reads from a UTF-8 encoded .properties file. If the encoding of the .jsp page itself is in ASCII, then I can't get the characters to show up at all any more. I have to save the .jsp page as UTF-8. Added set JAVA_OPTS=-Dfile.encoding=UTF-8 to my catalina.bat file Then, If I try to set a character set in my page header, it messes up. This works in some cases... %@ page language=java import=java.util.* contentType=text/html % response.getCharacterEncoding() = ISO-8859-1 The really scary part is that with no meta or charset actually set, that the browser(IE) correctly changes to UTF-8 and displays the content fine. But if I change the actual file encoding of the .jsp page from UTF-8 back to ASCII. Then IE does not change to UTF-8 and the page is messed up again. Why does the actual encoding of the .jsp file itself dictate the response sent to the client? It appears that the actual encoding of the source file someone how gets past along and then I'm unable to alter the character encoding, and if I try, it just causes everything to go to hell. This use to work before 5.0.27, but now doesn't, even though all data and pages are encoded in UTF-8. %@ page language=java import=java.util.* contentType=text/html; charset=UTF-8 % response.getCharacterEncoding() = UTF-8 Before 5.0.27, all I had to do to get my output in UTF-8 was ... contentType=text/html; charset=UTF-8 Now I have to mess with the actual .jsp file page encodings and still can't get most to work properly as well as none of my servlets will return correct UTF-8 data. I have tried setting pageEncoding in the page tag as well with no luck. Thanks for anyone's insight or help on this, its never fun to find out that something that had been working quite solid , up and blows up for no good reason. Current dev machine is on windows xp by the way, vanilla install of Tomcat 5.0.28. I will be setting this up on a Linux box for more testing shortly. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: UTF-8 Encoding Issue Since 5.0.27 ( gun in my mouth )
OK. I have a simple test case and all seems to be well. See the end of this
message for the contents of my test files.
My environment:
Win XP SP2 - brave I know but all has been OK so far ;)
JDK 1.4.2_05
Tomcat 5.0 branch, HEAD (latest) from CVS (very close to 5.0.28)
Points to note:
1. All my test files are ASCII files.
2. I had all sorts of problems with non-ASCII properties files. I didn't get to
the bottom of it but I think Windows was adding junk to the start of the file if
it was UTF-8 encoded. Maybe having the first line as a comment would fix this
but I haven't tested this.
3. There were times where Eclipse and Windows were reporting the exact same file
as having different encodings. There is something odd here but I didn't look at
this any further.
4. I had property file issues with 4.1.HEAD as well as 5.0.HEAD.
5. The downside of using ASCII files is that entering the UTF-8 characters by
hand is a real pain. A simple conversion app should fix this though.
6. Apart from the property file issue, everything seems fine.
Test files follow.
Hope this helps,
Mark
PS I noticed that you cross-posted to the dev list. Please don't do this. Any
message cross-posted is less likely rather than more likely to get a response.
=== utf8.jsp
%@ page language=java import=java.lang.*,java.util.*
contentType=text/html; charset=UTF-8 %
!DOCTYPE HTML PUBLIC -//W3C//DTD HTML 4.01 Transitional//EN
html
head
titleUTF-8 Encoding issue/title
/head
body
pText from JSP page (which is ASCII encoded)./p
form action=utf8.jsp method=post
pEnglishinput type=radio value=en name=language /p
pJapaneseinput type=radio value=ja name=language /p
input type=submit value=Post form data /
/form
pText from resources bundle:/p
%
String language = request.getParameter(language);
if (language == null) {
language=en;
}
Locale locale = null;
if (language.equalsIgnoreCase(en)) {
locale = Locale.ENGLISH;
} else {
locale = Locale.JAPAN;
}
ResourceBundle bundle = ResourceBundle.getBundle(foo.bar.LocalStrings,
locale);
out.println(p + bundle.getString(test) + /p);
%
p%=request.getParameter(language) %/p
/body
/html
= LocalStrings_en.properties =
test=Test string from resources bundle
= LocalStrings_ja.properties =
test=\u30d5\u30a1\u30a4\u30eb\u30ed
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Re: UTF-8 encoding
Hello Nikki Just send UTF8 encoded data and everything will be allright. Yes, that seems to work for me at the moment, though I am relying on default settings because I do not even specify UTF-8. (Java defaults to Unicode anyway.) I'm only using LATIN-1 characters at the moment, so I cannot comment on what would happen if I was working with (say) Chinese characters. I have to leave it at that because this is something I shall be looking into later. All the best! Harry Simply I don't get it. You send data over HTTP. You can send data as you wish. What about servlet serving images? Just send UTF8 encoded data and everything will be allright. No way Tomcat knows do you want to send cyrrilic letter or french accent letter. It's up to you. Niki Harry Mantheakis wrote: - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: UTF-8 encoding
Simply I don't get it. You send data over HTTP. You can send data as you wish. What about servlet serving images? Just send UTF8 encoded data and everything will be allright. No way Tomcat knows do you want to send cyrrilic letter or french accent letter. It's up to you. Niki Harry Mantheakis wrote: Okay, thanks Yoav. I got the source, and I can see what's happening - thanks to Google - at this URL: http://java.sun.com/blueprints/code/jps131/src/com/sun/j2ee/blueprints/encod ingfilter/web/EncodingFilter.java.html The 'doFilter' method sets the encoding for the *request* which does not seem to address the original question, which was asking how to 'force tomcat to send data in UTF-8 encoding'. Interesting filter nevertheless! It is a subject that concerns me. Kind regards Harry Hi, implement a EncodingFilter class Where's the interface? javax.servlet.Filter is the interface. He probably had http://java.sun.com/blueprints/code/jps131/api/com/sun/j2ee/blueprints/e ncodingfilter/web/EncodingFilter.html in mind. Yoav Shapira - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: UTF-8 encoding
Hi, you can specify the utf-8 encoding with a filter. All you need to do is implement a EncodingFilter class, and then in your deployment descriptor add the filter element as follows: filter filter-nameEncodingFilter/filter-name display-nameEncodingFilter/display-name descriptionUTF-8 encoding/description filter-classorg.mysite.EncodingFilter/filter-class init-param param-nametargetEncoding/param-name param-valueutf-8/param-value /init-param /filter Hope this helps:) -Yan -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Sent: Monday, April 05, 2004 6:49 AM To: Tomcat Users List Subject: UTF-8 encoding Hi! I have a web-application which on the serverside needs UTF-8 encoding. I tried to install and run apache/tomcat on a Windows-XP environment, and the server says, the encoding is not UTF-8. same applicationwith the same apache/tomcat version runs correctly on a windows 2000 environment. Is this a XP specific problem and is there any possibility to force tomcat to send data in UTF-8 encoding. Best regards bab - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: UTF-8 encoding
implement a EncodingFilter class Where's the interface? Hi, you can specify the utf-8 encoding with a filter. All you need to do is implement a EncodingFilter class, and then in your deployment descriptor add the filter element as follows: filter filter-nameEncodingFilter/filter-name display-nameEncodingFilter/display-name descriptionUTF-8 encoding/description filter-classorg.mysite.EncodingFilter/filter-class init-param param-nametargetEncoding/param-name param-valueutf-8/param-value /init-param /filter Hope this helps:) -Yan -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Sent: Monday, April 05, 2004 6:49 AM To: Tomcat Users List Subject: UTF-8 encoding Hi! I have a web-application which on the serverside needs UTF-8 encoding. I tried to install and run apache/tomcat on a Windows-XP environment, and the server says, the encoding is not UTF-8. same applicationwith the same apache/tomcat version runs correctly on a windows 2000 environment. Is this a XP specific problem and is there any possibility to force tomcat to send data in UTF-8 encoding. Best regards bab - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: UTF-8 encoding
Hi, implement a EncodingFilter class Where's the interface? javax.servlet.Filter is the interface. He probably had http://java.sun.com/blueprints/code/jps131/api/com/sun/j2ee/blueprints/e ncodingfilter/web/EncodingFilter.html in mind. Yoav Shapira This e-mail, including any attachments, is a confidential business communication, and may contain information that is confidential, proprietary and/or privileged. This e-mail is intended only for the individual(s) to whom it is addressed, and may not be saved, copied, printed, disclosed or used by anyone else. If you are not the(an) intended recipient, please immediately delete this e-mail from your computer system and notify the sender. Thank you. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: UTF-8 encoding
javax.servlet.Filter is the interface. He probably had http://java.sun.com/blueprints/code/jps131/api/com/sun/j2ee/blueprints/ e ncodingfilter/web/EncodingFilter.html in mind. BTW, swap .java for .html (or google with the above) to see the full java source code for the blueprint encoding filter implementation. Yoav Shapira This e-mail, including any attachments, is a confidential business communication, and may contain information that is confidential, proprietary and/or privileged. This e-mail is intended only for the individual(s) to whom it is addressed, and may not be saved, copied, printed, disclosed or used by anyone else. If you are not the(an) intended recipient, please immediately delete this e-mail from your computer system and notify the sender. Thank you. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: UTF-8 encoding
Okay, thanks Yoav. I got the source, and I can see what's happening - thanks to Google - at this URL: http://java.sun.com/blueprints/code/jps131/src/com/sun/j2ee/blueprints/encod ingfilter/web/EncodingFilter.java.html The 'doFilter' method sets the encoding for the *request* which does not seem to address the original question, which was asking how to 'force tomcat to send data in UTF-8 encoding'. Interesting filter nevertheless! It is a subject that concerns me. Kind regards Harry Hi, implement a EncodingFilter class Where's the interface? javax.servlet.Filter is the interface. He probably had http://java.sun.com/blueprints/code/jps131/api/com/sun/j2ee/blueprints/e ncodingfilter/web/EncodingFilter.html in mind. Yoav Shapira - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
