I don't know much about DB support that others have been commenting on, but in your snippet of code, if longDesc is already a Java String (that is UniCode),
then I'm not sure you can "convert" it as easily as you think ... you might try using
longDesc.getBytes("UTF8")
instead of just 'getBytes()'.
By default, getBytes() will use the plaform's default encoding, which might cause loss of data or misinterpretation of the bytes that make up the string. In other words, getBytes() doesn't really just return the byte array that makes up the string ... some sort of conversion always takes place.
David
| Ravi Varanasi <[EMAIL PROTECTED]>
09/04/2003 07:26 PM
|
To: [EMAIL PROTECTED] cc: [EMAIL PROTECTED] Subject: Re: UTF-8 problem with Xerces-J2 |
Thanks for all the inputs. I could isolate the issue to the way data is
written to the database. following is the code snippet for updating the
database.
1) Database column type Oracle LONG (LONGVARCHAR)
2) I am using OraclePreparedStatement using Oracle OCI driver.
longDesc is the String I created out of XML data in characters call-back
method.
I have created a temp String with UTF-8 encoding and using that stream to
insert into db in lines 1 & 2. Still the data is inserted as some junk in
database,
To verify if any data is lost during the conversion (while creating myStr),
I wrote the contents to a File and the data looks good in the output file
(out1.xml).
=========================================================================
1 String myStr = new String(longDesc.getBytes(), "UTF-8");
2 ostmt.setAsciiStream(7, new StringBufferInputStream(myStr),
myStr.length() );
3
4 OutputStreamWriter ostream = new OutputStreamWriter( new
FileOutputStream("/export/home/atgifce/DR/out1.xml"), "UTF-8" );
5 ostream.write(myStr, 0, myStr.length() );
6 ostream.close();
=========================================================================
Thanks for all the help !
PS: I have tried to write as BinaryStream as well. It did not work either.
Ravi Varanasi
408 517 7675
|---------+---------------------------->
| | mohammedi padaria|
| | <[EMAIL PROTECTED]|
| | om> |
| | |
| | 09/04/2003 01:30 |
| | PM |
| | Please respond to|
| | xerces-j-user |
| | |
|---------+---------------------------->
>---------------------------------------------------------------------------------------------------------------------------------------------|
| |
| To: [EMAIL PROTECTED] |
| cc: |
| Subject: Re: UTF-8 problem with Xerces-J2 |
>---------------------------------------------------------------------------------------------------------------------------------------------|
Also make sure that you are not using FileReader to get your Input stream.
It will read data in character and not in bytes. If you are using
FileReader ... change it to FileInputStream
Jeffrey Rodriguez <[EMAIL PROTECTED]> wrote:
Hi Ravi,
Your InputSource is OK, you should not have to set this since the parser
would
autodectect that your data is UTF-8.
>4) Comig to the important question, do I convert the data to UTF-8 ?
Answer
>is NO. The apache documentation says that the encoding is "retained" when
a
>ByteStream is passed in to the parse method (as InputSource). So, the
char
Where did you read this?
>array I get in characters call-back method must have char data encoded in
>UTF-8. Is it not correct ? Since the parser does not guarantee that
entire
What the handler defines back as a character event return is an array of
char ( Java char ).
>char data is sent in a single call back metod call, I am constucting a
>String using the char array. And the String constructor does not take the
>encoding parameter. Is t! here any other way I can get String with UTF-8
Of course it doesn't since String in Java are Unicode.
>encoding ? I can not use byte array because if I covert char[] to byte
[],
>there is a good possibility of data loss. Following is the code in my
>characters call back method. So, in essence, I am assuming that the
char[]
>I get has UTF-8 data. Please suggest if it is not correct ! !
>
I don't think this may be correct, but I will let others correct me. I
think
that your
array of char is just that, an array of Unicode character representing
your
UTF-8 data
byte stream.
What do you call to save your data to Oracle? JDBC and array of bytes?
Does
this call requires a stream?
You will have then to use an OutputStreamWriter with the proper encoding
name to
UTF-8 , and a ByteArrayOutputStream. Or just as easily just take the
char[]
which you
collect in your stack, create one instance of String a! nd use the
String.getBytes("UTF8")
method to get a byte[] than then you can store in Oracle.
>Please note that elementStack is a data structure I am using to store
some
>data. Pl ignore it.
Yes...
Hope this helps,
>From: Ravi Varanasi
>Reply-To: [EMAIL PROTECTED]
>To: [EMAIL PROTECTED]
>CC: [EMAIL PROTECTED]
>Subject: Re: UTF-8 problem with Xerces-J2
>Date: Thu, 4 Sep 2003 11:30:36 -0700
>
>
>
>
>
>
>Hi Jeffrey,
> Thanks for the reply. Following are the answers to your questions :-
>
>1) XML doc has encoding defined as UTF-8. This is the first stmt in the
XML
>file :
><?xml version="1.0" encoding="UTF-8"?>
>
>2) I am using InputSource with UTF-8 encoding set. Code snippet:
>
> InputSource ipSource = new InputSource();
> ipSource.setEncoding("UTF-8");
> ipSource.setByteStream( new FileInputStream( new File(inputFile) )
>);
> parser.parse(ipSource);
>
>3) Oracle totally supports UTF-8. I stored some UTF-8 data before ( using
>SQL scripts) and it worked fine.
>
>4) Comig to the important question, do I convert the data to UTF-8 ?
Answer
>is NO. The apache documentation says that the encoding is "retained" when
a
>ByteStream is passed in to the parse method (as InputSource). So, the
char
>array I get in characters call-back method must have char data encoded in
>UTF-8. Is it not correct ? Since the parser does not guarantee that
entire
>char data is sent in a single call back metod call, I am constucting a
>String using the char array. And the String constructor does not take the
>encoding parameter. Is there any other way I can get String with UTF-8
>encoding ? I can not use byte array because if ! I covert char[] to byte
[],
>there is a good possibility of data loss. Following is the code in my
>characters call back method. So, in essence, I am assuming that the
char[]
>I get has UTF-8 data. Please suggest if it is not correct ! !
>
>Please note that elementStack is a data structure I am using to store
some
>data. Pl ignore it.
>
>---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> public void characters(char ch[], int start, int length) throws
>SAXException {
> String currData = new String(ch, start, length);
>
> if (elementStack != null) {
> XMLElement currElement = (XMLElement) elementStack.peek();
> currElement.appendData(currData.trim());
> }
>
>---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>
>Thanks for the help,
>
>Ravi Varanasi
>408 517 7675
>
>
>
>|---------+---------------------------->
>| | "Jeffrey |
>| | Rodriguez" |
>| | >| | ail.com> |
>| | |
>| | 09/04/2003 11:02 |
>| | AM |
>| | Please respond to|
>| | xerces-j-user |
>| | |
>|---------+---------------------------->
>
>
>---------------------------------------------------------------------------------------------------------------------------------------------|
> |
> |
> | To: [EMAIL PROTECTED]
> |
> | cc:
> |
> | Subject: Re: UTF-8 problem with Xerces-J2
> |
>
>
>---------------------------------------------------------------------------------------------------------------------------------------------|
>
>
>
>
> >
> >Hi,
> > I am trying to parse an UTF-8 encoded document (which has lots of
> >UTF-8 characters) using Xerces SAX parser. I am running this program on
>Sun
>
>So what does your encodingDecl look like if any in your document?
>
>There is no problem with Xerces J parsing UTF8 data.
>
> >Solaris box with JDK 1.3.1_05. I save the data in XML (after parsing)
to
>
>What do you mean by "save the data", how? Remember that the parser will
get
>
>you
>back the data as "Java" char (aka Unicode, UTF16). Do you transcode the
>data
>back into UTF-8
>or does Oracle do that?
>
> >Oracle Database (which has UTF-8 encoding ). When I try to display the
> >conten! t in a HTML after retrieving from database, I see some weired
> >characters. Can any one suggest the reason ?
>
>If you had UTF16 data back from the parser and stored that into a UTF8 I
>think that would
>be problematic if you don't convert to UTF8.
>UTF8 is multibyte , and UTF16 is double byte data. UTF8 and UTF16 from
>U+0000 to U+007F
>map to each other (more correctly to said Unicode code point map within
>that
>range to UTF-8,
>therefore form some values if they are store directly into a UTF8 data
>repository they map
>map correctly but data outside this range wll not.
>
> >
> >
> > I am assuming that the UTF-8 format is supported by Xerces. I have
>
>Good assumption since xml parser must be able to read both UTF-8 and
UTF-16
>
>documents.
>
>
> > created a InputSource for the XML file and using it as the parameter
&! gt; >for
> > parse method.
>
>Did you use the InputSource and provided an encoding?
>
> > I am using OraclePreparedStatement because the column in which data
>is
> > stored is LONG. Do I need to do anything specific to let Oracle know
> > that it is UTF-8 data ?
>
>You said that Oracle stores data as UTF-8??? right. You should this
>question in a Oracle
>discussion group just to be sure.
>
> > The encoding is specified properly in Jsp using both JSP Page param &
> > HTML meta directive.
> >
>
>Yes, try to see first that your data is correctly store as UTF-8 in
Oracle.
>
>To test this pick
>a multibyte with more than one byte like.
>
>The Russian Sheah, in UTF-8 I think it is "000416" that should be the
>value
>stored into UTF-8.
>The value that the parser will give you back is Ud096 ( why? Because Java
>chars are Unicode...).
>
>Hope this helps,
>
> Jeffrey Rodriguez
> Silicon Valley
>
> >
> >Any help is appreciated.
> >
> >Thanks in advance.
> >
> >Ravi Varanasi
> >408 517 7675
> >
> >
> >---------------------------------------------------------------------
> >To unsubscribe, e-mail: [EMAIL PROTECTED]
> >For additional commands, e-mail: [EMAIL PROTECTED]
> >
>
>_________________________________________________________________
>Compare Cable, DSL or Satellite plans: As low as $29.95.
>https://broadband.msn.com
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: [EMAIL PROTECTED]
>For additional commands, e-mail: [EMAIL PROTECTED]
>
>
>
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: [EMAIL PROTECTED]
>For additional commands, e-mail: [EMAIL PROTECTED]
>
_________________________________________________________________
Get a FREE computer virus scan online from McAfee.
http://clinic.mcafee.com/clinic/ibuy/campaign.asp?cid=3963
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Do you Yahoo!?
Yahoo! SiteBuilder - Free, easy-to-use web site design software
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
