Sam, why don't you open a JIRA issue and we can move the discussion there? I assume the majority of people are not interested in the Unicode pains we are experiencing. ;-)
Thanks, Paolo Sam Tunnicliffe wrote: > It seems that NodeLib, or rather NodecSSE it uses under the covers, > has problems round tripping certain strings. How the data gets into > the larger system is still an issue but somewhat orthogonal here. > > String s = "Hello \uDAE0 World"; > Node literal = Node.createLiteral(s); > ByteBuffer bb = NodeLib.encode(literal); > NodeLib.decode(bb); > > blows up during the decode - looking at the stacktraces, this seems to > be the what causes the problems in committing the transaction. Should > we expect NodeLib's encode() & decode() to be symmetrical? > > Cheers, > Sam > > > On 21 March 2012 12:37, Paolo Castagna <[email protected]> wrote: >> Hi Andy >> >> Andy Seaborne wrote: >>> On 21/03/12 12:01, Paolo Castagna wrote: >>>> Paolo Castagna wrote: >>>>> Sam contributed this test case: >>>>> https://github.com/castagna/jena-examples/blob/master/src/main/resources/data/single-bad-triple.nt >>>>> >>>>> https://github.com/castagna/jena-examples/blob/master/src/main/java/dev/Run3.java >>>>> >>>>> Looking at this, right now. >>>> Getting closer... do we have a bug here? >>>> >>>> Test case is here: >>>> https://github.com/castagna/jena-examples/blob/master/src/main/java/dev/TestTDBUnicode.java >>>> >>>> >>>> This is the data (i.e. a single triple with the Unicode character \uDAE0 >>>> in a literal value):<s> <p> "Hello \uDAE0 World" . >>> You need >>> >>> private static final String str_literal = "Hello \\uDAE0 World"; >>> >>> else it is a Java \u >> Ack, silly mistake. >> >> I still don't understand why with "Hello \\uDAE0 World" test_03() fails. >> >>> >>> Using >>> >>> Dataset dataset = TDBFactory.createDataset ( ) ; >>> >>> is way easier for testing. >> Added test_04() which fails with the same exception: >> org.openjena.riot.RiotParseException: [line: 1, col: 2 ] Broken token: Hello >> >> I was using TDBFactory.createDataset ( location ) because I noticed that if >> FileOps.clearDirectory( path ) is commented out and the test is executed >> twice there is another exception: >> >> 12:35:00 WARN NodeTableTrans :: Txn[1]/W journalStartOffset not >> zero: 109/0x6D >> ************* UNEXPECTED [1] >> >> >> Different ids for file:///opt/workspaces/jena/jena-examples/s: allocated: >> expected [000000000000006D], got [0000000000000000] >> label = nodes >> txn = Transaction: 1 : Mode=WRITE : State=PREPARING : >> /opt/workspaces/jena/jena-examples/target/tdb/ >> offset = 109 >> journalStartOffset = 109 >> journal = nodes.dat-jrnl >> >> com.hp.hpl.jena.tdb.TDBException: Different ids for >> file:///opt/workspaces/jena/jena-examples/s: allocated: expected >> [000000000000006D], got [0000000000000000] >> at >> com.hp.hpl.jena.tdb.transaction.NodeTableTrans.inconsistent(NodeTableTrans.java:212) >> at >> com.hp.hpl.jena.tdb.transaction.NodeTableTrans.append(NodeTableTrans.java:200) >> at >> com.hp.hpl.jena.tdb.transaction.NodeTableTrans.writeNodeJournal(NodeTableTrans.java:306) >> at >> com.hp.hpl.jena.tdb.transaction.NodeTableTrans.commitPrepare(NodeTableTrans.java:266) >> at >> com.hp.hpl.jena.tdb.transaction.Transaction.prepare(Transaction.java:131) >> at >> com.hp.hpl.jena.tdb.transaction.Transaction.commit(Transaction.java:112) >> at >> com.hp.hpl.jena.tdb.transaction.DatasetGraphTxn.commit(DatasetGraphTxn.java:40) >> at >> com.hp.hpl.jena.tdb.transaction.DatasetGraphTransaction._commit(DatasetGraphTransaction.java:106) >> at >> com.hp.hpl.jena.tdb.migrate.DatasetGraphTrackActive.commit(DatasetGraphTrackActive.java:60) >> at >> com.hp.hpl.jena.sparql.core.DatasetImpl.commit(DatasetImpl.java:143) >> at dev.TestTDBUnicode.test_03(TestTDBUnicode.java:66) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >> at java.lang.reflect.Method.invoke(Method.java:597) >> at >> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44) >> at >> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) >> at >> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41) >> at >> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20) >> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263) >> at >> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:69) >> at >> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:48) >> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231) >> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60) >> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229) >> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50) >> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222) >> at org.junit.runners.ParentRunner.run(ParentRunner.java:292) >> at >> org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:46) >> at >> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38) >> at >> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467) >> at >> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683) >> at >> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390) >> at >> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197) >> >> But, I'd like to understand the first problem first. >> >> Paolo >> >>> Andy
