Re: Hive job submsiion failed with exception ”java.io.UTFDataFormatException“
Hi lapalette. Sorry, It seems its impossible to change marshaller due to GridClientJdkMarshaller hardcoded. I've created a ticket https://issues.apache.org/jira/browse/IGNITE-4219 It would be great if you provide more information here or in ticked about hadoop property value that causes this error. On Mon, Nov 14, 2016 at 4:13 AM, lapalette wrote: > Hi, Andery： > Sorry for late reply, I found that the way I changed Marshaller may > be > incorrect, because after I changed the configuration options, the error log > was all the same and was still about ”JDKmashaller“. > I use the JDK version is 1.7 .0_07, ignite 1.6 , and i'm running the > TPC-DS test on Ignite-Enhanced Hive, the test method address is > "https://github.com/hortonworks/hive-testbench.git”. > To change the Marshaller, I just added this property to the > configuration file witch used to start ingite : > > class="org.apache.ignite.configuration.IgniteConfiguration"> > class="org.apache.ignite.configuration.IgniteConfiguration" > singleton="true"> > ... > > class="org.apache.ignite.marshaller.optimized.OptimizedMarshaller"> > true > > > ... > > Thanks. > > > > -- > View this message in context: http://apache-ignite-users. > 70518.x6.nabble.com/Hive-job-submsiion-failed-with-exception-java-io- > UTFDataFormatException-tp8863p8931.html > Sent from the Apache Ignite Users mailing list archive at Nabble.com. >
Re: Hive job submsiion failed with exception ”java.io.UTFDataFormatException“
Hi, Andery： Sorry for late reply, I found that the way I changed Marshaller may be incorrect, because after I changed the configuration options, the error log was all the same and was still about ”JDKmashaller“. I use the JDK version is 1.7 .0_07, ignite 1.6 , and i'm running the TPC-DS test on Ignite-Enhanced Hive, the test method address is "https://github.com/hortonworks/hive-testbench.git”. To change the Marshaller, I just added this property to the configuration file witch used to start ingite : ... true ... Thanks. -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Hive-job-submsiion-failed-with-exception-java-io-UTFDataFormatException-tp8863p8931.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Hive job submsiion failed with exception ”java.io.UTFDataFormatException“
Hi lapalette, Would you like to explain the way you have change Marshaller? It seems OptimizedMarshaler does not have these limitations, and its possible it was not configured in correct way. On Fri, Nov 11, 2016 at 4:30 PM, Andrey Mashenkov wrote: > Hi lapalette, > > 1. Does this error appears in ignite 1.7 and it's not present in version > 1.6? > 2. Did you get same error with another marshallers? Would you please > provide stacktraces with other marshallers? > 3. What version of JVM do you use? Have you try to upgrade JVM? So, its > JDK internals limitation could be differ from version to version. > 4. As I understand right, you try to run some performance test, isn't it? > Anyway, would you please provide a code so I could reproduce this error. > > > On Fri, Nov 11, 2016 at 5:21 AM, lapalette wrote: > >> Hi，Andery： >> Thanks for your attention, but I tried use “OptimizedMarshaller” >> and >> "JdkMarshaller", but it did not work. I use the ignite version 1.6, and >> do I >> need to upgrade to 1.7? Or how can I modify the limitation of the >> ObjectOutputStream. >> Thanks. >> >> >> >> -- >> View this message in context: http://apache-ignite-users.705 >> 18.x6.nabble.com/Hive-job-submsiion-failed-with-exception- >> java-io-UTFDataFormatException-tp8863p8893.html >> Sent from the Apache Ignite Users mailing list archive at Nabble.com. >> > >
Re: Hive job submsiion failed with exception ”java.io.UTFDataFormatException“
Hi lapalette, 1. Does this error appears in ignite 1.7 and it's not present in version 1.6? 2. Did you get same error with another marshallers? Would you please provide stacktraces with other marshallers? 3. What version of JVM do you use? Have you try to upgrade JVM? So, its JDK internals limitation could be differ from version to version. 4. As I understand right, you try to run some performance test, isn't it? Anyway, would you please provide a code so I could reproduce this error. On Fri, Nov 11, 2016 at 5:21 AM, lapalette wrote: > Hi，Andery： > Thanks for your attention, but I tried use “OptimizedMarshaller” and > "JdkMarshaller", but it did not work. I use the ignite version 1.6, and do > I > need to upgrade to 1.7? Or how can I modify the limitation of the > ObjectOutputStream. > Thanks. > > > > -- > View this message in context: http://apache-ignite-users. > 70518.x6.nabble.com/Hive-job-submsiion-failed-with-exception-java-io- > UTFDataFormatException-tp8863p8893.html > Sent from the Apache Ignite Users mailing list archive at Nabble.com. >
Re: Hive job submsiion failed with exception ”java.io.UTFDataFormatException“
Hi，Andery： Thanks for your attention, but I tried use “OptimizedMarshaller” and "JdkMarshaller", but it did not work. I use the ignite version 1.6, and do I need to upgrade to 1.7? Or how can I modify the limitation of the ObjectOutputStream. Thanks. -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Hive-job-submsiion-failed-with-exception-java-io-UTFDataFormatException-tp8863p8893.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Hive job submsiion failed with exception ”java.io.UTFDataFormatException“
Hi lapalette, There is a limitation in ObjectOutputStream, it fails to write UTF strings longer than 65k bytes. Have you try to use another marshaller? https://ignite.apache.org/releases/1.7.0/javadoc/index.html?org/apache/ignite/marshaller/Marshaller.html On Thu, Nov 10, 2016 at 6:20 AM, lapalette wrote: > Hi，there is a error when i run TPC-DS testsets on ignite-enhanced hive2.0 > over hadoop2.6.0, and here is the log: > > java.io.IOException: Failed to submit job. > at > org.apache.ignite.internal.processors.hadoop.proto.HadoopClientProtocol. > submitJob(HadoopClientProtocol.java:128) > at > org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal( > JobSubmitter.java:536) > at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1296) > at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1293) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs( > UserGroupInformation.java:1628) > at org.apache.hadoop.mapreduce.Job.submit(Job.java:1293) > at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:562) > at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:557) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs( > UserGroupInformation.java:1628) > at org.apache.hadoop.mapred.JobClient.submitJobInternal( > JobClient.java:557) > at org.apache.hadoop.mapred.JobClient.submitJob(JobClient. > java:548) > at > org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:432) > at > org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:138) > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:158) > at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential( > TaskRunner.java:101) > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1840) > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1584) > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1361) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1184) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1172) > at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd( > CliDriver.java:233) > at org.apache.hadoop.hive.cli.CliDriver.processCmd( > CliDriver.java:184) > at org.apache.hadoop.hive.cli.CliDriver.processLine( > CliDriver.java:400) > at org.apache.hadoop.hive.cli.CliDriver.processLine( > CliDriver.java:336)å > at org.apache.hadoop.hive.cli.CliDriver.processReader( > CliDriver.java:433) > at org.apache.hadoop.hive.cli.CliDriver.processFile( > CliDriver.java:449) > at org.apache.hadoop.hive.cli.CliDriver.processCmd( > CliDriver.java:152) > at org.apache.hadoop.hive.cli.CliDriver.processLine( > CliDriver.java:400) > at org.apache.hadoop.hive.cli.CliDriver.executeDriver( > CliDriver.java:778) > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:717) > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:645) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java: > 57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke( > DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at org.apache.hadoop.util.RunJar.run(RunJar.java:221) > at org.apache.hadoop.util.RunJar.main(RunJar.java:136) > Caused by: class > org.apache.ignite.internal.client.GridServerUnreachableException: Failed > to > communicate with grid nodes (maximum count of retries reached). > at > org.apache.ignite.internal.client.impl.GridClientAbstractProjection. > withReconnectHandling(GridClientAbstractProjection.java:153) > at > org.apache.ignite.internal.client.impl.GridClientComputeImpl.executeAsync( > GridClientComputeImpl.java:132) > at > org.apache.ignite.internal.client.impl.GridClientComputeImpl.execute( > GridClientComputeImpl.java:121) > at > org.apache.ignite.internal.processors.hadoop.proto.HadoopClientProtocol. > submitJob(HadoopClientProtocol.java:119) > ... 40 more > Caused by: class > org.apache.ignite.internal.client.impl.connection. > GridClientConnectionResetException: > Failed to send message over connection (will try to reconnect): > /192.168.100.31:11211 > at > org.apache.ignite.internal.client.impl.connection. > GridClientNioTcpConnection.makeRequest(GridClientNioTcpConnection. > java:495) > at > org.apache.ignite.internal.client.impl.connection. > GridClientNioTcpConnection.makeRequest(GridClientNioTcpConnection. > java:416)
Hive job submsiion failed with exception ”java.io.UTFDataFormatException“
Hi，there is a error when i run TPC-DS testsets on ignite-enhanced hive2.0 over hadoop2.6.0, and here is the log: java.io.IOException: Failed to submit job. at org.apache.ignite.internal.processors.hadoop.proto.HadoopClientProtocol.submitJob(HadoopClientProtocol.java:128) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:536) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1296) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1293) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1293) at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:562) at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:557) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:557) at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:548) at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:432) at org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:138) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:158) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:101) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1840) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1584) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1361) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1184) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1172) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:400) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:336)å at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:433) at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:449) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:152) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:400) at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:778) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:717) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:645) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) Caused by: class org.apache.ignite.internal.client.GridServerUnreachableException: Failed to communicate with grid nodes (maximum count of retries reached). at org.apache.ignite.internal.client.impl.GridClientAbstractProjection.withReconnectHandling(GridClientAbstractProjection.java:153) at org.apache.ignite.internal.client.impl.GridClientComputeImpl.executeAsync(GridClientComputeImpl.java:132) at org.apache.ignite.internal.client.impl.GridClientComputeImpl.execute(GridClientComputeImpl.java:121) at org.apache.ignite.internal.processors.hadoop.proto.HadoopClientProtocol.submitJob(HadoopClientProtocol.java:119) ... 40 more Caused by: class org.apache.ignite.internal.client.impl.connection.GridClientConnectionResetException: Failed to send message over connection (will try to reconnect): /192.168.100.31:11211 at org.apache.ignite.internal.client.impl.connection.GridClientNioTcpConnection.makeRequest(GridClientNioTcpConnection.java:495) at org.apache.ignite.internal.client.impl.connection.GridClientNioTcpConnection.makeRequest(GridClientNioTcpConnection.java:416) at org.apache.ignite.internal.client.impl.connection.GridClientNioTcpConnection.makeRequest(GridClientNioTcpConnection.java:402) at org.apache.ignite.internal.client.impl.connection.GridClientNioTcpConnection.makeRequest(GridClientNioTcpConnection.java:380) at org.apache.ignite.internal.client.impl.connection.GridClientNioTcpConnection.execute(GridClientNioTcpConnection.java:797) at org.apache.ignite.internal.client.impl.GridClientComputeImpl$4.apply(GridClientComputeImpl.java:135) at org.apac
