[jira] [Updated] (HIVE-14711) HQL in which involves with join between txt and orc tables returns incorrect result
[ https://issues.apache.org/jira/browse/HIVE-14711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qiuzhuang Lian updated HIVE-14711: -- Attachment: schema.sql schema > HQL in which involves with join between txt and orc tables returns incorrect > result > --- > > Key: HIVE-14711 > URL: https://issues.apache.org/jira/browse/HIVE-14711 > Project: Hive > Issue Type: Bug > Components: ORC, SQL >Affects Versions: 1.2.1 > Environment: Hive 1.2.1 + Hadoop 2.6.2 >Reporter: Qiuzhuang Lian >Assignee: Owen O'Malley > Attachments: schema.sql > > > We have the scenario of ORC with ACID joining with TXT table and it can't > retrieve that ID based row which should. > Here is the HQL: > select a.id a_id,b.id b_id from txtdb.order a, orcdb.order_cln b where a.id = > b.id and b.id = '8a708a864faf8ef6014fb70426d62a67'; > Table txtdb.order is TXT format and orcdb.order_cln is ORC format. Both of > tables have this ID based row. But the result is empty. > Surprisingly, if I switch the position of TXT and ORC as follows, > select a.id a_id,b.id b_id from orcdb.order_cln a, txtdb.order b where a.id = > b.id and b.id = '8a708a864faf8ef6014fb70426d62a67'; > Then this ID based row returns correctly., -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-14711) HQL in which involves with join between txt and orc tables returns incorrect result
[ https://issues.apache.org/jira/browse/HIVE-14711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15727576#comment-15727576 ] Qiuzhuang Lian edited comment on HIVE-14711 at 12/7/16 3:42 AM: Attache is the schema for orc/txt table creation. BTW, txt table use partition with int date type while orc table doesn't. was (Author: qiuzhuang.lian): schema > HQL in which involves with join between txt and orc tables returns incorrect > result > --- > > Key: HIVE-14711 > URL: https://issues.apache.org/jira/browse/HIVE-14711 > Project: Hive > Issue Type: Bug > Components: ORC, SQL >Affects Versions: 1.2.1 > Environment: Hive 1.2.1 + Hadoop 2.6.2 >Reporter: Qiuzhuang Lian >Assignee: Owen O'Malley > Attachments: schema.sql > > > We have the scenario of ORC with ACID joining with TXT table and it can't > retrieve that ID based row which should. > Here is the HQL: > select a.id a_id,b.id b_id from txtdb.order a, orcdb.order_cln b where a.id = > b.id and b.id = '8a708a864faf8ef6014fb70426d62a67'; > Table txtdb.order is TXT format and orcdb.order_cln is ORC format. Both of > tables have this ID based row. But the result is empty. > Surprisingly, if I switch the position of TXT and ORC as follows, > select a.id a_id,b.id b_id from orcdb.order_cln a, txtdb.order b where a.id = > b.id and b.id = '8a708a864faf8ef6014fb70426d62a67'; > Then this ID based row returns correctly., -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13432) ACID ORC CompactorMR job throws java.lang.ArrayIndexOutOfBoundsException: 7
[ https://issues.apache.org/jira/browse/HIVE-13432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15303470#comment-15303470 ] Qiuzhuang Lian commented on HIVE-13432: --- Hi Matt, Since we are blocked by this issue, can you please help take a look at this? Many thanks. > ACID ORC CompactorMR job throws java.lang.ArrayIndexOutOfBoundsException: 7 > --- > > Key: HIVE-13432 > URL: https://issues.apache.org/jira/browse/HIVE-13432 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 1.2.1 > Environment: Hadoop 2.6.2+Hive 1.2.1 >Reporter: Qiuzhuang Lian >Assignee: Matt McCline > > After initiating HIVE ACID ORC table compaction, the CompactorMR job throws > exception: > Error: java.lang.ArrayIndexOutOfBoundsException: 7 > at > org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$StructTreeReader.(TreeReaderFactory.java:1968) > at > org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2368) > at > org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$StructTreeReader.(TreeReaderFactory.java:1969) > at > org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2368) > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderFactory.createTreeReader(RecordReaderFactory.java:69) > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.(RecordReaderImpl.java:202) > at > org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rowsOptions(ReaderImpl.java:539) > at > org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$ReaderPair.(OrcRawRecordMerger.java:183) > at > org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.(OrcRawRecordMerger.java:466) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRawReader(OrcInputFormat.java:1308) > at > org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:512) > at > org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:491) > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > As a result, we see hadoop exception stack, > 297 failed with state FAILED due to: Task failed > task_1458819387386_11297_m_08 > Job failed as tasks failed. failedMaps:1 failedReduces:0 > 2016-04-06 11:30:57,891 INFO [dn209006-27]: mapreduce.Job > (Job.java:monitorAndPrintJob(1392)) - Counters: 14 > Job Counters > Failed map tasks=16 > Killed map tasks=7 > Launched map tasks=23 > Other local map tasks=13 > Data-local map tasks=6 > Rack-local map tasks=4 > Total time spent by all maps in occupied slots (ms)=412592 > Total time spent by all reduces in occupied slots (ms)=0 > Total time spent by all map tasks (ms)=206296 > Total vcore-seconds taken by all map tasks=206296 > Total megabyte-seconds taken by all map tasks=422494208 > Map-Reduce Framework > CPU time spent (ms)=0 > Physical memory (bytes) snapshot=0 > Virtual memory (bytes) snapshot=0 > 2016-04-06 11:30:57,891 ERROR [dn209006-27]: compactor.Worker > (Worker.java:run(176)) - Caught exception while trying to compact > lqz.my_orc_acid_table. Marking clean to avoid repeated failures, > java.io.IOException: Job failed! > at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:836) > at > org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.run(CompactorMR.java:186) > at org.apache.hadoop.hive.ql.txn.compactor.Worker.run(Worker.java:162) > 2016-04-06 11:30:57,894 ERROR [dn209006-27]: txn.CompactionTxnHandler > (CompactionTxnHandler.java:markCleaned(327)) - Expected to remove at least > one row from completed_txn_components when marking compaction entry as clean! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13432) ACID ORC CompactorMR job throws java.lang.ArrayIndexOutOfBoundsException: 7
[ https://issues.apache.org/jira/browse/HIVE-13432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15323902#comment-15323902 ] Qiuzhuang Lian commented on HIVE-13432: --- Sorry for delay response. Yes, our hive-site.xml uses CombineHiveInputFormat. I should have sent you our hive-site.xml. Please check the attached to see if any more clue. > ACID ORC CompactorMR job throws java.lang.ArrayIndexOutOfBoundsException: 7 > --- > > Key: HIVE-13432 > URL: https://issues.apache.org/jira/browse/HIVE-13432 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 1.2.1 > Environment: Hadoop 2.6.2+Hive 1.2.1 >Reporter: Qiuzhuang Lian >Assignee: Matt McCline > > After initiating HIVE ACID ORC table compaction, the CompactorMR job throws > exception: > Error: java.lang.ArrayIndexOutOfBoundsException: 7 > at > org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$StructTreeReader.(TreeReaderFactory.java:1968) > at > org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2368) > at > org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$StructTreeReader.(TreeReaderFactory.java:1969) > at > org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2368) > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderFactory.createTreeReader(RecordReaderFactory.java:69) > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.(RecordReaderImpl.java:202) > at > org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rowsOptions(ReaderImpl.java:539) > at > org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$ReaderPair.(OrcRawRecordMerger.java:183) > at > org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.(OrcRawRecordMerger.java:466) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRawReader(OrcInputFormat.java:1308) > at > org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:512) > at > org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:491) > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > As a result, we see hadoop exception stack, > 297 failed with state FAILED due to: Task failed > task_1458819387386_11297_m_08 > Job failed as tasks failed. failedMaps:1 failedReduces:0 > 2016-04-06 11:30:57,891 INFO [dn209006-27]: mapreduce.Job > (Job.java:monitorAndPrintJob(1392)) - Counters: 14 > Job Counters > Failed map tasks=16 > Killed map tasks=7 > Launched map tasks=23 > Other local map tasks=13 > Data-local map tasks=6 > Rack-local map tasks=4 > Total time spent by all maps in occupied slots (ms)=412592 > Total time spent by all reduces in occupied slots (ms)=0 > Total time spent by all map tasks (ms)=206296 > Total vcore-seconds taken by all map tasks=206296 > Total megabyte-seconds taken by all map tasks=422494208 > Map-Reduce Framework > CPU time spent (ms)=0 > Physical memory (bytes) snapshot=0 > Virtual memory (bytes) snapshot=0 > 2016-04-06 11:30:57,891 ERROR [dn209006-27]: compactor.Worker > (Worker.java:run(176)) - Caught exception while trying to compact > lqz.my_orc_acid_table. Marking clean to avoid repeated failures, > java.io.IOException: Job failed! > at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:836) > at > org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.run(CompactorMR.java:186) > at org.apache.hadoop.hive.ql.txn.compactor.Worker.run(Worker.java:162) > 2016-04-06 11:30:57,894 ERROR [dn209006-27]: txn.CompactionTxnHandler > (CompactionTxnHandler.java:markCleaned(327)) - Expected to remove at least > one row from completed_txn_components when marking compaction entry as clean! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13432) ACID ORC CompactorMR job throws java.lang.ArrayIndexOutOfBoundsException: 7
[ https://issues.apache.org/jira/browse/HIVE-13432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qiuzhuang Lian updated HIVE-13432: -- Attachment: orc_hive-site.xml hive-site.xml to pr-produce the compaction failure for HIVE-13432 > ACID ORC CompactorMR job throws java.lang.ArrayIndexOutOfBoundsException: 7 > --- > > Key: HIVE-13432 > URL: https://issues.apache.org/jira/browse/HIVE-13432 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 1.2.1 > Environment: Hadoop 2.6.2+Hive 1.2.1 >Reporter: Qiuzhuang Lian >Assignee: Matt McCline > Attachments: orc_hive-site.xml > > > After initiating HIVE ACID ORC table compaction, the CompactorMR job throws > exception: > Error: java.lang.ArrayIndexOutOfBoundsException: 7 > at > org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$StructTreeReader.(TreeReaderFactory.java:1968) > at > org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2368) > at > org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$StructTreeReader.(TreeReaderFactory.java:1969) > at > org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2368) > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderFactory.createTreeReader(RecordReaderFactory.java:69) > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.(RecordReaderImpl.java:202) > at > org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rowsOptions(ReaderImpl.java:539) > at > org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$ReaderPair.(OrcRawRecordMerger.java:183) > at > org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.(OrcRawRecordMerger.java:466) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRawReader(OrcInputFormat.java:1308) > at > org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:512) > at > org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:491) > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > As a result, we see hadoop exception stack, > 297 failed with state FAILED due to: Task failed > task_1458819387386_11297_m_08 > Job failed as tasks failed. failedMaps:1 failedReduces:0 > 2016-04-06 11:30:57,891 INFO [dn209006-27]: mapreduce.Job > (Job.java:monitorAndPrintJob(1392)) - Counters: 14 > Job Counters > Failed map tasks=16 > Killed map tasks=7 > Launched map tasks=23 > Other local map tasks=13 > Data-local map tasks=6 > Rack-local map tasks=4 > Total time spent by all maps in occupied slots (ms)=412592 > Total time spent by all reduces in occupied slots (ms)=0 > Total time spent by all map tasks (ms)=206296 > Total vcore-seconds taken by all map tasks=206296 > Total megabyte-seconds taken by all map tasks=422494208 > Map-Reduce Framework > CPU time spent (ms)=0 > Physical memory (bytes) snapshot=0 > Virtual memory (bytes) snapshot=0 > 2016-04-06 11:30:57,891 ERROR [dn209006-27]: compactor.Worker > (Worker.java:run(176)) - Caught exception while trying to compact > lqz.my_orc_acid_table. Marking clean to avoid repeated failures, > java.io.IOException: Job failed! > at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:836) > at > org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.run(CompactorMR.java:186) > at org.apache.hadoop.hive.ql.txn.compactor.Worker.run(Worker.java:162) > 2016-04-06 11:30:57,894 ERROR [dn209006-27]: txn.CompactionTxnHandler > (CompactionTxnHandler.java:markCleaned(327)) - Expected to remove at least > one row from completed_txn_components when marking compaction entry as clean! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13432) ACID ORC CompactorMR job throws java.lang.ArrayIndexOutOfBoundsException: 7
[ https://issues.apache.org/jira/browse/HIVE-13432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qiuzhuang Lian updated HIVE-13432: -- Attachment: (was: orc_hive-site.xml) > ACID ORC CompactorMR job throws java.lang.ArrayIndexOutOfBoundsException: 7 > --- > > Key: HIVE-13432 > URL: https://issues.apache.org/jira/browse/HIVE-13432 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 1.2.1 > Environment: Hadoop 2.6.2+Hive 1.2.1 >Reporter: Qiuzhuang Lian >Assignee: Matt McCline > > After initiating HIVE ACID ORC table compaction, the CompactorMR job throws > exception: > Error: java.lang.ArrayIndexOutOfBoundsException: 7 > at > org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$StructTreeReader.(TreeReaderFactory.java:1968) > at > org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2368) > at > org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$StructTreeReader.(TreeReaderFactory.java:1969) > at > org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2368) > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderFactory.createTreeReader(RecordReaderFactory.java:69) > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.(RecordReaderImpl.java:202) > at > org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rowsOptions(ReaderImpl.java:539) > at > org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$ReaderPair.(OrcRawRecordMerger.java:183) > at > org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.(OrcRawRecordMerger.java:466) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRawReader(OrcInputFormat.java:1308) > at > org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:512) > at > org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:491) > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > As a result, we see hadoop exception stack, > 297 failed with state FAILED due to: Task failed > task_1458819387386_11297_m_08 > Job failed as tasks failed. failedMaps:1 failedReduces:0 > 2016-04-06 11:30:57,891 INFO [dn209006-27]: mapreduce.Job > (Job.java:monitorAndPrintJob(1392)) - Counters: 14 > Job Counters > Failed map tasks=16 > Killed map tasks=7 > Launched map tasks=23 > Other local map tasks=13 > Data-local map tasks=6 > Rack-local map tasks=4 > Total time spent by all maps in occupied slots (ms)=412592 > Total time spent by all reduces in occupied slots (ms)=0 > Total time spent by all map tasks (ms)=206296 > Total vcore-seconds taken by all map tasks=206296 > Total megabyte-seconds taken by all map tasks=422494208 > Map-Reduce Framework > CPU time spent (ms)=0 > Physical memory (bytes) snapshot=0 > Virtual memory (bytes) snapshot=0 > 2016-04-06 11:30:57,891 ERROR [dn209006-27]: compactor.Worker > (Worker.java:run(176)) - Caught exception while trying to compact > lqz.my_orc_acid_table. Marking clean to avoid repeated failures, > java.io.IOException: Job failed! > at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:836) > at > org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.run(CompactorMR.java:186) > at org.apache.hadoop.hive.ql.txn.compactor.Worker.run(Worker.java:162) > 2016-04-06 11:30:57,894 ERROR [dn209006-27]: txn.CompactionTxnHandler > (CompactionTxnHandler.java:markCleaned(327)) - Expected to remove at least > one row from completed_txn_components when marking compaction entry as clean! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13432) ACID ORC CompactorMR job throws java.lang.ArrayIndexOutOfBoundsException: 7
[ https://issues.apache.org/jira/browse/HIVE-13432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qiuzhuang Lian updated HIVE-13432: -- Attachment: orc_hive-site.xml > ACID ORC CompactorMR job throws java.lang.ArrayIndexOutOfBoundsException: 7 > --- > > Key: HIVE-13432 > URL: https://issues.apache.org/jira/browse/HIVE-13432 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 1.2.1 > Environment: Hadoop 2.6.2+Hive 1.2.1 >Reporter: Qiuzhuang Lian >Assignee: Matt McCline > Attachments: orc_hive-site.xml > > > After initiating HIVE ACID ORC table compaction, the CompactorMR job throws > exception: > Error: java.lang.ArrayIndexOutOfBoundsException: 7 > at > org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$StructTreeReader.(TreeReaderFactory.java:1968) > at > org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2368) > at > org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$StructTreeReader.(TreeReaderFactory.java:1969) > at > org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2368) > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderFactory.createTreeReader(RecordReaderFactory.java:69) > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.(RecordReaderImpl.java:202) > at > org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rowsOptions(ReaderImpl.java:539) > at > org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$ReaderPair.(OrcRawRecordMerger.java:183) > at > org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.(OrcRawRecordMerger.java:466) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRawReader(OrcInputFormat.java:1308) > at > org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:512) > at > org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:491) > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > As a result, we see hadoop exception stack, > 297 failed with state FAILED due to: Task failed > task_1458819387386_11297_m_08 > Job failed as tasks failed. failedMaps:1 failedReduces:0 > 2016-04-06 11:30:57,891 INFO [dn209006-27]: mapreduce.Job > (Job.java:monitorAndPrintJob(1392)) - Counters: 14 > Job Counters > Failed map tasks=16 > Killed map tasks=7 > Launched map tasks=23 > Other local map tasks=13 > Data-local map tasks=6 > Rack-local map tasks=4 > Total time spent by all maps in occupied slots (ms)=412592 > Total time spent by all reduces in occupied slots (ms)=0 > Total time spent by all map tasks (ms)=206296 > Total vcore-seconds taken by all map tasks=206296 > Total megabyte-seconds taken by all map tasks=422494208 > Map-Reduce Framework > CPU time spent (ms)=0 > Physical memory (bytes) snapshot=0 > Virtual memory (bytes) snapshot=0 > 2016-04-06 11:30:57,891 ERROR [dn209006-27]: compactor.Worker > (Worker.java:run(176)) - Caught exception while trying to compact > lqz.my_orc_acid_table. Marking clean to avoid repeated failures, > java.io.IOException: Job failed! > at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:836) > at > org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.run(CompactorMR.java:186) > at org.apache.hadoop.hive.ql.txn.compactor.Worker.run(Worker.java:162) > 2016-04-06 11:30:57,894 ERROR [dn209006-27]: txn.CompactionTxnHandler > (CompactionTxnHandler.java:markCleaned(327)) - Expected to remove at least > one row from completed_txn_components when marking compaction entry as clean! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Issue Comment Deleted] (HIVE-13432) ACID ORC CompactorMR job throws java.lang.ArrayIndexOutOfBoundsException: 7
[ https://issues.apache.org/jira/browse/HIVE-13432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qiuzhuang Lian updated HIVE-13432: -- Comment: was deleted (was: hive-site.xml to pr-produce the compaction failure for HIVE-13432) > ACID ORC CompactorMR job throws java.lang.ArrayIndexOutOfBoundsException: 7 > --- > > Key: HIVE-13432 > URL: https://issues.apache.org/jira/browse/HIVE-13432 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 1.2.1 > Environment: Hadoop 2.6.2+Hive 1.2.1 >Reporter: Qiuzhuang Lian >Assignee: Matt McCline > Attachments: orc_hive-site.xml > > > After initiating HIVE ACID ORC table compaction, the CompactorMR job throws > exception: > Error: java.lang.ArrayIndexOutOfBoundsException: 7 > at > org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$StructTreeReader.(TreeReaderFactory.java:1968) > at > org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2368) > at > org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$StructTreeReader.(TreeReaderFactory.java:1969) > at > org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2368) > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderFactory.createTreeReader(RecordReaderFactory.java:69) > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.(RecordReaderImpl.java:202) > at > org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rowsOptions(ReaderImpl.java:539) > at > org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$ReaderPair.(OrcRawRecordMerger.java:183) > at > org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.(OrcRawRecordMerger.java:466) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRawReader(OrcInputFormat.java:1308) > at > org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:512) > at > org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:491) > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > As a result, we see hadoop exception stack, > 297 failed with state FAILED due to: Task failed > task_1458819387386_11297_m_08 > Job failed as tasks failed. failedMaps:1 failedReduces:0 > 2016-04-06 11:30:57,891 INFO [dn209006-27]: mapreduce.Job > (Job.java:monitorAndPrintJob(1392)) - Counters: 14 > Job Counters > Failed map tasks=16 > Killed map tasks=7 > Launched map tasks=23 > Other local map tasks=13 > Data-local map tasks=6 > Rack-local map tasks=4 > Total time spent by all maps in occupied slots (ms)=412592 > Total time spent by all reduces in occupied slots (ms)=0 > Total time spent by all map tasks (ms)=206296 > Total vcore-seconds taken by all map tasks=206296 > Total megabyte-seconds taken by all map tasks=422494208 > Map-Reduce Framework > CPU time spent (ms)=0 > Physical memory (bytes) snapshot=0 > Virtual memory (bytes) snapshot=0 > 2016-04-06 11:30:57,891 ERROR [dn209006-27]: compactor.Worker > (Worker.java:run(176)) - Caught exception while trying to compact > lqz.my_orc_acid_table. Marking clean to avoid repeated failures, > java.io.IOException: Job failed! > at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:836) > at > org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.run(CompactorMR.java:186) > at org.apache.hadoop.hive.ql.txn.compactor.Worker.run(Worker.java:162) > 2016-04-06 11:30:57,894 ERROR [dn209006-27]: txn.CompactionTxnHandler > (CompactionTxnHandler.java:markCleaned(327)) - Expected to remove at least > one row from completed_txn_components when marking compaction entry as clean! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13998) CompactorMR for ORC ACID throws NPE
[ https://issues.apache.org/jira/browse/HIVE-13998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15326219#comment-15326219 ] Qiuzhuang Lian commented on HIVE-13998: --- it seems that here piece of code needs to double-checking: public Properties toProperties() { Properties props = new Properties(); props.putAll(this); return props; } > CompactorMR for ORC ACID throws NPE > --- > > Key: HIVE-13998 > URL: https://issues.apache.org/jira/browse/HIVE-13998 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 1.2.1 > Environment: Hadoop 2.6.2+Hive 1.2.1 >Reporter: Qiuzhuang Lian > > When job CompactorMR is running, we see NPE stack as follows, > 2016-06-12 14:20:34,463 INFO [Thread-51] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Added > attempt_1464322345445_83343_m_03_1 to list of failed maps > 2016-06-12 14:20:34,597 INFO [IPC Server handler 7 on 40704] > org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt > attempt_1464322345445_83343_m_02_0 is : 0.0 > 2016-06-12 14:20:34,599 FATAL [IPC Server handler 12 on 40704] > org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: > attempt_1464322345445_83343_m_02_0 - exited : > java.lang.NullPointerException > at java.util.Hashtable.put(Hashtable.java:514) > at java.util.Hashtable.putAll(Hashtable.java:587) > at > org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$StringableMap.toProperties(CompactorMR.java:608) > at > org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.getWriter(CompactorMR.java:541) > at > org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:517) > at > org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:491) > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > 2016-06-12 14:20:34,599 INFO [IPC Server handler 12 on 40704] > org.apache.hadoop.mapred.TaskAttemptListenerImpl: Diagnostics report from > attempt_1464322345445_83343_m_02_0: Error: java.lang.NullPointerException > at java.util.Hashtable.put(Hashtable.java:514) > at java.util.Hashtable.putAll(Hashtable.java:587) > at > org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$StringableMap.toProperties(CompactorMR.java:608) > at > org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.getWriter(CompactorMR.java:541) > at > org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:517) > at > org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:491) > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > 2016-06-12 14:20:34,599 INFO [AsyncDispatcher event handler] > org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics > report from attempt_1464322345445_83343_m_02_0: Error: > java.lang.NullPointerException > at java.util.Hashtable.put(Hashtable.java:514) > at java.util.Hashtable.putAll(Hashtable.java:587) > at > org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$StringableMap.toProperties(CompactorMR.java:608) > at > org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.getWriter(CompactorMR.java:541) > at > org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:517) > at > org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:491) > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) > at org.apache.hadoop.mapred.Yar
[jira] [Commented] (HIVE-13432) ACID ORC CompactorMR job throws java.lang.ArrayIndexOutOfBoundsException: 7
[ https://issues.apache.org/jira/browse/HIVE-13432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15326232#comment-15326232 ] Qiuzhuang Lian commented on HIVE-13432: --- I remove this configuration when running compactorMR job and it looks ok now. BTW, is this CombineHiveInputFormat issue related to the recent fix of HIVE-13968? > ACID ORC CompactorMR job throws java.lang.ArrayIndexOutOfBoundsException: 7 > --- > > Key: HIVE-13432 > URL: https://issues.apache.org/jira/browse/HIVE-13432 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 1.2.1 > Environment: Hadoop 2.6.2+Hive 1.2.1 >Reporter: Qiuzhuang Lian >Assignee: Matt McCline > Attachments: orc_hive-site.xml > > > After initiating HIVE ACID ORC table compaction, the CompactorMR job throws > exception: > Error: java.lang.ArrayIndexOutOfBoundsException: 7 > at > org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$StructTreeReader.(TreeReaderFactory.java:1968) > at > org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2368) > at > org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$StructTreeReader.(TreeReaderFactory.java:1969) > at > org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2368) > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderFactory.createTreeReader(RecordReaderFactory.java:69) > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.(RecordReaderImpl.java:202) > at > org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rowsOptions(ReaderImpl.java:539) > at > org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$ReaderPair.(OrcRawRecordMerger.java:183) > at > org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.(OrcRawRecordMerger.java:466) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRawReader(OrcInputFormat.java:1308) > at > org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:512) > at > org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:491) > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > As a result, we see hadoop exception stack, > 297 failed with state FAILED due to: Task failed > task_1458819387386_11297_m_08 > Job failed as tasks failed. failedMaps:1 failedReduces:0 > 2016-04-06 11:30:57,891 INFO [dn209006-27]: mapreduce.Job > (Job.java:monitorAndPrintJob(1392)) - Counters: 14 > Job Counters > Failed map tasks=16 > Killed map tasks=7 > Launched map tasks=23 > Other local map tasks=13 > Data-local map tasks=6 > Rack-local map tasks=4 > Total time spent by all maps in occupied slots (ms)=412592 > Total time spent by all reduces in occupied slots (ms)=0 > Total time spent by all map tasks (ms)=206296 > Total vcore-seconds taken by all map tasks=206296 > Total megabyte-seconds taken by all map tasks=422494208 > Map-Reduce Framework > CPU time spent (ms)=0 > Physical memory (bytes) snapshot=0 > Virtual memory (bytes) snapshot=0 > 2016-04-06 11:30:57,891 ERROR [dn209006-27]: compactor.Worker > (Worker.java:run(176)) - Caught exception while trying to compact > lqz.my_orc_acid_table. Marking clean to avoid repeated failures, > java.io.IOException: Job failed! > at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:836) > at > org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.run(CompactorMR.java:186) > at org.apache.hadoop.hive.ql.txn.compactor.Worker.run(Worker.java:162) > 2016-04-06 11:30:57,894 ERROR [dn209006-27]: txn.CompactionTxnHandler > (CompactionTxnHandler.java:markCleaned(327)) - Expected to remove at least > one row from completed_txn_components when marking compaction entry as clean! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10242) ACID: insert overwrite prevents create table command
[ https://issues.apache.org/jira/browse/HIVE-10242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15379099#comment-15379099 ] Qiuzhuang Lian commented on HIVE-10242: --- In our production env, we still got this issue with hive-1.2.1. The error stack thrown from HS2 is as follows, 2016-07-15 04:30:00,028 ERROR [HiveServer2-Background-Pool: Thread-2903721]: lockmgr.DbTxnManager (DbTxnManager.java:heartbeat(295)) - Unable to find lock 214155 2016-07-15 04:30:00,028 WARN [HiveServer2-Background-Pool: Thread-2903721]: exec.Heartbeater (Heartbeater.java:heartbeat(83)) - Failed trying to heartbeat No record of lock could be found, may have timed out 2016-07-15 04:30:00,031 ERROR [HiveServer2-Background-Pool: Thread-2903721]: exec.Task (SessionState.java:printError(960)) - Ended Job = job_140599841_119253 with exception 'java.io.IOException(org.apache.hadoop.hive.ql.lockmgr.LockException: No record of lock could be found, may have timed out)' java.io.IOException: org.apache.hadoop.hive.ql.lockmgr.LockException: No record of lock could be found, may have timed out at org.apache.hadoop.hive.ql.exec.Heartbeater.heartbeat(Heartbeater.java:84) at org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:242) at org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:549) at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:437) at org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:137) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1653) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1412) > ACID: insert overwrite prevents create table command > > > Key: HIVE-10242 > URL: https://issues.apache.org/jira/browse/HIVE-10242 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Fix For: 1.2.0 > > Attachments: HIVE-10242.2.patch, HIVE-10242.3.patch, > HIVE-10242.4.patch, HIVE-10242.patch > > > 1. insert overwirte table DB.T1 select ... from T2: this takes X lock on > DB.T1 and S lock on T2. > X lock makes sense because we don't want anyone reading T1 while it's > overwritten. S lock on T2 prevents if from being dropped while the query is > in progress. > 2. create table DB.T3: takes S lock on DB. > This S lock gets blocked by X lock on T1. S lock prevents the DB from being > dropped while create table is executed. > If the insert statement is long running, this blocks DDL ops on the same > database. This is a usability issue. > There is no good reason why X lock on a table within a DB and S lock on DB > should be in conflict. > (this is different from a situation where X lock is on a partition and S lock > is on the table to which this partition belongs. Here it makes sense. > Basically there is no SQL way to address all tables in a DB but you can > easily refer to all partitions of a table) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-14004) Minor compaction produces ArrayIndexOutOfBoundsException: 7 in SchemaEvolution.getFileType
[ https://issues.apache.org/jira/browse/HIVE-14004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qiuzhuang Lian reassigned HIVE-14004: - Assignee: Qiuzhuang Lian (was: Owen O'Malley) > Minor compaction produces ArrayIndexOutOfBoundsException: 7 in > SchemaEvolution.getFileType > -- > > Key: HIVE-14004 > URL: https://issues.apache.org/jira/browse/HIVE-14004 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 2.2.0 >Reporter: Eugene Koifman >Assignee: Qiuzhuang Lian > Fix For: 2.2.0, 2.1.1 > > Attachments: HIVE-14004.01.patch, HIVE-14004.02.patch, > HIVE-14004.03.patch, HIVE-14004.patch > > > Easiest way to repro is to add TestTxnCommands2 > {noformat} > @Test > public void testCompactWithDelete() throws Exception { > int[][] tableData = {{1,2},{3,4}}; > runStatementOnDriver("insert into " + Table.ACIDTBL + "(a,b) " + > makeValuesClause(tableData)); > runStatementOnDriver("alter table "+ Table.ACIDTBL + " compact 'MAJOR'"); > Worker t = new Worker(); > t.setThreadId((int) t.getId()); > t.setHiveConf(hiveConf); > AtomicBoolean stop = new AtomicBoolean(); > AtomicBoolean looped = new AtomicBoolean(); > stop.set(true); > t.init(stop, looped); > t.run(); > runStatementOnDriver("delete from " + Table.ACIDTBL + " where b = 4"); > runStatementOnDriver("update " + Table.ACIDTBL + " set b = -2 where b = > 2"); > runStatementOnDriver("alter table "+ Table.ACIDTBL + " compact 'MINOR'"); > t.run(); > } > {noformat} > to TestTxnCommands2 and run it. > Test won't fail but if you look > in target/tmp/log/hive.log for the following exception (from Minor > compaction). > {noformat} > 2016-06-09T18:36:39,071 WARN [Thread-190[]]: mapred.LocalJobRunner > (LocalJobRunner.java:run(560)) - job_local1233973168_0005 > java.lang.Exception: java.lang.ArrayIndexOutOfBoundsException: 7 > at > org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462) > ~[hadoop-mapreduce-client-common-2.6.1.jar:?] > at > org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522) > [hadoop-mapreduce-client-common-2.6.1.jar:?] > Caused by: java.lang.ArrayIndexOutOfBoundsException: 7 > at > org.apache.orc.impl.SchemaEvolution.getFileType(SchemaEvolution.java:67) > ~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT] > at > org.apache.orc.impl.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2031) > ~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT] > at > org.apache.orc.impl.TreeReaderFactory$StructTreeReader.(TreeReaderFactory.java:1716) > ~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT] > at > org.apache.orc.impl.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2077) > ~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT] > at > org.apache.orc.impl.TreeReaderFactory$StructTreeReader.(TreeReaderFactory.java:1716) > ~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT] > at > org.apache.orc.impl.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2077) > ~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT] > at > org.apache.orc.impl.RecordReaderImpl.(RecordReaderImpl.java:208) > ~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT] > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.(RecordReaderImpl.java:63) > ~[classes/:?] > at > org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rowsOptions(ReaderImpl.java:365) > ~[classes/:?] > at > org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$ReaderPair.(OrcRawRecordMerger.java:207) > ~[classes/:?] > at > org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.(OrcRawRecordMerger.java:508) > ~[classes/:?] > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRawReader(OrcInputFormat.java:1977) > ~[classes/:?] > at > org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:630) > ~[classes/:?] > at > org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:609) > ~[classes/:?] > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) > ~[hadoop-mapreduce-client-core-2.6.1.jar:?] > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450) > ~[hadoop-mapreduce-client-core-2.6.1.jar:?] > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) > ~[hadoop-mapreduce-client-core-2.6.1.jar:?] > at > org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243) > ~[hadoop-mapreduce-client-common-2.6.1.jar:?] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > ~[?:1.7.0_71] > at java.util.concurrent.FutureTa
[jira] [Updated] (HIVE-14711) HQL in which involves with join between txt and orc tables returns incorrect result
[ https://issues.apache.org/jira/browse/HIVE-14711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qiuzhuang Lian updated HIVE-14711: -- Description: We have the scenario of ORC with ACID joining with TXT table and it can't retrieve that ID based row which should. Here is the HQL: select a.id a_id,b.id b_id from txtdb.order a, orcdb.order_cln b where a.id = b.id and b.id = '8a708a864faf8ef6014fb70426d62a67'; Table txtdb.order is TXT format and orcdb.order_cln is ORC format. Both of tables have this ID based row. But the result is empty. Surprisingly, if I switch the position of TXT and ORC as follows, select a.id a_id,b.id b_id from orcdb.order_cln a, txtdb.order b where a.id = b.id and b.id = '8a708a864faf8ef6014fb70426d62a67'; Then this ID based row returns correctly., was: We have the scenario of ORC with ACID joining with TXT table and it can't retrieve that ID based row which should. Here is the HQL: select a.id a_id,b.id b_id from txtdb.order a, orcdb.order_cln b where a.id = b.id and b.id = '8a708a864faf8ef6014fb70426d62a67'; table txtdb.order is TXT format and orcdb.order_cln is ORC format. Both of tables have this ID based row. But the result is empty. Surprisingly, if I switch the position of TXT and ORC as follows, select a.id a_id,b.id b_id from orcdb.order_cln a, txtdb.order b where a.id = b.id and b.id = '8a708a864faf8ef6014fb70426d62a67'; Then this ID based row returns correctly., > HQL in which involves with join between txt and orc tables returns incorrect > result > --- > > Key: HIVE-14711 > URL: https://issues.apache.org/jira/browse/HIVE-14711 > Project: Hive > Issue Type: Bug > Components: ORC, SQL >Affects Versions: 1.2.1 > Environment: Hive 1.2.1 + Hadoop 2.6.2 >Reporter: Qiuzhuang Lian > > We have the scenario of ORC with ACID joining with TXT table and it can't > retrieve that ID based row which should. > Here is the HQL: > select a.id a_id,b.id b_id from txtdb.order a, orcdb.order_cln b where a.id = > b.id and b.id = '8a708a864faf8ef6014fb70426d62a67'; > Table txtdb.order is TXT format and orcdb.order_cln is ORC format. Both of > tables have this ID based row. But the result is empty. > Surprisingly, if I switch the position of TXT and ORC as follows, > select a.id a_id,b.id b_id from orcdb.order_cln a, txtdb.order b where a.id = > b.id and b.id = '8a708a864faf8ef6014fb70426d62a67'; > Then this ID based row returns correctly., -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-14711) HQL in which involves with join between txt and orc tables returns incorrect result
[ https://issues.apache.org/jira/browse/HIVE-14711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qiuzhuang Lian reassigned HIVE-14711: - Assignee: Owen O'Malley > HQL in which involves with join between txt and orc tables returns incorrect > result > --- > > Key: HIVE-14711 > URL: https://issues.apache.org/jira/browse/HIVE-14711 > Project: Hive > Issue Type: Bug > Components: ORC, SQL >Affects Versions: 1.2.1 > Environment: Hive 1.2.1 + Hadoop 2.6.2 >Reporter: Qiuzhuang Lian >Assignee: Owen O'Malley > > We have the scenario of ORC with ACID joining with TXT table and it can't > retrieve that ID based row which should. > Here is the HQL: > select a.id a_id,b.id b_id from txtdb.order a, orcdb.order_cln b where a.id = > b.id and b.id = '8a708a864faf8ef6014fb70426d62a67'; > Table txtdb.order is TXT format and orcdb.order_cln is ORC format. Both of > tables have this ID based row. But the result is empty. > Surprisingly, if I switch the position of TXT and ORC as follows, > select a.id a_id,b.id b_id from orcdb.order_cln a, txtdb.order b where a.id = > b.id and b.id = '8a708a864faf8ef6014fb70426d62a67'; > Then this ID based row returns correctly., -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14711) HQL in which involves with join between txt and orc tables returns incorrect result
[ https://issues.apache.org/jira/browse/HIVE-14711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15469429#comment-15469429 ] Qiuzhuang Lian commented on HIVE-14711: --- Hi Gopal, both of txt and orc tables are string type for id column. > HQL in which involves with join between txt and orc tables returns incorrect > result > --- > > Key: HIVE-14711 > URL: https://issues.apache.org/jira/browse/HIVE-14711 > Project: Hive > Issue Type: Bug > Components: ORC, SQL >Affects Versions: 1.2.1 > Environment: Hive 1.2.1 + Hadoop 2.6.2 >Reporter: Qiuzhuang Lian >Assignee: Owen O'Malley > > We have the scenario of ORC with ACID joining with TXT table and it can't > retrieve that ID based row which should. > Here is the HQL: > select a.id a_id,b.id b_id from txtdb.order a, orcdb.order_cln b where a.id = > b.id and b.id = '8a708a864faf8ef6014fb70426d62a67'; > Table txtdb.order is TXT format and orcdb.order_cln is ORC format. Both of > tables have this ID based row. But the result is empty. > Surprisingly, if I switch the position of TXT and ORC as follows, > select a.id a_id,b.id b_id from orcdb.order_cln a, txtdb.order b where a.id = > b.id and b.id = '8a708a864faf8ef6014fb70426d62a67'; > Then this ID based row returns correctly., -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13432) ACID ORC CompactorMR job throws java.lang.ArrayIndexOutOfBoundsException: 7
[ https://issues.apache.org/jira/browse/HIVE-13432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15227974#comment-15227974 ] Qiuzhuang Lian commented on HIVE-13432: --- Can someone confirm that this is related to issue of https://issues.apache.org/jira/browse/HIVE-11981? Thanks. > ACID ORC CompactorMR job throws java.lang.ArrayIndexOutOfBoundsException: 7 > --- > > Key: HIVE-13432 > URL: https://issues.apache.org/jira/browse/HIVE-13432 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 1.2.1 > Environment: Hadoop 2.6.2+Hive 1.2.1 >Reporter: Qiuzhuang Lian > > After initiating HIVE ACID ORC table compaction, the CompactorMR job throws > exception: > Error: java.lang.ArrayIndexOutOfBoundsException: 7 > at > org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$StructTreeReader.(TreeReaderFactory.java:1968) > at > org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2368) > at > org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$StructTreeReader.(TreeReaderFactory.java:1969) > at > org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2368) > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderFactory.createTreeReader(RecordReaderFactory.java:69) > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.(RecordReaderImpl.java:202) > at > org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rowsOptions(ReaderImpl.java:539) > at > org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$ReaderPair.(OrcRawRecordMerger.java:183) > at > org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.(OrcRawRecordMerger.java:466) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRawReader(OrcInputFormat.java:1308) > at > org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:512) > at > org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:491) > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > As a result, we see hadoop exception stack, > 297 failed with state FAILED due to: Task failed > task_1458819387386_11297_m_08 > Job failed as tasks failed. failedMaps:1 failedReduces:0 > 2016-04-06 11:30:57,891 INFO [dn209006-27]: mapreduce.Job > (Job.java:monitorAndPrintJob(1392)) - Counters: 14 > Job Counters > Failed map tasks=16 > Killed map tasks=7 > Launched map tasks=23 > Other local map tasks=13 > Data-local map tasks=6 > Rack-local map tasks=4 > Total time spent by all maps in occupied slots (ms)=412592 > Total time spent by all reduces in occupied slots (ms)=0 > Total time spent by all map tasks (ms)=206296 > Total vcore-seconds taken by all map tasks=206296 > Total megabyte-seconds taken by all map tasks=422494208 > Map-Reduce Framework > CPU time spent (ms)=0 > Physical memory (bytes) snapshot=0 > Virtual memory (bytes) snapshot=0 > 2016-04-06 11:30:57,891 ERROR [dn209006-27]: compactor.Worker > (Worker.java:run(176)) - Caught exception while trying to compact > lqz.my_orc_acid_table. Marking clean to avoid repeated failures, > java.io.IOException: Job failed! > at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:836) > at > org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.run(CompactorMR.java:186) > at org.apache.hadoop.hive.ql.txn.compactor.Worker.run(Worker.java:162) > 2016-04-06 11:30:57,894 ERROR [dn209006-27]: txn.CompactionTxnHandler > (CompactionTxnHandler.java:markCleaned(327)) - Expected to remove at least > one row from completed_txn_components when marking compaction entry as clean! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11981) ORC Schema Evolution Issues (Vectorized, ACID, and Non-Vectorized)
[ https://issues.apache.org/jira/browse/HIVE-11981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15229872#comment-15229872 ] Qiuzhuang Lian commented on HIVE-11981: --- When compacting ORC table, we got error as follows, , kind: INT ] innerStructSubtype -1 at org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$StructTreeReader.(TreeReaderFactory.java:2081) at org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2504) at org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$StructTreeReader.(TreeReaderFactory.java:2084) at org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2504) at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.(RecordReaderImpl.java:210) at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rowsOptions(ReaderImpl.java:662) at org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$ReaderPair.(OrcRawRecordMerger.java:212) at org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.(OrcRawRecordMerger.java:516) at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRawReader(OrcInputFormat.java:1755) at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:572) at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:550) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) 16/04/07 15:57:35 INFO mapreduce.Job: map 100% reduce 0% 16/04/07 15:57:35 INFO mapreduce.Job: Job job_1458819387386_13740 failed with state FAILED due to: Task failed task_1458819387386_13740_m_00 Job failed as tasks failed. failedMaps:1 failedReduces:0 Env: hadoop 2.6.2+hive 2.0 p.s. our orc table contains tinyint column type. > ORC Schema Evolution Issues (Vectorized, ACID, and Non-Vectorized) > -- > > Key: HIVE-11981 > URL: https://issues.apache.org/jira/browse/HIVE-11981 > Project: Hive > Issue Type: Bug > Components: Hive, Transactions >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Labels: TODOC2.0 > Fix For: 2.0.0 > > Attachments: HIVE-11981.01.patch, HIVE-11981.02.patch, > HIVE-11981.03.patch, HIVE-11981.05.patch, HIVE-11981.06.patch, > HIVE-11981.07.patch, HIVE-11981.08.patch, HIVE-11981.09.patch, > HIVE-11981.091.patch, HIVE-11981.092.patch, HIVE-11981.093.patch, > HIVE-11981.094.patch, HIVE-11981.095.patch, HIVE-11981.096.patch, > HIVE-11981.097.patch, HIVE-11981.098.patch, HIVE-11981.099.patch, > HIVE-11981.0991.patch, HIVE-11981.0992.patch, ORC Schema Evolution Issues.docx > > > High priority issues with schema evolution for the ORC file format. > Schema evolution here is limited to adding new columns and a few cases of > column type-widening (e.g. int to bigint). > Renaming columns, deleting column, moving columns and other schema evolution > were not pursued due to lack of importance and lack of time. Also, it > appears a much more sophisticated metadata would be needed to support them. > The biggest issues for users have been adding new columns for ACID table > (HIVE-11421 Support Schema evolution for ACID tables) and vectorization > (HIVE-10598 Vectorization borks when column is added to table). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11981) ORC Schema Evolution Issues (Vectorized, ACID, and Non-Vectorized)
[ https://issues.apache.org/jira/browse/HIVE-11981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15229977#comment-15229977 ] Qiuzhuang Lian commented on HIVE-11981: --- nope, our table doesn't have any struct. here is the DDL, CREATE TABLE `my_orc_table`( `id` string, `store_no_from` string, `store_name_from` string, `store_no_to` string, `store_name_to` string, `order_unit_no_from` string, `order_unit_name_from` string, `order_unit_no_to` string, `order_unit_name_to` string, `store_no` string, `store_name` string, `company_no` string, `order_unit_no` string, `order_unit_name` string, `item_no` string, `item_code` string, `item_name` string, `brand_no` string, `brand_name` string, `category_no` string, `sku_no` string, `size_no` string, `size_kind` string, `bill_no` string, `status` tinyint, `bill_type` int, `in_out_flag` tinyint, `ref_bill_no` string, `ref_bill_type` int, `biz_type` int, `account_type` tinyint, `bill_date` date, `cost` decimal(12,2), `balance_offset` int, `balance_qty` int, `factory_in_offset` int, `factory_in_qty` int, `factory_in_diff_offset` int, `factory_in_diff_qty` int, `transit_in_offset` int, `transit_in_qty` int, `transit_out_offset` int, `transit_out_qty` int, `in_diff_offset` int, `in_diff_qty` int, `out_diff_offset` int, `out_diff_qty` int, `transit_in_account_offset` int, `transit_in_account_qty` int, `transit_out_account_offset` int, `transit_out_account_qty` int, `in_diff_account_offset` int, `in_diff_account_qty` int, `out_diff_account_offset` int, `out_diff_account_qty` int, `lock_offset` int, `lock_qty` int, `occupied_offset` int, `occupied_qty` int, `backup_offset` int, `backup_qty` int, `guest_bad_offset` int, `guest_bad_qty` int, `original_bad_offset` int, `original_bad_qty` int, `bad_transit_offset` int, `bad_transit_qty` int, `bad_diff_offset` int, `bad_diff_qty` int, `return_offset` int, `return_qty` int, `borrow_offset` int, `borrow_qty` int, `create_time` timestamp, `create_timestamp` timestamp, `update_time` timestamp, `sharding_flag` string, `yw_update_time` timestamp, `hive_create_time` timestamp, `biz_date` int) CLUSTERED BY ( id) INTO 10 BUCKETS ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' LOCATION 'hdfs://nn:9000/hive/warehouse/lqz.db/my_orc_table' TBLPROPERTIES ( 'COLUMN_STATS_ACCURATE'='{\"BASIC_STATS\":\"true\"}', 'last_modified_by'='hive', 'last_modified_time'='1460015324', 'numFiles'='23', 'numRows'='33828471', 'orc.compress'='SNAPPY', 'orc.create.index'='true', 'orc.stripe.size'='67108864', 'rawDataSize'='92332902940', 'totalSize'='1474582939', 'transactional'='true', 'transient_lastDdlTime'='1460015745') > ORC Schema Evolution Issues (Vectorized, ACID, and Non-Vectorized) > -- > > Key: HIVE-11981 > URL: https://issues.apache.org/jira/browse/HIVE-11981 > Project: Hive > Issue Type: Bug > Components: Hive, Transactions >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Labels: TODOC2.0 > Fix For: 2.0.0 > > Attachments: HIVE-11981.01.patch, HIVE-11981.02.patch, > HIVE-11981.03.patch, HIVE-11981.05.patch, HIVE-11981.06.patch, > HIVE-11981.07.patch, HIVE-11981.08.patch, HIVE-11981.09.patch, > HIVE-11981.091.patch, HIVE-11981.092.patch, HIVE-11981.093.patch, > HIVE-11981.094.patch, HIVE-11981.095.patch, HIVE-11981.096.patch, > HIVE-11981.097.patch, HIVE-11981.098.patch, HIVE-11981.099.patch, > HIVE-11981.0991.patch, HIVE-11981.0992.patch, ORC Schema Evolution Issues.docx > > > High priority issues with schema evolution for the ORC file format. > Schema evolution here is limited to adding new columns and a few cases of > column type-widening (e.g. int to bigint). > Renaming columns, deleting column, moving columns and other schema evolution > were not pursued due to lack of importance and lack of time. Also, it > appears a much more sophisticated metadata would be needed to support them. > The biggest issues for users have been adding new columns for ACID table > (HIVE-11421 Support Schema evolution for ACID tables) and vectorization > (HIVE-10598 Vectorization borks when column is added to table). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11981) ORC Schema Evolution Issues (Vectorized, ACID, and Non-Vectorized)
[ https://issues.apache.org/jira/browse/HIVE-11981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15230452#comment-15230452 ] Qiuzhuang Lian commented on HIVE-11981: --- For more info, when I compact the table in hive 1.2 version, we see TreeReaderFactory errors as reported in this issue: https://issues.apache.org/jira/browse/HIVE-13432 > ORC Schema Evolution Issues (Vectorized, ACID, and Non-Vectorized) > -- > > Key: HIVE-11981 > URL: https://issues.apache.org/jira/browse/HIVE-11981 > Project: Hive > Issue Type: Bug > Components: Hive, Transactions >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Labels: TODOC2.0 > Fix For: 2.0.0 > > Attachments: HIVE-11981.01.patch, HIVE-11981.02.patch, > HIVE-11981.03.patch, HIVE-11981.05.patch, HIVE-11981.06.patch, > HIVE-11981.07.patch, HIVE-11981.08.patch, HIVE-11981.09.patch, > HIVE-11981.091.patch, HIVE-11981.092.patch, HIVE-11981.093.patch, > HIVE-11981.094.patch, HIVE-11981.095.patch, HIVE-11981.096.patch, > HIVE-11981.097.patch, HIVE-11981.098.patch, HIVE-11981.099.patch, > HIVE-11981.0991.patch, HIVE-11981.0992.patch, ORC Schema Evolution Issues.docx > > > High priority issues with schema evolution for the ORC file format. > Schema evolution here is limited to adding new columns and a few cases of > column type-widening (e.g. int to bigint). > Renaming columns, deleting column, moving columns and other schema evolution > were not pursued due to lack of importance and lack of time. Also, it > appears a much more sophisticated metadata would be needed to support them. > The biggest issues for users have been adding new columns for ACID table > (HIVE-11421 Support Schema evolution for ACID tables) and vectorization > (HIVE-10598 Vectorization borks when column is added to table). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13432) ACID ORC CompactorMR job throws java.lang.ArrayIndexOutOfBoundsException: 7
[ https://issues.apache.org/jira/browse/HIVE-13432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15234424#comment-15234424 ] Qiuzhuang Lian commented on HIVE-13432: --- I will build hive 2.1/2.0.1 and let you know to see if it works. Many thanks. > ACID ORC CompactorMR job throws java.lang.ArrayIndexOutOfBoundsException: 7 > --- > > Key: HIVE-13432 > URL: https://issues.apache.org/jira/browse/HIVE-13432 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 1.2.1 > Environment: Hadoop 2.6.2+Hive 1.2.1 >Reporter: Qiuzhuang Lian >Assignee: Matt McCline > > After initiating HIVE ACID ORC table compaction, the CompactorMR job throws > exception: > Error: java.lang.ArrayIndexOutOfBoundsException: 7 > at > org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$StructTreeReader.(TreeReaderFactory.java:1968) > at > org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2368) > at > org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$StructTreeReader.(TreeReaderFactory.java:1969) > at > org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2368) > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderFactory.createTreeReader(RecordReaderFactory.java:69) > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.(RecordReaderImpl.java:202) > at > org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rowsOptions(ReaderImpl.java:539) > at > org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$ReaderPair.(OrcRawRecordMerger.java:183) > at > org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.(OrcRawRecordMerger.java:466) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRawReader(OrcInputFormat.java:1308) > at > org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:512) > at > org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:491) > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > As a result, we see hadoop exception stack, > 297 failed with state FAILED due to: Task failed > task_1458819387386_11297_m_08 > Job failed as tasks failed. failedMaps:1 failedReduces:0 > 2016-04-06 11:30:57,891 INFO [dn209006-27]: mapreduce.Job > (Job.java:monitorAndPrintJob(1392)) - Counters: 14 > Job Counters > Failed map tasks=16 > Killed map tasks=7 > Launched map tasks=23 > Other local map tasks=13 > Data-local map tasks=6 > Rack-local map tasks=4 > Total time spent by all maps in occupied slots (ms)=412592 > Total time spent by all reduces in occupied slots (ms)=0 > Total time spent by all map tasks (ms)=206296 > Total vcore-seconds taken by all map tasks=206296 > Total megabyte-seconds taken by all map tasks=422494208 > Map-Reduce Framework > CPU time spent (ms)=0 > Physical memory (bytes) snapshot=0 > Virtual memory (bytes) snapshot=0 > 2016-04-06 11:30:57,891 ERROR [dn209006-27]: compactor.Worker > (Worker.java:run(176)) - Caught exception while trying to compact > lqz.my_orc_acid_table. Marking clean to avoid repeated failures, > java.io.IOException: Job failed! > at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:836) > at > org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.run(CompactorMR.java:186) > at org.apache.hadoop.hive.ql.txn.compactor.Worker.run(Worker.java:162) > 2016-04-06 11:30:57,894 ERROR [dn209006-27]: txn.CompactionTxnHandler > (CompactionTxnHandler.java:markCleaned(327)) - Expected to remove at least > one row from completed_txn_components when marking compaction entry as clean! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13432) ACID ORC CompactorMR job throws java.lang.ArrayIndexOutOfBoundsException: 7
[ https://issues.apache.org/jira/browse/HIVE-13432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15234463#comment-15234463 ] Qiuzhuang Lian commented on HIVE-13432: --- Hi Matt, I build hive from hive git branch 2.0 and see patch of 12984, I try compaction again on the same ACID table but still see the same error, , kind: TIMESTAMP , kind: INT ] innerStructSubtype -1 at org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$StructTreeReader.(TreeReaderFactory.java:2092) at org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2518) at org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$StructTreeReader.(TreeReaderFactory.java:2098) at org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2518) at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.(RecordReaderImpl.java:210) at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rowsOptions(ReaderImpl.java:662) at org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$ReaderPair.(OrcRawRecordMerger.java:212) at org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.(OrcRawRecordMerger.java:512) at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRawReader(OrcInputFormat.java:1870) at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:575) at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:554) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) 16/04/11 11:54:21 INFO mapreduce.Job: map 100% reduce 0% 16/04/11 11:54:21 INFO mapreduce.Job: Job job_1458819387386_23797 failed with state FAILED due to: Task failed task_1458819387386_23797_m_01 Job failed as tasks failed. failedMaps:1 failedReduces:0 16/04/11 11:54:21 INFO mapreduce.Job: Counters: 14 Job Counters Failed map tasks=9 Killed map tasks=9 Launched map tasks=18 Other local map tasks=8 Data-local map tasks=4 Rack-local map tasks=6 Total time spent by all maps in occupied slots (ms)=405068 Total time spent by all reduces in occupied slots (ms)=0 Total time spent by all map tasks (ms)=202534 Total vcore-seconds taken by all map tasks=202534 Total megabyte-seconds taken by all map tasks=414789632 Map-Reduce Framework CPU time spent (ms)=0 Physical memory (bytes) snapshot=0 Virtual memory (bytes) snapshot=0 16/04/11 11:54:21 ERROR compactor.Worker: Caught exception while trying to compact id:80,dbname:lqz,tableName:my_acid_orc_table,partName:null,state:,type:MAJOR,runAs:null,tooManyAborts:false,highestTxnId:0. Marking clean to avoid repeated failures, java.io.IOException: Job failed! at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:836) at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.launchCompactionJob(CompactorMR.java:247) at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.run(CompactorMR.java:213) at org.apache.hadoop.hive.ql.txn.compactor.Worker.run(Worker.java:164) 16/04/11 11:54:34 INFO txn.AcidCompactionHistoryService: History reaper reaper ran for 0seconds. isAliveCounter=-2147483642 Please let me know if you need more info for this issue. Regards, Qiuzhuang > ACID ORC CompactorMR job throws java.lang.ArrayIndexOutOfBoundsException: 7 > --- > > Key: HIVE-13432 > URL: https://issues.apache.org/jira/browse/HIVE-13432 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 1.2.1 > Environment: Hadoop 2.6.2+Hive 1.2.1 >Reporter: Qiuzhuang Lian >Assignee: Matt McCline > > After initiating HIVE ACID ORC table compaction, the CompactorMR job throws > exception: > Error: java.lang.ArrayIndexOutOfBoundsException: 7 > at > org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$StructTreeReader.(TreeReaderFactory.java:1968) > at > org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2368) > at > org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$StructTreeReader.(TreeReaderFactory.java:1969) > at > org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2368) > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderFactory.createTreeReader(RecordReaderFactory.java:69) > at > org.apache.hadoop.
[jira] [Commented] (HIVE-13432) ACID ORC CompactorMR job throws java.lang.ArrayIndexOutOfBoundsException: 7
[ https://issues.apache.org/jira/browse/HIVE-13432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15234480#comment-15234480 ] Qiuzhuang Lian commented on HIVE-13432: --- For more info, I also see the regression to the ACID feature for delete statement, 16/04/11 12:38:43 WARN shims.HadoopShimsSecure: Can't fetch tasklog: TaskLogServlet is not supported in MR2 mode. Task with the most failures(4): - Task ID: task_1458819387386_23814_r_03 URL: http://nn209003:8088/taskdetails.jsp?jobid=job_1458819387386_23814&tipid=task_1458819387386_23814_r_03 - Diagnostic Messages for this Task: Error: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row (tag=0) {"key":{"reducesinkkey0":{"transactionid":0,"bucketid":43,"rowid":0}},"value":null} at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:257) at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:444) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row (tag=0) {"key":{"reducesinkkey0":{"transactionid":0,"bucketid":43,"rowid":0}},"value":null} at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:245) ... 7 more Caused by: java.lang.ArrayIndexOutOfBoundsException: 1 at org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:759) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:837) at org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:97) at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:236) ... 7 more 16/04/11 12:38:43 ERROR exec.Task: Task with the most failures(4): - Task ID: task_1458819387386_23814_r_03 URL: http://nn209003:8088/taskdetails.jsp?jobid=job_1458819387386_23814&tipid=task_1458819387386_23814_r_03 - Diagnostic Messages for this Task: Error: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row (tag=0) {"key":{"reducesinkkey0":{"transactionid":0,"bucketid":43,"rowid":0}},"value":null} at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:257) at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:444) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row (tag=0) {"key":{"reducesinkkey0":{"transactionid":0,"bucketid":43,"rowid":0}},"value":null} at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:245) ... 7 more Caused by: java.lang.ArrayIndexOutOfBoundsException: 1 at org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:759) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:837) at org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:97) at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:236) > ACID ORC CompactorMR job throws java.lang.ArrayIndexOutOfBoundsException: 7 > --- > > Key: HIVE-13432 > URL: https://issues.apache.org/jira/browse/HIVE-13432 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 1.2.1 > Environment: Hadoop 2.6.2+Hive 1.2.1 >Reporter: Qiuzhuang Lian >Assignee: Matt McCline > > After initiating HIVE ACID ORC table compaction, the CompactorMR job throws > exception: > Error: java.lang.ArrayIndexOutOfBoundsException: 7 > at > org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$StructTreeReader.(TreeReaderFactory.java:1968) > at > org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2368) > at > org.apache.hadoop.hive.ql.io.orc.TreeReade
[jira] [Commented] (HIVE-12064) prevent transactional=false
[ https://issues.apache.org/jira/browse/HIVE-12064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15234556#comment-15234556 ] Qiuzhuang Lian commented on HIVE-12064: --- We create the ACID table from txt table yet failed with here errors, 16/04/11 14:27:34 ERROR exec.DDLTask: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:The table must be bucketed and stored using an ACID compliant format (such as ORC)) at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:783) at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:4032) at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:322) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:158) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:101) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1834) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1578) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1355) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1178) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1166) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:236) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:187) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403) at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:782) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:721) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:648) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) Caused by: MetaException(message:The table must be bucketed and stored using an ACID compliant format (such as ORC)) at org.apache.hadoop.hive.metastore.TransactionalValidationListener.handleCreateTableTransactionalProp(TransactionalValidationListener.java:143) at org.apache.hadoop.hive.metastore.TransactionalValidationListener.handle(TransactionalValidationListener.java:58) This appears in hive branch-2.0. However, it works in hive 1.2.1 stable version. > prevent transactional=false > --- > > Key: HIVE-12064 > URL: https://issues.apache.org/jira/browse/HIVE-12064 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Critical > Fix For: 1.3.0, 2.1.0, 2.0.1 > > Attachments: HIVE-12064.2.patch, HIVE-12064.3.patch, > HIVE-12064.4.patch, HIVE-12064.5.patch, HIVE-12064.6.patch, > HIVE-12064.7.patch, HIVE-12064.branch-1.patch, HIVE-12064.branch-2.0.patch, > HIVE-12064.patch > > > currently a tblproperty transactional=true must be set to make a table behave > in ACID compliant way. > This is misleading in that it seems like changing it to transactional=false > makes the table non-acid but on disk layout of acid table is different than > plain tables. So changing this property may cause wrong data to be returned. > Should prevent transactional=false. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12064) prevent transactional=false
[ https://issues.apache.org/jira/browse/HIVE-12064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15236646#comment-15236646 ] Qiuzhuang Lian commented on HIVE-12064: --- Create a new ACID table based on txt table. > prevent transactional=false > --- > > Key: HIVE-12064 > URL: https://issues.apache.org/jira/browse/HIVE-12064 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Critical > Fix For: 1.3.0, 2.1.0, 2.0.1 > > Attachments: HIVE-12064.2.patch, HIVE-12064.3.patch, > HIVE-12064.4.patch, HIVE-12064.5.patch, HIVE-12064.6.patch, > HIVE-12064.7.patch, HIVE-12064.branch-1.patch, HIVE-12064.branch-2.0.patch, > HIVE-12064.patch > > > currently a tblproperty transactional=true must be set to make a table behave > in ACID compliant way. > This is misleading in that it seems like changing it to transactional=false > makes the table non-acid but on disk layout of acid table is different than > plain tables. So changing this property may cause wrong data to be returned. > Should prevent transactional=false. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12064) prevent transactional=false
[ https://issues.apache.org/jira/browse/HIVE-12064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15236647#comment-15236647 ] Qiuzhuang Lian commented on HIVE-12064: --- Create a new ACID table based on txt table. > prevent transactional=false > --- > > Key: HIVE-12064 > URL: https://issues.apache.org/jira/browse/HIVE-12064 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Critical > Fix For: 1.3.0, 2.1.0, 2.0.1 > > Attachments: HIVE-12064.2.patch, HIVE-12064.3.patch, > HIVE-12064.4.patch, HIVE-12064.5.patch, HIVE-12064.6.patch, > HIVE-12064.7.patch, HIVE-12064.branch-1.patch, HIVE-12064.branch-2.0.patch, > HIVE-12064.patch > > > currently a tblproperty transactional=true must be set to make a table behave > in ACID compliant way. > This is misleading in that it seems like changing it to transactional=false > makes the table non-acid but on disk layout of acid table is different than > plain tables. So changing this property may cause wrong data to be returned. > Should prevent transactional=false. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12064) prevent transactional=false
[ https://issues.apache.org/jira/browse/HIVE-12064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15238456#comment-15238456 ] Qiuzhuang Lian commented on HIVE-12064: --- Yes, code seems to check this condition. It is incompatible with 1.X which seems OK. At least, this should be put into documentation. I did create the ORC table without transaction enabled from txt table and it works. > prevent transactional=false > --- > > Key: HIVE-12064 > URL: https://issues.apache.org/jira/browse/HIVE-12064 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Critical > Fix For: 1.3.0, 2.1.0, 2.0.1 > > Attachments: HIVE-12064.2.patch, HIVE-12064.3.patch, > HIVE-12064.4.patch, HIVE-12064.5.patch, HIVE-12064.6.patch, > HIVE-12064.7.patch, HIVE-12064.branch-1.patch, HIVE-12064.branch-2.0.patch, > HIVE-12064.patch > > > currently a tblproperty transactional=true must be set to make a table behave > in ACID compliant way. > This is misleading in that it seems like changing it to transactional=false > makes the table non-acid but on disk layout of acid table is different than > plain tables. So changing this property may cause wrong data to be returned. > Should prevent transactional=false. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12155) hive exited with status 5
[ https://issues.apache.org/jira/browse/HIVE-12155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qiuzhuang Lian updated HIVE-12155: -- Summary: hive exited with status 5 (was: hive exit with status 5) > hive exited with status 5 > - > > Key: HIVE-12155 > URL: https://issues.apache.org/jira/browse/HIVE-12155 > Project: Hive > Issue Type: Bug > Components: CLI, Clients >Affects Versions: 1.2.1 > Environment: Sqoop 1.4.5 & hadoop 2.6 & hive 1.2.1 >Reporter: Qiuzhuang Lian > > We run sqoop-hive import job via RunJar to harness parallelisms runnings. > Sqoop hive import works very well but suddenly the sqoop-hive import job JVM > exits with "Hive exited with status 5" error during hive import phrase which > invokes HIVE CLI via java Process. Futhermore, we can't find any related > hive logs under /tmp/hive/hive_*.log. The error blocks all futher sqoop > import jobs. As a result, we have to restart system and it works well again. > The log detail is as follows, > Encountered IOException running import job: java.io.IOException: Hive exited > with status 5 > at > org.apache.sqoop.hive.HiveImport.executeExternalHiveScript(HiveImport.java:385) > at org.apache.sqoop.hive.HiveImport.executeScript(HiveImport.java:335) > at org.apache.sqoop.hive.HiveImport.importTable(HiveImport.java:239) > at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:511) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12155) hive exit with status 5
[ https://issues.apache.org/jira/browse/HIVE-12155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qiuzhuang Lian updated HIVE-12155: -- Description: We run sqoop-hive import job via RunJar to harness parallelisms runnings. Sqoop hive import works very well but suddenly the sqoop-hive import job JVM exits with "Hive exited with status 5" error during hive import phrase which invokes HIVE CLI via java Process. Futhermore, we can't find any related hive logs under /tmp/hive/hive_*.log. The error blocks all futher sqoop import jobs. As a result, we have to restart system and it works well again. The log detail is as follows, Encountered IOException running import job: java.io.IOException: Hive exited with status 5 at org.apache.sqoop.hive.HiveImport.executeExternalHiveScript(HiveImport.java:385) at org.apache.sqoop.hive.HiveImport.executeScript(HiveImport.java:335) at org.apache.sqoop.hive.HiveImport.importTable(HiveImport.java:239) at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:511) was: When run the sqoop-hive import job via RunJar to harness parallelisms job runnings. Sqoop hive import work very well but suddenly all sqoop-hive import job JVM exits with "Hive exited with status 5" hint during hive import phrase which invokes HIVE CLI via java process. Futhermore, we can not find any related hive logs under /tmp/hive/hive_*.log. This error blocks all futher sqoop import job. As a result, we have to restart system and it works well again. The log detail is as follows, Encountered IOException running import job: java.io.IOException: Hive exited with status 5 at org.apache.sqoop.hive.HiveImport.executeExternalHiveScript(HiveImport.java:385) at org.apache.sqoop.hive.HiveImport.executeScript(HiveImport.java:335) at org.apache.sqoop.hive.HiveImport.importTable(HiveImport.java:239) at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:511) > hive exit with status 5 > --- > > Key: HIVE-12155 > URL: https://issues.apache.org/jira/browse/HIVE-12155 > Project: Hive > Issue Type: Bug > Components: CLI, Clients >Affects Versions: 1.2.1 > Environment: Sqoop 1.4.5 & hadoop 2.6 & hive 1.2.1 >Reporter: Qiuzhuang Lian > > We run sqoop-hive import job via RunJar to harness parallelisms runnings. > Sqoop hive import works very well but suddenly the sqoop-hive import job JVM > exits with "Hive exited with status 5" error during hive import phrase which > invokes HIVE CLI via java Process. Futhermore, we can't find any related > hive logs under /tmp/hive/hive_*.log. The error blocks all futher sqoop > import jobs. As a result, we have to restart system and it works well again. > The log detail is as follows, > Encountered IOException running import job: java.io.IOException: Hive exited > with status 5 > at > org.apache.sqoop.hive.HiveImport.executeExternalHiveScript(HiveImport.java:385) > at org.apache.sqoop.hive.HiveImport.executeScript(HiveImport.java:335) > at org.apache.sqoop.hive.HiveImport.importTable(HiveImport.java:239) > at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:511) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12155) hive exited with status 5
[ https://issues.apache.org/jira/browse/HIVE-12155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qiuzhuang Lian updated HIVE-12155: -- Environment: sqoop 1.4.5 & hadoop 2.6 & hive 1.2.1 (was: Sqoop 1.4.5 & hadoop 2.6 & hive 1.2.1) > hive exited with status 5 > - > > Key: HIVE-12155 > URL: https://issues.apache.org/jira/browse/HIVE-12155 > Project: Hive > Issue Type: Bug > Components: CLI, Clients >Affects Versions: 1.2.1 > Environment: sqoop 1.4.5 & hadoop 2.6 & hive 1.2.1 >Reporter: Qiuzhuang Lian > > We run sqoop-hive import job via RunJar to harness parallelisms runnings. > Sqoop hive import works very well but suddenly the sqoop-hive import job JVM > exits with "Hive exited with status 5" error during hive import phrase which > invokes HIVE CLI via java Process. Futhermore, we can't find any related > hive logs under /tmp/hive/hive_*.log. The error blocks all futher sqoop > import jobs. As a result, we have to restart system and it works well again. > The log detail is as follows, > Encountered IOException running import job: java.io.IOException: Hive exited > with status 5 > at > org.apache.sqoop.hive.HiveImport.executeExternalHiveScript(HiveImport.java:385) > at org.apache.sqoop.hive.HiveImport.executeScript(HiveImport.java:335) > at org.apache.sqoop.hive.HiveImport.importTable(HiveImport.java:239) > at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:511) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9970) Hive on spark
[ https://issues.apache.org/jira/browse/HIVE-9970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15001843#comment-15001843 ] Qiuzhuang Lian commented on HIVE-9970: -- Hi Xuefu, I use spark trunk version of 1.6 + hadoop 2.6.0 + hive 1.2.1, when running the HQL in HIVE CLI, We got following error, 2015-11-12 16:31:17,245 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(569)) - 15/11/12 16:31:17 ERROR Utils: uncaught error in thread SparkListenerBus, stopping SparkContext 2015-11-12 16:31:17,246 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(569)) - java.lang.AbstractMethodError 2015-11-12 16:31:17,246 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(569)) -at org.apache.spark.scheduler.SparkListenerBus$class.onPostEvent(SparkListenerBus.scala:62) 2015-11-12 16:31:17,246 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(569)) -at org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31) 2015-11-12 16:31:17,246 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(569)) -at org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31) 2015-11-12 16:31:17,246 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(569)) -at org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:55) Any ideas? > Hive on spark > - > > Key: HIVE-9970 > URL: https://issues.apache.org/jira/browse/HIVE-9970 > Project: Hive > Issue Type: Bug >Reporter: Amithsha > > Hi all, > Recently i have configured Spark 1.2.0 and my environment is hadoop > 2.6.0 hive 1.1.0 Here i have tried hive on Spark while executing > insert into i am getting the following g error. > Query ID = hadoop2_20150313162828_8764adad-a8e4-49da-9ef5-35e4ebd6bc63 > Total jobs = 1 > Launching Job 1 out of 1 > In order to change the average load for a reducer (in bytes): > set hive.exec.reducers.bytes.per.reducer= > In order to limit the maximum number of reducers: > set hive.exec.reducers.max= > In order to set a constant number of reducers: > set mapreduce.job.reduces= > Failed to execute spark task, with exception > 'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to create > spark client.)' > FAILED: Execution Error, return code 1 from > org.apache.hadoop.hive.ql.exec.spark.SparkTask > Have added the spark-assembly jar in hive lib > And also in hive console using the command add jar followed by the steps > set spark.home=/opt/spark-1.2.1/; > add jar > /opt/spark-1.2.1/assembly/target/scala-2.10/spark-assembly-1.2.1-hadoop2.4.0.jar; > set hive.execution.engine=spark; > set spark.master=spark://xxx:7077; > set spark.eventLog.enabled=true; > set spark.executor.memory=512m; > set spark.serializer=org.apache.spark.serializer.KryoSerializer; > Can anyone suggest > Thanks & Regards > Amithsha -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10610) hive command fails to get hadoop version
[ https://issues.apache.org/jira/browse/HIVE-10610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15021879#comment-15021879 ] Qiuzhuang Lian commented on HIVE-10610: --- We still run into this issue in hive-1.2.1 with parallelly running hive shell embedded in Java, e.g. in sqoop hive import. Somtimes it throws error as follows per sqoop. According to log, it seems that Java shell process executing hive command is under non-existing linux dir. [20151123 14:29:24-DEBUG][HiveImport.java:334][RMI TCP Connection(60)-172.17.209.254]-- Using external Hive process. [20151123 14:29:24-INFO][LoggingAsyncSink.java:85][Thread-441]-- shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory [20151123 14:29:24-INFO][LoggingAsyncSink.java:85][Thread-441]-- chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory [20151123 14:29:24-INFO][LoggingAsyncSink.java:85][Thread-441]-- chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory [20151123 14:29:24-INFO][LoggingAsyncSink.java:85][Thread-441]-- shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory [20151123 14:29:24-INFO][LoggingAsyncSink.java:85][Thread-441]-- chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory [20151123 14:29:24-INFO][LoggingAsyncSink.java:85][Thread-441]-- chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory [20151123 14:29:24-INFO][LoggingAsyncSink.java:85][Thread-441]-- chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory [20151123 14:29:24-INFO][LoggingAsyncSink.java:85][Thread-440]-- Unable to determine Hadoop version information. [20151123 14:29:24-INFO][LoggingAsyncSink.java:85][Thread-440]-- 'hadoop version' returned: [20151123 14:29:24-INFO][LoggingAsyncSink.java:85][Thread-441]-- shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory [20151123 14:29:24-INFO][LoggingAsyncSink.java:85][Thread-441]-- chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory [20151123 14:29:24-INFO][LoggingAsyncSink.java:85][Thread-441]-- chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory [20151123 14:29:24-INFO][LoggingAsyncSink.java:85][Thread-441]-- chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory [20151123 14:29:24-INFO][LoggingAsyncSink.java:85][Thread-440]-- Error occurred during initialization of VM java.lang.Error: Properties init: Could not determine current working directory. at java.lang.System.initProperties(Native Method) at java.lang.System.initializeSystemClass(System.java:1119) [20151123 14:29:24-ERROR][ImportTool.java:609][RMI TCP Connection(60)-172.17.209.254]-- Encountered IOException running import job: java.io.IOException: Hive exited with status 5 > hive command fails to get hadoop version > > > Key: HIVE-10610 > URL: https://issues.apache.org/jira/browse/HIVE-10610 > Project: Hive > Issue Type: Bug >Reporter: Shwetha G S >Assignee: Shwetha G S > Fix For: 1.2.0 > > Attachments: HIVE-10610.patch > > > NO PRECOMMIT TESTS > If debug level logging is enabled, hive command fails with the following > exception: > {noformat} > apache-hive-1.2.0-SNAPSHOT-bin$ ./bin/hive > Unable to determine Hadoop version information from 13:54:07,683 > 'hadoop version' returned: > 2015-05-05 13:54:08,014 DEBUG - [main:] ~ version: 2.5.0-cdh5.3.3 > (VersionInfo:171) Hadoop 2.5.0-cdh5.3.3 Subversion > http://github.com/cloudera/hadoop -r 82a65209d6e9e4a2b41fdbcd8190c7ea38730627 > Compiled by jenkins on 2015-04-08T22:00Z Compiled with protoc 2.5.0 From > source with checksum 1531e104cdad7489656f44875f3334b This command was run > using > /Users/sshivalingamurthy/installs/hadoop-2.5.0-cdh5.3.3/share/hadoop/common/hadoop-common-2.5.0-cdh5.3.3.jar > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)