[jira] [Commented] (HIVE-15415) Random "java.util.ConcurrentModificationException"
[ https://issues.apache.org/jira/browse/HIVE-15415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16055361#comment-16055361 ] kxxu commented on HIVE-15415: - I also got this error, this is my stack trace 2017-06-20T16:23:05,526 ERROR [HiveServer2-Background-Pool: Thread-222]: operation.Operation (SQLOperation.java:run(337)) - Error running hive query: org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask. java.util.ConcurrentModificationException at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:401) ~[hive-service-2.1.2-SNAPSHOT.jar:2.1.2-SNAPSHOT] at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:244) ~[hive-service-2.1.2-SNAPSHOT.jar:2.1.2-SNAPSHOT] at org.apache.hive.service.cli.operation.SQLOperation.access$800(SQLOperation.java:91) ~[hive-service-2.1.2-SNAPSHOT.jar:2.1.2-SNAPSHOT] at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:334) [hive-service-2.1.2-SNAPSHOT.jar:2.1.2-SNAPSHOT] at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_121] at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_121] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) [hadoop-common.jar:?] at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:347) [hive-service-2.1.2-SNAPSHOT.jar:2.1.2-SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_121] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_121] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121] Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.util.ConcurrentModificationException at org.apache.hadoop.hive.ql.metadata.Hive.moveFile(Hive.java:2992) ~[hive-exec-2.1.2-SNAPSHOT.jar:2.1.2-SNAPSHOT] at org.apache.hadoop.hive.ql.metadata.Hive.replaceFiles(Hive.java:3248) ~[hive-exec-2.1.2-SNAPSHOT.jar:2.1.2-SNAPSHOT] at org.apache.hadoop.hive.ql.metadata.Hive.loadTable(Hive.java:1828) ~[hive-exec-2.1.2-SNAPSHOT.jar:2.1.2-SNAPSHOT] at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:356) ~[hive-exec-2.1.2-SNAPSHOT.jar:2.1.2-SNAPSHOT] at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197) ~[hive-exec-2.1.2-SNAPSHOT.jar:2.1.2-SNAPSHOT] at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100) ~[hive-exec-2.1.2-SNAPSHOT.jar:2.1.2-SNAPSHOT] at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2073) ~[hive-exec-2.1.2-SNAPSHOT.jar:2.1.2-SNAPSHOT] at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1744) ~[hive-exec-2.1.2-SNAPSHOT.jar:2.1.2-SNAPSHOT] at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1453) ~[hive-exec-2.1.2-SNAPSHOT.jar:2.1.2-SNAPSHOT] at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1171) ~[hive-exec-2.1.2-SNAPSHOT.jar:2.1.2-SNAPSHOT] at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1166) ~[hive-exec-2.1.2-SNAPSHOT.jar:2.1.2-SNAPSHOT] at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:242) ~[hive-service-2.1.2-SNAPSHOT.jar:2.1.2-SNAPSHOT] ... 11 more Caused by: java.util.ConcurrentModificationException at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901) ~[?:1.8.0_121] at java.util.ArrayList$Itr.next(ArrayList.java:851) ~[?:1.8.0_121] at java.util.AbstractCollection.toString(AbstractCollection.java:461) ~[?:1.8.0_121] at java.lang.String.valueOf(String.java:2994) ~[?:1.8.0_121] at java.lang.StringBuilder.append(StringBuilder.java:131) ~[?:1.8.0_121] at org.apache.hadoop.fs.permission.AclStatus.toString(AclStatus.java:98) ~[hadoop-common.jar:?] at org.apache.hadoop.hive.io.HdfsUtils.setFullFileStatus(HdfsUtils.java:75) ~[hive-exec-2.1.2-SNAPSHOT.jar:2.1.2-SNAPSHOT] at org.apache.hadoop.hive.ql.metadata.Hive$3.call(Hive.java:2969) ~[hive-exec-2.1.2-SNAPSHOT.jar:2.1.2-SNAPSHOT] at org.apache.hadoop.hive.ql.metadata.Hive$3.call(Hive.java:2962) ~[hive-exec-2.1.2-SNAPSHOT.jar:2.1.2-SNAPSHOT] ... 4 more > Random "java.util.ConcurrentModificationException" > -- > > Key: HIVE-15415 > URL: https://issues.apache.org/jira/browse/HIVE-15415 > Project: Hive > Issue Type: Bug >
[jira] [Commented] (HIVE-15415) Random "java.util.ConcurrentModificationException"
[
https://issues.apache.org/jira/browse/HIVE-15415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15742738#comment-15742738
]
Vihang Karajgaonkar commented on HIVE-15415:
Hi [~BigDataOrange] can you paste the stack trace here so that we can confirm
if it is same as HIVE-15355? Thanks!
> Random "java.util.ConcurrentModificationException"
> --
>
> Key: HIVE-15415
> URL: https://issues.apache.org/jira/browse/HIVE-15415
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 2.1.0
> Environment: Hadoop 2.7.3, Hive 2.1.0
>Reporter: Alexandre Linte
>
> I'm regularly facing Hive job failures through Oozie or through the beeline
> CLI. The jobs exit with an error "FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.MoveTask.
> java.util.ConcurrentModificationException (state=08S01,code=1)" but not 100%
> of the time.
> it's also important to underline that only one user is working on the table
> when the jobs are running.
> - stderr
> {noformat}
> Connecting to jdbc:hive2://hiveserver2.bigdata.fr:1/default
> Connected to: Apache Hive (version 2.1.0)
> Driver: Hive JDBC (version 2.1.0)
> Transaction isolation: TRANSACTION_REPEATABLE_READ
> No rows affected (1.475 seconds)
> No rows affected (0.004 seconds)
> No rows affected (0.004 seconds)
> No rows affected (58.977 seconds)
> No rows affected (5.524 seconds)
> No rows affected (5.235 seconds)
> Error: Error while processing statement: FAILED: Execution Error, return code
> 1 from org.apache.hadoop.hive.ql.exec.MoveTask.
> java.util.ConcurrentModificationException (state=08S01,code=1)
> Closing: 0: jdbc:hive2://hiveserver2.bigdata.fr:1/default
> Intercepting System.exit(2)
> {noformat}
> - stdout
> {noformat}
> Beeline command arguments :
> -u
> jdbc:hive2://hiveserver2.bigdata.fr:1/default
> -n
> my_user
> -p
> DUMMY
> -d
> org.apache.hive.jdbc.HiveDriver
> -f
> full_job
> -a
> delegationToken
> --hiveconf
> mapreduce.job.tags=oozie-75b060aacd7ec48c4ed637855e413280
> Fetching child yarn jobs
> tag id : oozie-75b060aacd7ec48c4ed637855e413280
> Child yarn jobs are found -
> =
> >>> Invoking Beeline command line now >>>
> 0: jdbc:hive2://hiveserver2.bigdata.fr> use my_db;
> 0: jdbc:hive2://hiveserver2.bigdata.fr> set hive.execution.engine=tez;
> 0: jdbc:hive2://hiveserver2.bigdata.fr> set tez.queue.name=tez_queue;
> 0: jdbc:hive2://hiveserver2.bigdata.fr>
> 0: jdbc:hive2://hiveserver2.bigdata.fr> insert overwrite table main_table
> ^M_fd_livcfm
> . . . . . . . . . . . . . . . . . . . . . . .> select
> . . . . . . . . . . . . . . . . . . . . . . .> col.co_cd as co_cd,
> . . . . . . . . . . . . . . . . . . . . . . .> col.line_co_cd as line_co_cd,
> . . . . . . . . . . . . . . . . . . . . . . .>
> unix_timestamp(min(tt.statut_dt)) ^M as statut_dt
> . . . . . . . . . . . . . . . . . . . . . . .> from
> dlk_scf_rn_customer_order_li ^Mne col
> . . . . . . . . . . . . . . . . . . . . . . .> join
> dlk_scf_rn_shipment_handling ^M_utility shu
> . . . . . . . . . . . . . . . . . . . . . . .> on shu.co_cd =col.co_cd
> . . . . . . . . . . . . . . . . . . . . . . .> and shu.line_co_cd =
> col.line_co_ ^Mcd
> . . . . . . . . . . . . . . . . . . . . . . .> join ( select
> scaler_internal_ref ^M, statut_dt,recep_number,state,reason
> . . . . . . . . . . . . . . . . . . . . . . .> from
> dlk_scf_rn_transport_trackin ^Mg where state='LIV' and reason='CFM' ) tt
> . . . . . . . . . . . . . . . . . . . . . . .> on
> concat('CAL',shu.c_waybill_no) ^M =tt.scaler_internal_ref group by
> col.co_cd,col.line_co_cd;
> Heart beat
> Heart beat
> 0: jdbc:hive2://hiveserver2.bigdata.fr>
> 0: jdbc:hive2://hiveserver2.bigdata.fr> insert overwrite table main_table
> ^M_fd_cae
> . . . . . . . . . . . . . . . . . . . . . . .> select
> . . . . . . . . . . . . . . . . . . . . . . .> po_cd as cae, line_po_cd as
> lcae, ^M origin_co_cd, origin_line_co_cd
> . . . . . . . . . . . . . . . . . . . . . . .> from
> dlk_scf_rn_purchase_order_li ^Mne
> . . . . . . . . . . . . . . . . . . . . . . .> where instr(po_cd,"7")=1;
> 0: jdbc:hive2://hiveserver2.bigdata.fr>
> 0: jdbc:hive2://hiveserver2.bigdata.fr> insert overwrite table main_table
> ^M_fd_cai
> . . . . . . . . . . . . . . . . . . . . . . .> select
> . . . . . . . . . . . . . . . . . . . . . . .> po_cd as cai, line_po_cd as
> lcai, ^M origin_co_cd, origin_line_co_cd
> . . . . . . . . . . . . . . . . . . . . . . .> from
> dlk_scf_rn_purchase_order_li ^Mne
> . . . . . . . . . . . . . . . . . . . . . . .> where
[jira] [Commented] (HIVE-15415) Random "java.util.ConcurrentModificationException"
[
https://issues.apache.org/jira/browse/HIVE-15415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15741905#comment-15741905
]
Alexandre Linte commented on HIVE-15415:
Hi [~rajesh.balamohan],
the use case seems to be very similar. In all cases, I can confirm that the
error happens only when doing an INSERT OVERWRITE query.
> Random "java.util.ConcurrentModificationException"
> --
>
> Key: HIVE-15415
> URL: https://issues.apache.org/jira/browse/HIVE-15415
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 2.1.0
> Environment: Hadoop 2.7.3, Hive 2.1.0
>Reporter: Alexandre Linte
>
> I'm regularly facing Hive job failures through Oozie or through the beeline
> CLI. The jobs exit with an error "FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.MoveTask.
> java.util.ConcurrentModificationException (state=08S01,code=1)" but not 100%
> of the time.
> it's also important to underline that only one user is working on the table
> when the jobs are running.
> - stderr
> {noformat}
> Connecting to jdbc:hive2://hiveserver2.bigdata.fr:1/default
> Connected to: Apache Hive (version 2.1.0)
> Driver: Hive JDBC (version 2.1.0)
> Transaction isolation: TRANSACTION_REPEATABLE_READ
> No rows affected (1.475 seconds)
> No rows affected (0.004 seconds)
> No rows affected (0.004 seconds)
> No rows affected (58.977 seconds)
> No rows affected (5.524 seconds)
> No rows affected (5.235 seconds)
> Error: Error while processing statement: FAILED: Execution Error, return code
> 1 from org.apache.hadoop.hive.ql.exec.MoveTask.
> java.util.ConcurrentModificationException (state=08S01,code=1)
> Closing: 0: jdbc:hive2://hiveserver2.bigdata.fr:1/default
> Intercepting System.exit(2)
> {noformat}
> - stdout
> {noformat}
> Beeline command arguments :
> -u
> jdbc:hive2://hiveserver2.bigdata.fr:1/default
> -n
> my_user
> -p
> DUMMY
> -d
> org.apache.hive.jdbc.HiveDriver
> -f
> full_job
> -a
> delegationToken
> --hiveconf
> mapreduce.job.tags=oozie-75b060aacd7ec48c4ed637855e413280
> Fetching child yarn jobs
> tag id : oozie-75b060aacd7ec48c4ed637855e413280
> Child yarn jobs are found -
> =
> >>> Invoking Beeline command line now >>>
> 0: jdbc:hive2://hiveserver2.bigdata.fr> use my_db;
> 0: jdbc:hive2://hiveserver2.bigdata.fr> set hive.execution.engine=tez;
> 0: jdbc:hive2://hiveserver2.bigdata.fr> set tez.queue.name=tez_queue;
> 0: jdbc:hive2://hiveserver2.bigdata.fr>
> 0: jdbc:hive2://hiveserver2.bigdata.fr> insert overwrite table main_table
> ^M_fd_livcfm
> . . . . . . . . . . . . . . . . . . . . . . .> select
> . . . . . . . . . . . . . . . . . . . . . . .> col.co_cd as co_cd,
> . . . . . . . . . . . . . . . . . . . . . . .> col.line_co_cd as line_co_cd,
> . . . . . . . . . . . . . . . . . . . . . . .>
> unix_timestamp(min(tt.statut_dt)) ^M as statut_dt
> . . . . . . . . . . . . . . . . . . . . . . .> from
> dlk_scf_rn_customer_order_li ^Mne col
> . . . . . . . . . . . . . . . . . . . . . . .> join
> dlk_scf_rn_shipment_handling ^M_utility shu
> . . . . . . . . . . . . . . . . . . . . . . .> on shu.co_cd =col.co_cd
> . . . . . . . . . . . . . . . . . . . . . . .> and shu.line_co_cd =
> col.line_co_ ^Mcd
> . . . . . . . . . . . . . . . . . . . . . . .> join ( select
> scaler_internal_ref ^M, statut_dt,recep_number,state,reason
> . . . . . . . . . . . . . . . . . . . . . . .> from
> dlk_scf_rn_transport_trackin ^Mg where state='LIV' and reason='CFM' ) tt
> . . . . . . . . . . . . . . . . . . . . . . .> on
> concat('CAL',shu.c_waybill_no) ^M =tt.scaler_internal_ref group by
> col.co_cd,col.line_co_cd;
> Heart beat
> Heart beat
> 0: jdbc:hive2://hiveserver2.bigdata.fr>
> 0: jdbc:hive2://hiveserver2.bigdata.fr> insert overwrite table main_table
> ^M_fd_cae
> . . . . . . . . . . . . . . . . . . . . . . .> select
> . . . . . . . . . . . . . . . . . . . . . . .> po_cd as cae, line_po_cd as
> lcae, ^M origin_co_cd, origin_line_co_cd
> . . . . . . . . . . . . . . . . . . . . . . .> from
> dlk_scf_rn_purchase_order_li ^Mne
> . . . . . . . . . . . . . . . . . . . . . . .> where instr(po_cd,"7")=1;
> 0: jdbc:hive2://hiveserver2.bigdata.fr>
> 0: jdbc:hive2://hiveserver2.bigdata.fr> insert overwrite table main_table
> ^M_fd_cai
> . . . . . . . . . . . . . . . . . . . . . . .> select
> . . . . . . . . . . . . . . . . . . . . . . .> po_cd as cai, line_po_cd as
> lcai, ^M origin_co_cd, origin_line_co_cd
> . . . . . . . . . . . . . . . . . . . . . . .> from
> dlk_scf_rn_purchase_order_li ^Mne
> . . . . . . . . . . . . . . . . . .
[jira] [Commented] (HIVE-15415) Random "java.util.ConcurrentModificationException"
[
https://issues.apache.org/jira/browse/HIVE-15415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15741622#comment-15741622
]
Rajesh Balamohan commented on HIVE-15415:
-
HIVE-15355?
> Random "java.util.ConcurrentModificationException"
> --
>
> Key: HIVE-15415
> URL: https://issues.apache.org/jira/browse/HIVE-15415
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 2.1.0
> Environment: Hadoop 2.7.3, Hive 2.1.0
>Reporter: Alexandre Linte
>
> I'm regularly facing Hive job failures through Oozie or through the beeline
> CLI. The jobs exit with an error "FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.MoveTask.
> java.util.ConcurrentModificationException (state=08S01,code=1)" but not 100%
> of the time.
> it's also important to underline that only one user is working on the table
> when the jobs are running.
> - stderr
> {noformat}
> Connecting to jdbc:hive2://hiveserver2.bigdata.fr:1/default
> Connected to: Apache Hive (version 2.1.0)
> Driver: Hive JDBC (version 2.1.0)
> Transaction isolation: TRANSACTION_REPEATABLE_READ
> No rows affected (1.475 seconds)
> No rows affected (0.004 seconds)
> No rows affected (0.004 seconds)
> No rows affected (58.977 seconds)
> No rows affected (5.524 seconds)
> No rows affected (5.235 seconds)
> Error: Error while processing statement: FAILED: Execution Error, return code
> 1 from org.apache.hadoop.hive.ql.exec.MoveTask.
> java.util.ConcurrentModificationException (state=08S01,code=1)
> Closing: 0: jdbc:hive2://hiveserver2.bigdata.fr:1/default
> Intercepting System.exit(2)
> {noformat}
> - stdout
> {noformat}
> Beeline command arguments :
> -u
> jdbc:hive2://hiveserver2.bigdata.fr:1/default
> -n
> my_user
> -p
> DUMMY
> -d
> org.apache.hive.jdbc.HiveDriver
> -f
> full_job
> -a
> delegationToken
> --hiveconf
> mapreduce.job.tags=oozie-75b060aacd7ec48c4ed637855e413280
> Fetching child yarn jobs
> tag id : oozie-75b060aacd7ec48c4ed637855e413280
> Child yarn jobs are found -
> =
> >>> Invoking Beeline command line now >>>
> 0: jdbc:hive2://hiveserver2.bigdata.fr> use my_db;
> 0: jdbc:hive2://hiveserver2.bigdata.fr> set hive.execution.engine=tez;
> 0: jdbc:hive2://hiveserver2.bigdata.fr> set tez.queue.name=tez_queue;
> 0: jdbc:hive2://hiveserver2.bigdata.fr>
> 0: jdbc:hive2://hiveserver2.bigdata.fr> insert overwrite table main_table
> ^M_fd_livcfm
> . . . . . . . . . . . . . . . . . . . . . . .> select
> . . . . . . . . . . . . . . . . . . . . . . .> col.co_cd as co_cd,
> . . . . . . . . . . . . . . . . . . . . . . .> col.line_co_cd as line_co_cd,
> . . . . . . . . . . . . . . . . . . . . . . .>
> unix_timestamp(min(tt.statut_dt)) ^M as statut_dt
> . . . . . . . . . . . . . . . . . . . . . . .> from
> dlk_scf_rn_customer_order_li ^Mne col
> . . . . . . . . . . . . . . . . . . . . . . .> join
> dlk_scf_rn_shipment_handling ^M_utility shu
> . . . . . . . . . . . . . . . . . . . . . . .> on shu.co_cd =col.co_cd
> . . . . . . . . . . . . . . . . . . . . . . .> and shu.line_co_cd =
> col.line_co_ ^Mcd
> . . . . . . . . . . . . . . . . . . . . . . .> join ( select
> scaler_internal_ref ^M, statut_dt,recep_number,state,reason
> . . . . . . . . . . . . . . . . . . . . . . .> from
> dlk_scf_rn_transport_trackin ^Mg where state='LIV' and reason='CFM' ) tt
> . . . . . . . . . . . . . . . . . . . . . . .> on
> concat('CAL',shu.c_waybill_no) ^M =tt.scaler_internal_ref group by
> col.co_cd,col.line_co_cd;
> Heart beat
> Heart beat
> 0: jdbc:hive2://hiveserver2.bigdata.fr>
> 0: jdbc:hive2://hiveserver2.bigdata.fr> insert overwrite table main_table
> ^M_fd_cae
> . . . . . . . . . . . . . . . . . . . . . . .> select
> . . . . . . . . . . . . . . . . . . . . . . .> po_cd as cae, line_po_cd as
> lcae, ^M origin_co_cd, origin_line_co_cd
> . . . . . . . . . . . . . . . . . . . . . . .> from
> dlk_scf_rn_purchase_order_li ^Mne
> . . . . . . . . . . . . . . . . . . . . . . .> where instr(po_cd,"7")=1;
> 0: jdbc:hive2://hiveserver2.bigdata.fr>
> 0: jdbc:hive2://hiveserver2.bigdata.fr> insert overwrite table main_table
> ^M_fd_cai
> . . . . . . . . . . . . . . . . . . . . . . .> select
> . . . . . . . . . . . . . . . . . . . . . . .> po_cd as cai, line_po_cd as
> lcai, ^M origin_co_cd, origin_line_co_cd
> . . . . . . . . . . . . . . . . . . . . . . .> from
> dlk_scf_rn_purchase_order_li ^Mne
> . . . . . . . . . . . . . . . . . . . . . . .> where instr(po_cd,"L")=1;
> 0: jdbc:hive2://hiveserver2.bigdata.fr>
> 0: jdbc:hive2://hiveserver2.bigdata.fr> insert overwrite table
