[jira] [Commented] (ATLAS-2129) Atlas shutdown during progress of bulk import throws ConcurrentModificationException
[ https://issues.apache.org/jira/browse/ATLAS-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16168307#comment-16168307 ] Madhan Neethiraj commented on ATLAS-2129: - +1 for the unit test fix. Committed to following branches: - master: http://git-wip-us.apache.org/repos/asf/atlas/commit/45b84fa2 - branch-0.8: http://git-wip-us.apache.org/repos/asf/atlas/commit/4d5e69a8 > Atlas shutdown during progress of bulk import throws > ConcurrentModificationException > > > Key: ATLAS-2129 > URL: https://issues.apache.org/jira/browse/ATLAS-2129 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 0.9-incubating >Reporter: Sharmadha Sainath >Assignee: Ashutosh Mestry >Priority: Blocker > Fix For: 0.9-incubating, 0.8.2 > > Attachments: ATLAS-2129-bulkImport-Refactor.patch, > ATLAS-2129-bulkImport-UnitTest-Fix.patch > > > 1. Exported an hive_db containing 300 hive_table entities from cluster1 into > a zip file . > 2. Tried to import into cluster2. > 3. When the import was in progress ( at 34%) , stopped Atlas. > 4. Following exception was seen in application logs of cluster2 : > {code} > 2017-09-11 10:17:09,192 INFO - [pool-2-thread-9 - > 83fe24a2-ff2b-4add-a94e-a54b09090912:] ~ bulkImport(): progress: 34% (of 301) > - > entity:last-imported:hive_table:[102]:(2d629a8e-5c94-40e8-b83f-8a9c91c6de8d) > (AtlasEntityStoreV1:238) > 2017-09-11 10:17:09,340 ERROR - [SIGTERM handler:] ~ Could not commit > transaction [1] due to exception (StandardTitanGraph:731) > java.util.ConcurrentModificationException > at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901) > at java.util.ArrayList$Itr.next(ArrayList.java:851) > at > atlas.shaded.titan.guava.common.collect.Iterators$7.computeNext(Iterators.java:701) > at > atlas.shaded.titan.guava.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) > at > atlas.shaded.titan.guava.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) > at > com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.prepareCommit(StandardTitanGraph.java:473) > at > com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.commit(StandardTitanGraph.java:654) > at > com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx.commit(StandardTitanTx.java:1337) > at > com.thinkaurelius.titan.graphdb.blueprints.TitanBlueprintsGraph.shutdown(TitanBlueprintsGraph.java:120) > at > com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.shutdownInternal(StandardTitanGraph.java:171) > at > com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.access$700(StandardTitanGraph.java:75) > at > com.thinkaurelius.titan.graphdb.database.StandardTitanGraph$ShutdownThread.start(StandardTitanGraph.java:756) > at > java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:102) > at > java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:46) > at java.lang.Shutdown.runHooks(Shutdown.java:123) > at java.lang.Shutdown.sequence(Shutdown.java:167) > at java.lang.Shutdown.exit(Shutdown.java:212) > at java.lang.Terminator$1.handle(Terminator.java:52) > at sun.misc.Signal$1.run(Signal.java:212) > at java.lang.Thread.run(Thread.java:748) > {code} > Other operations that are called during Atlas shut down such as Shutdown hook > , Stopping KafkaNotification service , NotificationHookConsumer , > HBaseBasedAuditRepository are not called. > 5.After that , restarted Atlas. Atlas functioned properly. > 6.Resumed import with import option , startGuid= %. Atlas was stopped when Import was going on at 34% > > 7. Import completed successfully. > 8.Post import, only entities from 33% to 100% were imported . Initial 32% of > the entities were not imported. > 9.Fired an import command again without any interruption and without > specifying the startGUID. All 300 hive_table entities and 1 hive_db were > imported successfully. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (ATLAS-2129) Atlas shutdown during progress of bulk import throws ConcurrentModificationException
[ https://issues.apache.org/jira/browse/ATLAS-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16163480#comment-16163480 ] Ashutosh Mestry commented on ATLAS-2129: [~ssainath] I don't think this should be a blocker. At best it can be marked as _Major_. > Atlas shutdown during progress of bulk import throws > ConcurrentModificationException > > > Key: ATLAS-2129 > URL: https://issues.apache.org/jira/browse/ATLAS-2129 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 0.9-incubating >Reporter: Sharmadha Sainath >Assignee: Ashutosh Mestry >Priority: Blocker > Attachments: ATLAS-2129-bulkImport-nested-txn.patch > > > 1. Exported an hive_db containing 300 hive_table entities from cluster1 into > a zip file . > 2. Tried to import into cluster2. > 3. When the import was in progress ( at 34%) , stopped Atlas. > 4. Following exception was seen in application logs of cluster2 : > {code} > 2017-09-11 10:17:09,192 INFO - [pool-2-thread-9 - > 83fe24a2-ff2b-4add-a94e-a54b09090912:] ~ bulkImport(): progress: 34% (of 301) > - > entity:last-imported:hive_table:[102]:(2d629a8e-5c94-40e8-b83f-8a9c91c6de8d) > (AtlasEntityStoreV1:238) > 2017-09-11 10:17:09,340 ERROR - [SIGTERM handler:] ~ Could not commit > transaction [1] due to exception (StandardTitanGraph:731) > java.util.ConcurrentModificationException > at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901) > at java.util.ArrayList$Itr.next(ArrayList.java:851) > at > atlas.shaded.titan.guava.common.collect.Iterators$7.computeNext(Iterators.java:701) > at > atlas.shaded.titan.guava.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) > at > atlas.shaded.titan.guava.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) > at > com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.prepareCommit(StandardTitanGraph.java:473) > at > com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.commit(StandardTitanGraph.java:654) > at > com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx.commit(StandardTitanTx.java:1337) > at > com.thinkaurelius.titan.graphdb.blueprints.TitanBlueprintsGraph.shutdown(TitanBlueprintsGraph.java:120) > at > com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.shutdownInternal(StandardTitanGraph.java:171) > at > com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.access$700(StandardTitanGraph.java:75) > at > com.thinkaurelius.titan.graphdb.database.StandardTitanGraph$ShutdownThread.start(StandardTitanGraph.java:756) > at > java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:102) > at > java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:46) > at java.lang.Shutdown.runHooks(Shutdown.java:123) > at java.lang.Shutdown.sequence(Shutdown.java:167) > at java.lang.Shutdown.exit(Shutdown.java:212) > at java.lang.Terminator$1.handle(Terminator.java:52) > at sun.misc.Signal$1.run(Signal.java:212) > at java.lang.Thread.run(Thread.java:748) > {code} > Other operations that are called during Atlas shut down such as Shutdown hook > , Stopping KafkaNotification service , NotificationHookConsumer , > HBaseBasedAuditRepository are not called. > 5.After that , restarted Atlas. Atlas functioned properly. > 6.Resumed import with import option , startGuid= %. Atlas was stopped when Import was going on at 34% > > 7. Import completed successfully. > 8.Post import, only entities from 33% to 100% were imported . Initial 32% of > the entities were not imported. > 9.Fired an import command again without any interruption and without > specifying the startGUID. All 300 hive_table entities and 1 hive_db were > imported successfully. -- This message was sent by Atlassian JIRA (v6.4.14#64029)