[jira] [Commented] (ATLAS-2129) Atlas shutdown during progress of bulk import throws ConcurrentModificationException

2017-09-15 Thread Madhan Neethiraj (JIRA)

[ 
https://issues.apache.org/jira/browse/ATLAS-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16168307#comment-16168307
 ] 

Madhan Neethiraj commented on ATLAS-2129:
-

+1 for the unit test fix.

Committed to following branches:
 - master: http://git-wip-us.apache.org/repos/asf/atlas/commit/45b84fa2
 - branch-0.8: http://git-wip-us.apache.org/repos/asf/atlas/commit/4d5e69a8

> Atlas shutdown during progress of bulk import throws 
> ConcurrentModificationException
> 
>
> Key: ATLAS-2129
> URL: https://issues.apache.org/jira/browse/ATLAS-2129
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: 0.9-incubating
>Reporter: Sharmadha Sainath
>Assignee: Ashutosh Mestry
>Priority: Blocker
> Fix For: 0.9-incubating, 0.8.2
>
> Attachments: ATLAS-2129-bulkImport-Refactor.patch, 
> ATLAS-2129-bulkImport-UnitTest-Fix.patch
>
>
> 1. Exported an hive_db containing 300 hive_table entities from cluster1 into 
> a zip file .
> 2. Tried to import into cluster2.
> 3. When the import was in progress ( at 34%) , stopped Atlas.
> 4. Following exception was seen in application logs of cluster2 :
> {code}
> 2017-09-11 10:17:09,192 INFO  - [pool-2-thread-9 - 
> 83fe24a2-ff2b-4add-a94e-a54b09090912:] ~ bulkImport(): progress: 34% (of 301) 
> - 
> entity:last-imported:hive_table:[102]:(2d629a8e-5c94-40e8-b83f-8a9c91c6de8d) 
> (AtlasEntityStoreV1:238)
> 2017-09-11 10:17:09,340 ERROR - [SIGTERM handler:] ~ Could not commit 
> transaction [1] due to exception (StandardTitanGraph:731)
> java.util.ConcurrentModificationException
>   at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901)
>   at java.util.ArrayList$Itr.next(ArrayList.java:851)
>   at 
> atlas.shaded.titan.guava.common.collect.Iterators$7.computeNext(Iterators.java:701)
>   at 
> atlas.shaded.titan.guava.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
>   at 
> atlas.shaded.titan.guava.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
>   at 
> com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.prepareCommit(StandardTitanGraph.java:473)
>   at 
> com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.commit(StandardTitanGraph.java:654)
>   at 
> com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx.commit(StandardTitanTx.java:1337)
>   at 
> com.thinkaurelius.titan.graphdb.blueprints.TitanBlueprintsGraph.shutdown(TitanBlueprintsGraph.java:120)
>   at 
> com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.shutdownInternal(StandardTitanGraph.java:171)
>   at 
> com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.access$700(StandardTitanGraph.java:75)
>   at 
> com.thinkaurelius.titan.graphdb.database.StandardTitanGraph$ShutdownThread.start(StandardTitanGraph.java:756)
>   at 
> java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:102)
>   at 
> java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:46)
>   at java.lang.Shutdown.runHooks(Shutdown.java:123)
>   at java.lang.Shutdown.sequence(Shutdown.java:167)
>   at java.lang.Shutdown.exit(Shutdown.java:212)
>   at java.lang.Terminator$1.handle(Terminator.java:52)
>   at sun.misc.Signal$1.run(Signal.java:212)
>   at java.lang.Thread.run(Thread.java:748)
> {code}
> Other operations that are called during Atlas shut down such as Shutdown hook 
> , Stopping KafkaNotification service , NotificationHookConsumer , 
> HBaseBasedAuditRepository are not called.
> 5.After that , restarted Atlas.  Atlas functioned properly.
> 6.Resumed import with import option , startGuid=  %. Atlas was stopped when Import was going on at 34% >
> 7. Import completed successfully.
> 8.Post import, only entities from 33% to 100% were imported . Initial 32% of 
> the entities were not imported.
> 9.Fired an import command again without any interruption and without 
> specifying the startGUID. All 300 hive_table entities and 1 hive_db were 
> imported successfully.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (ATLAS-2129) Atlas shutdown during progress of bulk import throws ConcurrentModificationException

2017-09-12 Thread Ashutosh Mestry (JIRA)

[ 
https://issues.apache.org/jira/browse/ATLAS-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16163480#comment-16163480
 ] 

Ashutosh Mestry commented on ATLAS-2129:


[~ssainath] I don't think this should be a blocker. At best it can be marked as 
_Major_.

> Atlas shutdown during progress of bulk import throws 
> ConcurrentModificationException
> 
>
> Key: ATLAS-2129
> URL: https://issues.apache.org/jira/browse/ATLAS-2129
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: 0.9-incubating
>Reporter: Sharmadha Sainath
>Assignee: Ashutosh Mestry
>Priority: Blocker
> Attachments: ATLAS-2129-bulkImport-nested-txn.patch
>
>
> 1. Exported an hive_db containing 300 hive_table entities from cluster1 into 
> a zip file .
> 2. Tried to import into cluster2.
> 3. When the import was in progress ( at 34%) , stopped Atlas.
> 4. Following exception was seen in application logs of cluster2 :
> {code}
> 2017-09-11 10:17:09,192 INFO  - [pool-2-thread-9 - 
> 83fe24a2-ff2b-4add-a94e-a54b09090912:] ~ bulkImport(): progress: 34% (of 301) 
> - 
> entity:last-imported:hive_table:[102]:(2d629a8e-5c94-40e8-b83f-8a9c91c6de8d) 
> (AtlasEntityStoreV1:238)
> 2017-09-11 10:17:09,340 ERROR - [SIGTERM handler:] ~ Could not commit 
> transaction [1] due to exception (StandardTitanGraph:731)
> java.util.ConcurrentModificationException
>   at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901)
>   at java.util.ArrayList$Itr.next(ArrayList.java:851)
>   at 
> atlas.shaded.titan.guava.common.collect.Iterators$7.computeNext(Iterators.java:701)
>   at 
> atlas.shaded.titan.guava.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
>   at 
> atlas.shaded.titan.guava.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
>   at 
> com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.prepareCommit(StandardTitanGraph.java:473)
>   at 
> com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.commit(StandardTitanGraph.java:654)
>   at 
> com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx.commit(StandardTitanTx.java:1337)
>   at 
> com.thinkaurelius.titan.graphdb.blueprints.TitanBlueprintsGraph.shutdown(TitanBlueprintsGraph.java:120)
>   at 
> com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.shutdownInternal(StandardTitanGraph.java:171)
>   at 
> com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.access$700(StandardTitanGraph.java:75)
>   at 
> com.thinkaurelius.titan.graphdb.database.StandardTitanGraph$ShutdownThread.start(StandardTitanGraph.java:756)
>   at 
> java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:102)
>   at 
> java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:46)
>   at java.lang.Shutdown.runHooks(Shutdown.java:123)
>   at java.lang.Shutdown.sequence(Shutdown.java:167)
>   at java.lang.Shutdown.exit(Shutdown.java:212)
>   at java.lang.Terminator$1.handle(Terminator.java:52)
>   at sun.misc.Signal$1.run(Signal.java:212)
>   at java.lang.Thread.run(Thread.java:748)
> {code}
> Other operations that are called during Atlas shut down such as Shutdown hook 
> , Stopping KafkaNotification service , NotificationHookConsumer , 
> HBaseBasedAuditRepository are not called.
> 5.After that , restarted Atlas.  Atlas functioned properly.
> 6.Resumed import with import option , startGuid=  %. Atlas was stopped when Import was going on at 34% >
> 7. Import completed successfully.
> 8.Post import, only entities from 33% to 100% were imported . Initial 32% of 
> the entities were not imported.
> 9.Fired an import command again without any interruption and without 
> specifying the startGUID. All 300 hive_table entities and 1 hive_db were 
> imported successfully.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)