Hi All, I am using Atlas in my system to store several database metadatas. I observe that it is extremely slow ( > 30mins) when I post > 5000 entities through the endpoint `/v2/entity/bulk` .
I checked there is an option storage.batch-loading in the Janusgraph https://docs.janusgraph.org/operations/bulk-loading/ to speed up creating entities. However, Atlas Server failed to start when I added the settings atlas.graph.storage.batch-loading=true in the atlas-application.properties and reported the following error --------------------- java.lang.IllegalArgumentException: Undefined type used in query: __type.AtlasGlossaryTermRelationshipStatus.defaultValue at org.janusgraph.graphdb.query.QueryUtil.getType(QueryUtil.java:89) at org.janusgraph.graphdb.query.vertex.BasicVertexCentricQueryBuilder.constructQueryWithoutProfile(BasicVertexCentricQueryBuilder.java:514) at org.janusgraph.graphdb.query.vertex.BasicVertexCentricQueryBuilder.constructQuery(BasicVertexCentricQueryBuilder.java:447) at org.janusgraph.graphdb.query.vertex.VertexCentricQueryBuilder.execute(VertexCentricQueryBuilder.java:73) at org.janusgraph.graphdb.query.vertex.VertexCentricQueryBuilder.properties(VertexCentricQueryBuilder.java:105) at org.janusgraph.graphdb.vertices.AbstractVertex.properties(AbstractVertex.java:198) at org.apache.tinkerpop.gremlin.structure.Vertex.property(Vertex.java:70) at org.apache.tinkerpop.gremlin.structure.Vertex.property(Vertex.java:38) at org.apache.atlas.repository.graphdb.janus.AtlasJanusElement.getProperty(AtlasJanusElement.java:66) at org.apache.atlas.repository.store.graph.v2.AtlasGraphUtilsV2.getProperty(AtlasGraphUtilsV2.java:283) at org.apache.atlas.repository.store.graph.v2.AtlasGraphUtilsV2.getProperty(AtlasGraphUtilsV2.java:271) at org.apache.atlas.repository.store.graph.v2.AtlasEnumDefStoreV2.toEnumDef(AtlasEnumDefStoreV2.java:329) at org.apache.atlas.repository.store.graph.v2.AtlasEnumDefStoreV2.toEnumDef(AtlasEnumDefStoreV2.java:304) at org.apache.atlas.repository.store.graph.v2.AtlasEnumDefStoreV2.getAll(AtlasEnumDefStoreV2.java:104) at org.apache.atlas.repository.store.graph.AtlasTypeDefGraphStore.init(AtlasTypeDefGraphStore.java:102) at org.apache.atlas.repository.store.graph.v2.AtlasTypeDefGraphStoreV2.init(AtlasTypeDefGraphStoreV2.java:118) at org.apache.atlas.repository.store.graph.v2.AtlasTypeDefGraphStoreV2$$FastClassBySpringCGLIB$$4680fa9.invoke(<generated>) at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:793) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) at org.apache.atlas.GraphTransactionInterceptor.invoke(GraphTransactionInterceptor.java:111) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:708) at org.apache.atlas.repository.store.graph.v2.AtlasTypeDefGraphStoreV2$$EnhancerBySpringCGLIB$$f2d1c5cd.init(<generated>) at org.apache.atlas.repository.store.bootstrap.AtlasTypeDefStoreInitializer.startInternal(AtlasTypeDefStoreInitializer.java:382) at org.apache.atlas.repository.store.bootstrap.AtlasTypeDefStoreInitializer.init(AtlasTypeDefStoreInitializer.java:119) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:389) at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:333) --------------------- I need the help to know whether we have any other idea to boost bulk importing entities in the endpoint `/v2/entity/bulk` . Thank you, Best Regards, Fengkuangtian Zhu
