[
https://issues.apache.org/jira/browse/ATLAS-3114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16811499#comment-16811499
]
Ayush Nigam commented on ATLAS-3114:
------------------------------------
Hi [~madhan.neethiraj] ,Thanks for looking into the issue. We are persisting
entities top-down,so first top entities we will create,the bottom ones and then
attach bottom entities to the top one later,and update the top one. So
rdbms_source and rdbms_db were successfully created and we are getting issues
only while we are trying to attach referenced entities..i.e. columns to table
and then trying to update table.So for now we can focus on table and columns
only.
Our assumption is that bulk upserts sometimes fail silently in concurrent
requests,we never faced this issue with single request although that too was
concurrent.
You can see the error in Error.txt at line 127 as :
AtlasStjava.util.concurrent.ExecutionException:
org.apache.atlas.AtlasServiceException: Metadata service API
com.intuit.idf.dataregistry.atlas.AtlasClientV2$API_V2@30b914f1 failed with
status 404 (Not Found) Response Body
(\{"errorCode":"ATLAS-404-00-00A","errorMessage":"Referenced entity
AtlasObjectId{guid='null', typeName='rdbms_column',
uniqueAttributes={qualifiedName:pool-1-thread-3:Table5:Column81}} is not
found"})
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at
com.intuit.idf.dataportal.alationbridge.AtlasClientV2Test.main(AtlasClientV2Test.java:69)
Caused by: org.apache.atlas.AtlasServiceException: Metadata service API
com.intuit.idf.dataregistry.atlas.AtlasClientV2$API_V2@30b914f1 failed with
status 404 (Not Found) Response Body
(\{"errorCode":"ATLAS-404-00-00A","errorMessage":"Referenced entity
AtlasObjectId{guid='null', typeName='rdbms_column',
uniqueAttributes={qualifiedName:pool-1-thread-3:Table5:Column81}} is not
found"})
at
org.apache.atlas.AtlasBaseClient.callAPIWithResource(AtlasBaseClient.java:395)
at
org.apache.atlas.AtlasBaseClient.callAPIWithResource(AtlasBaseClient.java:323)
at org.apache.atlas.AtlasBaseClient.callAPI(AtlasBaseClient.java:211)
at
com.intuit.idf.dataregistry.atlas.AtlasClientV2.createEntity(AtlasClientV2.java:547)
at
com.intuit.idf.dataportal.alationbridge.AtlasClientV2Test.lambda$0(AtlasClientV2Test.java:60)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
This is just the part of the code we have attached where we are facing
problems..top entities rdbms_source,rbdms_db are already created successfully
for us.
1) Explained below
2) We have attached just part of the code where we are facing issues,also is
table a mandatory attribute by just mentioning it in a relationship name?
Although your point is correct but should the code fail because of the wrong
name of endDef..although type is correct?
3) Does mentioning name of an endDef in relationship makes an attribute
mandatory? I have not faced this issue before,hence asking?(Same as above)
4) In line 59 of the code you can see we are setting mandatory attribute
'columns'
5) Yes,but we are following a top down approach rather than a bottom up
approach,so first we create a table with mandatory attribute columns as an
empty list then we do bulk create of columns and attach it to to table and
update the table.
> Issue with concurrent bulk inserts for entities
> -----------------------------------------------
>
> Key: ATLAS-3114
> URL: https://issues.apache.org/jira/browse/ATLAS-3114
> Project: Atlas
> Issue Type: Bug
> Reporter: Ayush Nigam
> Assignee: chaitali borole
> Priority: Major
> Attachments: AtlasClientV2Test.java, Error.txt, model.json
>
>
> We have a model with tables having attribute 'columns' in which we are
> attaching list of object ids for all columns once these are created. We are
> using clientV2 java APIs.
> We are doing bulk operation for columns and parallelizing the tables.
> Sometimes the issue is that bulk creation for columns is successful,i.e.
> atlas don't throw any exception but we get some columns as created,some as
> updated,whereas as none of the columns existed before.Even it misses out some
> entities while creating.Some are created and some are just silently missed
> without throwing an exception.
> So to sum up issue is there for concurrent bulk create/update calls.It works
> for concurrent single entity create/update calls.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
