Re: Review Request 65911: [ATLAS-2480]: Regression, Tag Propagation : Deleting the table with lineage and tag

[Madhan Neethiraj]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65911/#review198681
---


Fix it, then Ship it!





repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
Lines 338 (patched)


there is another deleteTypeVertex() method, with additional parameter 
'boolean force'. Shouldn't removeTagPropagation() be called from there as well? 
Please review.


- Madhan Neethiraj


On March 6, 2018, 3:34 a.m., Sarath Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65911/
> ---
> 
> (Updated March 6, 2018, 3:34 a.m.)
> 
> 
> Review request for atlas, Apoorv Naik, Ashutosh Mestry, and Madhan Neethiraj.
> 
> 
> Bugs: ATLAS-2480
> https://issues.apache.org/jira/browse/ATLAS-2480
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> When deleting the table with lineage and tag, propagated edges also needs to 
> be removed and impacted entity vertices needs to be updated.
> 
> 
> Diffs
> -
> 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
> 9d56fa96 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
>  2b62a687 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityGraphMapper.java
>  0ba06b94 
> 
> 
> Diff: https://reviews.apache.org/r/65911/diff/2/
> 
> 
> Testing
> ---
> 
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/139/console
> 
> 
> Thanks,
> 
> Sarath Subramanian
> 
>

[jira] [Updated] (ATLAS-2481) remove vertex properties traitNames and propagatedTraitNames from entity vertices

[Sarath Subramanian (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-2481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-2481:
--
Description: 
We currently store classification and propagated classification names in the 
entity vertex as properties (traitNames and propagatedTraitNames). This needs 
to be updated to fetch these values from outgoing edges from entity vertex.
 # Classification edge and propagated classification edge label needs to be 
updated to a static name - 'classifiedAs' and additional properties needs to be 
added to the edge:
 ## _name_ - name of the classification 
 ## _isPropagated_ - whether this tag is associated or propagated one
 # Modify basic and advanced (DSL) search to handle searching on edges instead 
of vertex property for tag based searches.

 

  was:
We currently store classification and propagated classification names in the 
entity vertex as properties (traitNames and propagatedTraitNames). This needs 
to be updated to fetch these values from outgoing edges from entity vertex.
 # Classification edge and propagated classification edge label needs to be 
updated to give a static name - 'classifiedAs' and additional properties needs 
to be added to the edge:
 ## name - name of the classification 
 ## isPropagated - whether this tag is associated or propagated one
 # Modify basic and advanced (DSL) search to handle this change - search edges 
instead of vertex property for tag based searches.

 


> remove vertex properties traitNames and propagatedTraitNames from entity 
> vertices
> -
>
> Key: ATLAS-2481
> URL: https://issues.apache.org/jira/browse/ATLAS-2481
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 1.0.0
>Reporter: Sarath Subramanian
>Assignee: Sarath Subramanian
>Priority: Major
> Fix For: 1.0.0
>
>
> We currently store classification and propagated classification names in the 
> entity vertex as properties (traitNames and propagatedTraitNames). This needs 
> to be updated to fetch these values from outgoing edges from entity vertex.
>  # Classification edge and propagated classification edge label needs to be 
> updated to a static name - 'classifiedAs' and additional properties needs to 
> be added to the edge:
>  ## _name_ - name of the classification 
>  ## _isPropagated_ - whether this tag is associated or propagated one
>  # Modify basic and advanced (DSL) search to handle searching on edges 
> instead of vertex property for tag based searches.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (ATLAS-2481) remove vertex properties traitNames and propagatedTraitNames from entity vertices

[Sarath Subramanian (JIRA)]

Sarath Subramanian created ATLAS-2481:
-

 Summary: remove vertex properties traitNames and 
propagatedTraitNames from entity vertices
 Key: ATLAS-2481
 URL: https://issues.apache.org/jira/browse/ATLAS-2481
 Project: Atlas
  Issue Type: Improvement
  Components:  atlas-core
Affects Versions: 1.0.0
Reporter: Sarath Subramanian
Assignee: Sarath Subramanian
 Fix For: 1.0.0


We currently store classification and propagated classification names in the 
entity vertex as properties (traitNames and propagatedTraitNames). This needs 
to be updated to fetch these values from outgoing edges from entity vertex.
 # Classification edge and propagated classification edge label needs to be 
updated to give a static name - 'classifiedAs' and additional properties needs 
to be added to the edge:
 ## name - name of the classification 
 ## isPropagated - whether this tag is associated or propagated one
 # Modify basic and advanced (DSL) search to handle this change - search edges 
instead of vertex property for tag based searches.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 65911: [ATLAS-2480]: Regression, Tag Propagation : Deleting the table with lineage and tag

[Sarath Subramanian]

> On March 5, 2018, 7:40 p.m., Madhan Neethiraj wrote:
> > repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
> > Lines 335 (patched)
> > 
> >
> > removeTagPropagation() expects a classificationVertex as argument; is 
> > the instanceVertex here a classificationVertex?

Yes, in this context - the vertex to delete is a classificationVertex. If the 
vertex is of type trait, delete the edge, trait vertex and all propagated edges


- Sarath


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65911/#review198679
---


On March 5, 2018, 7:34 p.m., Sarath Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65911/
> ---
> 
> (Updated March 5, 2018, 7:34 p.m.)
> 
> 
> Review request for atlas, Apoorv Naik, Ashutosh Mestry, and Madhan Neethiraj.
> 
> 
> Bugs: ATLAS-2480
> https://issues.apache.org/jira/browse/ATLAS-2480
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> When deleting the table with lineage and tag, propagated edges also needs to 
> be removed and impacted entity vertices needs to be updated.
> 
> 
> Diffs
> -
> 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
> 9d56fa96 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
>  2b62a687 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityGraphMapper.java
>  0ba06b94 
> 
> 
> Diff: https://reviews.apache.org/r/65911/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sarath Subramanian
> 
>

Re: Review Request 65911: [ATLAS-2480]: Regression, Tag Propagation : Deleting the table with lineage and tag

[Madhan Neethiraj]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65911/#review198679
---




repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
Lines 335 (patched)


removeTagPropagation() expects a classificationVertex as argument; is the 
instanceVertex here a classificationVertex?


- Madhan Neethiraj


On March 6, 2018, 3:34 a.m., Sarath Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65911/
> ---
> 
> (Updated March 6, 2018, 3:34 a.m.)
> 
> 
> Review request for atlas, Apoorv Naik, Ashutosh Mestry, and Madhan Neethiraj.
> 
> 
> Bugs: ATLAS-2480
> https://issues.apache.org/jira/browse/ATLAS-2480
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> When deleting the table with lineage and tag, propagated edges also needs to 
> be removed and impacted entity vertices needs to be updated.
> 
> 
> Diffs
> -
> 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
> 9d56fa96 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
>  2b62a687 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityGraphMapper.java
>  0ba06b94 
> 
> 
> Diff: https://reviews.apache.org/r/65911/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sarath Subramanian
> 
>

Review Request 65911: [ATLAS-2480]: Regression, Tag Propagation : Deleting the table with lineage and tag

[Sarath Subramanian]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65911/
---

Review request for atlas, Apoorv Naik, Ashutosh Mestry, and Madhan Neethiraj.


Bugs: ATLAS-2480
https://issues.apache.org/jira/browse/ATLAS-2480


Repository: atlas


Description
---

When deleting the table with lineage and tag, propagated edges also needs to be 
removed and impacted entity vertices needs to be updated.


Diffs
-

  repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
9d56fa96 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
 2b62a687 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityGraphMapper.java
 0ba06b94 


Diff: https://reviews.apache.org/r/65911/diff/1/


Testing
---


Thanks,

Sarath Subramanian

[jira] [Created] (ATLAS-2480) Regression,Tag Propagation : Deleting the table with lineage and tag

[Sarath Subramanian (JIRA)]

Sarath Subramanian created ATLAS-2480:
-

 Summary: Regression,Tag Propagation : Deleting the table with 
lineage and tag
 Key: ATLAS-2480
 URL: https://issues.apache.org/jira/browse/ATLAS-2480
 Project: Atlas
  Issue Type: Bug
  Components:  atlas-core
Affects Versions: 1.0.0
Reporter: Sarath Subramanian
Assignee: Sarath Subramanian
 Fix For: 1.0.0


1. Created an Hive table t1 and added tag tag1 to it.
2. Created a CTAS table t1_ctas from t1. 
3.Now tag1 is associated to both t1 and t1_ctas.
4.Now dropped the table t1 . 
Following exception is seen in application logs :
{code}
java.lang.IllegalArgumentException: Invalid edge label propagated:tag1: 
expected 2 or 3 label components but found 1
{code}

Attached the complete exception stack trace.

In following cases , the exception is not seen and works correctly:
1. Dropping a table associated to tag
2.Dropping the source table which has relationship with other table (CTAS)
3.Dropping t1_ctas table in the above scenario

Only when the table is source of relationship and has tags , this exception is 
seen.

The issue is not seen in BETA-1 versions.

CC : [~ssubramanian]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 65790: Data Migration Utility: Export

[Madhan Neethiraj]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65790/#review198672
---


Fix it, then Ship it!





tools/atlas-migration-utility/pom.xml
Lines 59 (patched)


'atlas-repository' is included twice.



tools/atlas-migration-utility/pom.xml
Lines 67 (patched)


Assembly is handled for other modules from distro/pom.xml. Consider moving 
this (and tools/atlas-migration-utility/src/assembly/bin.xml) into distro 
module.


- Madhan Neethiraj


On March 5, 2018, 5:39 a.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65790/
> ---
> 
> (Updated March 5, 2018, 5:39 a.m.)
> 
> 
> Review request for atlas, Apoorv Naik, Madhan Neethiraj, and Ruchi Solani.
> 
> 
> Bugs: ATLAS-2461
> https://issues.apache.org/jira/browse/ATLAS-2461
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Background**
> The data migration utility allows for exporting data from an Atlas instance 
> without the server running. This is important as it will prevent Atlas from 
> processing any requests.
> 
> **Approach**
> The migration is a new utility that will perform export of data using a 
> pre-defined export request file.
> 
> The approach used by this application:
> - Create an application context (_migrationContext.xml_). This context, 
> prevents instantiation of number of classes, most notably _webapp_, 
> _notifications_, _listeners_.
> - Create _ImportService_ using Spring's framework classes.
> 
> Here are the pieces:
> - _MigrationApp_: Contains _main_. It is the entry point for the app.
> - _Exporter_: Contains plumbing needed to use the new _migrationContext.xml_ 
> and create _ApplicationContext_ for use.
> - Dengenerate classes _EmptyNotification_, _EmptyNotificationChangeListener_. 
> These are necessary to launch the application in a way that no notifications 
> are sent out.
> - _atlas_migration.py_ Creates environment for the app to execute. It 
> launches the application.
> - _migration-export-request.json_ This can be modified for environments that 
> do not have latest improvements to _Export_.
> - _README_: Instructions on usage.
> 
> **Build**
> The project maven's assembly building plugin. The build will create a ZIP 
> file with the necessary files.
> 
> _mvn install package_ generates a ZIP file in _target_ directory.
> 
> **Usage (from README)**
> - Use Ambari to turn shutdown Atlas.
> - Unzip contents of the ZIP to a directory on the server using: 'unzip 
> atlas-migration-kit.zip'
> - cd atlas-migration-kit
> - Run 'python atlas_migration.py'. Use 'python atlas_migration.py 
> ' to export to a specific file.
> - To watch the progress: 'tail -f /var/log/atlas/application.log'.
> 
> 
> Diffs
> -
> 
>   pom.xml 7db1be78 
>   tools/atlas-migration-utility/pom.xml PRE-CREATION 
>   tools/atlas-migration-utility/src/assembly/bin.xml PRE-CREATION 
>   
> tools/atlas-migration-utility/src/main/java/org/apache/atlas/migration/Exporter.java
>  PRE-CREATION 
>   
> tools/atlas-migration-utility/src/main/java/org/apache/atlas/migration/NoOpNotification.java
>  PRE-CREATION 
>   
> tools/atlas-migration-utility/src/main/java/org/apache/atlas/migration/NoOpNotificationChangeListener.java
>  PRE-CREATION 
>   tools/atlas-migration-utility/src/main/resources/README PRE-CREATION 
>   tools/atlas-migration-utility/src/main/resources/atlas_migration.py 
> PRE-CREATION 
>   
> tools/atlas-migration-utility/src/main/resources/migration-export-request.json
>  PRE-CREATION 
>   tools/atlas-migration-utility/src/main/resources/migrationContext.xml 
> PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/65790/diff/5/
> 
> 
> Testing
> ---
> 
> **Unit tests**
> None.
> 
> **Functional tests**
> Exports from existing clusters.
> 
> 
> File Attachments
> 
> 
> Migration Utility
>   
> https://reviews.apache.org/media/uploaded/files/2018/03/05/4d42eda3-cba8-441d-8505-12271cffc569__atlas-migration-kit-0.8.3-SNAPSHOT-bin.zip
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>

Re: Version 1.0 features and v1 API compatibility

[Madhan Neethiraj]

Ismaël,

Atlas 1.0 server will continue to support API calls from applications that use 
Atlas 0.8.x libraries. However, updating an application to use newer version of 
Atlas libraries might require some code changes - mainly due to the following:
 1) ATLAS-2251: removal of earlier version of type-system implementation.
 List of renamed/relocated classes can be found in this JIRA.
 2) ATLAS-2265: upgrade to use newer version of Jackson libraries
 References to classes in earlier Jackson library have been replaced with 
corresponding newer version classes.

If you run into any specific issue, can you please include details?

Details of REST APIs in 1.0.0-snapshot version can be found at 
http://atlas.apache.org/api/v2/index.html.  The new APIs introduced in 1.0.0 
are about relationships (RelationshipREST, AtlasEntity.relationshipAttributes). 
The documentation doesn't include the marker to identity the version the 
API/attribute was introduced. We will have this information added to API 
documentation. In the meantime, if you have any specific questions, please 
reach out to the dev list.

Hope this helps.

Madhan

On 3/5/18, 1:28 AM, "Ismaël Mejía"  wrote:

Hello,

We worked with some colleagues in an integration with Atlas 0.7/0.8
almost one year ago, I was testing it with the latest releases and it
works flawlessly with the latest release 0.8.2. However with version
1.0.0-alpha and I found that even if the version seems to still
support the first version of the API (v1) all the package are in
different places, is this intended ? Because if this is the case well,
this breaks a bit the contract of backward compatibility.

And a second question, is there a list of features included in version
1.0 somewhere, or more concretely on the new v2 API? I have checked a
bit at the endpoints of the REST API but I was wondering if there is a
more user friendly doc

Thanks,
Ismaël

[jira] [Created] (ATLAS-2479) [Docs] Update Atlas user docs and REST API specs

[Apoorv Naik (JIRA)]

Apoorv Naik created ATLAS-2479:
--

 Summary: [Docs] Update Atlas user docs and REST API specs 
 Key: ATLAS-2479
 URL: https://issues.apache.org/jira/browse/ATLAS-2479
 Project: Atlas
  Issue Type: Bug
Affects Versions: 1.0.0-alpha
Reporter: Apoorv Naik
Assignee: Apoorv Naik
 Fix For: 1.0.0






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: EntityREST Bulk & relationships

[Madhan Neethiraj]

Nigel,

>  If the composition relationship is modelled using explicit attributes, I 
> believe I could  create my entire definition containing multiple of all the 
> above in a single EntityREST Bulk API request, using negative GUIDs as 
> placeholders for the entities that are being referenced & created within the 
> request.

Yes. This is correct.

>  If instead I don't use attributes, but use relationships to define the 
> composition, I cannot do this, since I need to create all the entities 
> initially (which could be done in one request) and then, based on the GUIDs 
> generated, issue  a series of calls to RelationshipREST to define the 
> relationships (relationships cannot be created as part of the bulk entity api 
> call)

Entity update will process the relationship attributes specified in 
"AtlasEntity.relationshipAttributes" and update the relationships accordingly. 
If an entity has an attribute in both " attributes" and 
"relationshipAttributes" (like hive_table.columns), then the values specified 
in "attributes" would be ignored; and the entity will be updated for the value 
specified in "relationshipAttributes". Given this, it is possible to use 
'relationshipAttributes', in the same way as 'attributes'. However, considering 
that relationships are dynamic (i.e. a relationship type can be added at 
runtime) and the entity-update API caller may not know all the relationships of 
an entity-type at compile time, I would think such API integration would only 
deal with attributes directly defined in an entity-type.

Hope this helps.

Madhan




On 3/5/18, 7:42 AM, "Nigel Jones"  wrote:

I'd like to check my understanding of the bulk api & relationships

Let's say I have a database with a table, which contains a few columns. 
(like hive)

If the composition relationship is modelled using explicit attributes, I 
believe I could  create my entire definition containing multiple of all the 
above in a single EntityREST Bulk API request, using negative GUIDs as 
placeholders for the entities that are being referenced & created within the 
request.

If instead I don't use attributes, but use relationships to define the 
composition, I cannot do this, since I need to create all the entities 
initially (which could be done in one request) and then, based on the GUIDs 
generated, issue  a series of calls to RelationshipREST to define the 
relationships (relationships cannot be created as part of the bulk entity api 
call)

is my understanding correct? Have I missed anything?

Thanks
Nigel.

Re: Review Request 65885: ATLAS-2470 - Add JanusGraph Cassandra support to Atlas

[David Radley]

> On March 4, 2018, 11:04 a.m., David Radley wrote:
> > distro/pom.xml
> > Lines 302 (patched)
> > 
> >
> > for consistency should we have  as well?
> 
> Pierre Padovani wrote:
> You cannot just have cassandra without a solr/elasticsearch install 
> unless you are using DSE (the Datastax Enterprise with integrates with solr). 
> Based on cursory reading over on JanusGraph, they do not recommend production 
> deployments of JanusGraph with Cassandra embedded as the performance profiles 
> are not entirely predictable. Again I would address this in the main 
> documentation task.

Yes I was thinking that cassandra-embedded to differentiate between the 
production version - maybe you can add this when the extenral cassandra support 
is put in.


- David


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65885/#review198584
---


On March 2, 2018, 4:51 p.m., Pierre Padovani wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65885/
> ---
> 
> (Updated March 2, 2018, 4:51 p.m.)
> 
> 
> Review request for atlas and David Radley.
> 
> 
> Bugs: ATLAS-2470
> https://issues.apache.org/jira/browse/ATLAS-2470
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This updates pom's to add the needed cassandra jars, and adds a dist profile 
> embedded-cassandra-solr.
> 
> 
> Diffs
> -
> 
>   distro/pom.xml 0103bef6 
>   distro/src/bin/atlas_config.py e6415cf4 
>   distro/src/bin/atlas_start.py 39be6b7c 
>   distro/src/bin/atlas_stop.py 66edd904 
>   distro/src/conf/cassandra.yml PRE-CREATION 
>   distro/src/conf/zookeeper/log4j.properties PRE-CREATION 
>   distro/src/conf/zookeeper/zoo.cfg PRE-CREATION 
>   distro/src/main/assemblies/standalone-package.xml 1881082e 
>   docs/src/site/twiki/InstallationSteps.twiki 6b9f0313 
>   graphdb/janus/pom.xml 143b775f 
> 
> 
> Diff: https://reviews.apache.org/r/65885/diff/1/
> 
> 
> Testing
> ---
> 
> Full build with the new embedded-cassandra-solr, and testing to make sure 
> Atlas comes up and is functional.
> 
> Be aware that we have been running Cassandra backed JanusGraph for months 
> with no issues.
> 
> 
> Thanks,
> 
> Pierre Padovani
> 
>

Re: Review Request 65885: ATLAS-2470 - Add JanusGraph Cassandra support to Atlas

[David Radley]

> On March 4, 2018, 11:07 a.m., David Radley wrote:
> > My review comments are from my initial look at the code; I hope to try 
> > running this patch to verify it works for me
> 
> Pierre Padovani wrote:
> I'll update the patch with the above changes today or tomorrow as I have 
> time.
> 
> Pierre Padovani wrote:
> I've updated the ATLAS-2470 with a new patch that should fix all of these 
> issues.
> 
> Pierre Padovani wrote:
> Hi David, I think I'm missing something. The instructions on the 'Using 
> GIT with Atlas' page are unclear on patch revisions. Do I create another 
> review, or do I somehow update this review?

It is best to use the same review for subsequent patches- so then it is 
possible to compare between versions. I have updated the wiki with this 
information - thanks.


- David


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65885/#review198593
---


On March 2, 2018, 4:51 p.m., Pierre Padovani wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65885/
> ---
> 
> (Updated March 2, 2018, 4:51 p.m.)
> 
> 
> Review request for atlas and David Radley.
> 
> 
> Bugs: ATLAS-2470
> https://issues.apache.org/jira/browse/ATLAS-2470
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This updates pom's to add the needed cassandra jars, and adds a dist profile 
> embedded-cassandra-solr.
> 
> 
> Diffs
> -
> 
>   distro/pom.xml 0103bef6 
>   distro/src/bin/atlas_config.py e6415cf4 
>   distro/src/bin/atlas_start.py 39be6b7c 
>   distro/src/bin/atlas_stop.py 66edd904 
>   distro/src/conf/cassandra.yml PRE-CREATION 
>   distro/src/conf/zookeeper/log4j.properties PRE-CREATION 
>   distro/src/conf/zookeeper/zoo.cfg PRE-CREATION 
>   distro/src/main/assemblies/standalone-package.xml 1881082e 
>   docs/src/site/twiki/InstallationSteps.twiki 6b9f0313 
>   graphdb/janus/pom.xml 143b775f 
> 
> 
> Diff: https://reviews.apache.org/r/65885/diff/1/
> 
> 
> Testing
> ---
> 
> Full build with the new embedded-cassandra-solr, and testing to make sure 
> Atlas comes up and is functional.
> 
> Be aware that we have been running Cassandra backed JanusGraph for months 
> with no issues.
> 
> 
> Thanks,
> 
> Pierre Padovani
> 
>

[jira] [Commented] (ATLAS-2478) Elasticsearch support is broken for JanusGraph

[David Radley (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-2478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16386672#comment-16386672
 ] 

David Radley commented on ATLAS-2478:
-

[~ppadovani] I am not an expert on this part of Atlas. My thought is that most 
atlas customer will not have moved to Janus yet, so will need to perform some 
sort of migration of existing data; I would think that this sort of change 
could be made during the migration. 

The only released version of Atlas is an Alpha release, which is not intended 
for production. I suggest getting this change into master, so migration scripts 
could take account of this as well.  

> Elasticsearch support is broken for JanusGraph
> --
>
> Key: ATLAS-2478
> URL: https://issues.apache.org/jira/browse/ATLAS-2478
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: 1.0.0-alpha
>Reporter: Pierre Padovani
>Assignee: Pierre Padovani
>Priority: Major
> Fix For: 1.0.0
>
>
> With JanusGraph the Elasticsearch support moved to 5.x+. This introduced a 
> change where fields that contained '.' (dots) were not allowed unless either 
> a specific cluster wide setting was enabled AND the mapping was formatted 
> such that each of the fields that contained a '.' could be considered part of 
> an object.
> Example:
> {code:java}
> foo.x
> foo.y
> foo.z{code}
>  Elasticsearch looks at these fields as if they are truly:
> {code:java}
> foo : {
>   x,
>   y,
>   z
> }{code}
> In the file:
> /atlas/common/src/main/java/org/apache/atlas/repository/Constants.java
> {code:java}
> /**
>  * Properties for type store graph.
>  */
> public static final String TYPE_CATEGORY_PROPERTY_KEY = 
> INTERNAL_PROPERTY_KEY_PREFIX + "type.category";
> public static final String VERTEX_TYPE_PROPERTY_KEY = 
> INTERNAL_PROPERTY_KEY_PREFIX + "type";
> public static final String TYPENAME_PROPERTY_KEY = 
> INTERNAL_PROPERTY_KEY_PREFIX + "type.name";
> public static final String TYPEDESCRIPTION_PROPERTY_KEY = 
> INTERNAL_PROPERTY_KEY_PREFIX + "type.description";
> public static final String TYPEVERSION_PROPERTY_KEY = 
> INTERNAL_PROPERTY_KEY_PREFIX + "type.version";
> public static final String TYPEOPTIONS_PROPERTY_KEY = 
> INTERNAL_PROPERTY_KEY_PREFIX + "type.options";
> {code}
> These are the only fields that cause Elasticsearch issue. As you can see a 
> field called 'type' is created, then additional fields type.name, 
> type.description etc. This will cause a mapping conflict exception in 
> Elasticsearch and it will refuse to create the mapping.
>  
> The easy fix is to simply replace the '.' with an '_' (underscore) but this 
> will be a backwards incompatible change for existing customers. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 65902: [ATLAS-2477]: Tag Propagation and Search : Search with tag and type doesn't list all the entities

[Madhan Neethiraj]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65902/#review198643
---


Ship it!




Ship It!

- Madhan Neethiraj


On March 5, 2018, 12:23 p.m., Sarath Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65902/
> ---
> 
> (Updated March 5, 2018, 12:23 p.m.)
> 
> 
> Review request for atlas, Apoorv Naik, Ashutosh Mestry, and Madhan Neethiraj.
> 
> 
> Bugs: ATLAS-2477
> https://issues.apache.org/jira/browse/ATLAS-2477
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1. Created an Hive table "employee" and CTAS table "employee_ctas".
> 2. Associated a tag tag1 to employee.Both employee and employee_ctas are 
> associated to tag1.
> 3. Basic search with classification = tag1 , listed 3 entities - employee , 
> employee_ctas and the process.
> 4. But Search with classification = tag1 , type = hive_table listed only the 
> employee . Expected that it would list the employee_ctas also.
> 5.With Advanced search, both the following queries
> 
> query =tag1
> typename = hive_table query = isa tag1
> 
> lists only the employee entity.
> 
> Hence , search with type  and tag lists only the entity to which the tag is 
> associated and not the propagated associations.
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/discovery/ClassificationSearchProcessor.java
>  4265e098 
>   
> repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java
>  016ad1b9 
>   repository/src/main/java/org/apache/atlas/query/GremlinClause.java 38411037 
>   repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java 
> 9a66636d 
>   
> repository/src/main/java/org/apache/atlas/util/AtlasGremlin2QueryProvider.java
>  65b99cd0 
>   repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java 
> fedb8a7e 
>   
> repository/src/test/java/org/apache/atlas/query/GremlinQueryComposerTest.java 
> 9c8aac5c 
> 
> 
> Diff: https://reviews.apache.org/r/65902/diff/2/
> 
> 
> Testing
> ---
> 
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/138/console
> 
> 
> Thanks,
> 
> Sarath Subramanian
> 
>

Re: Review Request 65870: ATLAS-2469 : Tag propagation from object to child object or derivative asset.

[Madhan Neethiraj]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65870/#review198627
---


Ship it!




Ship It!

- Madhan Neethiraj


On March 5, 2018, 2:16 p.m., pratik pandey wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65870/
> ---
> 
> (Updated March 5, 2018, 2:16 p.m.)
> 
> 
> Review request for atlas, keval bhatt, Madhan Neethiraj, Nixon Rodrigues, and 
> Sarath Subramanian.
> 
> 
> Bugs: ATLAS-2469
> https://issues.apache.org/jira/browse/ATLAS-2469
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> User Story:
> 
> As a data steward i need a scalable way to quickly and efficiently propagate 
> tags for efficient searches and tag based security. Likewise tags for 
> derivative dataset should be inherited from the parent. For example, if an 
> entity is tagged "Secret" then resulting entity created from a CTAS operation 
> should also be tagged "secret" to maintain the classification of the parent. 
> In the case where 2 or more datasets are aggregated the derivative dataset 
> should be a union of all parent tags.
> 
> Terms:
> 
> Child business terms should inherit the tags associated with the parent term.
> the option to propagate tags to child business terms in a hierarchy should be 
> provided
> Ability to update the propagated tags manually via UI or through the API
> Tagging a term should propagate to data assets that are already attached to 
> that business term as well
> Data assets
> 
> For supported components in HDP 2.6,if a derivative asset is created it 
> should inherit the tags and attributes from the original asset.
> the option to propagate tags to child entities should be provided (e.g. if 
> you tag a hdfs folder optionally tag all the files within it)
> Ability to update the propagated tags manually via UI or through the API
> Tagging a parent object should be inherited even after child creation 
> dynamically (unless a flag is set not to do this)
> Derived data assets should have the tags of the original data asset.
> conflict resolution - if the different values for attributes then a UX dialog 
> that prompts user for action needs to be provided. Once resolved, the 
> resolved value will be carry forth to derivative dataset.
> 
> 
> Diffs
> -
> 
>   dashboardv2/gruntfile.js 451a933 
>   dashboardv2/package.json 9237c35 
>   dashboardv2/public/css/scss/override.scss f1e2c6b 
>   dashboardv2/public/js/main.js ce0f4fe 
>   dashboardv2/public/js/templates/detail_page/DetailPageLayoutView_tmpl.html 
> 1578548 
>   dashboardv2/public/js/templates/search/SearchLayoutView_tmpl.html 5d3c3af 
>   dashboardv2/public/js/templates/search/SearchResultLayoutView_tmpl.html 
> 403b064 
>   dashboardv2/public/js/templates/site/SideNavLayoutView_tmpl.html b706609 
>   dashboardv2/public/js/templates/tag/AddTagModalView_tmpl.html a477532 
>   dashboardv2/public/js/templates/tag/AddTimezoneView_tmpl.html PRE-CREATION 
>   dashboardv2/public/js/templates/tag/CreateTagLayoutView_tmpl.html 40827ef 
>   dashboardv2/public/js/templates/tag/TagAttributeDetailLayoutView_tmpl.html 
> d418782 
>   dashboardv2/public/js/templates/tag/TagDetailTableLayoutView_tmpl.html 
> 76f225a 
>   dashboardv2/public/js/templates/tag/TagLayoutView_tmpl.html f786df4 
>   dashboardv2/public/js/utils/CommonViewFunction.js 9b9ffcc 
>   dashboardv2/public/js/utils/Enums.js a7b9a8b 
>   dashboardv2/public/js/utils/Messages.js 44604cf 
>   dashboardv2/public/js/views/detail_page/DetailPageLayoutView.js 802f6a4 
>   dashboardv2/public/js/views/search/SearchResultLayoutView.js 86520c9 
>   dashboardv2/public/js/views/tag/AddTagModalView.js d1c8c67 
>   dashboardv2/public/js/views/tag/AddTimezoneItemView.js PRE-CREATION 
>   dashboardv2/public/js/views/tag/CreateTagLayoutView.js e43a431 
>   dashboardv2/public/js/views/tag/TagAttributeDetailLayoutView.js 3d6b859 
>   dashboardv2/public/js/views/tag/TagDetailTableLayoutView.js c05b48d 
>   dashboardv2/public/js/views/tag/TagLayoutView.js 0edb378 
> 
> 
> Diff: https://reviews.apache.org/r/65870/diff/3/
> 
> 
> Testing
> ---
> 
> Done one round of sanity testing.
> 
> 
> Thanks,
> 
> pratik pandey
> 
>

EntityREST Bulk & relationships

[Nigel Jones]

I'd like to check my understanding of the bulk api & relationships

Let's say I have a database with a table, which contains a few columns. (like 
hive)

If the composition relationship is modelled using explicit attributes, I 
believe I could  create my entire definition containing multiple of all the 
above in a single EntityREST Bulk API request, using negative GUIDs as 
placeholders for the entities that are being referenced & created within the 
request.

If instead I don't use attributes, but use relationships to define the 
composition, I cannot do this, since I need to create all the entities 
initially (which could be done in one request) and then, based on the GUIDs 
generated, issue  a series of calls to RelationshipREST to define the 
relationships (relationships cannot be created as part of the bulk entity api 
call)

is my understanding correct? Have I missed anything?

Thanks
Nigel.

[jira] [Updated] (ATLAS-2469) Tag propagation from object to child object or derivative asset

[pratik pandey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-2469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

pratik pandey updated ATLAS-2469:
-
Attachment: ATLAS-2469.patch

> Tag propagation from object to child object or derivative asset
> ---
>
> Key: ATLAS-2469
> URL: https://issues.apache.org/jira/browse/ATLAS-2469
> Project: Atlas
>  Issue Type: New Feature
>Affects Versions: 1.0.0
>Reporter: pratik pandey
>Assignee: pratik pandey
>Priority: Blocker
> Fix For: 1.0.0
>
> Attachments: ATLAS-2469.patch
>
>
> User Story:
> As a data steward i need a scalable way to quickly and efficiently propagate 
> tags for efficient searches and tag based security. Likewise tags for 
> derivative dataset should be inherited from the parent. For example, if an 
> entity is tagged "Secret" then resulting entity created from a CTAS operation 
> should also be tagged "secret" to maintain the classification of the parent. 
> In the case where 2 or more datasets are aggregated the derivative dataset 
> should be a union of all parent tags.
> Terms:
> * Child business terms should inherit the tags associated with the parent 
> term.
> * the option to propagate tags to child business terms in a hierarchy should 
> be provided
> * Ability to update the propagated tags manually via UI or through the API
> * Tagging a term should propagate to data assets that are already attached to 
> that business term as well
> Data assets
> * For supported components in HDP 2.6,if a derivative asset is created it 
> should inherit the tags and attributes from the original asset.
> * the option to propagate tags to child entities should be provided (e.g. if 
> you tag a hdfs folder optionally tag all the files within it)
> * Ability to update the propagated tags manually via UI or through the API
> * Tagging a parent object should be inherited even after child creation 
> dynamically (unless a flag is set not to do this)
> * Derived data assets should have the tags of the original data asset.
> conflict resolution - if the different values for attributes then a UX dialog 
> that prompts user for action needs to be provided. Once resolved, the 
> resolved value will be carry forth to derivative dataset.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 65870: ATLAS-2469 : Tag propagation from object to child object or derivative asset.

[pratik pandey]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65870/
---

(Updated March 5, 2018, 2:16 p.m.)


Review request for atlas, keval bhatt, Madhan Neethiraj, Nixon Rodrigues, and 
Sarath Subramanian.


Changes
---

All notification text change from Tag to Classfication.


Bugs: ATLAS-2469
https://issues.apache.org/jira/browse/ATLAS-2469


Repository: atlas


Description
---

User Story:

As a data steward i need a scalable way to quickly and efficiently propagate 
tags for efficient searches and tag based security. Likewise tags for 
derivative dataset should be inherited from the parent. For example, if an 
entity is tagged "Secret" then resulting entity created from a CTAS operation 
should also be tagged "secret" to maintain the classification of the parent. In 
the case where 2 or more datasets are aggregated the derivative dataset should 
be a union of all parent tags.

Terms:

Child business terms should inherit the tags associated with the parent term.
the option to propagate tags to child business terms in a hierarchy should be 
provided
Ability to update the propagated tags manually via UI or through the API
Tagging a term should propagate to data assets that are already attached to 
that business term as well
Data assets

For supported components in HDP 2.6,if a derivative asset is created it should 
inherit the tags and attributes from the original asset.
the option to propagate tags to child entities should be provided (e.g. if you 
tag a hdfs folder optionally tag all the files within it)
Ability to update the propagated tags manually via UI or through the API
Tagging a parent object should be inherited even after child creation 
dynamically (unless a flag is set not to do this)
Derived data assets should have the tags of the original data asset.
conflict resolution - if the different values for attributes then a UX dialog 
that prompts user for action needs to be provided. Once resolved, the resolved 
value will be carry forth to derivative dataset.


Diffs (updated)
-

  dashboardv2/gruntfile.js 451a933 
  dashboardv2/package.json 9237c35 
  dashboardv2/public/css/scss/override.scss f1e2c6b 
  dashboardv2/public/js/main.js ce0f4fe 
  dashboardv2/public/js/templates/detail_page/DetailPageLayoutView_tmpl.html 
1578548 
  dashboardv2/public/js/templates/search/SearchLayoutView_tmpl.html 5d3c3af 
  dashboardv2/public/js/templates/search/SearchResultLayoutView_tmpl.html 
403b064 
  dashboardv2/public/js/templates/site/SideNavLayoutView_tmpl.html b706609 
  dashboardv2/public/js/templates/tag/AddTagModalView_tmpl.html a477532 
  dashboardv2/public/js/templates/tag/AddTimezoneView_tmpl.html PRE-CREATION 
  dashboardv2/public/js/templates/tag/CreateTagLayoutView_tmpl.html 40827ef 
  dashboardv2/public/js/templates/tag/TagAttributeDetailLayoutView_tmpl.html 
d418782 
  dashboardv2/public/js/templates/tag/TagDetailTableLayoutView_tmpl.html 
76f225a 
  dashboardv2/public/js/templates/tag/TagLayoutView_tmpl.html f786df4 
  dashboardv2/public/js/utils/CommonViewFunction.js 9b9ffcc 
  dashboardv2/public/js/utils/Enums.js a7b9a8b 
  dashboardv2/public/js/utils/Messages.js 44604cf 
  dashboardv2/public/js/views/detail_page/DetailPageLayoutView.js 802f6a4 
  dashboardv2/public/js/views/search/SearchResultLayoutView.js 86520c9 
  dashboardv2/public/js/views/tag/AddTagModalView.js d1c8c67 
  dashboardv2/public/js/views/tag/AddTimezoneItemView.js PRE-CREATION 
  dashboardv2/public/js/views/tag/CreateTagLayoutView.js e43a431 
  dashboardv2/public/js/views/tag/TagAttributeDetailLayoutView.js 3d6b859 
  dashboardv2/public/js/views/tag/TagDetailTableLayoutView.js c05b48d 
  dashboardv2/public/js/views/tag/TagLayoutView.js 0edb378 


Diff: https://reviews.apache.org/r/65870/diff/3/

Changes: https://reviews.apache.org/r/65870/diff/2-3/


Testing
---

Done one round of sanity testing.


Thanks,

pratik pandey

[jira] [Commented] (ATLAS-2478) Elasticsearch support is broken for JanusGraph

[Pierre Padovani (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-2478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16386101#comment-16386101
 ] 

Pierre Padovani commented on ATLAS-2478:


[~davidrad] Could I get your thoughts on this issue please? Include anyone else 
you think might have some input.

 

Thanks!

> Elasticsearch support is broken for JanusGraph
> --
>
> Key: ATLAS-2478
> URL: https://issues.apache.org/jira/browse/ATLAS-2478
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: 1.0.0-alpha
>Reporter: Pierre Padovani
>Assignee: Pierre Padovani
>Priority: Major
> Fix For: 1.0.0
>
>
> With JanusGraph the Elasticsearch support moved to 5.x+. This introduced a 
> change where fields that contained '.' (dots) were not allowed unless either 
> a specific cluster wide setting was enabled AND the mapping was formatted 
> such that each of the fields that contained a '.' could be considered part of 
> an object.
> Example:
> {code:java}
> foo.x
> foo.y
> foo.z{code}
>  Elasticsearch looks at these fields as if they are truly:
> {code:java}
> foo : {
>   x,
>   y,
>   z
> }{code}
> In the file:
> /atlas/common/src/main/java/org/apache/atlas/repository/Constants.java
> {code:java}
> /**
>  * Properties for type store graph.
>  */
> public static final String TYPE_CATEGORY_PROPERTY_KEY = 
> INTERNAL_PROPERTY_KEY_PREFIX + "type.category";
> public static final String VERTEX_TYPE_PROPERTY_KEY = 
> INTERNAL_PROPERTY_KEY_PREFIX + "type";
> public static final String TYPENAME_PROPERTY_KEY = 
> INTERNAL_PROPERTY_KEY_PREFIX + "type.name";
> public static final String TYPEDESCRIPTION_PROPERTY_KEY = 
> INTERNAL_PROPERTY_KEY_PREFIX + "type.description";
> public static final String TYPEVERSION_PROPERTY_KEY = 
> INTERNAL_PROPERTY_KEY_PREFIX + "type.version";
> public static final String TYPEOPTIONS_PROPERTY_KEY = 
> INTERNAL_PROPERTY_KEY_PREFIX + "type.options";
> {code}
> These are the only fields that cause Elasticsearch issue. As you can see a 
> field called 'type' is created, then additional fields type.name, 
> type.description etc. This will cause a mapping conflict exception in 
> Elasticsearch and it will refuse to create the mapping.
>  
> The easy fix is to simply replace the '.' with an '_' (underscore) but this 
> will be a backwards incompatible change for existing customers. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (ATLAS-2478) Elasticsearch support is broken for JanusGraph

[Pierre Padovani (JIRA)]

Pierre Padovani created ATLAS-2478:
--

 Summary: Elasticsearch support is broken for JanusGraph
 Key: ATLAS-2478
 URL: https://issues.apache.org/jira/browse/ATLAS-2478
 Project: Atlas
  Issue Type: Bug
  Components:  atlas-core
Affects Versions: 1.0.0-alpha
Reporter: Pierre Padovani
Assignee: Pierre Padovani
 Fix For: 1.0.0


With JanusGraph the Elasticsearch support moved to 5.x+. This introduced a 
change where fields that contained '.' (dots) were not allowed unless either a 
specific cluster wide setting was enabled AND the mapping was formatted such 
that each of the fields that contained a '.' could be considered part of an 
object.

Example:
{code:java}
foo.x
foo.y
foo.z{code}
 Elasticsearch looks at these fields as if they are truly:
{code:java}
foo : {
  x,
  y,
  z
}{code}
In the file:

/atlas/common/src/main/java/org/apache/atlas/repository/Constants.java
{code:java}
/**
 * Properties for type store graph.
 */
public static final String TYPE_CATEGORY_PROPERTY_KEY = 
INTERNAL_PROPERTY_KEY_PREFIX + "type.category";
public static final String VERTEX_TYPE_PROPERTY_KEY = 
INTERNAL_PROPERTY_KEY_PREFIX + "type";
public static final String TYPENAME_PROPERTY_KEY = INTERNAL_PROPERTY_KEY_PREFIX 
+ "type.name";
public static final String TYPEDESCRIPTION_PROPERTY_KEY = 
INTERNAL_PROPERTY_KEY_PREFIX + "type.description";
public static final String TYPEVERSION_PROPERTY_KEY = 
INTERNAL_PROPERTY_KEY_PREFIX + "type.version";
public static final String TYPEOPTIONS_PROPERTY_KEY = 
INTERNAL_PROPERTY_KEY_PREFIX + "type.options";

{code}
These are the only fields that cause Elasticsearch issue. As you can see a 
field called 'type' is created, then additional fields type.name, 
type.description etc. This will cause a mapping conflict exception in 
Elasticsearch and it will refuse to create the mapping.

 

The easy fix is to simply replace the '.' with an '_' (underscore) but this 
will be a backwards incompatible change for existing customers. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (ATLAS-2472) Update Atlas documentation for Cassandra support

[Pierre Padovani (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-2472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Padovani reassigned ATLAS-2472:
--

Assignee: Pierre Padovani

> Update Atlas documentation for Cassandra support
> 
>
> Key: ATLAS-2472
> URL: https://issues.apache.org/jira/browse/ATLAS-2472
> Project: Atlas
>  Issue Type: Sub-task
>Reporter: Pierre Padovani
>Assignee: Pierre Padovani
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (ATLAS-2471) Create Dockerfile support for Cassandra

[Pierre Padovani (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-2471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Padovani reassigned ATLAS-2471:
--

Assignee: Pierre Padovani

> Create Dockerfile support for Cassandra
> ---
>
> Key: ATLAS-2471
> URL: https://issues.apache.org/jira/browse/ATLAS-2471
> Project: Atlas
>  Issue Type: Sub-task
>Reporter: Pierre Padovani
>Assignee: Pierre Padovani
>Priority: Major
>
> Add an additional Dockerfile that builds a self contained container that runs 
> on Cassandra 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (ATLAS-2469) Tag propagation from object to child object or derivative asset

[pratik pandey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-2469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

pratik pandey updated ATLAS-2469:
-
Attachment: ATLAS-2469.patch

> Tag propagation from object to child object or derivative asset
> ---
>
> Key: ATLAS-2469
> URL: https://issues.apache.org/jira/browse/ATLAS-2469
> Project: Atlas
>  Issue Type: New Feature
>Affects Versions: 1.0.0
>Reporter: pratik pandey
>Assignee: pratik pandey
>Priority: Blocker
> Fix For: 1.0.0
>
> Attachments: ATLAS-2469.patch
>
>
> User Story:
> As a data steward i need a scalable way to quickly and efficiently propagate 
> tags for efficient searches and tag based security. Likewise tags for 
> derivative dataset should be inherited from the parent. For example, if an 
> entity is tagged "Secret" then resulting entity created from a CTAS operation 
> should also be tagged "secret" to maintain the classification of the parent. 
> In the case where 2 or more datasets are aggregated the derivative dataset 
> should be a union of all parent tags.
> Terms:
> * Child business terms should inherit the tags associated with the parent 
> term.
> * the option to propagate tags to child business terms in a hierarchy should 
> be provided
> * Ability to update the propagated tags manually via UI or through the API
> * Tagging a term should propagate to data assets that are already attached to 
> that business term as well
> Data assets
> * For supported components in HDP 2.6,if a derivative asset is created it 
> should inherit the tags and attributes from the original asset.
> * the option to propagate tags to child entities should be provided (e.g. if 
> you tag a hdfs folder optionally tag all the files within it)
> * Ability to update the propagated tags manually via UI or through the API
> * Tagging a parent object should be inherited even after child creation 
> dynamically (unless a flag is set not to do this)
> * Derived data assets should have the tags of the original data asset.
> conflict resolution - if the different values for attributes then a UX dialog 
> that prompts user for action needs to be provided. Once resolved, the 
> resolved value will be carry forth to derivative dataset.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 65885: ATLAS-2470 - Add JanusGraph Cassandra support to Atlas

[Pierre Padovani]

> On March 4, 2018, 11:07 a.m., David Radley wrote:
> > My review comments are from my initial look at the code; I hope to try 
> > running this patch to verify it works for me
> 
> Pierre Padovani wrote:
> I'll update the patch with the above changes today or tomorrow as I have 
> time.
> 
> Pierre Padovani wrote:
> I've updated the ATLAS-2470 with a new patch that should fix all of these 
> issues.

Hi David, I think I'm missing something. The instructions on the 'Using GIT 
with Atlas' page are unclear on patch revisions. Do I create another review, or 
do I somehow update this review?


- Pierre


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65885/#review198593
---


On March 2, 2018, 4:51 p.m., Pierre Padovani wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65885/
> ---
> 
> (Updated March 2, 2018, 4:51 p.m.)
> 
> 
> Review request for atlas and David Radley.
> 
> 
> Bugs: ATLAS-2470
> https://issues.apache.org/jira/browse/ATLAS-2470
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This updates pom's to add the needed cassandra jars, and adds a dist profile 
> embedded-cassandra-solr.
> 
> 
> Diffs
> -
> 
>   distro/pom.xml 0103bef6 
>   distro/src/bin/atlas_config.py e6415cf4 
>   distro/src/bin/atlas_start.py 39be6b7c 
>   distro/src/bin/atlas_stop.py 66edd904 
>   distro/src/conf/cassandra.yml PRE-CREATION 
>   distro/src/conf/zookeeper/log4j.properties PRE-CREATION 
>   distro/src/conf/zookeeper/zoo.cfg PRE-CREATION 
>   distro/src/main/assemblies/standalone-package.xml 1881082e 
>   docs/src/site/twiki/InstallationSteps.twiki 6b9f0313 
>   graphdb/janus/pom.xml 143b775f 
> 
> 
> Diff: https://reviews.apache.org/r/65885/diff/1/
> 
> 
> Testing
> ---
> 
> Full build with the new embedded-cassandra-solr, and testing to make sure 
> Atlas comes up and is functional.
> 
> Be aware that we have been running Cassandra backed JanusGraph for months 
> with no issues.
> 
> 
> Thanks,
> 
> Pierre Padovani
> 
>

[jira] [Updated] (ATLAS-2470) Basic support for Cassandra

[Pierre Padovani (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-2470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Padovani updated ATLAS-2470:
---
Attachment: ATLAS-2470-1.patch

> Basic support for Cassandra 
> 
>
> Key: ATLAS-2470
> URL: https://issues.apache.org/jira/browse/ATLAS-2470
> Project: Atlas
>  Issue Type: Sub-task
>Affects Versions: 1.0.0
>Reporter: Pierre Padovani
>Assignee: Pierre Padovani
>Priority: Major
> Fix For: 1.0.0
>
> Attachments: ATLAS-2470-1.patch, ATLAS-2470.patch
>
>
> Add the basic build support, and ability to run embedded Cassandra and solr 
> using a profile build:
> -Pdist,embedded-cassandra-solr



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: PreCommit-ATLAS-Build-Test #137-master-ATLAS-2477.1.patch

[Apache Jenkins Server]

See 


--
[...truncated 232.02 KB...]
at 
org.apache.atlas.query.DSLQueriesTest.queryAssert(DSLQueriesTest.java:256)
at org.apache.atlas.query.DSLQueriesTest.basic(DSLQueriesTest.java:234)

basic(org.apache.atlas.query.DSLQueriesTest)  Time elapsed: 0.125 sec  <<< 
FAILURE!
java.lang.AssertionError: `Log Data` expected [3] but found [4]
at org.testng.Assert.fail(Assert.java:94)
at org.testng.Assert.failNotEquals(Assert.java:496)
at org.testng.Assert.assertEquals(Assert.java:125)
at org.testng.Assert.assertEquals(Assert.java:372)
at 
org.apache.atlas.query.DSLQueriesTest.assertSearchResult(DSLQueriesTest.java:654)
at 
org.apache.atlas.query.DSLQueriesTest.queryAssert(DSLQueriesTest.java:256)
at org.apache.atlas.query.DSLQueriesTest.basic(DSLQueriesTest.java:234)

syntax(org.apache.atlas.query.DSLQueriesTest)  Time elapsed: 0.107 sec  <<< 
FAILURE!
java.lang.AssertionError: hive_table isa Dimension expected [3] but found [5]
at org.testng.Assert.fail(Assert.java:94)
at org.testng.Assert.failNotEquals(Assert.java:496)
at org.testng.Assert.assertEquals(Assert.java:125)
at org.testng.Assert.assertEquals(Assert.java:372)
at 
org.apache.atlas.query.DSLQueriesTest.assertSearchResult(DSLQueriesTest.java:654)
at 
org.apache.atlas.query.DSLQueriesTest.queryAssert(DSLQueriesTest.java:256)
at org.apache.atlas.query.DSLQueriesTest.syntax(DSLQueriesTest.java:368)

syntax(org.apache.atlas.query.DSLQueriesTest)  Time elapsed: 0.104 sec  <<< 
FAILURE!
java.lang.AssertionError: hive_table isa Dimension limit 3 offset 1 expected 
[2] but found [3]
at org.testng.Assert.fail(Assert.java:94)
at org.testng.Assert.failNotEquals(Assert.java:496)
at org.testng.Assert.assertEquals(Assert.java:125)
at org.testng.Assert.assertEquals(Assert.java:372)
at 
org.apache.atlas.query.DSLQueriesTest.assertSearchResult(DSLQueriesTest.java:654)
at 
org.apache.atlas.query.DSLQueriesTest.queryAssert(DSLQueriesTest.java:256)
at org.apache.atlas.query.DSLQueriesTest.syntax(DSLQueriesTest.java:368)


Results :

Failed tests: 
org.apache.atlas.query.DSLQueriesTest.basic(org.apache.atlas.query.DSLQueriesTest)
  Run 1: PASS
  Run 2: PASS
  Run 3: PASS
  Run 4: PASS
  Run 5: PASS
  Run 6: PASS
  Run 7: PASS
  Run 8: PASS
  Run 9: DSLQueriesTest.basic:234->queryAssert:256->assertSearchResult:654 
hive_table isa Dimension expected [3] but found [5]
  Run 10: PASS
  Run 11: PASS
  Run 12: PASS
  Run 13: PASS
  Run 14: PASS
  Run 15: PASS
  Run 16: PASS
  Run 17: PASS
  Run 18: PASS
  Run 19: PASS
  Run 20: PASS
  Run 21: PASS
  Run 22: PASS
  Run 23: PASS
  Run 24: DSLQueriesTest.basic:234->queryAssert:256->assertSearchResult:654 
Dimension expected [5] but found [9]
  Run 25: PASS
  Run 26: DSLQueriesTest.basic:234->queryAssert:256->assertSearchResult:654 ETL 
expected [5] but found [10]
  Run 27: DSLQueriesTest.basic:234->queryAssert:256->assertSearchResult:654 
Metric expected [5] but found [8]
  Run 28: PASS
  Run 29: DSLQueriesTest.basic:234->queryAssert:256->assertSearchResult:654 
`Log Data` expected [3] but found [4]
  Run 30: PASS
  Run 31: PASS

org.apache.atlas.query.DSLQueriesTest.syntax(org.apache.atlas.query.DSLQueriesTest)
  Run 1: PASS
  Run 2: PASS
  Run 3: PASS
  Run 4: PASS
  Run 5: PASS
  Run 6: PASS
  Run 7: PASS
  Run 8: PASS
  Run 9: PASS
  Run 10: PASS
  Run 11: PASS
  Run 12: PASS
  Run 13: PASS
  Run 14: PASS
  Run 15: PASS
  Run 16: PASS
  Run 17: PASS
  Run 18: PASS
  Run 19: PASS
  Run 20: PASS
  Run 21: PASS
  Run 22: PASS
  Run 23: PASS
  Run 24: PASS
  Run 25: PASS
  Run 26: DSLQueriesTest.syntax:368->queryAssert:256->assertSearchResult:654 
hive_table isa Dimension expected [3] but found [5]
  Run 27: PASS
  Run 28: PASS
  Run 29: PASS
  Run 30: DSLQueriesTest.syntax:368->queryAssert:256->assertSearchResult:654 
hive_table isa Dimension limit 3 offset 1 expected [2] but found [3]
  Run 31: PASS
  Run 32: PASS
  Run 33: PASS
  Run 34: PASS
  Run 35: PASS
  Run 36: PASS
  Run 37: PASS
  Run 38: PASS
  Run 39: PASS
  Run 40: PASS
  Run 41: PASS
  Run 42: PASS
  Run 43: PASS
  Run 44: PASS
  Run 45: PASS
  Run 46: PASS
  Run 47: PASS
  Run 48: PASS
  Run 49: PASS
  Run 50: PASS
  Run 51: PASS
  Run 52: PASS
  Run 53: PASS
  Run 54: PASS
  Run 55: PASS
  Run 56: PASS
  Run 57: PASS
  Run 58: PASS
  Run 59: PASS
  Run 60: PASS
  Run 61: PASS
  Run 62: PASS
  Run 63: PASS
  Run 64: PASS
  Run 65: PASS
  Run 66: PASS
  Run 67: PASS
  Run 68: PASS
  Run 69: PASS
  Run 70: PASS


Tests run: 351, Failures: 2, Errors: 0, Skipped: 0

[INFO] 
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Atlas Server Build Tools  SUCCESS [  0.541 s]
[INFO] 

Review Request 65902: [ATLAS-2477]: Tag Propagation and Search : Search with tag and type doesn't list all the entities

[Sarath Subramanian]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65902/
---

Review request for atlas, Apoorv Naik, Ashutosh Mestry, and Madhan Neethiraj.


Bugs: ATLAS-2477
https://issues.apache.org/jira/browse/ATLAS-2477


Repository: atlas


Description
---

1. Created an Hive table "employee" and CTAS table "employee_ctas".
2. Associated a tag tag1 to employee.Both employee and employee_ctas are 
associated to tag1.
3. Basic search with classification = tag1 , listed 3 entities - employee , 
employee_ctas and the process.
4. But Search with classification = tag1 , type = hive_table listed only the 
employee . Expected that it would list the employee_ctas also.
5.With Advanced search, both the following queries

query =tag1
typename = hive_table query = isa tag1

lists only the employee entity.

Hence , search with type  and tag lists only the entity to which the tag is 
associated and not the propagated associations.


Diffs
-

  
repository/src/main/java/org/apache/atlas/discovery/ClassificationSearchProcessor.java
 4265e098 
  
repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java 
016ad1b9 
  repository/src/main/java/org/apache/atlas/query/GremlinClause.java 38411037 
  repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java 
9a66636d 
  
repository/src/main/java/org/apache/atlas/util/AtlasGremlin2QueryProvider.java 
65b99cd0 


Diff: https://reviews.apache.org/r/65902/diff/1/


Testing
---


Thanks,

Sarath Subramanian

Re: Review Request 65870: ATLAS-2469 : Tag propagation from object to child object or derivative asset.

[pratik pandey]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65870/
---

(Updated March 5, 2018, 12:20 p.m.)


Review request for atlas, keval bhatt, Madhan Neethiraj, Nixon Rodrigues, and 
Sarath Subramanian.


Changes
---

Review Commnet handled.


Bugs: ATLAS-2469
https://issues.apache.org/jira/browse/ATLAS-2469


Repository: atlas


Description
---

User Story:

As a data steward i need a scalable way to quickly and efficiently propagate 
tags for efficient searches and tag based security. Likewise tags for 
derivative dataset should be inherited from the parent. For example, if an 
entity is tagged "Secret" then resulting entity created from a CTAS operation 
should also be tagged "secret" to maintain the classification of the parent. In 
the case where 2 or more datasets are aggregated the derivative dataset should 
be a union of all parent tags.

Terms:

Child business terms should inherit the tags associated with the parent term.
the option to propagate tags to child business terms in a hierarchy should be 
provided
Ability to update the propagated tags manually via UI or through the API
Tagging a term should propagate to data assets that are already attached to 
that business term as well
Data assets

For supported components in HDP 2.6,if a derivative asset is created it should 
inherit the tags and attributes from the original asset.
the option to propagate tags to child entities should be provided (e.g. if you 
tag a hdfs folder optionally tag all the files within it)
Ability to update the propagated tags manually via UI or through the API
Tagging a parent object should be inherited even after child creation 
dynamically (unless a flag is set not to do this)
Derived data assets should have the tags of the original data asset.
conflict resolution - if the different values for attributes then a UX dialog 
that prompts user for action needs to be provided. Once resolved, the resolved 
value will be carry forth to derivative dataset.


Diffs (updated)
-

  dashboardv2/gruntfile.js 451a933 
  dashboardv2/package.json 9237c35 
  dashboardv2/public/css/scss/override.scss f1e2c6b 
  dashboardv2/public/js/main.js ce0f4fe 
  dashboardv2/public/js/templates/detail_page/DetailPageLayoutView_tmpl.html 
1578548 
  dashboardv2/public/js/templates/search/SearchLayoutView_tmpl.html 5d3c3af 
  dashboardv2/public/js/templates/search/SearchResultLayoutView_tmpl.html 
403b064 
  dashboardv2/public/js/templates/site/SideNavLayoutView_tmpl.html b706609 
  dashboardv2/public/js/templates/tag/AddTagModalView_tmpl.html a477532 
  dashboardv2/public/js/templates/tag/AddTimezoneView_tmpl.html PRE-CREATION 
  dashboardv2/public/js/templates/tag/CreateTagLayoutView_tmpl.html 40827ef 
  dashboardv2/public/js/templates/tag/TagAttributeDetailLayoutView_tmpl.html 
d418782 
  dashboardv2/public/js/templates/tag/TagDetailTableLayoutView_tmpl.html 
76f225a 
  dashboardv2/public/js/templates/tag/TagLayoutView_tmpl.html f786df4 
  dashboardv2/public/js/utils/CommonViewFunction.js 9b9ffcc 
  dashboardv2/public/js/utils/Enums.js a7b9a8b 
  dashboardv2/public/js/views/detail_page/DetailPageLayoutView.js 802f6a4 
  dashboardv2/public/js/views/search/SearchResultLayoutView.js 86520c9 
  dashboardv2/public/js/views/tag/AddTagModalView.js d1c8c67 
  dashboardv2/public/js/views/tag/AddTimezoneItemView.js PRE-CREATION 
  dashboardv2/public/js/views/tag/CreateTagLayoutView.js e43a431 
  dashboardv2/public/js/views/tag/TagAttributeDetailLayoutView.js 3d6b859 
  dashboardv2/public/js/views/tag/TagDetailTableLayoutView.js c05b48d 
  dashboardv2/public/js/views/tag/TagLayoutView.js 0edb378 


Diff: https://reviews.apache.org/r/65870/diff/2/

Changes: https://reviews.apache.org/r/65870/diff/1-2/


Testing
---

Done one round of sanity testing.


Thanks,

pratik pandey

[jira] [Created] (ATLAS-2477) Tag Propagation and Search : Search with tag and type doesn't list all the entities

[Sarath Subramanian (JIRA)]

Sarath Subramanian created ATLAS-2477:
-

 Summary: Tag Propagation and Search : Search with tag and type 
doesn't list all the entities
 Key: ATLAS-2477
 URL: https://issues.apache.org/jira/browse/ATLAS-2477
 Project: Atlas
  Issue Type: Bug
  Components:  atlas-core
Affects Versions: 1.0.0
Reporter: Sarath Subramanian
Assignee: Sarath Subramanian
 Fix For: 1.0.0


1. Created an Hive table "employee" and CTAS table "employee_ctas".
2. Associated a tag tag1 to employee.Both employee and employee_ctas are 
associated to tag1.
3. Basic search with classification = tag1 , listed 3 entities - employee , 
employee_ctas and the process.
4. But Search with classification = tag1 , type = hive_table listed only the 
employee . Expected that it would list the employee_ctas also.
5.With Advanced search, both the following queries
{code}
query =tag1
typename = hive_table query = isa tag1
{code}
lists only the employee entity.

Hence , search with type  and tag lists only the entity to which the tag is 
associated and not the propagated associations.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Version 1.0 features and v1 API compatibility

[Ismaël Mejía]

Hello,

We worked with some colleagues in an integration with Atlas 0.7/0.8
almost one year ago, I was testing it with the latest releases and it
works flawlessly with the latest release 0.8.2. However with version
1.0.0-alpha and I found that even if the version seems to still
support the first version of the API (v1) all the package are in
different places, is this intended ? Because if this is the case well,
this breaks a bit the contract of backward compatibility.

And a second question, is there a list of features included in version
1.0 somewhere, or more concretely on the new v2 API? I have checked a
bit at the endpoints of the REST API but I was wondering if there is a
more user friendly doc

Thanks,
Ismaël