[VOTE] Release Apache Atlas version 2.1.0 - rc3
Atlas team, Apache Atlas 2.1.0 rc3, with following fixes since rc2, is now available for vote within dev community. ATLAS-3877: Purged entity audit throws 404 Entity not found error ATLAS-3882: Classic UI: Loader not disappearing while assigning category to term Links to the release artifacts are given below. Please review and vote. The vote will be open for at least 72 hours or until necessary votes are reached. [ ] +1 Approve [ ] +0 No opinion [ ] -1 Disapprove (and reason why) Thanks, Madhan List of all issues addressed in this release: https://issues.apache.org/jira/issues/?jql=project=ATLAS AND status=Resolved AND fixVersion=2.1.0 ORDER BY key DESC Git tag for the release: https://github.com/apache/atlas/tree/release-2.1.0-rc3 Sources for the release: https://dist.apache.org/repos/dist/dev/atlas/2.1.0-rc3/apache-atlas-2.1.0-sources.tar.gz Source release verification: PGP Signature: https://dist.apache.org/repos/dist/dev/atlas/2.1.0-rc3/apache-atlas-2.1.0-sources.tar.gz.asc SHA512 Hash: https://dist.apache.org/repos/dist/dev/atlas/2.1.0-rc3/apache-atlas-2.1.0-sources.tar.gz.sha512 MD5 Hash: https://dist.apache.org/repos/dist/dev/atlas/2.1.0-rc3/apache-atlas-2.1.0-sources.tar.gz.md5 Keys to verify the signature of the release artifacts are available at: https://dist.apache.org/repos/dist/release/atlas/KEYS New features/enhancements: - Quick Search: provides a simpler search experience with type-ahead suggestions - Business Metadata: enables augmenting entity-types with additional attributes, search entities using these attributes - Labels: ability to add/remove labels on entities, and search entities using labels - Custom Attributes: ability to add entity instance specific custom attributes i.e. attributes not defined in entity-def or business metadata - Entity Purge: added REST APIs to purge deleted entities Enhancements: - Search: ability to find entities by more than one classification - Performance: improvements in lineage retrieval and classification-propagation - Notification: ability to process notifications from multiple Kafka topics - Hive Hook: tracks process-executions via hive_process_execution entities - Hive Hook: captures DDL operations via hive_db_ddl and hive_table_ddl entities - Notification: introduced shell entities to record references to non-existing entities in notifications - Spark: added model to capture Spark entities, processes and relationships - AWS S3: introduced updated model to capture AWS S3 entities and relationships - ADLS-Gen2: introduced model to capture Azure Data Lake Storage Gen2 entities and relationships - Dependencies: JanusGraph 0.5.1, Tinkerpop 3.4.6, Spring Framework 4.3.20 - Authorization: updated to cover new features, like: business metadata, labels, purge - UI: multiple UI improvements, including a beta UI
[CANCEL] [VOTE] Release Apache Atlas version 2.1.0 - rc2
Atlas team, Thank you for validating rc2. Following issues were reported during rc2 validation, and fix for these issues were committed as well. I will generate rc3 shortly with these fixes. ATLAS-3877: Purged entity audit throws 404 Entity not found error ATLAS-3882: Classic UI: Loader not disappearing while assigning category to term Thanks, Madhan On 7/4/20, 11:50 PM, "Madhan Neethiraj" wrote: Atlas team, Apache Atlas 2.1.0 rc2 is now available for vote within dev community. Links to the release artifacts are given below. Please review and vote. The vote will be open for at least 72 hours or until necessary votes are reached. [ ] +1 Approve [ ] +0 No opinion [ ] -1 Disapprove (and reason why) Thanks, Madhan List of all issues addressed in this release: https://issues.apache.org/jira/issues/?jql=project=ATLAS AND status=Resolved AND fixVersion=2.1.0 ORDER BY key DESC Git tag for the release: https://github.com/apache/atlas/tree/release-2.1.0-rc2 Sources for the release: https://dist.apache.org/repos/dist/dev/atlas/2.1.0-rc2/apache-atlas-2.1.0-sources.tar.gz Source release verification: PGP Signature: https://dist.apache.org/repos/dist/dev/atlas/2.1.0-rc2/apache-atlas-2.1.0-sources.tar.gz.asc SHA512 Hash: https://dist.apache.org/repos/dist/dev/atlas/2.1.0-rc2/apache-atlas-2.1.0-sources.tar.gz.sha512 MD5 Hash: https://dist.apache.org/repos/dist/dev/atlas/2.1.0-rc2/apache-atlas-2.1.0-sources.tar.gz.md5 Keys to verify the signature of the release artifacts are available at: https://dist.apache.org/repos/dist/release/atlas/KEYS New features/enhancements: - Quick Search: provides a simpler search experience with type-ahead suggestions - Business Metadata: enables augmenting entity-types with additional attributes, search entities using these attributes - Labels: ability to add/remove labels on entities, and search entities using labels - Custom Attributes: ability to add entity instance specific custom attributes i.e. attributes not defined in entity-def or business metadata - Entity Purge: added REST APIs to purge deleted entities Enhancements: - Search: ability to find entities by more than one classification - Performance: improvements in lineage retrieval and classification-propagation - Notification: ability to process notifications from multiple Kafka topics - Hive Hook: tracks process-executions via hive_process_execution entities - Hive Hook: captures DDL operations via hive_db_ddl and hive_table_ddl entities - Notification: introduced shell entities to record references to non-existing entities in notifications - Spark: added model to capture Spark entities, processes and relationships - AWS S3: introduced updated model to capture AWS S3 entities and relationships - ADLS-Gen2: introduced model to capture Azure Data Lake Storage Gen2 entities and relationships - Dependencies: JanusGraph 0.5.1, Tinkerpop 3.4.6, Spring Framework 4.3.20 - Authorization: updated to cover new features, like: business metadata, labels, purge - UI: multiple UI improvements, including a beta UI
Re: Review Request 72638: ATLAS-3875: Enhancing AtlasClient by adding missing API endpoints
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72638/ --- (Updated July 10, 2020, 12:05 a.m.) Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, Sarath Subramanian, and Sidharth Mishra. Bugs: ATLAS-3875 https://issues.apache.org/jira/browse/ATLAS-3875 Repository: atlas Description --- There are many missing API endpoints in AtlasClientV2. This solution is adding functions corresponding to those missing APIs for the following Rest endpoints. 1. TypeRest 2. EntityRest 3. LineageRest 4. DiscoveryRest 5. GlossaryRest 6. RelationshipRest This will enable users to test and integrate with APIs more effectively via AtlasCient. Diffs (updated) - client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 7b6d1d0f3 client/common/src/main/java/org/apache/atlas/AtlasBaseClient.java e3d2ebc34 webapp/pom.xml 3c55b4dd4 webapp/src/test/java/org/apache/atlas/web/TestUtils.java e22a1c10d webapp/src/test/java/org/apache/atlas/web/integration/BaseResourceIT.java 680028892 webapp/src/test/java/org/apache/atlas/web/integration/BasicSearchIT.java 808f62354 webapp/src/test/java/org/apache/atlas/web/integration/EntityV2JerseyResourceIT.java cc883d615 webapp/src/test/java/org/apache/atlas/web/integration/GlossaryClientV2IT.java PRE-CREATION webapp/src/test/java/org/apache/atlas/web/integration/LineageClientV2IT.java PRE-CREATION webapp/src/test/java/org/apache/atlas/web/integration/TypedefsJerseyResourceIT.java 331ea2c47 webapp/src/test/resources/json/search-parameters/attribute-filters.json PRE-CREATION Diff: https://reviews.apache.org/r/72638/diff/4/ Changes: https://reviews.apache.org/r/72638/diff/3-4/ Testing --- IT has been added for newly created functions in AtlasClientV2. Validated by running IT on local system. Thanks, Jyoti Singh
Re: Review Request 72638: ATLAS-3875: Enhancing AtlasClient by adding missing API endpoints
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72638/ --- (Updated July 10, 2020, midnight) Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, Sarath Subramanian, and Sidharth Mishra. Bugs: ATLAS-3875 https://issues.apache.org/jira/browse/ATLAS-3875 Repository: atlas Description --- There are many missing API endpoints in AtlasClientV2. This solution is adding functions corresponding to those missing APIs for the following Rest endpoints. 1. TypeRest 2. EntityRest 3. LineageRest 4. DiscoveryRest 5. GlossaryRest 6. RelationshipRest This will enable users to test and integrate with APIs more effectively via AtlasCient. Diffs (updated) - client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 7b6d1d0f3 client/common/src/main/java/org/apache/atlas/AtlasBaseClient.java e3d2ebc34 webapp/pom.xml 3c55b4dd4 webapp/src/test/java/org/apache/atlas/web/TestUtils.java e22a1c10d webapp/src/test/java/org/apache/atlas/web/integration/BaseResourceIT.java 680028892 webapp/src/test/java/org/apache/atlas/web/integration/BasicSearchIT.java 808f62354 webapp/src/test/java/org/apache/atlas/web/integration/EntityV2JerseyResourceIT.java cc883d615 webapp/src/test/java/org/apache/atlas/web/integration/GlossaryClientV2IT.java PRE-CREATION webapp/src/test/java/org/apache/atlas/web/integration/LineageClientV2IT.java PRE-CREATION webapp/src/test/java/org/apache/atlas/web/integration/TypedefsJerseyResourceIT.java 331ea2c47 webapp/src/test/resources/json/search-parameters/attribute-filters.json PRE-CREATION Diff: https://reviews.apache.org/r/72638/diff/3/ Changes: https://reviews.apache.org/r/72638/diff/2-3/ Testing --- IT has been added for newly created functions in AtlasClientV2. Validated by running IT on local system. Thanks, Jyoti Singh
Re: Review Request 72666: Notification: Solution to Memory Build-up
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72666/ --- (Updated July 9, 2020, 6:17 p.m.) Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian. Changes --- Updates include: Improvement to initial buffer allocation. Bugs: ATLAS-3878 https://issues.apache.org/jira/browse/ATLAS-3878 Repository: atlas Description --- **Background** See JIRA for details. *Analysis* Using memory profiling tools, it was observed that large number of notification objects were created. These stayed in memory and later were promoted to higher generation, thereby taking even longer to be collected. **Approach** Using the fixed-buffer approach to address the problem of creating large number of small objects. New *FixedBufferList* This is an encapsulation over *ArrayList*. During initial allocation, list is populated with default values. Features: - Setting of values to these pre-allocated objects is achieved by first doing a *get* on the element and then assigning values to it. - *toList* fetches the sub-list from the encapsulating list. This uses the state within the class to fetch the right length for the returning array. New *NamedFixedBufferList* Maintains a per-thread *FixedBufferList*. This is necessary since the list is now part class's state. Modified *EntityAuditListenerV2* Uses the new classes. Modifed *EntityNotificationListener* Uses the new classes. **Verification** - Using the test setup, the memory usage was observed over a period of 24 hrs. - Memory usage and object allocation was obvserved using memory profiler. Diffs (updated) - intg/src/main/java/org/apache/atlas/utils/FixedBufferList.java PRE-CREATION intg/src/main/java/org/apache/atlas/utils/NamedFixedBufferList.java PRE-CREATION intg/src/test/java/org/apache/atlas/utils/FixedBufferListTest.java PRE-CREATION repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java 79527acfa webapp/src/main/java/org/apache/atlas/notification/EntityNotificationListenerV2.java a677b315c Diff: https://reviews.apache.org/r/72666/diff/2/ Changes: https://reviews.apache.org/r/72666/diff/1-2/ Testing --- **Unit testing** Unit tests added for the new classes. **Volume testing** Setup: - Node: Threads 40, Core: 40, Allocated Memory: 12 GB - Multiple Kafka queues ingesting data. - Bulk entity creation using custom script ingesting 100M entities. Memory usage stayed between 0 and 5% during the 24 hr period. **Pre-commit** https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/2035/ Thanks, Ashutosh Mestry
[jira] [Commented] (ATLAS-3880) BasicSearch: Multiple type/tag: Log invalid and allow searching valid type/tag names
[ https://issues.apache.org/jira/browse/ATLAS-3880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17154552#comment-17154552 ] ASF subversion and git services commented on ATLAS-3880: Commit f690755f79f9696e3214b9a3f158977b3bd5f4a7 in atlas's branch refs/heads/master from Pinal Shah [ https://gitbox.apache.org/repos/asf?p=atlas.git;h=f690755 ] ATLAS-3880 : BasicSearch: Multiple type/tag: Log invalid and allow searching valid type/tag names Signed-off-by: nixonrodrigues > BasicSearch: Multiple type/tag: Log invalid and allow searching valid > type/tag names > > > Key: ATLAS-3880 > URL: https://issues.apache.org/jira/browse/ATLAS-3880 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.0.0 >Reporter: Pinal >Assignee: Pinal >Priority: Major > Labels: BasicSearch, MultipleType/Tag > > If in basic search, request has combination of valid and invalid > typename/tags, allow searching for valid type names/tag names. > Example: > typeName:"valid,invalid" > Allow searching entities for valid type. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Review Request 72651: ATLAS-3880 : BasicSearch: Multiple type/tag: Log invalid and allow searching valid type/tag names
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72651/#review221164 --- Ship it! Ship It! - Nixon Rodrigues On July 8, 2020, 4:26 p.m., Pinal Shah wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/72651/ > --- > > (Updated July 8, 2020, 4:26 p.m.) > > > Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, > and Sarath Subramanian. > > > Bugs: ATLAS-3880 > https://issues.apache.org/jira/browse/ATLAS-3880 > > > Repository: atlas > > > Description > --- > > **Issue:** > If in basic search, request has combination of valid and invalid > typename/tags, it throws "errorCode": "ATLAS-400-00-00A" unknown/invalid > typeName for the invalid typeName. > > **WorkAround:** > Log invalid typeNames/tagNames and allow searching for valid > > **Example:** > typeName:"valid,invalid" > Allow searching entities for 'valid' type. > > > Diffs > - > > repository/src/main/java/org/apache/atlas/discovery/SearchContext.java > 04e82186b > > repository/src/test/java/org/apache/atlas/discovery/EntitySearchProcessorTest.java > b7ce97845 > webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java 140ed9a6e > > > Diff: https://reviews.apache.org/r/72651/diff/2/ > > > Testing > --- > > Manually tested > Precommit : https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2033 > (Failed in hive bridge) > > > Thanks, > > Pinal Shah > >
Re: Review Request 72651: ATLAS-3880 : BasicSearch: Multiple type/tag: Log invalid and allow searching valid type/tag names
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72651/#review221163 --- Ship it! Ship It! - Madhan Neethiraj On July 8, 2020, 4:26 p.m., Pinal Shah wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/72651/ > --- > > (Updated July 8, 2020, 4:26 p.m.) > > > Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, > and Sarath Subramanian. > > > Bugs: ATLAS-3880 > https://issues.apache.org/jira/browse/ATLAS-3880 > > > Repository: atlas > > > Description > --- > > **Issue:** > If in basic search, request has combination of valid and invalid > typename/tags, it throws "errorCode": "ATLAS-400-00-00A" unknown/invalid > typeName for the invalid typeName. > > **WorkAround:** > Log invalid typeNames/tagNames and allow searching for valid > > **Example:** > typeName:"valid,invalid" > Allow searching entities for 'valid' type. > > > Diffs > - > > repository/src/main/java/org/apache/atlas/discovery/SearchContext.java > 04e82186b > > repository/src/test/java/org/apache/atlas/discovery/EntitySearchProcessorTest.java > b7ce97845 > webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java 140ed9a6e > > > Diff: https://reviews.apache.org/r/72651/diff/2/ > > > Testing > --- > > Manually tested > Precommit : https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2033 (In > progress) > > > Thanks, > > Pinal Shah > >
[jira] [Created] (ATLAS-3886) Import hive metastore failed for ConversionException error
Lijun Ye created ATLAS-3886:
---
Summary: Import hive metastore failed for ConversionException error
Key: ATLAS-3886
URL: https://issues.apache.org/jira/browse/ATLAS-3886
Project: Atlas
Issue Type: Bug
Components: atlas-core, hive-integration
Affects Versions: 2.1.0
Environment: os: CentOS Linux release 7.6.1810 (Core)
jdk: java version "1.8.0_221"
Java(TM) SE Runtime Environment (build 1.8.0_221-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.221-b11, mixed mode)
hive: 2.3.2
atlas: 2.1.0-rc2
Reporter: Lijun Ye
When I import hive metastore, it occur this exception:
{code:java}
// code placeholder
org.apache.atlas.AtlasException: Failed to load application
propertiesorg.apache.atlas.AtlasException: Failed to load application
properties at
org.apache.atlas.ApplicationProperties.get(ApplicationProperties.java:147)
~[atlas-intg-2.1.0.jar:2.1.0] at
org.apache.atlas.ApplicationProperties.get(ApplicationProperties.java:100)
~[atlas-intg-2.1.0.jar:2.1.0] at
org.apache.atlas.hive.bridge.HiveMetaStoreBridge.main(HiveMetaStoreBridge.java:123)
[hive-bridge-2.1.0.jar:2.1.0]Caused by:
org.apache.commons.configuration.ConversionException:
'atlas.graph.index.search.solr.wait-searcher' doesn't map to a List object:
true, a java.lang.Boolean at
org.apache.commons.configuration.AbstractConfiguration.getList(AbstractConfiguration.java:1144)
~[commons-configuration-1.6.jar:1.6] at
org.apache.commons.configuration.AbstractConfiguration.getList(AbstractConfiguration.java:1109)
~[commons-configuration-1.6.jar:1.6] at
org.apache.commons.configuration.AbstractConfiguration.interpolatedConfiguration(AbstractConfiguration.java:1274)
~[commons-configuration-1.6.jar:1.6] at
org.apache.atlas.ApplicationProperties.get(ApplicationProperties.java:142)
~[atlas-intg-2.1.0.jar:2.1.0] ... 2 moreFailed to import Hive Meta Data!!!
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (ATLAS-3855) Bulk entity tag association and bulk api enhancement
[
https://issues.apache.org/jira/browse/ATLAS-3855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17154239#comment-17154239
]
ASF subversion and git services commented on ATLAS-3855:
Commit 06cd0cb35a285b3c4a7b0b3c662ce7263848f7b6 in atlas's branch
refs/heads/master from chaitali borole
[ https://gitbox.apache.org/repos/asf?p=atlas.git;h=06cd0cb ]
ATLAS-3855 :- Bulk entity tag association and bulk api enhancement,
authorization fix.
Signed-off-by: nixonrodrigues
> Bulk entity tag association and bulk api enhancement
>
>
> Key: ATLAS-3855
> URL: https://issues.apache.org/jira/browse/ATLAS-3855
> Project: Atlas
> Issue Type: Improvement
> Components: atlas-core
>Affects Versions: 2.0.0
>Reporter: chaitali borole
>Assignee: chaitali borole
>Priority: Major
> Labels: api-change
> Fix For: 2.1.0
>
>
> API : /api/atlas/v2/entity/bulk/classification
> Body :
> {code:java}
> {"classification":{"typeName":"PII","attributes":{},"propagate":true,"removePropagationsOnEntityDelete":false,"validityPeriods":[]},"entityGuids":["guid1","guid2"]}
> {code}
> when user doesn't have authorisation on either of guid1 or guid2 , the bulk
> entity call fails with 403.
> bulk api _v2/entity/bulk_ to retrieve a list of entities identified by its
> guids.
> This bulk api fails with 403 if some guids belong to entities on which user
> is *unauthorized* and other guids belong to entities on which user is
> *authorized*.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
