Re: Review Request 74608: ATLAS-4797 : Implement custom audit filters in Atlas
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74608/ --- (Updated April 4, 2024, 10:44 a.m.) Review request for atlas, Jayendra Parab, Mandar Ambawane, Pinal Shah, Prasad Pawar, Radhika Kundam, and Sidharth Mishra. Changes --- corrected operator case sensitivity logic Bugs: ATLAS-4797 https://issues.apache.org/jira/browse/ATLAS-4797 Repository: atlas Description (updated) --- Currently, Atlas has to persist all audit events without any constraints which can make audit data grow exponentially after some time. This feature is specifically to reduce Atlas audit storage where generation of audit events can be controlled based on the user's use-case/criteria Note: Audit data mentioned here is exclusive of Admin Audit Data which persists in atlas_janus table. Filtering will be applicable to the entity audit data which persists in ATLAS_ENTITY_AUDIT_EVENTS table. Precommit details : Latest PC build (PASSED) : build:https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1592/ Diffs (updated) - addons/hbase-bridge/pom.xml 278b6c6d3 addons/hbase-testing-util/pom.xml c4f3a99ee addons/hive-bridge-shim/pom.xml 39f16ceee addons/hive-bridge/pom.xml 356ac9542 addons/kafka-bridge/pom.xml 093db7229 addons/models/-Area0/0010-base_model.json a4a9248ec addons/sqoop-bridge/pom.xml 4b6eac98f authorization/pom.xml 7a1108487 distro/src/conf/atlas-application.properties b5734d7a8 graphdb/api/pom.xml 4ba89b20f graphdb/janus-hbase2/pom.xml 9ec087dbb intg/pom.xml 43a172c1b intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 0868f8b6e intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 77a6fd8c3 intg/src/main/java/org/apache/atlas/model/audit/EntityAuditEventV2.java 3afd27e7d repository/pom.xml 8fd744b44 repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java 9b011ab06 repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java b7f0dd5e2 repository/src/main/java/org/apache/atlas/repository/ogm/AtlasRuleDTO.java PRE-CREATION repository/src/main/java/org/apache/atlas/repository/ogm/DataAccess.java e63152475 repository/src/main/java/org/apache/atlas/rulesengine/AtlasEntityAuditFilterService.java PRE-CREATION repository/src/main/java/org/apache/atlas/rulesengine/AtlasRule.java PRE-CREATION repository/src/main/java/org/apache/atlas/rulesengine/AtlasRuleUtils.java PRE-CREATION repository/src/main/java/org/apache/atlas/rulesengine/AtlasRulesEngine.java PRE-CREATION repository/src/main/java/org/apache/atlas/rulesengine/RuleAction.java PRE-CREATION server-api/pom.xml de311c329 test-tools/src/main/resources/solr/core-template/solrconfig.xml 7a0e8dd16 webapp/src/main/java/org/apache/atlas/web/filters/ActiveServerFilter.java e19beccb1 webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 0025276bb webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java 268fca717 Diff: https://reviews.apache.org/r/74608/diff/13/ Changes: https://reviews.apache.org/r/74608/diff/12-13/ Testing --- Manual testing done Thanks, Sheetal Shah
[jira] [Commented] (ATLAS-4817) Change compression from 'GZ' to 'SNAPPY' in Atlas HBase tables
[ https://issues.apache.org/jira/browse/ATLAS-4817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17833819#comment-17833819 ] ASF subversion and git services commented on ATLAS-4817: Commit 80d2ffe1fb501a6ac5585909b872b9d137494883 in atlas's branch refs/heads/branch-2.0 from vinayak.marraiya [ https://gitbox.apache.org/repos/asf?p=atlas.git;h=80d2ffe1f ] ATLAS-4817 : Change compression from 'GZ' to 'SNAPPY' in Atlas HBase tables Signed-off-by: Pinal Shah > Change compression from 'GZ' to 'SNAPPY' in Atlas HBase tables > -- > > Key: ATLAS-4817 > URL: https://issues.apache.org/jira/browse/ATLAS-4817 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 2.3.0 >Reporter: VINAYAK MARRAIYA >Assignee: VINAYAK MARRAIYA >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (ATLAS-4817) Change compression from 'GZ' to 'SNAPPY' in Atlas HBase tables
[ https://issues.apache.org/jira/browse/ATLAS-4817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17833818#comment-17833818 ] ASF subversion and git services commented on ATLAS-4817: Commit ab89e78ed5737acbee409203d20c75bc10429b9d in atlas's branch refs/heads/master from vinayak.marraiya [ https://gitbox.apache.org/repos/asf?p=atlas.git;h=ab89e78ed ] ATLAS-4817 : Change compression from 'GZ' to 'SNAPPY' in Atlas HBase tables Signed-off-by: Pinal Shah > Change compression from 'GZ' to 'SNAPPY' in Atlas HBase tables > -- > > Key: ATLAS-4817 > URL: https://issues.apache.org/jira/browse/ATLAS-4817 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 2.3.0 >Reporter: VINAYAK MARRAIYA >Assignee: VINAYAK MARRAIYA >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Community over Code EU 2024: Start planning your trip!
[Note: You're receiving this email because you are subscribed to one or more project dev@ mailing lists at the Apache Software Foundation.] Dear community, We hope you are doing great, are you ready for Community Over Code EU? Check out the featured sessions, get your tickets with special discounts and start planning your trip. Save your spot! Take a look at our lineup of sessions, panelists and featured speakers and make your final choice: * EU policies and regulations affecting open source specialists working in OSPOs The panel will discuss how EU legislation affects the daily work of open source operations. Panelists will cover some recent policy updates, the challenges of staying compliant when managing open source contribution and usage within organizations, and their personal experiences in adapting to the changing European regulatory environment. * Doing for sustainability, what open source did for software In this keynote Asim Hussain will explain the history of Impact Framework, a coalition of hundreds of software practitioners with tangible solutions that directly foster meaningful change by measuring the environmental impacts of a piece of software. Don’t forget that we have special discounts for groups, students and Apache committers. Visit the website to discover more about these rates.[1] It's time for you to start planning your trip. Remember that we have prepared a “How to get there” guide that will be helpful to find out the best transportation, either train, bus, flight or boat to Bratislava from wherever you are coming from. Take a look at the different options and please reach out to us if you have any questions. We have available rooms -with a special rate- at the Radisson Blu Carlton Hotel, where the event will take place and at the Park Inn Hotel which is only 5 minutes walking from the venue. [2] However, you are free to choose any other accommodation options around the city. See you in Bratislava, Community Over Code EU Team [1]: https://eu.communityovercode.org/tickets/ "Register" [2]: https://eu.communityovercode.org/venue/ "Where to stay"
Participate in the ASF 25th Anniversary Campaign
Hi everyone, As part of The ASF’s 25th anniversary campaign[1], we will be celebrating projects and communities in multiple ways. We invite all projects and contributors to participate in the following ways: * Individuals - submit your first contribution: https://news.apache.org/foundation/entry/the-asf-launches-firstasfcontribution-campaign * Projects - share your public good story: https://docs.google.com/forms/d/1vuN-tUnBwpTgOE5xj3Z5AG1hsOoDNLBmGIqQHwQT6k8/viewform?edit_requested=true * Projects - submit a project spotlight for the blog: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=278466116 * Projects - contact the Voice of Apache podcast (formerly Feathercast) to be featured: https://feathercast.apache.org/help/ * Projects - use the 25th anniversary template and the #ASF25Years hashtag on social media: https://docs.google.com/presentation/d/1oDbMol3F_XQuCmttPYxBIOIjRuRBksUjDApjd8Ve3L8/edit#slide=id.g26b0919956e_0_13 If you have questions, email the Marketing & Publicity team at mark...@apache.org. Peace, BKP [1] https://apache.org/asf25years/ [NOTE: You are receiving this message because you are a contributor to an Apache Software Foundation project. The ASF will very occasionally send out messages relating to the Foundation to contributors and members, such as this one.] Brian Proffitt VP, Marketing & Publicity VP, Conferences
Re: Review Request 74951: ATLAS-4845 : Atlas Import is failing with fetchType: Incremental if there are no changes between two consecutive runs
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74951/#review226374 --- As per the changes in this patch, in case if no entities are found in the export, the export service will respond with an empty zip file with no files within and status code 204. Contradicting to this, behaviour as seen in the previous version for similar case of empty export is such : The export API call returns 3 json files (atlas-typesdef.json, atlas-export-order.json and atlas-export-info.json) in the zip output response with response code 200 and operation status "FAIL" Remark : Backward compatibility needs to be taken care of while handling empty export/zero delta for incremental export. Also, Operation Status to be marked as SUCCESS for every successful execution (even if there are no entities found for any request) unless there occurs any exception/error in the flow Suggestion : Make the changes based on a flag/setting which indicates an empty export; like so : omitZipResponseForEmptyExport ; the content-type in the response may also need to be set acccordingly - Sheetal Shah On April 2, 2024, 8:20 p.m., Priyanshi Shah wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/74951/ > --- > > (Updated April 2, 2024, 8:20 p.m.) > > > Review request for atlas, Jayendra Parab, Mandar Ambawane, Pinal Shah, and > Sheetal Shah. > > > Bugs: ATLAS-4845 > https://issues.apache.org/jira/browse/ATLAS-4845 > > > Repository: atlas > > > Description > --- > > Steps: > > 1. First export and import with changeMarker “0” - worked as expected. > 2. Try to run 2nd export + import without any changes in atlas, 2nd export > also worked and gave differentchangeMarker in output as expected. > > Expectation: > > a) Empty zip should be created if no entities is fetched. > > Problem: > > 2nd Import is getting failed. > > Solution: > > 1. Earlier export zip was created with default json files like > atlas-export-info.json, atlas-export-order.json and atlas-typesdef.json when > though no entities was fetched and hence, while performing import opertaion > failure was seen with Empty Zip File exception. > 2. Now, the problem is fixed by returning empty zip files if no entities was > fetched and the status code is set to 204. > > > Diffs > - > > repository/src/main/java/org/apache/atlas/repository/impexp/ZipSink.java > 5cec39dd6 > webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java > 3aa24 > > > Diff: https://reviews.apache.org/r/74951/diff/1/ > > > Testing > --- > > Manual testing for the same has been performed. > > PC is also successfull. > > https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1589/ > > > Thanks, > > Priyanshi Shah > >
Re: Review Request 74787: ATLAS-4817 : Change compression from 'GZ' to 'SNAPPY' in Atlas HBase tables
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74787/ --- (Updated April 3, 2024, 10:24 a.m.) Review request for atlas, Jayendra Parab and Pinal Shah. Repository: atlas Description --- Currently Atlas is using Compression Algorithm as 'GZ' chaning it to 'SNAPPY' Diffs - graphdb/janus-hbase2/src/main/java/org/janusgraph/diskstorage/hbase2/HBaseStoreManager.java f98fa7b76 repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java b7f0dd5e2 Diff: https://reviews.apache.org/r/74787/diff/2/ Testing (updated) --- 1) Unit tests New tests added. Volume testing +--+--+-+ | Description | Message | Duration| | | Count| | +--+--+-+ | Compression | | | | Alogrithm| 18000 | 4 hr 11 min | | as "GZ" | | | +--+--+-+ | Compression | | | | Alogrithm| 18000 | 1 hr 55 min | | as "SNAPPY" | | | +--+--+-+ 2) Precommit - https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1535/console Thanks, Vinayak Marraiya
[jira] [Created] (ATLAS-4846) Source the atlas-env file in ATLAS import script to get all the argument
Paresh Devalia created ATLAS-4846: - Summary: Source the atlas-env file in ATLAS import script to get all the argument Key: ATLAS-4846 URL: https://issues.apache.org/jira/browse/ATLAS-4846 Project: Atlas Issue Type: Bug Components: atlas-core Reporter: Paresh Devalia Assignee: Paresh Devalia User has to run manually export command to get all the argument which are set, to run the Atlas import script. The problem is not with the Atlas service itself but with these scripts: * /opt/cloudera/parcels/CDH/lib/atlas/hook-bin/import-kafka.sh * /opt/cloudera/parcels/CDH/lib/atlas/hook-bin/import-hbase.sh * /opt/cloudera/parcels/CDH/lib/atlas/hook-bin/import-hive.sh Need to set argument before this script are being called. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: Review Request 74738: ATLAS-4764: CuratorFrameWork should work only with Zookeeper HA mode
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74738/ --- (Updated April 3, 2024, 9:39 a.m.) Review request for atlas, Jayendra Parab, Mandar Ambawane, Nixon Rodrigues, Pinal Shah, and Sheetal Shah. Bugs: ATLAS-4764 https://issues.apache.org/jira/browse/ATLAS-4764 Repository: atlas Description --- Zookeeper Non-HA mode should not run through CuratorFramework. Diffs (updated) - webapp/src/main/java/org/apache/atlas/web/service/CuratorFactory.java 7c89055e0 Diff: https://reviews.apache.org/r/74738/diff/4/ Changes: https://reviews.apache.org/r/74738/diff/3-4/ Testing --- PC build: https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1591/console Testing : Tested locally. By default atlas.server.ha.enabled=false Thanks, Paresh Devalia
Re: Review Request 74738: ATLAS-4764: CuratorFrameWork should work only with Zookeeper HA mode
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74738/ --- (Updated April 3, 2024, 9:10 a.m.) Review request for atlas, Jayendra Parab, Mandar Ambawane, Nixon Rodrigues, Pinal Shah, and Sheetal Shah. Bugs: ATLAS-4764 https://issues.apache.org/jira/browse/ATLAS-4764 Repository: atlas Description --- Zookeeper Non-HA mode should not run through CuratorFramework. Diffs - webapp/src/main/java/org/apache/atlas/web/service/CuratorFactory.java 7c89055e0 Diff: https://reviews.apache.org/r/74738/diff/3/ Testing (updated) --- PC build: https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1591/console Testing : Tested locally. By default atlas.server.ha.enabled=false Thanks, Paresh Devalia
Re: Review Request 74738: ATLAS-4764: CuratorFrameWork should work only with Zookeeper HA mode
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74738/ --- (Updated April 3, 2024, 9:05 a.m.) Review request for atlas, Jayendra Parab, Mandar Ambawane, Nixon Rodrigues, Pinal Shah, and Sheetal Shah. Bugs: ATLAS-4764 https://issues.apache.org/jira/browse/ATLAS-4764 Repository: atlas Description --- Zookeeper Non-HA mode should not run through CuratorFramework. Diffs (updated) - webapp/src/main/java/org/apache/atlas/web/service/CuratorFactory.java 7c89055e0 Diff: https://reviews.apache.org/r/74738/diff/3/ Changes: https://reviews.apache.org/r/74738/diff/2-3/ Testing --- PC build: https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1591/console Thanks, Paresh Devalia
Re: Review Request 74738: ATLAS-4764: CuratorFrameWork should work only with Zookeeper HA mode
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74738/ --- (Updated April 3, 2024, 9:05 a.m.) Review request for atlas, Jayendra Parab, Mandar Ambawane, Nixon Rodrigues, Pinal Shah, and Sheetal Shah. Bugs: ATLAS-4764 https://issues.apache.org/jira/browse/ATLAS-4764 Repository: atlas Description --- Zookeeper Non-HA mode should not run through CuratorFramework. Diffs - distro/src/conf/atlas-application.properties b5734d7a8 webapp/src/main/java/org/apache/atlas/web/service/CuratorFactory.java 7c89055e0 Diff: https://reviews.apache.org/r/74738/diff/2/ Testing (updated) --- PC build: https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1591/console Thanks, Paresh Devalia
Re: Review Request 74951: ATLAS-4845 : Atlas Import is failing with fetchType: Incremental if there are no changes between two consecutive runs
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74951/#review226373 --- Ship it! Ship It! - Mandar Ambawane On April 2, 2024, 2:50 p.m., Priyanshi Shah wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/74951/ > --- > > (Updated April 2, 2024, 2:50 p.m.) > > > Review request for atlas, Jayendra Parab, Mandar Ambawane, Pinal Shah, and > Sheetal Shah. > > > Bugs: ATLAS-4845 > https://issues.apache.org/jira/browse/ATLAS-4845 > > > Repository: atlas > > > Description > --- > > Steps: > > 1. First export and import with changeMarker “0” - worked as expected. > 2. Try to run 2nd export + import without any changes in atlas, 2nd export > also worked and gave differentchangeMarker in output as expected. > > Expectation: > > a) Empty zip should be created if no entities is fetched. > > Problem: > > 2nd Import is getting failed. > > Solution: > > 1. Earlier export zip was created with default json files like > atlas-export-info.json, atlas-export-order.json and atlas-typesdef.json when > though no entities was fetched and hence, while performing import opertaion > failure was seen with Empty Zip File exception. > 2. Now, the problem is fixed by returning empty zip files if no entities was > fetched and the status code is set to 204. > > > Diffs > - > > repository/src/main/java/org/apache/atlas/repository/impexp/ZipSink.java > 5cec39dd6 > webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java > 3aa24 > > > Diff: https://reviews.apache.org/r/74951/diff/1/ > > > Testing > --- > > Manual testing for the same has been performed. > > PC is also successfull. > > https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1589/ > > > Thanks, > > Priyanshi Shah > >
Re: Review Request 74787: ATLAS-4817 : Change compression from 'GZ' to 'SNAPPY' in Atlas HBase tables
> On April 3, 2024, 8:23 a.m., Pinal Shah wrote: > > Can we add the tests performed in the description box > > Also Does SNAPPY took 4hr 11min for 18k messages ? less compared to GZ or > > it is mistake Editing above: time taken is more compared to GZ - Pinal --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74787/#review226371 --- On April 3, 2024, 4:19 a.m., Vinayak Marraiya wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/74787/ > --- > > (Updated April 3, 2024, 4:19 a.m.) > > > Review request for atlas, Jayendra Parab and Pinal Shah. > > > Repository: atlas > > > Description > --- > > Currently Atlas is using Compression Algorithm as 'GZ' chaning it to 'SNAPPY' > > > Diffs > - > > > graphdb/janus-hbase2/src/main/java/org/janusgraph/diskstorage/hbase2/HBaseStoreManager.java > f98fa7b76 > > repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java > b7f0dd5e2 > > > Diff: https://reviews.apache.org/r/74787/diff/2/ > > > Testing > --- > > 1) Unit tests >New tests added. > >Volume testing >+--+--+-+ >| Description | Message | Duration| >| | Count| | >+--+--+-+ >| Compression | | | >| Alogrithm| 18000 | 1 hr 55 min | >| as "GZ" | | | >+--+--+-+ >| Compression | | | >| Alogrithm| 18000 | 4 hr 11 min | >| as "SNAPPY" | | | >+--+--+-+ > > 2) Precommit - > https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1535/console > > > Thanks, > > Vinayak Marraiya > >
Re: Review Request 74787: ATLAS-4817 : Change compression from 'GZ' to 'SNAPPY' in Atlas HBase tables
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74787/#review226371 --- Ship it! Can we add the tests performed in the description box Also Does SNAPPY took 4hr 11min for 18k messages ? less compared to GZ or it is mistake - Pinal Shah On April 3, 2024, 4:19 a.m., Vinayak Marraiya wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/74787/ > --- > > (Updated April 3, 2024, 4:19 a.m.) > > > Review request for atlas, Jayendra Parab and Pinal Shah. > > > Repository: atlas > > > Description > --- > > Currently Atlas is using Compression Algorithm as 'GZ' chaning it to 'SNAPPY' > > > Diffs > - > > > graphdb/janus-hbase2/src/main/java/org/janusgraph/diskstorage/hbase2/HBaseStoreManager.java > f98fa7b76 > > repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java > b7f0dd5e2 > > > Diff: https://reviews.apache.org/r/74787/diff/2/ > > > Testing > --- > > 1) Unit tests >New tests added. > >Volume testing >+--+--+-+ >| Description | Message | Duration| >| | Count| | >+--+--+-+ >| Compression | | | >| Alogrithm| 18000 | 1 hr 55 min | >| as "GZ" | | | >+--+--+-+ >| Compression | | | >| Alogrithm| 18000 | 4 hr 11 min | >| as "SNAPPY" | | | >+--+--+-+ > > 2) Precommit - > https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1535/console > > > Thanks, > > Vinayak Marraiya > >
