Updating latest docs and links to 0.6 release
Project: http://git-wip-us.apache.org/repos/asf/incubator-atlas-website/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-atlas-website/commit/02b20682 Tree: http://git-wip-us.apache.org/repos/asf/incubator-atlas-website/tree/02b20682 Diff: http://git-wip-us.apache.org/repos/asf/incubator-atlas-website/diff/02b20682 Branch: refs/heads/asf-site Commit: 02b20682244f7e97e7ca93b58a2a2adedd4cdaa2 Parents: 5058c74 Author: Shwetha GS <[email protected]> Authored: Tue Jan 5 12:27:13 2016 +0530 Committer: Shwetha GS <[email protected]> Committed: Tue Jan 5 12:27:13 2016 +0530 ---------------------------------------------------------------------- Architecture.html | 20 +++- Bridge-Falcon.html | 261 ++++++++++++++++++++++++++++++++++++++++++ Bridge-Hive.html | 25 ++-- Bridge-Sqoop.html | 256 +++++++++++++++++++++++++++++++++++++++++ Configuration.html | 18 ++- HighAvailability.html | 18 ++- InstallationSteps.html | 152 ++++++++++++++++-------- Notification-Entity.html | 16 ++- QuickStart.html | 16 ++- Repository.html | 16 ++- Search.html | 24 ++-- Security.html | 147 ++++++++++++++++++++++-- TypeSystem.html | 16 ++- index.html | 20 +++- issue-tracking.html | 16 ++- license.html | 16 ++- mail-lists.html | 16 ++- project-info.html | 16 ++- source-repository.html | 16 ++- team-list.html | 27 ++++- 20 files changed, 964 insertions(+), 148 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-atlas-website/blob/02b20682/Architecture.html ---------------------------------------------------------------------- diff --git a/Architecture.html b/Architecture.html index aadd792..461d55c 100644 --- a/Architecture.html +++ b/Architecture.html @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia at 2015-12-30 + | Generated by Apache Maven Doxia at 2016-01-05 | Rendered using Apache Maven Fluido Skin 1.3.0 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20151230" /> + <meta name="Date-Revision-yyyymmdd" content="20160105" /> <meta http-equiv="Content-Language" content="en" /> <title>Apache Atlas – Architecture</title> <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" /> @@ -90,6 +90,9 @@ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Releases <b class="caret"></b></a> <ul class="dropdown-menu"> + <li> <a href="http://www.apache.org/dyn/closer.cgi/incubator/atlas/0.6.0-incubating/"; title="0.6-incubating">0.6-incubating</a> +</li> + <li> <a href="http://www.apache.org/dyn/closer.cgi/incubator/atlas/0.5.0-incubating/"; title="0.5-incubating">0.5-incubating</a> </li> </ul> @@ -98,6 +101,9 @@ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation <b class="caret"></b></a> <ul class="dropdown-menu"> + <li> <a href="0.6.0-incubating/index.html" title="0.6-incubating">0.6-incubating</a> +</li> + <li> <a href="0.5.0-incubating/index.html" title="0.5-incubating">0.5-incubating</a> </li> </ul> @@ -183,8 +189,8 @@ - <li id="publishDate" class="pull-right">Last Published: 2015-12-30</li> <li class="divider pull-right">|</li> - <li id="projectVersion" class="pull-right">Version: 0.6-incubating</li> + <li id="publishDate" class="pull-right">Last Published: 2016-01-05</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.7-incubating-SNAPSHOT</li> </ul> </div> @@ -215,7 +221,9 @@ <p>Atlas exposes notification interface and can be used for reliable entity registration by hook as well. The hook can send notification message containing the list of entities to be registered. Atlas service contains hook consumer that listens to these messages and registers the entities.</p> <p>Available bridges are:</p> <ul> -<li><a href="./Bridge-Hive.html">Hive Bridge</a></li></ul></div> +<li><a href="./Bridge-Hive.html">Hive Bridge</a></li> +<li><a href="./Bridge-Sqoop.html">Sqoop Bridge</a></li> +<li><a href="./Bridge-Falcon.html">Falcon Bridge</a></li></ul></div> <div class="section"> <h3><a name="Notification"></a>Notification</h3> <p>Notification is used for reliable entity registration from hooks and for entity/type change notifications. Atlas, by default, provides Kafka integration, but its possible to provide other implementations as well. Atlas service starts embedded Kafka server by default.</p> @@ -227,7 +235,7 @@ <footer> <div class="container"> - <div class="row span12">Copyright © 2015 + <div class="row span12">Copyright © 2015-2016 <a href="http://www.apache.org";>Apache Software Foundation</a>. All Rights Reserved. http://git-wip-us.apache.org/repos/asf/incubator-atlas-website/blob/02b20682/Bridge-Falcon.html ---------------------------------------------------------------------- diff --git a/Bridge-Falcon.html b/Bridge-Falcon.html new file mode 100644 index 0000000..bcf43c9 --- /dev/null +++ b/Bridge-Falcon.html @@ -0,0 +1,261 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2016-01-05 + | Rendered using Apache Maven Fluido Skin 1.3.0 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20160105" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Apache Atlas – Falcon Atlas Bridge</title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + + + <script type="text/javascript" src="./js/apache-maven-fluido-1.3.0.min.js"></script> + + + +<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> + + </head> + <body class="topBarEnabled"> + + + + + + <div id="topbar" class="navbar navbar-fixed-top "> + <div class="navbar-inner"> + <div class="container" style="width: 68%;"><div class="nav-collapse"> + + + <ul class="nav"> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Atlas <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="index.html" title="About">About</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="Wiki">Wiki</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="News">News</a> +</li> + + <li> <a href="https://git-wip-us.apache.org/repos/asf/incubator-atlas.git"; title="Git">Git</a> +</li> + + <li> <a href="https://issues.apache.org/jira/browse/ATLAS"; title="Jira">Jira</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/ATLAS/PoweredBy"; title="Powered by">Powered by</a> +</li> + + <li> <a href="http://blogs.apache.org/atlas/"; title="Blog">Blog</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Project Information <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="project-info.html" title="Summary">Summary</a> +</li> + + <li> <a href="mail-lists.html" title="Mailing Lists">Mailing Lists</a> +</li> + + <li> <a href="http://webchat.freenode.net?channels=apacheatlas&uio=d4"; title="IRC">IRC</a> +</li> + + <li> <a href="team-list.html" title="Team">Team</a> +</li> + + <li> <a href="issue-tracking.html" title="Issue Tracking">Issue Tracking</a> +</li> + + <li> <a href="source-repository.html" title="Source Repository">Source Repository</a> +</li> + + <li> <a href="license.html" title="License">License</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Releases <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="http://www.apache.org/dyn/closer.cgi/incubator/atlas/0.6.0-incubating/"; title="0.6-incubating">0.6-incubating</a> +</li> + + <li> <a href="http://www.apache.org/dyn/closer.cgi/incubator/atlas/0.5.0-incubating/"; title="0.5-incubating">0.5-incubating</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="0.6.0-incubating/index.html" title="0.6-incubating">0.6-incubating</a> +</li> + + <li> <a href="0.5.0-incubating/index.html" title="0.5-incubating">0.5-incubating</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">ASF <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="http://www.apache.org/foundation/how-it-works.html"; title="How Apache Works">How Apache Works</a> +</li> + + <li> <a href="http://www.apache.org/foundation/"; title="Foundation">Foundation</a> +</li> + + <li> <a href="http://www.apache.org/foundation/sponsorship.html"; title="Sponsoring Apache">Sponsoring Apache</a> +</li> + + <li> <a href="http://www.apache.org/foundation/thanks.html"; title="Thanks">Thanks</a> +</li> + </ul> + </li> + </ul> + + <form id="search-form" action="http://www.google.com/search"; method="get" class="navbar-search pull-right" > + + <input value="http://atlas.incubator.apache.org"; name="sitesearch" type="hidden"/> + <input class="search-query" name="q" id="query" type="text" /> +</form> +<script type="text/javascript" src="http://www.google.com/coop/cse/brand?form=search-form";></script> + + + + + + <iframe src="http://www.facebook.com/plugins/like.php?href=http://atlas.incubator.apache.org/atlas-docs&send=false&layout=button_count&show-faces=false&action=like&colorscheme=dark"; + scrolling="no" frameborder="0" + style="border:none; width:80px; height:20px; margin-top: 10px;" class="pull-right" ></iframe> + + <script type="text/javascript" src="https://apis.google.com/js/plusone.js";></script> + + <ul class="nav pull-right"><li style="margin-top: 10px;"> + + <div class="g-plusone" data-href="http://atlas.incubator.apache.org/atlas-docs"; data-size="medium" width="60px" align="right" ></div> + + </li></ul> + + + </div> + + </div> + </div> + </div> + + <div class="container"> + <div id="banner"> + <div class="pull-left"> + <a href=".." id="bannerLeft"> + <img src="images/atlas-logo.png" alt="Apache Atlas" width="200px" height="45px"/> + </a> + </div> + <div class="pull-right"> <a href="http://incubator.apache.org"; id="bannerRight"> + <img src="images/apache-incubator-logo.png" alt="Apache Incubator"/> + </a> + </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="http://www.apache.org"; class="externalLink" title="Apache"> + Apache</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="index.html" title="Atlas"> + Atlas</a> + </li> + <li class="divider ">/</li> + <li class="">Falcon Atlas Bridge</li> + + + + <li id="publishDate" class="pull-right">Last Published: 2016-01-05</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.7-incubating-SNAPSHOT</li> + + </ul> + </div> + + + + <div id="bodyColumn" > + + <div class="section"> +<h2><a name="Falcon_Atlas_Bridge"></a>Falcon Atlas Bridge</h2></div> +<div class="section"> +<h3><a name="Falcon_Model"></a>Falcon Model</h3> +<p>The default falcon modelling is available in org.apache.atlas.falcon.model.FalconDataModelGenerator. It defines the following types:</p> +<div class="source"> +<pre> +falcon_process(ClassType) - super types [Process] - attributes [timestamp, owned-by, tags] + +</pre></div> +<p>One falcon_process entity is created for every cluster that the falcon process is defined for.</p> +<p>The entities are created and de-duped using unique qualified name. They provide namespace and can be used for querying/lineage as well. The unique attributes are:</p> +<ul> +<li>falcon_process - attribute name - <process name>@<cluster name></li></ul></div> +<div class="section"> +<h3><a name="Falcon_Hook"></a>Falcon Hook</h3> +<p>Falcon supports listeners on falcon entity submission. This is used to add entities in Atlas using the model defined in org.apache.atlas.falcon.model.FalconDataModelGenerator. The hook submits the request to a thread pool executor to avoid blocking the command execution. The thread submits the entities as message to the notification server and atlas server reads these messages and registers the entities.</p> +<ul> +<li>Add 'org.apache.falcon.atlas.service.AtlasService' to application.services in <falcon-conf>/startup.properties</li> +<li>Link falcon hook jars in falcon classpath - 'ln -s <atlas-home>/hook/falcon/* <falcon-home>/server/webapp/falcon/WEB-INF/lib/'</li> +<li>Copy <atlas-conf>/client.properties and <atlas-conf>/atlas-application.properties to the falcon conf directory.</li></ul> +<p>The following properties in <atlas-conf>/client.properties control the thread pool and notification details:</p> +<ul> +<li>atlas.hook.falcon.synchronous - boolean, true to run the hook synchronously. default false</li> +<li>atlas.hook.falcon.numRetries - number of retries for notification failure. default 3</li> +<li>atlas.hook.falcon.minThreads - core number of threads. default 5</li> +<li>atlas.hook.falcon.maxThreads - maximum number of threads. default 5</li> +<li>atlas.hook.falcon.keepAliveTime - keep alive time in msecs. default 10</li> +<li>atlas.hook.falcon.queueSize - queue size for the threadpool. default 10000</li></ul> +<p>Refer <a href="./Configuration.html">Configuration</a> for notification related configurations</p></div> +<div class="section"> +<h3><a name="Limitations"></a>Limitations</h3> +<p></p> +<ul> +<li>Only the process entity creation is currently handled. This model will be expanded to include all Falcon metadata</li> +<li>In falcon cluster entity, cluster name used should be uniform across components like hive, falcon, sqoop etc. If used with ambari, ambari cluster name should be used for cluster entity</li></ul></div> + </div> + </div> + + <hr/> + + <footer> + <div class="container"> + <div class="row span12">Copyright © 2015-2016 + <a href="http://www.apache.org";>Apache Software Foundation</a>. + All Rights Reserved. + + </div> + + + <p id="poweredBy" class="pull-right"> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /> + </a> + </p> + + </div> + </footer> + </body> +</html> http://git-wip-us.apache.org/repos/asf/incubator-atlas-website/blob/02b20682/Bridge-Hive.html ---------------------------------------------------------------------- diff --git a/Bridge-Hive.html b/Bridge-Hive.html index 8067435..c725b1a 100644 --- a/Bridge-Hive.html +++ b/Bridge-Hive.html @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia at 2015-12-30 + | Generated by Apache Maven Doxia at 2016-01-05 | Rendered using Apache Maven Fluido Skin 1.3.0 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20151230" /> + <meta name="Date-Revision-yyyymmdd" content="20160105" /> <meta http-equiv="Content-Language" content="en" /> <title>Apache Atlas – Hive Atlas Bridge</title> <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" /> @@ -90,6 +90,9 @@ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Releases <b class="caret"></b></a> <ul class="dropdown-menu"> + <li> <a href="http://www.apache.org/dyn/closer.cgi/incubator/atlas/0.6.0-incubating/"; title="0.6-incubating">0.6-incubating</a> +</li> + <li> <a href="http://www.apache.org/dyn/closer.cgi/incubator/atlas/0.5.0-incubating/"; title="0.5-incubating">0.5-incubating</a> </li> </ul> @@ -98,6 +101,9 @@ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation <b class="caret"></b></a> <ul class="dropdown-menu"> + <li> <a href="0.6.0-incubating/index.html" title="0.6-incubating">0.6-incubating</a> +</li> + <li> <a href="0.5.0-incubating/index.html" title="0.5-incubating">0.5-incubating</a> </li> </ul> @@ -183,8 +189,8 @@ - <li id="publishDate" class="pull-right">Last Published: 2015-12-30</li> <li class="divider pull-right">|</li> - <li id="projectVersion" class="pull-right">Version: 0.6-incubating</li> + <li id="publishDate" class="pull-right">Last Published: 2016-01-05</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.7-incubating-SNAPSHOT</li> </ul> </div> @@ -234,7 +240,12 @@ hive_process(ClassType) - super types [Process] - attributes [startTime, endTime </property> </pre></div> -<p>Usage: <atlas package>/bin/import-hive.sh. The logs are in <atlas package>/logs/import-hive.log</p></div> +<p>Usage: <atlas package>/bin/import-hive.sh. The logs are in <atlas package>/logs/import-hive.log</p> +<p>If you you are importing metadata in a kerberized cluster you need to run the command like this:</p> +<p><atlas package>/bin/import-hive.sh -Dsun.security.jgss.debug=true -Djavax.security.auth.useSubjectCredsOnly=false -Djava.security.krb5.conf=[krb5.conf location] -Djava.security.auth.login.config=[jaas.conf location]</p> +<ul> +<li>krb5.conf is typically found at /etc/krb5.conf</li> +<li>for details about jaas.conf and a suggested location see the <a href="./Security.html">atlas security documentation</a></li></ul></div> <div class="section"> <h3><a name="Hive_Hook"></a>Hive Hook</h3> <p>Hive supports listeners on hive command execution using hive hooks. This is used to add/update/remove entities in Atlas using the model defined in org.apache.atlas.hive.model.HiveDataModelGenerator. The hook submits the request to a thread pool executor to avoid blocking the command execution. The thread submits the entities as message to the notification server and atlas server reads these messages and registers the entities. Follow these instructions in your hive set-up to add hive hook for Atlas:</p> @@ -259,7 +270,7 @@ hive_process(ClassType) - super types [Process] - attributes [startTime, endTime <p></p> <ul> <li>Add 'export HIVE_AUX_JARS_PATH=<atlas package>/hook/hive' in hive-env.sh of your hive configuration</li> -<li>Copy <atlas-conf>/client.properties and <atlas-conf>/application.properties to the hive conf directory.</li></ul> +<li>Copy <atlas-conf>/client.properties and <atlas-conf>/atlas-application.properties to the hive conf directory.</li></ul> <p>The following properties in <atlas-conf>/client.properties control the thread pool and notification details:</p> <ul> <li>atlas.hook.hive.synchronous - boolean, true to run the hook synchronously. default false</li> @@ -282,7 +293,7 @@ hive_process(ClassType) - super types [Process] - attributes [startTime, endTime <footer> <div class="container"> - <div class="row span12">Copyright © 2015 + <div class="row span12">Copyright © 2015-2016 <a href="http://www.apache.org";>Apache Software Foundation</a>. All Rights Reserved. http://git-wip-us.apache.org/repos/asf/incubator-atlas-website/blob/02b20682/Bridge-Sqoop.html ---------------------------------------------------------------------- diff --git a/Bridge-Sqoop.html b/Bridge-Sqoop.html new file mode 100644 index 0000000..9fa6414 --- /dev/null +++ b/Bridge-Sqoop.html @@ -0,0 +1,256 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2016-01-05 + | Rendered using Apache Maven Fluido Skin 1.3.0 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20160105" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Apache Atlas – Sqoop Atlas Bridge</title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + + + <script type="text/javascript" src="./js/apache-maven-fluido-1.3.0.min.js"></script> + + + +<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> + + </head> + <body class="topBarEnabled"> + + + + + + <div id="topbar" class="navbar navbar-fixed-top "> + <div class="navbar-inner"> + <div class="container" style="width: 68%;"><div class="nav-collapse"> + + + <ul class="nav"> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Atlas <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="index.html" title="About">About</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="Wiki">Wiki</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="News">News</a> +</li> + + <li> <a href="https://git-wip-us.apache.org/repos/asf/incubator-atlas.git"; title="Git">Git</a> +</li> + + <li> <a href="https://issues.apache.org/jira/browse/ATLAS"; title="Jira">Jira</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/ATLAS/PoweredBy"; title="Powered by">Powered by</a> +</li> + + <li> <a href="http://blogs.apache.org/atlas/"; title="Blog">Blog</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Project Information <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="project-info.html" title="Summary">Summary</a> +</li> + + <li> <a href="mail-lists.html" title="Mailing Lists">Mailing Lists</a> +</li> + + <li> <a href="http://webchat.freenode.net?channels=apacheatlas&uio=d4"; title="IRC">IRC</a> +</li> + + <li> <a href="team-list.html" title="Team">Team</a> +</li> + + <li> <a href="issue-tracking.html" title="Issue Tracking">Issue Tracking</a> +</li> + + <li> <a href="source-repository.html" title="Source Repository">Source Repository</a> +</li> + + <li> <a href="license.html" title="License">License</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Releases <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="http://www.apache.org/dyn/closer.cgi/incubator/atlas/0.6.0-incubating/"; title="0.6-incubating">0.6-incubating</a> +</li> + + <li> <a href="http://www.apache.org/dyn/closer.cgi/incubator/atlas/0.5.0-incubating/"; title="0.5-incubating">0.5-incubating</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="0.6.0-incubating/index.html" title="0.6-incubating">0.6-incubating</a> +</li> + + <li> <a href="0.5.0-incubating/index.html" title="0.5-incubating">0.5-incubating</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">ASF <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="http://www.apache.org/foundation/how-it-works.html"; title="How Apache Works">How Apache Works</a> +</li> + + <li> <a href="http://www.apache.org/foundation/"; title="Foundation">Foundation</a> +</li> + + <li> <a href="http://www.apache.org/foundation/sponsorship.html"; title="Sponsoring Apache">Sponsoring Apache</a> +</li> + + <li> <a href="http://www.apache.org/foundation/thanks.html"; title="Thanks">Thanks</a> +</li> + </ul> + </li> + </ul> + + <form id="search-form" action="http://www.google.com/search"; method="get" class="navbar-search pull-right" > + + <input value="http://atlas.incubator.apache.org"; name="sitesearch" type="hidden"/> + <input class="search-query" name="q" id="query" type="text" /> +</form> +<script type="text/javascript" src="http://www.google.com/coop/cse/brand?form=search-form";></script> + + + + + + <iframe src="http://www.facebook.com/plugins/like.php?href=http://atlas.incubator.apache.org/atlas-docs&send=false&layout=button_count&show-faces=false&action=like&colorscheme=dark"; + scrolling="no" frameborder="0" + style="border:none; width:80px; height:20px; margin-top: 10px;" class="pull-right" ></iframe> + + <script type="text/javascript" src="https://apis.google.com/js/plusone.js";></script> + + <ul class="nav pull-right"><li style="margin-top: 10px;"> + + <div class="g-plusone" data-href="http://atlas.incubator.apache.org/atlas-docs"; data-size="medium" width="60px" align="right" ></div> + + </li></ul> + + + </div> + + </div> + </div> + </div> + + <div class="container"> + <div id="banner"> + <div class="pull-left"> + <a href=".." id="bannerLeft"> + <img src="images/atlas-logo.png" alt="Apache Atlas" width="200px" height="45px"/> + </a> + </div> + <div class="pull-right"> <a href="http://incubator.apache.org"; id="bannerRight"> + <img src="images/apache-incubator-logo.png" alt="Apache Incubator"/> + </a> + </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="http://www.apache.org"; class="externalLink" title="Apache"> + Apache</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="index.html" title="Atlas"> + Atlas</a> + </li> + <li class="divider ">/</li> + <li class="">Sqoop Atlas Bridge</li> + + + + <li id="publishDate" class="pull-right">Last Published: 2016-01-05</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.7-incubating-SNAPSHOT</li> + + </ul> + </div> + + + + <div id="bodyColumn" > + + <div class="section"> +<h2><a name="Sqoop_Atlas_Bridge"></a>Sqoop Atlas Bridge</h2></div> +<div class="section"> +<h3><a name="Sqoop_Model"></a>Sqoop Model</h3> +<p>The default Sqoop modelling is available in org.apache.atlas.sqoop.model.SqoopDataModelGenerator. It defines the following types:</p> +<div class="source"> +<pre> +sqoop_operation_type(EnumType) - values [IMPORT, EXPORT, EVAL] +sqoop_dbstore_usage(EnumType) - values [TABLE, QUERY, PROCEDURE, OTHER] +sqoop_process(ClassType) - super types [Process] - attributes [name, operation, dbStore, hiveTable, commandlineOpts, startTime, endTime, userName] +sqoop_dbdatastore(ClassType) - super types [DataSet] - attributes [name, dbStoreType, storeUse, storeUri, source, description, ownerName] + +</pre></div> +<p>The entities are created and de-duped using unique qualified name. They provide namespace and can be used for querying as well: sqoop_process - attribute name - sqoop-dbStoreType-storeUri-endTime sqoop_dbdatastore - attribute name - dbStoreType-connectorUrl-source</p></div> +<div class="section"> +<h3><a name="Sqoop_Hook"></a>Sqoop Hook</h3> +<p>Sqoop added a <a href="./SqoopJobDataPublisher.html">SqoopJobDataPublisher</a> that publishes data to Atlas after completion of import Job. Today, only hiveImport is supported in sqoopHook. This is used to add entities in Atlas using the model defined in org.apache.atlas.sqoop.model.SqoopDataModelGenerator. Follow these instructions in your sqoop set-up to add sqoop hook for Atlas in <sqoop-conf>/sqoop-site.xml:</p> +<p></p> +<ul> +<li>Sqoop Job publisher class. Currently only one publishing class is supported</li></ul><property> <name>sqoop.job.data.publish.class</name> <value>org.apache.atlas.sqoop.hook.SqoopHook</value> </property> +<ul> +<li>Atlas cluster name</li></ul><property> <name>atlas.cluster.name</name> <value><clustername></value> </property> +<ul> +<li>Copy <atlas-conf>/atlas-application.properties and <atlas-conf>/client.properties to to the sqoop conf directory <sqoop-conf>/</li> +<li>Link <atlas-home>/hook/sqoop/*.jar in sqoop lib</li></ul> +<p>Refer <a href="./Configuration.html">Configuration</a> for notification related configurations</p></div> +<div class="section"> +<h3><a name="Limitations"></a>Limitations</h3> +<p></p> +<ul> +<li>Only the following sqoop operations are captured by sqoop hook currently - hiveImport</li></ul></div> + </div> + </div> + + <hr/> + + <footer> + <div class="container"> + <div class="row span12">Copyright © 2015-2016 + <a href="http://www.apache.org";>Apache Software Foundation</a>. + All Rights Reserved. + + </div> + + + <p id="poweredBy" class="pull-right"> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /> + </a> + </p> + + </div> + </footer> + </body> +</html> http://git-wip-us.apache.org/repos/asf/incubator-atlas-website/blob/02b20682/Configuration.html ---------------------------------------------------------------------- diff --git a/Configuration.html b/Configuration.html index 535bae4..eecc8e7 100644 --- a/Configuration.html +++ b/Configuration.html @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia at 2015-12-30 + | Generated by Apache Maven Doxia at 2016-01-05 | Rendered using Apache Maven Fluido Skin 1.3.0 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20151230" /> + <meta name="Date-Revision-yyyymmdd" content="20160105" /> <meta http-equiv="Content-Language" content="en" /> <title>Apache Atlas – Configuring Apache Atlas - Application Properties</title> <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" /> @@ -90,6 +90,9 @@ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Releases <b class="caret"></b></a> <ul class="dropdown-menu"> + <li> <a href="http://www.apache.org/dyn/closer.cgi/incubator/atlas/0.6.0-incubating/"; title="0.6-incubating">0.6-incubating</a> +</li> + <li> <a href="http://www.apache.org/dyn/closer.cgi/incubator/atlas/0.5.0-incubating/"; title="0.5-incubating">0.5-incubating</a> </li> </ul> @@ -98,6 +101,9 @@ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation <b class="caret"></b></a> <ul class="dropdown-menu"> + <li> <a href="0.6.0-incubating/index.html" title="0.6-incubating">0.6-incubating</a> +</li> + <li> <a href="0.5.0-incubating/index.html" title="0.5-incubating">0.5-incubating</a> </li> </ul> @@ -183,8 +189,8 @@ - <li id="publishDate" class="pull-right">Last Published: 2015-12-30</li> <li class="divider pull-right">|</li> - <li id="projectVersion" class="pull-right">Version: 0.6-incubating</li> + <li id="publishDate" class="pull-right">Last Published: 2016-01-05</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.7-incubating-SNAPSHOT</li> </ul> </div> @@ -195,7 +201,7 @@ <div class="section"> <h2><a name="Configuring_Apache_Atlas_-_Application_Properties"></a>Configuring Apache Atlas - Application Properties</h2> -<p>All configuration in Atlas uses java properties style configuration. The main configuration file is application.properties which is in the <b>conf</b> dir at the deployed location. It consists of the following sections:</p></div> +<p>All configuration in Atlas uses java properties style configuration. The main configuration file is atlas-application.properties which is in the <b>conf</b> dir at the deployed location. It consists of the following sections:</p></div> <div class="section"> <h3><a name="Graph_Configs"></a>Graph Configs</h3></div> <div class="section"> @@ -337,7 +343,7 @@ atlas.enableTLS=false <footer> <div class="container"> - <div class="row span12">Copyright © 2015 + <div class="row span12">Copyright © 2015-2016 <a href="http://www.apache.org";>Apache Software Foundation</a>. All Rights Reserved. http://git-wip-us.apache.org/repos/asf/incubator-atlas-website/blob/02b20682/HighAvailability.html ---------------------------------------------------------------------- diff --git a/HighAvailability.html b/HighAvailability.html index bca44aa..56cb376 100644 --- a/HighAvailability.html +++ b/HighAvailability.html @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia at 2015-12-30 + | Generated by Apache Maven Doxia at 2016-01-05 | Rendered using Apache Maven Fluido Skin 1.3.0 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20151230" /> + <meta name="Date-Revision-yyyymmdd" content="20160105" /> <meta http-equiv="Content-Language" content="en" /> <title>Apache Atlas – Fault Tolerance and High Availability Options</title> <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" /> @@ -90,6 +90,9 @@ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Releases <b class="caret"></b></a> <ul class="dropdown-menu"> + <li> <a href="http://www.apache.org/dyn/closer.cgi/incubator/atlas/0.6.0-incubating/"; title="0.6-incubating">0.6-incubating</a> +</li> + <li> <a href="http://www.apache.org/dyn/closer.cgi/incubator/atlas/0.5.0-incubating/"; title="0.5-incubating">0.5-incubating</a> </li> </ul> @@ -98,6 +101,9 @@ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation <b class="caret"></b></a> <ul class="dropdown-menu"> + <li> <a href="0.6.0-incubating/index.html" title="0.6-incubating">0.6-incubating</a> +</li> + <li> <a href="0.5.0-incubating/index.html" title="0.5-incubating">0.5-incubating</a> </li> </ul> @@ -183,8 +189,8 @@ - <li id="publishDate" class="pull-right">Last Published: 2015-12-30</li> <li class="divider pull-right">|</li> - <li id="projectVersion" class="pull-right">Version: 0.6-incubating</li> + <li id="publishDate" class="pull-right">Last Published: 2016-01-05</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.7-incubating-SNAPSHOT</li> </ul> </div> @@ -265,7 +271,7 @@ </pre></div> <p></p> <ul> -<li>In application.properties, set the following configuration:</li></ul> +<li>In atlas-application.properties, set the following configuration:</li></ul> <div class="source"> <pre> atlas.notification.embedded=false @@ -286,7 +292,7 @@ <footer> <div class="container"> - <div class="row span12">Copyright © 2015 + <div class="row span12">Copyright © 2015-2016 <a href="http://www.apache.org";>Apache Software Foundation</a>. All Rights Reserved. http://git-wip-us.apache.org/repos/asf/incubator-atlas-website/blob/02b20682/InstallationSteps.html ---------------------------------------------------------------------- diff --git a/InstallationSteps.html b/InstallationSteps.html index 6020b11..687708c 100644 --- a/InstallationSteps.html +++ b/InstallationSteps.html @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia at 2015-12-30 + | Generated by Apache Maven Doxia at 2016-01-05 | Rendered using Apache Maven Fluido Skin 1.3.0 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20151230" /> + <meta name="Date-Revision-yyyymmdd" content="20160105" /> <meta http-equiv="Content-Language" content="en" /> <title>Apache Atlas – Building & Installing Apache Atlas</title> <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" /> @@ -90,6 +90,9 @@ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Releases <b class="caret"></b></a> <ul class="dropdown-menu"> + <li> <a href="http://www.apache.org/dyn/closer.cgi/incubator/atlas/0.6.0-incubating/"; title="0.6-incubating">0.6-incubating</a> +</li> + <li> <a href="http://www.apache.org/dyn/closer.cgi/incubator/atlas/0.5.0-incubating/"; title="0.5-incubating">0.5-incubating</a> </li> </ul> @@ -98,6 +101,9 @@ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation <b class="caret"></b></a> <ul class="dropdown-menu"> + <li> <a href="0.6.0-incubating/index.html" title="0.6-incubating">0.6-incubating</a> +</li> + <li> <a href="0.5.0-incubating/index.html" title="0.5-incubating">0.5-incubating</a> </li> </ul> @@ -183,8 +189,8 @@ - <li id="publishDate" class="pull-right">Last Published: 2015-12-30</li> <li class="divider pull-right">|</li> - <li id="projectVersion" class="pull-right">Version: 0.6-incubating</li> + <li id="publishDate" class="pull-right">Last Published: 2016-01-05</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.7-incubating-SNAPSHOT</li> </ul> </div> @@ -226,7 +232,7 @@ mvn clean package -Pdist |- quick_start.py |- cputil.py |- conf - |- application.properties + |- atlas-application.properties |- client.properties |- atlas-env.sh |- log4j.xml @@ -257,11 +263,12 @@ mvn clean package -Pdist <div class="source"> <pre> tar -xzvf apache-atlas-${project.version}-bin.tar.gz -* cd atlas-${project.version} + +cd atlas-${project.version} </pre></div> <p><b>Configuring Atlas</b></p> -<p>By default config directory used by Atlas is {package dir}/conf. To override this set environment variable METADATA_CONF to the path of the conf dir.</p> +<p>By default config directory used by Atlas is {package dir}/conf. To override this set environment variable ATLAS_CONF to the path of the conf dir.</p> <p>atlas-env.sh has been added to the Atlas conf. This file can be used to set various environment variables that you need for you services. In addition you can set any other environment variables you might need. This file will be sourced by atlas scripts before any commands are executed. The following environment variables are available to set.</p> <div class="source"> <pre> @@ -269,73 +276,96 @@ tar -xzvf apache-atlas-${project.version}-bin.tar.gz #export JAVA_HOME= # any additional java opts you want to set. This will apply to both client and server operations -#export METADATA_OPTS= +#export ATLAS_OPTS= # any additional java opts that you want to set for client only -#export METADATA_CLIENT_OPTS= +#export ATLAS_CLIENT_OPTS= # java heap size we want to set for the client. Default is 1024MB -#export METADATA_CLIENT_HEAP= +#export ATLAS_CLIENT_HEAP= # any additional opts you want to set for atlas service. -#export METADATA_SERVER_OPTS= +#export ATLAS_SERVER_OPTS= # java heap size we want to set for the atlas server. Default is 1024MB -#export METADATA_SERVER_HEAP= +#export ATLAS_SERVER_HEAP= # What is is considered as atlas home dir. Default is the base locaion of the installed software -#export METADATA_HOME_DIR= +#export ATLAS_HOME_DIR= # Where log files are stored. Defatult is logs directory under the base install location -#export METADATA_LOG_DIR= +#export ATLAS_LOG_DIR= # Where pid files are stored. Defatult is logs directory under the base install location -#export METADATA_PID_DIR= +#export ATLAS_PID_DIR= # where the atlas titan db data is stored. Defatult is logs/data directory under the base install location -#export METADATA_DATA_DIR= +#export ATLAS_DATA_DIR= # Where do you want to expand the war file. By Default it is in /server/webapp dir under the base install dir. -#export METADATA_EXPANDED_WEBAPP_DIR= +#export ATLAS_EXPANDED_WEBAPP_DIR= </pre></div> -<p><b>NOTE for Mac OS users</b></p> +<p><b>NOTE for Mac OS users</b> If you are using a Mac OS, you will need to configure the ATLAS_SERVER_OPTS (explained above).</p> +<p>In {package dir}/conf/atlas-env.sh uncomment the following line</p> <div class="source"> <pre> -If you are using a Mac OS, you will need to configure the METADATA_SERVER_OPTS (explained above). - -In {package dir}/conf/atlas-env.sh uncomment the following line -#export METADATA_SERVER_OPTS= +#export ATLAS_SERVER_OPTS= -and change it to look as below -export METADATA_SERVER_OPTS="-Djava.awt.headless=true -Djava.security.krb5.realm= -Djava.security.krb5.kdc=" +</pre></div> +<p>and change it to look as below</p> +<div class="source"> +<pre> +export ATLAS_SERVER_OPTS="-Djava.awt.headless=true -Djava.security.krb5.realm= -Djava.security.krb5.kdc=" </pre></div> -<p>* Hbase as the Storage Backend for the Graph Repository</p> -<p>By default, Atlas uses Titan as the graph repository and is the only graph repository implementation available currently. The HBase versions currently supported are 1.1.x. For configuring ATLAS graph persistence on HBase, please go through the "Configuration - Graph persistence engine - HBase" section for more details.</p> -<p>Pre-requisites for running HBase as a distributed cluster * 3 or 5 <a href="./ZooKeeper.html">ZooKeeper</a> nodes * Atleast 3 <a href="./RegionServer.html">RegionServer</a> nodes. It would be ideal to run the <a href="./DataNodes.html">DataNodes</a> on the same hosts as the Region servers for data locality.</p> -<p>* Configuring SOLR as the Indexing Backend for the Graph Repository</p> +<p><b>Hbase as the Storage Backend for the Graph Repository</b></p> +<p>By default, Atlas uses Titan as the graph repository and is the only graph repository implementation available currently. The HBase versions currently supported are 1.1.x. For configuring ATLAS graph persistence on HBase, please see "Graph persistence engine - HBase" in the <a href="./Configuration.html">Configuration</a> section for more details.</p> +<p>Pre-requisites for running HBase as a distributed cluster</p> +<ul> +<li>3 or 5 <a href="./ZooKeeper.html">ZooKeeper</a> nodes</li> +<li>Atleast 3 <a href="./RegionServer.html">RegionServer</a> nodes. It would be ideal to run the <a href="./DataNodes.html">DataNodes</a> on the same hosts as the Region servers for data locality.</li></ul> +<p><b>Configuring SOLR as the Indexing Backend for the Graph Repository</b></p> <p>By default, Atlas uses Titan as the graph repository and is the only graph repository implementation available currently. For configuring Titan to work with Solr, please follow the instructions below</p> +<p></p> +<ul> +<li>Install solr if not already running. The version of SOLR supported is 5.2.1. Could be installed from <a class="externalLink" href="http://archive.apache.org/dist/lucene/solr/5.2.1/solr-5.2.1.tgz";>http://archive.apache.org/dist/lucene/solr/5.2.1/solr-5.2.1.tgz</a></li></ul> +<p></p> +<ul> +<li>Start solr in cloud mode.</li></ul><a href="./SolrCloud.html">SolrCloud</a> mode uses a <a href="./ZooKeeper.html">ZooKeeper</a> Service as a highly available, central location for cluster management. For a small cluster, running with an existing <a href="./ZooKeeper.html">ZooKeeper</a> quorum should be fine. For larger clusters, you would want to run separate multiple <a href="./ZooKeeper.html">ZooKeeper</a> quorum with atleast 3 servers. Note: Atlas currently supports solr in "cloud" mode only. "http" mode is not supported. For more information, refer solr documentation - <a class="externalLink" href="https://cwiki.apache.org/confluence/display/solr/SolrCloud";>https://cwiki.apache.org/confluence/display/solr/SolrCloud</a> +<p></p> +<ul> +<li>For e.g., to bring up a Solr node listening on port 8983 on a machine, you can use the command:</li></ul> <div class="source"> <pre> -* Install solr if not already running. The version of SOLR supported is 5.2.1. Could be installed from http://archive.apache.org/dist/lucene/solr/5.2.1/solr-5.2.1.tgz - -* Start solr in cloud mode. - SolrCloud mode uses a ZooKeeper Service as a highly available, central location for cluster management. - For a small cluster, running with an existing ZooKeeper quorum should be fine. For larger clusters, you would want to run separate multiple ZooKeeper quorum with atleast 3 servers. - Note: Atlas currently supports solr in "cloud" mode only. "http" mode is not supported. For more information, refer solr documentation - https://cwiki.apache.org/confluence/display/solr/SolrCloud - -* For e.g., to bring up a Solr node listening on port 8983 on a machine, you can use the command: - <verbatim> $SOLR_HOME/bin/solr start -c -z <zookeeper_host:port> -p 8983 </pre></div> -<p>* Run the following commands from SOLR_HOME directory to create collections in Solr corresponding to the indexes that Atlas uses. In the case that the ATLAS and SOLR instance are on 2 different hosts, first copy the required configuration files from ATLAS_HOME/conf/solr on the ATLAS instance host to the Solr instance host. SOLR_CONF in the below mentioned commands refer to the directory where the solr configuration files have been copied to on Solr host:</p> -<p>bin/solr create -c vertex_index -d SOLR_CONF -shards #numShards -replicationFactor #replicationFactor bin/solr create -c edge_index -d SOLR_CONF -shards #numShards -replicationFactor #replicationFactor bin/solr create -c fulltext_index -d SOLR_CONF -shards #numShards -replicationFactor #replicationFactor</p> +<p></p> +<ul> +<li>Run the following commands from SOLR_HOME directory to create collections in Solr corresponding to the indexes that Atlas uses. In the case that the ATLAS and SOLR instance are on 2 different hosts,</li></ul>first copy the required configuration files from ATLAS_HOME/conf/solr on the ATLAS instance host to the Solr instance host. SOLR_CONF in the below mentioned commands refer to the directory where the solr configuration files have been copied to on Solr host: +<div class="source"> +<pre> + bin/solr create -c vertex_index -d SOLR_CONF -shards #numShards -replicationFactor #replicationFactor + bin/solr create -c edge_index -d SOLR_CONF -shards #numShards -replicationFactor #replicationFactor + bin/solr create -c fulltext_index -d SOLR_CONF -shards #numShards -replicationFactor #replicationFactor + +</pre></div> <p>Note: If numShards and replicationFactor are not specified, they default to 1 which suffices if you are trying out solr with ATLAS on a single node instance. Otherwise specify numShards according to the number of hosts that are in the Solr cluster and the maxShardsPerNode configuration. The number of shards cannot exceed the total number of Solr nodes in your SolrCloud cluster.</p> <p>The number of replicas (replicationFactor) can be set according to the redundancy required.</p> -<p>* Change ATLAS configuration to point to the Solr instance setup. Please make sure the following configurations are set to the below values in ATLAS_HOME//conf/application.properties atlas.graph.index.search.backend=solr5 atlas.graph.index.search.solr.mode=cloud atlas.graph.index.search.solr.zookeeper-url=<the ZK quorum setup for solr as comma separated value> eg: 10.1.6.4:2181,10.1.6.5:2181</p> -<p>* Restart Atlas </verbatim></p> +<p></p> +<ul> +<li>Change ATLAS configuration to point to the Solr instance setup. Please make sure the following configurations are set to the below values in ATLAS_HOME/conf/atlas-application.properties</li></ul> +<div class="source"> +<pre> + atlas.graph.index.search.backend=solr5 + atlas.graph.index.search.solr.mode=cloud + atlas.graph.index.search.solr.zookeeper-url=<the ZK quorum setup for solr as comma separated value> eg: 10.1.6.4:2181,10.1.6.5:2181 + +</pre></div> +<p></p> +<ul> +<li>Restart Atlas</li></ul> <p>For more information on Titan solr configuration , please refer <a class="externalLink" href="http://s3.thinkaurelius.com/docs/titan/0.5.4/solr.htm";>http://s3.thinkaurelius.com/docs/titan/0.5.4/solr.htm</a></p> <p>Pre-requisites for running Solr in cloud mode * Memory - Solr is both memory and CPU intensive. Make sure the server running Solr has adequate memory, CPU and disk. Solr works well with 32GB RAM. Plan to provide as much memory as possible to Solr process * Disk - If the number of entities that need to be stored are large, plan to have at least 500 GB free space in the volume where Solr is going to store the index data * <a href="./SolrCloud.html">SolrCloud</a> has support for replication and sharding. It is highly recommended to use <a href="./SolrCloud.html">SolrCloud</a> with at least two Solr nodes running on different servers with replication enabled. If using <a href="./SolrCloud.html">SolrCloud</a>, then you also need <a href="./ZooKeeper.html">ZooKeeper</a> installed and configured with 3 or 5 <a href="./ZooKeeper.html">ZooKeeper</a> nodes</p> <p><b>Starting Atlas Server</b></p> @@ -344,33 +374,57 @@ export METADATA_SERVER_OPTS="-Djava.awt.headless=true -Djava.security.krb5. bin/atlas_start.py [-port <port>] </pre></div> -<p>By default, * To change the port, use -port option. * atlas server starts with conf from {package dir}/conf. To override this (to use the same conf with multiple atlas upgrades), set environment variable METADATA_CONF to the path of conf dir</p> +<p>By default,</p> +<ul> +<li>To change the port, use -port option.</li> +<li>atlas server starts with conf from {package dir}/conf. To override this (to use the same conf with multiple atlas upgrades), set environment variable ATLAS_CONF to the path of conf dir</li></ul> <p><b>Using Atlas</b></p> +<ul> +<li>Quick start model - sample model and data</li></ul> <div class="source"> <pre> -* Quick start model - sample model and data bin/quick_start.py [<atlas endpoint>] -* Verify if the server is up and running +</pre></div> +<p></p> +<ul> +<li>Verify if the server is up and running</li></ul> +<div class="source"> +<pre> curl -v http://localhost:21000/api/atlas/admin/version {"Version":"v0.1"} -* List the types in the repository +</pre></div> +<p></p> +<ul> +<li>List the types in the repository</li></ul> +<div class="source"> +<pre> curl -v http://localhost:21000/api/atlas/types {"results":["Process","Infrastructure","DataSet"],"count":3,"requestId":"1867493731@qtp-262860041-0 - 82d43a27-7c34-4573-85d1-a01525705091"} -* List the instances for a given type +</pre></div> +<p></p> +<ul> +<li>List the instances for a given type</li></ul> +<div class="source"> +<pre> curl -v http://localhost:21000/api/atlas/entities?type=hive_table {"requestId":"788558007@qtp-44808654-5","list":["cb9b5513-c672-42cb-8477-b8f3e537a162","ec985719-a794-4c98-b98f-0509bd23aac0","48998f81-f1d3-45a2-989a-223af5c1ed6e","a54b386e-c759-4651-8779-a099294244c4"]} curl -v http://localhost:21000/api/atlas/entities/list/hive_db -* Search for entities (instances) in the repository +</pre></div> +<p></p> +<ul> +<li>Search for entities (instances) in the repository</li></ul> +<div class="source"> +<pre> curl -v http://localhost:21000/api/atlas/discovery/search/dsl?query="from hive_table" </pre></div> <p><b>Dashboard</b></p> -<p>Once atlas is started, you can view the status of atlas entities using the Web-based dashboard. \You can open your browser at the corresponding port to use the web UI.</p> +<p>Once atlas is started, you can view the status of atlas entities using the Web-based dashboard. You can open your browser at the corresponding port to use the web UI.</p> <p><b>Stopping Atlas Server</b></p> <div class="source"> <pre> @@ -384,7 +438,7 @@ bin/atlas_stop.py <footer> <div class="container"> - <div class="row span12">Copyright © 2015 + <div class="row span12">Copyright © 2015-2016 <a href="http://www.apache.org";>Apache Software Foundation</a>. All Rights Reserved. http://git-wip-us.apache.org/repos/asf/incubator-atlas-website/blob/02b20682/Notification-Entity.html ---------------------------------------------------------------------- diff --git a/Notification-Entity.html b/Notification-Entity.html index ca427f6..184fd90 100644 --- a/Notification-Entity.html +++ b/Notification-Entity.html @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia at 2015-12-30 + | Generated by Apache Maven Doxia at 2016-01-05 | Rendered using Apache Maven Fluido Skin 1.3.0 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20151230" /> + <meta name="Date-Revision-yyyymmdd" content="20160105" /> <meta http-equiv="Content-Language" content="en" /> <title>Apache Atlas – Entity Change Notifications</title> <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" /> @@ -90,6 +90,9 @@ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Releases <b class="caret"></b></a> <ul class="dropdown-menu"> + <li> <a href="http://www.apache.org/dyn/closer.cgi/incubator/atlas/0.6.0-incubating/"; title="0.6-incubating">0.6-incubating</a> +</li> + <li> <a href="http://www.apache.org/dyn/closer.cgi/incubator/atlas/0.5.0-incubating/"; title="0.5-incubating">0.5-incubating</a> </li> </ul> @@ -98,6 +101,9 @@ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation <b class="caret"></b></a> <ul class="dropdown-menu"> + <li> <a href="0.6.0-incubating/index.html" title="0.6-incubating">0.6-incubating</a> +</li> + <li> <a href="0.5.0-incubating/index.html" title="0.5-incubating">0.5-incubating</a> </li> </ul> @@ -183,8 +189,8 @@ - <li id="publishDate" class="pull-right">Last Published: 2015-12-30</li> <li class="divider pull-right">|</li> - <li id="projectVersion" class="pull-right">Version: 0.6-incubating</li> + <li id="publishDate" class="pull-right">Last Published: 2016-01-05</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.7-incubating-SNAPSHOT</li> </ul> </div> @@ -232,7 +238,7 @@ <footer> <div class="container"> - <div class="row span12">Copyright © 2015 + <div class="row span12">Copyright © 2015-2016 <a href="http://www.apache.org";>Apache Software Foundation</a>. All Rights Reserved. http://git-wip-us.apache.org/repos/asf/incubator-atlas-website/blob/02b20682/QuickStart.html ---------------------------------------------------------------------- diff --git a/QuickStart.html b/QuickStart.html index a4664e5..2bb35b0 100644 --- a/QuickStart.html +++ b/QuickStart.html @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia at 2015-12-30 + | Generated by Apache Maven Doxia at 2016-01-05 | Rendered using Apache Maven Fluido Skin 1.3.0 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20151230" /> + <meta name="Date-Revision-yyyymmdd" content="20160105" /> <meta http-equiv="Content-Language" content="en" /> <title>Apache Atlas – Quick Start Guide</title> <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" /> @@ -90,6 +90,9 @@ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Releases <b class="caret"></b></a> <ul class="dropdown-menu"> + <li> <a href="http://www.apache.org/dyn/closer.cgi/incubator/atlas/0.6.0-incubating/"; title="0.6-incubating">0.6-incubating</a> +</li> + <li> <a href="http://www.apache.org/dyn/closer.cgi/incubator/atlas/0.5.0-incubating/"; title="0.5-incubating">0.5-incubating</a> </li> </ul> @@ -98,6 +101,9 @@ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation <b class="caret"></b></a> <ul class="dropdown-menu"> + <li> <a href="0.6.0-incubating/index.html" title="0.6-incubating">0.6-incubating</a> +</li> + <li> <a href="0.5.0-incubating/index.html" title="0.5-incubating">0.5-incubating</a> </li> </ul> @@ -183,8 +189,8 @@ - <li id="publishDate" class="pull-right">Last Published: 2015-12-30</li> <li class="divider pull-right">|</li> - <li id="projectVersion" class="pull-right">Version: 0.6-incubating</li> + <li id="publishDate" class="pull-right">Last Published: 2016-01-05</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.7-incubating-SNAPSHOT</li> </ul> </div> @@ -223,7 +229,7 @@ <footer> <div class="container"> - <div class="row span12">Copyright © 2015 + <div class="row span12">Copyright © 2015-2016 <a href="http://www.apache.org";>Apache Software Foundation</a>. All Rights Reserved. http://git-wip-us.apache.org/repos/asf/incubator-atlas-website/blob/02b20682/Repository.html ---------------------------------------------------------------------- diff --git a/Repository.html b/Repository.html index 20c0d33..ef67868 100644 --- a/Repository.html +++ b/Repository.html @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia at 2015-12-30 + | Generated by Apache Maven Doxia at 2016-01-05 | Rendered using Apache Maven Fluido Skin 1.3.0 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20151230" /> + <meta name="Date-Revision-yyyymmdd" content="20160105" /> <meta http-equiv="Content-Language" content="en" /> <title>Apache Atlas – Repository</title> <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" /> @@ -90,6 +90,9 @@ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Releases <b class="caret"></b></a> <ul class="dropdown-menu"> + <li> <a href="http://www.apache.org/dyn/closer.cgi/incubator/atlas/0.6.0-incubating/"; title="0.6-incubating">0.6-incubating</a> +</li> + <li> <a href="http://www.apache.org/dyn/closer.cgi/incubator/atlas/0.5.0-incubating/"; title="0.5-incubating">0.5-incubating</a> </li> </ul> @@ -98,6 +101,9 @@ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation <b class="caret"></b></a> <ul class="dropdown-menu"> + <li> <a href="0.6.0-incubating/index.html" title="0.6-incubating">0.6-incubating</a> +</li> + <li> <a href="0.5.0-incubating/index.html" title="0.5-incubating">0.5-incubating</a> </li> </ul> @@ -183,8 +189,8 @@ - <li id="publishDate" class="pull-right">Last Published: 2015-12-30</li> <li class="divider pull-right">|</li> - <li id="projectVersion" class="pull-right">Version: 0.6-incubating</li> + <li id="publishDate" class="pull-right">Last Published: 2016-01-05</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.7-incubating-SNAPSHOT</li> </ul> </div> @@ -204,7 +210,7 @@ <footer> <div class="container"> - <div class="row span12">Copyright © 2015 + <div class="row span12">Copyright © 2015-2016 <a href="http://www.apache.org";>Apache Software Foundation</a>. All Rights Reserved. http://git-wip-us.apache.org/repos/asf/incubator-atlas-website/blob/02b20682/Search.html ---------------------------------------------------------------------- diff --git a/Search.html b/Search.html index 7a8d042..8f89001 100644 --- a/Search.html +++ b/Search.html @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia at 2015-12-30 + | Generated by Apache Maven Doxia at 2016-01-05 | Rendered using Apache Maven Fluido Skin 1.3.0 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20151230" /> + <meta name="Date-Revision-yyyymmdd" content="20160105" /> <meta http-equiv="Content-Language" content="en" /> <title>Apache Atlas – Search</title> <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" /> @@ -90,6 +90,9 @@ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Releases <b class="caret"></b></a> <ul class="dropdown-menu"> + <li> <a href="http://www.apache.org/dyn/closer.cgi/incubator/atlas/0.6.0-incubating/"; title="0.6-incubating">0.6-incubating</a> +</li> + <li> <a href="http://www.apache.org/dyn/closer.cgi/incubator/atlas/0.5.0-incubating/"; title="0.5-incubating">0.5-incubating</a> </li> </ul> @@ -98,6 +101,9 @@ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation <b class="caret"></b></a> <ul class="dropdown-menu"> + <li> <a href="0.6.0-incubating/index.html" title="0.6-incubating">0.6-incubating</a> +</li> + <li> <a href="0.5.0-incubating/index.html" title="0.5-incubating">0.5-incubating</a> </li> </ul> @@ -183,8 +189,8 @@ - <li id="publishDate" class="pull-right">Last Published: 2015-12-30</li> <li class="divider pull-right">|</li> - <li id="projectVersion" class="pull-right">Version: 0.6-incubating</li> + <li id="publishDate" class="pull-right">Last Published: 2016-01-05</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.7-incubating-SNAPSHOT</li> </ul> </div> @@ -265,12 +271,13 @@ literal: booleanConstant | <ul> <li>The transitive closure of an Entity relationship can be expressed via the <i>Loop</i> expression. A</li></ul><i>Loop</i> expression can be any traversal (recursively a query) that represents a <i>Path</i> that ends in an Entity of the same <i>Type</i> as the starting Entity. <ul> -<li>The <i><a href="./WithPath.html">WithPath</a></i> clause can be used with transitive closure queries to retrieve the Path that</li></ul>connects the two related Entities. (We also provide a higher level interface for Closure Queries see scaladoc for 'org.apache.metadata.query.ClosureQuery') +<li>The <i><a href="./WithPath.html">WithPath</a></i> clause can be used with transitive closure queries to retrieve the Path that</li></ul>connects the two related Entities. (We also provide a higher level interface for Closure Queries see scaladoc for 'org.apache.atlas.query.ClosureQuery') <ul> <li>There are couple of Predicate functions different from SQL: <ul> <li><i>is</i> or <i>isa</i>can be used to filter Entities that have a particular Trait.</li> -<li><i>has</i> can be used to filter Entities that have a value for a particular Attribute.</li></ul></li></ul></div> +<li><i>has</i> can be used to filter Entities that have a value for a particular Attribute.</li></ul></li> +<li>When querying for a space delimited multiple-word identifier, it need to be enclosed within</li></ul>backquote (`)</div> <div class="section"> <h4><a name="DSL_Examples"></a>DSL Examples</h4> <p></p> @@ -281,7 +288,8 @@ literal: booleanConstant | <li>DB is <a href="./JdbcAccess.html">JdbcAccess</a></li> <li>Column where Column isa PII</li> <li>Table where name="sales_fact", columns</li> -<li>Table where name="sales_fact", columns as column select column.name, column.dataType, column.comment</li></ul></div> +<li>Table where name="sales_fact", columns as column select column.name, column.dataType, column.comment</li> +<li>`Log Data`</li></ul></div> <div class="section"> <h3><a name="Full-text_Search"></a>Full-text Search</h3> <p>Atlas also exposes a lucene style full-text search capability.</p></div> @@ -292,7 +300,7 @@ literal: booleanConstant | <footer> <div class="container"> - <div class="row span12">Copyright © 2015 + <div class="row span12">Copyright © 2015-2016 <a href="http://www.apache.org";>Apache Software Foundation</a>. All Rights Reserved. http://git-wip-us.apache.org/repos/asf/incubator-atlas-website/blob/02b20682/Security.html ---------------------------------------------------------------------- diff --git a/Security.html b/Security.html index 0ba9472..8af726c 100644 --- a/Security.html +++ b/Security.html @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia at 2015-12-30 + | Generated by Apache Maven Doxia at 2016-01-05 | Rendered using Apache Maven Fluido Skin 1.3.0 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20151230" /> + <meta name="Date-Revision-yyyymmdd" content="20160105" /> <meta http-equiv="Content-Language" content="en" /> <title>Apache Atlas – Security Features of Apache Atlas</title> <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" /> @@ -90,6 +90,9 @@ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Releases <b class="caret"></b></a> <ul class="dropdown-menu"> + <li> <a href="http://www.apache.org/dyn/closer.cgi/incubator/atlas/0.6.0-incubating/"; title="0.6-incubating">0.6-incubating</a> +</li> + <li> <a href="http://www.apache.org/dyn/closer.cgi/incubator/atlas/0.5.0-incubating/"; title="0.5-incubating">0.5-incubating</a> </li> </ul> @@ -98,6 +101,9 @@ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation <b class="caret"></b></a> <ul class="dropdown-menu"> + <li> <a href="0.6.0-incubating/index.html" title="0.6-incubating">0.6-incubating</a> +</li> + <li> <a href="0.5.0-incubating/index.html" title="0.5-incubating">0.5-incubating</a> </li> </ul> @@ -183,8 +189,8 @@ - <li id="publishDate" class="pull-right">Last Published: 2015-12-30</li> <li class="divider pull-right">|</li> - <li id="projectVersion" class="pull-right">Version: 0.6-incubating</li> + <li id="publishDate" class="pull-right">Last Published: 2016-01-05</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.7-incubating-SNAPSHOT</li> </ul> </div> @@ -204,7 +210,7 @@ <li>SPNEGO-based HTTP Authentication</li></ul></div> <div class="section"> <h4><a name="SSL"></a>SSL</h4> -<p>Both SSL one-way (server authentication) and two-way (server and client authentication) are supported. The following application properties (properties configured in the application.properties file) are available for configuring SSL:</p> +<p>Both SSL one-way (server authentication) and two-way (server and client authentication) are supported. The following application properties (properties configured in the atlas-application.properties file) are available for configuring SSL:</p> <p></p> <ul> <li><code>atlas.enableTLS</code> (false|true) [default: false] - enable/disable the SSL listener</li> @@ -256,7 +262,7 @@ </pre></div> <p></p> <ul> -<li>Update Atlas METADATA_OPTS to include ‘java.security.auth.login.config’ set to the above Atlas JAAS configuration file. +<li>Update Atlas ATLAS_OPTS to include ‘java.security.auth.login.config’ set to the above Atlas JAAS configuration file. <ul> <li>For example, <code>-Djava.security.auth.login.config=/etc/atlas/conf/atlas-jaas.conf</code></li></ul></li></ul></div> <div class="section"> @@ -289,8 +295,131 @@ <p>The property required for authenticating to the server (if authentication is enabled):</p> <p></p> <ul> -<li><code>atlas.http.authentication.type</code> (simple|kerberos) [default: simple] - the authentication type</li></ul> -<p>If the authentication type specified is 'kerberos', then the kerberos ticket cache will be accessed for authenticating to the server (Therefore the client is required to authenticate to the KDC prior to communication with the server using 'kinit' or a similar mechanism).</p></div> +<li><code>atlas.http.authentication.type</code> (simple|kerberos) [default: simple] - the authentication type</li></ul></div> +<div class="section"> +<h4><a name="SOLR_Kerberos_configuration"></a>SOLR Kerberos configuration</h4> +<p>If the authentication type specified is 'kerberos', then the kerberos ticket cache will be accessed for authenticating to the server (Therefore the client is required to authenticate to the KDC prior to communication with the server using 'kinit' or a similar mechanism).</p> +<p>See <a class="externalLink" href="https://cwiki.apache.org/confluence/display/RANGER/How+to+configure+Solr+Cloud+with+Kerberos+for+Ranger+0.5";>the Apache SOLR Kerberos configuration</a>.</p> +<p></p> +<ul> +<li>Add principal and generate the keytab file for solr. Create a keytab per host for each host where Solr is going to run and use the principal name with the host (e.g. addprinc -randkey solr/${HOST1}@EXAMPLE.COM. Replace ${HOST1} with the actual host names).</li></ul> +<div class="source"> +<pre> + kadmin.local + kadmin.local: addprinc -randkey solr/<hostname>@EXAMPLE.COM + kadmin.local: xst -k solr.keytab solr/<hostname>@EXAMPLE.COM + kadmin.local: quit + +</pre></div> +<p></p> +<ul> +<li>Add principal and generate the keytab file for authenticating HTTP request. (Note that if Ambari is used to Kerberize the cluster, the keytab /etc/security/keytabs/spnego.service.keytab can be used)</li></ul> +<div class="source"> +<pre> + kadmin.local + kadmin.local: addprinc -randkey HTTP/<hostname>@EXAMPLE.COM + kadmin.local: xst -k HTTP.keytab HTTP/<hostname>@EXAMPLE.COM + kadmin.local: quit + +</pre></div> +<p></p> +<ul> +<li>Copy the keytab file to all the hosts running Solr.</li></ul> +<div class="source"> +<pre> + cp solr.keytab /etc/security/keytabs/ + chmod 400 /etc/security/keytabs/solr.keytab + + cp HTTP.keytab /etc/security/keytabs/ + chmod 400 /etc/security/keytabs/HTTP.keytab + +</pre></div> +<p></p> +<ul> +<li>Create path in Zookeeper for storing the Solr configs and other parameters.</li></ul> +<div class="source"> +<pre> + $SOLR_INSTALL_HOME/server/scripts/cloud-scripts/zkcli.sh -zkhost $ZK_HOST:2181 -cmd makepath solr + +</pre></div> +<p></p> +<ul> +<li>Upload the configuration to Zookeeper.</li></ul> +<div class="source"> +<pre> + $SOLR_INSTALL_HOME/server/scripts/cloud-scripts/zkcli.sh -cmd upconfig -zkhost $ZK_HOST:2181/solr -confname basic_configs -confdir $SOLR_INSTALL_HOME/server/solr/configsets/basic_configs/conf + +</pre></div> +<p></p> +<ul> +<li>Create the JAAS configuration.</li></ul> +<div class="source"> +<pre> + vi /etc/solr/conf/solr_jaas.conf + + Client { + com.sun.security.auth.module.Krb5LoginModule required + useKeyTab=true + keyTab="/etc/security/keytabs/solr.keytab" + storeKey=true + useTicketCache=true + debug=true + principal="solr/<hostname>@EXAMPLE.COM"; + }; + +</pre></div> +<p></p> +<ul> +<li>Copy /etc/solr/conf/solr_jaas.conf to all hosts running Solr.</li></ul> +<p></p> +<ul> +<li>Edit solr.in.sh in $SOLR_INSTALL_HOME/bin/</li></ul> +<div class="source"> +<pre> + vi $SOLR_INSTALL_HOME/bin/solr.in.sh + + SOLR_JAAS_FILE=/etc/solr/conf/solr_jaas.conf + SOLR_HOST=`hostname -f` + ZK_HOST="$ZK_HOST1:2181,$ZK_HOST2:2181,$ZK_HOST3:2181/solr" + KERBEROS_REALM="EXAMPLE.COM" + SOLR_KEYTAB=/etc/solr/conf/solr.keytab + SOLR_KERB_PRINCIPAL=HTTP@${KERBEROS_REALM} + SOLR_KERB_KEYTAB=/etc/solr/conf/HTTP.keytab + SOLR_AUTHENTICATION_CLIENT_CONFIGURER="org.apache.solr.client.solrj.impl.Krb5HttpClientConfigurer" + SOLR_AUTHENTICATION_OPTS=" -DauthenticationPlugin=org.apache.solr.security.KerberosPlugin -Djava.security.auth.login.config=${SOLR_JAAS_FILE} -Dsolr.kerberos.principal=${SOLR_KERB_PRINCIPAL} -Dsolr.kerberos.keytab=${SOLR_KERB_KEYTAB} -Dsolr.kerberos.cookie.domain=${SOLR_HOST} -Dhost=${SOLR_HOST} -Dsolr.kerberos.name.rules=DEFAULT" + +</pre></div> +<p></p> +<ul> +<li>Copy solr.in.sh to all hosts running Solr.</li></ul> +<p></p> +<ul> +<li>Set up Solr to use the Kerberos plugin by uploading the security.json.</li></ul> +<div class="source"> +<pre> + $SOLR_INSTALL_HOME/server/scripts/cloud-scripts/zkcli.sh -zkhost <zk host>:2181 -cmd put /security.json '{"authentication":{"class": "org.apache.solr.security.KerberosPlugin"}}' + +</pre></div> +<p></p> +<ul> +<li>Start Solr.</li></ul> +<div class="source"> +<pre> + $SOLR_INSTALL_HOME/bin/solr start -cloud -z $ZK_HOST1:2181,$ZK_HOST2:2181,$ZK_HOST3:2181 -noprompt + +</pre></div> +<p></p> +<ul> +<li>Test Solr</li></ul> +<div class="source"> +<pre> + kinit -k -t /etc/security/keytabs/HTTP.keytab HTTP/<host>@EXAMPLE.COM + curl --negotiate -u : "http://<host>:8983/solr/" + +</pre></div> +<p></p> +<ul> +<li>Create collections in Solr corresponding to the indexes that Atlas uses and change the Atlas configuration to point to the Solr instance setup as described in the <a href="./InstallationSteps.html">Install Steps</a>.</li></ul></div> </div> </div> @@ -298,7 +427,7 @@ <footer> <div class="container"> - <div class="row span12">Copyright © 2015 + <div class="row span12">Copyright © 2015-2016 <a href="http://www.apache.org";>Apache Software Foundation</a>. All Rights Reserved. http://git-wip-us.apache.org/repos/asf/incubator-atlas-website/blob/02b20682/TypeSystem.html ---------------------------------------------------------------------- diff --git a/TypeSystem.html b/TypeSystem.html index d45e18b..de01779 100644 --- a/TypeSystem.html +++ b/TypeSystem.html @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia at 2015-12-30 + | Generated by Apache Maven Doxia at 2016-01-05 | Rendered using Apache Maven Fluido Skin 1.3.0 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20151230" /> + <meta name="Date-Revision-yyyymmdd" content="20160105" /> <meta http-equiv="Content-Language" content="en" /> <title>Apache Atlas – Type System</title> <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" /> @@ -90,6 +90,9 @@ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Releases <b class="caret"></b></a> <ul class="dropdown-menu"> + <li> <a href="http://www.apache.org/dyn/closer.cgi/incubator/atlas/0.6.0-incubating/"; title="0.6-incubating">0.6-incubating</a> +</li> + <li> <a href="http://www.apache.org/dyn/closer.cgi/incubator/atlas/0.5.0-incubating/"; title="0.5-incubating">0.5-incubating</a> </li> </ul> @@ -98,6 +101,9 @@ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation <b class="caret"></b></a> <ul class="dropdown-menu"> + <li> <a href="0.6.0-incubating/index.html" title="0.6-incubating">0.6-incubating</a> +</li> + <li> <a href="0.5.0-incubating/index.html" title="0.5-incubating">0.5-incubating</a> </li> </ul> @@ -183,8 +189,8 @@ - <li id="publishDate" class="pull-right">Last Published: 2015-12-30</li> <li class="divider pull-right">|</li> - <li id="projectVersion" class="pull-right">Version: 0.6-incubating</li> + <li id="publishDate" class="pull-right">Last Published: 2016-01-05</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.7-incubating-SNAPSHOT</li> </ul> </div> @@ -227,7 +233,7 @@ <footer> <div class="container"> - <div class="row span12">Copyright © 2015 + <div class="row span12">Copyright © 2015-2016 <a href="http://www.apache.org";>Apache Software Foundation</a>. All Rights Reserved.
