http://git-wip-us.apache.org/repos/asf/atlas-website/blob/e9346ee2/1.0.0-rc0/Import-API.html ---------------------------------------------------------------------- diff --git a/1.0.0-rc0/Import-API.html b/1.0.0-rc0/Import-API.html new file mode 100644 index 0000000..8e182d6 --- /dev/null +++ b/1.0.0-rc0/Import-API.html @@ -0,0 +1,293 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/twiki/Import-API.twiki at 2018-05-26 + | Rendered using Apache Maven Fluido Skin 1.7 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20180526" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Apache Atlas – Import API</title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.7.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + <script type="text/javascript" src="./js/apache-maven-fluido-1.7.min.js"></script> + </head> + <body class="topBarEnabled"> + <div id="topbar" class="navbar navbar-fixed-top "> + <div class="navbar-inner"> + <div class="container" style="width: 68%;"><div class="nav-collapse"> + <ul class="nav"> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Atlas <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="index.html" title="About">About</a></li> + <li><a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="Wiki">Wiki</a></li> + <li><a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="News">News</a></li> + <li><a href="https://git-wip-us.apache.org/repos/asf/atlas.git"; title="Git">Git</a></li> + <li><a href="https://issues.apache.org/jira/browse/ATLAS"; title="Jira">Jira</a></li> + <li><a href="https://cwiki.apache.org/confluence/display/ATLAS/PoweredBy"; title="Powered by">Powered by</a></li> + <li><a href="http://blogs.apache.org/atlas/"; title="Blog">Blog</a></li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Project Information <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="project-info.html" title="Summary">Summary</a></li> + <li><a href="mail-lists.html" title="Mailing Lists">Mailing Lists</a></li> + <li><a href="http://webchat.freenode.net?channels=apacheatlas&uio=d4"; title="IRC">IRC</a></li> + <li><a href="team-list.html" title="Team">Team</a></li> + <li><a href="issue-tracking.html" title="Issue Tracking">Issue Tracking</a></li> + <li><a href="source-repository.html" title="Source Repository">Source Repository</a></li> + <li><a href="license.html" title="License">License</a></li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Releases <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="http://www.apache.org/dyn/closer.cgi/atlas/1.0.0-alpha/"; title="1.0.0-alpha">1.0.0-alpha</a></li> + <li><a href="http://www.apache.org/dyn/closer.cgi/atlas/1.0.0-rc0/"; title="1.0.0-rc0">1.0.0-rc0</a></li> + <li><a href="http://www.apache.org/dyn/closer.cgi/atlas/0.8.2/"; title="0.8.2">0.8.2</a></li> + <li><a href="http://www.apache.org/dyn/closer.cgi/atlas/0.8.1/"; title="0.8.1">0.8.1</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.8.0-incubating/"; title="0.8-incubating">0.8-incubating</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.7.1-incubating/"; title="0.7.1-incubating">0.7.1-incubating</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.7.0-incubating/"; title="0.7-incubating">0.7-incubating</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.6.0-incubating/"; title="0.6-incubating">0.6-incubating</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.5.0-incubating/"; title="0.5-incubating">0.5-incubating</a></li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="../index.html" title="latest">latest</a></li> + <li><a href="../1.0.0-alpha/index.html" title="1.0.0-alpha">1.0.0-alpha</a></li> + <li><a href="../1.0.0-rc0/index.html" title="1.0.0-rc0">1.0.0-rc0</a></li> + <li><a href="../0.8.2/index.html" title="0.8.2">0.8.2</a></li> + <li><a href="../0.8.1/index.html" title="0.8.1">0.8.1</a></li> + <li><a href="../0.8.0-incubating/index.html" title="0.8-incubating">0.8-incubating</a></li> + <li><a href="../0.7.1-incubating/index.html" title="0.7.1-incubating">0.7.1-incubating</a></li> + <li><a href="../0.7.0-incubating/index.html" title="0.7-incubating">0.7-incubating</a></li> + <li><a href="../0.6.0-incubating/index.html" title="0.6-incubating">0.6-incubating</a></li> + <li><a href="../0.5.0-incubating/index.html" title="0.5-incubating">0.5-incubating</a></li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">ASF <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="http://www.apache.org/foundation/how-it-works.html"; title="How Apache Works">How Apache Works</a></li> + <li><a href="https://www.apache.org/events/current-event"; title="Events">Events</a></li> + <li><a href="https://www.apache.org/licenses/"; title="License">License</a></li> + <li><a href="http://www.apache.org/foundation/"; title="Foundation">Foundation</a></li> + <li><a href="http://www.apache.org/foundation/sponsorship.html"; title="Sponsoring Apache">Sponsoring Apache</a></li> + <li><a href="http://www.apache.org/foundation/thanks.html"; title="Thanks">Thanks</a></li> + </ul> + </li> + </ul> +<form id="search-form" action="https://www.google.com/search"; method="get" class="navbar-search pull-right" > + <input value="http://atlas.apache.org"; name="sitesearch" type="hidden"/> + <input class="search-query" name="q" id="query" type="text" /> +</form> +<script type="text/javascript">asyncJs( 'https://cse.google.com/brand?form=search-form' )</script> + <iframe src="https://www.facebook.com/plugins/like.php?href=http://atlas.apache.org/atlas-docs&send=false&layout=button_count&show-faces=false&action=like&colorscheme=dark"; + scrolling="no" frameborder="0" + style="border:none; width:100px; height:20px; margin-top: 10px;" class="pull-right" ></iframe> + <script type="text/javascript">asyncJs( 'https://apis.google.com/js/plusone.js' )</script> + <ul class="nav pull-right"><li style="margin-top: 10px;"> + <div class="g-plusone" data-href="http://atlas.apache.org/atlas-docs"; data-size="medium" width="60px" align="right" ></div> + </li></ul> + </div> + </div> + </div> + </div> + <div class="container"> + <div id="banner"> + <div class="pull-left"><a href=".." id="bannerLeft"><img src="images/atlas-logo.png" alt="Apache Atlas" width="200px" height="45px"/></a></div> + <div class="pull-right"></div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + <li class=""><a href="http://www.apache.org"; class="externalLink" title="Apache">Apache</a><span class="divider">/</span></li> + <li class=""><a href="index.html" title="Atlas">Atlas</a><span class="divider">/</span></li> + <li class="active ">Import API</li> + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-05-26</li> + <li id="projectVersion" class="pull-right">Version: 1.0.0-rc0</li> + </ul> + </div> + <div id="bodyColumn" > +<div class="section"> +<h2><a name="Import_API"></a>Import API</h2> +<p>The general approach is:</p> +<ul> +<li>Consumer makes a ZIP file available for import operation. See details below for the 2 flavors of the API.</li> +<li>The API if successful, will return the results of the operation.</li> +<li>Error will be returned on failure of the call.</li></ul></div> +<div class="section"> +<h4><a name="Import_ZIP_File_Using_POST"></a>Import ZIP File Using POST</h4> +<p></p> +<table border="0" class="table table-striped"> +<tr class="a"> +<th>Title</th> +<th>Import API</th></tr> +<tr class="b"> +<td><i>Example</i></td> +<td>See Examples sections below.</td></tr> +<tr class="a"> +<td><i>Description</i></td> +<td>Provide the contents of the file to be imported in the request body.</td></tr> +<tr class="b"> +<td><i>URL</i></td> +<td><i>api/atlas/admin/import</i></td></tr> +<tr class="a"> +<td><i>Method</i></td> +<td><i>POST</i></td></tr> +<tr class="b"> +<td><i>URL Parameters</i></td> +<td><i>None</i></td></tr> +<tr class="a"> +<td><i>Data Parameters</i></td> +<td><i>None</i></td></tr> +<tr class="b"> +<td><i>Success Response</i></td> +<td><i>AtlasImporResult</i> is returned as JSON. See details below.</td></tr> +<tr class="a"> +<td><i>Error Response</i></td> +<td>Errors that are handled within the system will be returned as <i>AtlasBaseException</i>.</td></tr></table></div> +<div class="section"> +<h4><a name="Import_ZIP_File_Available_on_Server"></a>Import ZIP File Available on Server</h4> +<p></p> +<table border="0" class="table table-striped"> +<tr class="a"> +<th>Title</th> +<th>Import API</th></tr> +<tr class="b"> +<td><i>Example</i></td> +<td>See Examples sections below.</td></tr> +<tr class="a"> +<td><i>Description</i></td> +<td>Provide the path of the file to be imported.</td></tr> +<tr class="b"> +<td><i>URL</i></td> +<td><i>api/atlas/admin/importfile</i></td></tr> +<tr class="a"> +<td><i>Method</i></td> +<td><i>POST</i></td></tr> +<tr class="b"> +<td><i>URL Parameters</i></td> +<td><i>None</i></td></tr> +<tr class="a"> +<td><i>Data Parameters</i></td> +<td><i>None</i></td></tr> +<tr class="b"> +<td><i>Success Response</i></td> +<td><i>AtlasImporResult</i> is returned as JSON. See details below.</td></tr> +<tr class="a"> +<td><i>Error Response</i></td> +<td>Errors that are handled within the system will be returned as <i>AtlasBaseException</i>.</td></tr> +<tr class="b"> +<td><i>Notes</i></td> +<td>The file to be imported needs to be present on the server at the location specified by the <i>FILENAME</i> parameter.</td></tr></table><b><i>Method Signature for Import</i></b> +<div class="source"><pre class="prettyprint"> +@POST +@Path("/import") +@Produces("application/json; charset=UTF-8") +@Consumes("multipart/form-data") + +</pre></div> +<p><b><i>Method Signature for Import File</i></b></p> +<div class="source"><pre class="prettyprint"> +@POST +@Path("/importfile") +@Produces("application/json; charset=UTF-8") +@Consumes("application/json") + +</pre></div> +<p><b><i>Import Options</i></b> Please see <b><i><a href="./Import-API-Options.html">here</a></i></b> for the available options during import process.</p> +<p><b><i>AtlasImportResult Response</i></b> The API will return the results of the import operation in the format defined by the <i>AtlasImportResult</i>:</p> +<ul> +<li><i>AtlasImportParameters</i>: This contains a collection of name value pair of the options that are applied during the import operation.</li> +<li><i>Metrics</i>: Operation metrics. These include details on the number of types imported, number of entities imported, etc.</li> +<li><i>Processed Entities</i>: Contains list of GUIDs for the entities that were processed.</li> +<li><i>Operation Status</i>: Overall status of the operation. Values are <i>SUCCESS</i>, PARTIAL_SUCCESS, <i>FAIL</i>.</li></ul></div> +<div class="section"> +<h4><a name="Examples_Using_CURL_Calls"></a>Examples Using CURL Calls</h4> +<p>The call below performs Import of <i>QuickStart</i> database using POST.</p> +<div class="source"><pre class="prettyprint"> +curl -g -X POST -u adminuser:password -H "Content-Type: multipart/form-data" + -H "Cache-Control: no-cache" + -F request=@importOptions.json + -F data=@quickStartDB.zip + "http://localhost:21000/api/atlas/admin/import" + +</pre></div> +<p>The <i>request</i> parameter is optional. If import has to be run without any options use:</p> +<div class="source"><pre class="prettyprint"> +curl -g -X POST -u adminuser:password -H "Content-Type: multipart/form-data" + -H "Cache-Control: no-cache" + -F data=@quickStartDB.zip + "http://localhost:21000/api/atlas/admin/import" + +</pre></div> +<p>The call below performs Import of <i>QuickStart</i> database using a ZIP file available on server.</p> +<div class="source"><pre class="prettyprint"> +curl -X POST -u adminuser:password -H "Cache-Control: no-cache" -d ./importOptions.json +"http://localhost:21000/api/atlas/admin/importFile" > quickStartDB-import-result.json + +</pre></div> +<p>Below is the <i>AtlasImportResult</i> JSON for an import that contains <i>hive</i>db_.</p> +<p>The <i>processedEntities</i> contains the <i>guids</i> of all the entities imported.</p> +<p>The <i>metrics</i> contain a breakdown of the types and entities imported along with the operation performed on them viz. <i>created</i> or <i>updated</i>.</p> +<div class="source"><pre class="prettyprint"> +{ + "request": { + "options": {} + }, + "userName": "admin", + "clientIpAddress": "10.0.2.2", + "hostName": "10.0.2.15", + "timeStamp": 1491285622823, + "metrics": { + "duration": 9143, + "typedef:enum": 0, + "typedef:struct": 0, + "entity:hive_column:created": 461, + "entity:hive_storagedesc:created": 20, + "entity:hive_process:created": 12, + "entity:hive_db:created": 5, + "entity:hive_table:created": 20, + "entity:hdfs_path:created": 2, + "typedef:entitydef": 0, + "typedef:classification": 3 + }, + "processedEntities": [ + "2c4aa713-030b-4fb3-98b1-1cab23d9ac81", + "e4aa71ed-70fd-4fa7-9dfb-8250a573e293", + + ... + + "ea0f9bdb-1dfc-4e48-9848-a006129929f9", + "b5e2cb41-3e7d-4468-84e1-d87c320e75f9" + ], + "operationStatus": "SUCCESS" +} + +</pre></div></div> + </div> + </div> + <hr/> + <footer> + <div class="container"> + <div class="row"> +<p><a href="https://www.apache.org/foundation/contributing";><img src="https://www.apache.org/images/SupportApache-small.png"; alt="Support the ASF" id="asf-logo" height="20" width="20" /></a>Copyright © 2011-2018 The Apache Software Foundation. Licensed under the <a href="https://www.apache.org/licenses/";>Apache License, Version 2.0</a>.<br/> +Apache Atlas, Atlas, Apache, the Apache feather logo are trademarks of the <a href="https://www.apache.org";>Apache Software Foundation</a>.<br/> +All other marks mentioned may be trademarks or registered trademarks of their respective owners.</p> + </div> + <p id="poweredBy" class="pull-right"><a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"><img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /></a> +</p> + </div> + </footer> + </body> +</html>
http://git-wip-us.apache.org/repos/asf/atlas-website/blob/e9346ee2/1.0.0-rc0/Import-Export-API.html ---------------------------------------------------------------------- diff --git a/1.0.0-rc0/Import-Export-API.html b/1.0.0-rc0/Import-Export-API.html new file mode 100644 index 0000000..6f2ce84 --- /dev/null +++ b/1.0.0-rc0/Import-Export-API.html @@ -0,0 +1,168 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/twiki/Import-Export-API.twiki at 2018-05-26 + | Rendered using Apache Maven Fluido Skin 1.7 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20180526" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Apache Atlas – Export & Import REST APIs</title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.7.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + <script type="text/javascript" src="./js/apache-maven-fluido-1.7.min.js"></script> + </head> + <body class="topBarEnabled"> + <div id="topbar" class="navbar navbar-fixed-top "> + <div class="navbar-inner"> + <div class="container" style="width: 68%;"><div class="nav-collapse"> + <ul class="nav"> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Atlas <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="index.html" title="About">About</a></li> + <li><a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="Wiki">Wiki</a></li> + <li><a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="News">News</a></li> + <li><a href="https://git-wip-us.apache.org/repos/asf/atlas.git"; title="Git">Git</a></li> + <li><a href="https://issues.apache.org/jira/browse/ATLAS"; title="Jira">Jira</a></li> + <li><a href="https://cwiki.apache.org/confluence/display/ATLAS/PoweredBy"; title="Powered by">Powered by</a></li> + <li><a href="http://blogs.apache.org/atlas/"; title="Blog">Blog</a></li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Project Information <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="project-info.html" title="Summary">Summary</a></li> + <li><a href="mail-lists.html" title="Mailing Lists">Mailing Lists</a></li> + <li><a href="http://webchat.freenode.net?channels=apacheatlas&uio=d4"; title="IRC">IRC</a></li> + <li><a href="team-list.html" title="Team">Team</a></li> + <li><a href="issue-tracking.html" title="Issue Tracking">Issue Tracking</a></li> + <li><a href="source-repository.html" title="Source Repository">Source Repository</a></li> + <li><a href="license.html" title="License">License</a></li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Releases <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="http://www.apache.org/dyn/closer.cgi/atlas/1.0.0-alpha/"; title="1.0.0-alpha">1.0.0-alpha</a></li> + <li><a href="http://www.apache.org/dyn/closer.cgi/atlas/1.0.0-rc0/"; title="1.0.0-rc0">1.0.0-rc0</a></li> + <li><a href="http://www.apache.org/dyn/closer.cgi/atlas/0.8.2/"; title="0.8.2">0.8.2</a></li> + <li><a href="http://www.apache.org/dyn/closer.cgi/atlas/0.8.1/"; title="0.8.1">0.8.1</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.8.0-incubating/"; title="0.8-incubating">0.8-incubating</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.7.1-incubating/"; title="0.7.1-incubating">0.7.1-incubating</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.7.0-incubating/"; title="0.7-incubating">0.7-incubating</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.6.0-incubating/"; title="0.6-incubating">0.6-incubating</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.5.0-incubating/"; title="0.5-incubating">0.5-incubating</a></li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="../index.html" title="latest">latest</a></li> + <li><a href="../1.0.0-alpha/index.html" title="1.0.0-alpha">1.0.0-alpha</a></li> + <li><a href="../1.0.0-rc0/index.html" title="1.0.0-rc0">1.0.0-rc0</a></li> + <li><a href="../0.8.2/index.html" title="0.8.2">0.8.2</a></li> + <li><a href="../0.8.1/index.html" title="0.8.1">0.8.1</a></li> + <li><a href="../0.8.0-incubating/index.html" title="0.8-incubating">0.8-incubating</a></li> + <li><a href="../0.7.1-incubating/index.html" title="0.7.1-incubating">0.7.1-incubating</a></li> + <li><a href="../0.7.0-incubating/index.html" title="0.7-incubating">0.7-incubating</a></li> + <li><a href="../0.6.0-incubating/index.html" title="0.6-incubating">0.6-incubating</a></li> + <li><a href="../0.5.0-incubating/index.html" title="0.5-incubating">0.5-incubating</a></li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">ASF <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="http://www.apache.org/foundation/how-it-works.html"; title="How Apache Works">How Apache Works</a></li> + <li><a href="https://www.apache.org/events/current-event"; title="Events">Events</a></li> + <li><a href="https://www.apache.org/licenses/"; title="License">License</a></li> + <li><a href="http://www.apache.org/foundation/"; title="Foundation">Foundation</a></li> + <li><a href="http://www.apache.org/foundation/sponsorship.html"; title="Sponsoring Apache">Sponsoring Apache</a></li> + <li><a href="http://www.apache.org/foundation/thanks.html"; title="Thanks">Thanks</a></li> + </ul> + </li> + </ul> +<form id="search-form" action="https://www.google.com/search"; method="get" class="navbar-search pull-right" > + <input value="http://atlas.apache.org"; name="sitesearch" type="hidden"/> + <input class="search-query" name="q" id="query" type="text" /> +</form> +<script type="text/javascript">asyncJs( 'https://cse.google.com/brand?form=search-form' )</script> + <iframe src="https://www.facebook.com/plugins/like.php?href=http://atlas.apache.org/atlas-docs&send=false&layout=button_count&show-faces=false&action=like&colorscheme=dark"; + scrolling="no" frameborder="0" + style="border:none; width:100px; height:20px; margin-top: 10px;" class="pull-right" ></iframe> + <script type="text/javascript">asyncJs( 'https://apis.google.com/js/plusone.js' )</script> + <ul class="nav pull-right"><li style="margin-top: 10px;"> + <div class="g-plusone" data-href="http://atlas.apache.org/atlas-docs"; data-size="medium" width="60px" align="right" ></div> + </li></ul> + </div> + </div> + </div> + </div> + <div class="container"> + <div id="banner"> + <div class="pull-left"><a href=".." id="bannerLeft"><img src="images/atlas-logo.png" alt="Apache Atlas" width="200px" height="45px"/></a></div> + <div class="pull-right"></div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + <li class=""><a href="http://www.apache.org"; class="externalLink" title="Apache">Apache</a><span class="divider">/</span></li> + <li class=""><a href="index.html" title="Atlas">Atlas</a><span class="divider">/</span></li> + <li class="active ">Export & Import REST APIs</li> + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-05-26</li> + <li id="projectVersion" class="pull-right">Version: 1.0.0-rc0</li> + </ul> + </div> + <div id="bodyColumn" > +<div class="section"> +<h2><a name="Export_.26_Import_REST_APIs"></a>Export & Import REST APIs</h2></div> +<div class="section"> +<h4><a name="Background"></a>Background</h4> +<p>The Import-Export APIs for Atlas facilitate transfer of data to and from a cluster that has Atlas provisioned.</p> +<p>The APIs when integrated with backup and/or disaster recovery process will ensure participation of Atlas.</p></div> +<div class="section"> +<h4><a name="Introduction"></a>Introduction</h4> +<p>There are 2 broad categories viz. Export & Import. The details of the APIs are discussed below.</p> +<p>The APIs are available only to <i>admin</i> user.</p> +<p>Only a single import or export operation can be performed at a given time. The operations have a potential for generating large amount. They can also put pressure on resources. This restriction tries to alleviate this problem.</p> +<p>For Import-Export APIs relating to HDFS path, can be found <a href="./Import-Export-HDFS-Path.html">here</a>.</p> +<p>For additional information please refer to the following:</p> +<ul> +<li><a class="externalLink" href="https://issues.apache.org/jira/browse/ATLAS-1503";>ATLAS-1503</a> Original Import-Export API requirements.</li> +<li><a class="externalLink" href="https://issues.apache.org/jira/browse/ATLAS-1618";>ATLAS-1618</a> Export API Scope Specification.</li></ul></div> +<div class="section"> +<h4><a name="Errors"></a>Errors</h4> +<p>If an import or export operation is initiated while another is in progress, the consumer will receive this error:</p> +<div class="source"><pre class="prettyprint"> +"ATLAS5005E": "Another import or export is in progress. Please try again." + +</pre></div> +<p>Unhandled errors will be returned as Internal error code 500.</p></div> +<div class="section"> +<h3><a name="REST_API_Reference"></a>REST API Reference</h3> +<p></p> +<ul> +<li><b><i><a href="./Export-API.html">Export</a></i></b></li> +<li><b><i><a href="./Export-HDFS-API.html">Export HDFS</a></i></b></li> +<li><b><i><a href="./Import-API.html">Import</a></i></b></li> +<li><b><i><a href="./Import-API-Options.html">Import Options</a></i></b></li></ul></div> + </div> + </div> + <hr/> + <footer> + <div class="container"> + <div class="row"> +<p><a href="https://www.apache.org/foundation/contributing";><img src="https://www.apache.org/images/SupportApache-small.png"; alt="Support the ASF" id="asf-logo" height="20" width="20" /></a>Copyright © 2011-2018 The Apache Software Foundation. Licensed under the <a href="https://www.apache.org/licenses/";>Apache License, Version 2.0</a>.<br/> +Apache Atlas, Atlas, Apache, the Apache feather logo are trademarks of the <a href="https://www.apache.org";>Apache Software Foundation</a>.<br/> +All other marks mentioned may be trademarks or registered trademarks of their respective owners.</p> + </div> + <p id="poweredBy" class="pull-right"><a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"><img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /></a> +</p> + </div> + </footer> + </body> +</html> http://git-wip-us.apache.org/repos/asf/atlas-website/blob/e9346ee2/1.0.0-rc0/InstallationSteps.html ---------------------------------------------------------------------- diff --git a/1.0.0-rc0/InstallationSteps.html b/1.0.0-rc0/InstallationSteps.html new file mode 100644 index 0000000..242722e --- /dev/null +++ b/1.0.0-rc0/InstallationSteps.html @@ -0,0 +1,458 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/twiki/InstallationSteps.twiki at 2018-05-26 + | Rendered using Apache Maven Fluido Skin 1.7 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20180526" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Apache Atlas – Building & Installing Apache Atlas</title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.7.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + <script type="text/javascript" src="./js/apache-maven-fluido-1.7.min.js"></script> + </head> + <body class="topBarEnabled"> + <div id="topbar" class="navbar navbar-fixed-top "> + <div class="navbar-inner"> + <div class="container" style="width: 68%;"><div class="nav-collapse"> + <ul class="nav"> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Atlas <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="index.html" title="About">About</a></li> + <li><a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="Wiki">Wiki</a></li> + <li><a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="News">News</a></li> + <li><a href="https://git-wip-us.apache.org/repos/asf/atlas.git"; title="Git">Git</a></li> + <li><a href="https://issues.apache.org/jira/browse/ATLAS"; title="Jira">Jira</a></li> + <li><a href="https://cwiki.apache.org/confluence/display/ATLAS/PoweredBy"; title="Powered by">Powered by</a></li> + <li><a href="http://blogs.apache.org/atlas/"; title="Blog">Blog</a></li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Project Information <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="project-info.html" title="Summary">Summary</a></li> + <li><a href="mail-lists.html" title="Mailing Lists">Mailing Lists</a></li> + <li><a href="http://webchat.freenode.net?channels=apacheatlas&uio=d4"; title="IRC">IRC</a></li> + <li><a href="team-list.html" title="Team">Team</a></li> + <li><a href="issue-tracking.html" title="Issue Tracking">Issue Tracking</a></li> + <li><a href="source-repository.html" title="Source Repository">Source Repository</a></li> + <li><a href="license.html" title="License">License</a></li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Releases <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="http://www.apache.org/dyn/closer.cgi/atlas/1.0.0-alpha/"; title="1.0.0-alpha">1.0.0-alpha</a></li> + <li><a href="http://www.apache.org/dyn/closer.cgi/atlas/1.0.0-rc0/"; title="1.0.0-rc0">1.0.0-rc0</a></li> + <li><a href="http://www.apache.org/dyn/closer.cgi/atlas/0.8.2/"; title="0.8.2">0.8.2</a></li> + <li><a href="http://www.apache.org/dyn/closer.cgi/atlas/0.8.1/"; title="0.8.1">0.8.1</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.8.0-incubating/"; title="0.8-incubating">0.8-incubating</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.7.1-incubating/"; title="0.7.1-incubating">0.7.1-incubating</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.7.0-incubating/"; title="0.7-incubating">0.7-incubating</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.6.0-incubating/"; title="0.6-incubating">0.6-incubating</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.5.0-incubating/"; title="0.5-incubating">0.5-incubating</a></li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="../index.html" title="latest">latest</a></li> + <li><a href="../1.0.0-alpha/index.html" title="1.0.0-alpha">1.0.0-alpha</a></li> + <li><a href="../1.0.0-rc0/index.html" title="1.0.0-rc0">1.0.0-rc0</a></li> + <li><a href="../0.8.2/index.html" title="0.8.2">0.8.2</a></li> + <li><a href="../0.8.1/index.html" title="0.8.1">0.8.1</a></li> + <li><a href="../0.8.0-incubating/index.html" title="0.8-incubating">0.8-incubating</a></li> + <li><a href="../0.7.1-incubating/index.html" title="0.7.1-incubating">0.7.1-incubating</a></li> + <li><a href="../0.7.0-incubating/index.html" title="0.7-incubating">0.7-incubating</a></li> + <li><a href="../0.6.0-incubating/index.html" title="0.6-incubating">0.6-incubating</a></li> + <li><a href="../0.5.0-incubating/index.html" title="0.5-incubating">0.5-incubating</a></li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">ASF <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="http://www.apache.org/foundation/how-it-works.html"; title="How Apache Works">How Apache Works</a></li> + <li><a href="https://www.apache.org/events/current-event"; title="Events">Events</a></li> + <li><a href="https://www.apache.org/licenses/"; title="License">License</a></li> + <li><a href="http://www.apache.org/foundation/"; title="Foundation">Foundation</a></li> + <li><a href="http://www.apache.org/foundation/sponsorship.html"; title="Sponsoring Apache">Sponsoring Apache</a></li> + <li><a href="http://www.apache.org/foundation/thanks.html"; title="Thanks">Thanks</a></li> + </ul> + </li> + </ul> +<form id="search-form" action="https://www.google.com/search"; method="get" class="navbar-search pull-right" > + <input value="http://atlas.apache.org"; name="sitesearch" type="hidden"/> + <input class="search-query" name="q" id="query" type="text" /> +</form> +<script type="text/javascript">asyncJs( 'https://cse.google.com/brand?form=search-form' )</script> + <iframe src="https://www.facebook.com/plugins/like.php?href=http://atlas.apache.org/atlas-docs&send=false&layout=button_count&show-faces=false&action=like&colorscheme=dark"; + scrolling="no" frameborder="0" + style="border:none; width:100px; height:20px; margin-top: 10px;" class="pull-right" ></iframe> + <script type="text/javascript">asyncJs( 'https://apis.google.com/js/plusone.js' )</script> + <ul class="nav pull-right"><li style="margin-top: 10px;"> + <div class="g-plusone" data-href="http://atlas.apache.org/atlas-docs"; data-size="medium" width="60px" align="right" ></div> + </li></ul> + </div> + </div> + </div> + </div> + <div class="container"> + <div id="banner"> + <div class="pull-left"><a href=".." id="bannerLeft"><img src="images/atlas-logo.png" alt="Apache Atlas" width="200px" height="45px"/></a></div> + <div class="pull-right"></div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + <li class=""><a href="http://www.apache.org"; class="externalLink" title="Apache">Apache</a><span class="divider">/</span></li> + <li class=""><a href="index.html" title="Atlas">Atlas</a><span class="divider">/</span></li> + <li class="active ">Building & Installing Apache Atlas</li> + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-05-26</li> + <li id="projectVersion" class="pull-right">Version: 1.0.0-rc0</li> + </ul> + </div> + <div id="bodyColumn" > +<div class="section"> +<h3><a name="Building_.26_Installing_Apache_Atlas"></a>Building & Installing Apache Atlas</h3></div> +<div class="section"> +<h4><a name="Building_Apache_Atlas"></a>Building Apache Atlas</h4> +<div class="source"><pre class="prettyprint"> +git clone https://github.com/apache/atlas.git atlas +cd atlas +git checkout branch-1.0 +export MAVEN_OPTS="-Xms2g -Xmx2g" +mvn clean -DskipTests install +</pre></div></div> +<div class="section"> +<h4><a name="Packaging_Apache_Atlas"></a>Packaging Apache Atlas</h4> +<p>To create Apache Atlas package for deployment in an environment having functional Apache HBase and Apache Solr instances, build with the following command:</p> +<div class="source"><pre class="prettyprint"> +mvn clean -DskipTests package -Pdist +</pre></div> +<p></p> +<ul> +<li>NOTES: +<ul> +<li>Remove option '-DskipTests' to run unit and integration tests</li> +<li>To build a distribution without minified js,css file, build with <i>skipMinify</i> profile. By default js and css files are minified.</li></ul></li></ul> +<p>Above will build Apache Atlas for an environment having functional HBase and Solr instances. Apache Atlas needs to be setup with the following to run in this environment:</p> +<ul> +<li>Configure atlas.graph.storage.hostname (see "Graph persistence engine - HBase" in the <a href="./Configuration.html">Configuration</a> section).</li> +<li>Configure atlas.graph.index.search.solr.zookeeper-url (see "Graph Search Index - Solr" in the <a href="./Configuration.html">Configuration</a> section).</li> +<li>Set HBASE_CONF_DIR to point to a valid Apache HBase config directory (see "Graph persistence engine - HBase" in the <a href="./Configuration.html">Configuration</a> section).</li> +<li>Create indices in Apache Solr (see "Graph Search Index - Solr" in the <a href="./Configuration.html">Configuration</a> section).</li></ul></div> +<div class="section"> +<h4><a name="Packaging_Apache_Atlas_with_embedded_Apache_HBase_.26_Apache_Solr"></a>Packaging Apache Atlas with embedded Apache HBase & Apache Solr</h4> +<p>To create Apache Atlas package that includes Apache HBase and Apache Solr, build with the embedded-hbase-solr profile as shown below:</p> +<div class="source"><pre class="prettyprint"> +mvn clean -DskipTests package -Pdist,embedded-hbase-solr +</pre></div> +<p>Using the embedded-hbase-solr profile will configure Apache Atlas so that an Apache HBase instance and an Apache Solr instance will be started and stopped along with the Apache Atlas server.</p> +<p>NOTE: This distribution profile is only intended to be used for single node development not in production.</p></div> +<div class="section"> +<h4><a name="Packaging_Apache_Atlas_with_embedded_Apache_Cassandra_.26_Apache_Solr"></a>Packaging Apache Atlas with embedded Apache Cassandra & Apache Solr</h4> +<p>To create Apache Atlas package that includes Apache Cassandra and Apache Solr, build with the embedded-cassandra-solr profile as shown below:</p> +<div class="source"><pre class="prettyprint"> +mvn clean package -Pdist,embedded-cassandra-solr +</pre></div> +<p>Using the embedded-cassandra-solr profile will configure Apache Atlas so that an Apache Cassandra instance and an Apache Solr instance will be started and stopped along with the Atlas server.</p> +<p>NOTE: This distribution profile is only intended to be used for single node development not in production.</p></div> +<div class="section"> +<h4><a name="Apache_Atlas_Package"></a>Apache Atlas Package</h4> +<p>Build will create following files, which are used to install Apache Atlas.</p> +<div class="source"><pre class="prettyprint"> +distro/target/apache-atlas-${project.version}-bin.tar.gz +distro/target/apache-atlas-${project.version}-hbase-hook.tar.gz +distro/target/apache-atlas-${project.version}-hive-hook.gz +distro/target/apache-atlas-${project.version}-kafka-hook.gz +distro/target/apache-atlas-${project.version}-sources.tar.gz +distro/target/apache-atlas-${project.version}-sqoop-hook.tar.gz +distro/target/apache-atlas-${project.version}-storm-hook.tar.gz +</pre></div></div> +<div class="section"> +<h4><a name="Installing_.26_Running_Apache_Atlas"></a>Installing & Running Apache Atlas</h4></div> +<div class="section"> +<h5><a name="Installing_Apache_Atlas"></a>Installing Apache Atlas</h5> +<p>From the directory you would like Apache Atlas to be installed, run the following commands:</p> +<div class="source"><pre class="prettyprint"> +tar -xzvf apache-atlas-${project.version}-bin.tar.gz +cd atlas-${project.version} +</pre></div></div> +<div class="section"> +<h5><a name="Running_Apache_Atlas_with_Local_Apache_HBase_.26_Apache_Solr"></a>Running Apache Atlas with Local Apache HBase & Apache Solr</h5> +<p>To run Apache Atlas with local Apache HBase & Apache Solr instances that are started/stopped along with Atlas start/stop, run following commands:</p> +<div class="source"><pre class="prettyprint"> +export MANAGE_LOCAL_HBASE=true +export MANAGE_LOCAL_SOLR=true + +bin/atlas_start.py +</pre></div></div> +<div class="section"> +<h5><a name="Using_Apache_Atlas"></a>Using Apache Atlas</h5> +<p></p> +<ul> +<li>To verify if Apache Atlas server is up and running, run curl command as shown below:</li></ul> +<div class="source"><pre class="prettyprint"> + curl -u username:password http://localhost:21000/api/atlas/admin/version + + {"Description":"Metadata Management and Data Governance Platform over Hadoop","Version":"1.0.0","Name":"apache-atlas"} +</pre></div> +<p></p> +<ul> +<li>Run quick start to load sample model and data</li></ul> +<div class="source"><pre class="prettyprint"> + bin/quick_start.py + Enter username for atlas :- + Enter password for atlas :- + +</pre></div> +<p></p> +<ul> +<li>Access Apache Atlas UI using a browser: <a class="externalLink" href="http://localhost:21000";>http://localhost:21000</a></li></ul></div> +<div class="section"> +<h5><a name="Stopping_Apache_Atlas_Server"></a>Stopping Apache Atlas Server</h5> +<p>To stop Apache Atlas, run following command:</p> +<div class="source"><pre class="prettyprint"> +bin/atlas_stop.py +</pre></div></div> +<div class="section"> +<h4><a name="Configuring_Apache_Atlas"></a>Configuring Apache Atlas</h4> +<p>By default config directory used by Apache Atlas is <i>{package dir}/conf</i>. To override this set environment variable ATLAS_CONF to the path of the conf dir.</p> +<p>Environment variables needed to run Apache Atlas can be set in <i>atlas-env.sh</i> file in the conf directory. This file will be sourced by Apache Atlas scripts before any commands are executed. The following environment variables are available to set.</p> +<div class="source"><pre class="prettyprint"> +# The java implementation to use. If JAVA_HOME is not found we expect java and jar to be in path +#export JAVA_HOME= + +# any additional java opts you want to set. This will apply to both client and server operations +#export ATLAS_OPTS= + +# any additional java opts that you want to set for client only +#export ATLAS_CLIENT_OPTS= + +# java heap size we want to set for the client. Default is 1024MB +#export ATLAS_CLIENT_HEAP= + +# any additional opts you want to set for atlas service. +#export ATLAS_SERVER_OPTS= + +# java heap size we want to set for the atlas server. Default is 1024MB +#export ATLAS_SERVER_HEAP= + +# What is is considered as atlas home dir. Default is the base location of the installed software +#export ATLAS_HOME_DIR= + +# Where log files are stored. Defatult is logs directory under the base install location +#export ATLAS_LOG_DIR= + +# Where pid files are stored. Defatult is logs directory under the base install location +#export ATLAS_PID_DIR= + +# Where do you want to expand the war file. By Default it is in /server/webapp dir under the base install dir. +#export ATLAS_EXPANDED_WEBAPP_DIR= +</pre></div> +<p><b>Settings to support large number of metadata objects</b></p> +<p>If you plan to store large number of metadata objects, it is recommended that you use values tuned for better GC performance of the JVM.</p> +<p>The following values are common server side options:</p> +<div class="source"><pre class="prettyprint"> +export ATLAS_SERVER_OPTS="-server -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+PrintTenuringDistribution -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=dumps/atlas_server.hprof -Xloggc:logs/gc-worker.log -verbose:gc -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=1m -XX:+PrintGCDetails -XX:+PrintHeapAtGC -XX:+PrintGCTimeStamps" +</pre></div> +<p>The <tt>-XX:SoftRefLRUPolicyMSPerMB</tt> option was found to be particularly helpful to regulate GC performance for query heavy workloads with many concurrent users.</p> +<p>The following values are recommended for JDK 8:</p> +<div class="source"><pre class="prettyprint"> +export ATLAS_SERVER_HEAP="-Xms15360m -Xmx15360m -XX:MaxNewSize=5120m -XX:MetaspaceSize=100M -XX:MaxMetaspaceSize=512m" +</pre></div> +<p><b>NOTE for Mac OS users</b> If you are using a Mac OS, you will need to configure the ATLAS_SERVER_OPTS (explained above).</p> +<p>In <i>{package dir}/conf/atlas-env.sh</i> uncomment the following line</p> +<div class="source"><pre class="prettyprint"> +#export ATLAS_SERVER_OPTS= +</pre></div> +<p>and change it to look as below</p> +<div class="source"><pre class="prettyprint"> +export ATLAS_SERVER_OPTS="-Djava.awt.headless=true -Djava.security.krb5.realm= -Djava.security.krb5.kdc=" +</pre></div> +<p><b>Configuring Apache HBase as the storage backend for the Graph Repository</b></p> +<p>By default, Apache Atlas uses <a href="./JanusGraph.html">JanusGraph</a> as the graph repository and is the only graph repository implementation available currently. Apache HBase versions currently supported are 1.1.x. For configuring Apache Atlas graph persistence on Apache HBase, please see "Graph persistence engine - HBase" in the <a href="./Configuration.html">Configuration</a> section for more details.</p> +<p>Apache HBase tables used by Apache Atlas can be set using the following configurations:</p> +<div class="source"><pre class="prettyprint"> +atlas.graph.storage.hbase.table=atlas +atlas.audit.hbase.tablename=apache_atlas_entity_audit +</pre></div> +<p><b>Configuring Apache Solr as the indexing backend for the Graph Repository</b></p> +<p>By default, Apache Atlas uses <a href="./JanusGraph.html">JanusGraph</a> as the graph repository and is the only graph repository implementation available currently. For configuring <a href="./JanusGraph.html">JanusGraph</a> to work with Apache Solr, please follow the instructions below</p> +<p></p> +<ul> +<li>Install Apache Solr if not already running. The version of Apache Solr supported is 5.5.1. Could be installed from <a class="externalLink" href="http://archive.apache.org/dist/lucene/solr/5.5.1/solr-5.5.1.tgz";>http://archive.apache.org/dist/lucene/solr/5.5.1/solr-5.5.1.tgz</a></li></ul> +<p></p> +<ul> +<li>Start Apache Solr in cloud mode.</li></ul>SolrCloud mode uses a ZooKeeper Service as a highly available, central location for cluster management. For a small cluster, running with an existing ZooKeeper quorum should be fine. For larger clusters, you would want to run separate multiple ZooKeeper quorum with at least 3 servers. Note: Apache Atlas currently supports Apache Solr in "cloud" mode only. "http" mode is not supported. For more information, refer Apache Solr documentation - <a class="externalLink" href="https://cwiki.apache.org/confluence/display/solr/SolrCloud";>https://cwiki.apache.org/confluence/display/solr/SolrCloud</a> +<p></p> +<ul> +<li>For e.g., to bring up an Apache Solr node listening on port 8983 on a machine, you can use the command:</li></ul> +<div class="source"><pre class="prettyprint"> + $SOLR_HOME/bin/solr start -c -z <zookeeper_host:port> -p 8983 +</pre></div> +<p></p> +<ul> +<li>Run the following commands from SOLR_BIN (e.g. $SOLR_HOME/bin) directory to create collections in Apache Solr corresponding to the indexes that Apache Atlas uses. In the case that the Apache Atlas and Apache Solr instances are on 2 different hosts, first copy the required configuration files from ATLAS_HOME/conf/solr on the Apache Atlas instance host to Apache Solr instance host. SOLR_CONF in the below mentioned commands refer to the directory where Apache Solr configuration files have been copied to on Apache Solr host:</li></ul> +<div class="source"><pre class="prettyprint"> + $SOLR_BIN/solr create -c vertex_index -d SOLR_CONF -shards #numShards -replicationFactor #replicationFactor + $SOLR_BIN/solr create -c edge_index -d SOLR_CONF -shards #numShards -replicationFactor #replicationFactor + $SOLR_BIN/solr create -c fulltext_index -d SOLR_CONF -shards #numShards -replicationFactor #replicationFactor +</pre></div> +<p>Note: If numShards and replicationFactor are not specified, they default to 1 which suffices if you are trying out solr with ATLAS on a single node instance. Otherwise specify numShards according to the number of hosts that are in the Solr cluster and the maxShardsPerNode configuration. The number of shards cannot exceed the total number of Solr nodes in your !SolrCloud cluster.</p> +<p>The number of replicas (replicationFactor) can be set according to the redundancy required.</p> +<p>Also note that Apache Solr will automatically be called to create the indexes when Apache Atlas server is started if the SOLR_BIN and SOLR_CONF environment variables are set and the search indexing backend is set to 'solr5'.</p> +<p></p> +<ul> +<li>Change ATLAS configuration to point to Apache Solr instance setup. Please make sure the following configurations are set to the below values in ATLAS_HOME/conf/atlas-application.properties</li></ul> +<div class="source"><pre class="prettyprint"> + atlas.graph.index.search.backend=solr + atlas.graph.index.search.solr.mode=cloud + atlas.graph.index.search.solr.zookeeper-url=<the ZK quorum setup for solr as comma separated value> eg: 10.1.6.4:2181,10.1.6.5:2181 + atlas.graph.index.search.solr.zookeeper-connect-timeout=<SolrCloud Zookeeper Connection Timeout>. Default value is 60000 ms + atlas.graph.index.search.solr.zookeeper-session-timeout=<SolrCloud Zookeeper Session Timeout>. Default value is 60000 ms +</pre></div> +<p>For more information on <a href="./JanusGraph.html">JanusGraph</a> solr configuration , please refer <a class="externalLink" href="http://docs.janusgraph.org/0.2.0/solr.html";>http://docs.janusgraph.org/0.2.0/solr.html</a></p> +<p>Pre-requisites for running Apache Solr in cloud mode * Memory - Apache Solr is both memory and CPU intensive. Make sure the server running Apache Solr has adequate memory, CPU and disk. Apache Solr works well with 32GB RAM. Plan to provide as much memory as possible to Apache Solr process * Disk - If the number of entities that need to be stored are large, plan to have at least 500 GB free space in the volume where Apache Solr is going to store the index data * SolrCloud has support for replication and sharding. It is highly recommended to use SolrCloud with at least two Apache Solr nodes running on different servers with replication enabled. If using SolrCloud, then you also need ZooKeeper installed and configured with 3 or 5 ZooKeeper nodes</p> +<p><b>Configuring Elasticsearch as the indexing backend for the Graph Repository (Tech Preview)</b></p> +<p>By default, Apache Atlas uses <a href="./JanusGraph.html">JanusGraph</a> as the graph repository and is the only graph repository implementation available currently. For configuring <a href="./JanusGraph.html">JanusGraph</a> to work with Elasticsearch, please follow the instructions below</p> +<p></p> +<ul> +<li>Install an Elasticsearch cluster. The version currently supported is 5.6.4, and can be acquired from: <a class="externalLink" href="https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.6.4.tar.gz";>https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.6.4.tar.gz</a></li></ul> +<p></p> +<ul> +<li>For simple testing a single Elasticsearch node can be started by using the 'elasticsearch' command in the bin directory of the Elasticsearch distribution.</li></ul> +<p></p> +<ul> +<li>Change Apache Atlas configuration to point to the Elasticsearch instance setup. Please make sure the following configurations are set to the below values in ATLAS_HOME/conf/atlas-application.properties</li></ul> +<div class="source"><pre class="prettyprint"> + atlas.graph.index.search.backend=elasticsearch + atlas.graph.index.search.hostname=<the hostname(s) of the Elasticsearch master nodes comma separated> + atlas.graph.index.search.elasticsearch.client-only=true +</pre></div> +<p>For more information on <a href="./JanusGraph.html">JanusGraph</a> configuration for elasticsearch, please refer <a class="externalLink" href="http://docs.janusgraph.org/0.2.0/elasticsearch.html";>http://docs.janusgraph.org/0.2.0/elasticsearch.html</a></p> +<p><b>Configuring Kafka Topics</b></p> +<p>Apache Atlas uses Apache Kafka to ingest metadata from other components at runtime. This is described in the <a href="./Architecture.html">Architecture page</a> in more detail. Depending on the configuration of Apache Kafka, sometimes you might need to setup the topics explicitly before using Apache Atlas. To do so, Apache Atlas provides a script <tt>bin/atlas_kafka_setup.py</tt> which can be run from Apache Atlas server. In some environments, the hooks might start getting used first before Apache Atlas server itself is setup. In such cases, the topics can be run on the hosts where hooks are installed using a similar script <tt>hook-bin/atlas_kafka_setup_hook.py</tt>. Both these use configuration in <tt>atlas-application.properties</tt> for setting up the topics. Please refer to the <a href="./Configuration.html">Configuration page</a> for these details.</p></div> +<div class="section"> +<h5><a name="Setting_up_Apache_Atlas"></a>Setting up Apache Atlas</h5> +<p>There are a few steps that setup dependencies of Apache Atlas. One such example is setting up the <a href="./JanusGraph.html">JanusGraph</a> schema in the storage backend of choice. In a simple single server setup, these are automatically setup with default configuration when the server first accesses these dependencies.</p> +<p>However, there are scenarios when we may want to run setup steps explicitly as one time operations. For example, in a multiple server scenario using <a href="./HighAvailability.html">High Availability</a>, it is preferable to run setup steps from one of the server instances the first time, and then start the services.</p> +<p>To run these steps one time, execute the command <tt>bin/atlas_start.py -setup</tt> from a single Apache Atlas server instance.</p> +<p>However, Apache Atlas server does take care of parallel executions of the setup steps. Also, running the setup steps multiple times is idempotent. Therefore, if one chooses to run the setup steps as part of server startup, for convenience, then they should enable the configuration option <tt>atlas.server.run.setup.on.start</tt> by defining it with the value <tt>true</tt> in the <tt>atlas-application.properties</tt> file.</p></div> +<div class="section"> +<h4><a name="Examples:_calling_Apache_Atlas_REST_APIs"></a>Examples: calling Apache Atlas REST APIs</h4> +<p>Here are few examples of calling Apache Atlas REST APIs via curl command.</p> +<ul> +<li>List the types in the repository</li></ul> +<div class="source"><pre class="prettyprint"> + curl -u username:password http://localhost:21000/api/atlas/v2/types/typedefs/headers + [ {"guid":"fa421be8-c21b-4cf8-a226-fdde559ad598","name":"Referenceable","category":"ENTITY"}, + {"guid":"7f3f5712-521d-450d-9bb2-ba996b6f2a4e","name":"Asset","category":"ENTITY"}, + {"guid":"84b02fa0-e2f4-4cc4-8b24-d2371cd00375","name":"DataSet","category":"ENTITY"}, + {"guid":"f93975d5-5a5c-41da-ad9d-eb7c4f91a093","name":"Process","category":"ENTITY"}, + {"guid":"79dcd1f9-f350-4f7b-b706-5bab416f8206","name":"Infrastructure","category":"ENTITY"} + ] +</pre></div> +<p></p> +<ul> +<li>List the instances for a given type</li></ul> +<div class="source"><pre class="prettyprint"> + curl -u username:password http://localhost:21000/api/atlas/v2/search/basic?typeName=hive_db + { + "queryType":"BASIC", + "searchParameters":{ + "typeName":"hive_db", + "excludeDeletedEntities":false, + "includeClassificationAttributes":false, + "includeSubTypes":true, + "includeSubClassifications":true, + "limit":100, + "offset":0 + }, + "entities":[ + { + "typeName":"hive_db", + "guid":"5d900c19-094d-4681-8a86-4eb1d6ffbe89", + "status":"ACTIVE", + "displayText":"default", + "classificationNames":[], + "attributes":{ + "owner":"public", + "createTime":null, + "qualifiedName":"default@cl1", + "name":"default", + "description":"Default Hive database" + } + }, + { + "typeName":"hive_db", + "guid":"3a0b14b0-ab85-4b65-89f2-e418f3f7f77c", + "status":"ACTIVE", + "displayText":"finance", + "classificationNames":[], + "attributes":{ + "owner":"hive", + "createTime":null, + "qualifiedName":"finance@cl1", + "name":"finance", + "description":null + } + } + ] + } +</pre></div> +<p></p> +<ul> +<li>Search for entities</li></ul> +<div class="source"><pre class="prettyprint"> + curl -u username:password http://localhost:21000/api/atlas/v2/search/dsl?query=hive_db%20where%20name='default' + { + "queryType":"DSL", + "queryText":"hive_db where name='default'", + "entities":[ + { + "typeName":"hive_db", + "guid":"5d900c19-094d-4681-8a86-4eb1d6ffbe89", + "status":"ACTIVE", + "displayText":"default", + "classificationNames":[], + "attributes":{ + "owner":"public", + "createTime":null, + "qualifiedName":"default@cl1", + "name":"default", + "description": + "Default Hive database" + } + } + ] + } +</pre></div></div> +<div class="section"> +<h4><a name="Troubleshooting"></a>Troubleshooting</h4></div> +<div class="section"> +<h5><a name="Setup_issues"></a>Setup issues</h5> +<p>If the setup of Apache Atlas service fails due to any reason, the next run of setup (either by an explicit invocation of <tt>atlas_start.py -setup</tt> or by enabling the configuration option <tt>atlas.server.run.setup.on.start</tt>) will fail with a message such as <tt>A previous setup run may not have completed cleanly.</tt>. In such cases, you would need to manually ensure the setup can run and delete the Zookeeper node at <tt>/apache_atlas/setup_in_progress</tt> before attempting to run setup again.</p> +<p>If the setup failed due to Apache HBase schema setup errors, it may be necessary to repair Apache HBase schema. If no data has been stored, one can also disable and drop the Apache HBase tables used by Apache Atlas and run setup again.</p></div> + </div> + </div> + <hr/> + <footer> + <div class="container"> + <div class="row"> +<p><a href="https://www.apache.org/foundation/contributing";><img src="https://www.apache.org/images/SupportApache-small.png"; alt="Support the ASF" id="asf-logo" height="20" width="20" /></a>Copyright © 2011-2018 The Apache Software Foundation. Licensed under the <a href="https://www.apache.org/licenses/";>Apache License, Version 2.0</a>.<br/> +Apache Atlas, Atlas, Apache, the Apache feather logo are trademarks of the <a href="https://www.apache.org";>Apache Software Foundation</a>.<br/> +All other marks mentioned may be trademarks or registered trademarks of their respective owners.</p> + </div> + <p id="poweredBy" class="pull-right"><a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"><img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /></a> +</p> + </div> + </footer> + </body> +</html> http://git-wip-us.apache.org/repos/asf/atlas-website/blob/e9346ee2/1.0.0-rc0/Migration-0.8-to-1.0.html ---------------------------------------------------------------------- diff --git a/1.0.0-rc0/Migration-0.8-to-1.0.html b/1.0.0-rc0/Migration-0.8-to-1.0.html new file mode 100644 index 0000000..81c6e50 --- /dev/null +++ b/1.0.0-rc0/Migration-0.8-to-1.0.html @@ -0,0 +1,265 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/twiki/Migration-0.8-to-1.0.twiki at 2018-05-26 + | Rendered using Apache Maven Fluido Skin 1.7 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20180526" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Apache Atlas – Migrating data from Apache Atlas 0.8 to Apache Atlas 1.0</title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.7.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + <script type="text/javascript" src="./js/apache-maven-fluido-1.7.min.js"></script> + </head> + <body class="topBarEnabled"> + <div id="topbar" class="navbar navbar-fixed-top "> + <div class="navbar-inner"> + <div class="container" style="width: 68%;"><div class="nav-collapse"> + <ul class="nav"> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Atlas <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="index.html" title="About">About</a></li> + <li><a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="Wiki">Wiki</a></li> + <li><a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="News">News</a></li> + <li><a href="https://git-wip-us.apache.org/repos/asf/atlas.git"; title="Git">Git</a></li> + <li><a href="https://issues.apache.org/jira/browse/ATLAS"; title="Jira">Jira</a></li> + <li><a href="https://cwiki.apache.org/confluence/display/ATLAS/PoweredBy"; title="Powered by">Powered by</a></li> + <li><a href="http://blogs.apache.org/atlas/"; title="Blog">Blog</a></li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Project Information <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="project-info.html" title="Summary">Summary</a></li> + <li><a href="mail-lists.html" title="Mailing Lists">Mailing Lists</a></li> + <li><a href="http://webchat.freenode.net?channels=apacheatlas&uio=d4"; title="IRC">IRC</a></li> + <li><a href="team-list.html" title="Team">Team</a></li> + <li><a href="issue-tracking.html" title="Issue Tracking">Issue Tracking</a></li> + <li><a href="source-repository.html" title="Source Repository">Source Repository</a></li> + <li><a href="license.html" title="License">License</a></li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Releases <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="http://www.apache.org/dyn/closer.cgi/atlas/1.0.0-alpha/"; title="1.0.0-alpha">1.0.0-alpha</a></li> + <li><a href="http://www.apache.org/dyn/closer.cgi/atlas/1.0.0-rc0/"; title="1.0.0-rc0">1.0.0-rc0</a></li> + <li><a href="http://www.apache.org/dyn/closer.cgi/atlas/0.8.2/"; title="0.8.2">0.8.2</a></li> + <li><a href="http://www.apache.org/dyn/closer.cgi/atlas/0.8.1/"; title="0.8.1">0.8.1</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.8.0-incubating/"; title="0.8-incubating">0.8-incubating</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.7.1-incubating/"; title="0.7.1-incubating">0.7.1-incubating</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.7.0-incubating/"; title="0.7-incubating">0.7-incubating</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.6.0-incubating/"; title="0.6-incubating">0.6-incubating</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.5.0-incubating/"; title="0.5-incubating">0.5-incubating</a></li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="../index.html" title="latest">latest</a></li> + <li><a href="../1.0.0-alpha/index.html" title="1.0.0-alpha">1.0.0-alpha</a></li> + <li><a href="../1.0.0-rc0/index.html" title="1.0.0-rc0">1.0.0-rc0</a></li> + <li><a href="../0.8.2/index.html" title="0.8.2">0.8.2</a></li> + <li><a href="../0.8.1/index.html" title="0.8.1">0.8.1</a></li> + <li><a href="../0.8.0-incubating/index.html" title="0.8-incubating">0.8-incubating</a></li> + <li><a href="../0.7.1-incubating/index.html" title="0.7.1-incubating">0.7.1-incubating</a></li> + <li><a href="../0.7.0-incubating/index.html" title="0.7-incubating">0.7-incubating</a></li> + <li><a href="../0.6.0-incubating/index.html" title="0.6-incubating">0.6-incubating</a></li> + <li><a href="../0.5.0-incubating/index.html" title="0.5-incubating">0.5-incubating</a></li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">ASF <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="http://www.apache.org/foundation/how-it-works.html"; title="How Apache Works">How Apache Works</a></li> + <li><a href="https://www.apache.org/events/current-event"; title="Events">Events</a></li> + <li><a href="https://www.apache.org/licenses/"; title="License">License</a></li> + <li><a href="http://www.apache.org/foundation/"; title="Foundation">Foundation</a></li> + <li><a href="http://www.apache.org/foundation/sponsorship.html"; title="Sponsoring Apache">Sponsoring Apache</a></li> + <li><a href="http://www.apache.org/foundation/thanks.html"; title="Thanks">Thanks</a></li> + </ul> + </li> + </ul> +<form id="search-form" action="https://www.google.com/search"; method="get" class="navbar-search pull-right" > + <input value="http://atlas.apache.org"; name="sitesearch" type="hidden"/> + <input class="search-query" name="q" id="query" type="text" /> +</form> +<script type="text/javascript">asyncJs( 'https://cse.google.com/brand?form=search-form' )</script> + <iframe src="https://www.facebook.com/plugins/like.php?href=http://atlas.apache.org/atlas-docs&send=false&layout=button_count&show-faces=false&action=like&colorscheme=dark"; + scrolling="no" frameborder="0" + style="border:none; width:100px; height:20px; margin-top: 10px;" class="pull-right" ></iframe> + <script type="text/javascript">asyncJs( 'https://apis.google.com/js/plusone.js' )</script> + <ul class="nav pull-right"><li style="margin-top: 10px;"> + <div class="g-plusone" data-href="http://atlas.apache.org/atlas-docs"; data-size="medium" width="60px" align="right" ></div> + </li></ul> + </div> + </div> + </div> + </div> + <div class="container"> + <div id="banner"> + <div class="pull-left"><a href=".." id="bannerLeft"><img src="images/atlas-logo.png" alt="Apache Atlas" width="200px" height="45px"/></a></div> + <div class="pull-right"></div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + <li class=""><a href="http://www.apache.org"; class="externalLink" title="Apache">Apache</a><span class="divider">/</span></li> + <li class=""><a href="index.html" title="Atlas">Atlas</a><span class="divider">/</span></li> + <li class="active ">Migrating data from Apache Atlas 0.8 to Apache Atlas 1.0</li> + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-05-26</li> + <li id="projectVersion" class="pull-right">Version: 1.0.0-rc0</li> + </ul> + </div> + <div id="bodyColumn" > +<div class="section"> +<h2><a name="Migrating_data_from_Apache_Atlas_0.8_to_Apache_Atlas_1.0"></a>Migrating data from Apache Atlas 0.8 to Apache Atlas 1.0</h2> +<p>Apache Atlas 1.0 uses JanusGraph graph database to store its type and entity details. Prior versions of Apache Atlas use Titan 0.5.4 graph database. The two databases use different formats for storage. For deployments upgrading from earlier version Apache Atlas, the data in Titan 0.5.4 graph database should be migrated to JanusGraph graph database.</p> +<p>In addition to the change to the graph database, Apache Atlas 1.0 introduces few optimizations that require different internal representation compared to previous versions. Migration steps detailed below will transform data to be compliant with the new internal representation.</p></div> +<div class="section"> +<h4><a name="Migration_Steps"></a>Migration Steps</h4> +<p>Migration of data is done in following steps:</p> +<ul> +<li>Planning the migration.</li> +<li>Export Apache Atlas 0.8 data to a directory on the file system.</li> +<li>Import data from exported files into Apache Atlas 1.0.</li></ul></div> +<div class="section"> +<h5><a name="Planning_the_migration"></a>Planning the migration</h5> +<p>The duration of migration of data from Apache Atlas 0.8 to Apache Atlas 1.0 can be significant, depending upon the amount of data present in Apache Atlas. This section helps you to estimate the time to migrate, so that you can plan the upgrade process better.</p> +<p>To estimate the time needed to export data, first you need to find the number of entities in Apache Atlas 0.8. This can be done by running the following DSL query:</p> +<div class="source"><pre class="prettyprint"> +Referenceable select count() + +</pre></div> +<p>Assuming Apache Atlas is deployed in a quad-core CPU with 4 GB of RAM allocated:</p> +<ul> +<li>Export from Apache Atlas 0.8 will process approximately 2 million entities per hour.</li> +<li>Import into Apache Atlas 1.0 will process approximately 0.7 million entities per hour.</li></ul></div> +<div class="section"> +<h5><a name="Exporting_data_from_Apache_Atlas_0.8"></a>Exporting data from Apache Atlas 0.8</h5> +<p><i>Atlas Migration Export Utility</i> from Apache Atlas branch-0.8 should be used to export the data from Apache Atlas 0.8 deployments. The implementation of which can be found <a class="externalLink" href="https://github.com/apache/atlas/tree/branch-0.8/tools/atlas-migration-exporter";>here</a>.</p> +<p>To build this utility:</p> +<ul> +<li>Navigate to root directory of Apache Atlas branch-0.8 local repo (say, <i>/home/atlas/</i>)</li> +<li>Build using the command: <i>mvn clean -DskipTests package -Pdist</i>.</li> +<li>After successful build, the <i>Atlas Migration Utility</i> can be found in this <i>distro/target</i> directory. (Say, <i>/home/atlas/distro/target/apache-atlas-0.8.3-SNAPSHOT-bin/apache-atlas-0.8.3-SNAPSHOT/tools/migration-exporter</i>)</li></ul> +<p>Move the <i>Atlas Migration Utility</i> directory to the Apache Atlas 0.8 cluster.</p> +<p>Follow these steps to export the data:</p> +<ul> +<li>Shutdown <i>Apache Atlas 0.8</i>, so that the database is not updated while the migration is in progress.</li> +<li>Execute the following command to export Apache Atlas data in Titan graph database to the specified directory:</li></ul> +<div class="source"><pre class="prettyprint"> + atlas_migration_export.py -d <output directory> +</pre></div> +<p>Example:</p> +<div class="source"><pre class="prettyprint"> + /home/atlas-migration-utility/atlas_migration_export.py -d /home/atlas-0.8-data + +</pre></div> +<p>On successful execution, <i>Atlas Migration Utility</i> tool will display messages like these:</p> +<div class="source"><pre class="prettyprint"> +atlas-migration-export: starting migration export. Log file location /var/log/atlas/atlas-migration-exporter.log +atlas-migration-export: initializing +atlas-migration-export: initialized +atlas-migration-export: exporting typesDef to file /home/atlas-0.8-data/atlas-migration-typesdef.json +atlas-migration-export: exported typesDef to file /home/atlas-0.8-data/atlas-migration-typesdef.json +atlas-migration-export: exporting data to file /home/atlas-0.8-data/atlas-migration-data.json +atlas-migration-export: exported data to file /home/atlas-0.8-data/atlas-migration-data.json +atlas-migration-export: completed migration export! + +</pre></div> +<p>More details on the progress of export can be found in a log file named <i>atlas-migration-exporter.log</i>, in the log directory specified in <i>atlas-log4j.xml</i>.</p></div> +<div class="section"> +<h4><a name="Before_importing_into_Apache_Atlas_1.0"></a>Before importing into Apache Atlas 1.0</h4> +<p></p> +<ul> +<li>For Apache Atlas deployments that use Solr as index store, please ensure that existing Apache Atlas specific collections are deleted or renamed before installing Apache Atlas 1.0.</li></ul> +<p>Apache Atlas specific Solr collections can be deleted using CURL commands shown below:</p> +<div class="source"><pre class="prettyprint"> +curl 'http://<solrHost:port>/solr/admin/collections?action=DELETE&name=vertex_index' +curl 'http://<solrHost:port>/solr/admin/collections?action=DELETE&name=edge_index' +curl 'http://<solrHost:port>/solr/admin/collections?action=DELETE&name=fulltext_index' +</pre></div> +<p></p> +<ul> +<li>Create Solr collections for Apache Atlas 1.0</li></ul>Apache Atlas specific Solr collections can be created using CURL commands shown below: +<div class="source"><pre class="prettyprint"> +curl 'http://<solrHost:port>/solr/admin/collections?action=CREATE&name=vertex_index&numShards=1&replicationFactor=1&collection.configName=atlas_configs' +curl 'http://<solrHost:port>/solr/admin/collections?action=CREATE&name=edge_index&numShards=1&replicationFactor=1&collection.configName=atlas_configs' +curl 'http://<solrHost:port>/solr/admin/collections?action=CREATE&name=fulltext_index&numShards=1&replicationFactor=1&collection.configName=atlas_configs' +</pre></div> +<p></p> +<ul> +<li>For Apache Atlas deployments that use HBase as backend store, please note that HBase table used by earlier version can't be used by Apache Atlas 1.0. If you are constrained on disk storage space, the table used by earlier version can be removed after successful export of data. +<ul> +<li>Apache Atlas 0.8 uses HBase table named 'atlas_titan' (by default)</li> +<li>Apache Atlas 1.0 uses HBase table named 'atlas_janus' (by default)</li></ul></li></ul> +<p></p> +<ul> +<li>Install Apache Atlas 1.0. Do not start yet!</li></ul> +<p></p> +<ul> +<li>Make sure the directory containing exported data is accessible to Apache Atlas 1.0 instance.</li></ul></div> +<div class="section"> +<h5><a name="Importing_Data_into_Apache_Atlas_1.0"></a>Importing Data into Apache Atlas 1.0</h5> +<p>Please follow the steps below to import the data exported above into Apache Atlas 1.0:</p> +<ul> +<li>Specify the location of the directory containing exported data in following property to <i>atlas-application.properties</i>:</li></ul> +<div class="source"><pre class="prettyprint"> +atlas.migration.data.filename=<location of the directory containing exported data> +</pre></div> +<p></p> +<ul> +<li>Start Apache Atlas 1.0. Apache Atlas will start in migration mode. It will start importing data from the specified directory.</li></ul> +<p></p> +<ul> +<li>Monitor the progress of import process with the following curl command:</li></ul> +<div class="source"><pre class="prettyprint"> +curl -X GET -u admin:<password> -H "Content-Type: application/json" -H "Cache-Control: no-cache" http://<atlasHost>:port/api/atlas/admin/status +</pre></div> +<p>Progress of import will be indicated by a message like this:</p> +<div class="source"><pre class="prettyprint"> +{"Status":"MIGRATING","MigrationStatus":{"operationStatus":"IN_PROGRESS","startTime":1526512275110,"endTime":1526512302750,"currentIndex":10,"currentCounter":101,"totalCount":0}} + +</pre></div> +<p>Successful completion of the operation will show a message like this:</p> +<div class="source"><pre class="prettyprint"> +{"Status":"MIGRATING","MigrationStatus":{"operationStatus":"SUCCESS","startTime":1526512275110,"endTime":1526512302750,"currentIndex":0,"currentCounter":0,"totalCount":371}} + +</pre></div> +<p>Once migration import is complete, i.e. <i>operationStatus</i> is <i>SUCCESS</i>, follow the steps given below to restart Apache Atlas in <i>ACTIVE</i> mode for regular use:</p> +<ul> +<li>Stop Apache Atlas 1.0.</li> +<li>Remove property <i>atlas.migration.data.filename</i> from <i>atlas-application.properties</i>.</li> +<li>Start Apache Atlas 1.0.</li></ul></div> +<div class="section"> +<h4><a name="Atlas_Entity_Defaults_for_Migrated_Data"></a>Atlas Entity Defaults for Migrated Data</h4> +<p>Apache Atlas 1.0 introduces number of new features. For data that is migrated, the following defaults are set:</p> +<ul> +<li>All classifications will have <i>isPropagate</i> set to <i>false</i>.</li> +<li>Taxonomy terms present in Apache Atlas 0.8, if any, will be converted to classification.</li></ul></div> +<div class="section"> +<h5><a name="Handling_of_Entity_Definitions_that_use_Classifications_as_Types"></a>Handling of Entity Definitions that use Classifications as Types</h5> +<p>This features is no longer supported. Classifications that are used as types in <i>attribute definitions</i> (<i><a href="./AttributeDefs.html">AttributeDefs</a></i>) are converted in to new types whose name has <i>legacy</i> prefix. These are then handled like any other type. Creation of such types was prevented in an earlier release, hence only type definitions have potential to exist. Care has been taken to handle entities of this type as well.</p></div> + </div> + </div> + <hr/> + <footer> + <div class="container"> + <div class="row"> +<p><a href="https://www.apache.org/foundation/contributing";><img src="https://www.apache.org/images/SupportApache-small.png"; alt="Support the ASF" id="asf-logo" height="20" width="20" /></a>Copyright © 2011-2018 The Apache Software Foundation. Licensed under the <a href="https://www.apache.org/licenses/";>Apache License, Version 2.0</a>.<br/> +Apache Atlas, Atlas, Apache, the Apache feather logo are trademarks of the <a href="https://www.apache.org";>Apache Software Foundation</a>.<br/> +All other marks mentioned may be trademarks or registered trademarks of their respective owners.</p> + </div> + <p id="poweredBy" class="pull-right"><a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"><img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /></a> +</p> + </div> + </footer> + </body> +</html>
![https://www.apache.org/images/SupportApache-small.png"](https://www.apache.org/images/SupportApache-small.png"){kind=link}