http://git-wip-us.apache.org/repos/asf/atlas-website/blob/2d4dc1a0/0.8.3/InstallationSteps.html ---------------------------------------------------------------------- diff --git a/0.8.3/InstallationSteps.html b/0.8.3/InstallationSteps.html new file mode 100644 index 0000000..1bf4d15 --- /dev/null +++ b/0.8.3/InstallationSteps.html @@ -0,0 +1,394 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/twiki/InstallationSteps.twiki at 2018-10-31 + | Rendered using Apache Maven Fluido Skin 1.7 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20181031" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Apache Atlas РBuilding & Installing Apache Atlas</title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.7.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + <script type="text/javascript" src="./js/apache-maven-fluido-1.7.min.js"></script> + </head> + <body class="topBarEnabled"> + <div id="topbar" class="navbar navbar-fixed-top "> + <div class="navbar-inner"> + <div class="container" style="width: 68%;"><div class="nav-collapse"> + <ul class="nav"> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Atlas <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="index.html" title="About">About</a></li> + <li><a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="Wiki">Wiki</a></li> + <li><a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="News">News</a></li> + <li><a href="https://git-wip-us.apache.org/repos/asf/atlas.git"; title="Git">Git</a></li> + <li><a href="https://issues.apache.org/jira/browse/ATLAS"; title="Jira">Jira</a></li> + <li><a href="https://cwiki.apache.org/confluence/display/ATLAS/PoweredBy"; title="Powered by">Powered by</a></li> + <li><a href="http://blogs.apache.org/atlas/"; title="Blog">Blog</a></li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Project Information <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="project-info.html" title="Summary">Summary</a></li> + <li><a href="mail-lists.html" title="Mailing Lists">Mailing Lists</a></li> + <li><a href="http://webchat.freenode.net?channels=apacheatlas&uio=d4"; title="IRC">IRC</a></li> + <li><a href="team-list.html" title="Team">Team</a></li> + <li><a href="issue-tracking.html" title="Issue Tracking">Issue Tracking</a></li> + <li><a href="source-repository.html" title="Source Repository">Source Repository</a></li> + <li><a href="license.html" title="License">License</a></li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Releases <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="http://www.apache.org/dyn/closer.cgi/atlas/0.8.2/"; title="0.8.2">0.8.2</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.8.1/"; title="0.8.1">0.8.1</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.8.0-incubating/"; title="0.8-incubating">0.8-incubating</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.7.1-incubating/"; title="0.7.1-incubating">0.7.1-incubating</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.7.0-incubating/"; title="0.7-incubating">0.7-incubating</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.6.0-incubating/"; title="0.6-incubating">0.6-incubating</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.5.0-incubating/"; title="0.5-incubating">0.5-incubating</a></li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="../index.html" title="latest">latest</a></li> + <li><a href="../0.8.2/index.html" title="0.8.2">0.8.2</a></li> + <li><a href="../0.8.1/index.html" title="0.8.1">0.8.1</a></li> + <li><a href="../0.8.0-incubating/index.html" title="0.8-incubating">0.8-incubating</a></li> + <li><a href="../0.7.1-incubating/index.html" title="0.7.1-incubating">0.7.1-incubating</a></li> + <li><a href="../0.7.0-incubating/index.html" title="0.7-incubating">0.7-incubating</a></li> + <li><a href="../0.6.0-incubating/index.html" title="0.6-incubating">0.6-incubating</a></li> + <li><a href="../0.5.0-incubating/index.html" title="0.5-incubating">0.5-incubating</a></li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">ASF <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="http://www.apache.org/foundation/how-it-works.html"; title="How Apache Works">How Apache Works</a></li> + <li><a href="http://www.apache.org/foundation/"; title="Foundation">Foundation</a></li> + <li><a href="http://www.apache.org/foundation/sponsorship.html"; title="Sponsoring Apache">Sponsoring Apache</a></li> + <li><a href="http://www.apache.org/foundation/thanks.html"; title="Thanks">Thanks</a></li> + </ul> + </li> + </ul> +<form id="search-form" action="https://www.google.com/search"; method="get" class="navbar-search pull-right" > + <input value="http://atlas.apache.org"; name="sitesearch" type="hidden"/> + <input class="search-query" name="q" id="query" type="text" /> +</form> +<script type="text/javascript">asyncJs( 'https://cse.google.com/brand?form=search-form' )</script> + <iframe src="https://www.facebook.com/plugins/like.php?href=http://atlas.apache.org/atlas-docs&send=false&layout=button_count&show-faces=false&action=like&colorscheme=dark"; + scrolling="no" frameborder="0" + style="border:none; width:100px; height:20px; margin-top: 10px;" class="pull-right" ></iframe> + <script type="text/javascript">asyncJs( 'https://apis.google.com/js/plusone.js' )</script> + <ul class="nav pull-right"><li style="margin-top: 10px;"> + <div class="g-plusone" data-href="http://atlas.apache.org/atlas-docs"; data-size="medium" width="60px" align="right" ></div> + </li></ul> + </div> + </div> + </div> + </div> + <div class="container"> + <div id="banner"> + <div class="pull-left"><a href=".." id="bannerLeft"><img src="images/atlas-logo.png" alt="Apache Atlas" width="200px" height="45px"/></a></div> + <div class="pull-right"></div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + <li class=""><a href="http://www.apache.org"; class="externalLink" title="Apache">Apache</a><span class="divider">/</span></li> + <li class=""><a href="index.html" title="Atlas">Atlas</a><span class="divider">/</span></li> + <li class="active ">Building & Installing Apache Atlas</li> + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-10-31</li> + <li id="projectVersion" class="pull-right">Version: 0.8.3</li> + </ul> + </div> + <div id="bodyColumn" > +<div class="section"> +<h3><a name="Building_.26_Installing_Apache_Atlas"></a>Building & Installing Apache Atlas</h3></div> +<div class="section"> +<h4><a name="Building_Atlas"></a>Building Atlas</h4> +<div class="source"><pre class="prettyprint"> +git clone https://git-wip-us.apache.org/repos/asf/atlas.git atlas +cd atlas +git checkout branch-0.8 +export MAVEN_OPTS="-Xms2g -Xmx2g" +mvn clean -DskipTests install +</pre></div></div> +<div class="section"> +<h4><a name="Packaging_Atlas"></a>Packaging Atlas</h4> +<p>To create Apache Atlas package for deployment in an environment having functional HBase and Solr instances, build with the following command:</p> +<div class="source"><pre class="prettyprint"> +mvn clean -DskipTests package -Pdist +</pre></div> +<p></p> +<ul> +<li>NOTES: +<ul> +<li>Remove option '-DskipTests' to run unit and integration tests</li> +<li>To build a distribution without minified js,css file, build with skipMinify profile. By default js and css files are minified.</li></ul></li></ul> +<p>Above will build Atlas for an environment having functional HBase and Solr instances. Atlas needs to be setup with the following to run in this environment:</p> +<ul> +<li>Configure atlas.graph.storage.hostname (see "Graph persistence engine - HBase" in the <a href="./Configuration.html">Configuration</a> section).</li> +<li>Configure atlas.graph.index.search.solr.zookeeper-url (see "Graph Search Index - Solr" in the <a href="./Configuration.html">Configuration</a> section).</li> +<li>Set HBASE_CONF_DIR to point to a valid HBase config directory (see "Graph persistence engine - HBase" in the <a href="./Configuration.html">Configuration</a> section).</li> +<li>Create the SOLR indices (see "Graph Search Index - Solr" in the <a href="./Configuration.html">Configuration</a> section).</li></ul></div> +<div class="section"> +<h4><a name="Packaging_Atlas_with_Embedded_HBase_.26_Solr"></a>Packaging Atlas with Embedded HBase & Solr</h4> +<p>To create Apache Atlas package that includes HBase and Solr, build with the embedded-hbase-solr profile as shown below:</p> +<div class="source"><pre class="prettyprint"> +mvn clean -DskipTests package -Pdist,embedded-hbase-solr +</pre></div> +<p>Using the embedded-hbase-solr profile will configure Atlas so that an HBase instance and a Solr instance will be started and stopped along with the Atlas server by default.</p></div> +<div class="section"> +<h4><a name="Apache_Atlas_Package"></a>Apache Atlas Package</h4> +<p>Build will create following files, which are used to install Apache Atlas.</p> +<div class="source"><pre class="prettyprint"> +distro/target/apache-atlas-${project.version}-bin.tar.gz +distro/target/apache-atlas-${project.version}-sources.tar.gz +</pre></div></div> +<div class="section"> +<h4><a name="Installing_.26_Running_Atlas"></a>Installing & Running Atlas</h4></div> +<div class="section"> +<h5><a name="Installing_Atlas"></a>Installing Atlas</h5> +<div class="source"><pre class="prettyprint"> +tar -xzvf apache-atlas-${project.version}-bin.tar.gz + +cd atlas-${project.version} +</pre></div></div> +<div class="section"> +<h5><a name="Configuring_Atlas"></a>Configuring Atlas</h5> +<p>By default config directory used by Atlas is {package dir}/conf. To override this set environment variable ATLAS_CONF to the path of the conf dir.</p> +<p>Environment variables needed to run Atlas can be set in atlas-env.sh file in the conf directory. This file will be sourced by Atlas scripts before any commands are executed. The following environment variables are available to set.</p> +<div class="source"><pre class="prettyprint"> +# The java implementation to use. If JAVA_HOME is not found we expect java and jar to be in path +#export JAVA_HOME= + +# any additional java opts you want to set. This will apply to both client and server operations +#export ATLAS_OPTS= + +# any additional java opts that you want to set for client only +#export ATLAS_CLIENT_OPTS= + +# java heap size we want to set for the client. Default is 1024MB +#export ATLAS_CLIENT_HEAP= + +# any additional opts you want to set for atlas service. +#export ATLAS_SERVER_OPTS= + +# java heap size we want to set for the atlas server. Default is 1024MB +#export ATLAS_SERVER_HEAP= + +# What is is considered as atlas home dir. Default is the base locaion of the installed software +#export ATLAS_HOME_DIR= + +# Where log files are stored. Defatult is logs directory under the base install location +#export ATLAS_LOG_DIR= + +# Where pid files are stored. Defatult is logs directory under the base install location +#export ATLAS_PID_DIR= + +# where the atlas titan db data is stored. Defatult is logs/data directory under the base install location +#export ATLAS_DATA_DIR= + +# Where do you want to expand the war file. By Default it is in /server/webapp dir under the base install dir. +#export ATLAS_EXPANDED_WEBAPP_DIR= +</pre></div> +<p><b>Settings to support large number of metadata objects</b></p> +<p>If you plan to store several tens of thousands of metadata objects, it is recommended that you use values tuned for better GC performance of the JVM.</p> +<p>The following values are common server side options:</p> +<div class="source"><pre class="prettyprint"> +export ATLAS_SERVER_OPTS="-server -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+PrintTenuringDistribution -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=dumps/atlas_server.hprof -Xloggc:logs/gc-worker.log -verbose:gc -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=1m -XX:+PrintGCDetails -XX:+PrintHeapAtGC -XX:+PrintGCTimeStamps" +</pre></div> +<p>The <tt>-XX:SoftRefLRUPolicyMSPerMB</tt> option was found to be particularly helpful to regulate GC performance for query heavy workloads with many concurrent users.</p> +<p>The following values are recommended for JDK 7:</p> +<div class="source"><pre class="prettyprint"> +export ATLAS_SERVER_HEAP="-Xms15360m -Xmx15360m -XX:MaxNewSize=3072m -XX:PermSize=100M -XX:MaxPermSize=512m" +</pre></div> +<p>The following values are recommended for JDK 8:</p> +<div class="source"><pre class="prettyprint"> +export ATLAS_SERVER_HEAP="-Xms15360m -Xmx15360m -XX:MaxNewSize=5120m -XX:MetaspaceSize=100M -XX:MaxMetaspaceSize=512m" +</pre></div> +<p><b>NOTE for Mac OS users</b> If you are using a Mac OS, you will need to configure the ATLAS_SERVER_OPTS (explained above).</p> +<p>In {package dir}/conf/atlas-env.sh uncomment the following line</p> +<div class="source"><pre class="prettyprint"> +#export ATLAS_SERVER_OPTS= +</pre></div> +<p>and change it to look as below</p> +<div class="source"><pre class="prettyprint"> +export ATLAS_SERVER_OPTS="-Djava.awt.headless=true -Djava.security.krb5.realm= -Djava.security.krb5.kdc=" +</pre></div> +<p><b>Hbase as the Storage Backend for the Graph Repository</b></p> +<p>By default, Atlas uses Titan as the graph repository and is the only graph repository implementation available currently. The HBase versions currently supported are 1.1.x. For configuring ATLAS graph persistence on HBase, please see "Graph persistence engine - HBase" in the <a href="./Configuration.html">Configuration</a> section for more details.</p> +<p>HBase tables used by Atlas can be set using the following configurations:</p> +<div class="source"><pre class="prettyprint"> +atlas.graph.storage.hbase.table=apache_atlas_titan +atlas.audit.hbase.tablename=apache_atlas_entity_audit +</pre></div> +<p><b>Configuring SOLR as the Indexing Backend for the Graph Repository</b></p> +<p>By default, Atlas uses Titan as the graph repository and is the only graph repository implementation available currently. For configuring Titan to work with Solr, please follow the instructions below</p> +<p></p> +<ul> +<li>Install solr if not already running. The version of SOLR supported is 5.2.1. Could be installed from <a class="externalLink" href="http://archive.apache.org/dist/lucene/solr/5.2.1/solr-5.2.1.tgz";>http://archive.apache.org/dist/lucene/solr/5.2.1/solr-5.2.1.tgz</a></li></ul> +<p></p> +<ul> +<li>Start solr in cloud mode.</li></ul>SolrCloud mode uses a ZooKeeper Service as a highly available, central location for cluster management. For a small cluster, running with an existing ZooKeeper quorum should be fine. For larger clusters, you would want to run separate multiple ZooKeeper quorum with atleast 3 servers. Note: Atlas currently supports solr in "cloud" mode only. "http" mode is not supported. For more information, refer solr documentation - <a class="externalLink" href="https://cwiki.apache.org/confluence/display/solr/SolrCloud";>https://cwiki.apache.org/confluence/display/solr/SolrCloud</a> +<p></p> +<ul> +<li>For e.g., to bring up a Solr node listening on port 8983 on a machine, you can use the command:</li></ul> +<div class="source"><pre class="prettyprint"> + $SOLR_HOME/bin/solr start -c -z <zookeeper_host:port> -p 8983 +</pre></div> +<p></p> +<ul> +<li>Run the following commands from SOLR_BIN (e.g. $SOLR_HOME/bin) directory to create collections in Solr corresponding to the indexes that Atlas uses. In the case that the ATLAS and SOLR instance are on 2 different hosts, first copy the required configuration files from ATLAS_HOME/conf/solr on the ATLAS instance host to the Solr instance host. SOLR_CONF in the below mentioned commands refer to the directory where the solr configuration files have been copied to on Solr host:</li></ul> +<div class="source"><pre class="prettyprint"> + $SOLR_BIN/solr create -c vertex_index -d SOLR_CONF -shards #numShards -replicationFactor #replicationFactor + $SOLR_BIN/solr create -c edge_index -d SOLR_CONF -shards #numShards -replicationFactor #replicationFactor + $SOLR_BIN/solr create -c fulltext_index -d SOLR_CONF -shards #numShards -replicationFactor #replicationFactor +</pre></div> +<p>Note: If numShards and replicationFactor are not specified, they default to 1 which suffices if you are trying out solr with ATLAS on a single node instance. Otherwise specify numShards according to the number of hosts that are in the Solr cluster and the maxShardsPerNode configuration. The number of shards cannot exceed the total number of Solr nodes in your !SolrCloud cluster.</p> +<p>The number of replicas (replicationFactor) can be set according to the redundancy required.</p> +<p>Also note that solr will automatically be called to create the indexes when the Atlas server is started if the SOLR_BIN and SOLR_CONF environment variables are set and the search indexing backend is set to 'solr5'.</p> +<p></p> +<ul> +<li>Change ATLAS configuration to point to the Solr instance setup. Please make sure the following configurations are set to the below values in ATLAS_HOME/conf/atlas-application.properties</li></ul> +<div class="source"><pre class="prettyprint"> + atlas.graph.index.search.backend=solr5 + atlas.graph.index.search.solr.mode=cloud + atlas.graph.index.search.solr.zookeeper-url=<the ZK quorum setup for solr as comma separated value> eg: 10.1.6.4:2181,10.1.6.5:2181 + atlas.graph.index.search.solr.zookeeper-connect-timeout=<SolrCloud Zookeeper Connection Timeout>. Default value is 60000 ms + atlas.graph.index.search.solr.zookeeper-session-timeout=<SolrCloud Zookeeper Session Timeout>. Default value is 60000 ms +</pre></div> +<p></p> +<ul> +<li>Restart Atlas</li></ul> +<p>For more information on Titan solr configuration , please refer <a class="externalLink" href="http://s3.thinkaurelius.com/docs/titan/0.5.4/solr.htm";>http://s3.thinkaurelius.com/docs/titan/0.5.4/solr.htm</a></p> +<p>Pre-requisites for running Solr in cloud mode * Memory - Solr is both memory and CPU intensive. Make sure the server running Solr has adequate memory, CPU and disk. Solr works well with 32GB RAM. Plan to provide as much memory as possible to Solr process * Disk - If the number of entities that need to be stored are large, plan to have at least 500 GB free space in the volume where Solr is going to store the index data * SolrCloud has support for replication and sharding. It is highly recommended to use SolrCloud with at least two Solr nodes running on different servers with replication enabled. If using SolrCloud, then you also need ZooKeeper installed and configured with 3 or 5 ZooKeeper nodes</p> +<p><b>Configuring Kafka Topics</b></p> +<p>Atlas uses Kafka to ingest metadata from other components at runtime. This is described in the <a href="./Architecture.html">Architecture page</a> in more detail. Depending on the configuration of Kafka, sometimes you might need to setup the topics explicitly before using Atlas. To do so, Atlas provides a script <tt>bin/atlas_kafka_setup.py</tt> which can be run from the Atlas server. In some environments, the hooks might start getting used first before Atlas server itself is setup. In such cases, the topics can be run on the hosts where hooks are installed using a similar script <tt>hook-bin/atlas_kafka_setup_hook.py</tt>. Both these use configuration in <tt>atlas-application.properties</tt> for setting up the topics. Please refer to the <a href="./Configuration.html">Configuration page</a> for these details.</p></div> +<div class="section"> +<h5><a name="Setting_up_Atlas"></a>Setting up Atlas</h5> +<p>There are a few steps that setup dependencies of Atlas. One such example is setting up the Titan schema in the storage backend of choice. In a simple single server setup, these are automatically setup with default configuration when the server first accesses these dependencies.</p> +<p>However, there are scenarios when we may want to run setup steps explicitly as one time operations. For example, in a multiple server scenario using <a href="./HighAvailability.html">High Availability</a>, it is preferable to run setup steps from one of the server instances the first time, and then start the services.</p> +<p>To run these steps one time, execute the command <tt>bin/atlas_start.py -setup</tt> from a single Atlas server instance.</p> +<p>However, the Atlas server does take care of parallel executions of the setup steps. Also, running the setup steps multiple times is idempotent. Therefore, if one chooses to run the setup steps as part of server startup, for convenience, then they should enable the configuration option <tt>atlas.server.run.setup.on.start</tt> by defining it with the value <tt>true</tt> in the <tt>atlas-application.properties</tt> file.</p></div> +<div class="section"> +<h5><a name="Starting_Atlas_Server"></a>Starting Atlas Server</h5> +<div class="source"><pre class="prettyprint"> +bin/atlas_start.py [-port <port>] +</pre></div></div> +<div class="section"> +<h4><a name="Using_Atlas"></a>Using Atlas</h4> +<p></p> +<ul> +<li>Verify if the server is up and running</li></ul> +<div class="source"><pre class="prettyprint"> + curl -v -u username:password http://localhost:21000/api/atlas/admin/version + {"Version":"v0.1"} +</pre></div> +<p></p> +<ul> +<li>Access Atlas UI using a browser: <a class="externalLink" href="http://localhost:21000";>http://localhost:21000</a></li></ul> +<p></p> +<ul> +<li>Quick start model - sample model and data</li></ul> +<div class="source"><pre class="prettyprint"> + bin/quick_start.py [<atlas endpoint>] +</pre></div> +<p></p> +<ul> +<li>List the types in the repository</li></ul> +<div class="source"><pre class="prettyprint"> + curl -v -u username:password http://localhost:21000/api/atlas/v2/types/typedefs/headers + [ {"guid":"fa421be8-c21b-4cf8-a226-fdde559ad598","name":"Referenceable","category":"ENTITY"}, + {"guid":"7f3f5712-521d-450d-9bb2-ba996b6f2a4e","name":"Asset","category":"ENTITY"}, + {"guid":"84b02fa0-e2f4-4cc4-8b24-d2371cd00375","name":"DataSet","category":"ENTITY"}, + {"guid":"f93975d5-5a5c-41da-ad9d-eb7c4f91a093","name":"Process","category":"ENTITY"}, + {"guid":"79dcd1f9-f350-4f7b-b706-5bab416f8206","name":"Infrastructure","category":"ENTITY"} + ] +</pre></div> +<p></p> +<ul> +<li>List the instances for a given type</li></ul> +<div class="source"><pre class="prettyprint"> + curl -v -u username:password http://localhost:21000/api/atlas/v2/search/basic?typeName=hive_db + { + "queryType":"BASIC", + "searchParameters":{ + "typeName":"hive_db", + "excludeDeletedEntities":false, + "includeClassificationAttributes":false, + "includeSubTypes":true, + "includeSubClassifications":true, + "limit":100, + "offset":0 + }, + "entities":[ + { + "typeName":"hive_db", + "guid":"5d900c19-094d-4681-8a86-4eb1d6ffbe89", + "status":"ACTIVE", + "displayText":"default", + "classificationNames":[], + "attributes":{ + "owner":"public", + "createTime":null, + "qualifiedName":"default@cl1", + "name":"default", + "description":"Default Hive database" + } + }, + { + "typeName":"hive_db", + "guid":"3a0b14b0-ab85-4b65-89f2-e418f3f7f77c", + "status":"ACTIVE", + "displayText":"finance", + "classificationNames":[], + "attributes":{ + "owner":"hive", + "createTime":null, + "qualifiedName":"finance@cl1", + "name":"finance", + "description":null + } + } + ] + } +</pre></div> +<p></p> +<ul> +<li>Search for entities</li></ul>curl -v -u username:password <a class="externalLink" href="http://localhost:21000/api/atlas/v2/search/dsl?query=hive_db%20where%20name='default'";>http://localhost:21000/api/atlas/v2/search/dsl?query=hive_db%20where%20name='default'</a> { "queryType":"DSL", "queryText":"hive_db where name='default'", "entities":[ { "typeName":"hive_db", "guid":"5d900c19-094d-4681-8a86-4eb1d6ffbe89", "status":"ACTIVE", "displayText":"default", "classificationNames":[], "attributes":{ "owner":"public", "createTime":null, "qualifiedName":"default@cl1", "name":"default", "description": "Default Hive database" } } ] }</verbatim></div> +<div class="section"> +<h4><a name="Stopping_Atlas_Server"></a>Stopping Atlas Server</h4> +<div class="source"><pre class="prettyprint"> +bin/atlas_stop.py +</pre></div></div> +<div class="section"> +<h4><a name="Troubleshooting"></a>Troubleshooting</h4></div> +<div class="section"> +<h5><a name="Setup_issues"></a>Setup issues</h5> +<p>If the setup of Atlas service fails due to any reason, the next run of setup (either by an explicit invocation of <tt>atlas_start.py -setup</tt> or by enabling the configuration option <tt>atlas.server.run.setup.on.start</tt>) will fail with a message such as <tt>A previous setup run may not have completed cleanly.</tt>. In such cases, you would need to manually ensure the setup can run and delete the Zookeeper node at <tt>/apache_atlas/setup_in_progress</tt> before attempting to run setup again.</p> +<p>If the setup failed due to HBase Titan schema setup errors, it may be necessary to repair the HBase schema. If no data has been stored, one can also disable and drop the HBase tables used by Atlas and run setup again.</p></div> + </div> + </div> + <hr/> + <footer> + <div class="container"> + <div class="row"> +Copyright ̩ 2018 The Apache Software Foundation, Licensed under the Apache License, Version 2.0. + </div> + <p id="poweredBy" class="pull-right"><a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"><img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /></a> +</p> + </div> + </footer> + </body> +</html>
http://git-wip-us.apache.org/repos/asf/atlas-website/blob/2d4dc1a0/0.8.3/Notification-Entity.html ---------------------------------------------------------------------- diff --git a/0.8.3/Notification-Entity.html b/0.8.3/Notification-Entity.html new file mode 100644 index 0000000..4aa0bb0 --- /dev/null +++ b/0.8.3/Notification-Entity.html @@ -0,0 +1,158 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/twiki/Notification-Entity.twiki at 2018-10-31 + | Rendered using Apache Maven Fluido Skin 1.7 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20181031" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Apache Atlas – Entity Change Notifications</title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.7.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + <script type="text/javascript" src="./js/apache-maven-fluido-1.7.min.js"></script> + </head> + <body class="topBarEnabled"> + <div id="topbar" class="navbar navbar-fixed-top "> + <div class="navbar-inner"> + <div class="container" style="width: 68%;"><div class="nav-collapse"> + <ul class="nav"> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Atlas <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="index.html" title="About">About</a></li> + <li><a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="Wiki">Wiki</a></li> + <li><a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="News">News</a></li> + <li><a href="https://git-wip-us.apache.org/repos/asf/atlas.git"; title="Git">Git</a></li> + <li><a href="https://issues.apache.org/jira/browse/ATLAS"; title="Jira">Jira</a></li> + <li><a href="https://cwiki.apache.org/confluence/display/ATLAS/PoweredBy"; title="Powered by">Powered by</a></li> + <li><a href="http://blogs.apache.org/atlas/"; title="Blog">Blog</a></li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Project Information <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="project-info.html" title="Summary">Summary</a></li> + <li><a href="mail-lists.html" title="Mailing Lists">Mailing Lists</a></li> + <li><a href="http://webchat.freenode.net?channels=apacheatlas&uio=d4"; title="IRC">IRC</a></li> + <li><a href="team-list.html" title="Team">Team</a></li> + <li><a href="issue-tracking.html" title="Issue Tracking">Issue Tracking</a></li> + <li><a href="source-repository.html" title="Source Repository">Source Repository</a></li> + <li><a href="license.html" title="License">License</a></li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Releases <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="http://www.apache.org/dyn/closer.cgi/atlas/0.8.2/"; title="0.8.2">0.8.2</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.8.1/"; title="0.8.1">0.8.1</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.8.0-incubating/"; title="0.8-incubating">0.8-incubating</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.7.1-incubating/"; title="0.7.1-incubating">0.7.1-incubating</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.7.0-incubating/"; title="0.7-incubating">0.7-incubating</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.6.0-incubating/"; title="0.6-incubating">0.6-incubating</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.5.0-incubating/"; title="0.5-incubating">0.5-incubating</a></li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="../index.html" title="latest">latest</a></li> + <li><a href="../0.8.2/index.html" title="0.8.2">0.8.2</a></li> + <li><a href="../0.8.1/index.html" title="0.8.1">0.8.1</a></li> + <li><a href="../0.8.0-incubating/index.html" title="0.8-incubating">0.8-incubating</a></li> + <li><a href="../0.7.1-incubating/index.html" title="0.7.1-incubating">0.7.1-incubating</a></li> + <li><a href="../0.7.0-incubating/index.html" title="0.7-incubating">0.7-incubating</a></li> + <li><a href="../0.6.0-incubating/index.html" title="0.6-incubating">0.6-incubating</a></li> + <li><a href="../0.5.0-incubating/index.html" title="0.5-incubating">0.5-incubating</a></li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">ASF <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="http://www.apache.org/foundation/how-it-works.html"; title="How Apache Works">How Apache Works</a></li> + <li><a href="http://www.apache.org/foundation/"; title="Foundation">Foundation</a></li> + <li><a href="http://www.apache.org/foundation/sponsorship.html"; title="Sponsoring Apache">Sponsoring Apache</a></li> + <li><a href="http://www.apache.org/foundation/thanks.html"; title="Thanks">Thanks</a></li> + </ul> + </li> + </ul> +<form id="search-form" action="https://www.google.com/search"; method="get" class="navbar-search pull-right" > + <input value="http://atlas.apache.org"; name="sitesearch" type="hidden"/> + <input class="search-query" name="q" id="query" type="text" /> +</form> +<script type="text/javascript">asyncJs( 'https://cse.google.com/brand?form=search-form' )</script> + <iframe src="https://www.facebook.com/plugins/like.php?href=http://atlas.apache.org/atlas-docs&send=false&layout=button_count&show-faces=false&action=like&colorscheme=dark"; + scrolling="no" frameborder="0" + style="border:none; width:100px; height:20px; margin-top: 10px;" class="pull-right" ></iframe> + <script type="text/javascript">asyncJs( 'https://apis.google.com/js/plusone.js' )</script> + <ul class="nav pull-right"><li style="margin-top: 10px;"> + <div class="g-plusone" data-href="http://atlas.apache.org/atlas-docs"; data-size="medium" width="60px" align="right" ></div> + </li></ul> + </div> + </div> + </div> + </div> + <div class="container"> + <div id="banner"> + <div class="pull-left"><a href=".." id="bannerLeft"><img src="images/atlas-logo.png" alt="Apache Atlas" width="200px" height="45px"/></a></div> + <div class="pull-right"></div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + <li class=""><a href="http://www.apache.org"; class="externalLink" title="Apache">Apache</a><span class="divider">/</span></li> + <li class=""><a href="index.html" title="Atlas">Atlas</a><span class="divider">/</span></li> + <li class="active ">Entity Change Notifications</li> + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-10-31</li> + <li id="projectVersion" class="pull-right">Version: 0.8.3</li> + </ul> + </div> + <div id="bodyColumn" > +<div class="section"> +<h2><a name="Entity_Change_Notifications"></a>Entity Change Notifications</h2> +<p>To receive Atlas entity notifications a consumer should be obtained through the notification interface. Entity change notifications are sent every time a change is made to an entity. Operations that result in an entity change notification are:</p> +<ul> +<li><code>ENTITY_CREATE</code> - Create a new entity.</li> +<li><code>ENTITY_UPDATE</code> - Update an attribute of an existing entity.</li> +<li><code>TRAIT_ADD</code> - Add a trait to an entity.</li> +<li><code>TRAIT_DELETE</code> - Delete a trait from an entity.</li></ul> +<div class="source"><pre class="prettyprint"> + // Obtain provider through injection… + Provider<NotificationInterface> provider; + + // Get the notification interface + NotificationInterface notification = provider.get(); + + // Create consumers + List<NotificationConsumer<EntityNotification>> consumers = + notification.createConsumers(NotificationInterface.NotificationType.ENTITIES, 1); + +</pre></div> +<p>The consumer exposes the Iterator interface that should be used to get the entity notifications as they are posted. The hasNext() method blocks until a notification is available.</p> +<div class="source"><pre class="prettyprint"> + while(consumer.hasNext()) { + EntityNotification notification = consumer.next(); + + IReferenceableInstance entity = notification.getEntity(); + … + } + +</pre></div></div> + </div> + </div> + <hr/> + <footer> + <div class="container"> + <div class="row"> +Copyright é 2018 The Apache Software Foundation, Licensed under the Apache License, Version 2.0. + </div> + <p id="poweredBy" class="pull-right"><a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"><img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /></a> +</p> + </div> + </footer> + </body> +</html> http://git-wip-us.apache.org/repos/asf/atlas-website/blob/2d4dc1a0/0.8.3/QuickStart.html ---------------------------------------------------------------------- diff --git a/0.8.3/QuickStart.html b/0.8.3/QuickStart.html new file mode 100644 index 0000000..f33eaf7 --- /dev/null +++ b/0.8.3/QuickStart.html @@ -0,0 +1,151 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/twiki/QuickStart.twiki at 2018-10-31 + | Rendered using Apache Maven Fluido Skin 1.7 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20181031" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Apache Atlas – Quick Start</title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.7.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + <script type="text/javascript" src="./js/apache-maven-fluido-1.7.min.js"></script> + </head> + <body class="topBarEnabled"> + <div id="topbar" class="navbar navbar-fixed-top "> + <div class="navbar-inner"> + <div class="container" style="width: 68%;"><div class="nav-collapse"> + <ul class="nav"> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Atlas <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="index.html" title="About">About</a></li> + <li><a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="Wiki">Wiki</a></li> + <li><a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="News">News</a></li> + <li><a href="https://git-wip-us.apache.org/repos/asf/atlas.git"; title="Git">Git</a></li> + <li><a href="https://issues.apache.org/jira/browse/ATLAS"; title="Jira">Jira</a></li> + <li><a href="https://cwiki.apache.org/confluence/display/ATLAS/PoweredBy"; title="Powered by">Powered by</a></li> + <li><a href="http://blogs.apache.org/atlas/"; title="Blog">Blog</a></li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Project Information <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="project-info.html" title="Summary">Summary</a></li> + <li><a href="mail-lists.html" title="Mailing Lists">Mailing Lists</a></li> + <li><a href="http://webchat.freenode.net?channels=apacheatlas&uio=d4"; title="IRC">IRC</a></li> + <li><a href="team-list.html" title="Team">Team</a></li> + <li><a href="issue-tracking.html" title="Issue Tracking">Issue Tracking</a></li> + <li><a href="source-repository.html" title="Source Repository">Source Repository</a></li> + <li><a href="license.html" title="License">License</a></li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Releases <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="http://www.apache.org/dyn/closer.cgi/atlas/0.8.2/"; title="0.8.2">0.8.2</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.8.1/"; title="0.8.1">0.8.1</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.8.0-incubating/"; title="0.8-incubating">0.8-incubating</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.7.1-incubating/"; title="0.7.1-incubating">0.7.1-incubating</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.7.0-incubating/"; title="0.7-incubating">0.7-incubating</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.6.0-incubating/"; title="0.6-incubating">0.6-incubating</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.5.0-incubating/"; title="0.5-incubating">0.5-incubating</a></li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="../index.html" title="latest">latest</a></li> + <li><a href="../0.8.2/index.html" title="0.8.2">0.8.2</a></li> + <li><a href="../0.8.1/index.html" title="0.8.1">0.8.1</a></li> + <li><a href="../0.8.0-incubating/index.html" title="0.8-incubating">0.8-incubating</a></li> + <li><a href="../0.7.1-incubating/index.html" title="0.7.1-incubating">0.7.1-incubating</a></li> + <li><a href="../0.7.0-incubating/index.html" title="0.7-incubating">0.7-incubating</a></li> + <li><a href="../0.6.0-incubating/index.html" title="0.6-incubating">0.6-incubating</a></li> + <li><a href="../0.5.0-incubating/index.html" title="0.5-incubating">0.5-incubating</a></li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">ASF <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="http://www.apache.org/foundation/how-it-works.html"; title="How Apache Works">How Apache Works</a></li> + <li><a href="http://www.apache.org/foundation/"; title="Foundation">Foundation</a></li> + <li><a href="http://www.apache.org/foundation/sponsorship.html"; title="Sponsoring Apache">Sponsoring Apache</a></li> + <li><a href="http://www.apache.org/foundation/thanks.html"; title="Thanks">Thanks</a></li> + </ul> + </li> + </ul> +<form id="search-form" action="https://www.google.com/search"; method="get" class="navbar-search pull-right" > + <input value="http://atlas.apache.org"; name="sitesearch" type="hidden"/> + <input class="search-query" name="q" id="query" type="text" /> +</form> +<script type="text/javascript">asyncJs( 'https://cse.google.com/brand?form=search-form' )</script> + <iframe src="https://www.facebook.com/plugins/like.php?href=http://atlas.apache.org/atlas-docs&send=false&layout=button_count&show-faces=false&action=like&colorscheme=dark"; + scrolling="no" frameborder="0" + style="border:none; width:100px; height:20px; margin-top: 10px;" class="pull-right" ></iframe> + <script type="text/javascript">asyncJs( 'https://apis.google.com/js/plusone.js' )</script> + <ul class="nav pull-right"><li style="margin-top: 10px;"> + <div class="g-plusone" data-href="http://atlas.apache.org/atlas-docs"; data-size="medium" width="60px" align="right" ></div> + </li></ul> + </div> + </div> + </div> + </div> + <div class="container"> + <div id="banner"> + <div class="pull-left"><a href=".." id="bannerLeft"><img src="images/atlas-logo.png" alt="Apache Atlas" width="200px" height="45px"/></a></div> + <div class="pull-right"></div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + <li class=""><a href="http://www.apache.org"; class="externalLink" title="Apache">Apache</a><span class="divider">/</span></li> + <li class=""><a href="index.html" title="Atlas">Atlas</a><span class="divider">/</span></li> + <li class="active ">Quick Start</li> + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-10-31</li> + <li id="projectVersion" class="pull-right">Version: 0.8.3</li> + </ul> + </div> + <div id="bodyColumn" > +<div class="section"> +<h2><a name="Quick_Start"></a>Quick Start</h2></div> +<div class="section"> +<h3><a name="Introduction"></a>Introduction</h3> +<p>Quick start is a simple client that adds a few sample type definitions modeled after the example shown below. It also adds sample entities along with traits as shown in the instance graph below.</p></div> +<div class="section"> +<h4><a name="Example_Type_Definitions"></a>Example Type Definitions</h4> +<p><img src="images/twiki/guide-class-diagram.png" alt="" /></p></div> +<div class="section"> +<h4><a name="Example_Instance_Graph"></a>Example Instance Graph</h4> +<p><img src="images/twiki/guide-instance-graph.png" alt="" /></p></div> +<div class="section"> +<h3><a name="Running_the_example"></a>Running the example</h3> +<p>This will add sample types and instances along with traits as shown in the instance graph above.</p> +<p></p> +<ul> +<li>bin/quick-start.sh</li></ul></div> +<div class="section"> +<h3><a name="Dashboard"></a>Dashboard</h3> +<p>A simple dashboard with search is available.</p> +<p></p> +<ul> +<li><a class="externalLink" href="http://localhost:21000/";>http://localhost:21000/</a></li></ul></div> + </div> + </div> + <hr/> + <footer> + <div class="container"> + <div class="row"> +Copyright é 2018 The Apache Software Foundation, Licensed under the Apache License, Version 2.0. + </div> + <p id="poweredBy" class="pull-right"><a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"><img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /></a> +</p> + </div> + </footer> + </body> +</html> http://git-wip-us.apache.org/repos/asf/atlas-website/blob/2d4dc1a0/0.8.3/Search.html ---------------------------------------------------------------------- diff --git a/0.8.3/Search.html b/0.8.3/Search.html new file mode 100644 index 0000000..36cc04e --- /dev/null +++ b/0.8.3/Search.html @@ -0,0 +1,417 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/twiki/Search.twiki at 2018-10-31 + | Rendered using Apache Maven Fluido Skin 1.7 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20181031" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Apache Atlas – Search</title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.7.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + <script type="text/javascript" src="./js/apache-maven-fluido-1.7.min.js"></script> + </head> + <body class="topBarEnabled"> + <div id="topbar" class="navbar navbar-fixed-top "> + <div class="navbar-inner"> + <div class="container" style="width: 68%;"><div class="nav-collapse"> + <ul class="nav"> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Atlas <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="index.html" title="About">About</a></li> + <li><a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="Wiki">Wiki</a></li> + <li><a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="News">News</a></li> + <li><a href="https://git-wip-us.apache.org/repos/asf/atlas.git"; title="Git">Git</a></li> + <li><a href="https://issues.apache.org/jira/browse/ATLAS"; title="Jira">Jira</a></li> + <li><a href="https://cwiki.apache.org/confluence/display/ATLAS/PoweredBy"; title="Powered by">Powered by</a></li> + <li><a href="http://blogs.apache.org/atlas/"; title="Blog">Blog</a></li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Project Information <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="project-info.html" title="Summary">Summary</a></li> + <li><a href="mail-lists.html" title="Mailing Lists">Mailing Lists</a></li> + <li><a href="http://webchat.freenode.net?channels=apacheatlas&uio=d4"; title="IRC">IRC</a></li> + <li><a href="team-list.html" title="Team">Team</a></li> + <li><a href="issue-tracking.html" title="Issue Tracking">Issue Tracking</a></li> + <li><a href="source-repository.html" title="Source Repository">Source Repository</a></li> + <li><a href="license.html" title="License">License</a></li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Releases <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="http://www.apache.org/dyn/closer.cgi/atlas/0.8.2/"; title="0.8.2">0.8.2</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.8.1/"; title="0.8.1">0.8.1</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.8.0-incubating/"; title="0.8-incubating">0.8-incubating</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.7.1-incubating/"; title="0.7.1-incubating">0.7.1-incubating</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.7.0-incubating/"; title="0.7-incubating">0.7-incubating</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.6.0-incubating/"; title="0.6-incubating">0.6-incubating</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.5.0-incubating/"; title="0.5-incubating">0.5-incubating</a></li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="../index.html" title="latest">latest</a></li> + <li><a href="../0.8.2/index.html" title="0.8.2">0.8.2</a></li> + <li><a href="../0.8.1/index.html" title="0.8.1">0.8.1</a></li> + <li><a href="../0.8.0-incubating/index.html" title="0.8-incubating">0.8-incubating</a></li> + <li><a href="../0.7.1-incubating/index.html" title="0.7.1-incubating">0.7.1-incubating</a></li> + <li><a href="../0.7.0-incubating/index.html" title="0.7-incubating">0.7-incubating</a></li> + <li><a href="../0.6.0-incubating/index.html" title="0.6-incubating">0.6-incubating</a></li> + <li><a href="../0.5.0-incubating/index.html" title="0.5-incubating">0.5-incubating</a></li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">ASF <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="http://www.apache.org/foundation/how-it-works.html"; title="How Apache Works">How Apache Works</a></li> + <li><a href="http://www.apache.org/foundation/"; title="Foundation">Foundation</a></li> + <li><a href="http://www.apache.org/foundation/sponsorship.html"; title="Sponsoring Apache">Sponsoring Apache</a></li> + <li><a href="http://www.apache.org/foundation/thanks.html"; title="Thanks">Thanks</a></li> + </ul> + </li> + </ul> +<form id="search-form" action="https://www.google.com/search"; method="get" class="navbar-search pull-right" > + <input value="http://atlas.apache.org"; name="sitesearch" type="hidden"/> + <input class="search-query" name="q" id="query" type="text" /> +</form> +<script type="text/javascript">asyncJs( 'https://cse.google.com/brand?form=search-form' )</script> + <iframe src="https://www.facebook.com/plugins/like.php?href=http://atlas.apache.org/atlas-docs&send=false&layout=button_count&show-faces=false&action=like&colorscheme=dark"; + scrolling="no" frameborder="0" + style="border:none; width:100px; height:20px; margin-top: 10px;" class="pull-right" ></iframe> + <script type="text/javascript">asyncJs( 'https://apis.google.com/js/plusone.js' )</script> + <ul class="nav pull-right"><li style="margin-top: 10px;"> + <div class="g-plusone" data-href="http://atlas.apache.org/atlas-docs"; data-size="medium" width="60px" align="right" ></div> + </li></ul> + </div> + </div> + </div> + </div> + <div class="container"> + <div id="banner"> + <div class="pull-left"><a href=".." id="bannerLeft"><img src="images/atlas-logo.png" alt="Apache Atlas" width="200px" height="45px"/></a></div> + <div class="pull-right"></div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + <li class=""><a href="http://www.apache.org"; class="externalLink" title="Apache">Apache</a><span class="divider">/</span></li> + <li class=""><a href="index.html" title="Atlas">Atlas</a><span class="divider">/</span></li> + <li class="active ">Search</li> + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-10-31</li> + <li id="projectVersion" class="pull-right">Version: 0.8.3</li> + </ul> + </div> + <div id="bodyColumn" > +<div class="section"> +<h2><a name="Search"></a>Search</h2> +<p>Atlas exposes search over the metadata in two ways:</p> +<ul> +<li>Basic Search</li> +<li>Advanced Search (DSL or Full-Text)</li></ul></div> +<div class="section"> +<h3><a name="Basic_search"></a>Basic search</h3> +<p>The basic search allows you to query using typename of an entity, associated classification/tag and has support for filtering on the entity attribute(s) as well as the classification/tag attributes.</p> +<p>The entire query structure can be represented using the following JSON structure (called SearchParameters)</p> +<div class="source"><pre class="prettyprint"> +{ + "typeName": "hive_table", + "excludeDeletedEntities": true, + "classification" : "", + "query": "", + "limit": 25, + "offset": 0, + "entityFilters": { + "attributeName": "name", + "operator": "contains", + "attributeValue": "testtable" + }, + "tagFilters": null, + "attributes": [""] +} + +</pre></div> +<p><b><i>Field description</i></b></p> +<p></p> +<ul> +<li>typeName: The type of entity to look for</li> +<li>excludeDeletedEntities: Should the search include deleted entities too (default: true)</li> +<li>classification: Only include entities with given Classification/tag</li> +<li>query: Any free text occurrence that the entity should have (generic/wildcard queries might be slow)</li> +<li>limit: Max number of results to fetch</li> +<li>offset: Starting offset of the result set (useful for pagination)</li> +<li>entityFilters: Entity Attribute filter(s)</li> +<li>tagFilters: Classification/tag Attribute filter(s)</li> +<li>attributes: Attributes to include in the search result (default: include any attribute present in the filter)</li></ul> +<p>Attribute based filtering can be done on multiple attributes with AND/OR condition.</p> +<p><b>NOTE: The tagFilters and entityFilters field have same JSON structure.</b></p> +<p><b><i>Examples of filtering (for hive_table attributes)</i></b></p> +<ul> +<li>Single attribute</li></ul> +<div class="source"><pre class="prettyprint"> + { + "typeName": "hive_table", + "excludeDeletedEntities": true, + "classification" : "", + "query": "", + "limit": 50, + "offset": 0, + "entityFilters": { + "attributeName": "name", + "operator": "contains", + "attributeValue": "testtable" + }, + "tagFilters": null, + "attributes": [""] + } + +</pre></div> +<p></p> +<ul> +<li>Multi-attribute with OR</li></ul> +<div class="source"><pre class="prettyprint"> + { + "typeName": "hive_table", + "excludeDeletedEntities": true, + "classification" : "", + "query": "", + "limit": 50, + "offset": 0, + "entityFilters": { + "condition": "OR", + "criterion": [ + { + "attributeName": "name", + "operator": "contains", + "attributeValue": "testtable" + }, + { + "attributeName": "owner", + "operator": "eq", + "attributeValue": "admin" + } + ] + }, + "tagFilters": null, + "attributes": [""] + } + +</pre></div> +<p></p> +<ul> +<li>Multi-attribute with AND</li></ul> +<div class="source"><pre class="prettyprint"> + { + "typeName": "hive_table", + "excludeDeletedEntities": true, + "classification" : "", + "query": "", + "limit": 50, + "offset": 0, + "entityFilters": { + "condition": "AND", + "criterion": [ + { + "attributeName": "name", + "operator": "contains", + "attributeValue": "testtable" + }, + { + "attributeName": "owner", + "operator": "eq", + "attributeValue": "admin" + } + ] + }, + "tagFilters": null, + "attributes": [""] + } + +</pre></div> +<p><b><i>Supported operators for filtering</i></b></p> +<p></p> +<ul> +<li>LT (symbols: <, lt) works with Numeric, Date attributes</li> +<li>GT (symbols: >, gt) works with Numeric, Date attributes</li> +<li>LTE (symbols: <=, lte) works with Numeric, Date attributes</li> +<li>GTE (symbols: >=, gte) works with Numeric, Date attributes</li> +<li>EQ (symbols: eq, =) works with Numeric, Date, String attributes</li> +<li>NEQ (symbols: neq, !=) works with Numeric, Date, String attributes</li> +<li>LIKE (symbols: like, LIKE) works with String attributes</li> +<li>STARTS_WITH (symbols: startsWith, STARTSWITH) works with String attributes</li> +<li>ENDS_WITH (symbols: endsWith, ENDSWITH) works with String attributes</li> +<li>CONTAINS (symbols: contains, CONTAINS) works with String attributes</li></ul> +<p><b><i>CURL Samples</i></b></p> +<div class="source"><pre class="prettyprint"> + curl -sivk -g + -u <user>:<password> + -X POST + -d '{ + "typeName": "hive_table", + "excludeDeletedEntities": true, + "classification" : "", + "query": "", + "limit": 50, + "offset": 0, + "entityFilters": { + "condition": "AND", + "criterion": [ + { + "attributeName": "name", + "operator": "contains", + "attributeValue": "testtable" + }, + { + "attributeName": "owner", + "operator": "eq", + "attributeValue": "admin" + } + ] + }, + "tagFilters": null, + "attributes": [""] + }' + <protocol>://<atlas_host>:<atlas_port>/api/atlas/v2/search/basic + +</pre></div></div> +<div class="section"> +<h3><a name="Advanced_Search"></a>Advanced Search</h3></div> +<div class="section"> +<h4><a name="Search_DSL_Grammar"></a>Search DSL Grammar</h4> +<p>The DSL exposes an SQL like query language for searching the metadata based on the type system. The grammar for the DSL is below.</p> +<div class="source"><pre class="prettyprint"> +queryWithPath: query ~ opt(WITHPATH) + +query: querySrc ~ opt(loopExpression) ~ opt(groupByExpr) ~ opt(selectClause) ~ opt(orderby) ~ opt(limitOffset) + +querySrc: rep1sep(singleQrySrc, opt(COMMA)) + +singleQrySrc = FROM ~ fromSrc ~ opt(WHERE) ~ opt(expr ^? notIdExpression) | + WHERE ~ (expr ^? notIdExpression) | + expr ^? notIdExpression | + fromSrc ~ opt(WHERE) ~ opt(expr ^? notIdExpression) + +fromSrc: identifier ~ AS ~ alias | identifier + +groupByExpr = GROUPBY ~ (LPAREN ~> rep1sep(selectExpression, COMMA) <~ RPAREN) + +orderby: ORDERBY ~ expr ~ opt (sortOrder) + +limitOffset: LIMIT ~ lmt ~ opt (offset) + +offset: OFFSET ~ offsetValue + +sortOrder = ASC | DESC + +loopExpression: LOOP ~ (LPAREN ~> query <~ RPAREN) ~ opt(intConstant <~ TIMES) ~ opt(AS ~> alias) + +selectClause: SELECT ~ rep1sep(selectExpression, COMMA) + +countClause = COUNT ~ LPAREN ~ RPAREN + +maxClause = MAX ~ (LPAREN ~> expr <~ RPAREN) + +minClause = MIN ~ (LPAREN ~> expr <~ RPAREN) + +sumClause = SUM ~ (LPAREN ~> expr <~ RPAREN) + +selectExpression: expr ~ opt(AS ~> alias) + +expr: compE ~ opt(rep(exprRight)) + +exprRight: (AND | OR) ~ compE + +compE: + arithE ~ (LT | LTE | EQ | NEQ | GT | GTE) ~ arithE | + arithE ~ (ISA | IS) ~ ident | + arithE ~ HAS ~ ident | + arithE | countClause | maxClause | minClause | sumClause + +arithE: multiE ~ opt(rep(arithERight)) + +arithERight: (PLUS | MINUS) ~ multiE + +multiE: atomE ~ opt(rep(multiERight)) + +multiERight: (STAR | DIV) ~ atomE + +atomE: literal | identifier | LPAREN ~> expr <~ RPAREN + +identifier: rep1sep(ident, DOT) + +alias: ident | stringLit + +literal: booleanConstant | + intConstant | + longConstant | + floatConstant | + doubleConstant | + stringLit + +</pre></div> +<p>Grammar language: {noformat} opt(a) => a is optional ~ => a combinator. 'a ~ b' means a followed by b rep => zero or more rep1sep => one or more, separated by second arg. {noformat}</p> +<p>Language Notes:</p> +<ul> +<li>A <b>SingleQuery</b> expression can be used to search for entities of a <i>Trait</i> or <i>Class</i>.</li></ul>Entities can be filtered based on a 'Where Clause' and Entity Attributes can be retrieved based on a 'Select Clause'. +<ul> +<li>An Entity Graph can be traversed/joined by combining one or more SingleQueries.</li> +<li>An attempt is made to make the expressions look SQL like by accepting keywords "SELECT",</li></ul>"FROM", and "WHERE"; but these are optional and users can simply think in terms of Entity Graph Traversals. +<ul> +<li>The transitive closure of an Entity relationship can be expressed via the <i>Loop</i> expression. A</li></ul><i>Loop</i> expression can be any traversal (recursively a query) that represents a <i>Path</i> that ends in an Entity of the same <i>Type</i> as the starting Entity. +<ul> +<li>The <i>WithPath</i> clause can be used with transitive closure queries to retrieve the Path that</li></ul>connects the two related Entities. (We also provide a higher level interface for Closure Queries see scaladoc for 'org.apache.atlas.query.ClosureQuery') +<ul> +<li>GROUPBY is optional. Group by can be specified with aggregate methods like max, min, sum and count. When group by is specified aggregated results are returned based on the method specified in select clause. Select expression is mandatory with group by expression.</li> +<li>ORDERBY is optional. When order by clause is specified, case insensitive sorting is done based on the column specified.</li></ul>For sorting in descending order specify 'DESC' after order by clause. If no order by is specified, then no default sorting is applied. +<ul> +<li>LIMIT is optional. It limits the maximum number of objects to be fetched starting from specified optional offset. If no offset is specified count starts from beginning.</li> +<li>There are couple of Predicate functions different from SQL:</li></ul>* <i>is</i> or <i>isa</i>can be used to filter Entities that have a particular Trait. * <i>has</i> can be used to filter Entities that have a value for a particular Attribute. +<ul> +<li>Any identifiers or constants with special characters(space,$,",{,}) should be enclosed within backquote (`)</li></ul></div> +<div class="section"> +<h5><a name="DSL_Examples"></a>DSL Examples</h5> +<p>For the model, Asset - attributes name, owner, description DB - supertype Asset - attributes clusterName, parameters, comment Column - extends Asset - attributes type, comment Table - supertype Asset - db, columns, parameters, comment Traits - PII, Log Data</p> +<p>DSL queries: * from DB</p> +<ul> +<li>DB where name="Reporting" select name, owner</li> +<li>DB where name="Reporting" select name, owner orderby name</li> +<li>DB where name="Reporting" select name limit 10</li> +<li>DB where name="Reporting" select name, owner limit 10 offset 0</li> +<li>DB where name="Reporting" select name, owner orderby name limit 10 offset 5</li> +<li>DB where name="Reporting" select name, owner orderby name desc limit 10 offset 5</li> +<li>DB has name</li> +<li>DB is JdbcAccess</li> +<li>Column where Column isa PII</li> +<li>Table where name="sales_fact", columns</li> +<li>Table where name="sales_fact", columns as column select column.name, column.dataType, column.comment</li> +<li>DB groupby(owner) select owner, count()</li> +<li>DB groupby(owner) select owner, max(name)</li> +<li>DB groupby(owner) select owner, min(name)</li> +<li>from Person select count() as 'count', max(Person.age) as 'max', min(Person.age)</li> +<li>`Log Data`</li></ul></div> +<div class="section"> +<h4><a name="Full-text_Search"></a>Full-text Search</h4> +<p>Atlas also exposes a lucene style full-text search capability.</p></div> + </div> + </div> + <hr/> + <footer> + <div class="container"> + <div class="row"> +Copyright é 2018 The Apache Software Foundation, Licensed under the Apache License, Version 2.0. + </div> + <p id="poweredBy" class="pull-right"><a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"><img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /></a> +</p> + </div> + </footer> + </body> +</html> http://git-wip-us.apache.org/repos/asf/atlas-website/blob/2d4dc1a0/0.8.3/StormAtlasHook.html ---------------------------------------------------------------------- diff --git a/0.8.3/StormAtlasHook.html b/0.8.3/StormAtlasHook.html new file mode 100644 index 0000000..f6e9354 --- /dev/null +++ b/0.8.3/StormAtlasHook.html @@ -0,0 +1,197 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/twiki/StormAtlasHook.twiki at 2018-10-31 + | Rendered using Apache Maven Fluido Skin 1.7 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20181031" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Apache Atlas – Storm Atlas Bridge</title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.7.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + <script type="text/javascript" src="./js/apache-maven-fluido-1.7.min.js"></script> + </head> + <body class="topBarEnabled"> + <div id="topbar" class="navbar navbar-fixed-top "> + <div class="navbar-inner"> + <div class="container" style="width: 68%;"><div class="nav-collapse"> + <ul class="nav"> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Atlas <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="index.html" title="About">About</a></li> + <li><a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="Wiki">Wiki</a></li> + <li><a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="News">News</a></li> + <li><a href="https://git-wip-us.apache.org/repos/asf/atlas.git"; title="Git">Git</a></li> + <li><a href="https://issues.apache.org/jira/browse/ATLAS"; title="Jira">Jira</a></li> + <li><a href="https://cwiki.apache.org/confluence/display/ATLAS/PoweredBy"; title="Powered by">Powered by</a></li> + <li><a href="http://blogs.apache.org/atlas/"; title="Blog">Blog</a></li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Project Information <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="project-info.html" title="Summary">Summary</a></li> + <li><a href="mail-lists.html" title="Mailing Lists">Mailing Lists</a></li> + <li><a href="http://webchat.freenode.net?channels=apacheatlas&uio=d4"; title="IRC">IRC</a></li> + <li><a href="team-list.html" title="Team">Team</a></li> + <li><a href="issue-tracking.html" title="Issue Tracking">Issue Tracking</a></li> + <li><a href="source-repository.html" title="Source Repository">Source Repository</a></li> + <li><a href="license.html" title="License">License</a></li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Releases <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="http://www.apache.org/dyn/closer.cgi/atlas/0.8.2/"; title="0.8.2">0.8.2</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.8.1/"; title="0.8.1">0.8.1</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.8.0-incubating/"; title="0.8-incubating">0.8-incubating</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.7.1-incubating/"; title="0.7.1-incubating">0.7.1-incubating</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.7.0-incubating/"; title="0.7-incubating">0.7-incubating</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.6.0-incubating/"; title="0.6-incubating">0.6-incubating</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.5.0-incubating/"; title="0.5-incubating">0.5-incubating</a></li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="../index.html" title="latest">latest</a></li> + <li><a href="../0.8.2/index.html" title="0.8.2">0.8.2</a></li> + <li><a href="../0.8.1/index.html" title="0.8.1">0.8.1</a></li> + <li><a href="../0.8.0-incubating/index.html" title="0.8-incubating">0.8-incubating</a></li> + <li><a href="../0.7.1-incubating/index.html" title="0.7.1-incubating">0.7.1-incubating</a></li> + <li><a href="../0.7.0-incubating/index.html" title="0.7-incubating">0.7-incubating</a></li> + <li><a href="../0.6.0-incubating/index.html" title="0.6-incubating">0.6-incubating</a></li> + <li><a href="../0.5.0-incubating/index.html" title="0.5-incubating">0.5-incubating</a></li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">ASF <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="http://www.apache.org/foundation/how-it-works.html"; title="How Apache Works">How Apache Works</a></li> + <li><a href="http://www.apache.org/foundation/"; title="Foundation">Foundation</a></li> + <li><a href="http://www.apache.org/foundation/sponsorship.html"; title="Sponsoring Apache">Sponsoring Apache</a></li> + <li><a href="http://www.apache.org/foundation/thanks.html"; title="Thanks">Thanks</a></li> + </ul> + </li> + </ul> +<form id="search-form" action="https://www.google.com/search"; method="get" class="navbar-search pull-right" > + <input value="http://atlas.apache.org"; name="sitesearch" type="hidden"/> + <input class="search-query" name="q" id="query" type="text" /> +</form> +<script type="text/javascript">asyncJs( 'https://cse.google.com/brand?form=search-form' )</script> + <iframe src="https://www.facebook.com/plugins/like.php?href=http://atlas.apache.org/atlas-docs&send=false&layout=button_count&show-faces=false&action=like&colorscheme=dark"; + scrolling="no" frameborder="0" + style="border:none; width:100px; height:20px; margin-top: 10px;" class="pull-right" ></iframe> + <script type="text/javascript">asyncJs( 'https://apis.google.com/js/plusone.js' )</script> + <ul class="nav pull-right"><li style="margin-top: 10px;"> + <div class="g-plusone" data-href="http://atlas.apache.org/atlas-docs"; data-size="medium" width="60px" align="right" ></div> + </li></ul> + </div> + </div> + </div> + </div> + <div class="container"> + <div id="banner"> + <div class="pull-left"><a href=".." id="bannerLeft"><img src="images/atlas-logo.png" alt="Apache Atlas" width="200px" height="45px"/></a></div> + <div class="pull-right"></div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + <li class=""><a href="http://www.apache.org"; class="externalLink" title="Apache">Apache</a><span class="divider">/</span></li> + <li class=""><a href="index.html" title="Atlas">Atlas</a><span class="divider">/</span></li> + <li class="active ">Storm Atlas Bridge</li> + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-10-31</li> + <li id="projectVersion" class="pull-right">Version: 0.8.3</li> + </ul> + </div> + <div id="bodyColumn" > +<div class="section"> +<h2><a name="Storm_Atlas_Bridge"></a>Storm Atlas Bridge</h2></div> +<div class="section"> +<h3><a name="Introduction"></a>Introduction</h3> +<p>Apache Storm is a distributed real-time computation system. Storm makes it easy to reliably process unbounded streams of data, doing for real-time processing what Hadoop did for batch processing. The process is essentially a DAG of nodes, which is called <b>topology</b>.</p> +<p>Apache Atlas is a metadata repository that enables end-to-end data lineage, search and associate business classification.</p> +<p>The goal of this integration is to push the operational topology metadata along with the underlying data source(s), target(s), derivation processes and any available business context so Atlas can capture the lineage for this topology.</p> +<p>There are 2 parts in this process detailed below:</p> +<ul> +<li>Data model to represent the concepts in Storm</li> +<li>Storm Atlas Hook to update metadata in Atlas</li></ul></div> +<div class="section"> +<h3><a name="Storm_Data_Model"></a>Storm Data Model</h3> +<p>A data model is represented as Types in Atlas. It contains the descriptions of various nodes in the topology graph, such as spouts and bolts and the corresponding producer and consumer types.</p> +<p>The following types are added in Atlas.</p> +<p></p> +<ul> +<li>storm_topology - represents the coarse-grained topology. A storm_topology derives from an Atlas Process type and hence can be used to inform Atlas about lineage.</li> +<li>Following data sets are added - kafka_topic, jms_topic, hbase_table, hdfs_data_set. These all derive from an Atlas Dataset type and hence form the end points of a lineage graph.</li> +<li>storm_spout - Data Producer having outputs, typically Kafka, JMS</li> +<li>storm_bolt - Data Consumer having inputs and outputs, typically Hive, HBase, HDFS, etc.</li></ul> +<p>The Storm Atlas hook auto registers dependent models like the Hive data model if it finds that these are not known to the Atlas server.</p> +<p>The data model for each of the types is described in the class definition at org.apache.atlas.storm.model.StormDataModel.</p></div> +<div class="section"> +<h3><a name="Storm_Atlas_Hook"></a>Storm Atlas Hook</h3> +<p>Atlas is notified when a new topology is registered successfully in Storm. Storm provides a hook, backtype.storm.ISubmitterHook, at the Storm client used to submit a storm topology.</p> +<p>The Storm Atlas hook intercepts the hook post execution and extracts the metadata from the topology and updates Atlas using the types defined. Atlas implements the Storm client hook interface in org.apache.atlas.storm.hook.StormAtlasHook.</p></div> +<div class="section"> +<h3><a name="Limitations"></a>Limitations</h3> +<p>The following apply for the first version of the integration.</p> +<p></p> +<ul> +<li>Only new topology submissions are registered with Atlas, any lifecycle changes are not reflected in Atlas.</li> +<li>The Atlas server needs to be online when a Storm topology is submitted for the metadata to be captured.</li> +<li>The Hook currently does not support capturing lineage for custom spouts and bolts.</li></ul></div> +<div class="section"> +<h3><a name="Installation"></a>Installation</h3> +<p>The Storm Atlas Hook needs to be manually installed in Storm on the client side. The hook artifacts are available at: $ATLAS_PACKAGE/hook/storm</p> +<p>Storm Atlas hook jars need to be copied to $STORM_HOME/extlib. Replace STORM_HOME with storm installation path.</p> +<p>Restart all daemons after you have installed the atlas hook into Storm.</p></div> +<div class="section"> +<h3><a name="Configuration"></a>Configuration</h3></div> +<div class="section"> +<h4><a name="Storm_Configuration"></a>Storm Configuration</h4> +<p>The Storm Atlas Hook needs to be configured in Storm client config in <b>$STORM_HOME/conf/storm.yaml</b> as:</p> +<div class="source"><pre class="prettyprint"> +storm.topology.submission.notifier.plugin.class: "org.apache.atlas.storm.hook.StormAtlasHook" + +</pre></div> +<p>Also set a 'cluster name' that would be used as a namespace for objects registered in Atlas. This name would be used for namespacing the Storm topology, spouts and bolts.</p> +<p>The other objects like data sets should ideally be identified with the cluster name of the components that generate them. For e.g. Hive tables and databases should be identified using the cluster name set in Hive. The Storm Atlas hook will pick this up if the Hive configuration is available in the Storm topology jar that is submitted on the client and the cluster name is defined there. This happens similarly for HBase data sets. In case this configuration is not available, the cluster name set in the Storm configuration will be used.</p> +<div class="source"><pre class="prettyprint"> +atlas.cluster.name: "cluster_name" + +</pre></div> +<p>In <b>$STORM_HOME/conf/storm_env.ini</b>, set an environment variable as follows:</p> +<div class="source"><pre class="prettyprint"> +STORM_JAR_JVM_OPTS:"-Datlas.conf=$ATLAS_HOME/conf/" + +</pre></div> +<p>where ATLAS_HOME is pointing to where ATLAS is installed.</p> +<p>You could also set this up programatically in Storm Config as:</p> +<div class="source"><pre class="prettyprint"> + Config stormConf = new Config(); + ... + stormConf.put(Config.STORM_TOPOLOGY_SUBMISSION_NOTIFIER_PLUGIN, + org.apache.atlas.storm.hook.StormAtlasHook.class.getName()); + +</pre></div></div> + </div> + </div> + <hr/> + <footer> + <div class="container"> + <div class="row"> +Copyright é 2018 The Apache Software Foundation, Licensed under the Apache License, Version 2.0. + </div> + <p id="poweredBy" class="pull-right"><a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"><img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /></a> +</p> + </div> + </footer> + </body> +</html>
