http://git-wip-us.apache.org/repos/asf/asterixdb-site/blob/d00eca89/docs/0.9.4/ncservice.html ---------------------------------------------------------------------- diff --git a/docs/0.9.4/ncservice.html b/docs/0.9.4/ncservice.html new file mode 100644 index 0000000..f65561d --- /dev/null +++ b/docs/0.9.4/ncservice.html @@ -0,0 +1,918 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia Site Renderer 1.8.1 from src/site/markdown/ncservice.md at 2018-10-02 + | Rendered using Apache Maven Fluido Skin 1.7 +--> +<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20181002" /> + <meta http-equiv="Content-Language" content="en" /> + <title>AsterixDB – Table of Contents</title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.7.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + <script type="text/javascript" src="./js/apache-maven-fluido-1.7.min.js"></script> + + </head> + <body class="topBarDisabled"> + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"><a href="./" id="bannerLeft"><img src="images/asterixlogo.png" alt="AsterixDB"/></a></div> + <div class="pull-right"></div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + <li id="publishDate">Last Published: 2018-10-02</li> + <li id="projectVersion" class="pull-right">Version: 0.9.4</li> + <li class="pull-right"><a href="index.html" title="Documentation Home">Documentation Home</a></li> + </ul> + </div> + <div class="row-fluid"> + <div id="leftColumn" class="span2"> + <div class="well sidebar-nav"> + <ul class="nav nav-list"> + <li class="nav-header">Get Started - Installation</li> + <li class="active"><a href="#"><span class="none"></span>Option 1: using NCService</a></li> + <li><a href="ansible.html" title="Option 2: using Ansible"><span class="none"></span>Option 2: using Ansible</a></li> + <li><a href="aws.html" title="Option 3: using Amazon Web Services"><span class="none"></span>Option 3: using Amazon Web Services</a></li> + <li class="nav-header">AsterixDB Primer</li> + <li><a href="sqlpp/primer-sqlpp.html" title="Option 1: using SQL++"><span class="none"></span>Option 1: using SQL++</a></li> + <li><a href="aql/primer.html" title="Option 2: using AQL"><span class="none"></span>Option 2: using AQL</a></li> + <li class="nav-header">Data Model</li> + <li><a href="datamodel.html" title="The Asterix Data Model"><span class="none"></span>The Asterix Data Model</a></li> + <li class="nav-header">Queries - SQL++</li> + <li><a href="sqlpp/manual.html" title="The SQL++ Query Language"><span class="none"></span>The SQL++ Query Language</a></li> + <li><a href="sqlpp/builtins.html" title="Builtin Functions"><span class="none"></span>Builtin Functions</a></li> + <li class="nav-header">Queries - AQL</li> + <li><a href="aql/manual.html" title="The Asterix Query Language (AQL)"><span class="none"></span>The Asterix Query Language (AQL)</a></li> + <li><a href="aql/builtins.html" title="Builtin Functions"><span class="none"></span>Builtin Functions</a></li> + <li class="nav-header">API/SDK</li> + <li><a href="api.html" title="HTTP API"><span class="none"></span>HTTP API</a></li> + <li><a href="csv.html" title="CSV Output"><span class="none"></span>CSV Output</a></li> + <li class="nav-header">Advanced Features</li> + <li><a href="aql/fulltext.html" title="Support of Full-text Queries"><span class="none"></span>Support of Full-text Queries</a></li> + <li><a href="aql/externaldata.html" title="Accessing External Data"><span class="none"></span>Accessing External Data</a></li> + <li><a href="feeds/tutorial.html" title="Support for Data Ingestion"><span class="none"></span>Support for Data Ingestion</a></li> + <li><a href="udf.html" title="User Defined Functions"><span class="none"></span>User Defined Functions</a></li> + <li><a href="aql/filters.html" title="Filter-Based LSM Index Acceleration"><span class="none"></span>Filter-Based LSM Index Acceleration</a></li> + <li><a href="aql/similarity.html" title="Support of Similarity Queries"><span class="none"></span>Support of Similarity Queries</a></li> +</ul> + <hr /> + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> +<a href="./" title="AsterixDB" class="builtBy"><img class="builtBy" alt="AsterixDB" src="images/asterixlogo.png" /></a> + </div> + </div> + </div> + <div id="bodyColumn" class="span10" > +<!-- + ! Licensed to the Apache Software Foundation (ASF) under one + ! or more contributor license agreements. See the NOTICE file + ! distributed with this work for additional information + ! regarding copyright ownership. The ASF licenses this file + ! to you under the Apache License, Version 2.0 (the + ! "License"); you may not use this file except in compliance + ! with the License. You may obtain a copy of the License at + ! + ! http://www.apache.org/licenses/LICENSE-2.0 + ! + ! Unless required by applicable law or agreed to in writing, + ! software distributed under the License is distributed on an + ! "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + ! KIND, either express or implied. See the License for the + ! specific language governing permissions and limitations + ! under the License. + !--> +<div class="section"> +<h2><a name="Table_of_Contents"></a><a name="toc" id="toc">Table of Contents</a></h2> +<ul> + +<li><a href="#quickstart">Quick Start</a></li> +<li><a href="#Small_cluster">Starting a small single-machine cluster using the NCService</a></li> +<li><a href="#Multi_machine">Deploying AsterixDB via NCService in a multi-machine setup</a></li> +<li><a href="#Parameters">Available Configuration Parameters</a></li> +</ul> +<h1><a name="quickstart" id="quickstart">Quick Start</a></h1> +<p>The fastest way to get set up with a single-machine sample instance of AsterixDB is to use the included sample helper scripts. To do so, in the extracted <tt>asterix-server</tt> directory, navigate to <tt>opt/local/bin/</tt></p> + +<div> +<div> +<pre class="source">user@localhost:~/ +$cd asterix-server/ +user@localhost:~/asterix-server +$cd opt/local/bin +</pre></div></div> + +<p>This folder should contain 4 scripts, two pairs of <tt>.sh</tt> and <tt>.bat</tt> files respectively. <tt>start-sample-cluster.sh</tt> will simply start a basic sample cluster using the configuration files located in <tt>opt/local/conf/</tt>.</p> + +<div> +<div> +<pre class="source">user@localhost:~/a/o/l/bin +$./start-sample-cluster.sh +CLUSTERDIR=/home/user/asterix-server/opt/local +INSTALLDIR=/home/user/asterix-server +LOGSDIR=/home/user/asterix-server/samples/opt/logs + +INFO: Starting sample cluster... +INFO: Waiting up to 30 seconds for cluster 127.0.0.1:19002 to be available. +INFO: Cluster started and is ACTIVE. +user@localhost:~/a/o/l/bin +$ +</pre></div></div> + +<p>Now, there should be a running AsterixDB cluster on the machine. To go to the Web Interface, visit <a class="externalLink" href="http://localhost:19001">http://localhost:19001</a></p> + +<div class="source"> + +<div class="source"><pre class="prettyprint linenums"> +<img src="images/asterixdb_interface.png" alt="The AsterixDB Web Interface" /> +<i>Fig. 1</i>: The AsterixDB Web Interface +</pre></div> +</div> + +<h1><a name="Small_cluster" id="Small_cluster">Starting a small single-machine cluster using NCService</a></h1> +<p>The above cluster was started using a script, but below is a description in detail of how precisely this was achieved. The config files here are analagous to the ones within <tt>samples/local/conf</tt>.</p> +<p>When running a cluster using the <tt>NCService</tt> there are 3 different kinds of processes involved:</p> +<ul> + +<li><tt>NCDriver</tt>, also known as the Node Controller or NC for short. This is the process that does the actual work of queries and data management within the AsterixDB cluster</li> +<li><tt>NCService</tt> configures and starts the <tt>NCDriver</tt> process. It is a simple daemon whose only purpose is to wait for the <tt>CCDriver</tt> process to call upon it to initiate cluster bootup.</li> +<li><tt>CCDriver</tt>, which is the Cluster Controller process, also known as the CC. This process manages the distribution of tasks to all NCs, as well as providing the parameters of each NC to the NCService upon cluster startup. It also hosts the Web interface and query compiler and optimizer.</li> +</ul> +<p>The cluster startup follows a particular sequence, which is as follows:</p> +<ol style="list-style-type: decimal"> + +<li>Each host on which an NC is desired and is mentioned in the configuration, the <tt>NCService</tt> daemon is started first. It will listen and wait for the CC to contact it.</li> +<li>The one host on which the CC is to be placed is started with an appropriate configuration file.</li> +<li>The CC contacts all <tt>NCService</tt> daemons and the <tt>NCService</tt> subsequently starts and <tt>NCDriver</tt> process with the configration supplied by the <tt>CC</tt></li> +<li>Each <tt>NCDriver</tt> then contacts the CC to register itself as started</li> +</ol> +<p>This process is briefly illustrated in the diagram below:</p> + +<div class="source"> + +<div class="source"><pre class="prettyprint linenums"> +<img src="images/ncservice.png" alt="The AsterixDB Web Interface" /> +<i>Fig. 2</i>: NCService startup sequence +</pre></div> +</div> + +<p>To start a small cluster consisting of 2 NodeControllers (<tt>red</tt> and <tt>blue</tt>) and 1 ClusterController (<tt>cc</tt>) on a single machine only 2 configuration files are required. The first one is</p> +<p><tt>blue.conf</tt>:</p> + +<div> +<div> +<pre class="source">[ncservice] +port=9091 +</pre></div></div> + +<p>It is a configuration file for the second <tt>NCService</tt>. This contains only the port that the <tt>NCService</tt> of the second NodeControllers listens to as it is non-standard. The first <tt>NCService</tt> does not need a configuration file, as it only uses default parameters. In a distributed environment with 1 NodeController per machine, no <tt>NCService</tt> needs a configuration file.</p> +<p>The second configuration file is</p> +<p><tt>cc.conf</tt>:</p> + +<div> +<div> +<pre class="source">[nc/red] +txn.log.dir=/tmp/asterix/red/txnlog +core.dump.dir=/tmp/asterix/red/coredump +iodevices=/tmp/asterix/red + +[nc/blue] +port=9091 +txn.log.dir=/tmp/asterix/blue/txnlog +core.dump.dir=/tmp/asterix/blue/coredump +iodevices=/tmp/asterix/blue + +[nc] +app.class=org.apache.asterix.hyracks.bootstrap.NCApplicationEntryPoint +address=127.0.0.1 +command=asterixnc + +[cc] +address = 127.0.0.1 +console.listen.port = 12345 +</pre></div></div> + +<p>This is the configuration file for the cluster and it contains information that each <tt>NCService</tt> will use when starting the corresponding <tt>NCDriver</tt> as well as information for the <tt>CCDriver</tt>.</p> +<p>To start the cluster simply use the following steps</p> +<ol style="list-style-type: decimal"> + +<li> + +<p>Change directory into the asterix-server binary folder</p> + +<div> +<div> +<pre class="source">user@localhost:~/ +$cd asterix-server/ +user@localhost:~/asterix-server +$cd samples/local/bin +</pre></div></div> +</li> +<li> + +<p>Start the 2 <tt>NCServices</tt> for <tt>red</tt> and <tt>blue</tt>.</p> + +<div> +<div> +<pre class="source">user@localhost:~/asterix-server +$bin/asterixncservice -config-file blue.conf > blue-service.log 2>&1 & +user@localhost:~/asterix-server +$bin/asterixncservice >red-service.log 2>&1 & +</pre></div></div> +</li> +<li> + +<p>Start the <tt>CCDriver</tt>.</p> + +<div> +<div> +<pre class="source">user@localhost:~/asterix-server +$bin/asterixcc -config-file cc.conf > cc.log 2>&1 & +</pre></div></div> +</li> +</ol> +<p>The <tt>CCDriver</tt> will connect to the <tt>NCServices</tt> and thus initiate the configuration and the start of the <tt>NCDrivers</tt>. After running these scripts, <tt>jps</tt> should show a result similar to this:</p> + +<div> +<div> +<pre class="source">user@localhost:~/asterix-server +$jps +13184 NCService +13200 NCDriver +13185 NCService +13186 CCDriver +13533 Jps +13198 NCDriver +</pre></div></div> + +<p>The logs for the <tt>NCDrivers</tt> will be in <tt>$BASEDIR/logs</tt>.</p> +<p>To stop the cluster again simply run</p> + +<div> +<div> +<pre class="source">$ kill `jps | egrep '(CDriver|NCService)' | awk '{print $1}'` +</pre></div></div> + +<p>to kill all processes.</p> +<h1><a name="Multi_machine" id="Multi_machine">Deploying AsterixDB via NCService in a multi-machine setup</a></h1> +<p>Deploying on multiple machines only differs in the configuration file and where each process is actually resident. Take for example a deployment on 3 machines, <tt>cacofonix-1</tt>,<tt>cacofonix-2</tt>,and <tt>cacofonix-3</tt>. <tt>cacofonix-1</tt> will be the CC, and <tt>cacofonix-2</tt> and <tt>cacofonix-3</tt> will be the two NCs, respectively. The configuration would be as follows:</p> +<p><tt>cc.conf</tt>:</p> + +<div> +<div> +<pre class="source">[nc/red] +txn.log.dir=/lv_scratch/asterix/red/txnlog +core.dump.dir=/lv_scratch/asterix/red/coredump +iodevices=/lv_scratch/asterix/red +address=cacofonix-2 + +[nc/blue] +txn.log.dir=/lv_scratch/asterix/blue/txnlog +core.dump.dir=/lv_scratch/asterix/blue/coredump +iodevices=/lv_scratch/asterix/blue +address=cacofonix-3 + +[nc] +app.class=org.apache.asterix.hyracks.bootstrap.NCApplicationEntryPoint +storagedir=storage +command=asterixnc + +[cc] +address = cacofonix-1 +</pre></div></div> + +<p>To deploy, first the <tt>asterix-server</tt> binary must be present on each machine. Any method to transfer the archive to each machine will work, but here <tt>scp</tt> will be used for simplicity’s sake.</p> + +<div> +<div> +<pre class="source">user@localhost:~ +$for f in {1,2,3}; scp asterix-server.zip cacofonix-$f:~/; end +</pre></div></div> + +<p>Then unzip the binary on each machine. First, start the <tt>NCService</tt> processes on each of the slave machines. Any way of getting a shell on the machine is fine, be it physical or via <tt>ssh</tt>.</p> + +<div> +<div> +<pre class="source">user@cacofonix-2 12:41:42 ~/asterix-server/ +$ bin/asterixncservice > red-service.log 2>&1 & + + +user@cacofonix-3 12:41:42 ~/asterix-server/ +$ bin/asterixncservice > blue-service.log 2>&1 & +</pre></div></div> + +<p>Now that each <tt>NCService</tt> is waiting, the CC can be started.</p> + +<div> +<div> +<pre class="source">user@cacofonix-1 12:41:42 ~/asterix-server/ +$ bin/asterixcc -config-file cc.conf > cc.log 2>&1 & +</pre></div></div> + +<p>The cluster should now be started and the Web UI available on the CC host at the default port.</p> +<h1><a name="Parameters" id="Parameters">Available Configuration Parameters</a></h1> +<p>The following parameters are for the master process, under the “[cc]” section.</p> +<table border="0" class="table table-striped"> +<thead> + +<tr class="a"> +<th> Section </th> +<th> Parameter </th> +<th> Meaning </th> +<th> Default </th></tr> +</thead><tbody> + +<tr class="b"> +<td> cc </td> +<td> active.port </td> +<td> The listen port of the active server </td> +<td> 19003 </td></tr> +<tr class="a"> +<td> cc </td> +<td> address </td> +<td> Default bind address for all services on this cluster controller </td> +<td> 127.0.0.1 </td></tr> +<tr class="b"> +<td> cc </td> +<td> api.port </td> +<td> The listen port of the API server </td> +<td> 19002 </td></tr> +<tr class="a"> +<td> cc </td> +<td> app.class </td> +<td> Application CC main class </td> +<td> org.apache.asterix.hyracks.bootstrap.CCApplication </td></tr> +<tr class="b"> +<td> cc </td> +<td> client.listen.address </td> +<td> Sets the IP Address to listen for connections from clients </td> +<td> same as address </td></tr> +<tr class="a"> +<td> cc </td> +<td> client.listen.port </td> +<td> Sets the port to listen for connections from clients </td> +<td> 1098 </td></tr> +<tr class="b"> +<td> cc </td> +<td> cluster.listen.address </td> +<td> Sets the IP Address to listen for connections from NCs </td> +<td> same as address </td></tr> +<tr class="a"> +<td> cc </td> +<td> cluster.listen.port </td> +<td> Sets the port to listen for connections from node controllers </td> +<td> 1099 </td></tr> +<tr class="b"> +<td> cc </td> +<td> cluster.public.address </td> +<td> Address that NCs should use to contact this CC </td> +<td> same as cluster.listen.address </td></tr> +<tr class="a"> +<td> cc </td> +<td> cluster.public.port </td> +<td> Port that NCs should use to contact this CC </td> +<td> same as cluster.listen.port </td></tr> +<tr class="b"> +<td> cc </td> +<td> cluster.topology </td> +<td> Sets the XML file that defines the cluster topology </td> +<td> <undefined> </td></tr> +<tr class="a"> +<td> cc </td> +<td> console.listen.address </td> +<td> Sets the listen address for the Cluster Controller </td> +<td> same as address </td></tr> +<tr class="b"> +<td> cc </td> +<td> console.listen.port </td> +<td> Sets the http port for the Cluster Controller) </td> +<td> 16001 </td></tr> +<tr class="a"> +<td> cc </td> +<td> cores.multiplier </td> +<td> The factor to multiply by the number of cores to determine maximum query concurrent execution level </td> +<td> 3 </td></tr> +<tr class="b"> +<td> cc </td> +<td> heartbeat.max.misses </td> +<td> Sets the maximum number of missed heartbeats before a node is marked as dead </td> +<td> 5 </td></tr> +<tr class="a"> +<td> cc </td> +<td> heartbeat.period </td> +<td> Sets the time duration between two heartbeats from each node controller in milliseconds </td> +<td> 10000 </td></tr> +<tr class="b"> +<td> cc </td> +<td> job.history.size </td> +<td> Limits the number of historical jobs remembered by the system to the specified value </td> +<td> 10 </td></tr> +<tr class="a"> +<td> cc </td> +<td> job.manager.class </td> +<td> Specify the implementation class name for the job manager </td> +<td> org.apache.hyracks.control.cc.job.JobManager </td></tr> +<tr class="b"> +<td> cc </td> +<td> job.queue.capacity </td> +<td> The maximum number of jobs to queue before rejecting new jobs </td> +<td> 4096 </td></tr> +<tr class="a"> +<td> cc </td> +<td> job.queue.class </td> +<td> Specify the implementation class name for the job queue </td> +<td> org.apache.hyracks.control.cc.scheduler.FIFOJobQueue </td></tr> +<tr class="b"> +<td> cc </td> +<td> profile.dump.period </td> +<td> Sets the time duration between two profile dumps from each node controller in milliseconds; 0 to disable </td> +<td> 0 </td></tr> +<tr class="a"> +<td> cc </td> +<td> result.sweep.threshold </td> +<td> The duration within which an instance of the result cleanup should be invoked in milliseconds </td> +<td> 60000 </td></tr> +<tr class="b"> +<td> cc </td> +<td> result.ttl </td> +<td> Limits the amount of time results for asynchronous jobs should be retained by the system in milliseconds </td> +<td> 86400000 </td></tr> +<tr class="a"> +<td> cc </td> +<td> root.dir </td> +<td> Sets the root folder used for file operations </td> +<td> ${java.io.tmpdir}/asterixdb/ClusterControllerService </td></tr> +<tr class="b"> +<td> cc </td> +<td> web.port </td> +<td> The listen port of the legacy query interface </td> +<td> 19001 </td></tr> +<tr class="a"> +<td> cc </td> +<td> web.queryinterface.port </td> +<td> The listen port of the query web interface </td> +<td> 19006 </td></tr> +</tbody> +</table> +<p>The following parameters for slave processes, under “[nc]” sections.</p> +<table border="0" class="table table-striped"> +<thead> + +<tr class="a"> +<th> Section </th> +<th> Parameter </th> +<th> Meaning </th> +<th> Default </th></tr> +</thead><tbody> + +<tr class="b"> +<td> nc </td> +<td> address </td> +<td> Default IP Address to bind listeners on this NC. All services will bind on this address unless a service-specific listen address is supplied. </td> +<td> 127.0.0.1 </td></tr> +<tr class="a"> +<td> nc </td> +<td> app.class </td> +<td> Application NC Main Class </td> +<td> org.apache.asterix.hyracks.bootstrap.NCApplication </td></tr> +<tr class="b"> +<td> nc </td> +<td> cluster.address </td> +<td> Cluster Controller address (required unless specified in config file) </td> +<td> <undefined> </td></tr> +<tr class="a"> +<td> nc </td> +<td> cluster.connect.retries </td> +<td> Number of attempts to contact CC before giving up </td> +<td> 5 </td></tr> +<tr class="b"> +<td> nc </td> +<td> cluster.listen.address </td> +<td> IP Address to bind cluster listener on this NC </td> +<td> same as address </td></tr> +<tr class="a"> +<td> nc </td> +<td> cluster.listen.port </td> +<td> IP port to bind cluster listener </td> +<td> 0 </td></tr> +<tr class="b"> +<td> nc </td> +<td> cluster.port </td> +<td> Cluster Controller port </td> +<td> 1099 </td></tr> +<tr class="a"> +<td> nc </td> +<td> cluster.public.address </td> +<td> Public IP Address to announce cluster listener </td> +<td> same as public.address </td></tr> +<tr class="b"> +<td> nc </td> +<td> cluster.public.port </td> +<td> Public IP port to announce cluster listener </td> +<td> same as cluster.listen.port </td></tr> +<tr class="a"> +<td> nc </td> +<td> command </td> +<td> Command NCService should invoke to start the NCDriver </td> +<td> hyracksnc </td></tr> +<tr class="b"> +<td> nc </td> +<td> core.dump.dir </td> +<td> The directory where node core dumps should be written </td> +<td> ${java.io.tmpdir}/asterixdb/coredump </td></tr> +<tr class="a"> +<td> nc </td> +<td> data.listen.address </td> +<td> IP Address to bind data listener </td> +<td> same as address </td></tr> +<tr class="b"> +<td> nc </td> +<td> data.listen.port </td> +<td> IP port to bind data listener </td> +<td> 0 </td></tr> +<tr class="a"> +<td> nc </td> +<td> data.public.address </td> +<td> Public IP Address to announce data listener </td> +<td> same as public.address </td></tr> +<tr class="b"> +<td> nc </td> +<td> data.public.port </td> +<td> Public IP port to announce data listener </td> +<td> same as data.listen.port </td></tr> +<tr class="a"> +<td> nc </td> +<td> iodevices </td> +<td> Comma separated list of IO Device mount points </td> +<td> ${java.io.tmpdir}/asterixdb/iodevice </td></tr> +<tr class="b"> +<td> nc </td> +<td> jvm.args </td> +<td> JVM args to pass to the NCDriver </td> +<td> <undefined> </td></tr> +<tr class="a"> +<td> nc </td> +<td> messaging.listen.address </td> +<td> IP Address to bind messaging listener </td> +<td> same as address </td></tr> +<tr class="b"> +<td> nc </td> +<td> messaging.listen.port </td> +<td> IP port to bind messaging listener </td> +<td> 0 </td></tr> +<tr class="a"> +<td> nc </td> +<td> messaging.public.address </td> +<td> Public IP Address to announce messaging listener </td> +<td> same as public.address </td></tr> +<tr class="b"> +<td> nc </td> +<td> messaging.public.port </td> +<td> Public IP port to announce messaging listener </td> +<td> same as messaging.listen.port </td></tr> +<tr class="a"> +<td> nc </td> +<td> ncservice.address </td> +<td> Address the CC should use to contact the NCService associated with this NC </td> +<td> same as public.address </td></tr> +<tr class="b"> +<td> nc </td> +<td> ncservice.pid </td> +<td> PID of the NCService which launched this NCDriver </td> +<td> -1 </td></tr> +<tr class="a"> +<td> nc </td> +<td> ncservice.port </td> +<td> Port the CC should use to contact the NCService associated with this NC </td> +<td> 9090 </td></tr> +<tr class="b"> +<td> nc </td> +<td> net.buffer.count </td> +<td> Number of network buffers per input/output channel </td> +<td> 1 </td></tr> +<tr class="a"> +<td> nc </td> +<td> net.thread.count </td> +<td> Number of threads to use for Network I/O </td> +<td> 1 </td></tr> +<tr class="b"> +<td> nc </td> +<td> public.address </td> +<td> Default public address that other processes should use to contact this NC. All services will advertise this address unless a service-specific public address is supplied. </td> +<td> same as address </td></tr> +<tr class="a"> +<td> nc </td> +<td> result.listen.address </td> +<td> IP Address to bind dataset result distribution listener </td> +<td> same as address </td></tr> +<tr class="b"> +<td> nc </td> +<td> result.listen.port </td> +<td> IP port to bind dataset result distribution listener </td> +<td> 0 </td></tr> +<tr class="a"> +<td> nc </td> +<td> result.manager.memory </td> +<td> Memory usable for result caching at this Node Controller in bytes </td> +<td> -1 (-1 B) </td></tr> +<tr class="b"> +<td> nc </td> +<td> result.public.address </td> +<td> Public IP Address to announce dataset result distribution listener </td> +<td> same as public.address </td></tr> +<tr class="a"> +<td> nc </td> +<td> result.public.port </td> +<td> Public IP port to announce dataset result distribution listener </td> +<td> same as result.listen.port </td></tr> +<tr class="b"> +<td> nc </td> +<td> result.sweep.threshold </td> +<td> The duration within which an instance of the result cleanup should be invoked in milliseconds </td> +<td> 60000 </td></tr> +<tr class="a"> +<td> nc </td> +<td> result.ttl </td> +<td> Limits the amount of time results for asynchronous jobs should be retained by the system in milliseconds </td> +<td> 86400000 </td></tr> +<tr class="b"> +<td> nc </td> +<td> storage.buffercache.maxopenfiles </td> +<td> The maximum number of open files in the buffer cache </td> +<td> 2147483647 </td></tr> +<tr class="a"> +<td> nc </td> +<td> storage.buffercache.pagesize </td> +<td> The page size in bytes for pages in the buffer cache </td> +<td> 131072 (128 kB) </td></tr> +<tr class="b"> +<td> nc </td> +<td> storage.buffercache.size </td> +<td> The size of memory allocated to the disk buffer cache. The value should be a multiple of the buffer cache page size. </td> +<td> 1/4 of the JVM allocated memory </td></tr> +<tr class="a"> +<td> nc </td> +<td> storage.lsm.bloomfilter.falsepositiverate </td> +<td> The maximum acceptable false positive rate for bloom filters associated with LSM indexes </td> +<td> 0.01 </td></tr> +<tr class="b"> +<td> nc </td> +<td> storage.memorycomponent.globalbudget </td> +<td> The size of memory allocated to the memory components. The value should be a multiple of the memory component page size </td> +<td> 1/4 of the JVM allocated memory </td></tr> +<tr class="a"> +<td> nc </td> +<td> storage.memorycomponent.numcomponents </td> +<td> The number of memory components to be used per lsm index </td> +<td> 2 </td></tr> +<tr class="b"> +<td> nc </td> +<td> storage.memorycomponent.pagesize </td> +<td> The page size in bytes for pages allocated to memory components </td> +<td> 131072 (128 kB) </td></tr> +<tr class="a"> +<td> nc </td> +<td> storage.metadata.memorycomponent.numpages </td> +<td> The number of pages to allocate for a metadata memory component </td> +<td> 8 </td></tr> +<tr class="b"> +<td> nc </td> +<td> txn.log.dir </td> +<td> The directory where transaction logs should be stored </td> +<td> ${java.io.tmpdir}/asterixdb/txn-log </td></tr> +</tbody> +</table> +<p>The following parameters are configured under the “[common]” section.</p> +<table border="0" class="table table-striped"> +<thead> + +<tr class="a"> +<th> Section </th> +<th> Parameter </th> +<th> Meaning </th> +<th> Default </th></tr> +</thead><tbody> + +<tr class="b"> +<td> common </td> +<td> active.memory.global.budget </td> +<td> The memory budget (in bytes) for the active runtime </td> +<td> 67108864 (64 MB) </td></tr> +<tr class="a"> +<td> common </td> +<td> compiler.framesize </td> +<td> The page size (in bytes) for computation </td> +<td> 32768 (32 kB) </td></tr> +<tr class="b"> +<td> common </td> +<td> compiler.groupmemory </td> +<td> The memory budget (in bytes) for a group by operator instance in a partition </td> +<td> 33554432 (32 MB) </td></tr> +<tr class="a"> +<td> common </td> +<td> compiler.joinmemory </td> +<td> The memory budget (in bytes) for a join operator instance in a partition </td> +<td> 33554432 (32 MB) </td></tr> +<tr class="b"> +<td> common </td> +<td> compiler.parallelism </td> +<td> The degree of parallelism for query execution. Zero means to use the storage parallelism as the query execution parallelism, while other integer values dictate the number of query execution parallel partitions. The system will fall back to use the number of all available CPU cores in the cluster as the degree of parallelism if the number set by a user is too large or too small </td> +<td> 0 </td></tr> +<tr class="a"> +<td> common </td> +<td> compiler.sortmemory </td> +<td> The memory budget (in bytes) for a sort operator instance in a partition </td> +<td> 33554432 (32 MB) </td></tr> +<tr class="b"> +<td> common </td> +<td> compiler.textsearchmemory </td> +<td> The memory budget (in bytes) for an inverted-index-search operator instance in a partition </td> +<td> 33554432 (32 MB) </td></tr> +<tr class="a"> +<td> common </td> +<td> log.level </td> +<td> The logging level for master and slave processes </td> +<td> WARNING </td></tr> +<tr class="b"> +<td> common </td> +<td> max.wait.active.cluster </td> +<td> The max pending time (in seconds) for cluster startup. After the threshold, if the cluster still is not up and running, it is considered unavailable </td> +<td> 60 </td></tr> +<tr class="a"> +<td> common </td> +<td> messaging.frame.count </td> +<td> Number of reusable frames for NC to NC messaging </td> +<td> 512 </td></tr> +<tr class="b"> +<td> common </td> +<td> messaging.frame.size </td> +<td> The frame size to be used for NC to NC messaging </td> +<td> 4096 (4 kB) </td></tr> +<tr class="a"> +<td> common </td> +<td> metadata.callback.port </td> +<td> IP port to bind metadata callback listener (0 = random port) </td> +<td> 0 </td></tr> +<tr class="b"> +<td> common </td> +<td> metadata.listen.port </td> +<td> IP port to bind metadata listener (0 = random port) </td> +<td> 0 </td></tr> +<tr class="a"> +<td> common </td> +<td> metadata.node </td> +<td> the node which should serve as the metadata node </td> +<td> <undefined> </td></tr> +<tr class="b"> +<td> common </td> +<td> metadata.registration.timeout.secs </td> +<td> how long in seconds to wait for the metadata node to register with the CC </td> +<td> 60 </td></tr> +<tr class="a"> +<td> common </td> +<td> replication.log.batchsize </td> +<td> The size in bytes to replicate in each batch </td> +<td> 4096 (4 kB) </td></tr> +<tr class="b"> +<td> common </td> +<td> replication.log.buffer.numpages </td> +<td> The number of log buffer pages </td> +<td> 8 </td></tr> +<tr class="a"> +<td> common </td> +<td> replication.log.buffer.pagesize </td> +<td> The size in bytes of each log buffer page </td> +<td> 131072 (128 kB) </td></tr> +<tr class="b"> +<td> common </td> +<td> replication.max.remote.recovery.attempts </td> +<td> The maximum number of times to attempt to recover from a replica on failure before giving up </td> +<td> 5 </td></tr> +<tr class="a"> +<td> common </td> +<td> replication.timeout </td> +<td> The time in seconds to timeout when trying to contact a replica, before assuming it is dead </td> +<td> 15 </td></tr> +<tr class="b"> +<td> common </td> +<td> storage.max.active.writable.datasets </td> +<td> The maximum number of datasets that can be concurrently modified </td> +<td> 8 </td></tr> +<tr class="a"> +<td> common </td> +<td> txn.commitprofiler.enabled </td> +<td> Enable output of commit profiler logs </td> +<td> false </td></tr> +<tr class="b"> +<td> common </td> +<td> txn.commitprofiler.reportinterval </td> +<td> Interval (in seconds) to report commit profiler logs </td> +<td> 5 </td></tr> +<tr class="a"> +<td> common </td> +<td> txn.job.recovery.memorysize </td> +<td> The memory budget (in bytes) used for recovery </td> +<td> 67108864 (64 MB) </td></tr> +<tr class="b"> +<td> common </td> +<td> txn.lock.escalationthreshold </td> +<td> The maximum number of entity locks to obtain before upgrading to a dataset lock </td> +<td> 1000 </td></tr> +<tr class="a"> +<td> common </td> +<td> txn.lock.shrinktimer </td> +<td> The time (in milliseconds) where under utilization of resources will trigger a shrink phase </td> +<td> 5000 </td></tr> +<tr class="b"> +<td> common </td> +<td> txn.lock.timeout.sweepthreshold </td> +<td> Interval (in milliseconds) for checking lock timeout </td> +<td> 10000 </td></tr> +<tr class="a"> +<td> common </td> +<td> txn.lock.timeout.waitthreshold </td> +<td> Time out (in milliseconds) of waiting for a lock </td> +<td> 60000 </td></tr> +<tr class="b"> +<td> common </td> +<td> txn.log.buffer.numpages </td> +<td> The number of pages in the transaction log tail </td> +<td> 8 </td></tr> +<tr class="a"> +<td> common </td> +<td> txn.log.buffer.pagesize </td> +<td> The page size (in bytes) for transaction log buffer </td> +<td> 131072 (128 kB) </td></tr> +<tr class="b"> +<td> common </td> +<td> txn.log.checkpoint.history </td> +<td> The number of checkpoints to keep in the transaction log </td> +<td> 0 </td></tr> +<tr class="a"> +<td> common </td> +<td> txn.log.checkpoint.lsnthreshold </td> +<td> The checkpoint threshold (in terms of LSNs (log sequence numbers) that have been written to the transaction log, i.e., the length of the transaction log) for transaction logs </td> +<td> 67108864 (64 MB) </td></tr> +<tr class="b"> +<td> common </td> +<td> txn.log.checkpoint.pollfrequency </td> +<td> The frequency (in seconds) the checkpoint thread should check to see if a checkpoint should be written </td> +<td> 120 </td></tr> +<tr class="a"> +<td> common </td> +<td> txn.log.partitionsize </td> +<td> The maximum size (in bytes) of each transaction log file </td> +<td> 268435456 (256 MB) </td></tr> +</tbody> +</table> +<p>For the optional NCService process configuration file, the following parameters, under “[ncservice]” section.</p> +<table border="0" class="table table-striped"> +<thead> + +<tr class="a"> +<th> Parameter </th> +<th> Meaning </th> +<th> Default </th></tr> +</thead><tbody> + +<tr class="b"> +<td> address </td> +<td> The address the NCService listens on for commands from the CC </td> +<td> (all addresses) </td></tr> +<tr class="a"> +<td> port </td> +<td> The port for the NCService listens on for commands from the CC </td> +<td> 9090 </td></tr> +<tr class="b"> +<td> logdir </td> +<td> Directory where NCService logs should be written (‘-’ indicates that output should go to stdout) </td> +<td> ${app.home}/logs (${user.home} if ‘app.home’ not present in NCService Java system properties. </td></tr> +</tbody> +</table></div> + </div> + </div> + </div> + <hr/> + <footer> + <div class="container-fluid"> + <div class="row-fluid"> +<div class="row-fluid">Apache AsterixDB, AsterixDB, Apache, the Apache + feather logo, and the Apache AsterixDB project logo are either + registered trademarks or trademarks of The Apache Software + Foundation in the United States and other countries. + All other marks mentioned may be trademarks or registered + trademarks of their respective owners. + </div> + </div> + </div> + </footer> + </body> +</html>
http://git-wip-us.apache.org/repos/asf/asterixdb-site/blob/d00eca89/docs/0.9.4/overview.html ---------------------------------------------------------------------- diff --git a/docs/0.9.4/overview.html b/docs/0.9.4/overview.html new file mode 100644 index 0000000..4f087c8 --- /dev/null +++ b/docs/0.9.4/overview.html @@ -0,0 +1,157 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia Site Renderer 1.8.1 from src/site/markdown/overview.md at 2018-10-02 + | Rendered using Apache Maven Fluido Skin 1.7 +--> +<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20181002" /> + <meta http-equiv="Content-Language" content="en" /> + <title>AsterixDB – AsterixDB: A Big Data Management System</title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.7.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + <script type="text/javascript" src="./js/apache-maven-fluido-1.7.min.js"></script> + + </head> + <body class="topBarDisabled"> + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"><a href="./" id="bannerLeft"><img src="images/asterixlogo.png" alt="AsterixDB"/></a></div> + <div class="pull-right"></div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + <li id="publishDate">Last Published: 2018-10-02</li> + <li id="projectVersion" class="pull-right">Version: 0.9.4</li> + <li class="pull-right"><a href="index.html" title="Documentation Home">Documentation Home</a></li> + </ul> + </div> + <div class="row-fluid"> + <div id="leftColumn" class="span2"> + <div class="well sidebar-nav"> + <ul class="nav nav-list"> + <li class="nav-header">Get Started - Installation</li> + <li><a href="ncservice.html" title="Option 1: using NCService"><span class="none"></span>Option 1: using NCService</a></li> + <li><a href="ansible.html" title="Option 2: using Ansible"><span class="none"></span>Option 2: using Ansible</a></li> + <li><a href="aws.html" title="Option 3: using Amazon Web Services"><span class="none"></span>Option 3: using Amazon Web Services</a></li> + <li class="nav-header">AsterixDB Primer</li> + <li><a href="sqlpp/primer-sqlpp.html" title="Option 1: using SQL++"><span class="none"></span>Option 1: using SQL++</a></li> + <li><a href="aql/primer.html" title="Option 2: using AQL"><span class="none"></span>Option 2: using AQL</a></li> + <li class="nav-header">Data Model</li> + <li><a href="datamodel.html" title="The Asterix Data Model"><span class="none"></span>The Asterix Data Model</a></li> + <li class="nav-header">Queries - SQL++</li> + <li><a href="sqlpp/manual.html" title="The SQL++ Query Language"><span class="none"></span>The SQL++ Query Language</a></li> + <li><a href="sqlpp/builtins.html" title="Builtin Functions"><span class="none"></span>Builtin Functions</a></li> + <li class="nav-header">Queries - AQL</li> + <li><a href="aql/manual.html" title="The Asterix Query Language (AQL)"><span class="none"></span>The Asterix Query Language (AQL)</a></li> + <li><a href="aql/builtins.html" title="Builtin Functions"><span class="none"></span>Builtin Functions</a></li> + <li class="nav-header">API/SDK</li> + <li><a href="api.html" title="HTTP API"><span class="none"></span>HTTP API</a></li> + <li><a href="csv.html" title="CSV Output"><span class="none"></span>CSV Output</a></li> + <li class="nav-header">Advanced Features</li> + <li><a href="aql/fulltext.html" title="Support of Full-text Queries"><span class="none"></span>Support of Full-text Queries</a></li> + <li><a href="aql/externaldata.html" title="Accessing External Data"><span class="none"></span>Accessing External Data</a></li> + <li><a href="feeds/tutorial.html" title="Support for Data Ingestion"><span class="none"></span>Support for Data Ingestion</a></li> + <li><a href="udf.html" title="User Defined Functions"><span class="none"></span>User Defined Functions</a></li> + <li><a href="aql/filters.html" title="Filter-Based LSM Index Acceleration"><span class="none"></span>Filter-Based LSM Index Acceleration</a></li> + <li><a href="aql/similarity.html" title="Support of Similarity Queries"><span class="none"></span>Support of Similarity Queries</a></li> +</ul> + <hr /> + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> +<a href="./" title="AsterixDB" class="builtBy"><img class="builtBy" alt="AsterixDB" src="images/asterixlogo.png" /></a> + </div> + </div> + </div> + <div id="bodyColumn" class="span10" > +<!-- + ! Licensed to the Apache Software Foundation (ASF) under one + ! or more contributor license agreements. See the NOTICE file + ! distributed with this work for additional information + ! regarding copyright ownership. The ASF licenses this file + ! to you under the Apache License, Version 2.0 (the + ! "License"); you may not use this file except in compliance + ! with the License. You may obtain a copy of the License at + ! + ! http://www.apache.org/licenses/LICENSE-2.0 + ! + ! Unless required by applicable law or agreed to in writing, + ! software distributed under the License is distributed on an + ! "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + ! KIND, either express or implied. See the License for the + ! specific language governing permissions and limitations + ! under the License. + !--> +<h1>AsterixDB: A Big Data Management System</h1> +<div class="section"> +<h2><a name="Table_of_Contents"></a><a name="toc" id="toc">Table of Contents</a></h2> +<ul> + +<li><a href="#WhatIsAsterixDB">What Is AsterixDB?</a></li> +<li><a href="#GettingAndUsingAsterixDB">Getting and Using AsterixDB</a></li> +</ul></div> +<div class="section"> +<h2><a name="What_Is_AsterixDB.3F_.5BBack_to_TOC.5D"></a><a name="WhatIsAsterixDB" id="WhatIsAsterixDB">What Is AsterixDB?</a> <font size="4"><a href="#toc">[Back to TOC]</a></font></h2> +<p>In a nutshell, AsterixDB is a full-function BDMS (Big Data Management System) with a rich feature set that distinguishes it from pretty much any other Big Data platform that’s out and available today. We believe that its feature set makes it well-suited to modern needs such as web data warehousing and social data storage and analysis. AsterixDB has:</p> +<ul> + +<li>A semistructured NoSQL style data model (ADM) resulting from extending JSON with object database ideas</li> +<li>An expressive and declarative query language (AQL) that supports a broad range of queries and analysis over semistructured data</li> +<li>A parallel runtime query execution engine, Apache Hyracks, that has been scale-tested on up to 1000+ cores and 500+ disks</li> +<li>Partitioned LSM-based data storage and indexing to support efficient ingestion and management of semistructured data</li> +<li>Support for query access to externally stored data (e.g., data in HDFS) as well as to data stored natively by AsterixDB</li> +<li>A rich set of primitive data types, including spatial and temporal data in addition to integer, floating point, and textual data</li> +<li>Secondary indexing options that include B+ trees, R trees, and inverted keyword (exact and fuzzy) index types</li> +<li>Support for fuzzy and spatial queries as well as for more traditional parametric queries</li> +<li>Basic transactional (concurrency and recovery) capabilities akin to those of a NoSQL store</li> +</ul></div> +<div class="section"> +<h2><a name="Getting_and_Using_AsterixDB_.5BBack_to_TOC.5D"></a><a name="GettingAndUsingAsterixDB" id="GettingAndUsingAsterixDB">Getting and Using AsterixDB</a> <font size="4"><a href="#toc">[Back to TOC]</a></font></h2> +<p>You are most likely here because you are interested in getting your hands on AsterixDB—so you would like to know how to get it, how to set it up, and how to use it. The following is a list of the supporting documents that we have available today:</p> +<ol style="list-style-type: decimal"> + +<li> + +<p><a href="install.html">Installing AsterixDB using Managix</a> : This is our installation guide, and it is where you should start. This document will tell you how to obtain, install, and manage instances of <a class="externalLink" href="https://asterixdb.apache.org/download.html">AsterixDB</a>, including both single-machine setup (for developers) as well as cluster installations (for deployment in its intended form).</p> +</li> +<li> + +<p><a href="aql/primer.html">AsterixDB 101: An ADM and AQL Primer</a> : This is a first-timers introduction to the user model of the AsterixDB BDMS, by which we mean the view of AsterixDB as seen from the perspective of an “average user” or Big Data application developer. The AsterixDB user model consists of its data modeling features (ADM) and its query capabilities (AQL). This document presents a tiny “social data warehousing” example and uses it as a backdrop for describing, by example, the key features of AsterixDB. By working through this document, you will learn how to define the artifacts needed to manage data in AsterixDB, how to load data into the system, how to use most of the basic features of its query language, and how to insert and delete data dynamically.</p> +</li> +<li> + +<p><a href="aql/datamodel.html">Asterix Data Model (ADM)</a>, <a href="aql/functions.html">Asterix Functions</a>, <a href="aql/allens.html">Asterix functions for Allen’s Relations</a>, and <a href="aql/manual.html">Asterix Query Language (AQL)</a> : These are reference documents that catalog the primitive data types and built-in functions available in AQL and the reference manual for AQL itself.</p> +</li> +<li> + +<p><a href="api.html">REST API to AsterixDB</a> : Access to data in an AsterixDB instance is provided via a REST-based API. This is a short document that describes the REST API entry points and their URL syntax.</p> +</li> +</ol> +<p>To all who have now come this far: Thanks for your interest in AsterixDB, and for kicking its tires in its Beta form. In addition to getting the system and trying it out, please sign up as a member of the AsterixDB user mailing list (users (at) asterixdb.apache.org) so that you can contact us easily with your questions, issues, and other feedback. We want AsterixDB to be a “big hit” some day, and we are anxious to see what users do with it and to learn from that feedback what we should be working on most urgently in the next phase of the project.</p></div> + </div> + </div> + </div> + <hr/> + <footer> + <div class="container-fluid"> + <div class="row-fluid"> +<div class="row-fluid">Apache AsterixDB, AsterixDB, Apache, the Apache + feather logo, and the Apache AsterixDB project logo are either + registered trademarks or trademarks of The Apache Software + Foundation in the United States and other countries. + All other marks mentioned may be trademarks or registered + trademarks of their respective owners. + </div> + </div> + </div> + </footer> + </body> +</html>