http://git-wip-us.apache.org/repos/asf/asterixdb-site/blob/b2f1d3e6/content/docs/0.9.4/aql/externaldata.html ---------------------------------------------------------------------- diff --git a/content/docs/0.9.4/aql/externaldata.html b/content/docs/0.9.4/aql/externaldata.html index 6c06dc9..5cbe512 100644 --- a/content/docs/0.9.4/aql/externaldata.html +++ b/content/docs/0.9.4/aql/externaldata.html @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia Site Renderer 1.8.1 from src/site/markdown/aql/externaldata.md at 2018-10-02 + | Generated by Apache Maven Doxia Site Renderer 1.8.1 from src/site/markdown/aql/externaldata.md at 2018-10-11 | Rendered using Apache Maven Fluido Skin 1.7 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20181002" /> + <meta name="Date-Revision-yyyymmdd" content="20181011" /> <meta http-equiv="Content-Language" content="en" /> <title>AsterixDB – Accessing External Data in AsterixDB</title> <link rel="stylesheet" href="../css/apache-maven-fluido-1.7.min.css" /> @@ -26,8 +26,8 @@ <div id="breadcrumbs"> <ul class="breadcrumb"> - <li id="publishDate">Last Published: 2018-10-02</li> - <li id="projectVersion" class="pull-right">Version: 0.9.4</li> + <li id="publishDate">Last Published: 2018-10-11</li> + <li id="projectVersion" class="pull-right">Version: 0.9.5-SNAPSHOT</li> <li class="pull-right"><a href="../index.html" title="Documentation Home">Documentation Home</a></li> </ul> </div> @@ -40,26 +40,26 @@ <li><a href="../ansible.html" title="Option 2: using Ansible"><span class="none"></span>Option 2: using Ansible</a></li> <li><a href="../aws.html" title="Option 3: using Amazon Web Services"><span class="none"></span>Option 3: using Amazon Web Services</a></li> <li class="nav-header">AsterixDB Primer</li> - <li><a href="../sqlpp/primer-sqlpp.html" title="Option 1: using SQL++"><span class="none"></span>Option 1: using SQL++</a></li> - <li><a href="../aql/primer.html" title="Option 2: using AQL"><span class="none"></span>Option 2: using AQL</a></li> + <li><a href="../sqlpp/primer-sqlpp.html" title="Using SQL++"><span class="none"></span>Using SQL++</a></li> <li class="nav-header">Data Model</li> <li><a href="../datamodel.html" title="The Asterix Data Model"><span class="none"></span>The Asterix Data Model</a></li> - <li class="nav-header">Queries - SQL++</li> + <li class="nav-header">Queries</li> <li><a href="../sqlpp/manual.html" title="The SQL++ Query Language"><span class="none"></span>The SQL++ Query Language</a></li> <li><a href="../sqlpp/builtins.html" title="Builtin Functions"><span class="none"></span>Builtin Functions</a></li> - <li class="nav-header">Queries - AQL</li> - <li><a href="../aql/manual.html" title="The Asterix Query Language (AQL)"><span class="none"></span>The Asterix Query Language (AQL)</a></li> - <li><a href="../aql/builtins.html" title="Builtin Functions"><span class="none"></span>Builtin Functions</a></li> <li class="nav-header">API/SDK</li> <li><a href="../api.html" title="HTTP API"><span class="none"></span>HTTP API</a></li> <li><a href="../csv.html" title="CSV Output"><span class="none"></span>CSV Output</a></li> <li class="nav-header">Advanced Features</li> - <li><a href="../aql/fulltext.html" title="Support of Full-text Queries"><span class="none"></span>Support of Full-text Queries</a></li> <li class="active"><a href="#"><span class="none"></span>Accessing External Data</a></li> - <li><a href="../feeds/tutorial.html" title="Support for Data Ingestion"><span class="none"></span>Support for Data Ingestion</a></li> + <li><a href="../feeds.html" title="Data Ingestion with Feeds"><span class="none"></span>Data Ingestion with Feeds</a></li> <li><a href="../udf.html" title="User Defined Functions"><span class="none"></span>User Defined Functions</a></li> - <li><a href="../aql/filters.html" title="Filter-Based LSM Index Acceleration"><span class="none"></span>Filter-Based LSM Index Acceleration</a></li> - <li><a href="../aql/similarity.html" title="Support of Similarity Queries"><span class="none"></span>Support of Similarity Queries</a></li> + <li><a href="../sqlpp/filters.html" title="Filter-Based LSM Index Acceleration"><span class="none"></span>Filter-Based LSM Index Acceleration</a></li> + <li><a href="../sqlpp/fulltext.html" title="Support of Full-text Queries"><span class="none"></span>Support of Full-text Queries</a></li> + <li><a href="../sqlpp/similarity.html" title="Support of Similarity Queries"><span class="none"></span>Support of Similarity Queries</a></li> + <li class="nav-header">Deprecated</li> + <li><a href="../aql/primer.html" title="AsterixDB Primer: Using AQL"><span class="none"></span>AsterixDB Primer: Using AQL</a></li> + <li><a href="../aql/manual.html" title="Queries: The Asterix Query Language (AQL)"><span class="none"></span>Queries: The Asterix Query Language (AQL)</a></li> + <li><a href="../aql/builtins.html" title="Queries: Builtin Functions (AQL)"><span class="none"></span>Queries: Builtin Functions (AQL)</a></li> </ul> <hr /> <div id="poweredBy">
http://git-wip-us.apache.org/repos/asf/asterixdb-site/blob/b2f1d3e6/content/docs/0.9.4/aql/filters.html ---------------------------------------------------------------------- diff --git a/content/docs/0.9.4/aql/filters.html b/content/docs/0.9.4/aql/filters.html deleted file mode 100644 index 8c18a52..0000000 --- a/content/docs/0.9.4/aql/filters.html +++ /dev/null @@ -1,142 +0,0 @@ -<!DOCTYPE html> -<!-- - | Generated by Apache Maven Doxia Site Renderer 1.8.1 from src/site/markdown/aql/filters.md at 2018-10-02 - | Rendered using Apache Maven Fluido Skin 1.7 ---> -<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> - <head> - <meta charset="UTF-8" /> - <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20181002" /> - <meta http-equiv="Content-Language" content="en" /> - <title>AsterixDB – Filter-Based LSM Index Acceleration</title> - <link rel="stylesheet" href="../css/apache-maven-fluido-1.7.min.css" /> - <link rel="stylesheet" href="../css/site.css" /> - <link rel="stylesheet" href="../css/print.css" media="print" /> - <script type="text/javascript" src="../js/apache-maven-fluido-1.7.min.js"></script> - - </head> - <body class="topBarDisabled"> - <div class="container-fluid"> - <div id="banner"> - <div class="pull-left"><a href=".././" id="bannerLeft"><img src="../images/asterixlogo.png" alt="AsterixDB"/></a></div> - <div class="pull-right"></div> - <div class="clear"><hr/></div> - </div> - - <div id="breadcrumbs"> - <ul class="breadcrumb"> - <li id="publishDate">Last Published: 2018-10-02</li> - <li id="projectVersion" class="pull-right">Version: 0.9.4</li> - <li class="pull-right"><a href="../index.html" title="Documentation Home">Documentation Home</a></li> - </ul> - </div> - <div class="row-fluid"> - <div id="leftColumn" class="span2"> - <div class="well sidebar-nav"> - <ul class="nav nav-list"> - <li class="nav-header">Get Started - Installation</li> - <li><a href="../ncservice.html" title="Option 1: using NCService"><span class="none"></span>Option 1: using NCService</a></li> - <li><a href="../ansible.html" title="Option 2: using Ansible"><span class="none"></span>Option 2: using Ansible</a></li> - <li><a href="../aws.html" title="Option 3: using Amazon Web Services"><span class="none"></span>Option 3: using Amazon Web Services</a></li> - <li class="nav-header">AsterixDB Primer</li> - <li><a href="../sqlpp/primer-sqlpp.html" title="Option 1: using SQL++"><span class="none"></span>Option 1: using SQL++</a></li> - <li><a href="../aql/primer.html" title="Option 2: using AQL"><span class="none"></span>Option 2: using AQL</a></li> - <li class="nav-header">Data Model</li> - <li><a href="../datamodel.html" title="The Asterix Data Model"><span class="none"></span>The Asterix Data Model</a></li> - <li class="nav-header">Queries - SQL++</li> - <li><a href="../sqlpp/manual.html" title="The SQL++ Query Language"><span class="none"></span>The SQL++ Query Language</a></li> - <li><a href="../sqlpp/builtins.html" title="Builtin Functions"><span class="none"></span>Builtin Functions</a></li> - <li class="nav-header">Queries - AQL</li> - <li><a href="../aql/manual.html" title="The Asterix Query Language (AQL)"><span class="none"></span>The Asterix Query Language (AQL)</a></li> - <li><a href="../aql/builtins.html" title="Builtin Functions"><span class="none"></span>Builtin Functions</a></li> - <li class="nav-header">API/SDK</li> - <li><a href="../api.html" title="HTTP API"><span class="none"></span>HTTP API</a></li> - <li><a href="../csv.html" title="CSV Output"><span class="none"></span>CSV Output</a></li> - <li class="nav-header">Advanced Features</li> - <li><a href="../aql/fulltext.html" title="Support of Full-text Queries"><span class="none"></span>Support of Full-text Queries</a></li> - <li><a href="../aql/externaldata.html" title="Accessing External Data"><span class="none"></span>Accessing External Data</a></li> - <li><a href="../feeds/tutorial.html" title="Support for Data Ingestion"><span class="none"></span>Support for Data Ingestion</a></li> - <li><a href="../udf.html" title="User Defined Functions"><span class="none"></span>User Defined Functions</a></li> - <li class="active"><a href="#"><span class="none"></span>Filter-Based LSM Index Acceleration</a></li> - <li><a href="../aql/similarity.html" title="Support of Similarity Queries"><span class="none"></span>Support of Similarity Queries</a></li> -</ul> - <hr /> - <div id="poweredBy"> - <div class="clear"></div> - <div class="clear"></div> - <div class="clear"></div> - <div class="clear"></div> -<a href=".././" title="AsterixDB" class="builtBy"><img class="builtBy" alt="AsterixDB" src="../images/asterixlogo.png" /></a> - </div> - </div> - </div> - <div id="bodyColumn" class="span10" > -<!-- - ! Licensed to the Apache Software Foundation (ASF) under one - ! or more contributor license agreements. See the NOTICE file - ! distributed with this work for additional information - ! regarding copyright ownership. The ASF licenses this file - ! to you under the Apache License, Version 2.0 (the - ! "License"); you may not use this file except in compliance - ! with the License. You may obtain a copy of the License at - ! - ! http://www.apache.org/licenses/LICENSE-2.0 - ! - ! Unless required by applicable law or agreed to in writing, - ! software distributed under the License is distributed on an - ! "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - ! KIND, either express or implied. See the License for the - ! specific language governing permissions and limitations - ! under the License. - !--> -<h1>Filter-Based LSM Index Acceleration</h1> -<div class="section"> -<h2><a name="Table_of_Contents"></a><a name="toc" id="toc">Table of Contents</a></h2> -<ul> - -<li><a href="#Motivation">Motivation</a></li> -<li><a href="#FiltersInAsterixDB">Filters in AsterixDB</a></li> -<li><a href="#FiltersAndMergePolicies">Filters and Merge Policies</a></li> -</ul></div> -<div class="section"> -<h2><a name="Motivation_.5BBack_to_TOC.5D"></a><a name="Motivation" id="Motivation">Motivation</a> <font size="4"><a href="#toc">[Back to TOC]</a></font></h2> -<p>Traditional relational databases usually employ conventional index structures such as B+ trees due to their low read latency. However, such traditional index structures use in-place writes to perform updates, resulting in costly random writes to disk. Today’s emerging applications often involve insert-intensive workloads for which the cost of random writes prohibits efficient ingestion of data. Consequently, popular NoSQL systems such as Cassandra, HBase, LevelDB, BigTable, etc. have adopted Log-Structured Merge (LSM) Trees as their storage structure. LSM-trees avoids the cost of random writes by batching updates into a component of the index that resides in main memory – an <i>in-memory component</i>. When the space occupancy of the in-memory component exceeds a specified threshold, its entries are <i>flushed</i> to disk forming a new component – a <i>disk component</i>. As disk components accumulate on disk, they are periodically merged together subject to a <i>merge policy</i> that decides when and what to merge. The benefit of the LSM-trees comes at the cost of possibly sacrificing read efficiency, but, it has been shown in previous studies that these inefficiencies can be mostly mitigated.</p> -<p>AsterixDB has also embraced LSM-trees, not just by using them as primary indexes, but also by using the same LSM-ification technique for all of its secondary index structures. In particular, AsterixDB adopted a generic framework for converting a class of indexes (that includes conventional B+ trees, R trees, and inverted indexes) into LSM-based secondary indexes, allowing higher data ingestion rates. In fact, for certain index structures, our results have shown that using an LSM-based version of an index can be made to significantly outperform its conventional counterpart for <i>both</i> ingestion and query speed (an example of such an index being the R-tree for spatial data).</p> -<p>Since an LSM-based index naturally partitions data into multiple disk components, it is possible, when answering certain queries, to exploit partitioning to only access some components and safely filter out the remaining components, thus reducing query times. For instance, referring to our <a href="primer.html#ADM:_Modeling_Semistructed_Data_in_AsterixDB">TinySocial</a> example, suppose a user always retrieves tweets from the <tt>TweetMessages</tt> dataset based on the <tt>send-time</tt> field (e.g., tweets posted in the last 24 hours). Since there is not a secondary index on the <tt>send-time</tt> field, the only available option for AsterixDB would be to scan the whole <tt>TweetMessages</tt> dataset and then apply the predicate as a post-processing step. However, if disk components of the primary index were tagged with the minimum and maximum timestamp values of the objects they contain, we could utilize the tagged information to directly access the primary index and prune comp onents that do not match the query predicate. Thus, we could save substantial cost by avoiding scanning the whole dataset and only access the relevant components. We simply call such tagging information that are associated with components, filters. (Note that even if there were a secondary index on <tt>send-time</tt> field, using filters could save substantial cost by avoiding accessing the secondary index, followed by probing the primary index for every fetched entry.) Moreover, the same filtering technique can also be used with any secondary LSM index (e.g., an LSM R-tree), in case the query contains multiple predicates (e.g., spatial and temporal predicates), to obtain similar pruning power.</p></div> -<div class="section"> -<h2><a name="Filters_in_AsterixDB_.5BBack_to_TOC.5D"></a><a name="FiltersInAsterixDB" id="FiltersInAsterixDB">Filters in AsterixDB</a> <font size="4"><a href="#toc">[Back to TOC]</a></font></h2> -<p>We have added support for LSM-based filters to all of AsterixDB’s index types. To enable the use of filters, the user must specify the filter’s key when creating a dataset, as shown below:</p> -<div class="section"> -<div class="section"> -<h4><a name="Creating_a_Dataset_with_a_Filter"></a>Creating a Dataset with a Filter</h4> - -<div> -<div> -<pre class="source"> create dataset Tweets(TweetType) primary key tweetid with filter on send-time; -</pre></div></div> - -<p>Filters can be created on any totally ordered datatype (i.e., any field that can be indexed using a B+ -tree), such as integers, doubles, floats, UUIDs, datetimes, etc.</p> -<p>When a dataset with a filter is created, the name of the filter’s key field is persisted in the <tt>Metadata.Dataset</tt> dataset (which is the metadata dataset that stores the details of each dataset in an AsterixDB instance) so that DML operations against the dataset can recognize the existence of filters and can update them or utilize them accordingly. Creating a dataset with a filter in AsterixDB implies that the primary and all secondary indexes of that dataset will maintain filters on their disk components. Once a filtered dataset is created, the user can use the dataset normally (just like any other dataset). AsterixDB will automatically maintain the filters and will leverage them to efficiently answer queries whenever possible (i.e., when a query has predicates on the filter’s key).</p></div></div></div> -<div class="section"> -<h2><a name="Filters_and_Merge_Policies_.5BBack_to_TOC.5D"></a><a name="FiltersAndMergePolicies" id="FiltersAndMergePolicies">Filters and Merge Policies</a> <font size="4"><a href="#toc">[Back to TOC]</a></font></h2> -<p>The AsterixDB default merge policy, the prefix merge policy, relies on component sizes and the number of components to decide which components to merge. This merge policy has proven to provide excellent performance for both ingestion and queries. However, when evaluating our filtering solution with the prefix policy, we observed a behavior that can reduce filter effectiveness. In particular, we noticed that under the prefix merge policy, the disk components of a secondary index tend to be constantly merged into a single component. This is because the prefix policy relies on a single size parameter for all of the indexes of a dataset. This parameter is typically chosen based on the sizes of the disk components of the primary index, which tend to be much larger than the sizes of the secondary indexes’ disk components. This difference caused the prefix merge policy to behave similarly to the constant merge policy (i.e., relatively poorly) when applied to secondary indexes in the sense that the secondary indexes are constantly merged into a single disk component. Consequently, the effectiveness of filters on secondary indexes was greatly reduced under the prefix-merge policy, but they were still effective when probing the primary index. Based on this behavior, we developed a new merge policy, an improved version of the prefix policy, called the correlated-prefix policy. The basic idea of this policy is that it delegates the decision of merging the disk components of all the indexes in a dataset to the primary index. When the policy decides that the primary index needs to be merged (using the same decision criteria as for the prefix policy), then it will issue successive merge requests to the I/O scheduler on behalf of all other indexes associated with the same dataset. The end result is that secondary indexes will always have the same number of disk components as their primary index under the correlated-prefix merge policy. This has improved query perfo rmance, since disk components of secondary indexes now have a much better chance of being pruned.</p></div> - </div> - </div> - </div> - <hr/> - <footer> - <div class="container-fluid"> - <div class="row-fluid"> -<div class="row-fluid">Apache AsterixDB, AsterixDB, Apache, the Apache - feather logo, and the Apache AsterixDB project logo are either - registered trademarks or trademarks of The Apache Software - Foundation in the United States and other countries. - All other marks mentioned may be trademarks or registered - trademarks of their respective owners. - </div> - </div> - </div> - </footer> - </body> -</html> http://git-wip-us.apache.org/repos/asf/asterixdb-site/blob/b2f1d3e6/content/docs/0.9.4/aql/fulltext.html ---------------------------------------------------------------------- diff --git a/content/docs/0.9.4/aql/fulltext.html b/content/docs/0.9.4/aql/fulltext.html deleted file mode 100644 index 022e124..0000000 --- a/content/docs/0.9.4/aql/fulltext.html +++ /dev/null @@ -1,208 +0,0 @@ -<!DOCTYPE html> -<!-- - | Generated by Apache Maven Doxia Site Renderer 1.8.1 from src/site/markdown/aql/fulltext.md at 2018-10-02 - | Rendered using Apache Maven Fluido Skin 1.7 ---> -<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> - <head> - <meta charset="UTF-8" /> - <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20181002" /> - <meta http-equiv="Content-Language" content="en" /> - <title>AsterixDB – AsterixDB Support of Full-text search queries</title> - <link rel="stylesheet" href="../css/apache-maven-fluido-1.7.min.css" /> - <link rel="stylesheet" href="../css/site.css" /> - <link rel="stylesheet" href="../css/print.css" media="print" /> - <script type="text/javascript" src="../js/apache-maven-fluido-1.7.min.js"></script> - - </head> - <body class="topBarDisabled"> - <div class="container-fluid"> - <div id="banner"> - <div class="pull-left"><a href=".././" id="bannerLeft"><img src="../images/asterixlogo.png" alt="AsterixDB"/></a></div> - <div class="pull-right"></div> - <div class="clear"><hr/></div> - </div> - - <div id="breadcrumbs"> - <ul class="breadcrumb"> - <li id="publishDate">Last Published: 2018-10-02</li> - <li id="projectVersion" class="pull-right">Version: 0.9.4</li> - <li class="pull-right"><a href="../index.html" title="Documentation Home">Documentation Home</a></li> - </ul> - </div> - <div class="row-fluid"> - <div id="leftColumn" class="span2"> - <div class="well sidebar-nav"> - <ul class="nav nav-list"> - <li class="nav-header">Get Started - Installation</li> - <li><a href="../ncservice.html" title="Option 1: using NCService"><span class="none"></span>Option 1: using NCService</a></li> - <li><a href="../ansible.html" title="Option 2: using Ansible"><span class="none"></span>Option 2: using Ansible</a></li> - <li><a href="../aws.html" title="Option 3: using Amazon Web Services"><span class="none"></span>Option 3: using Amazon Web Services</a></li> - <li class="nav-header">AsterixDB Primer</li> - <li><a href="../sqlpp/primer-sqlpp.html" title="Option 1: using SQL++"><span class="none"></span>Option 1: using SQL++</a></li> - <li><a href="../aql/primer.html" title="Option 2: using AQL"><span class="none"></span>Option 2: using AQL</a></li> - <li class="nav-header">Data Model</li> - <li><a href="../datamodel.html" title="The Asterix Data Model"><span class="none"></span>The Asterix Data Model</a></li> - <li class="nav-header">Queries - SQL++</li> - <li><a href="../sqlpp/manual.html" title="The SQL++ Query Language"><span class="none"></span>The SQL++ Query Language</a></li> - <li><a href="../sqlpp/builtins.html" title="Builtin Functions"><span class="none"></span>Builtin Functions</a></li> - <li class="nav-header">Queries - AQL</li> - <li><a href="../aql/manual.html" title="The Asterix Query Language (AQL)"><span class="none"></span>The Asterix Query Language (AQL)</a></li> - <li><a href="../aql/builtins.html" title="Builtin Functions"><span class="none"></span>Builtin Functions</a></li> - <li class="nav-header">API/SDK</li> - <li><a href="../api.html" title="HTTP API"><span class="none"></span>HTTP API</a></li> - <li><a href="../csv.html" title="CSV Output"><span class="none"></span>CSV Output</a></li> - <li class="nav-header">Advanced Features</li> - <li class="active"><a href="#"><span class="none"></span>Support of Full-text Queries</a></li> - <li><a href="../aql/externaldata.html" title="Accessing External Data"><span class="none"></span>Accessing External Data</a></li> - <li><a href="../feeds/tutorial.html" title="Support for Data Ingestion"><span class="none"></span>Support for Data Ingestion</a></li> - <li><a href="../udf.html" title="User Defined Functions"><span class="none"></span>User Defined Functions</a></li> - <li><a href="../aql/filters.html" title="Filter-Based LSM Index Acceleration"><span class="none"></span>Filter-Based LSM Index Acceleration</a></li> - <li><a href="../aql/similarity.html" title="Support of Similarity Queries"><span class="none"></span>Support of Similarity Queries</a></li> -</ul> - <hr /> - <div id="poweredBy"> - <div class="clear"></div> - <div class="clear"></div> - <div class="clear"></div> - <div class="clear"></div> -<a href=".././" title="AsterixDB" class="builtBy"><img class="builtBy" alt="AsterixDB" src="../images/asterixlogo.png" /></a> - </div> - </div> - </div> - <div id="bodyColumn" class="span10" > -<!-- - ! Licensed to the Apache Software Foundation (ASF) under one - ! or more contributor license agreements. See the NOTICE file - ! distributed with this work for additional information - ! regarding copyright ownership. The ASF licenses this file - ! to you under the Apache License, Version 2.0 (the - ! "License"); you may not use this file except in compliance - ! with the License. You may obtain a copy of the License at - ! - ! http://www.apache.org/licenses/LICENSE-2.0 - ! - ! Unless required by applicable law or agreed to in writing, - ! software distributed under the License is distributed on an - ! "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - ! KIND, either express or implied. See the License for the - ! specific language governing permissions and limitations - ! under the License. - !--> -<h1>AsterixDB Support of Full-text search queries</h1> -<div class="section"> -<h2><a name="Table_of_Contents"></a><a name="toc" id="toc">Table of Contents</a></h2> -<ul> - -<li><a href="#Motivation">Motivation</a></li> -<li><a href="#Syntax">Syntax</a></li> -<li><a href="#FulltextIndex">Creating and utilizing a Full-text index</a></li> -</ul></div> -<div class="section"> -<h2><a name="Motivation_.5BBack_to_TOC.5D"></a><a name="Motivation" id="Motivation">Motivation</a> <font size="4"><a href="#toc">[Back to TOC]</a></font></h2> -<p>Full-Text Search (FTS) queries are widely used in applications where users need to find records that satisfy an FTS predicate, i.e., where simple string-based matching is not sufficient. These queries are important when finding documents that contain a certain keyword is crucial. FTS queries are different from substring matching queries in that FTS queries find their query predicates as exact keywords in the given string, rather than treating a query predicate as a sequence of characters. For example, an FTS query that finds “rain” correctly returns a document when it contains “rain” as a word. However, a substring-matching query returns a document whenever it contains “rain” as a substring, for instance, a document with “brain” or “training” would be returned as well.</p></div> -<div class="section"> -<h2><a name="Syntax_.5BBack_to_TOC.5D"></a><a name="Syntax" id="Syntax">Syntax</a> <font size="4"><a href="#toc">[Back to TOC]</a></font></h2> -<p>The syntax of AsterixDB FTS follows a portion of the XQuery FullText Search syntax. Two basic forms are as follows:</p> - -<div> -<div> -<pre class="source"> ftcontains(Expression1, Expression2, {FullTextOption}) - ftcontains(Expression1, Expression2) -</pre></div></div> - -<p>For example, we can execute the following query to find tweet messages where the <tt>message-text</tt> field includes “voice” as a word. Please note that an FTS search is case-insensitive. Thus, “Voice” or “voice” will be evaluated as the same word.</p> - -<div> -<div> -<pre class="source"> use dataverse TinySocial; - - for $msg in dataset TweetMessages - where ftcontains($msg.message-text, "voice", {"mode":"any"}) - return {"id": $msg.id} -</pre></div></div> - -<p>The DDL and DML of TinySocial can be found in <a href="primer.html#ADM:_Modeling_Semistructed_Data_in_AsterixDB">ADM: Modeling Semistructed Data in AsterixDB</a>.</p> -<p>The same query can be also expressed in the SQL++.</p> - -<div> -<div> -<pre class="source"> use TinySocial; - - select element {"id":msg.id} - from TweetMessages as msg - where TinySocial.ftcontains(msg.`message-text`, "voice", {"mode":"any"}) -</pre></div></div> - -<p>The <tt>Expression1</tt> is an expression that should be evaluable as a string at runtime as in the above example where <tt>$msg.message-text</tt> is a string field. The <tt>Expression2</tt> can be a string, an (un)ordered list of string value(s), or an expression. In the last case, the given expression should be evaluable into one of the first two types, i.e., into a string value or an (un)ordered list of string value(s).</p> -<p>The following examples are all valid expressions.</p> - -<div> -<div> -<pre class="source"> ... where ftcontains($msg.message-text, "sound") - ... where ftcontains($msg.message-text, "sound", {"mode":"any"}) - ... where ftcontains($msg.message-text, ["sound", "system"], {"mode":"any"}) - ... where ftcontains($msg.message-text, {{"speed", "stand", "customization"}}, {"mode":"all"}) - ... where ftcontains($msg.message-text, let $keyword_list := ["voice", "system"] return $keyword_list, {"mode":"all"}) - ... where ftcontains($msg.message-text, $keyword_list, {"mode":"any"}) -</pre></div></div> - -<p>In the last example above, <tt>$keyword_list</tt> should evaluate to a string or an (un)ordered list of string value(s).</p> -<p>The last <tt>FullTextOption</tt> parameter clarifies the given FTS request. If you omit the <tt>FullTextOption</tt> parameter, then the default value will be set for each possible option. Currently, we only have one option named <tt>mode</tt>. And as we extend the FTS feature, more options will be added. Please note that the format of <tt>FullTextOption</tt> is a record, thus you need to put the option(s) in a record <tt>{}</tt>. The <tt>mode</tt> option indicates whether the given FTS query is a conjunctive (AND) or disjunctive (OR) search request. This option can be either <tt>“any”</tt> or <tt>“all”</tt>. The default value for <tt>mode</tt> is <tt>“all”</tt>. If one specifies <tt>“any”</tt>, a disjunctive search will be conducted. For example, the following query will find documents whose <tt>message-text</tt> field contains “sound” or “system”, so a document will be returned if it contains eit her “sound”, “system”, or both of the keywords.</p> - -<div> -<div> -<pre class="source"> ... where ftcontains($msg.message-text, ["sound", "system"], {"mode":"any"}) -</pre></div></div> - -<p>The other option parameter,<tt>“all”</tt>, specifies a conjunctive search. The following examples will find the documents whose <tt>message-text</tt> field contains both “sound” and “system”. If a document contains only “sound” or “system” but not both, it will not be returned.</p> - -<div> -<div> -<pre class="source"> ... where ftcontains($msg.message-text, ["sound", "system"], {"mode":"all"}) - ... where ftcontains($msg.message-text, ["sound", "system"]) -</pre></div></div> - -<p>Currently AsterixDB doesn’t (yet) support phrase searches, so the following query will not work.</p> - -<div> -<div> -<pre class="source"> ... where ftcontains($msg.message-text, "sound system", {"mode":"any"}) -</pre></div></div> - -<p>As a workaround solution, the following query can be used to achieve a roughly similar goal. The difference is that the following queries will find documents where <tt>$msg.message-text</tt> contains both “sound” and “system”, but the order and adjacency of “sound” and “system” are not checked, unlike in a phrase search. As a result, the query below would also return documents with “sound system can be installed.”, “system sound is perfect.”, or “sound is not clear. You may need to install a new system.”</p> - -<div> -<div> -<pre class="source"> ... where ftcontains($msg.message-text, ["sound", "system"], {"mode":"all"}) - ... where ftcontains($msg.message-text, ["sound", "system"]) -</pre></div></div> -</div> -<div class="section"> -<h2><a name="Creating_and_utilizing_a_Full-text_index_.5BBack_to_TOC.5D"></a><a name="FulltextIndex" id="FulltextIndex">Creating and utilizing a Full-text index</a> <font size="4"><a href="#toc">[Back to TOC]</a></font></h2> -<p>When there is a full-text index on the field that is being searched, rather than scanning all records, AsterixDB can utilize that index to expedite the execution of a FTS query. To create a full-text index, you need to specify the index type as <tt>fulltext</tt> in your DDL statement. For instance, the following DDL statement create a full-text index on the TweetMessages.message-text attribute.</p> - -<div> -<div> -<pre class="source">create index messageFTSIdx on TweetMessages(message-text) type fulltext; -</pre></div></div></div> - </div> - </div> - </div> - <hr/> - <footer> - <div class="container-fluid"> - <div class="row-fluid"> -<div class="row-fluid">Apache AsterixDB, AsterixDB, Apache, the Apache - feather logo, and the Apache AsterixDB project logo are either - registered trademarks or trademarks of The Apache Software - Foundation in the United States and other countries. - All other marks mentioned may be trademarks or registered - trademarks of their respective owners. - </div> - </div> - </div> - </footer> - </body> -</html> http://git-wip-us.apache.org/repos/asf/asterixdb-site/blob/b2f1d3e6/content/docs/0.9.4/aql/js-sdk.html ---------------------------------------------------------------------- diff --git a/content/docs/0.9.4/aql/js-sdk.html b/content/docs/0.9.4/aql/js-sdk.html index 50700e9..c5902e4 100644 --- a/content/docs/0.9.4/aql/js-sdk.html +++ b/content/docs/0.9.4/aql/js-sdk.html @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia Site Renderer 1.8.1 from src/site/markdown/aql/js-sdk.md at 2018-10-02 + | Generated by Apache Maven Doxia Site Renderer 1.8.1 from src/site/markdown/aql/js-sdk.md at 2018-10-11 | Rendered using Apache Maven Fluido Skin 1.7 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20181002" /> + <meta name="Date-Revision-yyyymmdd" content="20181011" /> <meta http-equiv="Content-Language" content="en" /> <title>AsterixDB – AsterixDB Javascript SDK</title> <link rel="stylesheet" href="../css/apache-maven-fluido-1.7.min.css" /> @@ -26,8 +26,8 @@ <div id="breadcrumbs"> <ul class="breadcrumb"> - <li id="publishDate">Last Published: 2018-10-02</li> - <li id="projectVersion" class="pull-right">Version: 0.9.4</li> + <li id="publishDate">Last Published: 2018-10-11</li> + <li id="projectVersion" class="pull-right">Version: 0.9.5-SNAPSHOT</li> <li class="pull-right"><a href="../index.html" title="Documentation Home">Documentation Home</a></li> </ul> </div> @@ -40,26 +40,26 @@ <li><a href="../ansible.html" title="Option 2: using Ansible"><span class="none"></span>Option 2: using Ansible</a></li> <li><a href="../aws.html" title="Option 3: using Amazon Web Services"><span class="none"></span>Option 3: using Amazon Web Services</a></li> <li class="nav-header">AsterixDB Primer</li> - <li><a href="../sqlpp/primer-sqlpp.html" title="Option 1: using SQL++"><span class="none"></span>Option 1: using SQL++</a></li> - <li><a href="../aql/primer.html" title="Option 2: using AQL"><span class="none"></span>Option 2: using AQL</a></li> + <li><a href="../sqlpp/primer-sqlpp.html" title="Using SQL++"><span class="none"></span>Using SQL++</a></li> <li class="nav-header">Data Model</li> <li><a href="../datamodel.html" title="The Asterix Data Model"><span class="none"></span>The Asterix Data Model</a></li> - <li class="nav-header">Queries - SQL++</li> + <li class="nav-header">Queries</li> <li><a href="../sqlpp/manual.html" title="The SQL++ Query Language"><span class="none"></span>The SQL++ Query Language</a></li> <li><a href="../sqlpp/builtins.html" title="Builtin Functions"><span class="none"></span>Builtin Functions</a></li> - <li class="nav-header">Queries - AQL</li> - <li><a href="../aql/manual.html" title="The Asterix Query Language (AQL)"><span class="none"></span>The Asterix Query Language (AQL)</a></li> - <li><a href="../aql/builtins.html" title="Builtin Functions"><span class="none"></span>Builtin Functions</a></li> <li class="nav-header">API/SDK</li> <li><a href="../api.html" title="HTTP API"><span class="none"></span>HTTP API</a></li> <li><a href="../csv.html" title="CSV Output"><span class="none"></span>CSV Output</a></li> <li class="nav-header">Advanced Features</li> - <li><a href="../aql/fulltext.html" title="Support of Full-text Queries"><span class="none"></span>Support of Full-text Queries</a></li> <li><a href="../aql/externaldata.html" title="Accessing External Data"><span class="none"></span>Accessing External Data</a></li> - <li><a href="../feeds/tutorial.html" title="Support for Data Ingestion"><span class="none"></span>Support for Data Ingestion</a></li> + <li><a href="../feeds.html" title="Data Ingestion with Feeds"><span class="none"></span>Data Ingestion with Feeds</a></li> <li><a href="../udf.html" title="User Defined Functions"><span class="none"></span>User Defined Functions</a></li> - <li><a href="../aql/filters.html" title="Filter-Based LSM Index Acceleration"><span class="none"></span>Filter-Based LSM Index Acceleration</a></li> - <li><a href="../aql/similarity.html" title="Support of Similarity Queries"><span class="none"></span>Support of Similarity Queries</a></li> + <li><a href="../sqlpp/filters.html" title="Filter-Based LSM Index Acceleration"><span class="none"></span>Filter-Based LSM Index Acceleration</a></li> + <li><a href="../sqlpp/fulltext.html" title="Support of Full-text Queries"><span class="none"></span>Support of Full-text Queries</a></li> + <li><a href="../sqlpp/similarity.html" title="Support of Similarity Queries"><span class="none"></span>Support of Similarity Queries</a></li> + <li class="nav-header">Deprecated</li> + <li><a href="../aql/primer.html" title="AsterixDB Primer: Using AQL"><span class="none"></span>AsterixDB Primer: Using AQL</a></li> + <li><a href="../aql/manual.html" title="Queries: The Asterix Query Language (AQL)"><span class="none"></span>Queries: The Asterix Query Language (AQL)</a></li> + <li><a href="../aql/builtins.html" title="Queries: Builtin Functions (AQL)"><span class="none"></span>Queries: Builtin Functions (AQL)</a></li> </ul> <hr /> <div id="poweredBy"> http://git-wip-us.apache.org/repos/asf/asterixdb-site/blob/b2f1d3e6/content/docs/0.9.4/aql/manual.html ---------------------------------------------------------------------- diff --git a/content/docs/0.9.4/aql/manual.html b/content/docs/0.9.4/aql/manual.html index 7a45391..ce8e842 100644 --- a/content/docs/0.9.4/aql/manual.html +++ b/content/docs/0.9.4/aql/manual.html @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia Site Renderer 1.8.1 from src/site/markdown/aql/manual.md at 2018-10-02 + | Generated by Apache Maven Doxia Site Renderer 1.8.1 from src/site/markdown/aql/manual.md at 2018-10-11 | Rendered using Apache Maven Fluido Skin 1.7 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20181002" /> + <meta name="Date-Revision-yyyymmdd" content="20181011" /> <meta http-equiv="Content-Language" content="en" /> <title>AsterixDB – The Asterix Query Language, Version 1.0</title> <link rel="stylesheet" href="../css/apache-maven-fluido-1.7.min.css" /> @@ -26,8 +26,8 @@ <div id="breadcrumbs"> <ul class="breadcrumb"> - <li id="publishDate">Last Published: 2018-10-02</li> - <li id="projectVersion" class="pull-right">Version: 0.9.4</li> + <li id="publishDate">Last Published: 2018-10-11</li> + <li id="projectVersion" class="pull-right">Version: 0.9.5-SNAPSHOT</li> <li class="pull-right"><a href="../index.html" title="Documentation Home">Documentation Home</a></li> </ul> </div> @@ -40,26 +40,26 @@ <li><a href="../ansible.html" title="Option 2: using Ansible"><span class="none"></span>Option 2: using Ansible</a></li> <li><a href="../aws.html" title="Option 3: using Amazon Web Services"><span class="none"></span>Option 3: using Amazon Web Services</a></li> <li class="nav-header">AsterixDB Primer</li> - <li><a href="../sqlpp/primer-sqlpp.html" title="Option 1: using SQL++"><span class="none"></span>Option 1: using SQL++</a></li> - <li><a href="../aql/primer.html" title="Option 2: using AQL"><span class="none"></span>Option 2: using AQL</a></li> + <li><a href="../sqlpp/primer-sqlpp.html" title="Using SQL++"><span class="none"></span>Using SQL++</a></li> <li class="nav-header">Data Model</li> <li><a href="../datamodel.html" title="The Asterix Data Model"><span class="none"></span>The Asterix Data Model</a></li> - <li class="nav-header">Queries - SQL++</li> + <li class="nav-header">Queries</li> <li><a href="../sqlpp/manual.html" title="The SQL++ Query Language"><span class="none"></span>The SQL++ Query Language</a></li> <li><a href="../sqlpp/builtins.html" title="Builtin Functions"><span class="none"></span>Builtin Functions</a></li> - <li class="nav-header">Queries - AQL</li> - <li class="active"><a href="#"><span class="none"></span>The Asterix Query Language (AQL)</a></li> - <li><a href="../aql/builtins.html" title="Builtin Functions"><span class="none"></span>Builtin Functions</a></li> <li class="nav-header">API/SDK</li> <li><a href="../api.html" title="HTTP API"><span class="none"></span>HTTP API</a></li> <li><a href="../csv.html" title="CSV Output"><span class="none"></span>CSV Output</a></li> <li class="nav-header">Advanced Features</li> - <li><a href="../aql/fulltext.html" title="Support of Full-text Queries"><span class="none"></span>Support of Full-text Queries</a></li> <li><a href="../aql/externaldata.html" title="Accessing External Data"><span class="none"></span>Accessing External Data</a></li> - <li><a href="../feeds/tutorial.html" title="Support for Data Ingestion"><span class="none"></span>Support for Data Ingestion</a></li> + <li><a href="../feeds.html" title="Data Ingestion with Feeds"><span class="none"></span>Data Ingestion with Feeds</a></li> <li><a href="../udf.html" title="User Defined Functions"><span class="none"></span>User Defined Functions</a></li> - <li><a href="../aql/filters.html" title="Filter-Based LSM Index Acceleration"><span class="none"></span>Filter-Based LSM Index Acceleration</a></li> - <li><a href="../aql/similarity.html" title="Support of Similarity Queries"><span class="none"></span>Support of Similarity Queries</a></li> + <li><a href="../sqlpp/filters.html" title="Filter-Based LSM Index Acceleration"><span class="none"></span>Filter-Based LSM Index Acceleration</a></li> + <li><a href="../sqlpp/fulltext.html" title="Support of Full-text Queries"><span class="none"></span>Support of Full-text Queries</a></li> + <li><a href="../sqlpp/similarity.html" title="Support of Similarity Queries"><span class="none"></span>Support of Similarity Queries</a></li> + <li class="nav-header">Deprecated</li> + <li><a href="../aql/primer.html" title="AsterixDB Primer: Using AQL"><span class="none"></span>AsterixDB Primer: Using AQL</a></li> + <li class="active"><a href="#"><span class="none"></span>Queries: The Asterix Query Language (AQL)</a></li> + <li><a href="../aql/builtins.html" title="Queries: Builtin Functions (AQL)"><span class="none"></span>Queries: Builtin Functions (AQL)</a></li> </ul> <hr /> <div id="poweredBy"> http://git-wip-us.apache.org/repos/asf/asterixdb-site/blob/b2f1d3e6/content/docs/0.9.4/aql/primer.html ---------------------------------------------------------------------- diff --git a/content/docs/0.9.4/aql/primer.html b/content/docs/0.9.4/aql/primer.html index 46918a7..36d59dd 100644 --- a/content/docs/0.9.4/aql/primer.html +++ b/content/docs/0.9.4/aql/primer.html @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia Site Renderer 1.8.1 from src/site/markdown/aql/primer.md at 2018-10-02 + | Generated by Apache Maven Doxia Site Renderer 1.8.1 from src/site/markdown/aql/primer.md at 2018-10-11 | Rendered using Apache Maven Fluido Skin 1.7 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20181002" /> + <meta name="Date-Revision-yyyymmdd" content="20181011" /> <meta http-equiv="Content-Language" content="en" /> <title>AsterixDB – AsterixDB 101: An ADM and AQL Primer</title> <link rel="stylesheet" href="../css/apache-maven-fluido-1.7.min.css" /> @@ -26,8 +26,8 @@ <div id="breadcrumbs"> <ul class="breadcrumb"> - <li id="publishDate">Last Published: 2018-10-02</li> - <li id="projectVersion" class="pull-right">Version: 0.9.4</li> + <li id="publishDate">Last Published: 2018-10-11</li> + <li id="projectVersion" class="pull-right">Version: 0.9.5-SNAPSHOT</li> <li class="pull-right"><a href="../index.html" title="Documentation Home">Documentation Home</a></li> </ul> </div> @@ -40,26 +40,26 @@ <li><a href="../ansible.html" title="Option 2: using Ansible"><span class="none"></span>Option 2: using Ansible</a></li> <li><a href="../aws.html" title="Option 3: using Amazon Web Services"><span class="none"></span>Option 3: using Amazon Web Services</a></li> <li class="nav-header">AsterixDB Primer</li> - <li><a href="../sqlpp/primer-sqlpp.html" title="Option 1: using SQL++"><span class="none"></span>Option 1: using SQL++</a></li> - <li class="active"><a href="#"><span class="none"></span>Option 2: using AQL</a></li> + <li><a href="../sqlpp/primer-sqlpp.html" title="Using SQL++"><span class="none"></span>Using SQL++</a></li> <li class="nav-header">Data Model</li> <li><a href="../datamodel.html" title="The Asterix Data Model"><span class="none"></span>The Asterix Data Model</a></li> - <li class="nav-header">Queries - SQL++</li> + <li class="nav-header">Queries</li> <li><a href="../sqlpp/manual.html" title="The SQL++ Query Language"><span class="none"></span>The SQL++ Query Language</a></li> <li><a href="../sqlpp/builtins.html" title="Builtin Functions"><span class="none"></span>Builtin Functions</a></li> - <li class="nav-header">Queries - AQL</li> - <li><a href="../aql/manual.html" title="The Asterix Query Language (AQL)"><span class="none"></span>The Asterix Query Language (AQL)</a></li> - <li><a href="../aql/builtins.html" title="Builtin Functions"><span class="none"></span>Builtin Functions</a></li> <li class="nav-header">API/SDK</li> <li><a href="../api.html" title="HTTP API"><span class="none"></span>HTTP API</a></li> <li><a href="../csv.html" title="CSV Output"><span class="none"></span>CSV Output</a></li> <li class="nav-header">Advanced Features</li> - <li><a href="../aql/fulltext.html" title="Support of Full-text Queries"><span class="none"></span>Support of Full-text Queries</a></li> <li><a href="../aql/externaldata.html" title="Accessing External Data"><span class="none"></span>Accessing External Data</a></li> - <li><a href="../feeds/tutorial.html" title="Support for Data Ingestion"><span class="none"></span>Support for Data Ingestion</a></li> + <li><a href="../feeds.html" title="Data Ingestion with Feeds"><span class="none"></span>Data Ingestion with Feeds</a></li> <li><a href="../udf.html" title="User Defined Functions"><span class="none"></span>User Defined Functions</a></li> - <li><a href="../aql/filters.html" title="Filter-Based LSM Index Acceleration"><span class="none"></span>Filter-Based LSM Index Acceleration</a></li> - <li><a href="../aql/similarity.html" title="Support of Similarity Queries"><span class="none"></span>Support of Similarity Queries</a></li> + <li><a href="../sqlpp/filters.html" title="Filter-Based LSM Index Acceleration"><span class="none"></span>Filter-Based LSM Index Acceleration</a></li> + <li><a href="../sqlpp/fulltext.html" title="Support of Full-text Queries"><span class="none"></span>Support of Full-text Queries</a></li> + <li><a href="../sqlpp/similarity.html" title="Support of Similarity Queries"><span class="none"></span>Support of Similarity Queries</a></li> + <li class="nav-header">Deprecated</li> + <li class="active"><a href="#"><span class="none"></span>AsterixDB Primer: Using AQL</a></li> + <li><a href="../aql/manual.html" title="Queries: The Asterix Query Language (AQL)"><span class="none"></span>Queries: The Asterix Query Language (AQL)</a></li> + <li><a href="../aql/builtins.html" title="Queries: Builtin Functions (AQL)"><span class="none"></span>Queries: Builtin Functions (AQL)</a></li> </ul> <hr /> <div id="poweredBy"> http://git-wip-us.apache.org/repos/asf/asterixdb-site/blob/b2f1d3e6/content/docs/0.9.4/aql/similarity.html ---------------------------------------------------------------------- diff --git a/content/docs/0.9.4/aql/similarity.html b/content/docs/0.9.4/aql/similarity.html deleted file mode 100644 index ff71fcf..0000000 --- a/content/docs/0.9.4/aql/similarity.html +++ /dev/null @@ -1,310 +0,0 @@ -<!DOCTYPE html> -<!-- - | Generated by Apache Maven Doxia Site Renderer 1.8.1 from src/site/markdown/aql/similarity.md at 2018-10-02 - | Rendered using Apache Maven Fluido Skin 1.7 ---> -<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> - <head> - <meta charset="UTF-8" /> - <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20181002" /> - <meta http-equiv="Content-Language" content="en" /> - <title>AsterixDB – AsterixDB Support of Similarity Queries</title> - <link rel="stylesheet" href="../css/apache-maven-fluido-1.7.min.css" /> - <link rel="stylesheet" href="../css/site.css" /> - <link rel="stylesheet" href="../css/print.css" media="print" /> - <script type="text/javascript" src="../js/apache-maven-fluido-1.7.min.js"></script> - - </head> - <body class="topBarDisabled"> - <div class="container-fluid"> - <div id="banner"> - <div class="pull-left"><a href=".././" id="bannerLeft"><img src="../images/asterixlogo.png" alt="AsterixDB"/></a></div> - <div class="pull-right"></div> - <div class="clear"><hr/></div> - </div> - - <div id="breadcrumbs"> - <ul class="breadcrumb"> - <li id="publishDate">Last Published: 2018-10-02</li> - <li id="projectVersion" class="pull-right">Version: 0.9.4</li> - <li class="pull-right"><a href="../index.html" title="Documentation Home">Documentation Home</a></li> - </ul> - </div> - <div class="row-fluid"> - <div id="leftColumn" class="span2"> - <div class="well sidebar-nav"> - <ul class="nav nav-list"> - <li class="nav-header">Get Started - Installation</li> - <li><a href="../ncservice.html" title="Option 1: using NCService"><span class="none"></span>Option 1: using NCService</a></li> - <li><a href="../ansible.html" title="Option 2: using Ansible"><span class="none"></span>Option 2: using Ansible</a></li> - <li><a href="../aws.html" title="Option 3: using Amazon Web Services"><span class="none"></span>Option 3: using Amazon Web Services</a></li> - <li class="nav-header">AsterixDB Primer</li> - <li><a href="../sqlpp/primer-sqlpp.html" title="Option 1: using SQL++"><span class="none"></span>Option 1: using SQL++</a></li> - <li><a href="../aql/primer.html" title="Option 2: using AQL"><span class="none"></span>Option 2: using AQL</a></li> - <li class="nav-header">Data Model</li> - <li><a href="../datamodel.html" title="The Asterix Data Model"><span class="none"></span>The Asterix Data Model</a></li> - <li class="nav-header">Queries - SQL++</li> - <li><a href="../sqlpp/manual.html" title="The SQL++ Query Language"><span class="none"></span>The SQL++ Query Language</a></li> - <li><a href="../sqlpp/builtins.html" title="Builtin Functions"><span class="none"></span>Builtin Functions</a></li> - <li class="nav-header">Queries - AQL</li> - <li><a href="../aql/manual.html" title="The Asterix Query Language (AQL)"><span class="none"></span>The Asterix Query Language (AQL)</a></li> - <li><a href="../aql/builtins.html" title="Builtin Functions"><span class="none"></span>Builtin Functions</a></li> - <li class="nav-header">API/SDK</li> - <li><a href="../api.html" title="HTTP API"><span class="none"></span>HTTP API</a></li> - <li><a href="../csv.html" title="CSV Output"><span class="none"></span>CSV Output</a></li> - <li class="nav-header">Advanced Features</li> - <li><a href="../aql/fulltext.html" title="Support of Full-text Queries"><span class="none"></span>Support of Full-text Queries</a></li> - <li><a href="../aql/externaldata.html" title="Accessing External Data"><span class="none"></span>Accessing External Data</a></li> - <li><a href="../feeds/tutorial.html" title="Support for Data Ingestion"><span class="none"></span>Support for Data Ingestion</a></li> - <li><a href="../udf.html" title="User Defined Functions"><span class="none"></span>User Defined Functions</a></li> - <li><a href="../aql/filters.html" title="Filter-Based LSM Index Acceleration"><span class="none"></span>Filter-Based LSM Index Acceleration</a></li> - <li class="active"><a href="#"><span class="none"></span>Support of Similarity Queries</a></li> -</ul> - <hr /> - <div id="poweredBy"> - <div class="clear"></div> - <div class="clear"></div> - <div class="clear"></div> - <div class="clear"></div> -<a href=".././" title="AsterixDB" class="builtBy"><img class="builtBy" alt="AsterixDB" src="../images/asterixlogo.png" /></a> - </div> - </div> - </div> - <div id="bodyColumn" class="span10" > -<!-- - ! Licensed to the Apache Software Foundation (ASF) under one - ! or more contributor license agreements. See the NOTICE file - ! distributed with this work for additional information - ! regarding copyright ownership. The ASF licenses this file - ! to you under the Apache License, Version 2.0 (the - ! "License"); you may not use this file except in compliance - ! with the License. You may obtain a copy of the License at - ! - ! http://www.apache.org/licenses/LICENSE-2.0 - ! - ! Unless required by applicable law or agreed to in writing, - ! software distributed under the License is distributed on an - ! "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - ! KIND, either express or implied. See the License for the - ! specific language governing permissions and limitations - ! under the License. - !--> -<h1>AsterixDB Support of Similarity Queries</h1> -<div class="section"> -<h2><a name="Table_of_Contents"></a><a name="toc" id="toc">Table of Contents</a></h2> -<ul> - -<li><a href="#Motivation">Motivation</a></li> -<li><a href="#DataTypesAndSimilarityFunctions">Data Types and Similarity Functions</a></li> -<li><a href="#SimilaritySelectionQueries">Similarity Selection Queries</a></li> -<li><a href="#SimilarityJoinQueries">Similarity Join Queries</a></li> -<li><a href="#UsingIndexesToSupportSimilarityQueries">Using Indexes to Support Similarity Queries</a></li> -</ul></div> -<div class="section"> -<h2><a name="Motivation_.5BBack_to_TOC.5D"></a><a name="Motivation" id="Motivation">Motivation</a> <font size="4"><a href="#toc">[Back to TOC]</a></font></h2> -<p>Similarity queries are widely used in applications where users need to find objects that satisfy a similarity predicate, while exact matching is not sufficient. These queries are especially important for social and Web applications, where errors, abbreviations, and inconsistencies are common. As an example, we may want to find all the movies starring Schwarzenegger, while we don’t know the exact spelling of his last name (despite his popularity in both the movie industry and politics :-)). As another example, we want to find all the Facebook users who have similar friends. To meet this type of needs, AsterixDB supports similarity queries using efficient indexes and algorithms.</p></div> -<div class="section"> -<h2><a name="Data_Types_and_Similarity_Functions_.5BBack_to_TOC.5D"></a><a name="DataTypesAndSimilarityFunctions" id="DataTypesAndSimilarityFunctions">Data Types and Similarity Functions</a> <font size="4"><a href="#toc">[Back to TOC]</a></font></h2> -<p>AsterixDB supports <a class="externalLink" href="http://en.wikipedia.org/wiki/Levenshtein_distance";>edit distance</a> (on strings) and <a class="externalLink" href="http://en.wikipedia.org/wiki/Jaccard_index";>Jaccard</a> (on sets). For instance, in our <a href="primer.html#ADM:_Modeling_Semistructed_Data_in_AsterixDB">TinySocial</a> example, the <tt>friend-ids</tt> of a Facebook user forms a set of friends, and we can define a similarity between the sets of friends of two users. We can also convert a string to a set of grams of a length “n” (called “n-grams”) and define the Jaccard similarity between the two gram sets of the two strings. Formally, the “n-grams” of a string are its substrings of length “n”. For instance, the 3-grams of the string <tt>schwarzenegger</tt> are <tt>sch</tt>, <tt>chw</tt>, <tt>hwa</tt>, …, <tt>ger</tt>.</p> -<p>AsterixDB provides <a href="functions.html#Tokenizing_Functions">tokenization functions</a> to convert strings to sets, and the <a href="functions.html#Similarity_Functions">similarity functions</a>.</p></div> -<div class="section"> -<h2><a name="Similarity_Selection_Queries_.5BBack_to_TOC.5D"></a><a name="SimilaritySelectionQueries" id="SimilaritySelectionQueries">Similarity Selection Queries</a> <font size="4"><a href="#toc">[Back to TOC]</a></font></h2> -<p>The following query asks for all the Facebook users whose name is similar to <tt>Suzanna Tilson</tt>, i.e., their edit distance is at most 2.</p> - -<div> -<div> -<pre class="source"> use dataverse TinySocial; - - for $user in dataset('FacebookUsers') - let $ed := edit-distance($user.name, "Suzanna Tilson") - where $ed <= 2 - return $user -</pre></div></div> - -<p>The following query asks for all the Facebook users whose set of friend ids is similar to <tt>[1,5,9,10]</tt>, i.e., their Jaccard similarity is at least 0.6.</p> - -<div> -<div> -<pre class="source"> use dataverse TinySocial; - - for $user in dataset('FacebookUsers') - let $sim := similarity-jaccard($user.friend-ids, [1,5,9,10]) - where $sim >= 0.6f - return $user -</pre></div></div> - -<p>AsterixDB allows a user to use a similarity operator <tt>~=</tt> to express a condition by defining the similarity function and threshold using “set” statements earlier. For instance, the above query can be equivalently written as:</p> - -<div> -<div> -<pre class="source"> use dataverse TinySocial; - - set simfunction "jaccard"; - set simthreshold "0.6f"; - - for $user in dataset('FacebookUsers') - where $user.friend-ids ~= [1,5,9,10] - return $user -</pre></div></div> - -<p>In this query, we first declare Jaccard as the similarity function using <tt>simfunction</tt> and then specify the threshold <tt>0.6f</tt> using <tt>simthreshold</tt>.</p></div> -<div class="section"> -<h2><a name="Similarity_Join_Queries_.5BBack_to_TOC.5D"></a><a name="SimilarityJoinQueries" id="SimilarityJoinQueries">Similarity Join Queries</a> <font size="4"><a href="#toc">[Back to TOC]</a></font></h2> -<p>AsterixDB supports fuzzy joins between two sets. The following <a href="primer.html#Query_5_-_Fuzzy_Join">query</a> finds, for each Facebook user, all Twitter users with names similar to their name based on the edit distance.</p> - -<div> -<div> -<pre class="source"> use dataverse TinySocial; - - set simfunction "edit-distance"; - set simthreshold "3"; - - for $fbu in dataset FacebookUsers - return { - "id": $fbu.id, - "name": $fbu.name, - "similar-users": for $t in dataset TweetMessages - let $tu := $t.user - where $tu.name ~= $fbu.name - return { - "twitter-screenname": $tu.screen-name, - "twitter-name": $tu.name - } - }; -</pre></div></div> -</div> -<div class="section"> -<h2><a name="Using_Indexes_to_Support_Similarity_Queries_.5BBack_to_TOC.5D"></a><a name="UsingIndexesToSupportSimilarityQueries" id="UsingIndexesToSupportSimilarityQueries">Using Indexes to Support Similarity Queries</a> <font size="4"><a href="#toc">[Back to TOC]</a></font></h2> -<p>AsterixDB uses two types of indexes to support similarity queries, namely “ngram index” and “keyword index”.</p> -<div class="section"> -<h3><a name="NGram_Index"></a>NGram Index</h3> -<p>An “ngram index” is constructed on a set of strings. We generate n-grams for each string, and build an inverted list for each n-gram that includes the ids of the strings with this gram. A similarity query can be answered efficiently by accessing the inverted lists of the grams in the query and counting the number of occurrences of the string ids on these inverted lists. The similar idea can be used to answer queries with Jaccard similarity. A detailed description of these techniques is available at this <a class="externalLink" href="http://www.ics.uci.edu/~chenli/pub/icde2009-memreducer.pdf";>paper</a>.</p> -<p>For instance, the following DDL statements create an ngram index on the <tt>FacebookUsers.name</tt> attribute using an inverted index of 3-grams.</p> - -<div> -<div> -<pre class="source"> use dataverse TinySocial; - - create index fbUserIdx on FacebookUsers(name) type ngram(3); -</pre></div></div> - -<p>The number “3” in “ngram(3)” is the length “n” in the grams. This index can be used to optimize similarity queries on this attribute using <a href="functions.html#edit-distance">edit-distance</a>, <a href="functions.html#edit-distance-check">edit-distance-check</a>, <a href="functions.html#similarity-jaccard">similarity-jaccard</a>, or <a href="functions.html#similarity-jaccard-check">similarity-jaccard-check</a> queries on this attribute where the similarity is defined on sets of 3-grams. This index can also be used to optimize queries with the “<a href="(functions.html#contains">contains()</a>” predicate (i.e., substring matching) since it can be also be solved by counting on the inverted lists of the grams in the query string.</p> -<div class="section"> -<h4><a name="NGram_Index_usage_case_-_edit-distance"></a>NGram Index usage case - <a href="functions.html#edit-distance">edit-distance</a></h4> - -<div> -<div> -<pre class="source"> use dataverse TinySocial; - - for $user in dataset('FacebookUsers') - let $ed := edit-distance($user.name, "Suzanna Tilson") - where $ed <= 2 - return $user -</pre></div></div> -</div> -<div class="section"> -<h4><a name="NGram_Index_usage_case_-_edit-distance-check"></a>NGram Index usage case - <a href="functions.html#edit-distance-check">edit-distance-check</a></h4> - -<div> -<div> -<pre class="source"> use dataverse TinySocial; - - for $user in dataset('FacebookUsers') - let $ed := edit-distance-check($user.name, "Suzanna Tilson", 2) - where $ed[0] - return $ed[1] -</pre></div></div> -</div> -<div class="section"> -<h4><a name="NGram_Index_usage_case_-_similarity-jaccard"></a>NGram Index usage case - <a href="functions.html#similarity-jaccard">similarity-jaccard</a></h4> - -<div> -<div> -<pre class="source"> use dataverse TinySocial; - - for $user in dataset('FacebookUsers') - let $sim := similarity-jaccard($user.friend-ids, [1,5,9,10]) - where $sim >= 0.6f - return $user -</pre></div></div> -</div> -<div class="section"> -<h4><a name="NGram_Index_usage_case_-_similarity-jaccard-check"></a>NGram Index usage case - <a href="functions.html#similarity-jaccard-check">similarity-jaccard-check</a></h4> - -<div> -<div> -<pre class="source"> use dataverse TinySocial; - - for $user in dataset('FacebookUsers') - let $sim := similarity-jaccard-check($user.friend-ids, [1,5,9,10], 0.6f) - where $sim[0] - return $user -</pre></div></div> -</div> -<div class="section"> -<h4><a name="NGram_Index_usage_case_-_contains.28.29"></a>NGram Index usage case - <a href="(functions.html#contains">contains()</a></h4> - -<div> -<div> -<pre class="source"> use dataverse TinySocial; - - for $i in dataset('FacebookMessages') - where contains($i.message, "phone") - return {"mid": $i.message-id, "message": $i.message} -</pre></div></div> -</div></div> -<div class="section"> -<h3><a name="Keyword_Index"></a>Keyword Index</h3> -<p>A “keyword index” is constructed on a set of strings or sets (e.g., OrderedList, UnorderedList). Instead of generating grams as in an ngram index, we generate tokens (e.g., words) and for each token, construct an inverted list that includes the ids of the objects with this token. The following two examples show how to create keyword index on two different types:</p> -<div class="section"> -<h4><a name="Keyword_Index_on_String_Type"></a>Keyword Index on String Type</h4> - -<div> -<div> -<pre class="source"> use dataverse TinySocial; - - drop index FacebookMessages.fbMessageIdx if exists; - create index fbMessageIdx on FacebookMessages(message) type keyword; - - for $o in dataset('FacebookMessages') - let $jacc := similarity-jaccard-check(word-tokens($o.message), word-tokens("love like ccast"), 0.2f) - where $jacc[0] - return $o -</pre></div></div> -</div> -<div class="section"> -<h4><a name="Keyword_Index_on_UnorderedList_Type"></a>Keyword Index on UnorderedList Type</h4> - -<div> -<div> -<pre class="source"> use dataverse TinySocial; - - create index fbUserIdx_fids on FacebookUsers(friend-ids) type keyword; - - for $c in dataset('FacebookUsers') - let $jacc := similarity-jaccard-check($c.friend-ids, {{3,10}}, 0.5f) - where $jacc[0] - return $c -</pre></div></div> - -<p>As shown above, keyword index can be used to optimize queries with token-based similarity predicates, including <a href="functions.html#similarity-jaccard">similarity-jaccard</a> and <a href="functions.html#similarity-jaccard-check">similarity-jaccard-check</a>.</p></div></div></div> - </div> - </div> - </div> - <hr/> - <footer> - <div class="container-fluid"> - <div class="row-fluid"> -<div class="row-fluid">Apache AsterixDB, AsterixDB, Apache, the Apache - feather logo, and the Apache AsterixDB project logo are either - registered trademarks or trademarks of The Apache Software - Foundation in the United States and other countries. - All other marks mentioned may be trademarks or registered - trademarks of their respective owners. - </div> - </div> - </div> - </footer> - </body> -</html> http://git-wip-us.apache.org/repos/asf/asterixdb-site/blob/b2f1d3e6/content/docs/0.9.4/aws.html ---------------------------------------------------------------------- diff --git a/content/docs/0.9.4/aws.html b/content/docs/0.9.4/aws.html new file mode 100644 index 0000000..977e782 --- /dev/null +++ b/content/docs/0.9.4/aws.html @@ -0,0 +1,384 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia Site Renderer 1.8.1 from target/generated-site/markdown/aws.md at 2018-10-12 + | Rendered using Apache Maven Fluido Skin 1.7 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20181012" /> + <meta http-equiv="Content-Language" content="en" /> + <title>AsterixDB – Installation using Amazon Web Services</title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.7.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + <script type="text/javascript" src="./js/apache-maven-fluido-1.7.min.js"></script> + + </head> + <body class="topBarDisabled"> + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"><a href="./" id="bannerLeft"><img src="images/asterixlogo.png" alt="AsterixDB"/></a></div> + <div class="pull-right"></div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + <li id="publishDate">Last Published: 2018-10-12</li> + <li id="projectVersion" class="pull-right">Version: 0.9.4</li> + <li class="pull-right"><a href="index.html" title="Documentation Home">Documentation Home</a></li> + </ul> + </div> + <div class="row-fluid"> + <div id="leftColumn" class="span2"> + <div class="well sidebar-nav"> + <ul class="nav nav-list"> + <li class="nav-header">Get Started - Installation</li> + <li><a href="ncservice.html" title="Option 1: using NCService"><span class="none"></span>Option 1: using NCService</a></li> + <li><a href="ansible.html" title="Option 2: using Ansible"><span class="none"></span>Option 2: using Ansible</a></li> + <li class="active"><a href="#"><span class="none"></span>Option 3: using Amazon Web Services</a></li> + <li class="nav-header">AsterixDB Primer</li> + <li><a href="sqlpp/primer-sqlpp.html" title="Using SQL++"><span class="none"></span>Using SQL++</a></li> + <li class="nav-header">Data Model</li> + <li><a href="datamodel.html" title="The Asterix Data Model"><span class="none"></span>The Asterix Data Model</a></li> + <li class="nav-header">Queries</li> + <li><a href="sqlpp/manual.html" title="The SQL++ Query Language"><span class="none"></span>The SQL++ Query Language</a></li> + <li><a href="sqlpp/builtins.html" title="Builtin Functions"><span class="none"></span>Builtin Functions</a></li> + <li class="nav-header">API/SDK</li> + <li><a href="api.html" title="HTTP API"><span class="none"></span>HTTP API</a></li> + <li><a href="csv.html" title="CSV Output"><span class="none"></span>CSV Output</a></li> + <li class="nav-header">Advanced Features</li> + <li><a href="aql/externaldata.html" title="Accessing External Data"><span class="none"></span>Accessing External Data</a></li> + <li><a href="feeds.html" title="Data Ingestion with Feeds"><span class="none"></span>Data Ingestion with Feeds</a></li> + <li><a href="udf.html" title="User Defined Functions"><span class="none"></span>User Defined Functions</a></li> + <li><a href="sqlpp/filters.html" title="Filter-Based LSM Index Acceleration"><span class="none"></span>Filter-Based LSM Index Acceleration</a></li> + <li><a href="sqlpp/fulltext.html" title="Support of Full-text Queries"><span class="none"></span>Support of Full-text Queries</a></li> + <li><a href="sqlpp/similarity.html" title="Support of Similarity Queries"><span class="none"></span>Support of Similarity Queries</a></li> + <li class="nav-header">Deprecated</li> + <li><a href="aql/primer.html" title="AsterixDB Primer: Using AQL"><span class="none"></span>AsterixDB Primer: Using AQL</a></li> + <li><a href="aql/manual.html" title="Queries: The Asterix Query Language (AQL)"><span class="none"></span>Queries: The Asterix Query Language (AQL)</a></li> + <li><a href="aql/builtins.html" title="Queries: Builtin Functions (AQL)"><span class="none"></span>Queries: Builtin Functions (AQL)</a></li> +</ul> + <hr /> + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> +<a href="./" title="AsterixDB" class="builtBy"><img class="builtBy" alt="AsterixDB" src="images/asterixlogo.png" /></a> + </div> + </div> + </div> + <div id="bodyColumn" class="span10" > +<!-- + ! Licensed to the Apache Software Foundation (ASF) under one + ! or more contributor license agreements. See the NOTICE file + ! distributed with this work for additional information + ! regarding copyright ownership. The ASF licenses this file + ! to you under the Apache License, Version 2.0 (the + ! "License"); you may not use this file except in compliance + ! with the License. You may obtain a copy of the License at + ! + ! http://www.apache.org/licenses/LICENSE-2.0 + ! + ! Unless required by applicable law or agreed to in writing, + ! software distributed under the License is distributed on an + ! "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + ! KIND, either express or implied. See the License for the + ! specific language governing permissions and limitations + ! under the License. + !--> +<h1>Installation using Amazon Web Services</h1> +<div class="section"> +<h2><a name="Table_of_Contents"></a><a name="atoc" id="#toc">Table of Contents</a></h2> +<ul> + +<li><a href="#Introduction">Introduction</a></li> +<li><a href="#Prerequisites">Prerequisites</a></li> +<li><a href="#config">Cluster Configuration</a></li> +<li><a href="#lifecycle">Cluster Lifecycle Management</a></li> +</ul><!-- + ! Licensed to the Apache Software Foundation (ASF) under one + ! or more contributor license agreements. See the NOTICE file + ! distributed with this work for additional information + ! regarding copyright ownership. The ASF licenses this file + ! to you under the Apache License, Version 2.0 (the + ! "License"); you may not use this file except in compliance + ! with the License. You may obtain a copy of the License at + ! + ! http://www.apache.org/licenses/LICENSE-2.0 + ! + ! Unless required by applicable law or agreed to in writing, + ! software distributed under the License is distributed on an + ! "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + ! KIND, either express or implied. See the License for the + ! specific language governing permissions and limitations + ! under the License. + !--> +</div> +<div class="section"> +<h2><a name="Introduction" id="Introduction">Introduction</a></h2> +<p>Note that you can always manually launch a number of Amazon Web Services EC2 instances and then run the Ansible cluster installation scripts as described <a href="ansible.html">here</a> separately to manage the lifecycle of an AsterixDB cluster on those EC2 instances.</p> +<p>However, via this installation option, we provide a combo solution for automating both AWS EC2 and AsterixDB, where you can run only one script to deploy, start, stop, and terminate an AsterixDB cluster on AWS.</p></div> +<div class="section"> +<h2><a name="Prerequisites" id="Prerequisites">Prerequisites</a></h2> +<ul> + +<li> + +<p>Supported operating systems for the client: <b>Linux</b> and <b>MacOS</b></p> +</li> +<li> + +<p>Supported operating systems for Amazon Web Services instances: <b>Linux</b></p> +</li> +<li> + +<p>Install pip on your client machine:</p> +<p>CentOS</p> + +<div> +<div> +<pre class="source"> $ sudo yum install python-pip +</pre></div></div> + +<p>Ubuntu</p> + +<div> +<div> +<pre class="source"> $ sudo apt-get install python-pip +</pre></div></div> + +<p>macOS</p> + +<div> +<div> +<pre class="source"> $ brew install pip +</pre></div></div> +</li> +<li> + +<p>Install Ansible, boto, and boto3 on your client machine:</p> + +<div> +<div> +<pre class="source"> $ pip install ansible + $ pip install boto + $ pip install boto3 +</pre></div></div> + +<p>Note that you might need <tt>sudo</tt> depending on your system configuration.</p> +<p><b>Make sure that the version of Ansible is no less than 2.2.1.0</b>:</p> + +<div> +<div> +<pre class="source"> $ ansible --version + ansible 2.2.1.0 +</pre></div></div> + +<p><b>For users with macOS 10.11+</b>, please create a user-level Ansible configuration file at:</p> + +<div> +<div> +<pre class="source"> ~/.ansible.cfg +</pre></div></div> + +<p>and add the following configuration:</p> + +<div> +<div> +<pre class="source"> [ssh_connection] + control_path = %(directory)s/%%C +</pre></div></div> +</li> +<li> + +<p>Download the AsterixDB distribution package, unzip it, navigate to <tt>opt/aws/</tt></p> + +<div> +<div> +<pre class="source"> $ cd opt/aws +</pre></div></div> + +<p>The following files and directories are in the directory <tt>opt/aws</tt>:</p> + +<div> +<div> +<pre class="source"> README bin conf yaml +</pre></div></div> + +<p><tt>bin</tt> contains scripts that start and terminate an AWS-based cluster instance, according to the configuration specified in files under <tt>conf</tt>, and <tt>yaml</tt> contains internal Ansible scripts that the shell scripts in <tt>bin</tt> use.</p> +</li> +<li> + +<p>Create an AWS account and an IAM user.</p> +<p>Set up a security group that you’d like to use for your AWS cluster. <b>The security group should at least allow all TCP connections from anywhere.</b> Provide the name of the security group as the value for the <tt>group</tt> field in <tt>conf/aws_settings.yml</tt>.</p> +</li> +<li> + +<p>Retrieve your AWS EC2 key pair name and use that as the <tt>keypair</tt> in <tt>conf/aws_settings.yml</tt>;</p> +<p>retrieve your AWS IAM <tt>access key ID</tt> and use that as the <tt>access_key_id</tt> in <tt>conf/aws_settings.yml</tt>;</p> +<p>retrieve your AWS IAM <tt>secret access key</tt> and use that as the <tt>secret_access_key</tt> in <tt>conf/aws_settings.yml</tt>.</p> +<p>Note that you can only read or download <tt>access key ID</tt> and <tt>secret access key</tt> once from your AWS console. If you forget them, you have to create new keys and delete the old ones.</p> +</li> +<li> + +<p>Configure your ssh setting by editing <tt>~/.ssh/config</tt> and adding the following entry:</p> + +<div> +<div> +<pre class="source"> Host *.amazonaws.com + IdentityFile <path_of_private_key> +</pre></div></div> + +<p>Note that <path_of_private_key> should be replaced by the path to the file that stores the private key for the key pair that you uploaded to AWS and used in <tt>conf/aws_settings</tt>. For example:</p> + +<div> +<div> +<pre class="source"> Host *.amazonaws.com + IdentityFile ~/.ssh/id_rsa +</pre></div></div> +</li> +</ul></div> +<div class="section"> +<h2><a name="Cluster_Configuration"></a><a name="config" id="config">Cluster Configuration</a></h2> +<ul> + +<li> + +<p><b>AWS settings</b>. Edit <tt>conf/instance_settings.yml</tt>. The meaning of each parameter is listed as follows:</p> + +<div> +<div> +<pre class="source"> # The OS image id for ec2 instances. + image: ami-76fa4116 + + # The data center region for ec2 instances. + region: us-west-2 + + # The tag for each ec2 machine. Use different tags for isolation. + tag: scale_test + + # The name of a security group that appears in your AWS console. + group: default + + # The name of a key pair that appears in your AWS console. + keypair: <to be filled> + + # The AWS access key id for your IAM user. + access_key_id: <to be filled> + + # The AWS secret key for your IAM user. + secret_access_key: <to be filled> + + # The AWS instance type. A full list of available types are listed at: + # https://aws.amazon.com/ec2/instance-types/ + instance_type: t2.micro + + # The number of ec2 instances that construct a cluster. + count: 3 + + # The user name. + user: ec2-user + + # Whether to reuse one slave machine to host the master process. + cc_on_nc: false +</pre></div></div> + +<p><b>As described in <a href="#Prerequisites">prerequisites</a>, the following parameters must be customized:</b></p> + +<div> +<div> +<pre class="source"> # The tag for each ec2 machine. Use different tags for isolation. + tag: scale_test + + # The name of a security group that appears in your AWS console. + group: default + + # The name of a key pair that appears in your AWS console. + keypair: <to be filled> + + # The AWS access key id for your IAM user. + access_key_id: <to be filled> + + # The AWS secrety key for your IAM user. + secret_access_key: <to be filled> +</pre></div></div> +</li> +<li> + +<p><b>Remote working directories</b>. Edit <tt>conf/instance_settings.yml</tt> to change the remote binary directory (the variable “binarydir”) when necessary. By default, the binary directory will be under the home directory (as the value of Ansible builtin variable ansible_env.HOME) of the ssh user account on each node.</p> +</li> +</ul></div> +<div class="section"> +<h2><a name="Cluster_Lifecycle_Management"></a><a name="lifecycle" id="lifecycle">Cluster Lifecycle Management</a></h2> +<ul> + +<li> + +<p>Allocate AWS EC2 nodes (the number of nodes is specified in <tt>conf/instance_settings.yml</tt>) and deploy the binary to all allocated EC2 nodes:</p> + +<div> +<div> +<pre class="source"> bin/deploy.sh +</pre></div></div> +</li> +<li> + +<p>Before starting the AsterixDB cluster, you the instance configuration file <tt>conf/instance/cc.conf</tt> can be modified with the exception of the IP addresses/DNS names which are are generated and cannot be changed. All available parameters and their usage can be found <a href="ncservice.html#Parameters">here</a>.</p> +</li> +<li> + +<p>Launch your AsterixDB cluster on EC2:</p> + +<div> +<div> +<pre class="source"> bin/start.sh +</pre></div></div> + +<p>Now you can use the multi-node AsterixDB cluster on EC2 by by opening the master node listed in <tt>conf/instance/inventory</tt> at port <tt>19001</tt> (which can be customized in <tt>conf/instance/cc.conf</tt>) in your browser.</p> +</li> +<li> + +<p>If you want to stop the AWS-based AsterixDB cluster, run the following script:</p> + +<div> +<div> +<pre class="source"> bin/stop.sh +</pre></div></div> + +<p>Note that this only stops AsterixDB but does not stop the EC2 nodes.</p> +</li> +<li> + +<p>If you want to terminate the EC2 nodes that run the AsterixDB cluster, run the following script:</p> + +<div> +<div> +<pre class="source"> bin/terminate.sh +</pre></div></div> + +<p><b>Note that it will destroy everything in the AsterixDB cluster you installed and terminate all EC2 nodes for the cluster.</b></p> +</li> +</ul></div> + </div> + </div> + </div> + <hr/> + <footer> + <div class="container-fluid"> + <div class="row-fluid"> +<div class="row-fluid">Apache AsterixDB, AsterixDB, Apache, the Apache + feather logo, and the Apache AsterixDB project logo are either + registered trademarks or trademarks of The Apache Software + Foundation in the United States and other countries. + All other marks mentioned may be trademarks or registered + trademarks of their respective owners. + </div> + </div> + </div> + </footer> + </body> +</html>
