http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/ae77df2e/content/ddl-of-carbondata.html ---------------------------------------------------------------------- diff --git a/content/ddl-of-carbondata.html b/content/ddl-of-carbondata.html index 434f378..7f84786 100644 --- a/content/ddl-of-carbondata.html +++ b/content/ddl-of-carbondata.html @@ -52,6 +52,9 @@ aria-expanded="false"> Download <span class="caret"></span></a> <ul class="dropdown-menu"> <li> + <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.1/"; + target="_blank">Apache CarbonData 1.5.1</a></li> + <li> <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"; target="_blank">Apache CarbonData 1.5.0</a></li> <li> @@ -64,9 +67,6 @@ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"; target="_blank">Apache CarbonData 1.3.1</a></li> <li> - <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"; - target="_blank">Apache CarbonData 1.3.0</a></li> - <li> <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"; target="_blank">Release Archive</a></li> </ul> @@ -223,7 +223,7 @@ <p>CarbonData DDL statements are documented here,which includes:</p> <ul> <li> -<a href="#create-table">CREATE TABLE</a> +<p><a href="#create-table">CREATE TABLE</a></p> <ul> <li><a href="#dictionary-encoding-configuration">Dictionary Encoding</a></li> <li><a href="#inverted-index-configuration">Inverted Index</a></li> @@ -239,19 +239,24 @@ <li><a href="#string-longer-than-32000-characters">Extra Long String columns</a></li> <li><a href="#compression-for-table">Compression for Table</a></li> <li><a href="#bad-records-path">Bad Records Path</a></li> +<li><a href="#load-minimum-data-size">Load Minimum Input File Size</a></li> </ul> </li> -<li><a href="#create-table-as-select">CREATE TABLE AS SELECT</a></li> <li> -<a href="#create-external-table">CREATE EXTERNAL TABLE</a> +<p><a href="#create-table-as-select">CREATE TABLE AS SELECT</a></p> +</li> +<li> +<p><a href="#create-external-table">CREATE EXTERNAL TABLE</a></p> <ul> <li><a href="#create-external-table-on-managed-table-data-location">External Table on Transactional table location</a></li> <li><a href="#create-external-table-on-non-transactional-table-data-location">External Table on non-transactional table location</a></li> </ul> </li> -<li><a href="#create-database">CREATE DATABASE</a></li> <li> -<a href="#table-management">TABLE MANAGEMENT</a> +<p><a href="#create-database">CREATE DATABASE</a></p> +</li> +<li> +<p><a href="#table-management">TABLE MANAGEMENT</a></p> <ul> <li><a href="#show-table">SHOW TABLE</a></li> <li> @@ -271,7 +276,7 @@ </ul> </li> <li> -<a href="#partition">PARTITION</a> +<p><a href="#partition">PARTITION</a></p> <ul> <li> <a href="#standard-partition">STANDARD PARTITION(HIVE)</a> @@ -293,7 +298,9 @@ <li><a href="#drop-a-partition">DROP PARTITION</a></li> </ul> </li> -<li><a href="#bucketing">BUCKETING</a></li> +<li> +<p><a href="#bucketing">BUCKETING</a></p> +</li> </ul> <h2> <a id="create-table" class="anchor" href="#create-table" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>CREATE TABLE</h2> @@ -324,6 +331,10 @@ STORED AS carbondata <td>Columns to exclude from inverted index generation</td> </tr> <tr> +<td><a href="#inverted-index-configuration">INVERTED_INDEX</a></td> +<td>Columns to include for inverted index generation</td> +</tr> +<tr> <td><a href="#sort-columns-configuration">SORT_COLUMNS</a></td> <td>Columns to include in sort and its order of sort</td> </tr> @@ -403,6 +414,10 @@ STORED AS carbondata <td><a href="#bucketing">BUCKETCOLUMNS</a></td> <td>Columns which are to be placed in buckets</td> </tr> +<tr> +<td><a href="#load-minimum-data-size">LOAD_MIN_SIZE_INMB</a></td> +<td>Minimum input data size per node for data loading</td> +</tr> </tbody> </table> <p>Following are the guidelines for TBLPROPERTIES, CarbonData's additional table options can be set via carbon.properties.</p> @@ -419,9 +434,9 @@ Suggested use cases : do dictionary encoding for low cardinality columns, it mig <li> <h5> <a id="inverted-index-configuration" class="anchor" href="#inverted-index-configuration" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Inverted Index Configuration</h5> -<p>By default inverted index is enabled, it might help to improve compression ratio and query speed, especially for low cardinality columns which are in reward position. +<p>By default inverted index is disabled as store size will be reduced, it can be enabled by using a table property. It might help to improve compression ratio and query speed, especially for low cardinality columns which are in reward position. Suggested use cases : For high cardinality columns, you can disable the inverted index for improving the data loading performance.</p> -<pre><code>TBLPROPERTIES ('NO_INVERTED_INDEX'='column1, column3') +<pre><code>TBLPROPERTIES ('NO_INVERTED_INDEX'='column1', 'INVERTED_INDEX'='column2, column3') </code></pre> </li> <li> @@ -549,6 +564,8 @@ Following are 5 configurations:</p> <li>TIMESTAMP</li> <li>DATE</li> <li>BOOLEAN</li> +<li>FLOAT</li> +<li>BYTE</li> </ul> </li> <li> @@ -746,7 +763,7 @@ You can refer to SDKwriterTestCase for example.</p> <h5> <a id="compression-for-table" class="anchor" href="#compression-for-table" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Compression for table</h5> <p>Data compression is also supported by CarbonData. -By default, Snappy is used to compress the data. CarbonData also support ZSTD compressor. +By default, Snappy is used to compress the data. CarbonData also supports ZSTD compressor. User can specify the compressor in the table property:</p> <pre><code>TBLPROPERTIES('carbon.column.compressor'='snappy') </code></pre> @@ -770,7 +787,19 @@ The corresponding system property is configured in carbon.properties file as bel As the table path remains the same after rename therefore the user can use this property to specify bad records path for the table at the time of creation, so that the same path can be later viewed in table description for reference.</p> -<pre><code> TBLPROPERTIES('BAD_RECORD_PATH'='/opt/badrecords'') +<pre><code> TBLPROPERTIES('BAD_RECORD_PATH'='/opt/badrecords') +</code></pre> +</li> +<li> +<h5> +<a id="load-minimum-data-size" class="anchor" href="#load-minimum-data-size" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Load minimum data size</h5> +<p>This property indicates the minimum input data size per node for data loading. +By default it is not enabled. Setting a non-zero integer value will enable this feature. +This property is useful if you have a large cluster and only want a small portion of the nodes to process data loading. +For example, if you have a cluster with 10 nodes and the input data is about 1GB. Without this property, each node will process about 100MB input data and result in at least 10 data files. With this property configured with 512, only 2 nodes will be chosen to process the input data, each with about 512MB input and result in about 2 or 4 files based on the compress ratio. +Moreover, this property can also be specified in the load option. +Notice that once you enable this feature, for load balance, carbondata will ignore the data locality while assigning input data to nodes, this will cause more network traffic.</p> +<pre><code> TBLPROPERTIES('LOAD_MIN_SIZE_INMB'='256') </code></pre> </li> </ul> @@ -832,14 +861,14 @@ checkAnswer(sql("SELECT count(*) from source"), sql("SELECT count(*) from origin <h3> <a id="create-external-table-on-non-transactional-table-data-location" class="anchor" href="#create-external-table-on-non-transactional-table-data-location" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Create external table on Non-Transactional table data location.</h3> <p>Non-Transactional table data location will have only carbondata and carbonindex files, there will not be a metadata folder (table status and schema). -Our SDK module currently support writing data in this format.</p> +Our SDK module currently supports writing data in this format.</p> <p><strong>Example:</strong></p> <pre><code>sql( s"""CREATE EXTERNAL TABLE sdkOutputTable STORED AS carbondata LOCATION |'$writerPath' """.stripMargin) </code></pre> <p>Here writer path will have carbondata and index files. -This can be SDK output. Refer <a href="./sdk-guide.html">SDK Guide</a>.</p> +This can be SDK output or C++ SDK output. Refer <a href="./sdk-guide.html">SDK Guide</a> and <a href="./csdk-guide.html">C++ SDK Guide</a>.</p> <p><strong>Note:</strong></p> <ol> <li>Dropping of the external table should not delete the files present in the location.</li>
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/ae77df2e/content/dml-of-carbondata.html ---------------------------------------------------------------------- diff --git a/content/dml-of-carbondata.html b/content/dml-of-carbondata.html index a96578a..15ff807 100644 --- a/content/dml-of-carbondata.html +++ b/content/dml-of-carbondata.html @@ -52,6 +52,9 @@ aria-expanded="false"> Download <span class="caret"></span></a> <ul class="dropdown-menu"> <li> + <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.1/"; + target="_blank">Apache CarbonData 1.5.1</a></li> + <li> <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"; target="_blank">Apache CarbonData 1.5.0</a></li> <li> @@ -64,9 +67,6 @@ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"; target="_blank">Apache CarbonData 1.3.1</a></li> <li> - <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"; - target="_blank">Apache CarbonData 1.3.0</a></li> - <li> <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"; target="_blank">Release Archive</a></li> </ul> @@ -306,7 +306,7 @@ OPTIONS(property_name=property_value, ...) </tr> <tr> <td><a href="#sort-column-bounds">SORT_COLUMN_BOUNDS</a></td> -<td>How to parititon the sort columns to make the evenly distributed</td> +<td>How to partition the sort columns to make the evenly distributed</td> </tr> <tr> <td><a href="#single_pass">SINGLE_PASS</a></td> @@ -353,7 +353,7 @@ OPTIONS(property_name=property_value, ...) <li> <h5> <a id="commentchar" class="anchor" href="#commentchar" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>COMMENTCHAR:</h5> -<p>Comment Characters can be provided in the load command if user want to comment lines.</p> +<p>Comment Characters can be provided in the load command if user wants to comment lines.</p> <pre><code>OPTIONS('COMMENTCHAR'='#') </code></pre> </li> @@ -443,7 +443,7 @@ true: CSV file is with file header.</p> <p><strong>NOTE:</strong></p> <ul> <li>SORT_COLUMN_BOUNDS will be used only when the SORT_SCOPE is 'local_sort'.</li> -<li>Carbondata will use these bounds as ranges to process data concurrently during the final sort percedure. The records will be sorted and written out inside each partition. Since the partition is sorted, all records will be sorted.</li> +<li>Carbondata will use these bounds as ranges to process data concurrently during the final sort procedure. The records will be sorted and written out inside each partition. Since the partition is sorted, all records will be sorted.</li> <li>Since the actual order and literal order of the dictionary column are not necessarily the same, we do not recommend you to use this feature if the first sort column is 'dictionary_include'.</li> <li>The option works better if your CPU usage during loading is low. If your current system CPU usage is high, better not to use this option. Besides, it depends on the user to specify the bounds. If user does not know the exactly bounds to make the data distributed evenly among the bounds, loading performance will still be better than before or at least the same as before.</li> <li>Users can find more information about this option in the description of PR1953.</li> http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/ae77df2e/content/documentation.html ---------------------------------------------------------------------- diff --git a/content/documentation.html b/content/documentation.html index aeab9cc..e49cdae 100644 --- a/content/documentation.html +++ b/content/documentation.html @@ -52,6 +52,9 @@ aria-expanded="false"> Download <span class="caret"></span></a> <ul class="dropdown-menu"> <li> + <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.1/"; + target="_blank">Apache CarbonData 1.5.1</a></li> + <li> <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"; target="_blank">Apache CarbonData 1.5.0</a></li> <li> @@ -64,9 +67,6 @@ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"; target="_blank">Apache CarbonData 1.3.1</a></li> <li> - <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"; - target="_blank">Apache CarbonData 1.3.0</a></li> - <li> <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"; target="_blank">Release Archive</a></li> </ul> @@ -226,7 +226,7 @@ <p><strong>File Format Concepts:</strong> Start with the basics of understanding the <a href="./file-structure-of-carbondata.html#carbondata-file-format">CarbonData file format</a> and its <a href="./file-structure-of-carbondata.html">storage structure</a>. This will help to understand other parts of the documentation, including deployment, programming and usage guides.</p> <p><strong>Quick Start:</strong> <a href="./quick-start-guide.html#installing-and-configuring-carbondata-to-run-locally-with-spark-shell">Run an example program</a> on your local machine or <a href="https://github.com/apache/carbondata/tree/master/examples/spark2/src/main/scala/org/apache/carbondata/examples"; target=_blank>study some examples</a>.</p> <p><strong>CarbonData SQL Language Reference:</strong> CarbonData extends the Spark SQL language and adds several <a href="./ddl-of-carbondata.html">DDL</a> and <a href="./dml-of-carbondata.html">DML</a> statements to support operations on it.Refer to the <a href="./language-manual.html">Reference Manual</a> to understand the supported features and functions.</p> -<p><strong>Programming Guides:</strong> You can read our guides about <a href="./sdk-guide.html">APIs supported</a> to learn how to integrate CarbonData with your applications.</p> +<p><strong>Programming Guides:</strong> You can read our guides about <a href="./sdk-guide.html">Java APIs supported</a> or <a href="./csdk-guide.html">C++ APIs supported</a> to learn how to integrate CarbonData with your applications.</p> <h2> <a id="integration" class="anchor" href="#integration" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Integration</h2> <p>CarbonData can be integrated with popular Execution engines like <a href="./quick-start-guide.html#spark">Spark</a> and <a href="./quick-start-guide.html#presto">Presto</a>.Refer to the <a href="./quick-start-guide.html#integration">Installation and Configuration</a> section to understand all modes of Integrating CarbonData.</p> http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/ae77df2e/content/faq.html ---------------------------------------------------------------------- diff --git a/content/faq.html b/content/faq.html index 2068c17..a42bbb8 100644 --- a/content/faq.html +++ b/content/faq.html @@ -52,6 +52,9 @@ aria-expanded="false"> Download <span class="caret"></span></a> <ul class="dropdown-menu"> <li> + <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.1/"; + target="_blank">Apache CarbonData 1.5.1</a></li> + <li> <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"; target="_blank">Apache CarbonData 1.5.0</a></li> <li> @@ -64,9 +67,6 @@ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"; target="_blank">Apache CarbonData 1.3.1</a></li> <li> - <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"; - target="_blank">Apache CarbonData 1.3.0</a></li> - <li> <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"; target="_blank">Release Archive</a></li> </ul> @@ -390,17 +390,15 @@ TimeZone.setDefault(TimeZone.getTimeZone("Asia/Shanghai")) <h2> <a id="how-to-check-lru-cache-memory-footprint" class="anchor" href="#how-to-check-lru-cache-memory-footprint" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>How to check LRU cache memory footprint?</h2> <p>To observe the LRU cache memory footprint in the logs, configure the below properties in log4j.properties file.</p> -<pre><code>log4j.logger.org.apache.carbondata.core.memory.UnsafeMemoryManager = DEBUG -log4j.logger.org.apache.carbondata.core.cache.CarbonLRUCache = DEBUG +<pre><code>log4j.logger.org.apache.carbondata.core.cache.CarbonLRUCache = DEBUG </code></pre> -<p>These properties will enable the DEBUG log for the CarbonLRUCache and UnsafeMemoryManager which will print the information of memory consumed using which the LRU cache size can be decided. <strong>Note:</strong> Enabling the DEBUG log will degrade the query performance.</p> +<p>This property will enable the DEBUG log for the CarbonLRUCache and UnsafeMemoryManager which will print the information of memory consumed using which the LRU cache size can be decided. <strong>Note:</strong> Enabling the DEBUG log will degrade the query performance. Ensure carbon.max.driver.lru.cache.size is configured to observe the current cache size.</p> <p><strong>Example:</strong></p> -<pre><code>18/09/26 15:05:28 DEBUG UnsafeMemoryManager: pool-44-thread-1 Memory block (org.apache.carbondata.core.memory.MemoryBlock@21312095) is created with size 10. Total memory used 413Bytes, left 536870499Bytes -18/09/26 15:05:29 DEBUG CarbonLRUCache: main Required size for entry /home/target/store/default/stored_as_carbondata_table/Fact/Part0/Segment_0/0_1537954529044.carbonindexmerge :: 181 Current cache size :: 0 -18/09/26 15:05:30 DEBUG UnsafeMemoryManager: main Freeing memory of size: 105available memory: 536870836 -18/09/26 15:05:30 DEBUG UnsafeMemoryManager: main Freeing memory of size: 76available memory: 536870912 +<pre><code>18/09/26 15:05:29 DEBUG CarbonLRUCache: main Required size for entry /home/target/store/default/stored_as_carbondata_table/Fact/Part0/Segment_0/0_1537954529044.carbonindexmerge :: 181 Current cache size :: 0 18/09/26 15:05:30 INFO CarbonLRUCache: main Removed entry from InMemory lru cache :: /home/target/store/default/stored_as_carbondata_table/Fact/Part0/Segment_0/0_1537954529044.carbonindexmerge </code></pre> +<p><strong>Note:</strong> If <code>Removed entry from InMemory LRU cache</code> are frequently observed in logs, you may have to increase the configured LRU size.</p> +<p>To observe the LRU cache from heap dump, check the heap used by CarbonLRUCache class.</p> <h2> <a id="getting-tablestatuslock-issues-when-loading-data" class="anchor" href="#getting-tablestatuslock-issues-when-loading-data" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Getting tablestatus.lock issues When loading data</h2> <p><strong>Symptom</strong></p> http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/ae77df2e/content/file-structure-of-carbondata.html ---------------------------------------------------------------------- diff --git a/content/file-structure-of-carbondata.html b/content/file-structure-of-carbondata.html index baa34db..5230ba3 100644 --- a/content/file-structure-of-carbondata.html +++ b/content/file-structure-of-carbondata.html @@ -52,6 +52,9 @@ aria-expanded="false"> Download <span class="caret"></span></a> <ul class="dropdown-menu"> <li> + <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.1/"; + target="_blank">Apache CarbonData 1.5.1</a></li> + <li> <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"; target="_blank">Apache CarbonData 1.5.0</a></li> <li> @@ -64,9 +67,6 @@ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"; target="_blank">Apache CarbonData 1.3.1</a></li> <li> - <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"; - target="_blank">Apache CarbonData 1.3.0</a></li> - <li> <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"; target="_blank">Release Archive</a></li> </ul> @@ -313,8 +313,7 @@ Several encodings that may be used in CarbonData files.</li> <p><a href="../docs/images/2-3_3.png?raw=true" target="_blank" rel="noopener noreferrer"><img src="https://github.com/apache/carbondata/blob/master/docs/images/2-3_3.png?raw=true"; alt="V3" style="max-width:100%;"></a></p> <h4> <a id="footer-format" class="anchor" href="#footer-format" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Footer format</h4> -<p>Footer records each carbondata -All blocklet data distribution information and statistical related metadata information (minmax, startkey/endkey) inside the file.</p> +<p>Footer records each carbondata, all blocklet data distribution information and statistical related metadata information (minmax, startkey/endkey) inside the file.</p> <p><a href="../docs/images/2-3_4.png?raw=true" target="_blank" rel="noopener noreferrer"><img src="https://github.com/apache/carbondata/blob/master/docs/images/2-3_4.png?raw=true"; alt="Footer format" style="max-width:100%;"></a></p> <ol> <li>BlockletInfo3 is used to record the offset and length of all ColumnChunk3.</li> http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/ae77df2e/content/how-to-contribute-to-apache-carbondata.html ---------------------------------------------------------------------- diff --git a/content/how-to-contribute-to-apache-carbondata.html b/content/how-to-contribute-to-apache-carbondata.html index eae12e3..a6dc1ee 100644 --- a/content/how-to-contribute-to-apache-carbondata.html +++ b/content/how-to-contribute-to-apache-carbondata.html @@ -52,6 +52,9 @@ aria-expanded="false"> Download <span class="caret"></span></a> <ul class="dropdown-menu"> <li> + <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.1/"; + target="_blank">Apache CarbonData 1.5.1</a></li> + <li> <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"; target="_blank">Apache CarbonData 1.5.0</a></li> <li> @@ -64,9 +67,6 @@ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"; target="_blank">Apache CarbonData 1.3.1</a></li> <li> - <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"; - target="_blank">Apache CarbonData 1.3.0</a></li> - <li> <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"; target="_blank">Release Archive</a></li> </ul> http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/ae77df2e/content/index.html ---------------------------------------------------------------------- diff --git a/content/index.html b/content/index.html index ea86500..54bf380 100644 --- a/content/index.html +++ b/content/index.html @@ -54,6 +54,9 @@ class="caret"></span></a> <ul class="dropdown-menu"> <li> + <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.1/"; + target="_blank">Apache CarbonData 1.5.1</a></li> + <li> <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"; target="_blank">Apache CarbonData 1.5.0</a></li> <li> @@ -66,9 +69,6 @@ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"; target="_blank">Apache CarbonData 1.3.1</a></li> <li> - <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"; - target="_blank">Apache CarbonData 1.3.0</a></li> - <li> <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"; target="_blank">Release Archive</a></li> </ul> @@ -313,6 +313,13 @@ </h4> <div class="linkblock"> <div class="block-row"> + <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.1/"; + target="_blank">Apache CarbonData 1.5.1</a> + <span class="release-date">Dec 2018</span> + <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Apache+CarbonData+1.5.1+Release"; + class="whatsnew" target="_blank">what's new</a> + </div> + <div class="block-row"> <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"; target="_blank">Apache CarbonData 1.5.0</a> <span class="release-date">Oct 2018</span> @@ -478,7 +485,7 @@ to do is:</p> <ol class="orderlist"> <li> - <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"; + <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.1/"; target="_blank">Download</a>the latest release. </li> http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/ae77df2e/content/introduction.html ---------------------------------------------------------------------- diff --git a/content/introduction.html b/content/introduction.html index 51a46c2..0cfa369 100644 --- a/content/introduction.html +++ b/content/introduction.html @@ -52,6 +52,9 @@ aria-expanded="false"> Download <span class="caret"></span></a> <ul class="dropdown-menu"> <li> + <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.1/"; + target="_blank">Apache CarbonData 1.5.1</a></li> + <li> <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"; target="_blank">Apache CarbonData 1.5.0</a></li> <li> @@ -64,9 +67,6 @@ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"; target="_blank">Apache CarbonData 1.3.1</a></li> <li> - <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"; - target="_blank">Apache CarbonData 1.3.0</a></li> - <li> <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"; target="_blank">Release Archive</a></li> </ul> http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/ae77df2e/content/language-manual.html ---------------------------------------------------------------------- diff --git a/content/language-manual.html b/content/language-manual.html index 9f738b8..a95de91 100644 --- a/content/language-manual.html +++ b/content/language-manual.html @@ -52,6 +52,9 @@ aria-expanded="false"> Download <span class="caret"></span></a> <ul class="dropdown-menu"> <li> + <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.1/"; + target="_blank">Apache CarbonData 1.5.1</a></li> + <li> <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"; target="_blank">Apache CarbonData 1.5.0</a></li> <li> @@ -64,9 +67,6 @@ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"; target="_blank">Apache CarbonData 1.3.1</a></li> <li> - <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"; - target="_blank">Apache CarbonData 1.3.0</a></li> - <li> <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"; target="_blank">Release Archive</a></li> </ul> http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/ae77df2e/content/lucene-datamap-guide.html ---------------------------------------------------------------------- diff --git a/content/lucene-datamap-guide.html b/content/lucene-datamap-guide.html index f461ca5..ef819a5 100644 --- a/content/lucene-datamap-guide.html +++ b/content/lucene-datamap-guide.html @@ -52,6 +52,9 @@ aria-expanded="false"> Download <span class="caret"></span></a> <ul class="dropdown-menu"> <li> + <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.1/"; + target="_blank">Apache CarbonData 1.5.1</a></li> + <li> <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"; target="_blank">Apache CarbonData 1.5.0</a></li> <li> @@ -64,9 +67,6 @@ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"; target="_blank">Apache CarbonData 1.3.1</a></li> <li> - <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"; - target="_blank">Apache CarbonData 1.3.0</a></li> - <li> <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"; target="_blank">Release Archive</a></li> </ul> http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/ae77df2e/content/performance-tuning.html ---------------------------------------------------------------------- diff --git a/content/performance-tuning.html b/content/performance-tuning.html index e077e85..e539614 100644 --- a/content/performance-tuning.html +++ b/content/performance-tuning.html @@ -52,6 +52,9 @@ aria-expanded="false"> Download <span class="caret"></span></a> <ul class="dropdown-menu"> <li> + <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.1/"; + target="_blank">Apache CarbonData 1.5.1</a></li> + <li> <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"; target="_blank">Apache CarbonData 1.5.0</a></li> <li> @@ -64,9 +67,6 @@ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"; target="_blank">Apache CarbonData 1.3.1</a></li> <li> - <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"; - target="_blank">Apache CarbonData 1.3.0</a></li> - <li> <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"; target="_blank">Release Archive</a></li> </ul> @@ -399,8 +399,8 @@ You can configure CarbonData by tuning following properties in carbon.properties </tr> <tr> <td>carbon.sort.file.write.buffer.size</td> -<td>Default: 50000.</td> -<td>DataOutputStream buffer.</td> +<td>Default: 16384.</td> +<td>CarbonData sorts and writes data to intermediate files to limit the memory usage. This configuration determines the buffer size to be used for reading and writing such files.</td> </tr> <tr> <td>carbon.merge.sort.reader.thread</td> @@ -474,7 +474,7 @@ scenarios. After the completion of POC, some of the configurations impacting the <tr> <td>carbon.detail.batch.size</td> <td>spark/carbonlib/carbon.properties</td> -<td>Data loading</td> +<td>Querying</td> <td>The buffer size to store records, returned from the block scan.</td> <td>In limit scenario this parameter is very important. For example your query limit is 1000. But if we set this value to 3000 that means we get 3000 records from scan but spark will only take 1000 rows. So the 2000 remaining are useless. In one Finance test case after we set it to 100, in the limit 1000 scenario the performance increase about 2 times in comparison to if we set this value to 12000.</td> </tr> @@ -486,18 +486,11 @@ scenarios. After the completion of POC, some of the configurations impacting the <td>If this is set it to true CarbonData will use YARN local directories for multi-table load disk load balance, that will improve the data load performance.</td> </tr> <tr> -<td>carbon.use.multiple.temp.dir</td> -<td>spark/carbonlib/carbon.properties</td> -<td>Data loading</td> -<td>Whether to use multiple YARN local directories during table data loading for disk load balance</td> -<td>After enabling 'carbon.use.local.dir', if this is set to true, CarbonData will use all YARN local directories during data load for disk load balance, that will improve the data load performance. Please enable this property when you encounter disk hotspot problem during data loading.</td> -</tr> -<tr> <td>carbon.sort.temp.compressor</td> <td>spark/carbonlib/carbon.properties</td> <td>Data loading</td> <td>Specify the name of compressor to compress the intermediate sort temporary files during sort procedure in data loading.</td> -<td>The optional values are 'SNAPPY','GZIP','BZIP2','LZ4','ZSTD', and empty. By default, empty means that Carbondata will not compress the sort temp files. This parameter will be useful if you encounter disk bottleneck.</td> +<td>The optional values are 'SNAPPY','GZIP','BZIP2','LZ4','ZSTD', and empty. Specially, empty means that Carbondata will not compress the sort temp files. This parameter will be useful if you encounter disk bottleneck.</td> </tr> <tr> <td>carbon.load.skewedDataOptimization.enabled</td> http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/ae77df2e/content/preaggregate-datamap-guide.html ---------------------------------------------------------------------- diff --git a/content/preaggregate-datamap-guide.html b/content/preaggregate-datamap-guide.html index e4f1c91..5e0d4e3 100644 --- a/content/preaggregate-datamap-guide.html +++ b/content/preaggregate-datamap-guide.html @@ -52,6 +52,9 @@ aria-expanded="false"> Download <span class="caret"></span></a> <ul class="dropdown-menu"> <li> + <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.1/"; + target="_blank">Apache CarbonData 1.5.1</a></li> + <li> <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"; target="_blank">Apache CarbonData 1.5.0</a></li> <li> @@ -64,9 +67,6 @@ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"; target="_blank">Apache CarbonData 1.3.1</a></li> <li> - <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"; - target="_blank">Apache CarbonData 1.3.0</a></li> - <li> <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"; target="_blank">Release Archive</a></li> </ul> http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/ae77df2e/content/quick-start-guide.html ---------------------------------------------------------------------- diff --git a/content/quick-start-guide.html b/content/quick-start-guide.html index 4703ed3..a2f093d 100644 --- a/content/quick-start-guide.html +++ b/content/quick-start-guide.html @@ -52,6 +52,9 @@ aria-expanded="false"> Download <span class="caret"></span></a> <ul class="dropdown-menu"> <li> + <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.1/"; + target="_blank">Apache CarbonData 1.5.1</a></li> + <li> <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"; target="_blank">Apache CarbonData 1.5.0</a></li> <li> @@ -64,9 +67,6 @@ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"; target="_blank">Apache CarbonData 1.3.1</a></li> <li> - <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"; - target="_blank">Apache CarbonData 1.3.0</a></li> - <li> <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"; target="_blank">Release Archive</a></li> </ul> @@ -564,7 +564,7 @@ hdfs://<host_name>:port/user/hive/warehouse/carbon.store <h2> <a id="installing-and-configuring-carbondata-on-presto" class="anchor" href="#installing-and-configuring-carbondata-on-presto" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Installing and Configuring CarbonData on Presto</h2> <p><strong>NOTE:</strong> <strong>CarbonData tables cannot be created nor loaded from Presto. User need to create CarbonData Table and load data into it -either with <a href="#installing-and-configuring-carbondata-to-run-locally-with-spark-shell">Spark</a> or <a href="./sdk-guide.html">SDK</a>. +either with <a href="#installing-and-configuring-carbondata-to-run-locally-with-spark-shell">Spark</a> or <a href="./sdk-guide.html">SDK</a> or <a href="./csdk-guide.html">C++ SDK</a>. Once the table is created,it can be queried from Presto.</strong></p> <h3> <a id="installing-presto" class="anchor" href="#installing-presto" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Installing Presto</h3> http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/ae77df2e/content/release-guide.html ---------------------------------------------------------------------- diff --git a/content/release-guide.html b/content/release-guide.html index c40f316..dcdaba3 100644 --- a/content/release-guide.html +++ b/content/release-guide.html @@ -52,6 +52,9 @@ aria-expanded="false"> Download <span class="caret"></span></a> <ul class="dropdown-menu"> <li> + <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.1/"; + target="_blank">Apache CarbonData 1.5.1</a></li> + <li> <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"; target="_blank">Apache CarbonData 1.5.0</a></li> <li> @@ -64,9 +67,6 @@ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"; target="_blank">Apache CarbonData 1.3.1</a></li> <li> - <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"; - target="_blank">Apache CarbonData 1.3.0</a></li> - <li> <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"; target="_blank">Release Archive</a></li> </ul> http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/ae77df2e/content/s3-guide.html ---------------------------------------------------------------------- diff --git a/content/s3-guide.html b/content/s3-guide.html index 27c1c66..ba25dfb 100644 --- a/content/s3-guide.html +++ b/content/s3-guide.html @@ -52,6 +52,9 @@ aria-expanded="false"> Download <span class="caret"></span></a> <ul class="dropdown-menu"> <li> + <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.1/"; + target="_blank">Apache CarbonData 1.5.1</a></li> + <li> <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"; target="_blank">Apache CarbonData 1.5.0</a></li> <li> @@ -64,9 +67,6 @@ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"; target="_blank">Apache CarbonData 1.3.1</a></li> <li> - <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"; - target="_blank">Apache CarbonData 1.3.0</a></li> - <li> <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"; target="_blank">Release Archive</a></li> </ul> http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/ae77df2e/content/sdk-guide.html ---------------------------------------------------------------------- diff --git a/content/sdk-guide.html b/content/sdk-guide.html index f33d5f9..37d6b26 100644 --- a/content/sdk-guide.html +++ b/content/sdk-guide.html @@ -52,6 +52,9 @@ aria-expanded="false"> Download <span class="caret"></span></a> <ul class="dropdown-menu"> <li> + <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.1/"; + target="_blank">Apache CarbonData 1.5.1</a></li> + <li> <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"; target="_blank">Apache CarbonData 1.5.0</a></li> <li> @@ -64,9 +67,6 @@ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"; target="_blank">Apache CarbonData 1.3.1</a></li> <li> - <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"; - target="_blank">Apache CarbonData 1.3.0</a></li> - <li> <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"; target="_blank">Release Archive</a></li> </ul> @@ -227,10 +227,14 @@ </ol> <h1> <a id="sdk-writer" class="anchor" href="#sdk-writer" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>SDK Writer</h1> -<p>In the carbon jars package, there exist a carbondata-store-sdk-x.x.x-SNAPSHOT.jar, including SDK writer and reader.</p> +<p>In the carbon jars package, there exist a carbondata-store-sdk-x.x.x-SNAPSHOT.jar, including SDK writer and reader. +If user want to use SDK, except carbondata-store-sdk-x.x.x-SNAPSHOT.jar, +it needs carbondata-core-x.x.x-SNAPSHOT.jar, carbondata-common-x.x.x-SNAPSHOT.jar, +carbondata-format-x.x.x-SNAPSHOT.jar, carbondata-hadoop-x.x.x-SNAPSHOT.jar and carbondata-processing-x.x.x-SNAPSHOT.jar. +What's more, user also can use carbondata-sdk.jar directly.</p> <p>This SDK writer, writes carbondata file and carbonindex file at a given path. External client can make use of this writer to convert other format data or live data to create carbondata and index files. -These SDK writer output contains just a carbondata and carbonindex files. No metadata folder will be present.</p> +These SDK writer output contains just carbondata and carbonindex files. No metadata folder will be present.</p> <h2> <a id="quick-example" class="anchor" href="#quick-example" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Quick example</h2> <h3> @@ -267,7 +271,7 @@ These SDK writer output contains just a carbondata and carbonindex files. No met <span class="pl-smi">CarbonProperties</span><span class="pl-k">.</span>getInstance()<span class="pl-k">.</span>addProperty(<span class="pl-s"><span class="pl-pds">"</span>enable.offheap.sort<span class="pl-pds">"</span></span>, enableOffheap); - <span class="pl-smi">CarbonWriterBuilder</span> builder <span class="pl-k">=</span> <span class="pl-smi">CarbonWriter</span><span class="pl-k">.</span>builder()<span class="pl-k">.</span>outputPath(path)<span class="pl-k">.</span>withCsvInput(schema); + <span class="pl-smi">CarbonWriterBuilder</span> builder <span class="pl-k">=</span> <span class="pl-smi">CarbonWriter</span><span class="pl-k">.</span>builder()<span class="pl-k">.</span>outputPath(path)<span class="pl-k">.</span>withCsvInput(schema)<span class="pl-k">.</span>writtenBy(<span class="pl-s"><span class="pl-pds">"</span>SDK<span class="pl-pds">"</span></span>); <span class="pl-smi">CarbonWriter</span> writer <span class="pl-k">=</span> builder<span class="pl-k">.</span>build(); @@ -322,7 +326,7 @@ These SDK writer output contains just a carbondata and carbonindex files. No met <span class="pl-k">try</span> { <span class="pl-smi">CarbonWriter</span> writer <span class="pl-k">=</span> <span class="pl-smi">CarbonWriter</span><span class="pl-k">.</span>builder() .outputPath(path) - .withAvroInput(<span class="pl-k">new</span> <span class="pl-smi">org.apache.avro<span class="pl-k">.</span>Schema</span>.<span class="pl-smi">Parser</span>()<span class="pl-k">.</span>parse(avroSchema))<span class="pl-k">.</span>build(); + .withAvroInput(<span class="pl-k">new</span> <span class="pl-smi">org.apache.avro<span class="pl-k">.</span>Schema</span>.<span class="pl-smi">Parser</span>()<span class="pl-k">.</span>parse(avroSchema))<span class="pl-k">.</span>writtenBy(<span class="pl-s"><span class="pl-pds">"</span>SDK<span class="pl-pds">"</span></span>)<span class="pl-k">.</span>build(); <span class="pl-k">for</span> (<span class="pl-k">int</span> i <span class="pl-k">=</span> <span class="pl-c1">0</span>; i <span class="pl-k"><</span> <span class="pl-c1">100</span>; i<span class="pl-k">++</span>) { writer<span class="pl-k">.</span>write(record); @@ -360,7 +364,7 @@ These SDK writer output contains just a carbondata and carbonindex files. No met <span class="pl-smi">Schema</span> <span class="pl-smi">CarbonSchema</span> <span class="pl-k">=</span> <span class="pl-k">new</span> <span class="pl-smi">Schema</span>(fields); - <span class="pl-smi">CarbonWriterBuilder</span> builder <span class="pl-k">=</span> <span class="pl-smi">CarbonWriter</span><span class="pl-k">.</span>builder()<span class="pl-k">.</span>outputPath(path)<span class="pl-k">.</span>withJsonInput(<span class="pl-smi">CarbonSchema</span>); + <span class="pl-smi">CarbonWriterBuilder</span> builder <span class="pl-k">=</span> <span class="pl-smi">CarbonWriter</span><span class="pl-k">.</span>builder()<span class="pl-k">.</span>outputPath(path)<span class="pl-k">.</span>withJsonInput(<span class="pl-smi">CarbonSchema</span>)<span class="pl-k">.</span>writtenBy(<span class="pl-s"><span class="pl-pds">"</span>SDK<span class="pl-pds">"</span></span>); <span class="pl-c"><span class="pl-c">//</span> initialize json writer with carbon schema</span> <span class="pl-smi">CarbonWriter</span> writer <span class="pl-k">=</span> builder<span class="pl-k">.</span>build(); @@ -644,6 +648,8 @@ public CarbonWriterBuilder withLoadOptions(Map<String, String> options); * j. sort_scope -- "local_sort", "no_sort", "batch_sort". default value is "local_sort" * k. long_string_columns -- comma separated string columns which are more than 32k length. * default value is null. +* l. inverted_index -- comma separated string columns for which inverted index needs to be +* generated * * @return updated CarbonWriterBuilder */ @@ -667,6 +673,15 @@ public CarbonWriterBuilder withThreadSafe(short numOfThreads); */ public CarbonWriterBuilder withHadoopConf(Configuration conf) </code></pre> +<pre><code> /** + * Updates the hadoop configuration with the given key value + * + * @param key key word + * @param value value + * @return this object + */ + public CarbonWriterBuilder withHadoopConf(String key, String value); +</code></pre> <pre><code>/** * to build a {@link CarbonWriter}, which accepts row in CSV format * @@ -692,6 +707,23 @@ public CarbonWriterBuilder withAvroInput(org.apache.avro.Schema avroSchema); public CarbonWriterBuilder withJsonInput(Schema carbonSchema); </code></pre> <pre><code>/** +* To support writing the ApplicationName which is writing the carbondata file +* This is a mandatory API to call, else the build() call will fail with error. +* @param application name which is writing the carbondata files +* @return CarbonWriterBuilder +*/ +public CarbonWriterBuilder writtenBy(String appName) { +</code></pre> +<pre><code>/** +* sets the list of columns for which inverted index needs to generated +* @param invertedIndexColumns is a string array of columns for which inverted index needs to +* generated. +* If it is null or an empty array, inverted index will be generated for none of the columns +* @return updated CarbonWriterBuilder +*/ +public CarbonWriterBuilder invertedIndexFor(String[] invertedIndexColumns); +</code></pre> +<pre><code>/** * Build a {@link CarbonWriter} * This writer is not thread safe, * use withThreadSafe() configuration in multi thread environment @@ -702,9 +734,22 @@ public CarbonWriterBuilder withJsonInput(Schema carbonSchema); */ public CarbonWriter build() throws IOException, InvalidLoadOptionException; </code></pre> +<pre><code> /** + * Configure Row Record Reader for reading. + * + */ + public CarbonReaderBuilder withRowRecordReader() +</code></pre> <h3> <a id="class-orgapachecarbondatasdkfilecarbonwriter" class="anchor" href="#class-orgapachecarbondatasdkfilecarbonwriter" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Class org.apache.carbondata.sdk.file.CarbonWriter</h3> <pre><code>/** +* Create a {@link CarbonWriterBuilder} to build a {@link CarbonWriter} +*/ +public static CarbonWriterBuilder builder() { + return new CarbonWriterBuilder(); +} +</code></pre> +<pre><code>/** * Write an object to the file, the format of the object depends on the implementation * If AvroCarbonWriter, object is of type org.apache.avro.generic.GenericData.Record, * which is one row of data. @@ -720,13 +765,6 @@ public abstract void write(Object object) throws IOException; */ public abstract void close() throws IOException; </code></pre> -<pre><code>/** -* Create a {@link CarbonWriterBuilder} to build a {@link CarbonWriter} -*/ -public static CarbonWriterBuilder builder() { - return new CarbonWriterBuilder(); -} -</code></pre> <h3> <a id="class-orgapachecarbondatasdkfilefield" class="anchor" href="#class-orgapachecarbondatasdkfilefield" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Class org.apache.carbondata.sdk.file.Field</h3> <pre><code>/** @@ -824,6 +862,24 @@ External client can make use of this reader to read CarbonData files without Car */ public static CarbonReaderBuilder builder(String tablePath); </code></pre> +<pre><code>/** + * Breaks the list of CarbonRecordReader in CarbonReader into multiple + * CarbonReader objects, each iterating through some 'carbondata' files + * and return that list of CarbonReader objects + * + * If the no. of files is greater than maxSplits, then break the + * CarbonReader into maxSplits splits, with each split iterating + * through >= 1 file. + * + * If the no. of files is less than maxSplits, then return list of + * CarbonReader with size as the no. of files, with each CarbonReader + * iterating through exactly one file + * + * @param maxSplits: Int + * @return list of CarbonReader objects + */ + public List<CarbonReader> split(int maxSplits); +</code></pre> <pre><code> /** * Return true if has next row */ @@ -835,6 +891,11 @@ External client can make use of this reader to read CarbonData files without Car public T readNextRow(); </code></pre> <pre><code> /** + * Read and return next batch row objects + */ + public Object[] readNextBatchRow(); +</code></pre> +<pre><code> /** * Close reader */ public void close(); @@ -865,6 +926,14 @@ External client can make use of this reader to read CarbonData files without Car */ public CarbonReaderBuilder filter(Expression filterExpression); </code></pre> +<pre><code> /** + * Sets the batch size of records to read + * + * @param batch batch size + * @return updated CarbonReaderBuilder + */ + public CarbonReaderBuilder withBatch(int batch); +</code></pre> <pre><code>/** * To support hadoop configuration * @@ -873,6 +942,15 @@ External client can make use of this reader to read CarbonData files without Car */ public CarbonReaderBuilder withHadoopConf(Configuration conf); </code></pre> +<pre><code> /** + * Updates the hadoop configuration with the given key value + * + * @param key key word + * @param value value + * @return this object + */ + public CarbonReaderBuilder withHadoopConf(String key, String value); +</code></pre> <pre><code> /** * Build CarbonReader * @@ -892,6 +970,7 @@ External client can make use of this reader to read CarbonData files without Car * @return schema object * @throws IOException */ + @Deprecated public static Schema readSchemaInSchemaFile(String schemaFilePath); </code></pre> <pre><code> /** @@ -900,6 +979,7 @@ External client can make use of this reader to read CarbonData files without Car * @param dataFilePath complete path including carbondata file name * @return Schema object */ + @Deprecated public static Schema readSchemaInDataFile(String dataFilePath); </code></pre> <pre><code> /** @@ -909,8 +989,42 @@ External client can make use of this reader to read CarbonData files without Car * @return schema object * @throws IOException */ + @Deprecated public static Schema readSchemaInIndexFile(String indexFilePath); </code></pre> +<pre><code> /** + * read schema from path, + * path can be folder path,carbonindex file path, and carbondata file path + * and will not check all files schema + * + * @param path file/folder path + * @return schema + * @throws IOException + */ + public static Schema readSchema(String path); +</code></pre> +<pre><code> /** + * read schema from path, + * path can be folder path,carbonindex file path, and carbondata file path + * and user can decide whether check all files schema + * + * @param path file/folder path + * @param validateSchema whether check all files schema + * @return schema + * @throws IOException + */ + public static Schema readSchema(String path, boolean validateSchema); +</code></pre> +<pre><code> /** + * This method return the version details in formatted string by reading from carbondata file + * If application name is SDK_1.0.0 and this has written the carbondata file in carbondata 1.6 project version, + * then this API returns the String "SDK_1.0.0 in version: 1.6.0-SNAPSHOT" + * @param dataFilePath complete path including carbondata file name + * @return string with information of who has written this file in which carbondata project version + * @throws IOException + */ + public static String getVersionDetails(String dataFilePath); +</code></pre> <h3> <a id="class-orgapachecarbondatasdkfileschema-1" class="anchor" href="#class-orgapachecarbondatasdkfileschema-1" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Class org.apache.carbondata.sdk.file.Schema</h3> <pre><code> /** http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/ae77df2e/content/security.html ---------------------------------------------------------------------- diff --git a/content/security.html b/content/security.html index ce1bc30..75a2f65 100644 --- a/content/security.html +++ b/content/security.html @@ -45,6 +45,9 @@ aria-expanded="false">Download <span class="caret"></span></a> <ul class="dropdown-menu"> <li> + <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.1/"; + target="_blank">Apache CarbonData 1.5.1</a></li> + <li> <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"; target="_blank">Apache CarbonData 1.5.0</a></li> <li> http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/ae77df2e/content/segment-management-on-carbondata.html ---------------------------------------------------------------------- diff --git a/content/segment-management-on-carbondata.html b/content/segment-management-on-carbondata.html index 2f04025..dae0d0e 100644 --- a/content/segment-management-on-carbondata.html +++ b/content/segment-management-on-carbondata.html @@ -52,6 +52,9 @@ aria-expanded="false"> Download <span class="caret"></span></a> <ul class="dropdown-menu"> <li> + <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.1/"; + target="_blank">Apache CarbonData 1.5.1</a></li> + <li> <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"; target="_blank">Apache CarbonData 1.5.0</a></li> <li> @@ -64,9 +67,6 @@ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"; target="_blank">Apache CarbonData 1.3.1</a></li> <li> - <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"; - target="_blank">Apache CarbonData 1.3.0</a></li> - <li> <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"; target="_blank">Release Archive</a></li> </ul> http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/ae77df2e/content/streaming-guide.html ---------------------------------------------------------------------- diff --git a/content/streaming-guide.html b/content/streaming-guide.html index 7ea58ce..8d8cb82 100644 --- a/content/streaming-guide.html +++ b/content/streaming-guide.html @@ -52,6 +52,9 @@ aria-expanded="false"> Download <span class="caret"></span></a> <ul class="dropdown-menu"> <li> + <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.1/"; + target="_blank">Apache CarbonData 1.5.1</a></li> + <li> <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"; target="_blank">Apache CarbonData 1.5.0</a></li> <li> @@ -64,9 +67,6 @@ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"; target="_blank">Apache CarbonData 1.3.1</a></li> <li> - <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"; - target="_blank">Apache CarbonData 1.3.0</a></li> - <li> <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"; target="_blank">Release Archive</a></li> </ul> @@ -243,9 +243,10 @@ <li> <a href="#streaming-job-management">Streaming Job Management</a> <ul> -<li><a href="#start-stream">START STREAM</a></li> -<li><a href="#stop-stream">STOP STREAM</a></li> +<li><a href="#create-stream">CREATE STREAM</a></li> +<li><a href="#drop-stream">DROP STREAM</a></li> <li><a href="#show-streams">SHOW STREAMS</a></li> +<li><a href="#close-stream">CLOSE STREAM</a></li> </ul> </li> </ul> @@ -570,7 +571,7 @@ streaming table using following DDL.</p> sql( """ - |START STREAM job123 ON TABLE sink + |CREATE STREAM job123 ON TABLE sink |STMPROPERTIES( | 'trigger'='ProcessingTime', | 'interval'='1 seconds') @@ -580,7 +581,7 @@ streaming table using following DDL.</p> | WHERE id % 2 = 1 """.stripMargin) - sql("STOP STREAM job123") + sql("DROP STREAM job123") sql("SHOW STREAMS [ON TABLE tableName]") </code></pre> @@ -591,14 +592,14 @@ streaming table using following DDL.</p> <p>As above example shown:</p> <ul> <li> -<code>START STREAM jobName ON TABLE tableName</code> is used to start a streaming ingest job.</li> +<code>CREATE STREAM jobName ON TABLE tableName</code> is used to start a streaming ingest job.</li> <li> -<code>STOP STREAM jobName</code> is used to stop a streaming job by its name</li> +<code>DROP STREAM jobName</code> is used to stop a streaming job by its name</li> <li> <code>SHOW STREAMS [ON TABLE tableName]</code> is used to print streaming job information</li> </ul> <h5> -<a id="start-stream" class="anchor" href="#start-stream" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>START STREAM</h5> +<a id="create-stream" class="anchor" href="#create-stream" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>CREATE STREAM</h5> <p>When this is issued, carbon will start a structured streaming job to do the streaming ingestion. Before launching the job, system will validate:</p> <ul> <li> @@ -651,9 +652,24 @@ TBLPROPERTIES( <span class="pl-s"><span class="pl-pds">'</span>record_format<span class="pl-pds">'</span></span><span class="pl-k">=</span><span class="pl-s"><span class="pl-pds">'</span>csv<span class="pl-pds">'</span></span>, <span class="pl-k">//</span> can be csv <span class="pl-k">or</span> json, default is csv <span class="pl-s"><span class="pl-pds">'</span>delimiter<span class="pl-pds">'</span></span><span class="pl-k">=</span><span class="pl-s"><span class="pl-pds">'</span>|<span class="pl-pds">'</span></span> )</pre></div> +<ul> +<li>Then CREATE STREAM can be used to start the streaming ingest job from source table to sink table</li> +</ul> +<pre><code>CREATE STREAM job123 ON TABLE sink +STMPROPERTIES( + 'trigger'='ProcessingTime', + 'interval'='10 seconds' +) +AS + SELECT * + FROM source + WHERE id % 2 = 1 +</code></pre> <h5> -<a id="stop-stream" class="anchor" href="#stop-stream" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>STOP STREAM</h5> -<p>When this is issued, the streaming job will be stopped immediately. It will fail if the jobName specified is not exist.</p> +<a id="drop-stream" class="anchor" href="#drop-stream" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>DROP STREAM</h5> +<p>When <code>DROP STREAM</code> is issued, the streaming job will be stopped immediately. It will fail if the jobName specified is not exist.</p> +<pre><code>DROP STREAM job123 +</code></pre> <h5> <a id="show-streams" class="anchor" href="#show-streams" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>SHOW STREAMS</h5> <p><code>SHOW STREAMS ON TABLE tableName</code> command will print the streaming job information as following</p> @@ -680,6 +696,10 @@ TBLPROPERTIES( </tbody> </table> <p><code>SHOW STREAMS</code> command will show all stream jobs in the system.</p> +<h5> +<a id="alter-table-close-stream" class="anchor" href="#alter-table-close-stream" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>ALTER TABLE CLOSE STREAM</h5> +<p>When the streaming application is stopped, and user want to manually trigger data conversion from carbon streaming files to columnar files, one can use +<code>ALTER TABLE sink COMPACT 'CLOSE_STREAMING';</code></p> <script> $(function() { // Show selected style on nav item http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/ae77df2e/content/supported-data-types-in-carbondata.html ---------------------------------------------------------------------- diff --git a/content/supported-data-types-in-carbondata.html b/content/supported-data-types-in-carbondata.html index 4f584da..d873fab 100644 --- a/content/supported-data-types-in-carbondata.html +++ b/content/supported-data-types-in-carbondata.html @@ -52,6 +52,9 @@ aria-expanded="false"> Download <span class="caret"></span></a> <ul class="dropdown-menu"> <li> + <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.1/"; + target="_blank">Apache CarbonData 1.5.1</a></li> + <li> <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"; target="_blank">Apache CarbonData 1.5.0</a></li> <li> @@ -64,9 +67,6 @@ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"; target="_blank">Apache CarbonData 1.3.1</a></li> <li> - <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"; - target="_blank">Apache CarbonData 1.3.0</a></li> - <li> <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"; target="_blank">Release Archive</a></li> </ul> http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/ae77df2e/content/timeseries-datamap-guide.html ---------------------------------------------------------------------- diff --git a/content/timeseries-datamap-guide.html b/content/timeseries-datamap-guide.html index a9137d0..9550fa1 100644 --- a/content/timeseries-datamap-guide.html +++ b/content/timeseries-datamap-guide.html @@ -52,6 +52,9 @@ aria-expanded="false"> Download <span class="caret"></span></a> <ul class="dropdown-menu"> <li> + <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.1/"; + target="_blank">Apache CarbonData 1.5.1</a></li> + <li> <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"; target="_blank">Apache CarbonData 1.5.0</a></li> <li> @@ -64,9 +67,6 @@ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"; target="_blank">Apache CarbonData 1.3.1</a></li> <li> - <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"; - target="_blank">Apache CarbonData 1.3.0</a></li> - <li> <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"; target="_blank">Release Archive</a></li> </ul> http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/ae77df2e/content/usecases.html ---------------------------------------------------------------------- diff --git a/content/usecases.html b/content/usecases.html index 7e5e07c..b6b4d74 100644 --- a/content/usecases.html +++ b/content/usecases.html @@ -52,6 +52,9 @@ aria-expanded="false"> Download <span class="caret"></span></a> <ul class="dropdown-menu"> <li> + <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.1/"; + target="_blank">Apache CarbonData 1.5.1</a></li> + <li> <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"; target="_blank">Apache CarbonData 1.5.0</a></li> <li> @@ -64,9 +67,6 @@ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"; target="_blank">Apache CarbonData 1.3.1</a></li> <li> - <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"; - target="_blank">Apache CarbonData 1.3.0</a></li> - <li> <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"; target="_blank">Release Archive</a></li> </ul> @@ -330,12 +330,6 @@ <td>yarn application directory will be usually on a single disk.YARN would be configured with multiple disks to be used as temp or to assign randomly to applications. Using the yarn temp directory will allow carbon to use multiple disks and improve IO performance</td> </tr> <tr> -<td>Data Loading</td> -<td>carbon.use.multiple.temp.dir</td> -<td>TRUE</td> -<td>multiple disks to write sort files will lead to better IO and reduce the IO bottleneck</td> -</tr> -<tr> <td>Compaction</td> <td>carbon.compaction.level.threshold</td> <td>6,6</td> @@ -468,12 +462,6 @@ </tr> <tr> <td>Data Loading</td> -<td>carbon.use.multiple.temp.dir</td> -<td>TRUE</td> -<td>multiple disks to write sort files will lead to better IO and reduce the IO bottleneck</td> -</tr> -<tr> -<td>Data Loading</td> <td>sort.inmemory.size.in.mb</td> <td>92160</td> <td>Memory allocated to do inmemory sorting. When more memory is available in the node, configuring this will retain more sort blocks in memory so that the merge sort is faster due to no/very less IO</td> http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/ae77df2e/content/videogallery.html ---------------------------------------------------------------------- diff --git a/content/videogallery.html b/content/videogallery.html index d6b9dbe..7b7e66c 100644 --- a/content/videogallery.html +++ b/content/videogallery.html @@ -49,6 +49,9 @@ aria-expanded="false"> Download <span class="caret"></span></a> <ul class="dropdown-menu"> <li> + <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.1/"; + target="_blank">Apache CarbonData 1.5.1</a></li> + <li> <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"; target="_blank">Apache CarbonData 1.5.0</a></li> <li> @@ -61,9 +64,6 @@ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"; target="_blank">Apache CarbonData 1.3.1</a></li> <li> - <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"; - target="_blank">Apache CarbonData 1.3.0</a></li> - <li> <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"; target="_blank">Release Archive</a></li> </ul> http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/ae77df2e/src/main/resources/application.conf ---------------------------------------------------------------------- diff --git a/src/main/resources/application.conf b/src/main/resources/application.conf index 020b127..2f1b695 100644 --- a/src/main/resources/application.conf +++ b/src/main/resources/application.conf @@ -17,7 +17,7 @@ fileList=["configuration-parameters", "how-to-contribute-to-apache-carbondata", "introduction", "usecases", - "CSDK-guide", + "csdk-guide", "carbon-as-spark-datasource-guide" ] dataMapFileList=[ http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/ae77df2e/src/main/scala/html/header.html ---------------------------------------------------------------------- diff --git a/src/main/scala/html/header.html b/src/main/scala/html/header.html index 52e6681..196736f 100644 --- a/src/main/scala/html/header.html +++ b/src/main/scala/html/header.html @@ -52,6 +52,9 @@ aria-expanded="false"> Download <span class="caret"></span></a> <ul class="dropdown-menu"> <li> + <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.1/"; + target="_blank">Apache CarbonData 1.5.1</a></li> + <li> <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"; target="_blank">Apache CarbonData 1.5.0</a></li> <li> @@ -64,9 +67,6 @@ <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"; target="_blank">Apache CarbonData 1.3.1</a></li> <li> - <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"; - target="_blank">Apache CarbonData 1.3.0</a></li> - <li> <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"; target="_blank">Release Archive</a></li> </ul> http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/ae77df2e/src/main/scala/scripts/CSDK-guide ---------------------------------------------------------------------- diff --git a/src/main/scala/scripts/CSDK-guide b/src/main/scala/scripts/CSDK-guide deleted file mode 100644 index 2c8ffc0..0000000 --- a/src/main/scala/scripts/CSDK-guide +++ /dev/null @@ -1,11 +0,0 @@ -<script> -$(function() { - // Show selected style on nav item - $('.b-nav__api').addClass('selected'); - - if (!$('.b-nav__api').parent().hasClass('nav__item__with__subs--expanded')) { - // Display api subnav items - $('.b-nav__api').parent().toggleClass('nav__item__with__subs--expanded'); - } -}); -</script> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/ae77df2e/src/main/scala/scripts/csdk-guide ---------------------------------------------------------------------- diff --git a/src/main/scala/scripts/csdk-guide b/src/main/scala/scripts/csdk-guide new file mode 100644 index 0000000..2c8ffc0 --- /dev/null +++ b/src/main/scala/scripts/csdk-guide @@ -0,0 +1,11 @@ +<script> +$(function() { + // Show selected style on nav item + $('.b-nav__api').addClass('selected'); + + if (!$('.b-nav__api').parent().hasClass('nav__item__with__subs--expanded')) { + // Display api subnav items + $('.b-nav__api').parent().toggleClass('nav__item__with__subs--expanded'); + } +}); +</script> \ No newline at end of file
![https://github.com/apache/carbondata/blob/master/docs/images/2-3_3.png?raw=true"](https://github.com/apache/carbondata/blob/master/docs/images/2-3_3.png?raw=true"){kind=link}
![https://github.com/apache/carbondata/blob/master/docs/images/2-3_4.png?raw=true"](https://github.com/apache/carbondata/blob/master/docs/images/2-3_4.png?raw=true"){kind=link}