carbondata git commit: [CARBONDATA-2818] Upgrade presto integration version to 0.210
Repository: carbondata Updated Branches: refs/heads/master 629d62549 -> 8427771fc [CARBONDATA-2818] Upgrade presto integration version to 0.210 Upgrade presto integration version to 0.210 This closes #2733 Project: http://git-wip-us.apache.org/repos/asf/carbondata/repo Commit: http://git-wip-us.apache.org/repos/asf/carbondata/commit/8427771f Tree: http://git-wip-us.apache.org/repos/asf/carbondata/tree/8427771f Diff: http://git-wip-us.apache.org/repos/asf/carbondata/diff/8427771f Branch: refs/heads/master Commit: 8427771fc22071099186d3310500d5fcd6c419a6 Parents: 629d625 Author: chenliang613 Authored: Wed Sep 19 08:18:28 2018 +0800 Committer: Raghunandan S Committed: Fri Sep 28 11:39:48 2018 +0530 -- docs/quick-start-guide.md | 29 .../Presto_Cluster_Setup_For_Carbondata.md | 24 integration/presto/README.md| 12 +--- integration/presto/pom.xml | 4 +-- 4 files changed, 40 insertions(+), 29 deletions(-) -- http://git-wip-us.apache.org/repos/asf/carbondata/blob/8427771f/docs/quick-start-guide.md -- diff --git a/docs/quick-start-guide.md b/docs/quick-start-guide.md index 37c398c..0fdf055 100644 --- a/docs/quick-start-guide.md +++ b/docs/quick-start-guide.md @@ -300,24 +300,24 @@ Once the table is created,it can be queried from Presto.** ### Installing Presto - 1. Download the 0.187 version of Presto using: -`wget https://repo1.maven.org/maven2/com/facebook/presto/presto-server/0.187/presto-server-0.187.tar.gz` + 1. Download the 0.210 version of Presto using: +`wget https://repo1.maven.org/maven2/com/facebook/presto/presto-server/0.210/presto-server-0.210.tar.gz` - 2. Extract Presto tar file: `tar zxvf presto-server-0.187.tar.gz`. + 2. Extract Presto tar file: `tar zxvf presto-server-0.210.tar.gz`. 3. Download the Presto CLI for the coordinator and name it presto. ``` -wget https://repo1.maven.org/maven2/com/facebook/presto/presto-cli/0.187/presto-cli-0.187-executable.jar +wget https://repo1.maven.org/maven2/com/facebook/presto/presto-cli/0.210/presto-cli-0.210-executable.jar -mv presto-cli-0.187-executable.jar presto +mv presto-cli-0.210-executable.jar presto chmod +x presto ``` ### Create Configuration Files - 1. Create `etc` folder in presto-server-0.187 directory. + 1. Create `etc` folder in presto-server-0.210 directory. 2. Create `config.properties`, `jvm.config`, `log.properties`, and `node.properties` files. 3. Install uuid to generate a node.id. @@ -363,10 +363,15 @@ Once the table is created,it can be queried from Presto.** coordinator=true node-scheduler.include-coordinator=false http-server.http.port=8086 - query.max-memory=50GB - query.max-memory-per-node=2GB + query.max-memory=5GB + query.max-total-memory-per-node=5GB + query.max-memory-per-node=3GB + memory.heap-headroom-per-node=1GB discovery-server.enabled=true - discovery.uri=:8086 + discovery.uri=http://localhost:8086 + task.max-worker-threads=4 + optimizer.dictionary-aggregation=true + optimizer.optimize-hash-generation = false ``` The options `node-scheduler.include-coordinator=false` and `coordinator=true` indicate that the node is the coordinator and tells the coordinator not to do any of the computation work itself and to use the workers. @@ -383,7 +388,7 @@ Then, `query.max-memory=<30GB * number of nodes>`. ``` coordinator=false http-server.http.port=8086 - query.max-memory=50GB + query.max-memory=5GB query.max-memory-per-node=2GB discovery.uri=:8086 ``` @@ -405,12 +410,12 @@ Then, `query.max-memory=<30GB * number of nodes>`. ### Start Presto Server on all nodes ``` -./presto-server-0.187/bin/launcher start +./presto-server-0.210/bin/launcher start ``` To run it as a background process. ``` -./presto-server-0.187/bin/launcher run +./presto-server-0.210/bin/launcher run ``` To run it in foreground. http://git-wip-us.apache.org/repos/asf/carbondata/blob/8427771f/integration/presto/Presto_Cluster_Setup_For_Carbondata.md -- diff --git a/integration/presto/Presto_Cluster_Setup_For_Carbondata.md b/integration/presto/Presto_Cluster_Setup_For_Carbondata.md index 082b8fe..975e1fa 100644 --- a/integration/presto/Presto_Cluster_Setup_For_Carbondata.md +++ b/integration/presto/Presto_Cluster_Setup_For_Carbondata.md @@ -2,24 +2,24 @@ ## Installing Presto - 1. Download the 0.187 version of Presto using: - `wget https://repo1.maven.org/maven2/com/facebook/presto/presto-server/0.187/presto-server-0.187.tar.gz` + 1. Download the 0.210 version of Presto using: + `wget https://repo1.maven.org/mave
carbondata git commit: [CARBONDATA-3111] Readme updated some error links had been fixed
Repository: carbondata Updated Branches: refs/heads/master d79ba999f -> c2ae98744 [CARBONDATA-3111] Readme updated some error links had been fixed Readme updated some error links that link to datamap had been link to correct file This closes #2934 Project: http://git-wip-us.apache.org/repos/asf/carbondata/repo Commit: http://git-wip-us.apache.org/repos/asf/carbondata/commit/c2ae9874 Tree: http://git-wip-us.apache.org/repos/asf/carbondata/tree/c2ae9874 Diff: http://git-wip-us.apache.org/repos/asf/carbondata/diff/c2ae9874 Branch: refs/heads/master Commit: c2ae98744d56dc1227eefbdf5f6c28ed4f56c238 Parents: d79ba99 Author: RebieKong Authored: Wed Nov 21 13:28:13 2018 +0800 Committer: Raghunandan S Committed: Wed Nov 21 15:01:06 2018 +0530 -- README.md | 10 +- 1 file changed, 5 insertions(+), 5 deletions(-) -- http://git-wip-us.apache.org/repos/asf/carbondata/blob/c2ae9874/README.md -- diff --git a/README.md b/README.md index 8e5a4b0..2e77a54 100644 --- a/README.md +++ b/README.md @@ -55,11 +55,11 @@ CarbonData is built using Apache Maven, to [build CarbonData](https://github.com * [Configuring CarbonData](https://github.com/apache/carbondata/blob/master/docs/configuration-parameters.md) * [DataMap Developer Guide](https://github.com/apache/carbondata/blob/master/docs/datamap-developer-guide.md) * [Data Types](https://github.com/apache/carbondata/blob/master/docs/supported-data-types-in-carbondata.md) -* [CarbonData DataMap Management](https://github.com/apache/carbondata/blob/master/docs/datamap-management.md) - * [CarbonData BloomFilter DataMap](https://github.com/apache/carbondata/blob/master/docs/bloomfilter-datamap-guide.md) - * [CarbonData Lucene DataMap](https://github.com/apache/carbondata/blob/master/docs/lucene-datamap-guide.md) - * [CarbonData Pre-aggregate DataMap](https://github.com/apache/carbondata/blob/master/docs/preaggregate-datamap-guide.md) - * [CarbonData Timeseries DataMap](https://github.com/apache/carbondata/blob/master/docs/timeseries-datamap-guide.md) +* [CarbonData DataMap Management](https://github.com/apache/carbondata/blob/master/docs/datamap/datamap-management.md) + * [CarbonData BloomFilter DataMap](https://github.com/apache/carbondata/blob/master/docs/datamap/bloomfilter-datamap-guide.md) + * [CarbonData Lucene DataMap](https://github.com/apache/carbondata/blob/master/docs/datamap/lucene-datamap-guide.md) + * [CarbonData Pre-aggregate DataMap](https://github.com/apache/carbondata/blob/master/docs/datamap/preaggregate-datamap-guide.md) + * [CarbonData Timeseries DataMap](https://github.com/apache/carbondata/blob/master/docs/datamap/timeseries-datamap-guide.md) * [SDK Guide](https://github.com/apache/carbondata/blob/master/docs/sdk-guide.md) * [C++ SDK Guide](https://github.com/apache/carbondata/blob/master/docs/csdk-guide.md) * [Performance Tuning](https://github.com/apache/carbondata/blob/master/docs/performance-tuning.md)
carbondata git commit: [Documentation] Local dictionary Data which are not supported float and byte updated in the note Float and Byte data types which are not supported are added to note and formatti
Repository: carbondata Updated Branches: refs/heads/master c5de10cb1 -> f51c40add [Documentation] Local dictionary Data which are not supported float and byte updated in the note Float and Byte data types which are not supported are added to note and formatting issues handled This closes #2805 Project: http://git-wip-us.apache.org/repos/asf/carbondata/repo Commit: http://git-wip-us.apache.org/repos/asf/carbondata/commit/f51c40ad Tree: http://git-wip-us.apache.org/repos/asf/carbondata/tree/f51c40ad Diff: http://git-wip-us.apache.org/repos/asf/carbondata/diff/f51c40ad Branch: refs/heads/master Commit: f51c40addf385f8b07f9e4c722184d6f6438ba64 Parents: c5de10c Author: sgururajshetty Authored: Tue Oct 9 16:34:20 2018 +0530 Committer: Raghunandan S Committed: Fri Nov 23 16:43:31 2018 +0530 -- docs/configuration-parameters.md | 76 +-- docs/ddl-of-carbondata.md| 3 +- 2 files changed, 40 insertions(+), 39 deletions(-) -- http://git-wip-us.apache.org/repos/asf/carbondata/blob/f51c40ad/docs/configuration-parameters.md -- diff --git a/docs/configuration-parameters.md b/docs/configuration-parameters.md index c210cae..e9e5978 100644 --- a/docs/configuration-parameters.md +++ b/docs/configuration-parameters.md @@ -16,7 +16,7 @@ --> # Configuring CarbonData - This guide explains the configurations that can be used to tune CarbonData to achieve better performance.Most of the properties that control the internal settings have reasonable default values. They are listed along with the properties along with explanation. + This guide explains the configurations that can be used to tune CarbonData to achieve better performance. Most of the properties that control the internal settings have reasonable default values. They are listed along with the properties along with explanation. * [System Configuration](#system-configuration) * [Data Loading Configuration](#data-loading-configuration) @@ -36,17 +36,17 @@ This section provides the details of all the configurations required for the Car | carbon.badRecords.location | (none) | CarbonData can detect the records not conforming to defined table schema and isolate them as bad records. This property is used to specify where to store such bad records. | | carbon.streaming.auto.handoff.enabled | true | CarbonData supports storing of streaming data. To have high throughput for streaming, the data is written in Row format which is highly optimized for write, but performs poorly for query. When this property is true and when the streaming data size reaches ***carbon.streaming.segment.max.size***, CabonData will automatically convert the data to columnar format and optimize it for faster querying.**NOTE:** It is not recommended to keep the default value which is true. | | carbon.streaming.segment.max.size | 102400 | CarbonData writes streaming data in row format which is optimized for high write throughput. This property defines the maximum size of data to be held is row format, beyond which it will be converted to columnar format in order to support high performance query, provided ***carbon.streaming.auto.handoff.enabled*** is true. **NOTE:** Setting higher value will impact the streaming ingestion. The value has to be configured in bytes. | -| carbon.query.show.datamaps | true | CarbonData stores datamaps as independent tables so as to allow independent maintenance to some extent. When this property is true,which is by default, show tables command will list all the tables including datatmaps(eg: Preaggregate table), else datamaps will be excluded from the table list.**NOTE:** It is generally not required for the user to do any maintenance operations on these tables and hence not required to be seen.But it is shown by default so that user or admin can get clear understanding of the system for capacity planning. | -| carbon.segment.lock.files.preserve.hours | 48 | In order to support parallel data loading onto the same table, CarbonData sequences(locks) at the granularity of segments.Operations affecting the segment(like IUD, alter) are blocked from parallel operations. This property value indicates the number of hours the segment lock files will be preserved after dataload. These lock files will be deleted with the clean command after the configured number of hours. | -| carbon.timestamp.format | -MM-dd HH:mm:ss | CarbonData can understand data of timestamp type and process it in special manner.It can be so that the format of Timestamp data is different from that understood by CarbonData by default. This configuration allows users to specify the format of Timestamp in their data. | +| carbon.query.show.datamaps | true | CarbonData stores datamaps as independent tab
[1/8] carbondata-site git commit: Added 1.5.1 version information
Repository: carbondata-site
Updated Branches:
refs/heads/asf-site 4574eccb4 -> ae77df2e4
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/ae77df2e/src/site/markdown/ddl-of-carbondata.md
--
diff --git a/src/site/markdown/ddl-of-carbondata.md
b/src/site/markdown/ddl-of-carbondata.md
index 933a448..965f11c 100644
--- a/src/site/markdown/ddl-of-carbondata.md
+++ b/src/site/markdown/ddl-of-carbondata.md
@@ -33,7 +33,9 @@ CarbonData DDL statements are documented here,which includes:
* [Hive/Parquet folder Structure](#support-flat-folder-same-as-hiveparquet)
* [Extra Long String columns](#string-longer-than-32000-characters)
* [Compression for Table](#compression-for-table)
- * [Bad Records Path](#bad-records-path)
+ * [Bad Records Path](#bad-records-path)
+ * [Load Minimum Input File Size](#load-minimum-data-size)
+
* [CREATE TABLE AS SELECT](#create-table-as-select)
* [CREATE EXTERNAL TABLE](#create-external-table)
* [External Table on Transactional table
location](#create-external-table-on-managed-table-data-location)
@@ -84,6 +86,7 @@ CarbonData DDL statements are documented here,which includes:
| |
|
| [DICTIONARY_INCLUDE](#dictionary-encoding-configuration) | Columns for
which dictionary needs to be generated |
| [NO_INVERTED_INDEX](#inverted-index-configuration) | Columns to
exclude from inverted index generation|
+| [INVERTED_INDEX](#inverted-index-configuration) | Columns to
include for inverted index generation |
| [SORT_COLUMNS](#sort-columns-configuration) | Columns to
include in sort and its order of sort |
| [SORT_SCOPE](#sort-scope-configuration) | Sort scope of
the load.Options include no sort, local sort ,batch sort and global sort |
| [TABLE_BLOCKSIZE](#table-block-size-configuration) | Size of
blocks to write onto hdfs|
@@ -104,6 +107,7 @@ CarbonData DDL statements are documented here,which
includes:
| [LONG_STRING_COLUMNS](#string-longer-than-32000-characters) | Columns which
are greater than 32K characters|
| [BUCKETNUMBER](#bucketing) | Number of
buckets to be created |
| [BUCKETCOLUMNS](#bucketing) | Columns which
are to be placed in buckets|
+| [LOAD_MIN_SIZE_INMB](#load-minimum-data-size)| Minimum input
data size per node for data loading |
Following are the guidelines for TBLPROPERTIES, CarbonData's additional table
options can be set via carbon.properties.
@@ -120,11 +124,11 @@ CarbonData DDL statements are documented here,which
includes:
- # Inverted Index Configuration
- By default inverted index is enabled, it might help to improve
compression ratio and query speed, especially for low cardinality columns which
are in reward position.
+ By default inverted index is disabled as store size will be reduced, it
can be enabled by using a table property. It might help to improve compression
ratio and query speed, especially for low cardinality columns which are in
reward position.
Suggested use cases : For high cardinality columns, you can disable the
inverted index for improving the data loading performance.
```
- TBLPROPERTIES ('NO_INVERTED_INDEX'='column1, column3')
+ TBLPROPERTIES ('NO_INVERTED_INDEX'='column1', 'INVERTED_INDEX'='column2,
column3')
```
- # Sort Columns Configuration
@@ -245,7 +249,8 @@ CarbonData DDL statements are documented here,which
includes:
* TIMESTAMP
* DATE
* BOOLEAN
-
+ * FLOAT
+ * BYTE
* In case of multi-level complex dataType columns, primitive
string/varchar/char columns are considered for local dictionary generation.
System Level Properties for Local Dictionary:
@@ -445,7 +450,7 @@ CarbonData DDL statements are documented here,which
includes:
- # Compression for table
Data compression is also supported by CarbonData.
- By default, Snappy is used to compress the data. CarbonData also support
ZSTD compressor.
+ By default, Snappy is used to compress the data. CarbonData also supports
ZSTD compressor.
User can specify the compressor in the table property:
```
@@ -474,7 +479,19 @@ CarbonData DDL statements are documented here,which
includes:
be later viewed in table description for reference.
```
- TBLPROPERTIES('BAD_RECORD_PATH'='/opt/badrecords'')
+ TBLPROPERTIES('BAD_RECORD_PATH'='/opt/badrecords')
+ ```
+
+ - # Load minimum data size
+ This property indicates the minimum input data size per node for data
loadi
[3/8] carbondata-site git commit: Added 1.5.1 version information
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/ae77df2e/src/main/webapp/ddl-of-carbondata.html
--
diff --git a/src/main/webapp/ddl-of-carbondata.html
b/src/main/webapp/ddl-of-carbondata.html
index 434f378..7f84786 100644
--- a/src/main/webapp/ddl-of-carbondata.html
+++ b/src/main/webapp/ddl-of-carbondata.html
@@ -52,6 +52,9 @@
aria-expanded="false"> Download
+https://dist.apache.org/repos/dist/release/carbondata/1.5.1/";
+ target="_blank">Apache CarbonData
1.5.1
+
https://dist.apache.org/repos/dist/release/carbondata/1.5.0/";
target="_blank">Apache CarbonData
1.5.0
@@ -64,9 +67,6 @@
https://dist.apache.org/repos/dist/release/carbondata/1.3.1/";
target="_blank">Apache CarbonData
1.3.1
-https://dist.apache.org/repos/dist/release/carbondata/1.3.0/";
- target="_blank">Apache CarbonData
1.3.0
-
https://cwiki.apache.org/confluence/display/CARBONDATA/Releases";
target="_blank">Release Archive
@@ -223,7 +223,7 @@
CarbonData DDL statements are documented here,which includes:
-CREATE TABLE
+CREATE TABLE
Dictionary Encoding
Inverted Index
@@ -239,19 +239,24 @@
Extra Long String
columns
Compression for Table
Bad Records Path
+Load Minimum Input File Size
-CREATE TABLE AS SELECT
-CREATE EXTERNAL TABLE
+CREATE TABLE AS SELECT
+
+
+CREATE EXTERNAL TABLE
External
Table on Transactional table location
External
Table on non-transactional table location
-CREATE DATABASE
-TABLE MANAGEMENT
+CREATE DATABASE
+
+
+TABLE MANAGEMENT
SHOW TABLE
@@ -271,7 +276,7 @@
-PARTITION
+PARTITION
STANDARD PARTITION(HIVE)
@@ -293,7 +298,9 @@
DROP PARTITION
-BUCKETING
+
+BUCKETING
+
CREATE TABLE
@@ -324,6 +331,10 @@ STORED AS carbondata
Columns to exclude from inverted index generation
+INVERTED_INDEX
+Columns to include for inverted index generation
+
+
SORT_COLUMNS
Columns to include in sort and its order of sort
@@ -403,6 +414,10 @@ STORED AS carbondata
BUCKETCOLUMNS
Columns which are to be placed in buckets
+
+LOAD_MIN_SIZE_INMB
+Minimum input data size per node for data loading
+
Following are the guidelines for TBLPROPERTIES, CarbonData's additional
table options can be set via carbon.properties.
@@ -419,9 +434,9 @@ Suggested use cases : do dictionary encoding for low
cardinality columns, it mig
Inverted Index
Configuration
-By default inverted index is enabled, it might help to improve compression
ratio and query speed, especially for low cardinality columns which are in
reward position.
+By default inverted index is disabled as store size will be reduced, it can
be enabled by using a table property. It might help to improve compression
ratio and query speed, especially for low cardinality columns which are in
reward position.
Suggested use cases : For high cardinality columns, you can disable the
inverted index for improving the data loading performance.
-TBLPROPERTIES ('NO_INVERTED_INDEX'='column1, column3')
+TBLPROPERTIES ('NO_INVERTED_INDEX'='column1',
'INVERTED_INDEX'='column2, column3')
@@ -549,6 +564,8 @@ Following are 5 configurations:
TIMESTAMP
DATE
BOOLEAN
+FLOAT
+BYTE
@@ -746,7 +763,7 @@ You can refer to SDKwriterTestCase for example.
Compression for table
Data compression is also supported by CarbonData.
-By default, Snappy is used to compress the data. CarbonData also support ZSTD
compressor.
+By default, Snappy is used to compress the data. CarbonData also supports ZSTD
compressor.
User can specify the compressor in the table property:
TBLPROPERTIES('carbon.column.compressor'='snappy')
@@ -770,7 +787,19 @@ The corresponding system property is configured in
carbon.properties file as bel
As the table path remains the same after rename therefore the user can use
this property to
specify bad records path for the table at the time of creation, so that the
same path can
be later viewed in table description for reference.
- TBLPROPERTIES('BAD_RECORD_PATH'='/opt/badrecords'')
+ TBLPROPERTIES('BAD_RECORD_PATH'='/opt/badrecords')
+
+
+
+
+Load minimum data size
+This property indicates the minimum input data size per node for data
loading.
+By default it is not enabled. Setting a non-zero integer value will enable
this feature.
+This property is useful if you have a large cluster and only want a small
portion of the nodes to process data loading.
+For example, if you hav
[7/8] carbondata-site git commit: Added 1.5.1 version information
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/ae77df2e/content/configuration-parameters.html -- diff --git a/content/configuration-parameters.html b/content/configuration-parameters.html index 5c334eb..5cc7a45 100644 --- a/content/configuration-parameters.html +++ b/content/configuration-parameters.html @@ -52,6 +52,9 @@ aria-expanded="false"> Download +https://dist.apache.org/repos/dist/release/carbondata/1.5.1/"; + target="_blank">Apache CarbonData 1.5.1 + https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"; target="_blank">Apache CarbonData 1.5.0 @@ -64,9 +67,6 @@ https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"; target="_blank">Apache CarbonData 1.3.1 -https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"; - target="_blank">Apache CarbonData 1.3.0 - https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"; target="_blank">Release Archive @@ -220,7 +220,7 @@ Configuring CarbonData -This guide explains the configurations that can be used to tune CarbonData to achieve better performance.Most of the properties that control the internal settings have reasonable default values. They are listed along with the properties along with explanation. +This guide explains the configurations that can be used to tune CarbonData to achieve better performance. Most of the properties that control the internal settings have reasonable default values. They are listed along with the properties along with explanation. System Configuration Data Loading Configuration @@ -244,7 +244,7 @@ carbon.storelocation spark.sql.warehouse.dir property value -Location where CarbonData will create the store, and write the data in its custom format. If not specified,the path defaults to spark.sql.warehouse.dir property. NOTE: Store location should be in HDFS. +Location where CarbonData will create the store, and write the data in its custom format. If not specified,the path defaults to spark.sql.warehouse.dir property. NOTE: Store location should be in HDFS or S3. carbon.ddl.base.hdfs.url @@ -269,17 +269,17 @@ carbon.query.show.datamaps true -CarbonData stores datamaps as independent tables so as to allow independent maintenance to some extent. When this property is true,which is by default, show tables command will list all the tables including datatmaps(eg: Preaggregate table), else datamaps will be excluded from the table list.NOTE: It is generally not required for the user to do any maintenance operations on these tables and hence not required to be seen.But it is shown by default so that user or admin can get clear understanding of the system for capacity planning. +CarbonData stores datamaps as independent tables so as to allow independent maintenance to some extent. When this property is true,which is by default, show tables command will list all the tables including datatmaps(eg: Preaggregate table), else datamaps will be excluded from the table list.NOTE: It is generally not required for the user to do any maintenance operations on these tables and hence not required to be seen. But it is shown by default so that user or admin can get clear understanding of the system for capacity planning. carbon.segment.lock.files.preserve.hours 48 -In order to support parallel data loading onto the same table, CarbonData sequences(locks) at the granularity of segments.Operations affecting the segment(like IUD, alter) are blocked from parallel operations. This property value indicates the number of hours the segment lock files will be preserved after dataload. These lock files will be deleted with the clean command after the configured number of hours. +In order to support parallel data loading onto the same table, CarbonData sequences(locks) at the granularity of segments. Operations affecting the segment(like IUD, alter) are blocked from parallel operations. This property value indicates the number of hours the segment lock files will be preserved after dataload. These lock files will be deleted with the clean command after the configured number of hours. carbon.timestamp.format -MM-dd HH:mm:ss -CarbonData can understand data of timestamp type and process it in special manner.It can be so that the format of Timestamp data is different from that understood by
[6/8] carbondata-site git commit: Added 1.5.1 version information
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/ae77df2e/content/ddl-of-carbondata.html
--
diff --git a/content/ddl-of-carbondata.html b/content/ddl-of-carbondata.html
index 434f378..7f84786 100644
--- a/content/ddl-of-carbondata.html
+++ b/content/ddl-of-carbondata.html
@@ -52,6 +52,9 @@
aria-expanded="false"> Download
+https://dist.apache.org/repos/dist/release/carbondata/1.5.1/";
+ target="_blank">Apache CarbonData
1.5.1
+
https://dist.apache.org/repos/dist/release/carbondata/1.5.0/";
target="_blank">Apache CarbonData
1.5.0
@@ -64,9 +67,6 @@
https://dist.apache.org/repos/dist/release/carbondata/1.3.1/";
target="_blank">Apache CarbonData
1.3.1
-https://dist.apache.org/repos/dist/release/carbondata/1.3.0/";
- target="_blank">Apache CarbonData
1.3.0
-
https://cwiki.apache.org/confluence/display/CARBONDATA/Releases";
target="_blank">Release Archive
@@ -223,7 +223,7 @@
CarbonData DDL statements are documented here,which includes:
-CREATE TABLE
+CREATE TABLE
Dictionary Encoding
Inverted Index
@@ -239,19 +239,24 @@
Extra Long String
columns
Compression for Table
Bad Records Path
+Load Minimum Input File Size
-CREATE TABLE AS SELECT
-CREATE EXTERNAL TABLE
+CREATE TABLE AS SELECT
+
+
+CREATE EXTERNAL TABLE
External
Table on Transactional table location
External
Table on non-transactional table location
-CREATE DATABASE
-TABLE MANAGEMENT
+CREATE DATABASE
+
+
+TABLE MANAGEMENT
SHOW TABLE
@@ -271,7 +276,7 @@
-PARTITION
+PARTITION
STANDARD PARTITION(HIVE)
@@ -293,7 +298,9 @@
DROP PARTITION
-BUCKETING
+
+BUCKETING
+
CREATE TABLE
@@ -324,6 +331,10 @@ STORED AS carbondata
Columns to exclude from inverted index generation
+INVERTED_INDEX
+Columns to include for inverted index generation
+
+
SORT_COLUMNS
Columns to include in sort and its order of sort
@@ -403,6 +414,10 @@ STORED AS carbondata
BUCKETCOLUMNS
Columns which are to be placed in buckets
+
+LOAD_MIN_SIZE_INMB
+Minimum input data size per node for data loading
+
Following are the guidelines for TBLPROPERTIES, CarbonData's additional
table options can be set via carbon.properties.
@@ -419,9 +434,9 @@ Suggested use cases : do dictionary encoding for low
cardinality columns, it mig
Inverted Index
Configuration
-By default inverted index is enabled, it might help to improve compression
ratio and query speed, especially for low cardinality columns which are in
reward position.
+By default inverted index is disabled as store size will be reduced, it can
be enabled by using a table property. It might help to improve compression
ratio and query speed, especially for low cardinality columns which are in
reward position.
Suggested use cases : For high cardinality columns, you can disable the
inverted index for improving the data loading performance.
-TBLPROPERTIES ('NO_INVERTED_INDEX'='column1, column3')
+TBLPROPERTIES ('NO_INVERTED_INDEX'='column1',
'INVERTED_INDEX'='column2, column3')
@@ -549,6 +564,8 @@ Following are 5 configurations:
TIMESTAMP
DATE
BOOLEAN
+FLOAT
+BYTE
@@ -746,7 +763,7 @@ You can refer to SDKwriterTestCase for example.
Compression for table
Data compression is also supported by CarbonData.
-By default, Snappy is used to compress the data. CarbonData also support ZSTD
compressor.
+By default, Snappy is used to compress the data. CarbonData also supports ZSTD
compressor.
User can specify the compressor in the table property:
TBLPROPERTIES('carbon.column.compressor'='snappy')
@@ -770,7 +787,19 @@ The corresponding system property is configured in
carbon.properties file as bel
As the table path remains the same after rename therefore the user can use
this property to
specify bad records path for the table at the time of creation, so that the
same path can
be later viewed in table description for reference.
- TBLPROPERTIES('BAD_RECORD_PATH'='/opt/badrecords'')
+ TBLPROPERTIES('BAD_RECORD_PATH'='/opt/badrecords')
+
+
+
+
+Load minimum data size
+This property indicates the minimum input data size per node for data
loading.
+By default it is not enabled. Setting a non-zero integer value will enable
this feature.
+This property is useful if you have a large cluster and only want a small
portion of the nodes to process data loading.
+For example, if you have a cluster with 10 nodes and the input d
[4/8] carbondata-site git commit: Added 1.5.1 version information
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/ae77df2e/src/main/webapp/configuration-parameters.html -- diff --git a/src/main/webapp/configuration-parameters.html b/src/main/webapp/configuration-parameters.html index 5c334eb..5cc7a45 100644 --- a/src/main/webapp/configuration-parameters.html +++ b/src/main/webapp/configuration-parameters.html @@ -52,6 +52,9 @@ aria-expanded="false"> Download +https://dist.apache.org/repos/dist/release/carbondata/1.5.1/"; + target="_blank">Apache CarbonData 1.5.1 + https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"; target="_blank">Apache CarbonData 1.5.0 @@ -64,9 +67,6 @@ https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"; target="_blank">Apache CarbonData 1.3.1 -https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"; - target="_blank">Apache CarbonData 1.3.0 - https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"; target="_blank">Release Archive @@ -220,7 +220,7 @@ Configuring CarbonData -This guide explains the configurations that can be used to tune CarbonData to achieve better performance.Most of the properties that control the internal settings have reasonable default values. They are listed along with the properties along with explanation. +This guide explains the configurations that can be used to tune CarbonData to achieve better performance. Most of the properties that control the internal settings have reasonable default values. They are listed along with the properties along with explanation. System Configuration Data Loading Configuration @@ -244,7 +244,7 @@ carbon.storelocation spark.sql.warehouse.dir property value -Location where CarbonData will create the store, and write the data in its custom format. If not specified,the path defaults to spark.sql.warehouse.dir property. NOTE: Store location should be in HDFS. +Location where CarbonData will create the store, and write the data in its custom format. If not specified,the path defaults to spark.sql.warehouse.dir property. NOTE: Store location should be in HDFS or S3. carbon.ddl.base.hdfs.url @@ -269,17 +269,17 @@ carbon.query.show.datamaps true -CarbonData stores datamaps as independent tables so as to allow independent maintenance to some extent. When this property is true,which is by default, show tables command will list all the tables including datatmaps(eg: Preaggregate table), else datamaps will be excluded from the table list.NOTE: It is generally not required for the user to do any maintenance operations on these tables and hence not required to be seen.But it is shown by default so that user or admin can get clear understanding of the system for capacity planning. +CarbonData stores datamaps as independent tables so as to allow independent maintenance to some extent. When this property is true,which is by default, show tables command will list all the tables including datatmaps(eg: Preaggregate table), else datamaps will be excluded from the table list.NOTE: It is generally not required for the user to do any maintenance operations on these tables and hence not required to be seen. But it is shown by default so that user or admin can get clear understanding of the system for capacity planning. carbon.segment.lock.files.preserve.hours 48 -In order to support parallel data loading onto the same table, CarbonData sequences(locks) at the granularity of segments.Operations affecting the segment(like IUD, alter) are blocked from parallel operations. This property value indicates the number of hours the segment lock files will be preserved after dataload. These lock files will be deleted with the clean command after the configured number of hours. +In order to support parallel data loading onto the same table, CarbonData sequences(locks) at the granularity of segments. Operations affecting the segment(like IUD, alter) are blocked from parallel operations. This property value indicates the number of hours the segment lock files will be preserved after dataload. These lock files will be deleted with the clean command after the configured number of hours. carbon.timestamp.format -MM-dd HH:mm:ss -CarbonData can understand data of timestamp type and process it in special manner.It can be so that the format of Timestamp da
[8/8] carbondata-site git commit: Added 1.5.1 version information
Added 1.5.1 version information Project: http://git-wip-us.apache.org/repos/asf/carbondata-site/repo Commit: http://git-wip-us.apache.org/repos/asf/carbondata-site/commit/ae77df2e Tree: http://git-wip-us.apache.org/repos/asf/carbondata-site/tree/ae77df2e Diff: http://git-wip-us.apache.org/repos/asf/carbondata-site/diff/ae77df2e Branch: refs/heads/asf-site Commit: ae77df2e4b6a69e37daac94993f4e58e40fb7d1d Parents: 4574ecc Author: Raghunandan S Authored: Fri Dec 7 12:38:57 2018 +0530 Committer: Raghunandan S Committed: Fri Dec 7 17:53:57 2018 +0530 -- content/CSDK-guide.html | 369 +++-- content/WEB-INF/classes/application.conf| 2 +- content/bloomfilter-datamap-guide.html | 6 +- content/carbon-as-spark-datasource-guide.html | 6 +- content/configuration-parameters.html | 274 ++--- content/datamap-developer-guide.html| 6 +- content/datamap-management.html | 6 +- content/ddl-of-carbondata.html | 61 ++- content/dml-of-carbondata.html | 12 +- content/documentation.html | 8 +- content/faq.html| 18 +- content/file-structure-of-carbondata.html | 9 +- .../how-to-contribute-to-apache-carbondata.html | 6 +- content/index.html | 15 +- content/introduction.html | 6 +- content/language-manual.html| 6 +- content/lucene-datamap-guide.html | 6 +- content/performance-tuning.html | 21 +- content/preaggregate-datamap-guide.html | 6 +- content/quick-start-guide.html | 8 +- content/release-guide.html | 6 +- content/s3-guide.html | 6 +- content/sdk-guide.html | 144 ++- content/security.html | 3 + content/segment-management-on-carbondata.html | 6 +- content/streaming-guide.html| 44 +- content/supported-data-types-in-carbondata.html | 6 +- content/timeseries-datamap-guide.html | 6 +- content/usecases.html | 18 +- content/videogallery.html | 6 +- src/main/resources/application.conf | 2 +- src/main/scala/html/header.html | 6 +- src/main/scala/scripts/CSDK-guide | 11 - src/main/scala/scripts/csdk-guide | 11 + src/main/webapp/CSDK-guide.html | 369 +++-- src/main/webapp/bloomfilter-datamap-guide.html | 6 +- .../carbon-as-spark-datasource-guide.html | 6 +- src/main/webapp/configuration-parameters.html | 274 ++--- src/main/webapp/datamap-developer-guide.html| 6 +- src/main/webapp/datamap-management.html | 6 +- src/main/webapp/ddl-of-carbondata.html | 61 ++- src/main/webapp/dml-of-carbondata.html | 12 +- src/main/webapp/documentation.html | 8 +- src/main/webapp/faq.html| 18 +- .../webapp/file-structure-of-carbondata.html| 9 +- .../how-to-contribute-to-apache-carbondata.html | 6 +- src/main/webapp/index.html | 6 +- src/main/webapp/introduction.html | 6 +- src/main/webapp/language-manual.html| 6 +- src/main/webapp/lucene-datamap-guide.html | 6 +- src/main/webapp/performance-tuning.html | 21 +- src/main/webapp/preaggregate-datamap-guide.html | 6 +- src/main/webapp/quick-start-guide.html | 8 +- src/main/webapp/release-guide.html | 6 +- src/main/webapp/s3-guide.html | 6 +- src/main/webapp/sdk-guide.html | 144 ++- src/main/webapp/security.html | 3 + .../segment-management-on-carbondata.html | 6 +- src/main/webapp/streaming-guide.html| 44 +- .../supported-data-types-in-carbondata.html | 6 +- src/main/webapp/timeseries-datamap-guide.html | 6 +- src/main/webapp/usecases.html | 18 +- src/main/webapp/videogallery.html | 6 +- src/site/markdown/CSDK-guide.md | 398 ++- src/site/markdown/configuration-parameters.md | 130 +++--- src/site/markdown/ddl-of-carbondata.md | 33 +- src/site/markdown/dml-of-carbondata.md | 6 +- src/site/markdown/documentation.md | 2 +- src/site/markdown/faq.md| 8 +- .../markdown/file-structure-of-carbondata.md| 3 +- src/site/markdown/performance-tuning.md | 7 +- src/site/markdown/quick-start-guide.md | 2 +- src/site/markdown/sdk-guide.md | 165 +++- src/site/markdown/streaming-guide.md
[2/8] carbondata-site git commit: Added 1.5.1 version information
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/ae77df2e/src/site/markdown/configuration-parameters.md -- diff --git a/src/site/markdown/configuration-parameters.md b/src/site/markdown/configuration-parameters.md index 0a4565a..4aa2929 100644 --- a/src/site/markdown/configuration-parameters.md +++ b/src/site/markdown/configuration-parameters.md @@ -16,7 +16,7 @@ --> # Configuring CarbonData - This guide explains the configurations that can be used to tune CarbonData to achieve better performance.Most of the properties that control the internal settings have reasonable default values. They are listed along with the properties along with explanation. + This guide explains the configurations that can be used to tune CarbonData to achieve better performance. Most of the properties that control the internal settings have reasonable default values. They are listed along with the properties along with explanation. * [System Configuration](#system-configuration) * [Data Loading Configuration](#data-loading-configuration) @@ -31,68 +31,66 @@ This section provides the details of all the configurations required for the Car | Property | Default Value | Description | ||-|--| -| carbon.storelocation | spark.sql.warehouse.dir property value | Location where CarbonData will create the store, and write the data in its custom format. If not specified,the path defaults to spark.sql.warehouse.dir property. **NOTE:** Store location should be in HDFS. | +| carbon.storelocation | spark.sql.warehouse.dir property value | Location where CarbonData will create the store, and write the data in its custom format. If not specified,the path defaults to spark.sql.warehouse.dir property. **NOTE:** Store location should be in HDFS or S3. | | carbon.ddl.base.hdfs.url | (none) | To simplify and shorten the path to be specified in DDL/DML commands, this property is supported. This property is used to configure the HDFS relative path, the path configured in carbon.ddl.base.hdfs.url will be appended to the HDFS path configured in fs.defaultFS of core-site.xml. If this path is configured, then user need not pass the complete path while dataload. For example: If absolute path of the csv file is hdfs://10.18.101.155:54310/data/cnbc/2016/xyz.csv, the path "hdfs://10.18.101.155:54310" will come from property fs.defaultFS and user can configure the /data/cnbc/ as carbon.ddl.base.hdfs.url. Now while dataload user can specify the csv path as /2016/xyz.csv. | | carbon.badRecords.location | (none) | CarbonData can detect the records not conforming to defined table schema and isolate them as bad records. This property is used to specify where to store such bad records. | | carbon.streaming.auto.handoff.enabled | true | CarbonData supports storing of streaming data. To have high throughput for streaming, the data is written in Row format which is highly optimized for write, but performs poorly for query. When this property is true and when the streaming data size reaches ***carbon.streaming.segment.max.size***, CabonData will automatically convert the data to columnar format and optimize it for faster querying.**NOTE:** It is not recommended to keep the default value which is true. | | carbon.streaming.segment.max.size | 102400 | CarbonData writes streaming data in row format which is optimized for high write throughput. This property defines the maximum size of data to be held is row format, beyond which it will be converted to columnar format in order to support high performance query, provided ***carbon.streaming.auto.handoff.enabled*** is true. **NOTE:** Setting higher value will impact the streaming ingestion. The value has to be configured in bytes. | -| carbon.query.show.datamaps | true | CarbonData stores datamaps as independent tables so as to allow independent maintenance to some extent. When this property is true,which is by default, show tables command will list all the tables including datatmaps(eg: Preaggregate table), else datamaps will be excluded from the table list.**NOTE:** It is generally not required for the user to do any maintenance operations on these tables and hence not required to be seen.But it is shown by default so tha
[5/8] carbondata-site git commit: Added 1.5.1 version information
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/ae77df2e/src/main/webapp/CSDK-guide.html
--
diff --git a/src/main/webapp/CSDK-guide.html b/src/main/webapp/CSDK-guide.html
index 8168aaf..73e1d67 100644
--- a/src/main/webapp/CSDK-guide.html
+++ b/src/main/webapp/CSDK-guide.html
@@ -52,6 +52,9 @@
aria-expanded="false"> Download
+https://dist.apache.org/repos/dist/release/carbondata/1.5.1/";
+ target="_blank">Apache CarbonData
1.5.1
+
https://dist.apache.org/repos/dist/release/carbondata/1.5.0/";
target="_blank">Apache CarbonData
1.5.0
@@ -64,9 +67,6 @@
https://dist.apache.org/repos/dist/release/carbondata/1.3.1/";
target="_blank">Apache CarbonData
1.3.1
-https://dist.apache.org/repos/dist/release/carbondata/1.3.0/";
- target="_blank">Apache CarbonData
1.3.0
-
https://cwiki.apache.org/confluence/display/CARBONDATA/Releases";
target="_blank">Release Archive
@@ -219,119 +219,28 @@
-CSDK Guide
-CarbonData CSDK provides C++ interface to write and read carbon file.
-CSDK use JNI to invoke java SDK in C++ code.
+C++ SDK Guide
+CarbonData C++ SDK provides C++ interface to write and read carbon file.
+C++ SDK use JNI to invoke java SDK in C++ code.
-CSDK Reader
-This CSDK reader reads CarbonData file and carbonindex file at a given path.
+C++ SDK Reader
+This C++ SDK reader reads CarbonData file and carbonindex file at a given
path.
External client can make use of this reader to read CarbonData files in C++
code and without CarbonSession.
In the carbon jars package, there exist a carbondata-sdk.jar,
-including SDK reader for CSDK.
+including SDK reader for C++ SDK.
Quick example
-// 1. init JVM
-JavaVM *jvm;
-JNIEnv *initJVM() {
-JNIEnv *env;
-JavaVMInitArgs vm_args;
-int parNum = 3;
-int res;
-JavaVMOption options[parNum];
-
-options[0].optionString = "-Djava.compiler=NONE";
-options[1].optionString =
"-Djava.class.path=../../sdk/target/carbondata-sdk.jar";
-options[2].optionString = "-verbose:jni";
-vm_args.version = JNI_VERSION_1_8;
-vm_args.nOptions = parNum;
-vm_args.options = options;
-vm_args.ignoreUnrecognized = JNI_FALSE;
-
-res = JNI_CreateJavaVM(&jvm, (void **) &env, &vm_args);
-if (res < 0) {
-fprintf(stderr, "\nCan't create Java VM\n");
-exit(1);
-}
-
-return env;
-}
-
-// 2. create carbon reader and read data
-// 2.1 read data from local disk
-/**
- * test read data from local disk, without projection
- *
- * @param env jni env
- * @return
- */
-bool readFromLocalWithoutProjection(JNIEnv *env) {
-
-CarbonReader carbonReaderClass;
-carbonReaderClass.builder(env, "../resources/carbondata", "test");
-carbonReaderClass.build();
-
-while (carbonReaderClass.hasNext()) {
-jobjectArray row = carbonReaderClass.readNextRow();
-jsize length = env->GetArrayLength(row);
-int j = 0;
-for (j = 0; j < length; j++) {
-jobject element = env->GetObjectArrayElement(row, j);
-char *str = (char *) env->GetStringUTFChars((jstring) element,
JNI_FALSE);
-printf("%s\t", str);
-}
-printf("\n");
-}
-carbonReaderClass.close();
-}
-
-// 2.2 read data from S3
-
-/**
- * read data from S3
- * parameter is ak sk endpoint
- *
- * @param env jni env
- * @param argv argument vector
- * @return
- */
-bool readFromS3(JNIEnv *env, char *argv[]) {
-CarbonReader reader;
-
-char *args[3];
-// "your access key"
-args[0] = argv[1];
-// "your secret key"
-args[1] = argv[2];
-// "your endPoint"
-args[2] = argv[3];
-
-reader.builder(env, "s3a://sdk/WriterOutput", "test");
-reader.withHadoopConf(3, args);
-reader.build();
-printf("\nRead data from S3:\n");
-while (reader.hasNext()) {
-jobjectArray row = reader.readNextRow();
-jsize length = env->GetArrayLength(row);
-
-int j = 0;
-for (j = 0; j < length; j++) {
-jobject element = env->GetObjectArrayElement(row, j);
-char *str = (char *) env->GetStringUTFChars((jstring) element,
JNI_FALSE);
-printf("%s\t", str);
-}
-printf("\n");
-}
-
-reader.close();
-}
-
-// 3. destory JVM
-(jvm)->DestroyJavaVM();
-
-Find example code at main.cpp of CSDK module
+P
[02/21] carbondata git commit: [CARBONDATA-3143] Fixed local dictionary in presto
[CARBONDATA-3143] Fixed local dictionary in presto
Problem:
Currently, local dictionary columns are not working for presto as it is not
handled in the integration layer.
Solution:
Add local dictionary support to presto integration layer.
This closes #2972
Project: http://git-wip-us.apache.org/repos/asf/carbondata/repo
Commit: http://git-wip-us.apache.org/repos/asf/carbondata/commit/d909482e
Tree: http://git-wip-us.apache.org/repos/asf/carbondata/tree/d909482e
Diff: http://git-wip-us.apache.org/repos/asf/carbondata/diff/d909482e
Branch: refs/heads/branch-1.5
Commit: d909482e6d6d178fceaac0c67c064a1f19f870bc
Parents: 3524f51
Author: ravipesala
Authored: Mon Dec 3 18:27:33 2018 +0530
Committer: Raghunandan S
Committed: Mon Dec 17 18:50:23 2018 +0530
--
.../presto/CarbonColumnVectorWrapper.java | 2 +-
.../presto/readers/SliceStreamReader.java | 35 +++
.../PrestoAllDataTypeLocalDictTest.scala| 291 +++
.../integrationtest/PrestoAllDataTypeTest.scala | 2 +-
.../carbondata/presto/server/PrestoServer.scala | 4 +-
.../presto/util/CarbonDataStoreCreator.scala| 18 +-
6 files changed, 342 insertions(+), 10 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/carbondata/blob/d909482e/integration/presto/src/main/java/org/apache/carbondata/presto/CarbonColumnVectorWrapper.java
--
diff --git
a/integration/presto/src/main/java/org/apache/carbondata/presto/CarbonColumnVectorWrapper.java
b/integration/presto/src/main/java/org/apache/carbondata/presto/CarbonColumnVectorWrapper.java
index a80751f..f001488 100644
---
a/integration/presto/src/main/java/org/apache/carbondata/presto/CarbonColumnVectorWrapper.java
+++
b/integration/presto/src/main/java/org/apache/carbondata/presto/CarbonColumnVectorWrapper.java
@@ -244,7 +244,7 @@ public class CarbonColumnVectorWrapper implements
CarbonColumnVector {
}
@Override public CarbonColumnVector getDictionaryVector() {
-return this.columnVector;
+return this.columnVector.getDictionaryVector();
}
@Override public void putFloats(int rowId, int count, float[] src, int
srcIndex) {
http://git-wip-us.apache.org/repos/asf/carbondata/blob/d909482e/integration/presto/src/main/java/org/apache/carbondata/presto/readers/SliceStreamReader.java
--
diff --git
a/integration/presto/src/main/java/org/apache/carbondata/presto/readers/SliceStreamReader.java
b/integration/presto/src/main/java/org/apache/carbondata/presto/readers/SliceStreamReader.java
index ab270fc..04e5bb3 100644
---
a/integration/presto/src/main/java/org/apache/carbondata/presto/readers/SliceStreamReader.java
+++
b/integration/presto/src/main/java/org/apache/carbondata/presto/readers/SliceStreamReader.java
@@ -17,14 +17,19 @@
package org.apache.carbondata.presto.readers;
+import java.util.Optional;
+
import org.apache.carbondata.core.metadata.datatype.DataType;
+import org.apache.carbondata.core.scan.result.vector.CarbonDictionary;
import
org.apache.carbondata.core.scan.result.vector.impl.CarbonColumnVectorImpl;
import com.facebook.presto.spi.block.Block;
import com.facebook.presto.spi.block.BlockBuilder;
import com.facebook.presto.spi.block.DictionaryBlock;
+import com.facebook.presto.spi.block.VariableWidthBlock;
import com.facebook.presto.spi.type.Type;
import com.facebook.presto.spi.type.VarcharType;
+import io.airlift.slice.Slices;
import static io.airlift.slice.Slices.wrappedBuffer;
@@ -63,6 +68,36 @@ public class SliceStreamReader extends
CarbonColumnVectorImpl implements PrestoV
}
}
+ @Override public void setDictionary(CarbonDictionary dictionary) {
+super.setDictionary(dictionary);
+if (dictionary == null) {
+ dictionaryBlock = null;
+ return;
+}
+boolean[] nulls = new boolean[dictionary.getDictionarySize()];
+nulls[0] = true;
+nulls[1] = true;
+int[] dictOffsets = new int[dictionary.getDictionarySize() + 1];
+int size = 0;
+for (int i = 0; i < dictionary.getDictionarySize(); i++) {
+ if (dictionary.getDictionaryValue(i) != null) {
+dictOffsets[i] = size;
+size += dictionary.getDictionaryValue(i).length;
+ }
+}
+byte[] singleArrayDictValues = new byte[size];
+for (int i = 0; i < dictionary.getDictionarySize(); i++) {
+ if (dictionary.getDictionaryValue(i) != null) {
+System.arraycopy(dictionary.getDictionaryValue(i), 0,
singleArrayDictValues, dictOffsets[i],
+dictionary.getDictionaryValue(i).length);
+ }
+}
+dictOffsets[dictOffsets.length - 1] = size;
+dictionaryBlock = new VariableWidthBlock(dictionary.getDictionarySize(),
+Slices.wrappedBuffer(singleArrayDictValues), dictOffsets,
Optio
[12/21] carbondata git commit: [CARBONDATA-2563][CATALYST] Explain query with Order by operator is fired Spark Job which is increase explain query time even though isQueryStatisticsEnabled is false
[CARBONDATA-2563][CATALYST] Explain query with Order by operator is fired Spark
Job
which is increase explain query time even though isQueryStatisticsEnabled is
false
Even though isQueryStatisticsEnabled is false which means user doesnt wants to
see
statistics for explain command, still the engine tries to fetch the paritions
information
which causes a job execution in case of order by query, this is mainly because
spark engine does sampling for defifning certain range within paritions for
sorting process.
As part of solution the explain command process shall fetch the parition info
only if isQueryStatisticsEnabled true.
This closes #2974
Project: http://git-wip-us.apache.org/repos/asf/carbondata/repo
Commit: http://git-wip-us.apache.org/repos/asf/carbondata/commit/21762c54
Tree: http://git-wip-us.apache.org/repos/asf/carbondata/tree/21762c54
Diff: http://git-wip-us.apache.org/repos/asf/carbondata/diff/21762c54
Branch: refs/heads/branch-1.5
Commit: 21762c541df334816c660eb8e4e26aca1df9aa71
Parents: 93268ea
Author: s71955
Authored: Tue Dec 4 20:48:55 2018 +0530
Committer: Raghunandan S
Committed: Mon Dec 17 18:58:33 2018 +0530
--
.../carbondata/core/profiler/ExplainCollector.java | 6 +-
.../execution/command/table/CarbonExplainCommand.scala | 11 ---
2 files changed, 13 insertions(+), 4 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/carbondata/blob/21762c54/core/src/main/java/org/apache/carbondata/core/profiler/ExplainCollector.java
--
diff --git
a/core/src/main/java/org/apache/carbondata/core/profiler/ExplainCollector.java
b/core/src/main/java/org/apache/carbondata/core/profiler/ExplainCollector.java
index 8513dac..75568e4 100644
---
a/core/src/main/java/org/apache/carbondata/core/profiler/ExplainCollector.java
+++
b/core/src/main/java/org/apache/carbondata/core/profiler/ExplainCollector.java
@@ -171,7 +171,11 @@ public class ExplainCollector {
}
public static String getFormatedOutput() {
-return get().toString();
+if (null != get()) {
+ return get().toString();
+} else {
+ return null;
+}
}
@Override
http://git-wip-us.apache.org/repos/asf/carbondata/blob/21762c54/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/table/CarbonExplainCommand.scala
--
diff --git
a/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/table/CarbonExplainCommand.scala
b/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/table/CarbonExplainCommand.scala
index 8939c6a..aa7a541 100644
---
a/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/table/CarbonExplainCommand.scala
+++
b/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/table/CarbonExplainCommand.scala
@@ -47,12 +47,17 @@ case class CarbonExplainCommand(
}
private def collectProfiler(sparkSession: SparkSession): Seq[Row] = {
-val queryExecution =
-
sparkSession.sessionState.executePlan(child.asInstanceOf[ExplainCommand].logicalPlan)
try {
ExplainCollector.setup()
- queryExecution.toRdd.partitions
if (ExplainCollector.enabled()) {
+val queryExecution =
+
sparkSession.sessionState.executePlan(child.asInstanceOf[ExplainCommand].logicalPlan)
+queryExecution.toRdd.partitions
+// For count(*) queries the explain collector will be disabled, so
profiler
+// informations not required in such scenarios.
+if (null == ExplainCollector.getFormatedOutput) {
+ Seq.empty
+}
Seq(Row("== CarbonData Profiler ==\n" +
ExplainCollector.getFormatedOutput))
} else {
Seq.empty
[01/21] carbondata git commit: [CARBONDATA-3145] Avoid duplicate decoding for complex column pages while querying
Repository: carbondata
Updated Branches:
refs/heads/branch-1.5 442e2446e -> 951a7811a
[CARBONDATA-3145] Avoid duplicate decoding for complex column pages while
querying
Problem:
Column page is decoded for getting each row of a complex primitive column.
Solution:
Decode a page it once then use the same.
This closes #2975
Project: http://git-wip-us.apache.org/repos/asf/carbondata/repo
Commit: http://git-wip-us.apache.org/repos/asf/carbondata/commit/523515cd
Tree: http://git-wip-us.apache.org/repos/asf/carbondata/tree/523515cd
Diff: http://git-wip-us.apache.org/repos/asf/carbondata/diff/523515cd
Branch: refs/heads/branch-1.5
Commit: 523515cd6700f0c19d43de126525fd3c8dede0b9
Parents: d909482
Author: dhatchayani
Authored: Wed Dec 5 12:40:56 2018 +0530
Committer: Raghunandan S
Committed: Mon Dec 17 18:50:23 2018 +0530
--
.../core/scan/complextypes/ArrayQueryType.java | 11 ++--
.../scan/complextypes/ComplexQueryType.java | 14 +++-
.../scan/complextypes/PrimitiveQueryType.java | 11 ++--
.../core/scan/complextypes/StructQueryType.java | 14 ++--
.../core/scan/filter/GenericQueryType.java | 4 +-
.../executer/RowLevelFilterExecuterImpl.java| 7 +-
.../core/scan/result/BlockletScannedResult.java | 68 +---
7 files changed, 86 insertions(+), 43 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/carbondata/blob/523515cd/core/src/main/java/org/apache/carbondata/core/scan/complextypes/ArrayQueryType.java
--
diff --git
a/core/src/main/java/org/apache/carbondata/core/scan/complextypes/ArrayQueryType.java
b/core/src/main/java/org/apache/carbondata/core/scan/complextypes/ArrayQueryType.java
index a5f4234..8538edb 100644
---
a/core/src/main/java/org/apache/carbondata/core/scan/complextypes/ArrayQueryType.java
+++
b/core/src/main/java/org/apache/carbondata/core/scan/complextypes/ArrayQueryType.java
@@ -22,6 +22,7 @@ import java.io.IOException;
import java.nio.ByteBuffer;
import java.util.Map;
+import org.apache.carbondata.core.datastore.chunk.DimensionColumnPage;
import org.apache.carbondata.core.datastore.chunk.impl.DimensionRawColumnChunk;
import org.apache.carbondata.core.metadata.schema.table.column.CarbonDimension;
import org.apache.carbondata.core.scan.filter.GenericQueryType;
@@ -62,17 +63,17 @@ public class ArrayQueryType extends ComplexQueryType
implements GenericQueryType
}
public void
parseBlocksAndReturnComplexColumnByteArray(DimensionRawColumnChunk[]
rawColumnChunks,
- int rowNumber, int pageNumber, DataOutputStream dataOutputStream) throws
IOException {
-byte[] input = copyBlockDataChunk(rawColumnChunks, rowNumber, pageNumber);
+ DimensionColumnPage[][] dimensionColumnPages, int rowNumber, int
pageNumber,
+ DataOutputStream dataOutputStream) throws IOException {
+byte[] input = copyBlockDataChunk(rawColumnChunks, dimensionColumnPages,
rowNumber, pageNumber);
ByteBuffer byteArray = ByteBuffer.wrap(input);
int dataLength = byteArray.getInt();
dataOutputStream.writeInt(dataLength);
if (dataLength > 0) {
int dataOffset = byteArray.getInt();
for (int i = 0; i < dataLength; i++) {
-children
-.parseBlocksAndReturnComplexColumnByteArray(rawColumnChunks,
dataOffset++, pageNumber,
-dataOutputStream);
+children.parseBlocksAndReturnComplexColumnByteArray(rawColumnChunks,
dimensionColumnPages,
+dataOffset++, pageNumber, dataOutputStream);
}
}
}
http://git-wip-us.apache.org/repos/asf/carbondata/blob/523515cd/core/src/main/java/org/apache/carbondata/core/scan/complextypes/ComplexQueryType.java
--
diff --git
a/core/src/main/java/org/apache/carbondata/core/scan/complextypes/ComplexQueryType.java
b/core/src/main/java/org/apache/carbondata/core/scan/complextypes/ComplexQueryType.java
index 98f0715..704af89 100644
---
a/core/src/main/java/org/apache/carbondata/core/scan/complextypes/ComplexQueryType.java
+++
b/core/src/main/java/org/apache/carbondata/core/scan/complextypes/ComplexQueryType.java
@@ -19,6 +19,7 @@ package org.apache.carbondata.core.scan.complextypes;
import java.io.IOException;
+import org.apache.carbondata.core.datastore.chunk.DimensionColumnPage;
import org.apache.carbondata.core.datastore.chunk.impl.DimensionRawColumnChunk;
import org.apache.carbondata.core.scan.processor.RawBlockletColumnChunks;
@@ -40,9 +41,10 @@ public class ComplexQueryType {
* This method is also used by child.
*/
protected byte[] copyBlockDataChunk(DimensionRawColumnChunk[]
rawColumnChunks,
- int rowNumber, int pageNumber) {
+ DimensionColumnPage[][] dimensionColumnPages, int rowNumber, int
pageNumber) {
by
[14/21] carbondata git commit: [CARBONDATA-3017] Map DDL Support
[CARBONDATA-3017] Map DDL Support
Support Create DDL for Map type.
This closes #2980
Project: http://git-wip-us.apache.org/repos/asf/carbondata/repo
Commit: http://git-wip-us.apache.org/repos/asf/carbondata/commit/c9b136c2
Tree: http://git-wip-us.apache.org/repos/asf/carbondata/tree/c9b136c2
Diff: http://git-wip-us.apache.org/repos/asf/carbondata/diff/c9b136c2
Branch: refs/heads/branch-1.5
Commit: c9b136c26763baad9bb8f81aa3676217578d6ccc
Parents: 8935164
Author: manishnalla1994
Authored: Tue Oct 16 15:18:08 2018 +0530
Committer: Raghunandan S
Committed: Mon Dec 17 18:58:34 2018 +0530
--
.../apache/carbondata/core/util/CarbonUtil.java | 4 +
.../hadoop/api/CarbonTableOutputFormat.java | 17 +-
.../TestCreateDDLForComplexMapType.scala| 445 +++
.../LocalDictionarySupportLoadTableTest.scala | 17 +
.../spark/rdd/NewCarbonDataLoadRDD.scala| 10 +-
.../carbondata/spark/util/CarbonScalaUtil.scala | 1 +
.../spark/sql/catalyst/CarbonDDLSqlParser.scala | 43 +-
.../streaming/CarbonAppendableStreamSink.scala | 9 +-
.../spark/rdd/CarbonDataRDDFactory.scala| 4 +-
.../CarbonAlterTableCompactionCommand.scala | 6 +
.../management/CarbonLoadDataCommand.scala | 6 +-
.../table/CarbonCreateTableCommand.scala| 2 +-
.../spark/util/AllDictionaryTestCase.scala | 4 +-
.../util/ExternalColumnDictionaryTestCase.scala | 4 +-
.../TestStreamingTableWithRowParser.scala | 3 +-
.../loading/ComplexDelimitersEnum.java | 39 ++
.../loading/DataLoadProcessBuilder.java | 7 +-
.../loading/model/CarbonLoadModel.java | 38 +-
.../loading/model/CarbonLoadModelBuilder.java | 11 +-
.../processing/loading/model/LoadOption.java| 17 +-
.../loading/parser/CarbonParserFactory.java | 25 +-
.../loading/parser/impl/ArrayParserImpl.java| 6 +-
.../loading/parser/impl/MapParserImpl.java | 60 +++
.../loading/parser/impl/RowParserImpl.java | 8 +-
.../sdk/file/CarbonWriterBuilder.java | 1 +
.../streaming/parser/RowStreamParserImp.scala | 2 +
26 files changed, 715 insertions(+), 74 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/carbondata/blob/c9b136c2/core/src/main/java/org/apache/carbondata/core/util/CarbonUtil.java
--
diff --git a/core/src/main/java/org/apache/carbondata/core/util/CarbonUtil.java
b/core/src/main/java/org/apache/carbondata/core/util/CarbonUtil.java
index ac52728..fc4704e 100644
--- a/core/src/main/java/org/apache/carbondata/core/util/CarbonUtil.java
+++ b/core/src/main/java/org/apache/carbondata/core/util/CarbonUtil.java
@@ -597,6 +597,10 @@ public final class CarbonUtil {
*/
public static String delimiterConverter(String delimiter) {
switch (delimiter) {
+ case "\\001":
+ case "\\002":
+ case "\\003":
+ case "\\004":
case "|":
case "*":
case ".":
http://git-wip-us.apache.org/repos/asf/carbondata/blob/c9b136c2/hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonTableOutputFormat.java
--
diff --git
a/hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonTableOutputFormat.java
b/hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonTableOutputFormat.java
index dbd2f0e..16486d0 100644
---
a/hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonTableOutputFormat.java
+++
b/hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonTableOutputFormat.java
@@ -37,6 +37,7 @@ import org.apache.carbondata.core.util.CarbonThreadFactory;
import org.apache.carbondata.core.util.ObjectSerializationUtil;
import org.apache.carbondata.core.util.ThreadLocalSessionInfo;
import org.apache.carbondata.hadoop.internal.ObjectArrayWritable;
+import org.apache.carbondata.processing.loading.ComplexDelimitersEnum;
import org.apache.carbondata.processing.loading.DataLoadExecutor;
import org.apache.carbondata.processing.loading.TableProcessingOperations;
import
org.apache.carbondata.processing.loading.iterator.CarbonOutputIteratorWrapper;
@@ -338,11 +339,19 @@ public class CarbonTableOutputFormat extends
FileOutputFormat 1) {
- model.setComplexDelimiterLevel2(split[1]);
+model.setComplexDelimiter(split[0]);
+if (split.length > 2) {
+ model.setComplexDelimiter(split[1]);
+ model.setComplexDelimiter(split[2]);
+} else if (split.length > 1) {
+ model.setComplexDelimiter(split[1]);
}
model.setDateFormat(
conf.get(
http://git-wip-us.apache.org/repos/asf/carbondata/blob/c9b136c2/integration/spark-common-test/src/test/scala/org/apache
[11/21] carbondata git commit: [CARBONDATA-2755][Complex DataType Enhancements] Compaction Complex Types (STRUCT AND ARRAY)
[CARBONDATA-2755][Complex DataType Enhancements] Compaction Complex Types
(STRUCT AND ARRAY)
Cherry-Picked from PR #2516
This closes #2976
Project: http://git-wip-us.apache.org/repos/asf/carbondata/repo
Commit: http://git-wip-us.apache.org/repos/asf/carbondata/commit/e8d25bb3
Tree: http://git-wip-us.apache.org/repos/asf/carbondata/tree/e8d25bb3
Diff: http://git-wip-us.apache.org/repos/asf/carbondata/diff/e8d25bb3
Branch: refs/heads/branch-1.5
Commit: e8d25bb3b80a3deda13bc12773e8565c33747131
Parents: 21762c5
Author: sounakr
Authored: Tue Jul 17 10:35:32 2018 +0530
Committer: Raghunandan S
Committed: Mon Dec 17 18:58:33 2018 +0530
--
.../core/datastore/row/WriteStepRowUtil.java| 23 +-
.../core/scan/wrappers/ByteArrayWrapper.java| 10 +
.../complexType/TestCompactionComplexType.scala | 992 +++
.../complexType/TestComplexDataType.scala | 32 -
.../CarbonAlterTableCompactionCommand.scala | 7 -
.../processing/datatypes/PrimitiveDataType.java | 10 +-
.../converter/impl/FieldEncoderFactory.java | 2 +-
.../processing/merger/CarbonCompactionUtil.java | 33 +
.../store/CarbonFactDataHandlerModel.java | 89 +-
.../util/CarbonDataProcessorUtil.java | 7 +-
10 files changed, 1142 insertions(+), 63 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/carbondata/blob/e8d25bb3/core/src/main/java/org/apache/carbondata/core/datastore/row/WriteStepRowUtil.java
--
diff --git
a/core/src/main/java/org/apache/carbondata/core/datastore/row/WriteStepRowUtil.java
b/core/src/main/java/org/apache/carbondata/core/datastore/row/WriteStepRowUtil.java
index fe4e10e..49716ac 100644
---
a/core/src/main/java/org/apache/carbondata/core/datastore/row/WriteStepRowUtil.java
+++
b/core/src/main/java/org/apache/carbondata/core/datastore/row/WriteStepRowUtil.java
@@ -59,29 +59,40 @@ public class WriteStepRowUtil {
}
converted[DICTIONARY_DIMENSION] = dictDimensions;
+Object[] noDictAndComplexKeys =
+new Object[segmentProperties.getNumberOfNoDictionaryDimension() +
segmentProperties
+.getComplexDimensions().size()];
+
byte[][] noDictionaryKeys = ((ByteArrayWrapper)
row[0]).getNoDictionaryKeys();
-Object[] noDictKeys = new Object[noDictionaryKeys.length];
for (int i = 0; i < noDictionaryKeys.length; i++) {
// in case of compaction rows are collected from result collector and
are in byte[].
// Convert the no dictionary columns to original data,
// as load expects the no dictionary column with original data.
if
(DataTypeUtil.isPrimitiveColumn(noDicAndComplexColumns[i].getDataType())) {
-noDictKeys[i] = DataTypeUtil
+noDictAndComplexKeys[i] = DataTypeUtil
.getDataBasedOnDataTypeForNoDictionaryColumn(noDictionaryKeys[i],
noDicAndComplexColumns[i].getDataType());
// for timestamp the above method will give the original data, so it
should be
// converted again to the format to be loaded (without micros)
-if (null != noDictKeys[i]
+if (null != noDictAndComplexKeys[i]
&& noDicAndComplexColumns[i].getDataType() == DataTypes.TIMESTAMP)
{
- noDictKeys[i] = (long) noDictKeys[i] / 1000L;
+ noDictAndComplexKeys[i] = (long) noDictAndComplexKeys[i] / 1000L;
}
} else {
-noDictKeys[i] = noDictionaryKeys[i];
+noDictAndComplexKeys[i] = noDictionaryKeys[i];
}
}
+// For Complex Type Columns
+byte[][] complexKeys = ((ByteArrayWrapper) row[0]).getComplexTypesKeys();
+for (int i = segmentProperties.getNumberOfNoDictionaryDimension();
+ i < segmentProperties.getNumberOfNoDictionaryDimension() +
segmentProperties
+ .getComplexDimensions().size(); i++) {
+ noDictAndComplexKeys[i] = complexKeys[i];
+}
+
// no dictionary and complex dimension
-converted[NO_DICTIONARY_AND_COMPLEX] = noDictKeys;
+converted[NO_DICTIONARY_AND_COMPLEX] = noDictAndComplexKeys;
// measure
int measureCount = row.length - 1;
http://git-wip-us.apache.org/repos/asf/carbondata/blob/e8d25bb3/core/src/main/java/org/apache/carbondata/core/scan/wrappers/ByteArrayWrapper.java
--
diff --git
a/core/src/main/java/org/apache/carbondata/core/scan/wrappers/ByteArrayWrapper.java
b/core/src/main/java/org/apache/carbondata/core/scan/wrappers/ByteArrayWrapper.java
index 4cce8af..65f29d4 100644
---
a/core/src/main/java/org/apache/carbondata/core/scan/wrappers/ByteArrayWrapper.java
+++
b/core/src/main/java/org/apache/carbondata/core/scan/wrappers/ByteArrayWrapper.java
@@ -90,6 +90,16 @@ public class ByteArrayWrapper implements
Comparable, Seriali
[09/21] carbondata git commit: [CARBONDATA-3157] Added lazy load and direct vector fill support to Presto
[CARBONDATA-3157] Added lazy load and direct vector fill support to Presto
To improve the scan performance, integrate lazy loading and direct fil vector
features to Carbon Presto Integration.
This PR also fixes the query fail in case of multiple table join and filters
This closes #2978
Project: http://git-wip-us.apache.org/repos/asf/carbondata/repo
Commit: http://git-wip-us.apache.org/repos/asf/carbondata/commit/93268eaf
Tree: http://git-wip-us.apache.org/repos/asf/carbondata/tree/93268eaf
Diff: http://git-wip-us.apache.org/repos/asf/carbondata/diff/93268eaf
Branch: refs/heads/branch-1.5
Commit: 93268eaf02c504d8058a4ec24bf27fbdd9605e43
Parents: 26a5850
Author: ravipesala
Authored: Wed Dec 5 18:34:13 2018 +0530
Committer: Raghunandan S
Committed: Mon Dec 17 18:58:33 2018 +0530
--
.../safe/AbstractNonDictionaryVectorFiller.java | 43 ++-
.../adaptive/AdaptiveDeltaFloatingCodec.java| 4 +-
.../adaptive/AdaptiveDeltaIntegralCodec.java| 4 +-
.../adaptive/AdaptiveFloatingCodec.java | 4 +-
.../adaptive/AdaptiveIntegralCodec.java | 4 +-
.../encoding/compress/DirectCompressCodec.java | 4 +-
.../RestructureBasedVectorResultCollector.java | 3 +-
.../scan/result/vector/CarbonColumnVector.java | 2 +-
.../vector/impl/CarbonColumnVectorImpl.java | 24 +-
.../AbstractCarbonColumnarVector.java | 2 +-
.../ColumnarVectorWrapperDirectFactory.java | 22 +-
.../vector/impl/directread/SequentialFill.java | 38 +++
.../presto/CarbonColumnVectorWrapper.java | 2 +-
.../carbondata/presto/CarbonVectorBatch.java| 10 +-
.../carbondata/presto/CarbondataPageSource.java | 1 +
.../presto/CarbondataPageSourceProvider.java| 5 +
.../presto/ColumnarVectorWrapperDirect.java | 310 +++
.../PrestoCarbonVectorizedRecordReader.java | 9 +-
.../presto/impl/CarbonTableConfig.java | 10 +
.../presto/impl/CarbonTableReader.java | 2 +-
.../presto/readers/BooleanStreamReader.java | 10 +
.../presto/readers/SliceStreamReader.java | 2 +-
.../vectorreader/ColumnarVectorWrapper.java | 2 +-
.../ColumnarVectorWrapperDirect.java| 2 +-
24 files changed, 482 insertions(+), 37 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/carbondata/blob/93268eaf/core/src/main/java/org/apache/carbondata/core/datastore/chunk/store/impl/safe/AbstractNonDictionaryVectorFiller.java
--
diff --git
a/core/src/main/java/org/apache/carbondata/core/datastore/chunk/store/impl/safe/AbstractNonDictionaryVectorFiller.java
b/core/src/main/java/org/apache/carbondata/core/datastore/chunk/store/impl/safe/AbstractNonDictionaryVectorFiller.java
index 23ac4a9..38e28ae 100644
---
a/core/src/main/java/org/apache/carbondata/core/datastore/chunk/store/impl/safe/AbstractNonDictionaryVectorFiller.java
+++
b/core/src/main/java/org/apache/carbondata/core/datastore/chunk/store/impl/safe/AbstractNonDictionaryVectorFiller.java
@@ -24,6 +24,7 @@ import org.apache.carbondata.core.metadata.datatype.DataType;
import org.apache.carbondata.core.metadata.datatype.DataTypes;
import org.apache.carbondata.core.scan.result.vector.CarbonColumnVector;
import
org.apache.carbondata.core.scan.result.vector.impl.directread.ColumnarVectorWrapperDirectWithInvertedIndex;
+import
org.apache.carbondata.core.scan.result.vector.impl.directread.SequentialFill;
import org.apache.carbondata.core.util.ByteUtil;
import org.apache.carbondata.core.util.DataTypeUtil;
@@ -83,20 +84,39 @@ class StringVectorFiller extends
AbstractNonDictionaryVectorFiller {
@Override
public void fillVector(byte[] data, CarbonColumnVector vector) {
// start position will be used to store the current data position
+boolean addSequential = vector instanceof
ColumnarVectorWrapperDirectWithInvertedIndex
+|| vector instanceof SequentialFill;
+
int localOffset = 0;
ByteUtil.UnsafeComparer comparator = ByteUtil.UnsafeComparer.INSTANCE;
-for (int i = 0; i < numberOfRows; i++) {
- int length = (((data[localOffset] & 0xFF) << 8) | (data[localOffset + 1]
& 0xFF));
- localOffset += 2;
- if (comparator.equals(CarbonCommonConstants.MEMBER_DEFAULT_VAL_ARRAY, 0,
- CarbonCommonConstants.MEMBER_DEFAULT_VAL_ARRAY.length, data,
localOffset, length)) {
-vector.putNull(i);
- } else {
-vector.putArray(i, localOffset, length);
+// In case of inverted index and sequential fill, add data to vector
sequentially instead of
+// adding offsets and data separately.
+if (addSequential) {
+ for (int i = 0; i < numberOfRows; i++) {
+int length = (((data[localOffset] & 0xFF) << 8) | (data[localOffset +
1] & 0xFF));
+localOffset +=
[16/21] carbondata git commit: [CARBONDATA-3002] Fix some spell error
[CARBONDATA-3002] Fix some spell error
Fix some spell error
This closes #2890
Project: http://git-wip-us.apache.org/repos/asf/carbondata/repo
Commit: http://git-wip-us.apache.org/repos/asf/carbondata/commit/313ba0ea
Tree: http://git-wip-us.apache.org/repos/asf/carbondata/tree/313ba0ea
Diff: http://git-wip-us.apache.org/repos/asf/carbondata/diff/313ba0ea
Branch: refs/heads/branch-1.5
Commit: 313ba0ea63f90f34f1b27f8940030c2999bd818c
Parents: c9b136c
Author: xubo245
Authored: Thu Nov 1 11:16:43 2018 +0800
Committer: Raghunandan S
Committed: Mon Dec 17 18:58:34 2018 +0530
--
.../carbondata/core/datastore/FileReader.java | 6 ++--
.../core/datastore/impl/FileReaderImpl.java | 8 ++---
.../carbondata/core/locks/CarbonLockUtil.java | 4 +--
.../AbstractDetailQueryResultIterator.java | 8 ++---
.../carbondata/core/util/CarbonProperties.java | 8 ++---
.../sdv/register/TestRegisterCarbonTable.scala | 20 +--
...ithColumnMetCacheAndCacheLevelProperty.scala | 4 +--
.../createTable/TestCreateTableAsSelect.scala | 2 +-
.../DBLocationCarbonTableTestCase.scala | 4 +--
.../iud/DeleteCarbonTableTestCase.scala | 6 ++--
.../StandardPartitionTableLoadingTestCase.scala | 2 +-
.../apache/spark/rdd/CarbonMergeFilesRDD.scala | 2 +-
.../spark/rdd/CarbonDataRDDFactory.scala| 4 +--
.../sql/CarbonDatasourceHadoopRelation.scala| 2 +-
.../scala/org/apache/spark/sql/CarbonEnv.scala | 12 +++
.../org/apache/spark/sql/CarbonSource.scala | 8 ++---
.../datamap/CarbonDropDataMapCommand.scala | 2 +-
.../CarbonAlterTableCompactionCommand.scala | 2 +-
.../management/CarbonLoadDataCommand.scala | 2 +-
.../management/RefreshCarbonTableCommand.scala | 2 +-
.../spark/sql/execution/command/package.scala | 2 +-
.../CarbonAlterTableDropPartitionCommand.scala | 2 +-
.../CarbonAlterTableSplitPartitionCommand.scala | 2 +-
.../CarbonShowCarbonPartitionsCommand.scala | 2 +-
.../preaaggregate/PreAggregateUtil.scala| 8 ++---
.../CarbonAlterTableAddColumnCommand.scala | 2 +-
.../CarbonAlterTableDataTypeChangeCommand.scala | 2 +-
.../CarbonAlterTableDropColumnCommand.scala | 2 +-
.../schema/CarbonAlterTableRenameCommand.scala | 2 +-
.../CarbonCreateTableAsSelectCommand.scala | 2 +-
.../table/CarbonCreateTableCommand.scala| 4 +--
.../table/CarbonDescribeFormattedCommand.scala | 2 +-
.../command/table/CarbonDropTableCommand.scala | 2 +-
.../sql/execution/strategy/DDLStrategy.scala| 36 ++--
.../spark/sql/hive/CarbonFileMetastore.scala| 2 +-
.../sql/hive/CarbonPreAggregateRules.scala | 4 +--
.../sql/parser/CarbonSparkSqlParserUtil.scala | 2 +-
.../org/apache/spark/util/AlterTableUtil.scala | 16 -
.../org/apache/spark/util/CleanFiles.scala | 2 +-
.../org/apache/spark/util/Compaction.scala | 2 +-
.../apache/spark/util/DeleteSegmentByDate.scala | 2 +-
.../apache/spark/util/DeleteSegmentById.scala | 2 +-
.../org/apache/spark/util/TableAPIUtil.scala| 2 +-
.../org/apache/spark/util/TableLoader.scala | 2 +-
.../spark/sql/hive/CarbonSessionState.scala | 4 +--
.../spark/util/AllDictionaryTestCase.scala | 2 +-
.../spark/util/DictionaryLRUCacheTestCase.scala | 2 +-
.../util/ExternalColumnDictionaryTestCase.scala | 2 +-
.../carbondata/TestStreamingTableOpName.scala | 26 +++---
.../TestStreamingTableWithRowParser.scala | 2 +-
.../register/TestRegisterCarbonTable.scala | 22 ++--
.../loading/CarbonDataLoadConfiguration.java| 2 +-
.../loading/model/CarbonLoadModel.java | 2 +-
.../processing/util/CarbonLoaderUtil.java | 14
54 files changed, 146 insertions(+), 146 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/carbondata/blob/313ba0ea/core/src/main/java/org/apache/carbondata/core/datastore/FileReader.java
--
diff --git
a/core/src/main/java/org/apache/carbondata/core/datastore/FileReader.java
b/core/src/main/java/org/apache/carbondata/core/datastore/FileReader.java
index df0d745..2527f1d 100644
--- a/core/src/main/java/org/apache/carbondata/core/datastore/FileReader.java
+++ b/core/src/main/java/org/apache/carbondata/core/datastore/FileReader.java
@@ -56,7 +56,7 @@ public interface FileReader {
/**
* This method will be used to read int from file from postion(offset), here
- * length will be always 4 bacause int byte size if 4
+ * length will be always 4 because int byte size if 4
*
* @param filePath fully qualified file path
* @param offset reading start position,
@@ -66,7 +66,7 @@ public interface FileReader {
/**
* This method will be used to read long from file from postion(offset), here
[20/21] carbondata git commit: [CARBONDATA-3175]Fix Testcase failures in complex delimiters
[CARBONDATA-3175]Fix Testcase failures in complex delimiters
Changed the complex delimiter in testcases
This closes #2989
Project: http://git-wip-us.apache.org/repos/asf/carbondata/repo
Commit: http://git-wip-us.apache.org/repos/asf/carbondata/commit/951a7811
Tree: http://git-wip-us.apache.org/repos/asf/carbondata/tree/951a7811
Diff: http://git-wip-us.apache.org/repos/asf/carbondata/diff/951a7811
Branch: refs/heads/branch-1.5
Commit: 951a7811a416ea4c000a804d56ae2ac3433736c8
Parents: ef78720
Author: Indhumathi27
Authored: Fri Dec 14 16:26:00 2018 +0530
Committer: Raghunandan S
Committed: Mon Dec 17 18:58:34 2018 +0530
--
.../sdv/generated/ComplexDataTypeTestCase.scala | 35 ++--
1 file changed, 18 insertions(+), 17 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/carbondata/blob/951a7811/integration/spark-common-cluster-test/src/test/scala/org/apache/carbondata/cluster/sdv/generated/ComplexDataTypeTestCase.scala
--
diff --git
a/integration/spark-common-cluster-test/src/test/scala/org/apache/carbondata/cluster/sdv/generated/ComplexDataTypeTestCase.scala
b/integration/spark-common-cluster-test/src/test/scala/org/apache/carbondata/cluster/sdv/generated/ComplexDataTypeTestCase.scala
index 723c4f7..6c2e271 100644
---
a/integration/spark-common-cluster-test/src/test/scala/org/apache/carbondata/cluster/sdv/generated/ComplexDataTypeTestCase.scala
+++
b/integration/spark-common-cluster-test/src/test/scala/org/apache/carbondata/cluster/sdv/generated/ComplexDataTypeTestCase.scala
@@ -74,7 +74,7 @@ class ComplexDataTypeTestCase extends QueryTest with
BeforeAndAfterAll {
sql(
"create table test(person
struct>) stored by 'carbondata'")
-sql("insert into test values('1:abc:4.30:true:2017/08/09:2017/08/09')")
+sql("insert into test
values('1\002abc\0024.30\002true\0022017/08/09\0022017/08/09')")
checkAnswer(sql("select * from test"),
Seq(Row(Row(Row(1, "abc", 4.3, true, java.sql.Date.valueOf("2017-08-09"),
Timestamp.valueOf("2017-08-09 00:00:00.0"))
@@ -83,8 +83,8 @@ class ComplexDataTypeTestCase extends QueryTest with
BeforeAndAfterAll {
"create table test(p1 array,p2 array,p3 array,p4
array,p5 " +
"array,p6 array) stored by 'carbondata'")
sql(
- "insert into test
values('1$2$3','abc$def$mno','4.30$4.60$5.20','true$true$false'," +
- "'2017/08/09$2017/08/09$2017/07/07','2017/08/09$2017/08/09$2017/07/07')")
+ "insert into test
values('1\0012\0013','abc\001def\001mno','4.30\0014.60\0015.20','true\001true\001false',"
+
+
"'2017/08/09\0012017/08/09\0012017/07/07','2017/08/09\0012017/08/09\0012017/07/07')")
checkAnswer(sql("select * from test"),
Seq(Row(mutable.WrappedArray.make(Array(1, 2, 3)),
mutable.WrappedArray.make(Array("abc", "def", "mno")),
@@ -142,7 +142,7 @@ class ComplexDataTypeTestCase extends QueryTest with
BeforeAndAfterAll {
"create table test(person
struct>) stored by 'carbondata'
tblproperties" +
"('dictionary_include'='person')")
-sql("insert into test values('1:abc:4.30:true:2017/08/09:2017/08/09')")
+sql("insert into test
values('1\002abc\0024.30\002true\0022017/08/09\0022017/08/09')")
checkAnswer(sql("select * from test"),
Seq(Row(Row(Row(1,
"abc", 4.3, true, java.sql.Date.valueOf("2017-08-09"),
@@ -153,8 +153,8 @@ class ComplexDataTypeTestCase extends QueryTest with
BeforeAndAfterAll {
"array,p6 array) stored by 'carbondata' tblproperties" +
"('dictionary_include'='p1,p2,p3,p4,p5,p6')")
sql(
- "insert into test
values('1$2$3','abc$def$mno','4.30$4.60$5.20','true$true$false'," +
- "'2017/08/09$2017/08/09$2017/07/07','2017/08/09$2017/08/09$2017/07/07')")
+ "insert into test
values('1\0012\0013','abc\001def\001mno','4.30\0014.60\0015.20','true\001true\001false',"
+
+
"'2017/08/09\0012017/08/09\0012017/07/07','2017/08/09\0012017/08/09\0012017/07/07')")
checkAnswer(sql("select * from test"),
Seq(Row(mutable.WrappedArray.make(Array(1, 2, 3)),
mutable.WrappedArray.make(Array("abc&q
[19/21] carbondata git commit: [CARBONDATA-3166]Updated Document and added Column Compressor used in Describe Formatted
[CARBONDATA-3166]Updated Document and added Column Compressor used in Describe Formatted Updated Document and added column compressor used in Describe Formatted Command This closes #2986 Project: http://git-wip-us.apache.org/repos/asf/carbondata/repo Commit: http://git-wip-us.apache.org/repos/asf/carbondata/commit/89351649 Tree: http://git-wip-us.apache.org/repos/asf/carbondata/tree/89351649 Diff: http://git-wip-us.apache.org/repos/asf/carbondata/diff/89351649 Branch: refs/heads/branch-1.5 Commit: 893516498c00d287ddb97b9faf04821dbd36c6d7 Parents: 3c59721 Author: shardul-cr7 Authored: Thu Dec 13 14:12:18 2018 +0530 Committer: Raghunandan S Committed: Mon Dec 17 18:58:34 2018 +0530 -- docs/configuration-parameters.md | 2 +- .../execution/command/table/CarbonDescribeFormattedCommand.scala | 4 +++- 2 files changed, 4 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/carbondata/blob/89351649/docs/configuration-parameters.md -- diff --git a/docs/configuration-parameters.md b/docs/configuration-parameters.md index 4aa2929..db21c6a 100644 --- a/docs/configuration-parameters.md +++ b/docs/configuration-parameters.md @@ -91,7 +91,7 @@ This section provides the details of all the configurations required for the Car | carbon.dictionary.server.port | 2030 | Single Pass Loading enables single job to finish data loading with dictionary generation on the fly. It enhances performance in the scenarios where the subsequent data loading after initial load involves fewer incremental updates on the dictionary. Single pass loading can be enabled using the option ***carbon.options.single.pass***. When this option is specified, a dictionary server will be internally started to handle the dictionary generation and query requests. This configuration specifies the port on which the server need to listen for incoming requests. Port value ranges between 0-65535 | | carbon.load.directWriteToStorePath.enabled | false | During data load, all the carbondata files are written to local disk and finally copied to the target store location in HDFS/S3. Enabling this parameter will make carbondata files to be written directly onto target HDFS/S3 location bypassing the local disk.**NOTE:** Writing directly to HDFS/S3 saves local disk IO(once for writing the files and again for copying to HDFS/S3) there by improving the performance. But the drawback is when data loading fails or the application crashes, unwanted carbondata files will remain in the target HDFS/S3 location until it is cleared during next data load or by running *CLEAN FILES* DDL command | | carbon.options.serialization.null.format | \N | Based on the business scenarios, some columns might need to be loaded with null values. As null value cannot be written in csv files, some special characters might be adopted to specify null values. This configuration can be used to specify the null values format in the data being loaded. | -| carbon.column.compressor | snappy | CarbonData will compress the column values using the compressor specified by this configuration. Currently CarbonData supports 'snappy' and 'zstd' compressors. | +| carbon.column.compressor | snappy | CarbonData will compress the column values using the compressor specified by this configuration. Currently CarbonData supports 'snappy', 'zstd' and 'gzip' compressors. | | carbon.minmax.allowed.byte.count | 200 | CarbonData will write the min max values for string/varchar types column using the byte count specified by this configuration. Max value is 1000 bytes(500 characters) and Min value is 10 bytes(5 characters). **NOTE:** This property is useful for reducing the store size thereby improving the query performance but can lead to query degradation if value is not configured properly. | | ## Compaction Configuration http://git-wip-us.apache.org/repos/asf/carbondata/blob/89351649/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/table/CarbonDescribeFormattedCommand.scala -- diff --git a/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/table/CarbonDescribeFormattedCommand.scala b/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/table/CarbonDescribeFormattedCommand.scala index 151359e..2d560df 100644 --- a/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/table/CarbonDescribeFormattedCommand.scala +++ b/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/table/CarbonDescribeFormattedCommand.scala @@ -92,7 +92,9 @@ private[sql] case class CarbonDescribeFormattedCommand(
[07/21] carbondata git commit: [CARBONDATA-2838] Added SDV test cases for Local Dictionary Support
[CARBONDATA-2838] Added SDV test cases for Local Dictionary Support
Added SDV test cases for Local Dictionary Support
This closes #2617
Project: http://git-wip-us.apache.org/repos/asf/carbondata/repo
Commit: http://git-wip-us.apache.org/repos/asf/carbondata/commit/ee8bfd05
Tree: http://git-wip-us.apache.org/repos/asf/carbondata/tree/ee8bfd05
Diff: http://git-wip-us.apache.org/repos/asf/carbondata/diff/ee8bfd05
Branch: refs/heads/branch-1.5
Commit: ee8bfd05f99d4bb26713021473d2547e1508c62f
Parents: 216d1c5
Author: praveenmeenakshi56
Authored: Wed Aug 8 11:45:49 2018 +0530
Committer: Raghunandan S
Committed: Mon Dec 17 18:58:33 2018 +0530
--
.../sdv/generated/AlterTableTestCase.scala | 2 +-
.../LoadTableWithLocalDictionaryTestCase.scala | 325 +++
.../cluster/sdv/suite/SDVSuites.scala | 7 +-
3 files changed, 331 insertions(+), 3 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/carbondata/blob/ee8bfd05/integration/spark-common-cluster-test/src/test/scala/org/apache/carbondata/cluster/sdv/generated/AlterTableTestCase.scala
--
diff --git
a/integration/spark-common-cluster-test/src/test/scala/org/apache/carbondata/cluster/sdv/generated/AlterTableTestCase.scala
b/integration/spark-common-cluster-test/src/test/scala/org/apache/carbondata/cluster/sdv/generated/AlterTableTestCase.scala
index 90fa602..2cf1794 100644
---
a/integration/spark-common-cluster-test/src/test/scala/org/apache/carbondata/cluster/sdv/generated/AlterTableTestCase.scala
+++
b/integration/spark-common-cluster-test/src/test/scala/org/apache/carbondata/cluster/sdv/generated/AlterTableTestCase.scala
@@ -1017,7 +1017,7 @@ class AlterTableTestCase extends QueryTest with
BeforeAndAfterAll {
}
val prop = CarbonProperties.getInstance()
- val p1 = prop.getProperty("carbon.horizontal.compaction.enable",
CarbonCommonConstants.defaultIsHorizontalCompactionEnabled)
+ val p1 = prop.getProperty("carbon.horizontal.compaction.enable",
CarbonCommonConstants.CARBON_HORIZONTAL_COMPACTION_ENABLE_DEFAULT)
val p2 = prop.getProperty("carbon.horizontal.update.compaction.threshold",
CarbonCommonConstants.DEFAULT_UPDATE_DELTAFILE_COUNT_THRESHOLD_IUD_COMPACTION)
val p3 = prop.getProperty("carbon.horizontal.delete.compaction.threshold",
CarbonCommonConstants.DEFAULT_DELETE_DELTAFILE_COUNT_THRESHOLD_IUD_COMPACTION)
val p4 = prop.getProperty("carbon.compaction.level.threshold",
CarbonCommonConstants.DEFAULT_SEGMENT_LEVEL_THRESHOLD)
http://git-wip-us.apache.org/repos/asf/carbondata/blob/ee8bfd05/integration/spark-common-cluster-test/src/test/scala/org/apache/carbondata/cluster/sdv/generated/LoadTableWithLocalDictionaryTestCase.scala
--
diff --git
a/integration/spark-common-cluster-test/src/test/scala/org/apache/carbondata/cluster/sdv/generated/LoadTableWithLocalDictionaryTestCase.scala
b/integration/spark-common-cluster-test/src/test/scala/org/apache/carbondata/cluster/sdv/generated/LoadTableWithLocalDictionaryTestCase.scala
new file mode 100644
index 000..199358b
--- /dev/null
+++
b/integration/spark-common-cluster-test/src/test/scala/org/apache/carbondata/cluster/sdv/generated/LoadTableWithLocalDictionaryTestCase.scala
@@ -0,0 +1,325 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.carbondata.spark.testsuite.localdictionary
+
+import scala.collection.JavaConverters._
+import java.io.{File, PrintWriter}
+import java.util
+import java.util.Collections
+
+import org.apache.spark.sql.test.util.QueryTest
+import org.scalatest.{BeforeAndAfterAll, Ignore}
+
+import org.apache.carbondata.core.cache.dictionary.DictionaryByteArrayWrapper
+import org.apache.carbondata.core.constants.CarbonCommonConstants
+import org.apache.carbondata.core.datastore.block.TableBlockInfo
+import org.apache.carbondata.core.datastore.chunk.impl.DimensionRawColumnChunk
+import
org.apache.
[06/21] carbondata git commit: [CARBONDATA-2840] Added SDV testcases for Complex DataType Support
[CARBONDATA-2840] Added SDV testcases for Complex DataType Support
Added SDV testcases for Complex DataType Support
This closes #2621
Project: http://git-wip-us.apache.org/repos/asf/carbondata/repo
Commit: http://git-wip-us.apache.org/repos/asf/carbondata/commit/22bec100
Tree: http://git-wip-us.apache.org/repos/asf/carbondata/tree/22bec100
Diff: http://git-wip-us.apache.org/repos/asf/carbondata/diff/22bec100
Branch: refs/heads/branch-1.5
Commit: 22bec1001dd348e543d953cdb2eefe825700b877
Parents: ee8bfd0
Author: Indhumathi27
Authored: Wed Aug 8 15:33:36 2018 +0530
Committer: Raghunandan S
Committed: Mon Dec 17 18:58:33 2018 +0530
--
.../sdv/generated/ComplexDataTypeTestCase.scala | 395 +++
.../cluster/sdv/suite/SDVSuites.scala | 1 +
2 files changed, 396 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/carbondata/blob/22bec100/integration/spark-common-cluster-test/src/test/scala/org/apache/carbondata/cluster/sdv/generated/ComplexDataTypeTestCase.scala
--
diff --git
a/integration/spark-common-cluster-test/src/test/scala/org/apache/carbondata/cluster/sdv/generated/ComplexDataTypeTestCase.scala
b/integration/spark-common-cluster-test/src/test/scala/org/apache/carbondata/cluster/sdv/generated/ComplexDataTypeTestCase.scala
new file mode 100644
index 000..723c4f7
--- /dev/null
+++
b/integration/spark-common-cluster-test/src/test/scala/org/apache/carbondata/cluster/sdv/generated/ComplexDataTypeTestCase.scala
@@ -0,0 +1,395 @@
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.cluster.sdv.generated
+
+import java.io.{ByteArrayInputStream, ByteArrayOutputStream, DataInputStream,
File, InputStream}
+import java.sql.Timestamp
+
+import scala.collection.mutable
+
+import org.apache.avro.file.DataFileWriter
+import org.apache.avro.generic.{GenericDatumReader, GenericDatumWriter,
GenericRecord}
+import org.apache.avro.io.{DecoderFactory, Encoder}
+import org.apache.commons.io.FileUtils
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.common.util.QueryTest
+import org.apache.spark.sql.test.TestQueryExecutor
+import org.scalatest.BeforeAndAfterAll
+
+import org.apache.carbondata.core.constants.CarbonCommonConstants
+import org.apache.carbondata.core.util.CarbonProperties
+import org.apache.carbondata.sdk.file.CarbonWriter
+
+/**
+ * Test Class for ComplexDataTypeTestCase to verify all scenerios
+ */
+
+class ComplexDataTypeTestCase extends QueryTest with BeforeAndAfterAll {
+
+ val filePath = TestQueryExecutor.integrationPath +
"/spark-common-test/src/test/resources"
+ val writerPath =
+s"${ resourcesPath }" + "/SparkCarbonFileFormat/WriterOutputComplex/"
+
+ override def beforeAll(): Unit = {
+FileUtils.deleteDirectory(new File(writerPath))
+sql("DROP TABLE IF EXISTS complexcarbontable")
+sql("DROP TABLE IF EXISTS test")
+sql("DROP TABLE IF EXISTS sdkOutputTable")
+ }
+
+ override def afterAll(): Unit = {
+FileUtils.deleteDirectory(new File(writerPath))
+sql("DROP TABLE IF EXISTS complexcarbontable")
+sql("DROP TABLE IF EXISTS test")
+sql("DROP TABLE IF EXISTS sdkOutputTable")
+CarbonProperties.getInstance()
+ .addProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT,
+CarbonCommonConstants.CARBON_TIMESTAMP_DEFAULT_FORMAT)
+ .addProperty(CarbonCommonConstants.CARBON_DATE_FORMAT,
+CarbonCommonConstants.CARBON_DATE_DEFAULT_FORMAT)
+ }
+
+ // check create table with complex data type and insert into complex table
+ test("test Complex_DataType-001") {
+CarbonProperties.getInstance()
+ .addProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT, "/MM/dd")
+ .addProperty(CarbonCommonConstants.CARBON_DATE_FORMAT, "/MM/dd")
+sql("DROP TABLE IF EXISTS test")
+sql(
+ "create table test(person
struct>) stored by &#
[08/21] carbondata git commit: [CARBONDATA-3005]Support Gzip as column compressor
[CARBONDATA-3005]Support Gzip as column compressor
This PR is to add a new compressor "Gzip" and enhance the compressing
capabilities offered by CarbonData.
User can now use gzip as the compressor for loading the data.
Gzip can be set at System Properties level or also for particular table.
This closes #2847
Project: http://git-wip-us.apache.org/repos/asf/carbondata/repo
Commit: http://git-wip-us.apache.org/repos/asf/carbondata/commit/8f57294d
Tree: http://git-wip-us.apache.org/repos/asf/carbondata/tree/8f57294d
Diff: http://git-wip-us.apache.org/repos/asf/carbondata/diff/8f57294d
Branch: refs/heads/branch-1.5
Commit: 8f57294d14532457d6fd3877a182f66055d338c2
Parents: 22bec10
Author: shardul-cr7
Authored: Tue Oct 23 17:27:47 2018 +0530
Committer: Raghunandan S
Committed: Mon Dec 17 18:58:33 2018 +0530
--
.../compression/AbstractCompressor.java | 1 +
.../compression/CompressorFactory.java | 3 +-
.../datastore/compression/GzipCompressor.java | 134 +++
.../datastore/compression/ZstdCompressor.java | 5 -
.../dataload/TestLoadDataWithCompression.scala | 94 ++---
.../TestLoadWithSortTempCompressed.scala| 20 +++
6 files changed, 236 insertions(+), 21 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/carbondata/blob/8f57294d/core/src/main/java/org/apache/carbondata/core/datastore/compression/AbstractCompressor.java
--
diff --git
a/core/src/main/java/org/apache/carbondata/core/datastore/compression/AbstractCompressor.java
b/core/src/main/java/org/apache/carbondata/core/datastore/compression/AbstractCompressor.java
index 0724bdc..c554dc6 100644
---
a/core/src/main/java/org/apache/carbondata/core/datastore/compression/AbstractCompressor.java
+++
b/core/src/main/java/org/apache/carbondata/core/datastore/compression/AbstractCompressor.java
@@ -123,4 +123,5 @@ public abstract class AbstractCompressor implements
Compressor {
return false;
}
+ @Override public boolean supportUnsafe() { return false; }
}
http://git-wip-us.apache.org/repos/asf/carbondata/blob/8f57294d/core/src/main/java/org/apache/carbondata/core/datastore/compression/CompressorFactory.java
--
diff --git
a/core/src/main/java/org/apache/carbondata/core/datastore/compression/CompressorFactory.java
b/core/src/main/java/org/apache/carbondata/core/datastore/compression/CompressorFactory.java
index f7d4e06..b7779ba 100644
---
a/core/src/main/java/org/apache/carbondata/core/datastore/compression/CompressorFactory.java
+++
b/core/src/main/java/org/apache/carbondata/core/datastore/compression/CompressorFactory.java
@@ -36,7 +36,8 @@ public class CompressorFactory {
public enum NativeSupportedCompressor {
SNAPPY("snappy", SnappyCompressor.class),
-ZSTD("zstd", ZstdCompressor.class);
+ZSTD("zstd", ZstdCompressor.class),
+GZIP("gzip", GzipCompressor.class);
private String name;
private Class compressorClass;
http://git-wip-us.apache.org/repos/asf/carbondata/blob/8f57294d/core/src/main/java/org/apache/carbondata/core/datastore/compression/GzipCompressor.java
--
diff --git
a/core/src/main/java/org/apache/carbondata/core/datastore/compression/GzipCompressor.java
b/core/src/main/java/org/apache/carbondata/core/datastore/compression/GzipCompressor.java
new file mode 100644
index 000..b386913
--- /dev/null
+++
b/core/src/main/java/org/apache/carbondata/core/datastore/compression/GzipCompressor.java
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.core.datastore.compression;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+
+import org.apache.commons.compress.compressors.gzip.GzipCompressorInputStream;
+import org.apache.commons.compress.compressors.gzip.
[18/21] carbondata git commit: [CARBONDATA-3153] Complex delimiters change
[CARBONDATA-3153] Complex delimiters change
Changed the two Complex Delimiters used to '\001' and '\002'.
This closes #2979
Project: http://git-wip-us.apache.org/repos/asf/carbondata/repo
Commit: http://git-wip-us.apache.org/repos/asf/carbondata/commit/3c597215
Tree: http://git-wip-us.apache.org/repos/asf/carbondata/tree/3c597215
Diff: http://git-wip-us.apache.org/repos/asf/carbondata/diff/3c597215
Branch: refs/heads/branch-1.5
Commit: 3c597215b96edb880041c4d651e5dfa665f4d2bb
Parents: 992b8f4
Author: manishnalla1994
Authored: Fri Dec 7 14:55:58 2018 +0530
Committer: Raghunandan S
Committed: Mon Dec 17 18:58:34 2018 +0530
--
conf/dataload.properties.template | 4 +-
docs/dml-of-carbondata.md | 12 +-
.../hadoop/api/CarbonTableOutputFormat.java | 2 +-
.../complexType/TestAdaptiveComplexType.scala | 104
.../TestAdaptiveEncodingForNullValues.scala | 22 +-
.../complexType/TestCompactionComplexType.scala | 250 +--
.../complexType/TestComplexDataType.scala | 63 ++---
.../TestComplexTypeWithBigArray.scala | 2 +-
...estLoadDataWithHiveSyntaxDefaultFormat.scala | 2 +-
.../VarcharDataTypesBasicTestCase.scala | 4 +-
.../sql/commands/UsingCarbondataSuite.scala | 2 +-
.../datasource/SparkCarbonDataSourceTest.scala | 4 +-
.../carbondata/CarbonDataSourceSuite.scala | 2 +-
.../carbondata/TestStreamingTableOpName.scala | 19 +-
.../TestStreamingTableWithRowParser.scala | 19 +-
.../processing/loading/model/LoadOption.java| 4 +-
.../sdk/file/CarbonWriterBuilder.java | 4 +-
.../sdk/file/CSVCarbonWriterTest.java | 4 +-
18 files changed, 260 insertions(+), 263 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/carbondata/blob/3c597215/conf/dataload.properties.template
--
diff --git a/conf/dataload.properties.template
b/conf/dataload.properties.template
index 0b2f6cf..3b582b2 100644
--- a/conf/dataload.properties.template
+++ b/conf/dataload.properties.template
@@ -51,10 +51,10 @@ delimiter=,
#all_dictionary_path=
#complex column's level 1 delimiter
-#complex_delimiter_level_1=\\$
+#complex_delimiter_level_1='\\\001'
#complex column's level 2 delimiter
-#complex_delimiter_level_2=\\:
+#complex_delimiter_level_2='\\\002'
#timestamp type column's data format
#dateformat=
http://git-wip-us.apache.org/repos/asf/carbondata/blob/3c597215/docs/dml-of-carbondata.md
--
diff --git a/docs/dml-of-carbondata.md b/docs/dml-of-carbondata.md
index 65654a4..d26cf19 100644
--- a/docs/dml-of-carbondata.md
+++ b/docs/dml-of-carbondata.md
@@ -132,18 +132,18 @@ CarbonData DML statements are documented here,which
includes:
- # COMPLEX_DELIMITER_LEVEL_1:
-Split the complex type data column in a row (eg., a$b$c --> Array =
{a,b,c}).
+Split the complex type data column in a row (eg., a\001b\001c --> Array =
{a,b,c}).
```
-OPTIONS('COMPLEX_DELIMITER_LEVEL_1'='$')
+OPTIONS('COMPLEX_DELIMITER_LEVEL_1'='\\\001')
```
- # COMPLEX_DELIMITER_LEVEL_2:
-Split the complex type nested data column in a row. Applies level_1
delimiter & applies level_2 based on complex data type (eg., a:b$c:d --> Array>
= {{a,b},{c,d}}).
+Split the complex type nested data column in a row. Applies level_1
delimiter & applies level_2 based on complex data type (eg., a\002b\001c\002d
--> Array> = {{a,b},{c,d}}).
```
-OPTIONS('COMPLEX_DELIMITER_LEVEL_2'=':')
+OPTIONS('COMPLEX_DELIMITER_LEVEL_2'='\\\002')
```
- # ALL_DICTIONARY_PATH:
@@ -212,8 +212,8 @@ CarbonData DML statements are documented here,which
includes:
'FILEHEADER'='empno,empname,designation,doj,workgroupcategory,
workgroupcategoryname,deptno,deptname,projectcode,
projectjoindate,projectenddate,attendance,utilization,salary',
- 'MULTILINE'='true','ESCAPECHAR'='\','COMPLEX_DELIMITER_LEVEL_1'='$',
- 'COMPLEX_DELIMITER_LEVEL_2'=':',
+ 'MULTILINE'='true','ESCAPECHAR'='\','COMPLEX_DELIMITER_LEVEL_1'='\\\001',
+ 'COMPLEX_DELIMITER_LEVEL_2'='\\\002',
'ALL_DICTIONARY_PATH'='/opt/alldictionary/data.dictionary',
'SINGLE_PASS'='TRUE')
```
http://git-wip-us.apache.org/repos/asf/carbondata/blob/3c597215/hadoop/src/main/java/org/apache/carbondata/hado
[21/21] carbondata git commit: [CARBONDATA-3102] Fix NoClassDefFoundError when use thriftServer and beeline to read/write data from/to S3
[CARBONDATA-3102] Fix NoClassDefFoundError when use thriftServer and beeline to
read/write data from/to S3
This PR fix NoClassDefFoundError when use thriftServer and beeline to use table
on cloud storage
This closes #2925
Project: http://git-wip-us.apache.org/repos/asf/carbondata/repo
Commit: http://git-wip-us.apache.org/repos/asf/carbondata/commit/ef787200
Tree: http://git-wip-us.apache.org/repos/asf/carbondata/tree/ef787200
Diff: http://git-wip-us.apache.org/repos/asf/carbondata/diff/ef787200
Branch: refs/heads/branch-1.5
Commit: ef7872009a09b7b2fb37991a9765d4f865d98078
Parents: 313ba0e
Author: xubo245
Authored: Thu Nov 15 20:36:12 2018 +0800
Committer: Raghunandan S
Committed: Mon Dec 17 18:58:34 2018 +0530
--
.../apache/carbondata/benchmark/SimpleQueryBenchmark.scala | 2 +-
.../scala/org/apache/carbondata/examples/S3Example.scala | 8
integration/hive/pom.xml | 4 ++--
integration/presto/pom.xml | 2 +-
integration/spark2/pom.xml | 5 +
pom.xml | 1 +
6 files changed, 14 insertions(+), 8 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/carbondata/blob/ef787200/examples/spark2/src/main/scala/org/apache/carbondata/benchmark/SimpleQueryBenchmark.scala
--
diff --git
a/examples/spark2/src/main/scala/org/apache/carbondata/benchmark/SimpleQueryBenchmark.scala
b/examples/spark2/src/main/scala/org/apache/carbondata/benchmark/SimpleQueryBenchmark.scala
index ce69c66..595af20 100644
---
a/examples/spark2/src/main/scala/org/apache/carbondata/benchmark/SimpleQueryBenchmark.scala
+++
b/examples/spark2/src/main/scala/org/apache/carbondata/benchmark/SimpleQueryBenchmark.scala
@@ -273,7 +273,7 @@ object SimpleQueryBenchmark {
}
}
- // run testcases and print comparison result
+ // run test cases and print comparison result
private def runTest(spark: SparkSession, table1: String, table2: String):
Unit = {
val formatter = new SimpleDateFormat("-MM-dd HH:mm:ss")
val date = new Date
http://git-wip-us.apache.org/repos/asf/carbondata/blob/ef787200/examples/spark2/src/main/scala/org/apache/carbondata/examples/S3Example.scala
--
diff --git
a/examples/spark2/src/main/scala/org/apache/carbondata/examples/S3Example.scala
b/examples/spark2/src/main/scala/org/apache/carbondata/examples/S3Example.scala
index d3d0a37..9cc43d0 100644
---
a/examples/spark2/src/main/scala/org/apache/carbondata/examples/S3Example.scala
+++
b/examples/spark2/src/main/scala/org/apache/carbondata/examples/S3Example.scala
@@ -157,8 +157,8 @@ object S3Example {
}
def getSparkMaster(args: Array[String]): String = {
- if (args.length == 5) args(4)
- else if (args(3).contains("spark:") || args(3).contains("mesos:"))
args(3)
- else "local"
-}
+if (args.length == 5) args(4)
+else if (args(3).contains("spark:") || args(3).contains("mesos:")) args(3)
+else "local"
+ }
}
http://git-wip-us.apache.org/repos/asf/carbondata/blob/ef787200/integration/hive/pom.xml
--
diff --git a/integration/hive/pom.xml b/integration/hive/pom.xml
index 0128241..7b8b06e 100644
--- a/integration/hive/pom.xml
+++ b/integration/hive/pom.xml
@@ -86,12 +86,12 @@
org.apache.httpcomponents
httpclient
-4.3.4
+${httpclient.version}
org.apache.httpcomponents
httpcore
-4.3-alpha1
+${httpcore.version}
org.apache.hadoop
http://git-wip-us.apache.org/repos/asf/carbondata/blob/ef787200/integration/presto/pom.xml
--
diff --git a/integration/presto/pom.xml b/integration/presto/pom.xml
index 1c1b91c..828696b 100644
--- a/integration/presto/pom.xml
+++ b/integration/presto/pom.xml
@@ -543,7 +543,7 @@
org.apache.httpcomponents
httpcore
- 4.2
+ ${httpcore.version}
http://git-wip-us.apache.org/repos/asf/carbondata/blob/ef787200/integration/spark2/pom.xml
--
diff --git a/integration/spark2/pom.xml b/integration/spark2/pom.xml
index f127dce..aea3da3 100644
--- a/integration/spark2/pom.xml
+++ b/integration/spark2/pom.xml
@@ -129,6 +129,11 @@
+ org.apache.httpcomponents
+ httpclient
+ ${httpclient.version}
+
+
net.java.dev.jets3t
jets3t
[17/21] carbondata git commit: [CARBONDATA-3153] Complex delimiters change
http://git-wip-us.apache.org/repos/asf/carbondata/blob/3c597215/integration/spark2/src/test/scala/org/apache/spark/carbondata/TestStreamingTableOpName.scala
--
diff --git
a/integration/spark2/src/test/scala/org/apache/spark/carbondata/TestStreamingTableOpName.scala
b/integration/spark2/src/test/scala/org/apache/spark/carbondata/TestStreamingTableOpName.scala
index f5596f2..e3c2d88 100644
---
a/integration/spark2/src/test/scala/org/apache/spark/carbondata/TestStreamingTableOpName.scala
+++
b/integration/spark2/src/test/scala/org/apache/spark/carbondata/TestStreamingTableOpName.scala
@@ -2323,23 +2323,23 @@ class TestStreamingTableOpName extends QueryTest with
BeforeAndAfterAll {
stringBuilder.append(index.toString + "abc,name_" + index
+ ",city_" + index + "," + (1.00 *
index).toString + ",0.01,80.01" +
",1990-01-01,2010-01-01
10:01:01,2010-01-01 10:01:01" +
- ",school_" + index + ":school_" + index +
index + "$" + index)
+ ",school_" + index + "\002school_" +
index + index + "\001" + index)
} else if (index == 9) {
stringBuilder.append(index.toString + ",name_" + index
+ ",city_" + index + "," + (1.00 *
index).toString + ",0.04,80.04" +
",1990-01-04,2010-01-04
10:01:01,2010-01-04 10:01:01" +
- ",school_" + index + ":school_" + index +
index + "$" + index)
+ ",school_" + index + "\002school_" +
index + index + "\001" + index)
} else {
stringBuilder.append(index.toString + ",name_" + index
+ ",city_" + index + "," + (1.00 *
index).toString + ",0.01,80.01" +
",1990-01-01,2010-01-01
10:01:01,2010-01-01 10:01:01" +
- ",school_" + index + ":school_" + index +
index + "$" + index)
+ ",school_" + index + "\002school_" +
index + index + "\001" + index)
}
} else {
stringBuilder.append(index.toString + ",name_" + index
+ ",city_" + index + "," + (1.00 *
index).toString + ",0.01,80.01" +
",1990-01-01,2010-01-01 10:01:01,2010-01-01
10:01:01" +
- ",school_" + index + ":school_" + index +
index + "$" + index)
+ ",school_" + index + "\002school_" + index
+ index + "\001" + index)
}
stringBuilder.append("\n")
}
@@ -2474,7 +2474,7 @@ class TestStreamingTableOpName extends QueryTest with
BeforeAndAfterAll {
"1990-01-01",
"2010-01-01 10:01:01",
"2010-01-01 10:01:01",
-"school_" + id + ":school_" + id + id + "$" + id)
+"school_" + id + "\002school_" + id + id + "\001" + id)
}
spark.createDataFrame(csvRDD).toDF(
"id", "name", "city", "salary", "tax", "percent", "birthday",
"register", "updated", "file")
@@ -2489,7 +2489,7 @@ class TestStreamingTableOpName extends QueryTest with
BeforeAndAfterAll {
"1990-01-01",
"2010-01-01 10:01:01",
"2010-01-01 10:01:01",
-"school_" + id + ":school_" + id + id + "$" + id)
+"school_" + id + "\002school_" + id + id + "\001" + id)
}
spark.createDataFrame(csvRDD).toDF(
"id", "salary", "tax", "percent", "birthday", "register", "updated",
"file")
@@ -2594,11 +2594,8 @@ class TestStreamingTableOpName extends QueryTest with
BeforeAndAfterAll {
def executeBatchLoad(tableName: String): Unit = {
sql(
- s"""
- | LOAD DATA LOCAL INPATH '$dataFilePath'
- | INTO TABLE streaming.$tableName
- | OPTIONS('HEADER'='true')
- """.stripMargin)
+ s"LOAD DATA LOCAL INPATH '$dataFilePath' INTO TABLE streaming.$tableName
OPTIONS" +
+ "('HEADER'='true','COMPLEX_DELIMITER_LEVEL_1'='$',
'COMPLEX_DELIMITER_LEVEL_2'=':')")
}
def wrap(array: Array[String]) = {
http://git-wip-us.apache.org/repos/asf/carbondata/blob/3c597215/integration/spark2/src/test/scala/org/apache/spark/carbondata/TestStreamingTableWithRowParser.scala
--
diff --git
a/integration/spark2/src/test/scala/org/apache/spark/carbondata/TestStreamingTableWithRowParser.scala
b/integration/spark2/src/test/scala/org/apache/spark/carbondata/TestStreamingTableWithRowParser.scala
index 9beee59..985b9d9 100644
---
a/integration/spark2/src/test/scala/org/apache/spark/carbondata/TestStreamingTableWithRowParser.scala
+++
b/i
[10/21] carbondata git commit: [CARBONDATA-3116] Support set carbon.query.directQueryOnDataMap.enabled=true
[CARBONDATA-3116] Support set carbon.query.directQueryOnDataMap.enabled=true
This PR fix the errorï¼User can query on dataMap after set
carbon.query.directQueryOnDataMap.enabled=true.
This closes #2940
Project: http://git-wip-us.apache.org/repos/asf/carbondata/repo
Commit: http://git-wip-us.apache.org/repos/asf/carbondata/commit/26a58508
Tree: http://git-wip-us.apache.org/repos/asf/carbondata/tree/26a58508
Diff: http://git-wip-us.apache.org/repos/asf/carbondata/diff/26a58508
Branch: refs/heads/branch-1.5
Commit: 26a585083fb7887d66c85c3ee27fcae205595742
Parents: 7d7d547
Author: xubo245
Authored: Thu Nov 22 00:32:13 2018 +0800
Committer: Raghunandan S
Committed: Mon Dec 17 18:58:33 2018 +0530
--
.../preaggregate/TestPreAggCreateCommand.scala | 43
.../sql/optimizer/CarbonLateDecodeRule.scala| 8 ++--
2 files changed, 48 insertions(+), 3 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/carbondata/blob/26a58508/integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/preaggregate/TestPreAggCreateCommand.scala
--
diff --git
a/integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/preaggregate/TestPreAggCreateCommand.scala
b/integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/preaggregate/TestPreAggCreateCommand.scala
index 7851bd1..f07c417 100644
---
a/integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/preaggregate/TestPreAggCreateCommand.scala
+++
b/integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/preaggregate/TestPreAggCreateCommand.scala
@@ -17,6 +17,7 @@
package org.apache.carbondata.integration.spark.testsuite.preaggregate
+import java.io.File
import java.util
import java.util.concurrent.{Callable, ExecutorService, Executors, TimeUnit}
@@ -486,6 +487,48 @@ class TestPreAggCreateCommand extends QueryTest with
BeforeAndAfterAll {
executorService.shutdown()
}
+ test("support set carbon.query.directQueryOnDataMap.enabled=true") {
+val rootPath = new File(this.getClass.getResource("/").getPath
+ + "../../../..").getCanonicalPath
+val testData =
s"$rootPath/integration/spark-common-test/src/test/resources/sample.csv"
+sql("drop table if exists mainTable")
+sql(
+ s"""
+ | CREATE TABLE mainTable
+ | (id Int,
+ | name String,
+ | city String,
+ | age Int)
+ | STORED BY 'org.apache.carbondata.format'
+ """.stripMargin)
+
+sql(
+ s"""
+ | LOAD DATA LOCAL INPATH '$testData'
+ | into table mainTable
+ """.stripMargin)
+
+sql(
+ s"""
+ | create datamap preagg_sum on table mainTable
+ | using 'preaggregate'
+ | as select id,sum(age) from mainTable group by id
+ """.stripMargin)
+
+CarbonProperties.getInstance()
+ .addProperty(CarbonCommonConstants.VALIDATE_DIRECT_QUERY_ON_DATAMAP,
"true")
+
+sql("set carbon.query.directQueryOnDataMap.enabled=true")
+checkAnswer(sql("select count(*) from maintable_preagg_sum"), Row(4))
+sql("set carbon.query.directQueryOnDataMap.enabled=false")
+val exception: Exception = intercept[AnalysisException] {
+ sql("select count(*) from maintable_preagg_sum").collect()
+}
+assert(exception.getMessage.contains("Query On DataMap not supported"))
+CarbonProperties.getInstance()
+ .addProperty(CarbonCommonConstants.VALIDATE_DIRECT_QUERY_ON_DATAMAP,
"false")
+ }
+
class QueryTask(query: String) extends Callable[String] {
override def call(): String = {
var result = "SUCCESS"
http://git-wip-us.apache.org/repos/asf/carbondata/blob/26a58508/integration/spark2/src/main/scala/org/apache/spark/sql/optimizer/CarbonLateDecodeRule.scala
--
diff --git
a/integration/spark2/src/main/scala/org/apache/spark/sql/optimizer/CarbonLateDecodeRule.scala
b/integration/spark2/src/main/scala/org/apache/spark/sql/optimizer/CarbonLateDecodeRule.scala
index 48c6377..36eb9ce 100644
---
a/integration/spark2/src/main/scala/org/apache/spark/sql/optimizer/CarbonLateDecodeRule.scala
+++
b/integration/spark2/src/main/scala/org/apache/spark/sql/optimizer/CarbonLateDecodeRule.scala
@@ -119,7 +119,7 @@ class CarbonLateDecodeRule extends Rule[LogicalPlan] with
PredicateHelper {
if (vali
[04/21] carbondata git commit: [CARBONDATA-3141] Removed Carbon Table Detail Command Test case
[CARBONDATA-3141] Removed Carbon Table Detail Command Test case
Removed Carbon Table Detail Command Test case, as this is not used.
This closes #2968
Project: http://git-wip-us.apache.org/repos/asf/carbondata/repo
Commit: http://git-wip-us.apache.org/repos/asf/carbondata/commit/216d1c5b
Tree: http://git-wip-us.apache.org/repos/asf/carbondata/tree/216d1c5b
Diff: http://git-wip-us.apache.org/repos/asf/carbondata/diff/216d1c5b
Branch: refs/heads/branch-1.5
Commit: 216d1c5b267167b07aedbfb2ed94a7ad54584592
Parents: 523515c
Author: praveenmeenakshi56
Authored: Fri Nov 30 18:05:08 2018 +0530
Committer: Raghunandan S
Committed: Mon Dec 17 18:56:14 2018 +0530
--
.../schema/CarbonGetTableDetailCommand.scala| 61
.../CarbonGetTableDetailComandTestCase.scala| 56 --
2 files changed, 117 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/carbondata/blob/216d1c5b/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/schema/CarbonGetTableDetailCommand.scala
--
diff --git
a/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/schema/CarbonGetTableDetailCommand.scala
b/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/schema/CarbonGetTableDetailCommand.scala
deleted file mode 100644
index 90da68a..000
---
a/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/schema/CarbonGetTableDetailCommand.scala
+++ /dev/null
@@ -1,61 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- *http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.spark.sql.execution.command.schema
-
-import org.apache.spark.sql.{CarbonEnv, Row, SparkSession}
-import org.apache.spark.sql.catalyst.expressions.{Attribute,
AttributeReference}
-import org.apache.spark.sql.execution.command.DataCommand
-import org.apache.spark.sql.types.{LongType, StringType}
-
-import org.apache.carbondata.core.metadata.AbsoluteTableIdentifier
-import org.apache.carbondata.core.statusmanager.SegmentStatusManager
-import org.apache.carbondata.core.util.CarbonProperties
-
-/**
- * collect dynamic detail information of the table, including table size, last
modified time, etc.
- */
-case class CarbonGetTableDetailCommand(
-databaseName: String,
-tableNames: Option[Seq[String]])
- extends DataCommand {
-
- override def processData(sparkSession: SparkSession): Seq[Row] = {
-if (tableNames.isDefined) {
- tableNames.get.map { tablename =>
-val carbonTable = CarbonEnv.getCarbonTable(Option(databaseName),
- tablename)(sparkSession)
-
-Row(
- tablename,
- carbonTable.size,
- SegmentStatusManager
-
.getTableStatusLastModifiedTime(carbonTable.getAbsoluteTableIdentifier))
- }
-} else {
- Seq.empty[Row]
-}
- }
-
- override def output: Seq[Attribute] = {
-Seq(AttributeReference("table name", StringType, nullable = false)(),
- AttributeReference("table size", LongType, nullable = false)(),
- AttributeReference("last modified time", LongType, nullable = false)())
- }
-
- override protected def opName: String = "GET TABLE DETAIL"
-}
http://git-wip-us.apache.org/repos/asf/carbondata/blob/216d1c5b/integration/spark2/src/test/scala/org/apache/spark/sql/CarbonGetTableDetailComandTestCase.scala
--
diff --git
a/integration/spark2/src/test/scala/org/apache/spark/sql/CarbonGetTableDetailComandTestCase.scala
b/integration/spark2/src/test/scala/org/apache/spark/sql/CarbonGetTableDetailComandTestCase.scala
deleted file mode 100644
index 908bc75..000
---
a/integration/spark2/src/test/scala/org/apache/spark/sql/CarbonGetTableDetailComandTestCase.scala
+++ /dev/null
@@ -1,56 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding c
[13/21] carbondata git commit: [CARBONDATA-3158] support presto-carbon to read sdk cabron files
[CARBONDATA-3158] support presto-carbon to read sdk cabron files
problem:
Currently, carbon SDK files output (files without metadata folder and its
contents) are read by spark using an external table with carbon session.
But presto carbon integration doesn't support that. It can currently read
only the transactional table output files.
solution:
Hence we can enhance presto to read SDK output files. This will increase
the use cases for presto-carbon integration.
The above scenario can be achieved by inferring schema if metadata folder
not exists and
setting read committed scope to LatestFilesReadCommittedScope, if
non-transctional table output files are present.
This closes #2982
Project: http://git-wip-us.apache.org/repos/asf/carbondata/repo
Commit: http://git-wip-us.apache.org/repos/asf/carbondata/commit/992b8f40
Tree: http://git-wip-us.apache.org/repos/asf/carbondata/tree/992b8f40
Diff: http://git-wip-us.apache.org/repos/asf/carbondata/diff/992b8f40
Branch: refs/heads/branch-1.5
Commit: 992b8f4024a7ecacefd61dd805e29a60b5fd3d66
Parents: e8d25bb
Author: ajantha-bhat
Authored: Fri Dec 7 18:37:10 2018 +0530
Committer: Raghunandan S
Committed: Mon Dec 17 18:58:34 2018 +0530
--
integration/presto/pom.xml | 6 +
.../presto/CarbondataPageSourceProvider.java| 3 +
.../presto/impl/CarbonTableReader.java | 74 +++---
.../PrestoAllDataTypeLocalDictTest.scala| 30 +--
.../integrationtest/PrestoAllDataTypeTest.scala | 102
.../PrestoTestNonTransactionalTableFiles.scala | 233 +++
.../carbondata/presto/server/PrestoServer.scala | 35 ++-
7 files changed, 378 insertions(+), 105 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/carbondata/blob/992b8f40/integration/presto/pom.xml
--
diff --git a/integration/presto/pom.xml b/integration/presto/pom.xml
index d0556b9..1c1b91c 100644
--- a/integration/presto/pom.xml
+++ b/integration/presto/pom.xml
@@ -487,6 +487,12 @@
test
+ org.apache.carbondata
+ carbondata-store-sdk
+ ${project.version}
+ test
+
+
org.lz4
lz4-java
1.4.0
http://git-wip-us.apache.org/repos/asf/carbondata/blob/992b8f40/integration/presto/src/main/java/org/apache/carbondata/presto/CarbondataPageSourceProvider.java
--
diff --git
a/integration/presto/src/main/java/org/apache/carbondata/presto/CarbondataPageSourceProvider.java
b/integration/presto/src/main/java/org/apache/carbondata/presto/CarbondataPageSourceProvider.java
index bef246e..96024e4 100644
---
a/integration/presto/src/main/java/org/apache/carbondata/presto/CarbondataPageSourceProvider.java
+++
b/integration/presto/src/main/java/org/apache/carbondata/presto/CarbondataPageSourceProvider.java
@@ -139,6 +139,9 @@ public class CarbondataPageSourceProvider implements
ConnectorPageSourceProvider
Configuration conf = new Configuration();
conf.set(CarbonTableInputFormat.INPUT_SEGMENT_NUMBERS, "");
String carbonTablePath =
carbonTable.getAbsoluteTableIdentifier().getTablePath();
+ CarbonTableInputFormat
+ .setTransactionalTable(conf,
carbonTable.getTableInfo().isTransactionalTable());
+ CarbonTableInputFormat.setTableInfo(conf, carbonTable.getTableInfo());
conf.set(CarbonTableInputFormat.INPUT_DIR, carbonTablePath);
conf.set("query.id", queryId);
http://git-wip-us.apache.org/repos/asf/carbondata/blob/992b8f40/integration/presto/src/main/java/org/apache/carbondata/presto/impl/CarbonTableReader.java
--
diff --git
a/integration/presto/src/main/java/org/apache/carbondata/presto/impl/CarbonTableReader.java
b/integration/presto/src/main/java/org/apache/carbondata/presto/impl/CarbonTableReader.java
index 546bf9b..9677839 100755
---
a/integration/presto/src/main/java/org/apache/carbondata/presto/impl/CarbonTableReader.java
+++
b/integration/presto/src/main/java/org/apache/carbondata/presto/impl/CarbonTableReader.java
@@ -53,8 +53,10 @@ import org.apache.carbondata.core.scan.expression.Expression;
import org.apache.carbondata.core.statusmanager.LoadMetadataDetails;
import org.apache.carbondata.core.statusmanager.SegmentStatusManager;
import org.apache.carbondata.core.util.CarbonProperties;
+import org.apache.carbondata.core.util.CarbonUtil;
import org.apache.carbondata.core.util.path.CarbonTablePath;
import org.apache.carbondata.hadoop.CarbonInputSplit;
+import org.apache.carbondata.hadoop.api.CarbonInputFormat;
import org.apache.carbondata.hadoop.api.CarbonTableInputFormat;
import org.apache.carbondata.presto.PrestoFilterUtil;
@@ -284,18 +286,6 @@ public class Ca
[05/21] carbondata git commit: [CARBONDATA-3147] Fixed concurrent load issue
[CARBONDATA-3147] Fixed concurrent load issue
Problem: During datamap commit, tablestatus_UUID files were being renamed to
tablestatus due to which any new in progress entry was being skipped and
eventually during writing of success status 'Entry not found exception was
thrown'.
Solution: Instead of renaming the files, now we are reading both the files and
merging the contents. Then the updated details are written to the tablestatus
file.
This closes #2977
Project: http://git-wip-us.apache.org/repos/asf/carbondata/repo
Commit: http://git-wip-us.apache.org/repos/asf/carbondata/commit/7d7d547e
Tree: http://git-wip-us.apache.org/repos/asf/carbondata/tree/7d7d547e
Diff: http://git-wip-us.apache.org/repos/asf/carbondata/diff/7d7d547e
Branch: refs/heads/branch-1.5
Commit: 7d7d547eff033e8760dff59b7c0a5fece76657c7
Parents: 8f57294
Author: kunal642
Authored: Wed Dec 5 16:00:41 2018 +0530
Committer: Raghunandan S
Committed: Mon Dec 17 18:58:33 2018 +0530
--
.../spark/rdd/AggregateDataMapCompactor.scala | 69 ++---
.../spark/rdd/CarbonDataRDDFactory.scala| 3 +-
.../management/CarbonLoadDataCommand.scala | 2 +-
.../preaaggregate/PreAggregateListeners.scala | 142 ++-
.../preaaggregate/PreAggregateUtil.scala| 7 +-
.../processing/util/CarbonLoaderUtil.java | 10 +-
6 files changed, 174 insertions(+), 59 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/carbondata/blob/7d7d547e/integration/spark2/src/main/scala/org/apache/carbondata/spark/rdd/AggregateDataMapCompactor.scala
--
diff --git
a/integration/spark2/src/main/scala/org/apache/carbondata/spark/rdd/AggregateDataMapCompactor.scala
b/integration/spark2/src/main/scala/org/apache/carbondata/spark/rdd/AggregateDataMapCompactor.scala
index 2119e4c..6d28295 100644
---
a/integration/spark2/src/main/scala/org/apache/carbondata/spark/rdd/AggregateDataMapCompactor.scala
+++
b/integration/spark2/src/main/scala/org/apache/carbondata/spark/rdd/AggregateDataMapCompactor.scala
@@ -23,11 +23,12 @@ import scala.collection.JavaConverters._
import org.apache.spark.sql.{CarbonSession, SQLContext}
import org.apache.spark.sql.execution.command.CompactionModel
import org.apache.spark.sql.execution.command.management.CarbonLoadDataCommand
-import org.apache.spark.sql.execution.command.preaaggregate.PreAggregateUtil
+import
org.apache.spark.sql.execution.command.preaaggregate.{CommitPreAggregateListener,
PreAggregateUtil}
import org.apache.carbondata.core.constants.CarbonCommonConstants
import org.apache.carbondata.core.datamap.Segment
-import org.apache.carbondata.core.datastore.impl.FileFactory
+import org.apache.carbondata.core.metadata.schema.table.CarbonTable
+import org.apache.carbondata.core.mutate.CarbonUpdateUtil
import org.apache.carbondata.core.statusmanager.{SegmentStatus,
SegmentStatusManager}
import org.apache.carbondata.core.util.path.CarbonTablePath
import org.apache.carbondata.events.OperationContext
@@ -79,20 +80,32 @@ class AggregateDataMapCompactor(carbonLoadModel:
CarbonLoadModel,
CarbonSession.threadSet(CarbonCommonConstants.SUPPORT_DIRECT_QUERY_ON_DATAMAP,
"true")
loadCommand.processData(sqlContext.sparkSession)
-val newLoadMetaDataDetails = SegmentStatusManager.readLoadMetadata(
+// After load is completed for child table the UUID table status will
have 0.1 as success
+// and the table status file will have 0,1,2,3 as Success and 0.1 as
In Progress.
+// Therefore we will read the table status and write 0,1,2,3 as
Compacted as the commit
+// listener will take care of merging the UUID and the table status
files.
+val newMetadataDetails = SegmentStatusManager.readLoadMetadata(
carbonTable.getMetadataPath, uuid)
-val updatedLoadMetaDataDetails = newLoadMetaDataDetails collect {
- case load if loadMetaDataDetails.contains(load) =>
-load.setMergedLoadName(mergedLoadName)
-load.setSegmentStatus(SegmentStatus.COMPACTED)
-
load.setModificationOrdeletionTimesStamp(System.currentTimeMillis())
-load
- case other => other
-}
+val mergedContent = loadMetaDataDetails.asScala.map {
+ segment => segment.setSegmentStatus(SegmentStatus.COMPACTED)
+segment.setMergedLoadName(mergedLoadName)
+
segment.setModificationOrdeletionTimesStamp(CarbonUpdateUtil.readCurrentTime)
+segment
+} ++ newMetadataDetails
SegmentStatusManager.writeLoadDetailsIntoFile(
CarbonTablePath.getTableStatusFilePathWithUUID(carbonTable.getTablePath, uuid),
-updatedLoadMetaDataDetails)
-
carbonLoa
[15/21] carbondata git commit: [CARBONDATA-3002] Fix some spell error
http://git-wip-us.apache.org/repos/asf/carbondata/blob/313ba0ea/integration/spark2/src/main/scala/org/apache/spark/util/TableLoader.scala
--
diff --git
a/integration/spark2/src/main/scala/org/apache/spark/util/TableLoader.scala
b/integration/spark2/src/main/scala/org/apache/spark/util/TableLoader.scala
index b6667df..efaa191 100644
--- a/integration/spark2/src/main/scala/org/apache/spark/util/TableLoader.scala
+++ b/integration/spark2/src/main/scala/org/apache/spark/util/TableLoader.scala
@@ -82,7 +82,7 @@ object TableLoader {
val spark = TableAPIUtil.spark(storePath, s"TableLoader:
$dbName.$tableName")
-CarbonEnv.getInstance(spark).carbonMetastore.
+CarbonEnv.getInstance(spark).carbonMetaStore.
checkSchemasModifiedTimeAndReloadTable(TableIdentifier(tableName,
Some(dbName)))
loadTable(spark, Option(dbName), tableName, inputPaths, map)
}
http://git-wip-us.apache.org/repos/asf/carbondata/blob/313ba0ea/integration/spark2/src/main/spark2.1/org/apache/spark/sql/hive/CarbonSessionState.scala
--
diff --git
a/integration/spark2/src/main/spark2.1/org/apache/spark/sql/hive/CarbonSessionState.scala
b/integration/spark2/src/main/spark2.1/org/apache/spark/sql/hive/CarbonSessionState.scala
index 759539b..47b6b67 100644
---
a/integration/spark2/src/main/spark2.1/org/apache/spark/sql/hive/CarbonSessionState.scala
+++
b/integration/spark2/src/main/spark2.1/org/apache/spark/sql/hive/CarbonSessionState.scala
@@ -164,10 +164,10 @@ class CarbonHiveSessionCatalog(
carbonDatasourceHadoopRelation: CarbonDatasourceHadoopRelation): Boolean
= {
var isRefreshed = false
val storePath = CarbonProperties.getStorePath
-carbonEnv.carbonMetastore.
+carbonEnv.carbonMetaStore.
checkSchemasModifiedTimeAndReloadTable(identifier)
-val table = carbonEnv.carbonMetastore.getTableFromMetadataCache(
+val table = carbonEnv.carbonMetaStore.getTableFromMetadataCache(
carbonDatasourceHadoopRelation.carbonTable.getDatabaseName,
carbonDatasourceHadoopRelation.carbonTable.getTableName)
if (table.isEmpty || (table.isDefined &&
http://git-wip-us.apache.org/repos/asf/carbondata/blob/313ba0ea/integration/spark2/src/test/scala/org/apache/carbondata/spark/util/AllDictionaryTestCase.scala
--
diff --git
a/integration/spark2/src/test/scala/org/apache/carbondata/spark/util/AllDictionaryTestCase.scala
b/integration/spark2/src/test/scala/org/apache/carbondata/spark/util/AllDictionaryTestCase.scala
index ed5486b..bceb0fc 100644
---
a/integration/spark2/src/test/scala/org/apache/carbondata/spark/util/AllDictionaryTestCase.scala
+++
b/integration/spark2/src/test/scala/org/apache/carbondata/spark/util/AllDictionaryTestCase.scala
@@ -137,7 +137,7 @@ class AllDictionaryTestCase extends Spark2QueryTest with
BeforeAndAfterAll {
.config("spark.executor.heartbeatInterval", "600s")
.config("carbon.enable.vector.reader","false")
.getOrCreateCarbonSession(storeLocation, metastoredb)
-val catalog = CarbonEnv.getInstance(spark).carbonMetastore
+val catalog = CarbonEnv.getInstance(spark).carbonMetaStore
sampleRelation =
catalog.lookupRelation(Option(CarbonCommonConstants.DATABASE_DEFAULT_NAME),
"sample")(spark).asInstanceOf[CarbonRelation]
complexRelation =
catalog.lookupRelation(Option(CarbonCommonConstants.DATABASE_DEFAULT_NAME),
http://git-wip-us.apache.org/repos/asf/carbondata/blob/313ba0ea/integration/spark2/src/test/scala/org/apache/carbondata/spark/util/DictionaryLRUCacheTestCase.scala
--
diff --git
a/integration/spark2/src/test/scala/org/apache/carbondata/spark/util/DictionaryLRUCacheTestCase.scala
b/integration/spark2/src/test/scala/org/apache/carbondata/spark/util/DictionaryLRUCacheTestCase.scala
index 245ee7c..3c50a18 100644
---
a/integration/spark2/src/test/scala/org/apache/carbondata/spark/util/DictionaryLRUCacheTestCase.scala
+++
b/integration/spark2/src/test/scala/org/apache/carbondata/spark/util/DictionaryLRUCacheTestCase.scala
@@ -39,7 +39,7 @@ class DictionaryLRUCacheTestCase extends Spark2QueryTest with
BeforeAndAfterAll
var path : String = null
def checkDictionaryAccessCount(databaseName: String, tableName: String):
Unit = {
-val carbonTable =
CarbonEnv.getInstance(Spark2TestQueryExecutor.spark).carbonMetastore
+val carbonTable =
CarbonEnv.getInstance(Spark2TestQueryExecutor.spark).carbonMetaStore
.lookupRelation(Option(databaseName),
tableName)(Spark2TestQueryExecutor.spark)
.asInstanceOf[CarbonRelation].carbonTable
val absoluteTableIdentifier = carbonTable.getAbsoluteTableIdentifier
http://git-wip-us.apache.org/repos/asf/carbondata/blob/313ba0ea/integration/spark2/src/test/scala/org/apache/car
[03/21] carbondata git commit: [CARBONDATA-3154] Fix spark-2.1 test error
[CARBONDATA-3154] Fix spark-2.1 test error
Spark2.2.1 supports location, but Spark2.1.0 doesn't support location,
supports options(path 'your file path').
So we should change location to options(path ... ) and
create new directory before use "create table" in spark2.1.0.
This closes #2981
Project: http://git-wip-us.apache.org/repos/asf/carbondata/repo
Commit: http://git-wip-us.apache.org/repos/asf/carbondata/commit/3524f51d
Tree: http://git-wip-us.apache.org/repos/asf/carbondata/tree/3524f51d
Diff: http://git-wip-us.apache.org/repos/asf/carbondata/diff/3524f51d
Branch: refs/heads/branch-1.5
Commit: 3524f51df087059f1fa7ccd87f2082e4d2d36c20
Parents: 442e244
Author: xubo245
Authored: Sat Dec 8 00:01:43 2018 +0800
Committer: Raghunandan S
Committed: Mon Dec 17 18:50:23 2018 +0530
--
.../datasource/SparkCarbonDataSourceTest.scala | 112 +++
1 file changed, 91 insertions(+), 21 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/carbondata/blob/3524f51d/integration/spark-datasource/src/test/scala/org/apache/spark/sql/carbondata/datasource/SparkCarbonDataSourceTest.scala
--
diff --git
a/integration/spark-datasource/src/test/scala/org/apache/spark/sql/carbondata/datasource/SparkCarbonDataSourceTest.scala
b/integration/spark-datasource/src/test/scala/org/apache/spark/sql/carbondata/datasource/SparkCarbonDataSourceTest.scala
index c5d6a8c..470e0bf 100644
---
a/integration/spark-datasource/src/test/scala/org/apache/spark/sql/carbondata/datasource/SparkCarbonDataSourceTest.scala
+++
b/integration/spark-datasource/src/test/scala/org/apache/spark/sql/carbondata/datasource/SparkCarbonDataSourceTest.scala
@@ -19,13 +19,14 @@ package org.apache.spark.sql.carbondata.datasource
import java.io.File
import java.util
-import java.util.Arrays
import scala.collection.JavaConverters._
import scala.collection.mutable
+import org.apache.hadoop.fs.permission.{FsAction, FsPermission}
import org.apache.spark.sql.{AnalysisException, Row}
import org.apache.spark.sql.carbondata.datasource.TestUtil._
+import org.apache.spark.util.SparkUtil
import org.scalatest.{BeforeAndAfterAll, FunSuite}
import org.apache.carbondata.core.datamap.DataMapStoreManager
@@ -998,9 +999,19 @@ class SparkCarbonDataSourceTest extends FunSuite with
BeforeAndAfterAll {
i += 1
}
writer.close()
- spark.sql("create table complextable (stringfield string, structfield
struct) " +
-s"using carbon location '$path'")
+ if (SparkUtil.isSparkVersionEqualTo("2.1")) {
+if (!FileFactory.isFileExist(path)) {
+ FileFactory.createDirectoryAndSetPermission(path,
+new FsPermission(FsAction.ALL, FsAction.ALL, FsAction.ALL))
+}
+spark.sql("create table complextable (stringfield string, structfield
struct) " +
+ s"using carbon options(path '$path')")
+ } else if (SparkUtil.isSparkVersionXandAbove("2.2")) {
+spark.sql("create table complextable (stringfield string, structfield
struct) " +
+ s"using carbon location '$path'")
+ }
} catch {
case ex: Exception => throw new RuntimeException(ex)
case _ => None
@@ -1052,13 +1063,33 @@ class SparkCarbonDataSourceTest extends FunSuite with
BeforeAndAfterAll {
writer.close()
spark.sql("drop table if exists sorted_par")
spark.sql("drop table if exists sort_table")
- spark.sql(s"create table sort_table (age int, height double, name
string, address string," +
-s" salary long, bytefield byte) using carbon location '$path'")
-
FileFactory.deleteAllCarbonFilesOfDir(FileFactory.getCarbonFile(s"$warehouse1/../warehouse2"))
- spark.sql(s"create table sorted_par(age int, height double, name string,
address " +
-s"string," +
-s"salary long, bytefield byte) using parquet location " +
-s"'$warehouse1/../warehouse2'")
+ val path2 = s"$warehouse1/../warehouse2";
+ if (SparkUtil.isSparkVersionEqualTo("2.1")) {
+if (!FileFactory.isFileExist(path)) {
+ FileFactory.createDirectoryAndSetPermission(path,
+new FsPermission(FsAction.ALL, FsAction.ALL, FsAction.ALL))
+}
+spark.sql(s"create table sort_table (age int, height double, name
string, address string," +
+ s" salary long, bytefield byte) using carbon options(path '$path')")
+
FileFactory.deleteAllCarbonFilesOfDir(FileFactory.getCarbonFi
svn commit: r31589 - /dev/carbondata/KEYS
Author: raghunandan Date: Tue Dec 18 11:59:10 2018 New Revision: 31589 Log: Raghunandan key Modified: dev/carbondata/KEYS Modified: dev/carbondata/KEYS == --- dev/carbondata/KEYS (original) +++ dev/carbondata/KEYS Tue Dec 18 11:59:10 2018 @@ -233,3 +233,40 @@ DxVW1jcI+55rpRS9qJu0IuguD6VJR74h6QXH4sRm zPKxJZEBAdX4e9GCJqEkKWT266LVHBlpHvpA84kx4gMsoW7my12HzVPk =AvWt -END PGP PUBLIC KEY BLOCK- +pub rsa2048 2018-12-18 [SC] [expires: 2020-12-17] + B1913C9DA588D0C9E007EF9FBB0D2966FD6BFAF0 +uid [ultimate] Raghunandan +sig 3BB0D2966FD6BFAF0 2018-12-18 Raghunandan +sub rsa2048 2018-12-18 [E] [expires: 2020-12-17] +sig BB0D2966FD6BFAF0 2018-12-18 Raghunandan + +-BEGIN PGP PUBLIC KEY BLOCK- + +mQENBFwY2AQBCAC/Gr3xiPF4XEr1Ua34Z+HxTkAIWrDTTDIphda3agAz4BlqCof0 +iK5wjz9/sTouK1v/wdKpzuSu5A9fe6LzXvmhtc6ooB+caYSXWjRFtHT2WCCEC2L1 +3h1EIxLaTK9o+JSZ298sdvpn/LbfVx5gMrPvrGhML7U8AoTbgLFzOezCHAySd8nE +0TP6zSWNwTL/1zW76iTZWsUn5e/KiMXmACUuM3bK26bJkTtwwi7fXvuWujIF+2l/ +gYJ0DS18eBbrwHrk4IBoM83LoKdJ3+ydFfhQho0Ie0zlHNHMe57PG81BaqgEuJtZ +VUD3WjdvBtaOX9qO0cgio6GcDsJjQYcTId+lABEBAAG0JFJhZ2h1bmFuZGFuIDxy +YWdodW5hbmRhbkBhcGFjaGUub3JnPokBVAQTAQgAPhYhBLGRPJ2liNDJ4Afvn7sN +KWb9a/rwBQJcGNgEAhsDBQkDwmcABQsJCAcCBhUKCQgLAgQWAgMBAh4BAheAAAoJ +ELsNKWb9a/rwr4gIAJRA75SC1uOnDRcrYzY8XVNIe3ZPOoZs/Lf65ZKwefEu+a+S +4WD+FKFldrwYBFJsKZHOqm6R7rgxyF8TRdIXxk9MjdG+bEE9Z+GBLFBiVBBcrkeQ +zpQezOCn1mCmUxZloi5zOU2vW1UMxxD/O1ZWxofhtASeefMZ19BLwCF295E9Aymg +5iK26zYu/EGqfk3DFAMHzNixJd6ZbxjC+DKR6ReFbXm2X58VzZrWQRqZ6uj0rXkS +hLYED1VzKTcseKVdhFJvOjmRj/IZ/30iXACoLFV2xojEX3b2gkv2yhF2A14cyB6J +Hv1CRb6R85pWq8HeTwfzPGmVNgPIQPj1lHbxq/O5AQ0EXBjYBAEIAL01OSTOIZH8 +sLOpxNvPR6Amw0xJu3Kg8PzpFxAchVJd+n7PRRWT7a3U7uwCXCVwBZi+pBi2fn2v +7YIgmzPtAtCvPryqSr9gMklvZ6bguLX89KWoZbC3RnYe3qjTjXRzG5A8iP29PYOO +NgWOsynoQwrFQ4yKuEfD7EYAhIVkFuuLbdoKnfGQQvdZFdly1lWZI50lQgftXsh0 +kKp3PBjWH6iPzE5GKbJkROQUr6ak7O6ND9uOkAEj7M0wu52xR3KRFElNri192TDz +qGojrAID4tyQccfKJ/+pk39qi5C7e8h1UGjYqv45gxaySoUabjZHL93mMhFWZBoy +nKdDw/PPKz0AEQEAAYkBPAQYAQgAJhYhBLGRPJ2liNDJ4Afvn7sNKWb9a/rwBQJc +GNgEAhsMBQkDwmcAAAoJELsNKWb9a/rwVeAH/10BF9GvnymR7kZzZ5YtWx2TXWsq +kFwKlWOddlTB3iq2HbdYJofAJI+mx69Q+s0dv0zXrDkDqePrm57sYRP7V2pMD044 +QurqMEtEkGBmXgTcpzXBYtumqV/s2BBDtkKTZrPw4iymLD5JxpEtr7byJY69QypQ +Gw6qvZ3F7MIcabL4WipUmUtIrqz9ozeQVGfw6pLy3G3SN5TjbK8YpEHoKgiZGzD9 +KSjRTEbZHl/fZbuhlJ9Ed9P+hoVcEep6iwIwFbM+RDcVE64ySNtW7ISSgt0tl137 +Z/U3Rvql1b/ibImt0cHf/C9OrxynJtJYgKQPP/fZh/Q2f3+vjpDmtGwYA7E= +=8soE +-END PGP PUBLIC KEY BLOCK-
[carbondata] branch master updated: [CARBONDATA-3215] Optimize the documentation
This is an automated email from the ASF dual-hosted git repository.
raghunandan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/carbondata.git
The following commit(s) were added to refs/heads/master by this push:
new b828d0d [CARBONDATA-3215] Optimize the documentation
b828d0d is described below
commit b828d0da9c5b0d63b7285f7d801edc4f0a949f5a
Author: xubo245
AuthorDate: Fri Dec 28 20:37:16 2018 +0800
[CARBONDATA-3215] Optimize the documentation
When user use the Global dictionary, local dictionary，non-dictionary in the
code,
users maybe have some confusion. The same for mvdataMap and IndexDataMap. I
describe and list it in this PR.
1.describe Global dictionary, local dictionary，non-dictionary together in
doc
2.list mvdataMap and IndexDataMap
This closes #3033
---
docs/datamap-developer-guide.md | 8 +-
docs/ddl-of-carbondata.md | 166
2 files changed, 89 insertions(+), 85 deletions(-)
diff --git a/docs/datamap-developer-guide.md b/docs/datamap-developer-guide.md
index c74aa1b..e1fa355 100644
--- a/docs/datamap-developer-guide.md
+++ b/docs/datamap-developer-guide.md
@@ -19,16 +19,16 @@
### Introduction
DataMap is a data structure that can be used to accelerate certain query of
the table. Different DataMap can be implemented by developers.
-Currently, there are two 2 types of DataMap supported:
-1. IndexDataMap: DataMap that leverages index to accelerate filter query
-2. MVDataMap: DataMap that leverages Materialized View to accelerate OLAP
style query, like SPJG query (select, predicate, join, groupby)
+Currently, there are two types of DataMap supported:
+1. IndexDataMap: DataMap that leverages index to accelerate filter query.
Lucene DataMap and BloomFiler DataMap belong to this type of DataMaps.
+2. MVDataMap: DataMap that leverages Materialized View to accelerate olap
style query, like SPJG query (select, predicate, join, groupby). Preaggregate,
timeseries and mv DataMap belong to this type of DataMaps.
### DataMap Provider
When user issues `CREATE DATAMAP dm ON TABLE main USING 'provider'`, the
corresponding DataMapProvider implementation will be created and initialized.
Currently, the provider string can be:
1. preaggregate: A type of MVDataMap that do pre-aggregate of single table
2. timeseries: A type of MVDataMap that do pre-aggregate based on time
dimension of the table
-3. class name IndexDataMapFactory implementation: Developer can implement new
type of IndexDataMap by extending IndexDataMapFactory
+3. class name IndexDataMapFactory implementation: Developer can implement new
type of IndexDataMap by extending IndexDataMapFactory
When user issues `DROP DATAMAP dm ON TABLE main`, the corresponding
DataMapProvider interface will be called.
diff --git a/docs/ddl-of-carbondata.md b/docs/ddl-of-carbondata.md
index aaa2eda..b9b391b 100644
--- a/docs/ddl-of-carbondata.md
+++ b/docs/ddl-of-carbondata.md
@@ -21,13 +21,13 @@ CarbonData DDL statements are documented here,which
includes:
* [CREATE TABLE](#create-table)
* [Dictionary Encoding](#dictionary-encoding-configuration)
+ * [Local Dictionary](#local-dictionary-configuration)
* [Inverted Index](#inverted-index-configuration)
* [Sort Columns](#sort-columns-configuration)
* [Sort Scope](#sort-scope-configuration)
* [Table Block Size](#table-block-size-configuration)
* [Table Compaction](#table-compaction-configuration)
* [Streaming](#streaming)
- * [Local Dictionary](#local-dictionary-configuration)
* [Caching Column Min/Max](#caching-minmax-value-for-required-columns)
* [Caching Level](#caching-at-block-or-blocklet-level)
* [Hive/Parquet folder Structure](#support-flat-folder-same-as-hiveparquet)
@@ -121,8 +121,91 @@ CarbonData DDL statements are documented here,which
includes:
TBLPROPERTIES ('DICTIONARY_INCLUDE'='column1, column2')
```
- **NOTE**: Dictionary Include/Exclude for complex child columns is not
supported.
+ **NOTE**:
+ * Dictionary Include/Exclude for complex child columns is not supported.
+ * Dictionary is global. Except global dictionary, there are local
dictionary and non-dictionary in CarbonData.
+
+ - # Local Dictionary Configuration
+
+ Columns for which dictionary is not generated needs more storage space and
in turn more IO. Also since more data will have to be read during query, query
performance also would suffer.Generating dictionary per blocklet for such
columns would help in saving storage space and assist in improving query
performance as carbondata is optimized for handling dictionary encoded columns
more effectively.Generating dictionary internally per blocklet is termed as
local dictionary. Please refer to [...]
+ Local Dictionary helps in:
+ 1. Getting more compression.
+ 2. Filter queries and full scan queries will be
[carbondata] branch master updated: Optimize presto-guide
This is an automated email from the ASF dual-hosted git repository. raghunandan pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/carbondata.git The following commit(s) were added to refs/heads/master by this push: new 6605b54 Optimize presto-guide 6605b54 is described below commit 6605b54d58be5d61032978810832220eb921d35a Author: Liang Chen AuthorDate: Thu Jan 10 23:28:38 2019 +0800 Optimize presto-guide Some users made mistake: copy the assemble jar. Add more description to clarify, need to copy many jars from integration/presto/target/carbondata-presto-x.x.x-SNAPSHOT This closes #3065 --- docs/presto-guide.md | 11 ++- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/docs/presto-guide.md b/docs/presto-guide.md index d44b2f3..8b3a527 100644 --- a/docs/presto-guide.md +++ b/docs/presto-guide.md @@ -28,7 +28,9 @@ This tutorial provides a quick introduction to using current integration/presto ### Installing Presto 1. Download the 0.210 version of Presto using: - `wget https://repo1.maven.org/maven2/com/facebook/presto/presto-server/0.210/presto-server-0.210.tar.gz` + ``` + wget https://repo1.maven.org/maven2/com/facebook/presto/presto-server/0.210/presto-server-0.210.tar.gz + ``` 2. Extract Presto tar file: `tar zxvf presto-server-0.210.tar.gz`. @@ -144,9 +146,7 @@ To run it as a background process. To run it in foreground. ### Start Presto CLI -``` -./presto -``` + To connect to carbondata catalog use the following command: ``` @@ -220,7 +220,8 @@ Now you can use the Presto CLI on the coordinator to query data sources in the c Secondly: Create a folder named 'carbondata' under $PRESTO_HOME$/plugin and copy all jars from carbondata/integration/presto/target/carbondata-presto-x.x.x-SNAPSHOT to $PRESTO_HOME$/plugin/carbondata - + **NOTE:** Copying assemble jar alone will not work, need to copy all jars from integration/presto/target/carbondata-presto-x.x.x-SNAPSHOT + Thirdly: Create a carbondata.properties file under $PRESTO_HOME$/etc/catalog/ containing the following contents: ``` connector.name=carbondata
[carbondata] branch branch-1.5 updated: [maven-release-plugin] prepare release apache-carbondata-1.5.2-rc1
This is an automated email from the ASF dual-hosted git repository. raghunandan pushed a commit to branch branch-1.5 in repository https://gitbox.apache.org/repos/asf/carbondata.git The following commit(s) were added to refs/heads/branch-1.5 by this push: new a8235fa [maven-release-plugin] prepare release apache-carbondata-1.5.2-rc1 a8235fa is described below commit a8235fa3dd2d73497b6a9b7c57fd78fe589cd0cf Author: Raghunandan S AuthorDate: Mon Jan 21 19:53:03 2019 +0530 [maven-release-plugin] prepare release apache-carbondata-1.5.2-rc1 --- assembly/pom.xml | 2 +- common/pom.xml| 2 +- core/pom.xml | 2 +- datamap/bloom/pom.xml | 2 +- datamap/examples/pom.xml | 2 +- datamap/lucene/pom.xml| 2 +- datamap/mv/core/pom.xml | 2 +- datamap/mv/plan/pom.xml | 2 +- examples/spark2/pom.xml | 2 +- format/pom.xml| 2 +- hadoop/pom.xml| 2 +- integration/hive/pom.xml | 2 +- integration/presto/pom.xml| 2 +- integration/spark-common-test/pom.xml | 2 +- integration/spark-common/pom.xml | 2 +- integration/spark-datasource/pom.xml | 2 +- integration/spark2/pom.xml| 2 +- pom.xml | 4 ++-- processing/pom.xml| 2 +- store/sdk/pom.xml | 2 +- streaming/pom.xml | 2 +- tools/cli/pom.xml | 2 +- 22 files changed, 23 insertions(+), 23 deletions(-) diff --git a/assembly/pom.xml b/assembly/pom.xml index a3e1cfd..4b24730 100644 --- a/assembly/pom.xml +++ b/assembly/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.5.2-SNAPSHOT +1.5.2 ../pom.xml diff --git a/common/pom.xml b/common/pom.xml index e10b48b..e0022ef 100644 --- a/common/pom.xml +++ b/common/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.5.2-SNAPSHOT +1.5.2 ../pom.xml diff --git a/core/pom.xml b/core/pom.xml index 6a22d94..e7ebe7d 100644 --- a/core/pom.xml +++ b/core/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.5.2-SNAPSHOT +1.5.2 ../pom.xml diff --git a/datamap/bloom/pom.xml b/datamap/bloom/pom.xml index c245599..456911f 100644 --- a/datamap/bloom/pom.xml +++ b/datamap/bloom/pom.xml @@ -4,7 +4,7 @@ org.apache.carbondata carbondata-parent -1.5.2-SNAPSHOT +1.5.2 ../../pom.xml diff --git a/datamap/examples/pom.xml b/datamap/examples/pom.xml index c1823c2..c427198 100644 --- a/datamap/examples/pom.xml +++ b/datamap/examples/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.5.2-SNAPSHOT +1.5.2 ../../pom.xml diff --git a/datamap/lucene/pom.xml b/datamap/lucene/pom.xml index 74999aa..03527db 100644 --- a/datamap/lucene/pom.xml +++ b/datamap/lucene/pom.xml @@ -4,7 +4,7 @@ org.apache.carbondata carbondata-parent -1.5.2-SNAPSHOT +1.5.2 ../../pom.xml diff --git a/datamap/mv/core/pom.xml b/datamap/mv/core/pom.xml index 3209c91..d432993 100644 --- a/datamap/mv/core/pom.xml +++ b/datamap/mv/core/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.5.2-SNAPSHOT +1.5.2 ../../../pom.xml diff --git a/datamap/mv/plan/pom.xml b/datamap/mv/plan/pom.xml index 7809337..f2efc97 100644 --- a/datamap/mv/plan/pom.xml +++ b/datamap/mv/plan/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.5.2-SNAPSHOT +1.5.2 ../../../pom.xml diff --git a/examples/spark2/pom.xml b/examples/spark2/pom.xml index b47b0af..d723e54 100644 --- a/examples/spark2/pom.xml +++ b/examples/spark2/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.5.2-SNAPSHOT +1.5.2 ../../pom.xml diff --git a/format/pom.xml b/format/pom.xml index e923bfb..7afd6ef 100644 --- a/format/pom.xml +++ b/format/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.5.2-SNAPSHOT +1.5.2 ../pom.xml diff --git a/hadoop/pom.xml b/hadoop/pom.xml index fcba526..c9cdcc7 100644 --- a/hadoop/pom.xml +++ b/hadoop/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.5.2-SNAPSHOT +1.5.2 ../pom.xml diff --git a/integration/hive/pom.xml b/integration/hive/pom.xml index 7b8b06e..a45ab42 100644 --- a/integration/hive/pom.xml +++ b/integration/hive/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.5.2-SNAPSHOT +1.5.2 ../../pom.xml diff --git a/integration/presto/pom.xml b/integration/presto/pom.xml index fd955dd..78a75fc 100644 --- a/integration/presto/pom.xml +++ b/integration/presto/pom.xml
[carbondata] annotated tag apache-carbondata-1.5.2-rc1 created (now 5e7254e)
This is an automated email from the ASF dual-hosted git repository. raghunandan pushed a change to annotated tag apache-carbondata-1.5.2-rc1 in repository https://gitbox.apache.org/repos/asf/carbondata.git. at 5e7254e (tag) tagging a8235fa3dd2d73497b6a9b7c57fd78fe589cd0cf (commit) replaces apache-carbondata-1.5.1-rc2 by Raghunandan S on Mon Jan 21 19:53:38 2019 +0530 - Log - [maven-release-plugin] copy for tag apache-carbondata-1.5.2-rc1 --- No new revisions were added by this update.
[carbondata] branch branch-1.5 updated: [maven-release-plugin] prepare for next development iteration
This is an automated email from the ASF dual-hosted git repository. raghunandan pushed a commit to branch branch-1.5 in repository https://gitbox.apache.org/repos/asf/carbondata.git The following commit(s) were added to refs/heads/branch-1.5 by this push: new 202365e [maven-release-plugin] prepare for next development iteration 202365e is described below commit 202365eb5c60c9b569d200df992829659a3ec2f6 Author: Raghunandan S AuthorDate: Mon Jan 21 19:53:59 2019 +0530 [maven-release-plugin] prepare for next development iteration --- assembly/pom.xml | 2 +- common/pom.xml| 2 +- core/pom.xml | 2 +- datamap/bloom/pom.xml | 2 +- datamap/examples/pom.xml | 2 +- datamap/lucene/pom.xml| 2 +- datamap/mv/core/pom.xml | 2 +- datamap/mv/plan/pom.xml | 2 +- examples/spark2/pom.xml | 2 +- format/pom.xml| 2 +- hadoop/pom.xml| 2 +- integration/hive/pom.xml | 2 +- integration/presto/pom.xml| 2 +- integration/spark-common-test/pom.xml | 2 +- integration/spark-common/pom.xml | 2 +- integration/spark-datasource/pom.xml | 2 +- integration/spark2/pom.xml| 2 +- pom.xml | 4 ++-- processing/pom.xml| 2 +- store/sdk/pom.xml | 2 +- streaming/pom.xml | 2 +- tools/cli/pom.xml | 2 +- 22 files changed, 23 insertions(+), 23 deletions(-) diff --git a/assembly/pom.xml b/assembly/pom.xml index 4b24730..b8af897 100644 --- a/assembly/pom.xml +++ b/assembly/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.5.2 +1.5.3-SNAPSHOT ../pom.xml diff --git a/common/pom.xml b/common/pom.xml index e0022ef..0f01c7b 100644 --- a/common/pom.xml +++ b/common/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.5.2 +1.5.3-SNAPSHOT ../pom.xml diff --git a/core/pom.xml b/core/pom.xml index e7ebe7d..ea93d30 100644 --- a/core/pom.xml +++ b/core/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.5.2 +1.5.3-SNAPSHOT ../pom.xml diff --git a/datamap/bloom/pom.xml b/datamap/bloom/pom.xml index 456911f..9070287 100644 --- a/datamap/bloom/pom.xml +++ b/datamap/bloom/pom.xml @@ -4,7 +4,7 @@ org.apache.carbondata carbondata-parent -1.5.2 +1.5.3-SNAPSHOT ../../pom.xml diff --git a/datamap/examples/pom.xml b/datamap/examples/pom.xml index c427198..e88a906 100644 --- a/datamap/examples/pom.xml +++ b/datamap/examples/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.5.2 +1.5.3-SNAPSHOT ../../pom.xml diff --git a/datamap/lucene/pom.xml b/datamap/lucene/pom.xml index 03527db..6162f01 100644 --- a/datamap/lucene/pom.xml +++ b/datamap/lucene/pom.xml @@ -4,7 +4,7 @@ org.apache.carbondata carbondata-parent -1.5.2 +1.5.3-SNAPSHOT ../../pom.xml diff --git a/datamap/mv/core/pom.xml b/datamap/mv/core/pom.xml index d432993..7924305 100644 --- a/datamap/mv/core/pom.xml +++ b/datamap/mv/core/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.5.2 +1.5.3-SNAPSHOT ../../../pom.xml diff --git a/datamap/mv/plan/pom.xml b/datamap/mv/plan/pom.xml index f2efc97..1210331 100644 --- a/datamap/mv/plan/pom.xml +++ b/datamap/mv/plan/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.5.2 +1.5.3-SNAPSHOT ../../../pom.xml diff --git a/examples/spark2/pom.xml b/examples/spark2/pom.xml index d723e54..d6804c9 100644 --- a/examples/spark2/pom.xml +++ b/examples/spark2/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.5.2 +1.5.3-SNAPSHOT ../../pom.xml diff --git a/format/pom.xml b/format/pom.xml index 7afd6ef..8284565 100644 --- a/format/pom.xml +++ b/format/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.5.2 +1.5.3-SNAPSHOT ../pom.xml diff --git a/hadoop/pom.xml b/hadoop/pom.xml index c9cdcc7..e23c74b 100644 --- a/hadoop/pom.xml +++ b/hadoop/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.5.2 +1.5.3-SNAPSHOT ../pom.xml diff --git a/integration/hive/pom.xml b/integration/hive/pom.xml index a45ab42..e2c1504 100644 --- a/integration/hive/pom.xml +++ b/integration/hive/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.5.2 +1.5.3-SNAPSHOT ../../pom.xml diff --git a/integration/presto/pom.xml b/integration/presto/pom.xml index 78a75fc..0ecd4a3 100644 --- a/integration/presto/pom.xml +++ b/integration/presto/pom.xml @@ -22,7 +22,7
svn commit: r32085 - in /dev/carbondata/1.5.2-rc1: ./ apache-carbondata-1.5.2-source-release.zip apache-carbondata-1.5.2-source-release.zip.asc apache-carbondata-1.5.2-source-release.zip.md5 apache-ca
Author: raghunandan Date: Tue Jan 22 06:30:46 2019 New Revision: 32085 Log: upload for 1.5.2 rc1 Added: dev/carbondata/1.5.2-rc1/ dev/carbondata/1.5.2-rc1/apache-carbondata-1.5.2-source-release.zip (with props) dev/carbondata/1.5.2-rc1/apache-carbondata-1.5.2-source-release.zip.asc dev/carbondata/1.5.2-rc1/apache-carbondata-1.5.2-source-release.zip.md5 dev/carbondata/1.5.2-rc1/apache-carbondata-1.5.2-source-release.zip.sha512 Added: dev/carbondata/1.5.2-rc1/apache-carbondata-1.5.2-source-release.zip == Binary file - no diff available. Propchange: dev/carbondata/1.5.2-rc1/apache-carbondata-1.5.2-source-release.zip -- svn:mime-type = application/octet-stream Added: dev/carbondata/1.5.2-rc1/apache-carbondata-1.5.2-source-release.zip.asc == --- dev/carbondata/1.5.2-rc1/apache-carbondata-1.5.2-source-release.zip.asc (added) +++ dev/carbondata/1.5.2-rc1/apache-carbondata-1.5.2-source-release.zip.asc Tue Jan 22 06:30:46 2019 @@ -0,0 +1,11 @@ +-BEGIN PGP SIGNATURE- + +iQEzBAABCAAdFiEEsZE8naWI0MngB++fuw0pZv1r+vAFAlxFu5QACgkQuw0pZv1r ++vDIkQgAmw3o6nxP8t2kBNlv17OBtJaTsgmhTVI+C8G1xuBG2yNYFVPLLs4+mtJT +n7h+XQD9exNuSroZB460LB5NS9+vLRl8Ns1TzIEWFiWBq5wBBcy+ECfS61hEKxKL +uVMksAdndumTQPz3MlSJTJo9FcXZwnCybPbPNiFbQF1iq/VhocqrDYBLEokEDzru +y7ffcHKPCLJnboOosTQSnmymyv8RrAwLWwAY3VJSQr+h0aGVZRxDRx7lGXn6+IYF +wW6WQqdgVHOf12GpCTzqosmJwtQsyzSW/y0T1AmA85+BkBQDPiTRPi+z5G0c9ehC +RbOLvoGyk6cwb1tuw2Tp4ZH1eD1Ixw== +=ol7B +-END PGP SIGNATURE- Added: dev/carbondata/1.5.2-rc1/apache-carbondata-1.5.2-source-release.zip.md5 == --- dev/carbondata/1.5.2-rc1/apache-carbondata-1.5.2-source-release.zip.md5 (added) +++ dev/carbondata/1.5.2-rc1/apache-carbondata-1.5.2-source-release.zip.md5 Tue Jan 22 06:30:46 2019 @@ -0,0 +1 @@ +08ee94c249ae71418be5c990d413cc20 apache-carbondata-1.5.2-source-release.zip Added: dev/carbondata/1.5.2-rc1/apache-carbondata-1.5.2-source-release.zip.sha512 == --- dev/carbondata/1.5.2-rc1/apache-carbondata-1.5.2-source-release.zip.sha512 (added) +++ dev/carbondata/1.5.2-rc1/apache-carbondata-1.5.2-source-release.zip.sha512 Tue Jan 22 06:30:46 2019 @@ -0,0 +1 @@ +989b59a5deb1581e498e1033d36fd47ac9888512c77637955af772ca1a836f7b8f84faf6d7db7cd64d925c1ad97d3a2c9c8a0157452fd36da7ae317dc5b4 apache-carbondata-1.5.2-source-release.zip
[carbondata] branch master updated: [DOC] Update range_info partition example in ddl description
This is an automated email from the ASF dual-hosted git repository.
raghunandan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/carbondata.git
The following commit(s) were added to refs/heads/master by this push:
new df3d4c8 [DOC] Update range_info partition example in ddl description
df3d4c8 is described below
commit df3d4c8dc9dc8d314fc62ff3816cc42786d1543a
Author: qiuchenjian <807169...@qq.com>
AuthorDate: Tue Jan 22 10:36:30 2019 +0800
[DOC] Update range_info partition example in ddl description
This closes #3092
---
docs/ddl-of-carbondata.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/docs/ddl-of-carbondata.md b/docs/ddl-of-carbondata.md
index b9b391b..a1b0ce7 100644
--- a/docs/ddl-of-carbondata.md
+++ b/docs/ddl-of-carbondata.md
@@ -950,7 +950,7 @@ Users can specify which columns to include and exclude for
local dictionary gene
col_D DECIMAL(10,2),
col_E LONG
) partitioned by (col_F Timestamp)
- PARTITIONED BY 'carbondata'
+ STORED BY 'carbondata'
TBLPROPERTIES('PARTITION_TYPE'='RANGE',
'RANGE_INFO'='2015-01-01, 2016-01-01, 2017-01-01, 2017-02-01')
```
[carbondata] branch master updated: [DOC] Updated Presto guide
This is an automated email from the ASF dual-hosted git repository.
raghunandan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/carbondata.git
The following commit(s) were added to refs/heads/master by this push:
new b8def42 [DOC] Updated Presto guide
b8def42 is described below
commit b8def4226dc18b2192d014dd01c29b0adebd4bf9
Author: ravipesala
AuthorDate: Mon Jan 21 12:27:40 2019 +0530
[DOC] Updated Presto guide
Added more information about CarbonData configurations for Presto and
clearly highlighted the sections in the document
This closes #3088
---
docs/presto-guide.md | 57 +---
1 file changed, 27 insertions(+), 30 deletions(-)
diff --git a/docs/presto-guide.md b/docs/presto-guide.md
index 8b3a527..054f29f 100644
--- a/docs/presto-guide.md
+++ b/docs/presto-guide.md
@@ -101,6 +101,7 @@ This tutorial provides a quick introduction to using
current integration/presto
```
The options `node-scheduler.include-coordinator=false` and `coordinator=true`
indicate that the node is the coordinator and tells the coordinator not to do
any of the computation work itself and to use the workers.
+
**Note**: We recommend setting `query.max-memory-per-node` to half of the JVM
config max memory, though if your workload is highly concurrent, you may want
to use a lower value for `query.max-memory-per-node`.
Also relation between below two configuration-properties should be like:
@@ -167,8 +168,10 @@ Now you can use the Presto CLI on the coordinator to query
data sources in the c
* Download presto server (0.210 is suggested and supported) :
https://repo1.maven.org/maven2/com/facebook/presto/presto-server/
* Finish presto configuration following
https://prestodb.io/docs/current/installation/deployment.html.
A configuration example:
+
+ **config.properties**
+
```
- config.properties:
coordinator=true
node-scheduler.include-coordinator=true
http-server.http.port=8086
@@ -180,10 +183,13 @@ Now you can use the Presto CLI on the coordinator to
query data sources in the c
discovery.uri=http://localhost:8086
task.max-worker-threads=4
optimizer.dictionary-aggregation=true
- optimizer.optimize-hash-generation = false
+ optimizer.optimize-hash-generation = false
+ ```
- jvm.config:
+ **jvm.config**
+
+ ```
-server
-Xmx4G
-XX:+UseG1GC
@@ -193,12 +199,20 @@ Now you can use the Presto CLI on the coordinator to
query data sources in the c
-XX:+HeapDumpOnOutOfMemoryError
-XX:OnOutOfMemoryError=kill -9 %p
-XX:+TraceClassLoading
+ -Dcarbon.properties.filepath=/carbon.properties
- log.properties:
+ ```
+ `carbon.properties.filepath` property is used to set the carbon.properties
file path and it is recommended to set otherwise some features may not work.
Please check the above example.
+
+
+ **log.properties**
+ ```
com.facebook.presto=DEBUG
com.facebook.presto.server.PluginManager=DEBUG
+ ```
- node.properties:
+ **node.properties**
+ ```
node.environment=carbondata
node.id=----
node.data-dir=/Users/apple/DEMO/presto_test/data
@@ -220,21 +234,20 @@ Now you can use the Presto CLI on the coordinator to
query data sources in the c
Secondly: Create a folder named 'carbondata' under $PRESTO_HOME$/plugin and
copy all jars from
carbondata/integration/presto/target/carbondata-presto-x.x.x-SNAPSHOT
to $PRESTO_HOME$/plugin/carbondata
+
**NOTE:** Copying assemble jar alone will not work, need to copy all jars
from integration/presto/target/carbondata-presto-x.x.x-SNAPSHOT
Thirdly: Create a carbondata.properties file under
$PRESTO_HOME$/etc/catalog/ containing the following contents:
```
connector.name=carbondata
- carbondata-store={schema-store-path}
- enable.unsafe.in.query.processing=false
- carbon.unsafe.working.memory.in.mb={value}
- enable.unsafe.columnpage=false
- enable.unsafe.sort=false
-
+ hive.metastore.uri=thrift://:
```
- Replace the schema-store-path with the absolute path of the parent directory
of the schema.
- For example, if you have a schema named 'default' stored in
hdfs://namenode:9000/test/carbondata/,
- Then set carbondata-store=hdfs://namenode:9000/test/carbondata
+ Carbondata becomes one of the supported format of presto hive plugin, so the
configurations and setup is similar to hive connector of presto.
+ Please refer https://prestodb.io/docs/current/connector/hive.html for
more details.
+
+ **Note**: Since carbon can work only with hive metastore, it is necessary
that spark also connects to same metastore db for creating tables and updating
tables.
+ All the operations done on spark will be reflected in presto immediately.
+ It is mandatory to create Carbon tables from spark using CarbonData 1.5.2 or
greater version since input/output formats are updated in carbon table
[carbondata] branch master updated: [CARBONDATA-3263] Update doc for RANGE_COLUMN
This is an automated email from the ASF dual-hosted git repository.
raghunandan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/carbondata.git
The following commit(s) were added to refs/heads/master by this push:
new 134df2b [CARBONDATA-3263] Update doc for RANGE_COLUMN
134df2b is described below
commit 134df2bcae37537990603a2a1e26318357233eae
Author: QiangCai
AuthorDate: Tue Jan 22 11:27:04 2019 +0800
[CARBONDATA-3263] Update doc for RANGE_COLUMN
Added documentation for range_column feature support
This closes #3093
---
docs/ddl-of-carbondata.md | 12 +++-
docs/dml-of-carbondata.md | 23 ---
2 files changed, 31 insertions(+), 4 deletions(-)
diff --git a/docs/ddl-of-carbondata.md b/docs/ddl-of-carbondata.md
index a1b0ce7..4f9e47b 100644
--- a/docs/ddl-of-carbondata.md
+++ b/docs/ddl-of-carbondata.md
@@ -34,7 +34,8 @@ CarbonData DDL statements are documented here,which includes:
* [Extra Long String columns](#string-longer-than-32000-characters)
* [Compression for Table](#compression-for-table)
* [Bad Records Path](#bad-records-path)
- * [Load Minimum Input File Size](#load-minimum-data-size)
+ * [Load Minimum Input File Size](#load-minimum-data-size)
+ * [Range Column](#range-column)
* [CREATE TABLE AS SELECT](#create-table-as-select)
* [CREATE EXTERNAL TABLE](#create-external-table)
@@ -109,6 +110,7 @@ CarbonData DDL statements are documented here,which
includes:
| [BUCKETNUMBER](#bucketing) | Number of
buckets to be created |
| [BUCKETCOLUMNS](#bucketing) | Columns which
are to be placed in buckets|
| [LOAD_MIN_SIZE_INMB](#load-minimum-data-size)| Minimum input
data size per node for data loading |
+| [Range Column](#range-column)| partition
input data by range |
Following are the guidelines for TBLPROPERTIES, CarbonData's additional table
options can be set via carbon.properties.
@@ -495,6 +497,14 @@ CarbonData DDL statements are documented here,which
includes:
TBLPROPERTIES('LOAD_MIN_SIZE_INMB'='256')
```
+ - # Range Column
+ This property is used to specify a column to partition the input data by
range.
+ Only one column can be configured. During data loading, you can use
"global_sort_partitions" or "scale_factor" to avoid generating small files.
+
+ ```
+ TBLPROPERTIES('RANGE_COLUMN'='col1')
+ ```
+
## CREATE TABLE AS SELECT
This function allows user to create a Carbon table from any of the
Parquet/Hive/Carbon table. This is beneficial when the user wants to create
Carbon table from any other Parquet/Hive table and use the Carbon query engine
to query and achieve better query results for cases where Carbon is faster than
other file formats. Also this feature can be used for backing up the data.
diff --git a/docs/dml-of-carbondata.md b/docs/dml-of-carbondata.md
index d6e5932..b3fe517 100644
--- a/docs/dml-of-carbondata.md
+++ b/docs/dml-of-carbondata.md
@@ -66,7 +66,8 @@ CarbonData DML statements are documented here,which includes:
| [BAD_RECORDS_ACTION](#bad-records-handling) | Behavior of data
loading when bad record is found|
| [IS_EMPTY_DATA_BAD_RECORD](#bad-records-handling) | Whether empty data
of a column to be considered as bad record or not |
| [GLOBAL_SORT_PARTITIONS](#global_sort_partitions) | Number of
partition to use for shuffling of data during sorting |
-
+| [SCALE_FACTOR](#scale_factor) | Control the
partition size for RANGE_COLUMN feature |
+-
You can use the following options to load data:
- # DELIMITER:
@@ -268,15 +269,31 @@ CarbonData DML statements are documented here,which
includes:
- # GLOBAL_SORT_PARTITIONS:
If the SORT_SCOPE is defined as GLOBAL_SORT, then user can specify the
number of partitions to use while shuffling data for sort using
GLOBAL_SORT_PARTITIONS. If it is not configured, or configured less than 1,
then it uses the number of map task as reduce task. It is recommended that each
reduce task deal with 512MB-1GB data.
-
+For RANGE_COLUMN, GLOBAL_SORT_PARTITIONS is used to specify the number of
range partitions also.
```
OPTIONS('GLOBAL_SORT_PARTITIONS'='2')
```
- NOTE:
+ **NOTE:**
* GLOBAL_SORT_PARTITIONS should be Integer type, the range is
[1,Integer.MaxValue].
* It is only used when the SORT_SCOPE is GLOBAL_SORT.
+ - # SCALE_FACTOR
+
+ For RANGE_COLUMN, SCALE_FACTOR is used to control the number of range
partitions as following.
+ ```
+ splitSize = max(blocklet_size, (block_size - blocklet_size)) *
scale_factor
+ numPartitions = total s
[carbondata] branch master updated: [CARBONDATA-3232] Add example and doc for alluxio integration
This is an automated email from the ASF dual-hosted git repository.
raghunandan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/carbondata.git
The following commit(s) were added to refs/heads/master by this push:
new 028eb25 [CARBONDATA-3232] Add example and doc for alluxio integration
028eb25 is described below
commit 028eb25bea490c53aa8736da0dce2f7e8785ca97
Author: xubo245
AuthorDate: Mon Jan 7 20:27:37 2019 +0800
[CARBONDATA-3232] Add example and doc for alluxio integration
Optimize carbonData usage with alluxio:
1.Add doc
2.optimize the example
This closes #3054
---
README.md | 1 +
docs/alluxio-guide.md | 136 +
docs/documentation.md | 6 +-
docs/introduction.md | 4 +-
docs/quick-start-guide.md | 17 ++-
examples/spark2/pom.xml| 10 ++
.../carbondata/examples/AlluxioExample.scala | 115 -
.../carbondata/examples/util/ExampleUtils.scala| 13 +-
8 files changed, 264 insertions(+), 38 deletions(-)
diff --git a/README.md b/README.md
index a788cea..bed906f 100644
--- a/README.md
+++ b/README.md
@@ -70,6 +70,7 @@ CarbonData is built using Apache Maven, to [build
CarbonData](https://github.com
## Integration
* [Hive](https://github.com/apache/carbondata/blob/master/docs/hive-guide.md)
*
[Presto](https://github.com/apache/carbondata/blob/master/docs/presto-guide.md)
+*
[Alluxio](https://github.com/apache/carbondata/blob/master/docs/alluxio-guide.md)
## Other Technical Material
* [Apache CarbonData meetup
material](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=66850609)
diff --git a/docs/alluxio-guide.md b/docs/alluxio-guide.md
new file mode 100644
index 000..b1bfeeb
--- /dev/null
+++ b/docs/alluxio-guide.md
@@ -0,0 +1,136 @@
+
+
+
+# Alluxio guide
+This tutorial provides a brief introduction to using Alluxio.
+ - How to use Alluxio in CarbonData?
+- [Running alluxio example in CarbonData project by IDEA](#Running alluxio
example in CarbonData project by IDEA)
+- [CarbonData supports alluxio by spark-shell](#CarbonData supports
alluxio by spark-shell)
+- [CarbonData supports alluxio by spark-submit](#CarbonData supports
alluxio by spark-submit)
+
+## Running alluxio example in CarbonData project by IDEA
+
+### [Building
CarbonData](https://github.com/apache/carbondata/tree/master/build)
+ - Please refer to [Building
CarbonData](https://github.com/apache/carbondata/tree/master/build).
+ - Users need to install IDEA and scala plugin, and import CarbonData project.
+
+### Installing and starting Alluxio
+ - Please refer to
[https://www.alluxio.org/docs/1.8/en/Getting-Started.html#starting-alluxio](https://www.alluxio.org/docs/1.8/en/Getting-Started.html#starting-alluxio)
+ - Access the Alluxio web:
[http://localhost:1/home](http://localhost:1/home)
+
+### Running Example
+ - Please refer to
[AlluxioExample](https://github.com/apache/carbondata/blob/master/examples/spark2/src/main/scala/org/apache/carbondata/examples/AlluxioExample.scala)
+
+## CarbonData supports alluxio by spark-shell
+
+### [Building
CarbonData](https://github.com/apache/carbondata/tree/master/build)
+ - Please refer to [Building
CarbonData](https://github.com/apache/carbondata/tree/master/build).
+
+### Preparing Spark
+ - Please refer to
[http://spark.apache.org/docs/latest/](http://spark.apache.org/docs/latest/)
+
+### Downloading alluxio and uncompressing it
+ - Please refer to
[https://www.alluxio.org/download](https://www.alluxio.org/download)
+
+### Running spark-shell
+ - Running the command in spark path
+ ```$command
+./bin/spark-shell --jars
${CARBONDATA_PATH}/assembly/target/scala-2.11/apache-carbondata-1.6.0-SNAPSHOT-bin-spark2.2.1-hadoop2.7.2.jar,${ALLUXIO_PATH}/client/alluxio-1.8.1-client.jar
+```
+ - Testing use alluxio by CarbonSession
+ ```$scala
+import org.apache.spark.sql.CarbonSession._
+import org.apache.spark.sql.SparkSession
+
+val carbon =
SparkSession.builder().master("local").appName("test").getOrCreateCarbonSession("alluxio://localhost:19998/carbondata");
+carbon.sql("CREATE TABLE carbon_alluxio(id String,name String, city String,age
Int) STORED as carbondata");
+carbon.sql(s"LOAD DATA LOCAL INPATH
'${CARBONDATA_PATH}/integration/spark-common-test/src/test/resources/sample.csv'
into table carbon_alluxio");
+carbon.sql("select * from carbon_alluxio").show
+```
+ - Result
+ ```$scala
+ scala> carbon.sql("select * from carbon_alluxio").show
+ +---+--+-+---+
+ | id| name| city|age|
+ +---+--+-+---+
+ | 1| david| shenzhen| 31|
+ | 2| eason| shenzhen| 27|
+ | 3| jarry|wuhan| 35|
+
[carbondata] branch branch-1.5 updated: [maven-release-plugin] prepare release apache-carbondata-1.5.2-rc2
This is an automated email from the ASF dual-hosted git repository. raghunandan pushed a commit to branch branch-1.5 in repository https://gitbox.apache.org/repos/asf/carbondata.git The following commit(s) were added to refs/heads/branch-1.5 by this push: new 9e0ff5e [maven-release-plugin] prepare release apache-carbondata-1.5.2-rc2 9e0ff5e is described below commit 9e0ff5e4c06fecd2dc9253d6e02093f123f2e71b Author: Raghunandan S AuthorDate: Wed Jan 30 17:35:55 2019 +0530 [maven-release-plugin] prepare release apache-carbondata-1.5.2-rc2 --- assembly/pom.xml | 2 +- common/pom.xml| 2 +- core/pom.xml | 2 +- datamap/bloom/pom.xml | 2 +- datamap/examples/pom.xml | 2 +- datamap/lucene/pom.xml| 2 +- datamap/mv/core/pom.xml | 2 +- datamap/mv/plan/pom.xml | 2 +- examples/spark2/pom.xml | 2 +- format/pom.xml| 2 +- hadoop/pom.xml| 2 +- integration/hive/pom.xml | 2 +- integration/presto/pom.xml| 2 +- integration/spark-common-test/pom.xml | 2 +- integration/spark-common/pom.xml | 2 +- integration/spark-datasource/pom.xml | 2 +- integration/spark2/pom.xml| 2 +- pom.xml | 4 ++-- processing/pom.xml| 2 +- store/sdk/pom.xml | 2 +- streaming/pom.xml | 2 +- tools/cli/pom.xml | 2 +- 22 files changed, 23 insertions(+), 23 deletions(-) diff --git a/assembly/pom.xml b/assembly/pom.xml index a3e1cfd..4b24730 100644 --- a/assembly/pom.xml +++ b/assembly/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.5.2-SNAPSHOT +1.5.2 ../pom.xml diff --git a/common/pom.xml b/common/pom.xml index e386d2b..e9b482b 100644 --- a/common/pom.xml +++ b/common/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.5.2-SNAPSHOT +1.5.2 ../pom.xml diff --git a/core/pom.xml b/core/pom.xml index 6a22d94..e7ebe7d 100644 --- a/core/pom.xml +++ b/core/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.5.2-SNAPSHOT +1.5.2 ../pom.xml diff --git a/datamap/bloom/pom.xml b/datamap/bloom/pom.xml index c245599..456911f 100644 --- a/datamap/bloom/pom.xml +++ b/datamap/bloom/pom.xml @@ -4,7 +4,7 @@ org.apache.carbondata carbondata-parent -1.5.2-SNAPSHOT +1.5.2 ../../pom.xml diff --git a/datamap/examples/pom.xml b/datamap/examples/pom.xml index c1823c2..c427198 100644 --- a/datamap/examples/pom.xml +++ b/datamap/examples/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.5.2-SNAPSHOT +1.5.2 ../../pom.xml diff --git a/datamap/lucene/pom.xml b/datamap/lucene/pom.xml index 74999aa..03527db 100644 --- a/datamap/lucene/pom.xml +++ b/datamap/lucene/pom.xml @@ -4,7 +4,7 @@ org.apache.carbondata carbondata-parent -1.5.2-SNAPSHOT +1.5.2 ../../pom.xml diff --git a/datamap/mv/core/pom.xml b/datamap/mv/core/pom.xml index 3209c91..d432993 100644 --- a/datamap/mv/core/pom.xml +++ b/datamap/mv/core/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.5.2-SNAPSHOT +1.5.2 ../../../pom.xml diff --git a/datamap/mv/plan/pom.xml b/datamap/mv/plan/pom.xml index 7809337..f2efc97 100644 --- a/datamap/mv/plan/pom.xml +++ b/datamap/mv/plan/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.5.2-SNAPSHOT +1.5.2 ../../../pom.xml diff --git a/examples/spark2/pom.xml b/examples/spark2/pom.xml index e233226..29359f4 100644 --- a/examples/spark2/pom.xml +++ b/examples/spark2/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.5.2-SNAPSHOT +1.5.2 ../../pom.xml diff --git a/format/pom.xml b/format/pom.xml index ecc9cc7..a7a7a0d 100644 --- a/format/pom.xml +++ b/format/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.5.2-SNAPSHOT +1.5.2 ../pom.xml diff --git a/hadoop/pom.xml b/hadoop/pom.xml index fcba526..c9cdcc7 100644 --- a/hadoop/pom.xml +++ b/hadoop/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.5.2-SNAPSHOT +1.5.2 ../pom.xml diff --git a/integration/hive/pom.xml b/integration/hive/pom.xml index 7b8b06e..a45ab42 100644 --- a/integration/hive/pom.xml +++ b/integration/hive/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.5.2-SNAPSHOT +1.5.2 ../../pom.xml diff --git a/integration/presto/pom.xml b/integration/presto/pom.xml index 3d52041..75631c2 100644 --- a/integration/presto/pom.xml +++ b/integration/presto/pom.xml
[carbondata] annotated tag apache-carbondata-1.5.2-rc2 created (now aab4c10)
This is an automated email from the ASF dual-hosted git repository. raghunandan pushed a change to annotated tag apache-carbondata-1.5.2-rc2 in repository https://gitbox.apache.org/repos/asf/carbondata.git. at aab4c10 (tag) tagging 9e0ff5e4c06fecd2dc9253d6e02093f123f2e71b (commit) replaces apache-carbondata-1.5.1-rc2 by Raghunandan S on Wed Jan 30 17:36:59 2019 +0530 - Log - [maven-release-plugin] copy for tag apache-carbondata-1.5.2-rc2 --- No new revisions were added by this update.
[carbondata] branch branch-1.5 updated: [maven-release-plugin] prepare for next development iteration
This is an automated email from the ASF dual-hosted git repository. raghunandan pushed a commit to branch branch-1.5 in repository https://gitbox.apache.org/repos/asf/carbondata.git The following commit(s) were added to refs/heads/branch-1.5 by this push: new 441edbb [maven-release-plugin] prepare for next development iteration 441edbb is described below commit 441edbb0fbe667c7d345fc42dececf8619ff0bef Author: Raghunandan S AuthorDate: Wed Jan 30 17:37:20 2019 +0530 [maven-release-plugin] prepare for next development iteration --- assembly/pom.xml | 2 +- common/pom.xml| 2 +- core/pom.xml | 2 +- datamap/bloom/pom.xml | 2 +- datamap/examples/pom.xml | 2 +- datamap/lucene/pom.xml| 2 +- datamap/mv/core/pom.xml | 2 +- datamap/mv/plan/pom.xml | 2 +- examples/spark2/pom.xml | 2 +- format/pom.xml| 2 +- hadoop/pom.xml| 2 +- integration/hive/pom.xml | 2 +- integration/presto/pom.xml| 2 +- integration/spark-common-test/pom.xml | 2 +- integration/spark-common/pom.xml | 2 +- integration/spark-datasource/pom.xml | 2 +- integration/spark2/pom.xml| 2 +- pom.xml | 4 ++-- processing/pom.xml| 2 +- store/sdk/pom.xml | 2 +- streaming/pom.xml | 2 +- tools/cli/pom.xml | 2 +- 22 files changed, 23 insertions(+), 23 deletions(-) diff --git a/assembly/pom.xml b/assembly/pom.xml index 4b24730..b8af897 100644 --- a/assembly/pom.xml +++ b/assembly/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.5.2 +1.5.3-SNAPSHOT ../pom.xml diff --git a/common/pom.xml b/common/pom.xml index e9b482b..f10ef34 100644 --- a/common/pom.xml +++ b/common/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.5.2 +1.5.3-SNAPSHOT ../pom.xml diff --git a/core/pom.xml b/core/pom.xml index e7ebe7d..ea93d30 100644 --- a/core/pom.xml +++ b/core/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.5.2 +1.5.3-SNAPSHOT ../pom.xml diff --git a/datamap/bloom/pom.xml b/datamap/bloom/pom.xml index 456911f..9070287 100644 --- a/datamap/bloom/pom.xml +++ b/datamap/bloom/pom.xml @@ -4,7 +4,7 @@ org.apache.carbondata carbondata-parent -1.5.2 +1.5.3-SNAPSHOT ../../pom.xml diff --git a/datamap/examples/pom.xml b/datamap/examples/pom.xml index c427198..e88a906 100644 --- a/datamap/examples/pom.xml +++ b/datamap/examples/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.5.2 +1.5.3-SNAPSHOT ../../pom.xml diff --git a/datamap/lucene/pom.xml b/datamap/lucene/pom.xml index 03527db..6162f01 100644 --- a/datamap/lucene/pom.xml +++ b/datamap/lucene/pom.xml @@ -4,7 +4,7 @@ org.apache.carbondata carbondata-parent -1.5.2 +1.5.3-SNAPSHOT ../../pom.xml diff --git a/datamap/mv/core/pom.xml b/datamap/mv/core/pom.xml index d432993..7924305 100644 --- a/datamap/mv/core/pom.xml +++ b/datamap/mv/core/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.5.2 +1.5.3-SNAPSHOT ../../../pom.xml diff --git a/datamap/mv/plan/pom.xml b/datamap/mv/plan/pom.xml index f2efc97..1210331 100644 --- a/datamap/mv/plan/pom.xml +++ b/datamap/mv/plan/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.5.2 +1.5.3-SNAPSHOT ../../../pom.xml diff --git a/examples/spark2/pom.xml b/examples/spark2/pom.xml index 29359f4..1a07edf 100644 --- a/examples/spark2/pom.xml +++ b/examples/spark2/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.5.2 +1.5.3-SNAPSHOT ../../pom.xml diff --git a/format/pom.xml b/format/pom.xml index a7a7a0d..ceaea40 100644 --- a/format/pom.xml +++ b/format/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.5.2 +1.5.3-SNAPSHOT ../pom.xml diff --git a/hadoop/pom.xml b/hadoop/pom.xml index c9cdcc7..e23c74b 100644 --- a/hadoop/pom.xml +++ b/hadoop/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.5.2 +1.5.3-SNAPSHOT ../pom.xml diff --git a/integration/hive/pom.xml b/integration/hive/pom.xml index a45ab42..e2c1504 100644 --- a/integration/hive/pom.xml +++ b/integration/hive/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.5.2 +1.5.3-SNAPSHOT ../../pom.xml diff --git a/integration/presto/pom.xml b/integration/presto/pom.xml index 75631c2..91221d6 100644 --- a/integration/presto/pom.xml +++ b/integration/presto/pom.xml @@ -22,7 +22,7
svn commit: r32231 - in /dev/carbondata/1.5.2-rc2: ./ apache-carbondata-1.5.2-source-release.zip apache-carbondata-1.5.2-source-release.zip.asc apache-carbondata-1.5.2-source-release.zip.md5 apache-ca
Author: raghunandan Date: Wed Jan 30 16:41:50 2019 New Revision: 32231 Log: upload 1.5.2-rc2 Added: dev/carbondata/1.5.2-rc2/ dev/carbondata/1.5.2-rc2/apache-carbondata-1.5.2-source-release.zip (with props) dev/carbondata/1.5.2-rc2/apache-carbondata-1.5.2-source-release.zip.asc dev/carbondata/1.5.2-rc2/apache-carbondata-1.5.2-source-release.zip.md5 dev/carbondata/1.5.2-rc2/apache-carbondata-1.5.2-source-release.zip.sha512 Added: dev/carbondata/1.5.2-rc2/apache-carbondata-1.5.2-source-release.zip == Binary file - no diff available. Propchange: dev/carbondata/1.5.2-rc2/apache-carbondata-1.5.2-source-release.zip -- svn:mime-type = application/octet-stream Added: dev/carbondata/1.5.2-rc2/apache-carbondata-1.5.2-source-release.zip.asc == --- dev/carbondata/1.5.2-rc2/apache-carbondata-1.5.2-source-release.zip.asc (added) +++ dev/carbondata/1.5.2-rc2/apache-carbondata-1.5.2-source-release.zip.asc Wed Jan 30 16:41:50 2019 @@ -0,0 +1,11 @@ +-BEGIN PGP SIGNATURE- + +iQEzBAABCAAdFiEEsZE8naWI0MngB++fuw0pZv1r+vAFAlxRlNwACgkQuw0pZv1r ++vDFfQgAkZ2V+KHTvrszmZMLAp/2slOHNZQ36zkaUyoHDkSLzQLDVN6f6bUskAaT +UQoU3a4dgNciauRx54C2jvkrsup8gL4/KDvkOB+xZYfymvMvySfgEGrZPDib90Kr +BJf4Bdp6sXWCFAufjPBnJbldHoeV8VsR9v6yHmJdttLXv8T6W7gPegKNnBUdaSVq +JwdJagjJn27LdOe8iIiWxnhM2/sq2DgZbonvsDVG6JQbrYf5Qnjbi1Y2mnWw/DhW +0qLItBQaR+JHzVcAhB+aVmXDVIPTLfV+OhFhop4ybWQHq45QMLWmVuBRrt6EqDgh +mfQCdhtqE7YNxbx3Z5VE3QM4GOuTwA== +=IrUW +-END PGP SIGNATURE- Added: dev/carbondata/1.5.2-rc2/apache-carbondata-1.5.2-source-release.zip.md5 == --- dev/carbondata/1.5.2-rc2/apache-carbondata-1.5.2-source-release.zip.md5 (added) +++ dev/carbondata/1.5.2-rc2/apache-carbondata-1.5.2-source-release.zip.md5 Wed Jan 30 16:41:50 2019 @@ -0,0 +1 @@ +2a2b8c0cc2c90222a1499f3d6b5389a8 apache-carbondata-1.5.2-source-release.zip Added: dev/carbondata/1.5.2-rc2/apache-carbondata-1.5.2-source-release.zip.sha512 == --- dev/carbondata/1.5.2-rc2/apache-carbondata-1.5.2-source-release.zip.sha512 (added) +++ dev/carbondata/1.5.2-rc2/apache-carbondata-1.5.2-source-release.zip.sha512 Wed Jan 30 16:41:50 2019 @@ -0,0 +1 @@ +c904df9ce13ad2325777f7232c4e0abd6136c5add97486d9cee516519b04a354cb083e64a228dbf65c9d61d801b637e5bb5fab2d14573566175f97c8e588e271 apache-carbondata-1.5.2-source-release.zip
[carbondata-site] branch asf-site updated: adding 1.5.2 version information
This is an automated email from the ASF dual-hosted git repository.
raghunandan pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/carbondata-site.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 3aee30b adding 1.5.2 version information
3aee30b is described below
commit 3aee30b8ffcba557cb4cebb4e4764fffbe323ee6
Author: Raghunandan S
AuthorDate: Mon Feb 11 09:23:17 2019 +0530
adding 1.5.2 version information
---
content/quick-start-guide.html | 10 +-
src/main/webapp/quick-start-guide.html | 10 +-
2 files changed, 10 insertions(+), 10 deletions(-)
diff --git a/content/quick-start-guide.html b/content/quick-start-guide.html
index b321353..3b22632 100644
--- a/content/quick-start-guide.html
+++ b/content/quick-start-guide.html
@@ -255,18 +255,18 @@ EOF
Installing and
Configuring CarbonData on Presto
Hive
-https://github.com/apache/carbondata/blob/master/docs/hive-guide.html";
target=_blank>Installing and Configuring CarbonData on Hive
+Installing and Configuring
CarbonData on Hive
Integration with
Storage Engines
HDFS
-https://github.com/apache/carbondata/blob/master/docs/quick-start-guide.html#installing-and-configuring-carbondata-on-standalone-spark-cluster";>CarbonData
supports read and write with HDFS
+CarbonData
supports read and write with HDFS
S3
-https://github.com/apache/carbondata/blob/master/docs/s3-guide.html";
target=_blank>CarbonData supports read and write with S3
+CarbonData supports read and write
with S3
Alluxio
-https://github.com/apache/carbondata/blob/master/docs/alluxio-guide.html";
target=_blank>CarbonData supports read and write with Alluxio
+CarbonData supports read and
write with Alluxio
Installing and Configuring CarbonData to run locally
with Spark Shell
Apache Spark Shell provides a simple way to learn the API, as well as a
powerful tool to analyze data interactively. Please visit http://spark.apache.org/docs/latest/"; target=_blank rel="nofollow">Apache
Spark Documentation for more details on Spark shell.
@@ -748,4 +748,4 @@ $(function() {
$('.b-nav__quickstart').addClass('selected'); });
-
\ No newline at end of file
+
diff --git a/src/main/webapp/quick-start-guide.html
b/src/main/webapp/quick-start-guide.html
index b321353..3b22632 100644
--- a/src/main/webapp/quick-start-guide.html
+++ b/src/main/webapp/quick-start-guide.html
@@ -255,18 +255,18 @@ EOF
Installing and
Configuring CarbonData on Presto
Hive
-https://github.com/apache/carbondata/blob/master/docs/hive-guide.html";
target=_blank>Installing and Configuring CarbonData on Hive
+Installing and Configuring
CarbonData on Hive
Integration with
Storage Engines
HDFS
-https://github.com/apache/carbondata/blob/master/docs/quick-start-guide.html#installing-and-configuring-carbondata-on-standalone-spark-cluster";>CarbonData
supports read and write with HDFS
+CarbonData
supports read and write with HDFS
S3
-https://github.com/apache/carbondata/blob/master/docs/s3-guide.html";
target=_blank>CarbonData supports read and write with S3
+CarbonData supports read and write
with S3
Alluxio
-https://github.com/apache/carbondata/blob/master/docs/alluxio-guide.html";
target=_blank>CarbonData supports read and write with Alluxio
+CarbonData supports read and
write with Alluxio
Installing and Configuring CarbonData to run locally
with Spark Shell
Apache Spark Shell provides a simple way to learn the API, as well as a
powerful tool to analyze data interactively. Please visit http://spark.apache.org/docs/latest/"; target=_blank rel="nofollow">Apache
Spark Documentation for more details on Spark shell.
@@ -748,4 +748,4 @@ $(function() {
$('.b-nav__quickstart').addClass('selected'); });
-
\ No newline at end of file
+
[carbondata] 01/01: [maven-release-plugin] prepare release apache-carbondata-1.5.3-rc1
This is an automated email from the ASF dual-hosted git repository. raghunandan pushed a commit to annotated tag apache-carbondata-1.5.3-rc1 in repository https://gitbox.apache.org/repos/asf/carbondata.git commit 7f271d0aba272f9fbe9642a4900cd4da61eb43bb Author: Raghunandan S AuthorDate: Tue Apr 2 18:37:51 2019 +0530 [maven-release-plugin] prepare release apache-carbondata-1.5.3-rc1 --- assembly/pom.xml | 2 +- common/pom.xml| 2 +- core/pom.xml | 2 +- datamap/bloom/pom.xml | 2 +- datamap/examples/pom.xml | 2 +- datamap/lucene/pom.xml| 2 +- datamap/mv/core/pom.xml | 2 +- datamap/mv/plan/pom.xml | 2 +- examples/spark2/pom.xml | 2 +- format/pom.xml| 2 +- hadoop/pom.xml| 2 +- integration/hive/pom.xml | 2 +- integration/presto/pom.xml| 2 +- integration/spark-common-test/pom.xml | 2 +- integration/spark-common/pom.xml | 2 +- integration/spark-datasource/pom.xml | 2 +- integration/spark2/pom.xml| 2 +- pom.xml | 4 ++-- processing/pom.xml| 2 +- store/sdk/pom.xml | 2 +- streaming/pom.xml | 2 +- tools/cli/pom.xml | 2 +- 22 files changed, 23 insertions(+), 23 deletions(-) diff --git a/assembly/pom.xml b/assembly/pom.xml index b8af897..6398414 100644 --- a/assembly/pom.xml +++ b/assembly/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.5.3-SNAPSHOT +1.5.3 ../pom.xml diff --git a/common/pom.xml b/common/pom.xml index f10ef34..d5720db 100644 --- a/common/pom.xml +++ b/common/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.5.3-SNAPSHOT +1.5.3 ../pom.xml diff --git a/core/pom.xml b/core/pom.xml index ea93d30..6810f6e 100644 --- a/core/pom.xml +++ b/core/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.5.3-SNAPSHOT +1.5.3 ../pom.xml diff --git a/datamap/bloom/pom.xml b/datamap/bloom/pom.xml index 9070287..56460da 100644 --- a/datamap/bloom/pom.xml +++ b/datamap/bloom/pom.xml @@ -4,7 +4,7 @@ org.apache.carbondata carbondata-parent -1.5.3-SNAPSHOT +1.5.3 ../../pom.xml diff --git a/datamap/examples/pom.xml b/datamap/examples/pom.xml index e88a906..b649c3c 100644 --- a/datamap/examples/pom.xml +++ b/datamap/examples/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.5.3-SNAPSHOT +1.5.3 ../../pom.xml diff --git a/datamap/lucene/pom.xml b/datamap/lucene/pom.xml index 6162f01..ca5ca79 100644 --- a/datamap/lucene/pom.xml +++ b/datamap/lucene/pom.xml @@ -4,7 +4,7 @@ org.apache.carbondata carbondata-parent -1.5.3-SNAPSHOT +1.5.3 ../../pom.xml diff --git a/datamap/mv/core/pom.xml b/datamap/mv/core/pom.xml index 7924305..b4a6cfc 100644 --- a/datamap/mv/core/pom.xml +++ b/datamap/mv/core/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.5.3-SNAPSHOT +1.5.3 ../../../pom.xml diff --git a/datamap/mv/plan/pom.xml b/datamap/mv/plan/pom.xml index 1210331..7c11a1a 100644 --- a/datamap/mv/plan/pom.xml +++ b/datamap/mv/plan/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.5.3-SNAPSHOT +1.5.3 ../../../pom.xml diff --git a/examples/spark2/pom.xml b/examples/spark2/pom.xml index 1a07edf..9ee3ed4 100644 --- a/examples/spark2/pom.xml +++ b/examples/spark2/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.5.3-SNAPSHOT +1.5.3 ../../pom.xml diff --git a/format/pom.xml b/format/pom.xml index ceaea40..50dab21 100644 --- a/format/pom.xml +++ b/format/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.5.3-SNAPSHOT +1.5.3 ../pom.xml diff --git a/hadoop/pom.xml b/hadoop/pom.xml index e23c74b..20481e5 100644 --- a/hadoop/pom.xml +++ b/hadoop/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.5.3-SNAPSHOT +1.5.3 ../pom.xml diff --git a/integration/hive/pom.xml b/integration/hive/pom.xml index e2c1504..8e3153c 100644 --- a/integration/hive/pom.xml +++ b/integration/hive/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.5.3-SNAPSHOT +1.5.3 ../../pom.xml diff --git a/integration/presto/pom.xml b/integration/presto/pom.xml index 5253677..8a51db7 100644 --- a/integration/presto/pom.xml +++ b/integration/presto/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.5.3-SNAPSHOT +1.5.3 ../../pom.xml diff --git a/integration/spark-common
[carbondata] annotated tag apache-carbondata-1.5.3-rc1 created (now 684a3a0)
This is an automated email from the ASF dual-hosted git repository. raghunandan pushed a change to annotated tag apache-carbondata-1.5.3-rc1 in repository https://gitbox.apache.org/repos/asf/carbondata.git. at 684a3a0 (tag) tagging 7f271d0aba272f9fbe9642a4900cd4da61eb43bb (commit) replaces apache-carbondata-1.5.2-rc2 by Raghunandan S on Tue Apr 2 18:55:50 2019 +0530 - Log - [maven-release-plugin] copy for tag apache-carbondata-1.5.3-rc1 --- This annotated tag includes the following new commits: new 7f271d0 [maven-release-plugin] prepare release apache-carbondata-1.5.3-rc1 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[carbondata] branch branch-1.5 updated (b9f3c6f -> 4f95559)
This is an automated email from the ASF dual-hosted git repository. raghunandan pushed a change to branch branch-1.5 in repository https://gitbox.apache.org/repos/asf/carbondata.git. from b9f3c6f [HOTFIX]fixed loading issue for legacy store add 7f271d0 [maven-release-plugin] prepare release apache-carbondata-1.5.3-rc1 new 4f95559 [maven-release-plugin] prepare for next development iteration The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: assembly/pom.xml | 2 +- common/pom.xml| 2 +- core/pom.xml | 2 +- datamap/bloom/pom.xml | 2 +- datamap/examples/pom.xml | 2 +- datamap/lucene/pom.xml| 2 +- datamap/mv/core/pom.xml | 2 +- datamap/mv/plan/pom.xml | 2 +- examples/spark2/pom.xml | 2 +- format/pom.xml| 2 +- hadoop/pom.xml| 2 +- integration/hive/pom.xml | 2 +- integration/presto/pom.xml| 2 +- integration/spark-common-test/pom.xml | 2 +- integration/spark-common/pom.xml | 2 +- integration/spark-datasource/pom.xml | 2 +- integration/spark2/pom.xml| 2 +- pom.xml | 2 +- processing/pom.xml| 2 +- store/sdk/pom.xml | 2 +- streaming/pom.xml | 2 +- tools/cli/pom.xml | 2 +- 22 files changed, 22 insertions(+), 22 deletions(-)
[carbondata] 01/01: [maven-release-plugin] prepare for next development iteration
This is an automated email from the ASF dual-hosted git repository. raghunandan pushed a commit to branch branch-1.5 in repository https://gitbox.apache.org/repos/asf/carbondata.git commit 4f95559599404762b851f6a23af50eefd7df4d5e Author: Raghunandan S AuthorDate: Tue Apr 2 18:56:59 2019 +0530 [maven-release-plugin] prepare for next development iteration --- assembly/pom.xml | 2 +- common/pom.xml| 2 +- core/pom.xml | 2 +- datamap/bloom/pom.xml | 2 +- datamap/examples/pom.xml | 2 +- datamap/lucene/pom.xml| 2 +- datamap/mv/core/pom.xml | 2 +- datamap/mv/plan/pom.xml | 2 +- examples/spark2/pom.xml | 2 +- format/pom.xml| 2 +- hadoop/pom.xml| 2 +- integration/hive/pom.xml | 2 +- integration/presto/pom.xml| 2 +- integration/spark-common-test/pom.xml | 2 +- integration/spark-common/pom.xml | 2 +- integration/spark-datasource/pom.xml | 2 +- integration/spark2/pom.xml| 2 +- pom.xml | 4 ++-- processing/pom.xml| 2 +- store/sdk/pom.xml | 2 +- streaming/pom.xml | 2 +- tools/cli/pom.xml | 2 +- 22 files changed, 23 insertions(+), 23 deletions(-) diff --git a/assembly/pom.xml b/assembly/pom.xml index 6398414..a05cfe6 100644 --- a/assembly/pom.xml +++ b/assembly/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.5.3 +1.5.4-SNAPSHOT ../pom.xml diff --git a/common/pom.xml b/common/pom.xml index d5720db..0148424 100644 --- a/common/pom.xml +++ b/common/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.5.3 +1.5.4-SNAPSHOT ../pom.xml diff --git a/core/pom.xml b/core/pom.xml index 6810f6e..2e8f5da 100644 --- a/core/pom.xml +++ b/core/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.5.3 +1.5.4-SNAPSHOT ../pom.xml diff --git a/datamap/bloom/pom.xml b/datamap/bloom/pom.xml index 56460da..1af3b19 100644 --- a/datamap/bloom/pom.xml +++ b/datamap/bloom/pom.xml @@ -4,7 +4,7 @@ org.apache.carbondata carbondata-parent -1.5.3 +1.5.4-SNAPSHOT ../../pom.xml diff --git a/datamap/examples/pom.xml b/datamap/examples/pom.xml index b649c3c..dabf4cd 100644 --- a/datamap/examples/pom.xml +++ b/datamap/examples/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.5.3 +1.5.4-SNAPSHOT ../../pom.xml diff --git a/datamap/lucene/pom.xml b/datamap/lucene/pom.xml index ca5ca79..627e758 100644 --- a/datamap/lucene/pom.xml +++ b/datamap/lucene/pom.xml @@ -4,7 +4,7 @@ org.apache.carbondata carbondata-parent -1.5.3 +1.5.4-SNAPSHOT ../../pom.xml diff --git a/datamap/mv/core/pom.xml b/datamap/mv/core/pom.xml index b4a6cfc..a4b8c13 100644 --- a/datamap/mv/core/pom.xml +++ b/datamap/mv/core/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.5.3 +1.5.4-SNAPSHOT ../../../pom.xml diff --git a/datamap/mv/plan/pom.xml b/datamap/mv/plan/pom.xml index 7c11a1a..feed9e3 100644 --- a/datamap/mv/plan/pom.xml +++ b/datamap/mv/plan/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.5.3 +1.5.4-SNAPSHOT ../../../pom.xml diff --git a/examples/spark2/pom.xml b/examples/spark2/pom.xml index 9ee3ed4..09606ac 100644 --- a/examples/spark2/pom.xml +++ b/examples/spark2/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.5.3 +1.5.4-SNAPSHOT ../../pom.xml diff --git a/format/pom.xml b/format/pom.xml index 50dab21..c287422 100644 --- a/format/pom.xml +++ b/format/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.5.3 +1.5.4-SNAPSHOT ../pom.xml diff --git a/hadoop/pom.xml b/hadoop/pom.xml index 20481e5..38649f6 100644 --- a/hadoop/pom.xml +++ b/hadoop/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.5.3 +1.5.4-SNAPSHOT ../pom.xml diff --git a/integration/hive/pom.xml b/integration/hive/pom.xml index 8e3153c..df0161a 100644 --- a/integration/hive/pom.xml +++ b/integration/hive/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.5.3 +1.5.4-SNAPSHOT ../../pom.xml diff --git a/integration/presto/pom.xml b/integration/presto/pom.xml index 8a51db7..c74040c 100644 --- a/integration/presto/pom.xml +++ b/integration/presto/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.5.3 +1.5.4-SNAPSHOT ../../pom.xml diff --git a/integration/spark-common-test/pom.xml b/integration
svn commit: r33414 - in /dev/carbondata/1.5.3-rc1: ./ apache-carbondata-1.5.3-source-release.zip apache-carbondata-1.5.3-source-release.zip.asc apache-carbondata-1.5.3-source-release.zip.md5 apache-ca
Author: raghunandan Date: Wed Apr 3 02:52:41 2019 New Revision: 33414 Log: checkin 1.5.3-rc1 Added: dev/carbondata/1.5.3-rc1/ dev/carbondata/1.5.3-rc1/apache-carbondata-1.5.3-source-release.zip (with props) dev/carbondata/1.5.3-rc1/apache-carbondata-1.5.3-source-release.zip.asc dev/carbondata/1.5.3-rc1/apache-carbondata-1.5.3-source-release.zip.md5 dev/carbondata/1.5.3-rc1/apache-carbondata-1.5.3-source-release.zip.sha512 Added: dev/carbondata/1.5.3-rc1/apache-carbondata-1.5.3-source-release.zip == Binary file - no diff available. Propchange: dev/carbondata/1.5.3-rc1/apache-carbondata-1.5.3-source-release.zip -- svn:mime-type = application/octet-stream Added: dev/carbondata/1.5.3-rc1/apache-carbondata-1.5.3-source-release.zip.asc == --- dev/carbondata/1.5.3-rc1/apache-carbondata-1.5.3-source-release.zip.asc (added) +++ dev/carbondata/1.5.3-rc1/apache-carbondata-1.5.3-source-release.zip.asc Wed Apr 3 02:52:41 2019 @@ -0,0 +1,11 @@ +-BEGIN PGP SIGNATURE- + +iQEzBAABCAAdFiEEsZE8naWI0MngB++fuw0pZv1r+vAFAlyjZB0ACgkQuw0pZv1r ++vAtdQgAkXTu/nht5DleMHt3A6IL1XHF9hMvl3mjZYDg56zJrENPh5iXoGqespMI +t8UA+nBTCDVTBkliEdHjb/pfnyN/GzO7bOcUUJjCkaElcCsVpHeElQHCwn8dRUZR +xEW0pNi2DCd/X/VZ8nfFaEuH5ys1uGkERbFnVrcSJ7o6oQPAxfPzuZOTKTUlalQ3 +KXXx4cEtPNEkK1+LZ8xbZxmreyWWDzDqvRo8lrGIe40MeiEbKGTNsRmu+ycb1GC8 +pqbxl4eCRVDl+QiIL1OdTKYiCPya2qnprY2v3hpXkQfuJ/RzM82ArLjKrbfdzM43 +vjuQAcA1wVlWR/VhcDmWWBXKNJQN3A== +=VsGg +-END PGP SIGNATURE- Added: dev/carbondata/1.5.3-rc1/apache-carbondata-1.5.3-source-release.zip.md5 == --- dev/carbondata/1.5.3-rc1/apache-carbondata-1.5.3-source-release.zip.md5 (added) +++ dev/carbondata/1.5.3-rc1/apache-carbondata-1.5.3-source-release.zip.md5 Wed Apr 3 02:52:41 2019 @@ -0,0 +1 @@ +946044661c12701ec17aa7b2232e1426 apache-carbondata-1.5.3-source-release.zip Added: dev/carbondata/1.5.3-rc1/apache-carbondata-1.5.3-source-release.zip.sha512 == --- dev/carbondata/1.5.3-rc1/apache-carbondata-1.5.3-source-release.zip.sha512 (added) +++ dev/carbondata/1.5.3-rc1/apache-carbondata-1.5.3-source-release.zip.sha512 Wed Apr 3 02:52:41 2019 @@ -0,0 +1 @@ +c1760912cc10f07bf2859ed0358efb62ddc594e8635f1ab58f74839d8d36a21f12242fa6ca559fba07b7aa0e9e20cadc50e4cf5bbd61f9d70192cc8b685cf2c4 apache-carbondata-1.5.3-source-release.zip
[carbondata] branch master updated: [CARBONDATA-3462][DOC]Added documentation for index server
This is an automated email from the ASF dual-hosted git repository.
raghunandan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/carbondata.git
The following commit(s) were added to refs/heads/master by this push:
new ee78597 [CARBONDATA-3462][DOC]Added documentation for index server
ee78597 is described below
commit ee785973df241b67308bee773a5be3c0ba37317d
Author: kunal642
AuthorDate: Wed Jun 19 03:05:34 2019 +0530
[CARBONDATA-3462][DOC]Added documentation for index server
Added documentation for index server
This closes #3294
---
README.md | 3 +-
.../core/constants/CarbonCommonConstants.java | 2 +-
docs/index-server.md | 229 +
.../carbondata/indexserver/IndexServer.scala | 9 +-
4 files changed, 234 insertions(+), 9 deletions(-)
diff --git a/README.md b/README.md
index bed906f..3226a30 100644
--- a/README.md
+++ b/README.md
@@ -63,7 +63,8 @@ CarbonData is built using Apache Maven, to [build
CarbonData](https://github.com
* [SDK
Guide](https://github.com/apache/carbondata/blob/master/docs/sdk-guide.md)
* [C++ SDK
Guide](https://github.com/apache/carbondata/blob/master/docs/csdk-guide.md)
* [Performance
Tuning](https://github.com/apache/carbondata/blob/master/docs/performance-tuning.md)
-* [S3
Storage](https://github.com/apache/carbondata/blob/master/docs/s3-guide.md)
+* [S3
Storage](https://github.com/apache/carbondata/blob/master/docs/s3-guide.md)
+* [Distributed Index
Server](https://github.com/apache/carbondata/blob/master/docs/index-server.md)
* [Carbon as Spark's
Datasource](https://github.com/apache/carbondata/blob/master/docs/carbon-as-spark-datasource-guide.md)
* [FAQs](https://github.com/apache/carbondata/blob/master/docs/faq.md)
diff --git
a/core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java
b/core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java
index 9b5260f..510bcee 100644
---
a/core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java
+++
b/core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java
@@ -2223,7 +2223,7 @@ public final class CarbonCommonConstants {
/**
* min value for in memory serialization size
*/
- public static final int CARBON_INDEX_SERVER_SERIALIZATION_THRESHOLD_MIN =
100;
+ public static final int CARBON_INDEX_SERVER_SERIALIZATION_THRESHOLD_MIN = 0;
/**
* max value for in memory serialization size
diff --git a/docs/index-server.md b/docs/index-server.md
new file mode 100644
index 000..5dd15c5
--- /dev/null
+++ b/docs/index-server.md
@@ -0,0 +1,229 @@
+
+
+# Distributed Index Server
+
+## Background
+
+Carbon currently prunes and caches all block/blocklet datamap index
information into the driver for
+normal table, for Bloom/Index datamaps the JDBC driver will launch a job to
prune and cache the
+datamaps in executors.
+
+This causes the driver to become a bottleneck in the following ways:
+1. If the cache size becomes huge(70-80% of the driver memory) then there can
be excessive GC in
+the driver which can slow down the query and the driver may even go
OutOfMemory.
+2. LRU has to evict a lot of elements from the cache to accommodate the new
objects which would
+in turn slow down the queries.
+3. For bloom there is no guarantee that the next query goes to the same
executor to reuse the cache
+and hence cache could be duplicated in multiple executors.
+4. Multiple JDBC drivers need to maintain their own copy of the cache.
+
+Distributed Index Cache Server aims to solve the above mentioned problems.
+
+## Distribution
+When enabled, any query on a carbon table will be routed to the index server
service in form of
+a request. The request will consist of the table name, segments, filter
expression and other
+information used for pruning.
+
+In IndexServer service a pruning RDD is fired which will take care of the
pruning for that
+request. This RDD will be creating tasks based on the number of segments that
are applicable for
+pruning. It can happen that the user has specified segments to access for that
table, so only the
+specified segments would be applicable for pruning. Refer:
[query-data-with-specified-segments](https://github.com/apache/carbondata/blob/6e50c1c6fc1d6e82a4faf6dc6e0824299786ccc0/docs/segment-management-on-carbondata.md#query-data-with-specified-segments).
+IndexServer driver would have 2 important tasks, distributing the segments
equally among the
+available executors and keeping track of the executor where the segment is
cached.
+
+To achieve this 2 separate mappings would be maintained as follows.
+1. segment to executor location:
+This mapping will be maintained for each table and will enable the index
server to track the
+cache location for each segment.
+
+2. Cache size held by each exe
[carbondata] branch branch-1.6 updated: [maven-release-plugin] prepare release apache-CarbonData-1.6.0-rc2
This is an automated email from the ASF dual-hosted git repository. raghunandan pushed a commit to branch branch-1.6 in repository https://gitbox.apache.org/repos/asf/carbondata.git The following commit(s) were added to refs/heads/branch-1.6 by this push: new 9ca7891 [maven-release-plugin] prepare release apache-CarbonData-1.6.0-rc2 9ca7891 is described below commit 9ca7891d16313be66d8271c855f7c6f4c54c2e1b Author: Raghunandan S AuthorDate: Fri Aug 2 14:43:40 2019 +0530 [maven-release-plugin] prepare release apache-CarbonData-1.6.0-rc2 --- assembly/pom.xml | 2 +- common/pom.xml| 2 +- core/pom.xml | 2 +- datamap/bloom/pom.xml | 6 ++ datamap/examples/pom.xml | 6 ++ datamap/lucene/pom.xml| 6 ++ datamap/mv/core/pom.xml | 2 +- datamap/mv/plan/pom.xml | 2 +- examples/flink/pom.xml| 2 +- examples/spark2/pom.xml | 2 +- format/pom.xml| 2 +- hadoop/pom.xml| 2 +- integration/hive/pom.xml | 2 +- integration/presto/pom.xml| 2 +- integration/spark-common-test/pom.xml | 14 +++--- integration/spark-common/pom.xml | 2 +- integration/spark-datasource/pom.xml | 2 +- integration/spark2/pom.xml| 2 +- pom.xml | 4 ++-- processing/pom.xml| 2 +- store/sdk/pom.xml | 6 ++ streaming/pom.xml | 6 ++ tools/cli/pom.xml | 6 ++ 23 files changed, 36 insertions(+), 48 deletions(-) diff --git a/assembly/pom.xml b/assembly/pom.xml index 9ac2e78..004788c 100644 --- a/assembly/pom.xml +++ b/assembly/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.6.0-SNAPSHOT +1.6.0 ../pom.xml diff --git a/common/pom.xml b/common/pom.xml index 14cd52f..8e5ddaa 100644 --- a/common/pom.xml +++ b/common/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.6.0-SNAPSHOT +1.6.0 ../pom.xml diff --git a/core/pom.xml b/core/pom.xml index 41481af..b39a42e 100644 --- a/core/pom.xml +++ b/core/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.6.0-SNAPSHOT +1.6.0 ../pom.xml diff --git a/datamap/bloom/pom.xml b/datamap/bloom/pom.xml index 1e8c382..a29f77b 100644 --- a/datamap/bloom/pom.xml +++ b/datamap/bloom/pom.xml @@ -1,12 +1,10 @@ -http://maven.apache.org/POM/4.0.0"; - xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; - xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> +http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> 4.0.0 org.apache.carbondata carbondata-parent -1.6.0-SNAPSHOT +1.6.0 ../../pom.xml diff --git a/datamap/examples/pom.xml b/datamap/examples/pom.xml index 3720a1c..a9c179d 100644 --- a/datamap/examples/pom.xml +++ b/datamap/examples/pom.xml @@ -15,16 +15,14 @@ See the License for the specific language governing permissions and limitations under the License. --> -http://maven.apache.org/POM/4.0.0"; - xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; - xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> +http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> 4.0.0 org.apache.carbondata carbondata-parent -1.6.0-SNAPSHOT +1.6.0 ../../pom.xml diff --git a/datamap/lucene/pom.xml b/datamap/lucene/pom.xml index 3e93a83..1a23a52 100644 --- a/datamap/lucene/pom.xml +++ b/datamap/lucene/pom.xml @@ -1,12 +1,10 @@ -http://maven.apache.org/POM/4.0.0"; - xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; - xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> +http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> 4.0.0 org.apache.carbondata carbondata-parent -1.6.0-SNAPSHOT +1.6.0 ../../pom.xml diff --git a/datamap/mv/core/pom.xml b/datamap/mv/core/pom.xml index 0a1f0e2..5cb284d 100644 --- a/datamap/mv/core/pom.xml +++ b/datamap/mv/core/pom.
[carbondata] branch branch-1.6 updated: [maven-release-plugin] prepare for next development iteration
This is an automated email from the ASF dual-hosted git repository. raghunandan pushed a commit to branch branch-1.6 in repository https://gitbox.apache.org/repos/asf/carbondata.git The following commit(s) were added to refs/heads/branch-1.6 by this push: new 9724fd4 [maven-release-plugin] prepare for next development iteration 9724fd4 is described below commit 9724fd47c2d446c733c79943a42f03d4eb5fcc3d Author: Raghunandan S AuthorDate: Fri Aug 2 18:34:19 2019 +0530 [maven-release-plugin] prepare for next development iteration --- assembly/pom.xml | 2 +- common/pom.xml| 2 +- core/pom.xml | 2 +- datamap/bloom/pom.xml | 2 +- datamap/examples/pom.xml | 2 +- datamap/lucene/pom.xml| 2 +- datamap/mv/core/pom.xml | 2 +- datamap/mv/plan/pom.xml | 2 +- examples/flink/pom.xml| 2 +- examples/spark2/pom.xml | 2 +- format/pom.xml| 2 +- hadoop/pom.xml| 2 +- integration/hive/pom.xml | 2 +- integration/presto/pom.xml| 2 +- integration/spark-common-test/pom.xml | 2 +- integration/spark-common/pom.xml | 2 +- integration/spark-datasource/pom.xml | 2 +- integration/spark2/pom.xml| 2 +- pom.xml | 4 ++-- processing/pom.xml| 2 +- store/sdk/pom.xml | 2 +- streaming/pom.xml | 2 +- tools/cli/pom.xml | 2 +- 23 files changed, 24 insertions(+), 24 deletions(-) diff --git a/assembly/pom.xml b/assembly/pom.xml index 004788c..12d7e6e 100644 --- a/assembly/pom.xml +++ b/assembly/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.6.0 +1.6.1-SNAPSHOT ../pom.xml diff --git a/common/pom.xml b/common/pom.xml index 8e5ddaa..728314c 100644 --- a/common/pom.xml +++ b/common/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.6.0 +1.6.1-SNAPSHOT ../pom.xml diff --git a/core/pom.xml b/core/pom.xml index b39a42e..22982f3 100644 --- a/core/pom.xml +++ b/core/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.6.0 +1.6.1-SNAPSHOT ../pom.xml diff --git a/datamap/bloom/pom.xml b/datamap/bloom/pom.xml index a29f77b..8ba7846 100644 --- a/datamap/bloom/pom.xml +++ b/datamap/bloom/pom.xml @@ -4,7 +4,7 @@ org.apache.carbondata carbondata-parent -1.6.0 +1.6.1-SNAPSHOT ../../pom.xml diff --git a/datamap/examples/pom.xml b/datamap/examples/pom.xml index a9c179d..6e3b8ae 100644 --- a/datamap/examples/pom.xml +++ b/datamap/examples/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.6.0 +1.6.1-SNAPSHOT ../../pom.xml diff --git a/datamap/lucene/pom.xml b/datamap/lucene/pom.xml index 1a23a52..42a22b2 100644 --- a/datamap/lucene/pom.xml +++ b/datamap/lucene/pom.xml @@ -4,7 +4,7 @@ org.apache.carbondata carbondata-parent -1.6.0 +1.6.1-SNAPSHOT ../../pom.xml diff --git a/datamap/mv/core/pom.xml b/datamap/mv/core/pom.xml index 5cb284d..6af274d 100644 --- a/datamap/mv/core/pom.xml +++ b/datamap/mv/core/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.6.0 +1.6.1-SNAPSHOT ../../../pom.xml diff --git a/datamap/mv/plan/pom.xml b/datamap/mv/plan/pom.xml index fe1afb7..4b8c9be 100644 --- a/datamap/mv/plan/pom.xml +++ b/datamap/mv/plan/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.6.0 +1.6.1-SNAPSHOT ../../../pom.xml diff --git a/examples/flink/pom.xml b/examples/flink/pom.xml index 3d0b2e4..5013f67 100644 --- a/examples/flink/pom.xml +++ b/examples/flink/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.6.0 +1.6.1-SNAPSHOT ../../pom.xml diff --git a/examples/spark2/pom.xml b/examples/spark2/pom.xml index e303406..ad0d3ec 100644 --- a/examples/spark2/pom.xml +++ b/examples/spark2/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.6.0 +1.6.1-SNAPSHOT ../../pom.xml diff --git a/format/pom.xml b/format/pom.xml index 51135d8..81aa95b 100644 --- a/format/pom.xml +++ b/format/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.6.0 +1.6.1-SNAPSHOT ../pom.xml diff --git a/hadoop/pom.xml b/hadoop/pom.xml index 59f515e..bcb5696 100644 --- a/hadoop/pom.xml +++ b/hadoop/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.6.0 +1.6.1-SNAPSHOT ../pom.xml diff --git a/integration/hive/pom.xml b/integration/hive/pom.xml index 58b0796..dfa8810 100644 --- a/integration/hive/pom.xml +++ b/integration/hive/pom.xml @@ -22,7
[carbondata] annotated tag apache-CarbonData-1.6.0-rc2 created (now f4a0805)
This is an automated email from the ASF dual-hosted git repository. raghunandan pushed a change to annotated tag apache-CarbonData-1.6.0-rc2 in repository https://gitbox.apache.org/repos/asf/carbondata.git. at f4a0805 (tag) tagging 9ca7891d16313be66d8271c855f7c6f4c54c2e1b (commit) by Raghunandan S on Fri Aug 2 18:33:56 2019 +0530 - Log - [maven-release-plugin] copy for tag apache-CarbonData-1.6.0-rc2 --- No new revisions were added by this update.
svn commit: r35123 - in /dev/carbondata/1.6.0-rc2: ./ apache-carbondata-1.6.0-source-release.zip apache-carbondata-1.6.0-source-release.zip.asc apache-carbondata-1.6.0-source-release.zip.sha512
Author: raghunandan Date: Sat Aug 3 06:59:21 2019 New Revision: 35123 Log: Upload 1.6.0-RC2 Added: dev/carbondata/1.6.0-rc2/ dev/carbondata/1.6.0-rc2/apache-carbondata-1.6.0-source-release.zip (with props) dev/carbondata/1.6.0-rc2/apache-carbondata-1.6.0-source-release.zip.asc dev/carbondata/1.6.0-rc2/apache-carbondata-1.6.0-source-release.zip.sha512 Added: dev/carbondata/1.6.0-rc2/apache-carbondata-1.6.0-source-release.zip == Binary file - no diff available. Propchange: dev/carbondata/1.6.0-rc2/apache-carbondata-1.6.0-source-release.zip -- svn:mime-type = application/octet-stream Added: dev/carbondata/1.6.0-rc2/apache-carbondata-1.6.0-source-release.zip.asc == --- dev/carbondata/1.6.0-rc2/apache-carbondata-1.6.0-source-release.zip.asc (added) +++ dev/carbondata/1.6.0-rc2/apache-carbondata-1.6.0-source-release.zip.asc Sat Aug 3 06:59:21 2019 @@ -0,0 +1,11 @@ +-BEGIN PGP SIGNATURE- + +iQEzBAABCAAdFiEEsZE8naWI0MngB++fuw0pZv1r+vAFAl1EPPgACgkQuw0pZv1r ++vDRUggAsUbPARXDmtg9S9yGRQ708XoPKV1B+VNfpJdHAy2Lx3M5ochA4tpOWGib +5W99UigytOBBQD/qY19Z/mo8rxw+s0lbOkNlndwJnRdSF4SZV7hYCXXcXhKfbLpa +zZTDZXm8f16O5gfyB4f7ZzY8PJfBGlZUQN7aef9rsqWmAhLZC7gAYtG2iqaO4n28 +8fpx/2z3vNjnVJl1fCDQoQ9cPAQO9BwLX9dWnCDYxYG66PGTkcdGjxdWsH5Xr4/3 +RFGyKBeX+drTOeJrlO3iUtGtQ49rKiZkSWwsZBHfneI4+bwQRO0MEHIG7AwOPD+r +P8ldHoxwsCc2iUni7Y1xs2i5BemPwA== +=27AM +-END PGP SIGNATURE- Added: dev/carbondata/1.6.0-rc2/apache-carbondata-1.6.0-source-release.zip.sha512 == --- dev/carbondata/1.6.0-rc2/apache-carbondata-1.6.0-source-release.zip.sha512 (added) +++ dev/carbondata/1.6.0-rc2/apache-carbondata-1.6.0-source-release.zip.sha512 Sat Aug 3 06:59:21 2019 @@ -0,0 +1 @@ +9b1cae6a142cf9291cae0ede01b209809a2a574fbdac582c1859ed7bce79fd3d4cffc90dd5c3fc595dff57d043809cd92fc126a3ea294719b1a83071be3598c6 apache-carbondata-1.6.0-source-release.zip
[carbondata] branch branch-1.6 updated: [maven-release-plugin] prepare release apache-CarbonData-1.6.0-rc2
This is an automated email from the ASF dual-hosted git repository. raghunandan pushed a commit to branch branch-1.6 in repository https://gitbox.apache.org/repos/asf/carbondata.git The following commit(s) were added to refs/heads/branch-1.6 by this push: new 79b533f [maven-release-plugin] prepare release apache-CarbonData-1.6.0-rc2 79b533f is described below commit 79b533f4228614b7ef5a126a79885df8cf24203e Author: Raghunandan S AuthorDate: Tue Aug 13 10:41:49 2019 +0530 [maven-release-plugin] prepare release apache-CarbonData-1.6.0-rc2 --- assembly/pom.xml | 2 +- common/pom.xml| 2 +- core/pom.xml | 2 +- datamap/bloom/pom.xml | 6 ++ datamap/examples/pom.xml | 6 ++ datamap/lucene/pom.xml| 6 ++ datamap/mv/core/pom.xml | 2 +- datamap/mv/plan/pom.xml | 2 +- examples/flink/pom.xml| 2 +- examples/spark2/pom.xml | 2 +- format/pom.xml| 2 +- hadoop/pom.xml| 2 +- integration/hive/pom.xml | 2 +- integration/presto/pom.xml| 2 +- integration/spark-common-test/pom.xml | 14 +++--- integration/spark-common/pom.xml | 2 +- integration/spark-datasource/pom.xml | 2 +- integration/spark2/pom.xml| 2 +- pom.xml | 4 ++-- processing/pom.xml| 2 +- store/sdk/pom.xml | 6 ++ streaming/pom.xml | 6 ++ tools/cli/pom.xml | 6 ++ 23 files changed, 36 insertions(+), 48 deletions(-) diff --git a/assembly/pom.xml b/assembly/pom.xml index 9ac2e78..004788c 100644 --- a/assembly/pom.xml +++ b/assembly/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.6.0-SNAPSHOT +1.6.0 ../pom.xml diff --git a/common/pom.xml b/common/pom.xml index 14cd52f..8e5ddaa 100644 --- a/common/pom.xml +++ b/common/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.6.0-SNAPSHOT +1.6.0 ../pom.xml diff --git a/core/pom.xml b/core/pom.xml index 41481af..b39a42e 100644 --- a/core/pom.xml +++ b/core/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.6.0-SNAPSHOT +1.6.0 ../pom.xml diff --git a/datamap/bloom/pom.xml b/datamap/bloom/pom.xml index 1e8c382..a29f77b 100644 --- a/datamap/bloom/pom.xml +++ b/datamap/bloom/pom.xml @@ -1,12 +1,10 @@ -http://maven.apache.org/POM/4.0.0"; - xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; - xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> +http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> 4.0.0 org.apache.carbondata carbondata-parent -1.6.0-SNAPSHOT +1.6.0 ../../pom.xml diff --git a/datamap/examples/pom.xml b/datamap/examples/pom.xml index 3720a1c..a9c179d 100644 --- a/datamap/examples/pom.xml +++ b/datamap/examples/pom.xml @@ -15,16 +15,14 @@ See the License for the specific language governing permissions and limitations under the License. --> -http://maven.apache.org/POM/4.0.0"; - xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; - xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> +http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> 4.0.0 org.apache.carbondata carbondata-parent -1.6.0-SNAPSHOT +1.6.0 ../../pom.xml diff --git a/datamap/lucene/pom.xml b/datamap/lucene/pom.xml index 3e93a83..1a23a52 100644 --- a/datamap/lucene/pom.xml +++ b/datamap/lucene/pom.xml @@ -1,12 +1,10 @@ -http://maven.apache.org/POM/4.0.0"; - xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; - xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> +http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> 4.0.0 org.apache.carbondata carbondata-parent -1.6.0-SNAPSHOT +1.6.0 ../../pom.xml diff --git a/datamap/mv/core/pom.xml b/datamap/mv/core/pom.xml index 0a1f0e2..5cb284d 100644 --- a/datamap/mv/core/pom.xml +++ b/datamap/mv/core/pom.
[carbondata] annotated tag apache-CarbonData-1.6.0-rc3 created (now d738062)
This is an automated email from the ASF dual-hosted git repository. raghunandan pushed a change to annotated tag apache-CarbonData-1.6.0-rc3 in repository https://gitbox.apache.org/repos/asf/carbondata.git. at d738062 (tag) tagging 4729b4ccee18ada1898e27f130253ad06497f1fb (commit) by Raghunandan S on Tue Aug 13 11:44:11 2019 +0530 - Log - [maven-release-plugin] copy for tag apache-CarbonData-1.6.0-rc3 --- No new revisions were added by this update.
[carbondata] branch branch-1.6 updated: [maven-release-plugin] prepare release apache-CarbonData-1.6.0-rc3
This is an automated email from the ASF dual-hosted git repository. raghunandan pushed a commit to branch branch-1.6 in repository https://gitbox.apache.org/repos/asf/carbondata.git The following commit(s) were added to refs/heads/branch-1.6 by this push: new 4729b4c [maven-release-plugin] prepare release apache-CarbonData-1.6.0-rc3 4729b4c is described below commit 4729b4ccee18ada1898e27f130253ad06497f1fb Author: Raghunandan S AuthorDate: Tue Aug 13 11:41:44 2019 +0530 [maven-release-plugin] prepare release apache-CarbonData-1.6.0-rc3 --- assembly/pom.xml | 2 +- common/pom.xml| 2 +- core/pom.xml | 2 +- datamap/bloom/pom.xml | 6 ++ datamap/examples/pom.xml | 6 ++ datamap/lucene/pom.xml| 6 ++ datamap/mv/core/pom.xml | 2 +- datamap/mv/plan/pom.xml | 2 +- examples/flink/pom.xml| 2 +- examples/spark2/pom.xml | 2 +- format/pom.xml| 2 +- hadoop/pom.xml| 2 +- integration/hive/pom.xml | 2 +- integration/presto/pom.xml| 2 +- integration/spark-common-test/pom.xml | 14 +++--- integration/spark-common/pom.xml | 2 +- integration/spark-datasource/pom.xml | 2 +- integration/spark2/pom.xml| 2 +- pom.xml | 4 ++-- processing/pom.xml| 2 +- store/sdk/pom.xml | 6 ++ streaming/pom.xml | 6 ++ tools/cli/pom.xml | 6 ++ 23 files changed, 36 insertions(+), 48 deletions(-) diff --git a/assembly/pom.xml b/assembly/pom.xml index 9ac2e78..004788c 100644 --- a/assembly/pom.xml +++ b/assembly/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.6.0-SNAPSHOT +1.6.0 ../pom.xml diff --git a/common/pom.xml b/common/pom.xml index 14cd52f..8e5ddaa 100644 --- a/common/pom.xml +++ b/common/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.6.0-SNAPSHOT +1.6.0 ../pom.xml diff --git a/core/pom.xml b/core/pom.xml index 41481af..b39a42e 100644 --- a/core/pom.xml +++ b/core/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.6.0-SNAPSHOT +1.6.0 ../pom.xml diff --git a/datamap/bloom/pom.xml b/datamap/bloom/pom.xml index 1e8c382..a29f77b 100644 --- a/datamap/bloom/pom.xml +++ b/datamap/bloom/pom.xml @@ -1,12 +1,10 @@ -http://maven.apache.org/POM/4.0.0"; - xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; - xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> +http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> 4.0.0 org.apache.carbondata carbondata-parent -1.6.0-SNAPSHOT +1.6.0 ../../pom.xml diff --git a/datamap/examples/pom.xml b/datamap/examples/pom.xml index 3720a1c..a9c179d 100644 --- a/datamap/examples/pom.xml +++ b/datamap/examples/pom.xml @@ -15,16 +15,14 @@ See the License for the specific language governing permissions and limitations under the License. --> -http://maven.apache.org/POM/4.0.0"; - xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; - xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> +http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> 4.0.0 org.apache.carbondata carbondata-parent -1.6.0-SNAPSHOT +1.6.0 ../../pom.xml diff --git a/datamap/lucene/pom.xml b/datamap/lucene/pom.xml index 3e93a83..1a23a52 100644 --- a/datamap/lucene/pom.xml +++ b/datamap/lucene/pom.xml @@ -1,12 +1,10 @@ -http://maven.apache.org/POM/4.0.0"; - xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; - xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> +http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> 4.0.0 org.apache.carbondata carbondata-parent -1.6.0-SNAPSHOT +1.6.0 ../../pom.xml diff --git a/datamap/mv/core/pom.xml b/datamap/mv/core/pom.xml index 0a1f0e2..5cb284d 100644 --- a/datamap/mv/core/pom.xml +++ b/datamap/mv/core/pom.
[carbondata] branch branch-1.6 updated: [maven-release-plugin] prepare for next development iteration
This is an automated email from the ASF dual-hosted git repository. raghunandan pushed a commit to branch branch-1.6 in repository https://gitbox.apache.org/repos/asf/carbondata.git The following commit(s) were added to refs/heads/branch-1.6 by this push: new 72169e5 [maven-release-plugin] prepare for next development iteration 72169e5 is described below commit 72169e554f12d21418a785b8cb587c1b91cd009a Author: Raghunandan S AuthorDate: Tue Aug 13 11:44:31 2019 +0530 [maven-release-plugin] prepare for next development iteration --- assembly/pom.xml | 2 +- common/pom.xml| 2 +- core/pom.xml | 2 +- datamap/bloom/pom.xml | 2 +- datamap/examples/pom.xml | 2 +- datamap/lucene/pom.xml| 2 +- datamap/mv/core/pom.xml | 2 +- datamap/mv/plan/pom.xml | 2 +- examples/flink/pom.xml| 2 +- examples/spark2/pom.xml | 2 +- format/pom.xml| 2 +- hadoop/pom.xml| 2 +- integration/hive/pom.xml | 2 +- integration/presto/pom.xml| 2 +- integration/spark-common-test/pom.xml | 2 +- integration/spark-common/pom.xml | 2 +- integration/spark-datasource/pom.xml | 2 +- integration/spark2/pom.xml| 2 +- pom.xml | 4 ++-- processing/pom.xml| 2 +- store/sdk/pom.xml | 2 +- streaming/pom.xml | 2 +- tools/cli/pom.xml | 2 +- 23 files changed, 24 insertions(+), 24 deletions(-) diff --git a/assembly/pom.xml b/assembly/pom.xml index 004788c..12d7e6e 100644 --- a/assembly/pom.xml +++ b/assembly/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.6.0 +1.6.1-SNAPSHOT ../pom.xml diff --git a/common/pom.xml b/common/pom.xml index 8e5ddaa..728314c 100644 --- a/common/pom.xml +++ b/common/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.6.0 +1.6.1-SNAPSHOT ../pom.xml diff --git a/core/pom.xml b/core/pom.xml index b39a42e..22982f3 100644 --- a/core/pom.xml +++ b/core/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.6.0 +1.6.1-SNAPSHOT ../pom.xml diff --git a/datamap/bloom/pom.xml b/datamap/bloom/pom.xml index a29f77b..8ba7846 100644 --- a/datamap/bloom/pom.xml +++ b/datamap/bloom/pom.xml @@ -4,7 +4,7 @@ org.apache.carbondata carbondata-parent -1.6.0 +1.6.1-SNAPSHOT ../../pom.xml diff --git a/datamap/examples/pom.xml b/datamap/examples/pom.xml index a9c179d..6e3b8ae 100644 --- a/datamap/examples/pom.xml +++ b/datamap/examples/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.6.0 +1.6.1-SNAPSHOT ../../pom.xml diff --git a/datamap/lucene/pom.xml b/datamap/lucene/pom.xml index 1a23a52..42a22b2 100644 --- a/datamap/lucene/pom.xml +++ b/datamap/lucene/pom.xml @@ -4,7 +4,7 @@ org.apache.carbondata carbondata-parent -1.6.0 +1.6.1-SNAPSHOT ../../pom.xml diff --git a/datamap/mv/core/pom.xml b/datamap/mv/core/pom.xml index 5cb284d..6af274d 100644 --- a/datamap/mv/core/pom.xml +++ b/datamap/mv/core/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.6.0 +1.6.1-SNAPSHOT ../../../pom.xml diff --git a/datamap/mv/plan/pom.xml b/datamap/mv/plan/pom.xml index fe1afb7..4b8c9be 100644 --- a/datamap/mv/plan/pom.xml +++ b/datamap/mv/plan/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.6.0 +1.6.1-SNAPSHOT ../../../pom.xml diff --git a/examples/flink/pom.xml b/examples/flink/pom.xml index 3d0b2e4..5013f67 100644 --- a/examples/flink/pom.xml +++ b/examples/flink/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.6.0 +1.6.1-SNAPSHOT ../../pom.xml diff --git a/examples/spark2/pom.xml b/examples/spark2/pom.xml index e303406..ad0d3ec 100644 --- a/examples/spark2/pom.xml +++ b/examples/spark2/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.6.0 +1.6.1-SNAPSHOT ../../pom.xml diff --git a/format/pom.xml b/format/pom.xml index 51135d8..81aa95b 100644 --- a/format/pom.xml +++ b/format/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.6.0 +1.6.1-SNAPSHOT ../pom.xml diff --git a/hadoop/pom.xml b/hadoop/pom.xml index 59f515e..bcb5696 100644 --- a/hadoop/pom.xml +++ b/hadoop/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.6.0 +1.6.1-SNAPSHOT ../pom.xml diff --git a/integration/hive/pom.xml b/integration/hive/pom.xml index 58b0796..dfa8810 100644 --- a/integration/hive/pom.xml +++ b/integration/hive/pom.xml @@ -22,7
svn commit: r35220 - in /dev/carbondata/1.6.0-rc3: ./ apache-carbondata-1.6.0-source-release.asc apache-carbondata-1.6.0-source-release.zip apache-carbondata-1.6.0-source-release.zip.sha512
Author: raghunandan Date: Tue Aug 13 11:38:49 2019 New Revision: 35220 Log: Upload 1.6.0-RC3 Added: dev/carbondata/1.6.0-rc3/ dev/carbondata/1.6.0-rc3/apache-carbondata-1.6.0-source-release.asc dev/carbondata/1.6.0-rc3/apache-carbondata-1.6.0-source-release.zip (with props) dev/carbondata/1.6.0-rc3/apache-carbondata-1.6.0-source-release.zip.sha512 Added: dev/carbondata/1.6.0-rc3/apache-carbondata-1.6.0-source-release.asc == --- dev/carbondata/1.6.0-rc3/apache-carbondata-1.6.0-source-release.asc (added) +++ dev/carbondata/1.6.0-rc3/apache-carbondata-1.6.0-source-release.asc Tue Aug 13 11:38:49 2019 @@ -0,0 +1,11 @@ +-BEGIN PGP SIGNATURE- + +iQEzBAABCAAdFiEEsZE8naWI0MngB++fuw0pZv1r+vAFAl1SUpUACgkQuw0pZv1r ++vB5ngf9FLPAtN99tmV/gnWCnQTSsBRHsV0tMPe8MGSSFfN/l0aS3ZWK9vcZaXhK +n95IRbbNdfXE8L9GGnz/14SWMBOE/nox7RwHXS9WUReuxHicaTdbz8XkHQ7AsV3Y ++qSHQN2hruoBKJpQlPouX+jzKcXbB4hL2WTjdL3uRgtm0HEilOF1tpuXlvMpNUEE +irQSzmHw/jMqycWYTWql3Vis2vgAwLWUTuc0ZmvheroHbP5Tm++CvNVMDA8MKcq6 +xRgP62uEGIfH9wRt19mXg7c2xLwzGy+648e/TOlnS8YsvE+DcUkh9yQk4qeiFhfU +QVgiGtWivhOGhZfkz5UIiGceLfyG6A== +=HsCy +-END PGP SIGNATURE- Added: dev/carbondata/1.6.0-rc3/apache-carbondata-1.6.0-source-release.zip == Binary file - no diff available. Propchange: dev/carbondata/1.6.0-rc3/apache-carbondata-1.6.0-source-release.zip -- svn:mime-type = application/octet-stream Added: dev/carbondata/1.6.0-rc3/apache-carbondata-1.6.0-source-release.zip.sha512 == --- dev/carbondata/1.6.0-rc3/apache-carbondata-1.6.0-source-release.zip.sha512 (added) +++ dev/carbondata/1.6.0-rc3/apache-carbondata-1.6.0-source-release.zip.sha512 Tue Aug 13 11:38:49 2019 @@ -0,0 +1 @@ +e3f3caf7b9e59bed300180d896390be2384999d9445fd146662342eb61407f717eb87cce70bca90516be31fbe5dab44807d58d096d443b5bf5e3c59d70731e42 apache-carbondata-1.6.0-source-release.zip
[carbondata-site] branch asf-site updated: Corrected md file links in 1.5.4 & 1.6.0 version
This is an automated email from the ASF dual-hosted git repository. raghunandan pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/carbondata-site.git The following commit(s) were added to refs/heads/asf-site by this push: new 3f34beb Corrected md file links in 1.5.4 & 1.6.0 version 3f34beb is described below commit 3f34bebd89189b006c9bafac4d8635033645 Author: Raghunandan S AuthorDate: Wed Sep 11 21:09:22 2019 +0530 Corrected md file links in 1.5.4 & 1.6.0 version --- content/ddl-of-carbondata.html | 2 +- content/documentation.html | 2 +- content/index-server.html | 6 +++--- content/mv-datamap-guide.html | 2 +- content/quick-start-guide.html | 10 +- src/main/webapp/ddl-of-carbondata.html | 2 +- src/main/webapp/documentation.html | 2 +- src/main/webapp/index-server.html | 6 +++--- src/main/webapp/mv-datamap-guide.html | 2 +- src/main/webapp/quick-start-guide.html | 10 +- 10 files changed, 22 insertions(+), 22 deletions(-) diff --git a/content/ddl-of-carbondata.html b/content/ddl-of-carbondata.html index 5720f10..ae8c618 100644 --- a/content/ddl-of-carbondata.html +++ b/content/ddl-of-carbondata.html @@ -927,7 +927,7 @@ Our SDK module currently supports writing data in this format. """.stripMargin) Here writer path will have carbondata and index files. -This can be SDK output or C++ SDK output. Refer SDK Guide and C++ SDK Guide. +This can be SDK output or C++ SDK output. Refer SDK Guide and C++ SDK Guide. Note: Dropping of the external table should not delete the files present in the location. diff --git a/content/documentation.html b/content/documentation.html index 3ca010d..6b5ad92 100644 --- a/content/documentation.html +++ b/content/documentation.html @@ -235,7 +235,7 @@ File Format Concepts: Start with the basics of understanding the CarbonData file format and its storage structure. This will help to understand other parts of the documentation, including deployment, programming and usage guides. Quick Start: Run an example program on your local machine or https://github.com/apache/carbondata/tree/master/examples/spark2/src/main/scala/org/apache/carbondata/examples"; target=_blank>study some examples. CarbonData SQL Language Reference: CarbonData extends the Spark SQL language and adds several DDL and DML statements to support operations on it. Refer to the Reference Manual to understand the supported features and functions. -Programming Guides: You can read our guides about Java APIs supported or C++ APIs supported to learn how to integrate CarbonData with your applications. +Programming Guides: You can read our guides about Java APIs supported or C++ APIs supported to learn how to integrate CarbonData with your applications. Integration diff --git a/content/index-server.html b/content/index-server.html index f7bea90..9a67034 100644 --- a/content/index-server.html +++ b/content/index-server.html @@ -253,7 +253,7 @@ information used for pruning. In IndexServer service a pruning RDD is fired which will take care of the pruning for that request. This RDD will be creating tasks based on the number of segments that are applicable for pruning. It can happen that the user has specified segments to access for that table, so only the -specified segments would be applicable for pruning. Refer: https://github.com/apache/carbondata/blob/6e50c1c6fc1d6e82a4faf6dc6e0824299786ccc0/docs/segment-management-on-carbondata.html#query-data-with-specified-segments";>query-data-with-specified-segments. +specified segments would be applicable for pruning. Refer: query-data-with-specified-segments. IndexServer driver would have 2 important tasks, distributing the segments equally among the available executors and keeping track of the executor where the segment is cached. To achieve this 2 separate mappings would be maintained as follows. @@ -302,7 +302,7 @@ current query. The show metacache DDL has a new column called cache location will indicate whether the cache is from executor or driver. To drop cache the user has to enable/disable the index server using the dynamic configuration to clear the cache of the desired location. -Refer: https://github.com/apache/carbondata/blob/master/docs/ddl-of-carbondata.html#cache";>MetaCacheDDL +Refer: MetaCacheDDL Fallback In case of any failure the index server would fallback to embedded mode @@ -363,7 +363,7 @@ executed as a Privileged Action using the login user. carbon.disable.index.server.fallback false -Whether to enable/disable fallback for index server. Should be used for testing purposes only. Refer: Fallback +Whether to enable/disable fallback for index server. Should be used for testing purposes only. Refer: Fallback diff --git a/content/mv-datamap-guide.html b/content/mv-datamap-guide.html index a6a3ee2..20a14bd 1
[carbondata] branch branch-1.6 updated: [maven-release-plugin] prepare release apache-CarbonData-1.6.1-rc1
This is an automated email from the ASF dual-hosted git repository. raghunandan pushed a commit to branch branch-1.6 in repository https://gitbox.apache.org/repos/asf/carbondata.git The following commit(s) were added to refs/heads/branch-1.6 by this push: new cabde62 [maven-release-plugin] prepare release apache-CarbonData-1.6.1-rc1 cabde62 is described below commit cabde6252d4a527fbfeb7f17627c6dce3e357f84 Author: Raghunandan S AuthorDate: Mon Oct 7 10:21:59 2019 +0530 [maven-release-plugin] prepare release apache-CarbonData-1.6.1-rc1 --- assembly/pom.xml | 2 +- common/pom.xml| 2 +- core/pom.xml | 2 +- datamap/bloom/pom.xml | 2 +- datamap/examples/pom.xml | 2 +- datamap/lucene/pom.xml| 2 +- datamap/mv/core/pom.xml | 2 +- datamap/mv/plan/pom.xml | 2 +- examples/flink/pom.xml| 2 +- examples/spark2/pom.xml | 2 +- format/pom.xml| 2 +- hadoop/pom.xml| 2 +- integration/hive/pom.xml | 2 +- integration/presto/pom.xml| 2 +- integration/spark-common-test/pom.xml | 2 +- integration/spark-common/pom.xml | 2 +- integration/spark-datasource/pom.xml | 2 +- integration/spark2/pom.xml| 2 +- pom.xml | 4 ++-- processing/pom.xml| 2 +- store/sdk/pom.xml | 2 +- streaming/pom.xml | 2 +- tools/cli/pom.xml | 2 +- 23 files changed, 24 insertions(+), 24 deletions(-) diff --git a/assembly/pom.xml b/assembly/pom.xml index bf729c5..3b76af6 100644 --- a/assembly/pom.xml +++ b/assembly/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.6.1-SNAPSHOT +1.6.1 ../pom.xml diff --git a/common/pom.xml b/common/pom.xml index 728314c..b9399f5 100644 --- a/common/pom.xml +++ b/common/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.6.1-SNAPSHOT +1.6.1 ../pom.xml diff --git a/core/pom.xml b/core/pom.xml index 22982f3..115503e 100644 --- a/core/pom.xml +++ b/core/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.6.1-SNAPSHOT +1.6.1 ../pom.xml diff --git a/datamap/bloom/pom.xml b/datamap/bloom/pom.xml index 8ba7846..9aa736f 100644 --- a/datamap/bloom/pom.xml +++ b/datamap/bloom/pom.xml @@ -4,7 +4,7 @@ org.apache.carbondata carbondata-parent -1.6.1-SNAPSHOT +1.6.1 ../../pom.xml diff --git a/datamap/examples/pom.xml b/datamap/examples/pom.xml index 6e3b8ae..6ea0c62 100644 --- a/datamap/examples/pom.xml +++ b/datamap/examples/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.6.1-SNAPSHOT +1.6.1 ../../pom.xml diff --git a/datamap/lucene/pom.xml b/datamap/lucene/pom.xml index 42a22b2..2220aaf 100644 --- a/datamap/lucene/pom.xml +++ b/datamap/lucene/pom.xml @@ -4,7 +4,7 @@ org.apache.carbondata carbondata-parent -1.6.1-SNAPSHOT +1.6.1 ../../pom.xml diff --git a/datamap/mv/core/pom.xml b/datamap/mv/core/pom.xml index 6af274d..8d71cdd 100644 --- a/datamap/mv/core/pom.xml +++ b/datamap/mv/core/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.6.1-SNAPSHOT +1.6.1 ../../../pom.xml diff --git a/datamap/mv/plan/pom.xml b/datamap/mv/plan/pom.xml index 4b8c9be..c4f841a 100644 --- a/datamap/mv/plan/pom.xml +++ b/datamap/mv/plan/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.6.1-SNAPSHOT +1.6.1 ../../../pom.xml diff --git a/examples/flink/pom.xml b/examples/flink/pom.xml index 5013f67..8ba9eea 100644 --- a/examples/flink/pom.xml +++ b/examples/flink/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.6.1-SNAPSHOT +1.6.1 ../../pom.xml diff --git a/examples/spark2/pom.xml b/examples/spark2/pom.xml index ad0d3ec..518705e 100644 --- a/examples/spark2/pom.xml +++ b/examples/spark2/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.6.1-SNAPSHOT +1.6.1 ../../pom.xml diff --git a/format/pom.xml b/format/pom.xml index 81aa95b..d07ad34 100644 --- a/format/pom.xml +++ b/format/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.6.1-SNAPSHOT +1.6.1 ../pom.xml diff --git a/hadoop/pom.xml b/hadoop/pom.xml index bcb5696..7089ae2 100644 --- a/hadoop/pom.xml +++ b/hadoop/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.6.1-SNAPSHOT +1.6.1 ../pom.xml diff --git a/integration/hive/pom.xml b/integration/hive/pom.xml index dfa8810..f0d23d3 100644 --- a/integration/hive/pom.xml +++ b/integration/hive
[carbondata] annotated tag apache-CarbonData-1.6.1-rc1 created (now 5a54270)
This is an automated email from the ASF dual-hosted git repository. raghunandan pushed a change to annotated tag apache-CarbonData-1.6.1-rc1 in repository https://gitbox.apache.org/repos/asf/carbondata.git. at 5a54270 (tag) tagging cabde6252d4a527fbfeb7f17627c6dce3e357f84 (commit) replaces apache-CarbonData-1.6.0-rc3 by Raghunandan S on Mon Oct 7 10:22:40 2019 +0530 - Log - [maven-release-plugin] copy for tag apache-CarbonData-1.6.1-rc1 --- No new revisions were added by this update.
[carbondata] branch branch-1.6 updated: [maven-release-plugin] prepare for next development iteration
This is an automated email from the ASF dual-hosted git repository. raghunandan pushed a commit to branch branch-1.6 in repository https://gitbox.apache.org/repos/asf/carbondata.git The following commit(s) were added to refs/heads/branch-1.6 by this push: new 63ebaab [maven-release-plugin] prepare for next development iteration 63ebaab is described below commit 63ebaabaaadecf73856257573b9816d0a24a3a66 Author: Raghunandan S AuthorDate: Mon Oct 7 10:23:00 2019 +0530 [maven-release-plugin] prepare for next development iteration --- assembly/pom.xml | 2 +- common/pom.xml| 2 +- core/pom.xml | 2 +- datamap/bloom/pom.xml | 2 +- datamap/examples/pom.xml | 2 +- datamap/lucene/pom.xml| 2 +- datamap/mv/core/pom.xml | 2 +- datamap/mv/plan/pom.xml | 2 +- examples/flink/pom.xml| 2 +- examples/spark2/pom.xml | 2 +- format/pom.xml| 2 +- hadoop/pom.xml| 2 +- integration/hive/pom.xml | 2 +- integration/presto/pom.xml| 2 +- integration/spark-common-test/pom.xml | 2 +- integration/spark-common/pom.xml | 2 +- integration/spark-datasource/pom.xml | 2 +- integration/spark2/pom.xml| 2 +- pom.xml | 4 ++-- processing/pom.xml| 2 +- store/sdk/pom.xml | 2 +- streaming/pom.xml | 2 +- tools/cli/pom.xml | 2 +- 23 files changed, 24 insertions(+), 24 deletions(-) diff --git a/assembly/pom.xml b/assembly/pom.xml index 3b76af6..ea1e6e6 100644 --- a/assembly/pom.xml +++ b/assembly/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.6.1 +1.6.2-SNAPSHOT ../pom.xml diff --git a/common/pom.xml b/common/pom.xml index b9399f5..e506998 100644 --- a/common/pom.xml +++ b/common/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.6.1 +1.6.2-SNAPSHOT ../pom.xml diff --git a/core/pom.xml b/core/pom.xml index 115503e..53c4651 100644 --- a/core/pom.xml +++ b/core/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.6.1 +1.6.2-SNAPSHOT ../pom.xml diff --git a/datamap/bloom/pom.xml b/datamap/bloom/pom.xml index 9aa736f..6b82af5 100644 --- a/datamap/bloom/pom.xml +++ b/datamap/bloom/pom.xml @@ -4,7 +4,7 @@ org.apache.carbondata carbondata-parent -1.6.1 +1.6.2-SNAPSHOT ../../pom.xml diff --git a/datamap/examples/pom.xml b/datamap/examples/pom.xml index 6ea0c62..c337d8a 100644 --- a/datamap/examples/pom.xml +++ b/datamap/examples/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.6.1 +1.6.2-SNAPSHOT ../../pom.xml diff --git a/datamap/lucene/pom.xml b/datamap/lucene/pom.xml index 2220aaf..e764200 100644 --- a/datamap/lucene/pom.xml +++ b/datamap/lucene/pom.xml @@ -4,7 +4,7 @@ org.apache.carbondata carbondata-parent -1.6.1 +1.6.2-SNAPSHOT ../../pom.xml diff --git a/datamap/mv/core/pom.xml b/datamap/mv/core/pom.xml index 8d71cdd..10d9817 100644 --- a/datamap/mv/core/pom.xml +++ b/datamap/mv/core/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.6.1 +1.6.2-SNAPSHOT ../../../pom.xml diff --git a/datamap/mv/plan/pom.xml b/datamap/mv/plan/pom.xml index c4f841a..c3b384c 100644 --- a/datamap/mv/plan/pom.xml +++ b/datamap/mv/plan/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.6.1 +1.6.2-SNAPSHOT ../../../pom.xml diff --git a/examples/flink/pom.xml b/examples/flink/pom.xml index 8ba9eea..6944e41 100644 --- a/examples/flink/pom.xml +++ b/examples/flink/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.6.1 +1.6.2-SNAPSHOT ../../pom.xml diff --git a/examples/spark2/pom.xml b/examples/spark2/pom.xml index 518705e..42a27d4 100644 --- a/examples/spark2/pom.xml +++ b/examples/spark2/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.6.1 +1.6.2-SNAPSHOT ../../pom.xml diff --git a/format/pom.xml b/format/pom.xml index d07ad34..a5d5428 100644 --- a/format/pom.xml +++ b/format/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.6.1 +1.6.2-SNAPSHOT ../pom.xml diff --git a/hadoop/pom.xml b/hadoop/pom.xml index 7089ae2..9ddd870 100644 --- a/hadoop/pom.xml +++ b/hadoop/pom.xml @@ -22,7 +22,7 @@ org.apache.carbondata carbondata-parent -1.6.1 +1.6.2-SNAPSHOT ../pom.xml diff --git a/integration/hive/pom.xml b/integration/hive/pom.xml index f0d23d3..11ac363 100644 --- a/integration/hive/pom.xml +++ b/integration/hive/pom.xml @@ -22,7
svn commit: r36233 - in /dev/carbondata/1.6.1-rc1: ./ apache-carbondata-1.6.1-source-release.zip apache-carbondata-1.6.1-source-release.zip.asc apache-carbondata-1.6.1-source-release.zip.sha512
Author: raghunandan Date: Mon Oct 7 14:23:57 2019 New Revision: 36233 Log: Upload 1.6.1-RC1 Added: dev/carbondata/1.6.1-rc1/ dev/carbondata/1.6.1-rc1/apache-carbondata-1.6.1-source-release.zip (with props) dev/carbondata/1.6.1-rc1/apache-carbondata-1.6.1-source-release.zip.asc dev/carbondata/1.6.1-rc1/apache-carbondata-1.6.1-source-release.zip.sha512 Added: dev/carbondata/1.6.1-rc1/apache-carbondata-1.6.1-source-release.zip == Binary file - no diff available. Propchange: dev/carbondata/1.6.1-rc1/apache-carbondata-1.6.1-source-release.zip -- svn:mime-type = application/octet-stream Added: dev/carbondata/1.6.1-rc1/apache-carbondata-1.6.1-source-release.zip.asc == --- dev/carbondata/1.6.1-rc1/apache-carbondata-1.6.1-source-release.zip.asc (added) +++ dev/carbondata/1.6.1-rc1/apache-carbondata-1.6.1-source-release.zip.asc Mon Oct 7 14:23:57 2019 @@ -0,0 +1,11 @@ +-BEGIN PGP SIGNATURE- + +iQEzBAABCAAdFiEEsZE8naWI0MngB++fuw0pZv1r+vAFAl2bQh0ACgkQuw0pZv1r ++vAy2QgApN5uNnJpXtycYT4y284iyv7I6rKJVOpT+0zZB9rsAUKnOq/5d3W5QJIk +7qpCReE5hZGXIgLnTsOalJYhb6VLGGqPi0E2lNG13jypGRz8LEkLupVnAC2Uh0ni +kpBziKWa03NXjhxmSuB8AJo2z0inLbEtHW4C1usX0CvkSjfoodrl7rxdEnAaUSs4 +3+MP4CQus+eblMsNZffW0+2jHLtZ5PgacXMn1bEsfBcl6YC7elibMCHByMY7l7cP +XPBkqh4rITeRixaAdDs+d7YD0fF79FLxHPHRtL5CElo+iCPIEMTc++2bY10ZHm+9 ++2SXP5CHwnzIWAHGeu1HMSdwbaXeQw== +=gQhA +-END PGP SIGNATURE- Added: dev/carbondata/1.6.1-rc1/apache-carbondata-1.6.1-source-release.zip.sha512 == --- dev/carbondata/1.6.1-rc1/apache-carbondata-1.6.1-source-release.zip.sha512 (added) +++ dev/carbondata/1.6.1-rc1/apache-carbondata-1.6.1-source-release.zip.sha512 Mon Oct 7 14:23:57 2019 @@ -0,0 +1 @@ +77e1ed80fe5e5445d09fd1693f65578ba057334d901652678c9e36307bb4346c18285dc71d2cf17954fa8f1c823aea3a48a72fd70eaabeef06267b1ee2390949 apache-carbondata-1.6.1-source-release.zip
[carbondata-site] branch asf-site updated: Added 1.6.1 version information
This is an automated email from the ASF dual-hosted git repository.
raghunandan pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/carbondata-site.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 0203755 Added 1.6.1 version information
0203755 is described below
commit 020375541fedbdebc8d16b5e0c7c6f2b6ee343bf
Author: Raghunandan S
AuthorDate: Tue Oct 29 11:28:17 2019 +0530
Added 1.6.1 version information
---
content/CSDK-guide.html | 5 -
content/alluxio-guide.html| 5 -
content/bloomfilter-datamap-guide.html| 5 -
content/carbon-as-spark-datasource-guide.html | 5 -
content/configuration-parameters.html | 10 +-
content/datamap-developer-guide.html | 5 -
content/datamap-management.html | 7 +--
content/ddl-of-carbondata.html| 9 ++---
content/dml-of-carbondata.html| 5 -
content/documentation.html| 7 +--
content/faq.html | 5 -
content/file-structure-of-carbondata.html | 5 -
content/hive-guide.html | 5 -
content/how-to-contribute-to-apache-carbondata.html | 5 -
content/index-server.html | 7 +--
content/index.html| 19 +++
content/introduction.html | 5 -
content/language-manual.html | 5 -
content/lucene-datamap-guide.html | 5 -
content/mv-datamap-guide.html | 9 -
content/performance-tuning.html | 5 -
content/preaggregate-datamap-guide.html | 5 -
content/presto-guide.html | 5 -
content/quick-start-guide.html| 13 -
content/release-guide.html| 5 -
content/s3-guide.html | 5 -
content/sdk-guide.html| 5 -
content/security.html | 3 +++
content/segment-management-on-carbondata.html | 5 -
content/streaming-guide.html | 5 -
content/supported-data-types-in-carbondata.html | 5 -
content/timeseries-datamap-guide.html | 5 -
content/usecases.html | 5 -
content/videogallery.html | 3 +++
src/main/scala/html/header.html | 5 -
src/main/webapp/alluxio-guide.html| 5 -
src/main/webapp/bloomfilter-datamap-guide.html| 5 -
src/main/webapp/carbon-as-spark-datasource-guide.html | 5 -
src/main/webapp/configuration-parameters.html | 10 +-
src/main/webapp/{CSDK-guide.html => csdk-guide.html} | 5 -
src/main/webapp/datamap-developer-guide.html | 5 -
src/main/webapp/datamap-management.html | 7 +--
src/main/webapp/ddl-of-carbondata.html| 9 ++---
src/main/webapp/dml-of-carbondata.html| 5 -
src/main/webapp/documentation.html| 7 +--
src/main/webapp/faq.html | 5 -
src/main/webapp/file-structure-of-carbondata.html | 5 -
src/main/webapp/hive-guide.html | 5 -
.../how-to-contribute-to-apache-carbondata.html | 5 -
src/main/webapp/index-server.html | 7 +--
src/main/webapp/index.html| 3 +++
src/main/webapp/introduction.html | 5 -
src/main/webapp/language-manual.html | 5 -
src/main/webapp/lucene-datamap-guide.html | 5 -
src/main/webapp/mv-datamap-guide.html | 9 -
src/main/webapp/performance-tuning.html | 5 -
src/main/webapp/preaggregate-datamap-guide.html | 5 -
src/main/webapp/presto-guide.html | 5 -
src/main/webapp/quick-start-guide.html| 13 -
src/main/webapp/release-guide.html| 5 -
src/main/webapp/s3-guide.html | 5 -
src/main/webapp/sdk-guide.html| 5 -
src/main/webapp/security.html | 3 +++
src/main/webapp/segment-management-on-carbondata.html | 5 -
src/main/webapp/streaming-guide.html | 5 -
.../webapp/supported-data-types-in-carbondata.html| 5 -
src/main/webapp/timeseries-datamap-guide.html | 5 -
src/main/web
[carbondata-site] branch asf-site updated: Renamed CSDK-guide.html to csdk.html
This is an automated email from the ASF dual-hosted git repository. raghunandan pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/carbondata-site.git The following commit(s) were added to refs/heads/asf-site by this push: new e103fc4 Renamed CSDK-guide.html to csdk.html e103fc4 is described below commit e103fc4db5a8dd2bc4e343342178549691ba633d Author: Raghunandan S AuthorDate: Wed Oct 30 15:04:51 2019 +0530 Renamed CSDK-guide.html to csdk.html --- content/csdk-guide.html | 687 1 file changed, 687 insertions(+) diff --git a/content/csdk-guide.html b/content/csdk-guide.html new file mode 100644 index 000..d775939 --- /dev/null +++ b/content/csdk-guide.html @@ -0,0 +1,687 @@ + + + + + + + + +CarbonData + + + + + + + + + + + + +https://use.fontawesome.com/releases/v5.0.8/js/all.js"</a>;> + + + + + + + + + +Toggle navigation + + + + + + + + + + + + +Home + + Download + + +https://dist.apache.org/repos/dist/release/carbondata/1.6.1/"; + target="_blank">Apache CarbonData 1.6.1 + +https://dist.apache.org/repos/dist/release/carbondata/1.6.0/"; + target="_blank">Apache CarbonData 1.6.0 + +https://dist.apache.org/repos/dist/release/carbondata/1.5.4/"; + target="_blank">Apache CarbonData 1.5.4 + +https://dist.apache.org/repos/dist/release/carbondata/1.5.3/"; + target="_blank">Apache CarbonData 1.5.3 + +https://dist.apache.org/repos/dist/release/carbondata/1.5.2/"; + target="_blank">Apache CarbonData 1.5.2 + +https://dist.apache.org/repos/dist/release/carbondata/1.5.1/"; + target="_blank">Apache CarbonData 1.5.1 + +https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"; + target="_blank">Apache CarbonData 1.5.0 + +https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"; + target="_blank">Apache CarbonData 1.4.1 + +https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"; + target="_blank">Release Archive + + +Documentation + +Community + + +https://github.com/apache/carbondata/blob/master/docs/how-to-contribute-to-apache-carbondata.md"; + target="_blank">Contributing to CarbonData + +https://github.com/apache/carbondata/blob/master/docs/release-guide.md"; + target="_blank">Release Guide + +https://cwiki.apache.org/confluence/display/CARBONDATA/PMC+and+Committers+member+list"; + target="_blank">Project PMC and Committers + +https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=66850609"; + target="_blank">CarbonData Meetups +Apache CarbonData Security +https://issues.apache.org/jira/browse/CARBONDATA"; target="_blank">Apache +Jira +CarbonData Videos + + + +http://www.apache.org/"; class="apache_link hidden-xs dropdown-toggle" +
[carbondata-site] branch asf-site updated: Added new TPCH performance reports
This is an automated email from the ASF dual-hosted git repository. raghunandan pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/carbondata-site.git The following commit(s) were added to refs/heads/asf-site by this push: new c9a44ce Added new TPCH performance reports c9a44ce is described below commit c9a44ce69be5e09596db1e449c4350c5f00fe7b8 Author: Raghunandan S AuthorDate: Wed Oct 30 15:12:50 2019 +0530 Added new TPCH performance reports --- content/documentation.html | 2 +- src/main/webapp/documentation.html | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/content/documentation.html b/content/documentation.html index 395f0f3..77505dc 100644 --- a/content/documentation.html +++ b/content/documentation.html @@ -256,7 +256,7 @@ faster data format.Contributing to CarbonData doesn?t just mean writing code. He Wiki: You can read the https://cwiki.apache.org/confluence/display/CARBONDATA/CarbonData+Home"; rel="nofollow">Apache CarbonData wiki page for upcoming release plan, blogs and training materials. Summit: Presentations from past summits and conferences can be found https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=66850609"; rel="nofollow">here. Blogs: Blogs by external users can be found https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=67635497"; rel="nofollow">here. -Performance reports: TPC-H performance reports can be found https://cwiki.apache.org/confluence/display/CARBONDATA/Performance+-+TPCH+Report+of+CarbonData+%281.2+version%29+and+Parquet+on+Spark+Execution+Engine"; rel="nofollow">here. +Performance reports: TPC-H performance reports can be found https://cwiki.apache.org/confluence/display/CARBONDATA/TPCH+Report+of+CarbonData+%281.5.1+version%29+and+Parquet+on+Spark+Execution+Engine"; rel="nofollow">here. Trainings: Training records on design and code flows can be found https://cwiki.apache.org/confluence/display/CARBONDATA/CarbonData+Training+Materials"; rel="nofollow">here. diff --git a/src/main/webapp/documentation.html b/src/main/webapp/documentation.html index 395f0f3..77505dc 100644 --- a/src/main/webapp/documentation.html +++ b/src/main/webapp/documentation.html @@ -256,7 +256,7 @@ faster data format.Contributing to CarbonData doesn?t just mean writing code. He Wiki: You can read the https://cwiki.apache.org/confluence/display/CARBONDATA/CarbonData+Home"; rel="nofollow">Apache CarbonData wiki page for upcoming release plan, blogs and training materials. Summit: Presentations from past summits and conferences can be found https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=66850609"; rel="nofollow">here. Blogs: Blogs by external users can be found https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=67635497"; rel="nofollow">here. -Performance reports: TPC-H performance reports can be found https://cwiki.apache.org/confluence/display/CARBONDATA/Performance+-+TPCH+Report+of+CarbonData+%281.2+version%29+and+Parquet+on+Spark+Execution+Engine"; rel="nofollow">here. +Performance reports: TPC-H performance reports can be found https://cwiki.apache.org/confluence/display/CARBONDATA/TPCH+Report+of+CarbonData+%281.5.1+version%29+and+Parquet+on+Spark+Execution+Engine"; rel="nofollow">here. Trainings: Training records on design and code flows can be found https://cwiki.apache.org/confluence/display/CARBONDATA/CarbonData+Training+Materials"; rel="nofollow">here.
[carbondata-site] branch asf-site updated: Updated copyright from 2017-2018 to 2020-2021 (#73)
This is an automated email from the ASF dual-hosted git repository. raghunandan pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/carbondata-site.git The following commit(s) were added to refs/heads/asf-site by this push: new 59825be Updated copyright from 2017-2018 to 2020-2021 (#73) 59825be is described below commit 59825bebb4d7795737e474fc1ca5d6ea94225af2 Author: Karan-c980 <63015850+karan-c...@users.noreply.github.com> AuthorDate: Mon May 4 10:02:14 2020 +0530 Updated copyright from 2017-2018 to 2020-2021 (#73) * Updated copyright from 2017-2018 to 2020-2021 --- content/index.html | 2 +- src/main/webapp/index.html | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/content/index.html b/content/index.html index d14b329..5e8bb16 100644 --- a/content/index.html +++ b/content/index.html @@ -692,7 +692,7 @@ -Copyright © 2017-2018 Apache CarbonData. All rights reserved | +Copyright © 2020-2021 Apache CarbonData. All rights reserved | http://www.apache.org/"; class="term-links" target="_blank">Apache Software Foundation | http://www.apache.org/foundation/policies/privacy.html"; diff --git a/src/main/webapp/index.html b/src/main/webapp/index.html index 86ce35c..b25a63b 100644 --- a/src/main/webapp/index.html +++ b/src/main/webapp/index.html @@ -671,7 +671,7 @@ -Copyright © 2017-2018 Apache CarbonData. All rights reserved | +Copyright © 2020-2021 Apache CarbonData. All rights reserved | http://www.apache.org/"; class="term-links" target="_blank">Apache Software Foundation | http://www.apache.org/foundation/policies/privacy.html";
[carbondata-site] branch asf-site updated: Corrected Link to MV (#76)
This is an automated email from the ASF dual-hosted git repository. raghunandan pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/carbondata-site.git The following commit(s) were added to refs/heads/asf-site by this push: new 9fa2b4e Corrected Link to MV (#76) 9fa2b4e is described below commit 9fa2b4e9c162a7f8f6f45a2d80dd966b3fc4e0fa Author: sraghunandan AuthorDate: Tue Jun 2 18:21:19 2020 +0530 Corrected Link to MV (#76) This closes #76 --- content/alluxio-guide.html | 4 ++-- content/bloomfilter-index-guide.html| 4 ++-- content/carbon-as-spark-datasource-guide.html | 4 ++-- content/configuration-parameters.html | 4 ++-- content/csdk-guide.html | 4 ++-- content/ddl-of-carbondata.html | 4 ++-- content/dml-of-carbondata.html | 4 ++-- content/documentation.html | 4 ++-- content/faq.html| 4 ++-- content/file-structure-of-carbondata.html | 4 ++-- content/flink-integration-guide.html| 4 ++-- content/hive-guide.html | 4 ++-- content/how-to-contribute-to-apache-carbondata.html | 4 ++-- content/index-developer-guide.html | 4 ++-- content/index-management.html | 4 ++-- content/index-server.html | 4 ++-- content/introduction.html | 4 ++-- content/language-manual.html| 4 ++-- content/lucene-index-guide.html | 4 ++-- content/mv-guide.html | 4 ++-- content/performance-tuning.html | 4 ++-- content/prestodb-guide.html | 4 ++-- content/prestosql-guide.html| 4 ++-- content/quick-start-guide.html | 4 ++-- content/release-guide.html | 4 ++-- content/s3-guide.html | 4 ++-- content/scd-and-cdc-guide.html | 4 ++-- content/sdk-guide.html | 4 ++-- content/secondary-index-guide.html | 4 ++-- content/segment-management-on-carbondata.html | 4 ++-- content/spatial-index-guide.html| 4 ++-- content/streaming-guide.html| 4 ++-- content/supported-data-types-in-carbondata.html | 4 ++-- content/usecases.html | 4 ++-- src/main/scala/html/header.html | 4 ++-- src/main/webapp/alluxio-guide.html | 4 ++-- src/main/webapp/bloomfilter-index-guide.html| 4 ++-- src/main/webapp/carbon-as-spark-datasource-guide.html | 4 ++-- src/main/webapp/configuration-parameters.html | 4 ++-- src/main/webapp/csdk-guide.html | 4 ++-- src/main/webapp/ddl-of-carbondata.html | 4 ++-- src/main/webapp/dml-of-carbondata.html | 4 ++-- src/main/webapp/documentation.html | 4 ++-- src/main/webapp/faq.html| 4 ++-- src/main/webapp/file-structure-of-carbondata.html | 4 ++-- src/main/webapp/flink-integration-guide.html| 4 ++-- src/main/webapp/hive-guide.html | 4 ++-- src/main/webapp/how-to-contribute-to-apache-carbondata.html | 4 ++-- src/main/webapp/index-developer-guide.html | 4 ++-- src/main/webapp/index-management.html | 4 ++-- src/main/webapp/index-server.html | 4 ++-- src/main/webapp/introduction.html | 4 ++-- src/main/webapp/language-manual.html| 4 ++-- src/main/webapp/lucene-index-guide.html | 4 ++-- src/main/webapp/mv-guide.html | 4 ++-- src/main/webapp/performance-tuning.html | 4 ++-- src/main/webapp/prestodb-guide.html | 4 ++-- src/main/webapp/prestosql-guide.html| 4 ++-- src/main/webapp/quick-start-guide.html | 4 ++-- src/main/webapp/release-guide.html | 4 ++-- src/main/webapp/s3-guide.html | 4 ++-- src/main/webapp/scd-and-cdc-guide.html | 4 ++-- src/main/webapp/sdk-guide.html | 4 ++-- src/main/webapp/secondary-index-guide.html | 4 ++-- src/main
[carbondata-site] branch asf-site updated: Corrected Link to Indexes (#77)
This is an automated email from the ASF dual-hosted git repository. raghunandan pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/carbondata-site.git The following commit(s) were added to refs/heads/asf-site by this push: new 0810717 Corrected Link to Indexes (#77) 0810717 is described below commit 0810717be9bef6d7653731a96fe0bcdacf5d61c3 Author: sraghunandan AuthorDate: Wed Jun 3 10:24:53 2020 +0530 Corrected Link to Indexes (#77) This closes #77 --- content/alluxio-guide.html | 2 +- content/bloomfilter-index-guide.html| 4 ++-- content/carbon-as-spark-datasource-guide.html | 2 +- content/configuration-parameters.html | 2 +- content/csdk-guide.html | 2 +- content/ddl-of-carbondata.html | 2 +- content/dml-of-carbondata.html | 2 +- content/documentation.html | 2 +- content/faq.html| 2 +- content/file-structure-of-carbondata.html | 2 +- content/flink-integration-guide.html| 2 +- content/hive-guide.html | 2 +- content/how-to-contribute-to-apache-carbondata.html | 2 +- content/index-developer-guide.html | 4 ++-- content/index-management.html | 2 +- content/index-server.html | 2 +- content/introduction.html | 2 +- content/language-manual.html| 10 +- content/lucene-index-guide.html | 2 +- content/mv-guide.html | 2 +- content/performance-tuning.html | 8 content/prestodb-guide.html | 2 +- content/prestosql-guide.html| 2 +- content/quick-start-guide.html | 2 +- content/release-guide.html | 2 +- content/s3-guide.html | 2 +- content/scd-and-cdc-guide.html | 2 +- content/sdk-guide.html | 2 +- content/secondary-index-guide.html | 2 +- content/segment-management-on-carbondata.html | 2 +- content/spatial-index-guide.html| 2 +- content/streaming-guide.html| 2 +- content/supported-data-types-in-carbondata.html | 2 +- content/usecases.html | 2 +- src/main/scala/html/header.html | 2 +- src/main/webapp/alluxio-guide.html | 2 +- src/main/webapp/bloomfilter-index-guide.html| 4 ++-- src/main/webapp/carbon-as-spark-datasource-guide.html | 2 +- src/main/webapp/configuration-parameters.html | 2 +- src/main/webapp/csdk-guide.html | 2 +- src/main/webapp/ddl-of-carbondata.html | 2 +- src/main/webapp/dml-of-carbondata.html | 2 +- src/main/webapp/documentation.html | 2 +- src/main/webapp/faq.html| 2 +- src/main/webapp/file-structure-of-carbondata.html | 2 +- src/main/webapp/flink-integration-guide.html| 2 +- src/main/webapp/hive-guide.html | 2 +- src/main/webapp/how-to-contribute-to-apache-carbondata.html | 2 +- src/main/webapp/index-developer-guide.html | 4 ++-- src/main/webapp/index-management.html | 2 +- src/main/webapp/index-server.html | 2 +- src/main/webapp/introduction.html | 2 +- src/main/webapp/language-manual.html| 10 +- src/main/webapp/lucene-index-guide.html | 2 +- src/main/webapp/mv-guide.html | 2 +- src/main/webapp/performance-tuning.html | 8 src/main/webapp/prestodb-guide.html | 2 +- src/main/webapp/prestosql-guide.html| 2 +- src/main/webapp/quick-start-guide.html | 2 +- src/main/webapp/release-guide.html | 2 +- src/main/webapp/s3-guide.html | 2 +- src/main/webapp/scd-and-cdc-guide.html | 2 +- src/main/webapp/sdk-guide.html | 2 +- src/main/webapp/secondary-index-guide.html | 2 +- src/main/webapp/segment
[carbondata-site] branch asf-site updated: updated website for 2.1.0 (#78)
This is an automated email from the ASF dual-hosted git repository. raghunandan pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/carbondata-site.git The following commit(s) were added to refs/heads/asf-site by this push: new cdcd1ab updated website for 2.1.0 (#78) cdcd1ab is described below commit cdcd1ab890a1d6bfa5cd2032210bf5d03c2bfe6b Author: Kunal Kapoor AuthorDate: Mon Jan 4 12:53:40 2021 +0530 updated website for 2.1.0 (#78) --- content/WEB-INF/classes/META-INF/NOTICE| 2 +- content/alluxio-guide.html | 3 +++ content/bloomfilter-index-guide.html | 3 +++ content/carbon-as-spark-datasource-guide.html | 3 +++ content/configuration-parameters.html | 3 +++ content/csdk-guide.html| 3 +++ content/ddl-of-carbondata.html | 3 +++ content/dml-of-carbondata.html | 3 +++ content/documentation.html | 3 +++ content/faq.html | 3 +++ content/file-structure-of-carbondata.html | 3 +++ content/flink-integration-guide.html | 3 +++ content/hive-guide.html| 3 +++ content/how-to-contribute-to-apache-carbondata.html| 3 +++ content/index-developer-guide.html | 3 +++ content/index-management.html | 3 +++ content/index-server.html | 3 +++ content/index.html | 16 +--- content/introduction.html | 3 +++ content/language-manual.html | 3 +++ content/lucene-index-guide.html| 3 +++ content/mv-guide.html | 3 +++ content/pdf/TPCDS_Report_CarbonData2.1.0.pdf | Bin 0 -> 717503 bytes content/performance-tuning.html| 3 +++ content/prestodb-guide.html| 3 +++ content/prestosql-guide.html | 3 +++ content/quick-start-guide.html | 3 +++ content/release-guide.html | 3 +++ content/s3-guide.html | 3 +++ content/scd-and-cdc-guide.html | 3 +++ content/sdk-guide.html | 3 +++ content/secondary-index-guide.html | 3 +++ content/security.html | 3 +++ content/segment-management-on-carbondata.html | 3 +++ content/spatial-index-guide.html | 3 +++ content/streaming-guide.html | 3 +++ content/supported-data-types-in-carbondata.html| 3 +++ content/usecases.html | 3 +++ content/videogallery.html | 3 +++ src/main/scala/html/header.html| 3 +++ src/main/webapp/alluxio-guide.html | 3 +++ src/main/webapp/bloomfilter-index-guide.html | 3 +++ src/main/webapp/carbon-as-spark-datasource-guide.html | 3 +++ src/main/webapp/configuration-parameters.html | 3 +++ src/main/webapp/csdk-guide.html| 3 +++ src/main/webapp/ddl-of-carbondata.html | 3 +++ src/main/webapp/dml-of-carbondata.html | 3 +++ src/main/webapp/documentation.html | 3 +++ src/main/webapp/faq.html | 3 +++ src/main/webapp/file-structure-of-carbondata.html | 3 +++ src/main/webapp/flink-integration-guide.html | 3 +++ src/main/webapp/hive-guide.html| 3 +++ .../webapp/how-to-contribute-to-apache-carbondata.html | 3 +++ src/main/webapp/index-developer-guide.html | 3 +++ src/main/webapp/index-management.html | 3 +++ src/main/webapp/index-server.html | 3 +++ src/main/webapp/index.html | 7 +-- src/main/webapp/introduction.html | 3 +++ src/main/webapp/language-manual.html | 3 +++ src/main/webapp/lucene-index-guide.html| 3 +++ src/main/webapp/mv-guide.html | 3 +++ src/main/webapp/performance-tuning.html| 3 +++ src/main/webapp/prestodb-guide.html| 3 +++ src/main/webapp/prestosql-guide.html | 3 +++ src/main/webapp/quick-start-guide.html | 3 +++ src/main/webapp/release-guide.html | 3 +++ src/main/webapp/s3-guide.html | 3 +++ src/main/webapp/scd-and-cdc-guide.h
