[jira] [Created] (CARBONDATA-213) Remove thrift complier dependency

2016-09-05 Thread QiangCai (JIRA) 
QiangCai created CARBONDATA-213:
---

 Summary: Remove thrift complier dependency
 Key: CARBONDATA-213
 URL: https://issues.apache.org/jira/browse/CARBONDATA-213
 Project: CarbonData
  Issue Type: Bug
Reporter: QiangCai
Priority: Minor
 Fix For: 0.2.0-incubating






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-carbondata pull request #126: [CARBONDATA-212] Use SQLContext to r...

2016-09-05 Thread jackylk 
GitHub user jackylk opened a pull request:

https://github.com/apache/incubator-carbondata/pull/126

[CARBONDATA-212] Use SQLContext to read CarbonData files

In this PR, user can use SQLContext instead of CarbonContext to read 
CarbonData files.
See DatasourceExample.scala

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jackylk/incubator-carbondata ds1

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-carbondata/pull/126.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #126


commit 9293e1b5c0c6c14b5f3d8f7aec89a812d785a452
Author: jackylk 
Date:   2016-09-05T15:36:04Z

use SqlContext for read

commit f0f7a4a2f0144dc1088ba2e8c6d4a606dc50a36a
Author: jackylk 
Date:   2016-09-05T15:43:30Z

revert CarbonScan

commit cc6c749115c35effed493d9064d982438c11cb59
Author: jackylk 
Date:   2016-09-05T15:45:33Z

add class

commit c53b2ef268b487f602350d4c93bc3cdda70e8c39
Author: jackylk 
Date:   2016-09-05T16:36:13Z

fix stylecheck




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (CARBONDATA-212) Use SQLContext to read CarbonData file

2016-09-05 Thread Jacky Li (JIRA) 
Jacky Li created CARBONDATA-212:
---

 Summary: Use SQLContext to read CarbonData file
 Key: CARBONDATA-212
 URL: https://issues.apache.org/jira/browse/CARBONDATA-212
 Project: CarbonData
  Issue Type: New Feature
Reporter: Jacky Li
Assignee: Jacky Li
 Fix For: 0.2.0-incubating


User should be enable to use Spark's SQLContext to read CarbonData files. 
SQLContext will use datasource API to read corresponding CarbonData files.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-carbondata pull request #123: [CARBONDATA-204] Clear queryStatisti...

2016-09-05 Thread foryou2030 
Github user foryou2030 commented on a diff in the pull request:

https://github.com/apache/incubator-carbondata/pull/123#discussion_r77511283
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/carbon/querystatistics/DriverQueryStatisticsRecorder.java
 ---
@@ -78,106 +83,148 @@ public synchronized void 
recordStatisticsForDriver(QueryStatistic statistic, Str
*/
   public void logStatisticsAsTableDriver() {
 synchronized (lock) {
-  String tableInfo = collectDriverStatistics();
-  if (null != tableInfo) {
-LOGGER.statistic(tableInfo);
+  Iterator> entries =
+  queryStatisticsMap.entrySet().iterator();
+  while (entries.hasNext()) {
+Map.Entry entry = entries.next();
+String queryId = entry.getKey();
+// clear the unknown query statistics
+if(StringUtils.isEmpty(queryId)) {
+  queryStatisticsMap.remove(queryId);
--- End diff --

ok, handled


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-carbondata pull request #123: [CARBONDATA-204] Clear queryStatisti...

2016-09-05 Thread foryou2030 
Github user foryou2030 commented on a diff in the pull request:

https://github.com/apache/incubator-carbondata/pull/123#discussion_r77506707
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/carbon/querystatistics/DriverQueryStatisticsRecorder.java
 ---
@@ -78,106 +83,148 @@ public synchronized void 
recordStatisticsForDriver(QueryStatistic statistic, Str
*/
   public void logStatisticsAsTableDriver() {
 synchronized (lock) {
-  String tableInfo = collectDriverStatistics();
-  if (null != tableInfo) {
-LOGGER.statistic(tableInfo);
+  Iterator> entries =
+  queryStatisticsMap.entrySet().iterator();
+  while (entries.hasNext()) {
+Map.Entry entry = entries.next();
+String queryId = entry.getKey();
+// clear the unknown query statistics
+if(StringUtils.isEmpty(queryId)) {
+  queryStatisticsMap.remove(queryId);
--- End diff --

I tried this, but caused some exceptions


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-carbondata pull request #123: [CARBONDATA-204] Clear queryStatisti...

2016-09-05 Thread Vimal-Das 
Github user Vimal-Das commented on a diff in the pull request:

https://github.com/apache/incubator-carbondata/pull/123#discussion_r77496731
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/carbon/querystatistics/DriverQueryStatisticsRecorder.java
 ---
@@ -78,106 +83,148 @@ public synchronized void 
recordStatisticsForDriver(QueryStatistic statistic, Str
*/
   public void logStatisticsAsTableDriver() {
 synchronized (lock) {
-  String tableInfo = collectDriverStatistics();
-  if (null != tableInfo) {
-LOGGER.statistic(tableInfo);
+  Iterator> entries =
+  queryStatisticsMap.entrySet().iterator();
+  while (entries.hasNext()) {
+Map.Entry entry = entries.next();
+String queryId = entry.getKey();
+// clear the unknown query statistics
+if(StringUtils.isEmpty(queryId)) {
+  queryStatisticsMap.remove(queryId);
--- End diff --

use the Iterator.remove() for better safety


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-carbondata pull request #125: [CARBONDATA-209] add DROP TABLE in a...

2016-09-05 Thread asfgit 
Github user asfgit closed the pull request at:

https://github.com/apache/incubator-carbondata/pull/125


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-carbondata pull request #123: [CARBONDATA-204] Clear queryStatisti...

2016-09-05 Thread foryou2030 
Github user foryou2030 commented on a diff in the pull request:

https://github.com/apache/incubator-carbondata/pull/123#discussion_r77488614
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/carbon/querystatistics/DriverQueryStatisticsRecorder.java
 ---
@@ -78,106 +82,142 @@ public synchronized void 
recordStatisticsForDriver(QueryStatistic statistic, Str
*/
   public void logStatisticsAsTableDriver() {
 synchronized (lock) {
-  String tableInfo = collectDriverStatistics();
-  if (null != tableInfo) {
-LOGGER.statistic(tableInfo);
+  for (String key: queryStatisticsMap.keySet()) {
+// print 
sql_parse_t,load_meta_t,block_allocation_t,block_identification_t
+// or just print block_allocation_t,block_identification_t
+if (queryStatisticsMap.get(key).size() >= 2) {
+  String tableInfo = collectDriverStatistics(key);
+  if (null != tableInfo) {
+LOGGER.statistic(tableInfo);
+  }
+}
+// clear timeout query statistics
+if(StringUtils.isEmpty(key)) {
+  queryStatisticsMap.remove(key);
+} else {
+  long interval = System.nanoTime() - Long.parseLong(key);
+  if (interval > 
QueryStatisticsConstants.CLEAR_STATISTICS_TIMEOUT) {
+queryStatisticsMap.remove(key);
+  }
+}
   }
 }
   }
 
   /**
* Below method will parse queryStatisticsMap and put time into table
*/
-  public String collectDriverStatistics() {
-for (String key: queryStatisticsMap.keySet()) {
-  try {
-// TODO: get the finished query, and print Statistics
-if (queryStatisticsMap.get(key).size() > 3) {
-  String sql_parse_time = "";
-  String load_meta_time = "";
-  String block_allocation_time = "";
-  String block_identification_time = "";
-  Double driver_part_time_tmp = 0.0;
-  String splitChar = " ";
-  // get statistic time from the QueryStatistic
-  for (QueryStatistic statistic : queryStatisticsMap.get(key)) {
-switch (statistic.getMessage()) {
-  case QueryStatisticsConstants.SQL_PARSE:
-sql_parse_time += statistic.getTimeTaken() + splitChar;
-driver_part_time_tmp += statistic.getTimeTaken();
-break;
-  case QueryStatisticsConstants.LOAD_META:
-load_meta_time += statistic.getTimeTaken() + splitChar;
-driver_part_time_tmp += statistic.getTimeTaken();
-break;
-  case QueryStatisticsConstants.BLOCK_ALLOCATION:
-block_allocation_time += statistic.getTimeTaken() + 
splitChar;
-driver_part_time_tmp += statistic.getTimeTaken();
-break;
-  case QueryStatisticsConstants.BLOCK_IDENTIFICATION:
-block_identification_time += statistic.getTimeTaken() + 
splitChar;
-driver_part_time_tmp += statistic.getTimeTaken();
-break;
-  default:
-break;
-}
-  }
-  String driver_part_time = driver_part_time_tmp + splitChar;
-  // structure the query statistics info table
-  StringBuilder tableInfo = new StringBuilder();
-  int len1 = 8;
-  int len2 = 20;
-  int len3 = 21;
-  int len4 = 22;
-  String line = "+" + printLine("-", len1) + "+" + printLine("-", 
len2) + "+" +
-  printLine("-", len3) + "+" + printLine("-", len4) + "+";
-  String line2 = "|" + printLine(" ", len1) + "+" + printLine("-", 
len2) + "+" +
-  printLine(" ", len3) + "+" + printLine("-", len4) + "+";
-  // table header
-  tableInfo.append(line).append("\n");
-  tableInfo.append("|" + printLine(" ", (len1 - 
"Module".length())) + "Module" + "|" +
-  printLine(" ", (len2 - "Operation Step".length())) + 
"Operation Step" + "|" +
-  printLine(" ", (len3 + len4 + 1 - "Query Cost".length())) +
-  "Query Cost" + "|" + "\n");
-  // driver part
-  tableInfo.append(line).append("\n");
-  tableInfo.append("|" + printLine(" ", len1) + "|" +
-  printLine(" ", (len2 - "SQL parse".length())) + "SQL parse" 
+ "|" +
-  printLine(" ", len3) + "|" +
-  printLine(" ", (len4 - sql_parse_time.length())) + 
sql_parse_time + "|" + "\n");
-  tableInfo.append(line2).append("\n");
-  tableInfo.append("|" +printLine(" ", (len1 - "Driver".length())) 
+ "Driver" + "|"

[jira] [Created] (CARBONDATA-211) Support compress CarbonData file create table options

2016-09-05 Thread Jacky Li (JIRA) 
Jacky Li created CARBONDATA-211:
---

 Summary: Support compress CarbonData file create table options
 Key: CARBONDATA-211
 URL: https://issues.apache.org/jira/browse/CARBONDATA-211
 Project: CarbonData
  Issue Type: Bug
Reporter: Jacky Li


Currently CarbonData uses Snappy as codec to compress its columnar file, this 
Issue is targeted to support:
1. support compresssion codec other than Snappy, including LZO, LZ4, ZLIB
2. add table property in CREATE TABLE syntax. Users can specify the codec in 
CREATE TABLE statement.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CARBONDATA-210) Support loading compressed CSV file

2016-09-05 Thread Jacky Li (JIRA) 
Jacky Li created CARBONDATA-210:
---

 Summary: Support loading compressed CSV file
 Key: CARBONDATA-210
 URL: https://issues.apache.org/jira/browse/CARBONDATA-210
 Project: CarbonData
  Issue Type: Bug
Reporter: Jacky Li
 Fix For: 0.2.0-incubating


Support compressed CSV file with GZIP, LZO, LZ4



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)