date:20170709

[GitHub] carbondata issue #943: [CARBONDATA-1086]Added documentation for BATCH SORT S...

2017-07-09 Thread sgururajshetty

Github user sgururajshetty commented on the issue:

https://github.com/apache/carbondata/pull/943
  
@vandana7 there is a space missing after a . in a sentence.
Also under DML operation we will have the main section as sort_scope and 
not batch_sort. Then we will explain about the different types of SORT_SCOPE.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] carbondata issue #984: [CARBONDATA-1008] Make Caron table schema compatible ...

2017-07-09 Thread bhavya411

Github user bhavya411 commented on the issue:

https://github.com/apache/carbondata/pull/984
  
Sure Liang,  I will review it today.

Regards
Bhavya

On Sun, Jul 9, 2017 at 7:39 PM, å²çæµ·  wrote:

> @chenliang613  can you review this pr?
>
> â
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> ,
> or mute the thread
> 

> .
>



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] carbondata pull request #984: [CARBONDATA-1008] Make Caron table schema comp...

2017-07-09 Thread anubhav100

Github user anubhav100 commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/984#discussion_r126336311
  
--- Diff: integration/hive/hive-guide.md ---
@@ -41,16 +41,19 @@ mvn -DskipTests -Pspark-2.1 -Phadoop-2.7.2 clean package
 $HADOOP_HOME/bin/hadoop fs -put sample.csv /sample.csv
 ```
 
+
+Please set spark.carbon.hive.schema.compatibility.enable=true in 
spark-defaults.conf
 * Start Spark shell by running the following command in the Spark directory
 
+
 ```
-./bin/spark-shell --jars 
+./bin/spark-shell --jars 
--- End diff --

@cenyuhai carbon hive jar won't be required for running the spark shell i 
guess


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (CARBONDATA-1282) Query with large no of column gives codegeneration issue

2017-07-09 Thread Ashok Kumar (JIRA)


 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashok Kumar updated CARBONDATA-1282:

Description: When table is having around 2000 columns and query is being 
done on same table, carbon gives codegeneration issue stating generated code 
size exceeds 64Kb.   (was: When table is having around 2000 columns and query 
is being done on same table, carbon gives codegeneration issue stating 
generated code size exceeds 65Kb. )

> Query with large no of column gives codegeneration issue
> 
>
> Key: CARBONDATA-1282
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1282
> Project: CarbonData
>  Issue Type: Bug
>Reporter: Ashok Kumar
>Priority: Minor
>
> When table is having around 2000 columns and query is being done on same 
> table, carbon gives codegeneration issue stating generated code size exceeds 
> 64Kb. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (CARBONDATA-1282) Query with large no of column gives codegeneration issue

2017-07-09 Thread Ashok Kumar (JIRA)

Ashok Kumar created CARBONDATA-1282:
---

 Summary: Query with large no of column gives codegeneration issue
 Key: CARBONDATA-1282
 URL: https://issues.apache.org/jira/browse/CARBONDATA-1282
 Project: CarbonData
  Issue Type: Bug
Reporter: Ashok Kumar
Priority: Minor


When table is having around 2000 columns and query is being done on same table, 
carbon gives codegeneration issue stating generated code size exceeds 65Kb. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] carbondata issue #1152: [WIP] Secure Dictionary Server Implementation

2017-07-09 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1152
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2983/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] carbondata pull request #1152: [WIP] Secure Dictionary Server Implementation

2017-07-09 Thread sounakr

GitHub user sounakr opened a pull request:

https://github.com/apache/carbondata/pull/1152

[WIP] Secure Dictionary Server Implementation

Secure Dictionary Implementation Along with Non Secure.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sounakr/incubator-carbondata Secure_Dictionry

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/carbondata/pull/1152.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1152


commit d0aca3a94f07e3dc4d456e7e263dfe905dcafd8b
Author: sounak 
Date:   2017-07-06T15:18:54Z

Secure Dictionary Server Implementation

Secure Dictionary Implementation Along with Non Secure.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] carbondata issue #1152: [WIP] Secure Dictionary Server Implementation

2017-07-09 Thread asfgit

Github user asfgit commented on the issue:

https://github.com/apache/carbondata/pull/1152
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] carbondata issue #1152: [WIP] Secure Dictionary Server Implementation

2017-07-09 Thread asfgit

Github user asfgit commented on the issue:

https://github.com/apache/carbondata/pull/1152
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] carbondata issue #1146: [WIP] Change Query related RDD to use TableInfo

2017-07-09 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1146
  
Build Failed with Spark 1.6, Please check CI 
http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/393/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] carbondata issue #1146: [WIP] Change Query related RDD to use TableInfo

2017-07-09 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1146
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2982/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (CARBONDATA-1281) Disk hotspot found during data loading

2017-07-09 Thread xuchuanyin (JIRA)

xuchuanyin created CARBONDATA-1281:
--

 Summary: Disk hotspot found during data loading
 Key: CARBONDATA-1281
 URL: https://issues.apache.org/jira/browse/CARBONDATA-1281
 Project: CarbonData
  Issue Type: Improvement
  Components: core, data-load
Affects Versions: 1.1.0
Reporter: xuchuanyin


# Scenario

Currently we have do a massive data loading. The input data is about 71GB in 
CSV format，and have about 88million records. When using carbondata, we do not 
use any dictionary encoding. Our testing environment has three nodes and each 
of them have 11 disks as yarn executor directory. We submit the loading command 
through JDBCServer.The JDBCServer instance have three executors in total, one 
on each node respectively. The loading takes about 10minutes (+-3min vary from 
each time).

We have observed the nmon information during the loading and find：

1. lots of CPU waits in the first half of loading;

2. only one single disk has many writes and almost reaches its bottleneck (Avg. 
80M/s, Max. 150M/s on SAS Disk)

3. the other disks are quite idel

# Analyze

When do data loading, carbondata read and sort data locally(default scope) and 
write the temp files to local disk. In my case, there is only one executor in 
one node, so carbondata write all the temp file to one disk(container directory 
or yarn local directory), thus resulting into single disk hotspot.

# Modification

We should support multiple directory for writing temp files to avoid disk 
hotspot.

Ps: I have improve this in my environment and the result is pretty optimistic: 
the loading takes about 6minutes (10 minutes before improving).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] carbondata issue #1139: [CARBONDATA-1267] Add short_int case branch in delta...

2017-07-09 Thread xuchuanyin

Github user xuchuanyin commented on the issue:

https://github.com/apache/carbondata/pull/1139
  
The tests are passed.

Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] carbondata pull request #1105: [WIP] Implement range interval partition

2017-07-09 Thread lionelcao

Github user lionelcao commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/1105#discussion_r126328264
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/scan/partition/RangeIntervalPartitioner.java
 ---
@@ -0,0 +1,206 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.core.scan.partition;
+
+import java.io.Serializable;
+import java.text.SimpleDateFormat;
+import java.util.ArrayList;
+import java.util.Calendar;
+import java.util.List;
+
+import org.apache.carbondata.core.constants.CarbonCommonConstants;
+import org.apache.carbondata.core.metadata.datatype.DataType;
+import org.apache.carbondata.core.metadata.schema.PartitionInfo;
+import org.apache.carbondata.core.util.CarbonProperties;
+
+/**
+ * Range Interval Partitioner
+ */
+
+public class RangeIntervalPartitioner implements Partitioner {
+
+  private int numPartitions;
+  private RangeIntervalComparator comparator;
+  private List boundsList = new ArrayList<>();
+  private String intervalType;
+  private DataType partitionColumnDataType;
+
+  private SimpleDateFormat timestampFormatter = new 
SimpleDateFormat(CarbonProperties.getInstance()
+  .getProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT,
+  CarbonCommonConstants.CARBON_TIMESTAMP_DEFAULT_FORMAT));
+
+  private SimpleDateFormat dateFormatter = new 
SimpleDateFormat(CarbonProperties.getInstance()
+  .getProperty(CarbonCommonConstants.CARBON_DATE_FORMAT,
+  CarbonCommonConstants.CARBON_DATE_DEFAULT_FORMAT));
+
+  public RangeIntervalPartitioner(PartitionInfo partitionInfo) {
+List values = partitionInfo.getRangeIntervalInfo();
+partitionColumnDataType = 
partitionInfo.getColumnSchemaList().get(0).getDataType();
+numPartitions = values.size() - 1;
+// -1 is just for instead of null
+boundsList.add(-1L);
+for (int i = 0; i < numPartitions; i++) {
+  boundsList.add(PartitionUtil.getDataBasedOnDataType(values.get(i), 
partitionColumnDataType,
+  timestampFormatter, dateFormatter));
+}
+comparator = new RangeIntervalComparator();
+intervalType = values.get(numPartitions);
+  }
+
+  @Override public int numPartitions() {
+String numParStr = CarbonProperties.getInstance().getProperty(
+CarbonCommonConstants.MAX_PARTITION_FOR_RANGE_INTERVAL,
+CarbonCommonConstants.DEFAULT_MAX_PARTITION_FOR_RANGE_INTERVAL);
+return Integer.parseInt(numParStr);
+  }
+
+  @Override public int getPartition(Object key) {
+int partitionIndex = -1;
+Object lastBound = boundsList.get(boundsList.size() - 1);
+if (key == null) {
+  return 0;
+} else {
+  for (int i = 1; i <= numPartitions; i++) {
+if (comparator.compareTo(key, boundsList.get(i))) {
+  return i;
+}
+  }
+  switch (intervalType.toLowerCase()) {
+case "year":
+  partitionIndex = getDynamicPartitionForYear(key, lastBound);
+  break;
+case "month":
+  partitionIndex = getDynamicPartitionForMonth(key, lastBound);
+  break;
+case "week":
+  partitionIndex = getDynamicPartitionForWeek(key, lastBound);
+  break;
+case "day":
+  partitionIndex = getDynamicPartitionForDay(key, lastBound);
+  break;
+case "hour":
+  partitionIndex = getDynamicPartitionForHour(key, lastBound);
+  break;
+default:
+  partitionIndex = -1;
+  }
+  // if value's partitionId is great than max partition, return 0 for 
it.
+  if (partitionIndex >= numPartitions()) {
+return 0;
+  }
+  return partitionIndex;
+}
+  }
+
+  public int getDynamicPartitionForYear(Object key, Object lastBound) {
+int

[GitHub] carbondata pull request #1105: [WIP] Implement range interval partition

2017-07-09 Thread lionelcao

Github user lionelcao commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/1105#discussion_r126328243
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/scan/partition/RangeIntervalPartitioner.java
 ---
@@ -0,0 +1,206 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.core.scan.partition;
+
+import java.io.Serializable;
+import java.text.SimpleDateFormat;
+import java.util.ArrayList;
+import java.util.Calendar;
+import java.util.List;
+
+import org.apache.carbondata.core.constants.CarbonCommonConstants;
+import org.apache.carbondata.core.metadata.datatype.DataType;
+import org.apache.carbondata.core.metadata.schema.PartitionInfo;
+import org.apache.carbondata.core.util.CarbonProperties;
+
+/**
+ * Range Interval Partitioner
+ */
+
+public class RangeIntervalPartitioner implements Partitioner {
+
+  private int numPartitions;
+  private RangeIntervalComparator comparator;
+  private List boundsList = new ArrayList<>();
+  private String intervalType;
+  private DataType partitionColumnDataType;
+
+  private SimpleDateFormat timestampFormatter = new 
SimpleDateFormat(CarbonProperties.getInstance()
+  .getProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT,
+  CarbonCommonConstants.CARBON_TIMESTAMP_DEFAULT_FORMAT));
+
+  private SimpleDateFormat dateFormatter = new 
SimpleDateFormat(CarbonProperties.getInstance()
+  .getProperty(CarbonCommonConstants.CARBON_DATE_FORMAT,
+  CarbonCommonConstants.CARBON_DATE_DEFAULT_FORMAT));
+
+  public RangeIntervalPartitioner(PartitionInfo partitionInfo) {
+List values = partitionInfo.getRangeIntervalInfo();
+partitionColumnDataType = 
partitionInfo.getColumnSchemaList().get(0).getDataType();
+numPartitions = values.size() - 1;
+// -1 is just for instead of null
+boundsList.add(-1L);
+for (int i = 0; i < numPartitions; i++) {
+  boundsList.add(PartitionUtil.getDataBasedOnDataType(values.get(i), 
partitionColumnDataType,
+  timestampFormatter, dateFormatter));
+}
+comparator = new RangeIntervalComparator();
+intervalType = values.get(numPartitions);
+  }
+
+  @Override public int numPartitions() {
+String numParStr = CarbonProperties.getInstance().getProperty(
+CarbonCommonConstants.MAX_PARTITION_FOR_RANGE_INTERVAL,
+CarbonCommonConstants.DEFAULT_MAX_PARTITION_FOR_RANGE_INTERVAL);
+return Integer.parseInt(numParStr);
+  }
+
+  @Override public int getPartition(Object key) {
+int partitionIndex = -1;
+Object lastBound = boundsList.get(boundsList.size() - 1);
+if (key == null) {
+  return 0;
+} else {
+  for (int i = 1; i <= numPartitions; i++) {
+if (comparator.compareTo(key, boundsList.get(i))) {
+  return i;
+}
+  }
+  switch (intervalType.toLowerCase()) {
+case "year":
+  partitionIndex = getDynamicPartitionForYear(key, lastBound);
+  break;
+case "month":
+  partitionIndex = getDynamicPartitionForMonth(key, lastBound);
+  break;
+case "week":
+  partitionIndex = getDynamicPartitionForWeek(key, lastBound);
+  break;
+case "day":
+  partitionIndex = getDynamicPartitionForDay(key, lastBound);
+  break;
+case "hour":
+  partitionIndex = getDynamicPartitionForHour(key, lastBound);
+  break;
+default:
+  partitionIndex = -1;
+  }
+  // if value's partitionId is great than max partition, return 0 for 
it.
+  if (partitionIndex >= numPartitions()) {
+return 0;
+  }
+  return partitionIndex;
+}
+  }
+
+  public int getDynamicPartitionForYear(Object key, Object lastBound) {
+int

[GitHub] carbondata pull request #1105: [WIP] Implement range interval partition

2017-07-09 Thread lionelcao

Github user lionelcao commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/1105#discussion_r126328220
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/scan/partition/RangeIntervalPartitioner.java
 ---
@@ -0,0 +1,206 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.core.scan.partition;
+
+import java.io.Serializable;
+import java.text.SimpleDateFormat;
+import java.util.ArrayList;
+import java.util.Calendar;
+import java.util.List;
+
+import org.apache.carbondata.core.constants.CarbonCommonConstants;
+import org.apache.carbondata.core.metadata.datatype.DataType;
+import org.apache.carbondata.core.metadata.schema.PartitionInfo;
+import org.apache.carbondata.core.util.CarbonProperties;
+
+/**
+ * Range Interval Partitioner
+ */
+
+public class RangeIntervalPartitioner implements Partitioner {
+
+  private int numPartitions;
+  private RangeIntervalComparator comparator;
+  private List boundsList = new ArrayList<>();
+  private String intervalType;
+  private DataType partitionColumnDataType;
+
+  private SimpleDateFormat timestampFormatter = new 
SimpleDateFormat(CarbonProperties.getInstance()
+  .getProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT,
+  CarbonCommonConstants.CARBON_TIMESTAMP_DEFAULT_FORMAT));
+
+  private SimpleDateFormat dateFormatter = new 
SimpleDateFormat(CarbonProperties.getInstance()
+  .getProperty(CarbonCommonConstants.CARBON_DATE_FORMAT,
+  CarbonCommonConstants.CARBON_DATE_DEFAULT_FORMAT));
+
+  public RangeIntervalPartitioner(PartitionInfo partitionInfo) {
+List values = partitionInfo.getRangeIntervalInfo();
+partitionColumnDataType = 
partitionInfo.getColumnSchemaList().get(0).getDataType();
+numPartitions = values.size() - 1;
+// -1 is just for instead of null
+boundsList.add(-1L);
+for (int i = 0; i < numPartitions; i++) {
+  boundsList.add(PartitionUtil.getDataBasedOnDataType(values.get(i), 
partitionColumnDataType,
+  timestampFormatter, dateFormatter));
+}
+comparator = new RangeIntervalComparator();
+intervalType = values.get(numPartitions);
+  }
+
+  @Override public int numPartitions() {
+String numParStr = CarbonProperties.getInstance().getProperty(
+CarbonCommonConstants.MAX_PARTITION_FOR_RANGE_INTERVAL,
+CarbonCommonConstants.DEFAULT_MAX_PARTITION_FOR_RANGE_INTERVAL);
+return Integer.parseInt(numParStr);
+  }
+
+  @Override public int getPartition(Object key) {
+int partitionIndex = -1;
+Object lastBound = boundsList.get(boundsList.size() - 1);
+if (key == null) {
+  return 0;
+} else {
+  for (int i = 1; i <= numPartitions; i++) {
+if (comparator.compareTo(key, boundsList.get(i))) {
+  return i;
+}
+  }
+  switch (intervalType.toLowerCase()) {
+case "year":
+  partitionIndex = getDynamicPartitionForYear(key, lastBound);
+  break;
+case "month":
+  partitionIndex = getDynamicPartitionForMonth(key, lastBound);
+  break;
+case "week":
+  partitionIndex = getDynamicPartitionForWeek(key, lastBound);
+  break;
+case "day":
+  partitionIndex = getDynamicPartitionForDay(key, lastBound);
+  break;
+case "hour":
+  partitionIndex = getDynamicPartitionForHour(key, lastBound);
+  break;
+default:
+  partitionIndex = -1;
+  }
+  // if value's partitionId is great than max partition, return 0 for 
it.
+  if (partitionIndex >= numPartitions()) {
+return 0;
+  }
+  return partitionIndex;
+}
+  }
+
+  public int getDynamicPartitionForYear(Object key, Object lastBound) {
+int

[GitHub] carbondata pull request #1105: [WIP] Implement range interval partition

2017-07-09 Thread lionelcao

Github user lionelcao commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/1105#discussion_r126328139
  
--- Diff: 
integration/spark-common/src/main/scala/org/apache/carbondata/spark/util/CommonUtil.scala
 ---
@@ -188,7 +189,7 @@ object CommonUtil {
   rangeInfo.get.split(",").map(_.trim).foreach(
 isValid &= validateTypeConvert(partitionerFields(0), _))
 }
-case "RANGE_INTERVAL" => isValid = false
+case "RANGE_INTERVAL" => isValid = rangeIntervalInfo.isDefined
--- End diff --

Is there any validate function for range interval info?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] carbondata pull request #1105: [WIP] Implement range interval partition

2017-07-09 Thread lionelcao

Github user lionelcao commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/1105#discussion_r126328252
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/scan/partition/RangeIntervalPartitioner.java
 ---
@@ -0,0 +1,206 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.core.scan.partition;
+
+import java.io.Serializable;
+import java.text.SimpleDateFormat;
+import java.util.ArrayList;
+import java.util.Calendar;
+import java.util.List;
+
+import org.apache.carbondata.core.constants.CarbonCommonConstants;
+import org.apache.carbondata.core.metadata.datatype.DataType;
+import org.apache.carbondata.core.metadata.schema.PartitionInfo;
+import org.apache.carbondata.core.util.CarbonProperties;
+
+/**
+ * Range Interval Partitioner
+ */
+
+public class RangeIntervalPartitioner implements Partitioner {
+
+  private int numPartitions;
+  private RangeIntervalComparator comparator;
+  private List boundsList = new ArrayList<>();
+  private String intervalType;
+  private DataType partitionColumnDataType;
+
+  private SimpleDateFormat timestampFormatter = new 
SimpleDateFormat(CarbonProperties.getInstance()
+  .getProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT,
+  CarbonCommonConstants.CARBON_TIMESTAMP_DEFAULT_FORMAT));
+
+  private SimpleDateFormat dateFormatter = new 
SimpleDateFormat(CarbonProperties.getInstance()
+  .getProperty(CarbonCommonConstants.CARBON_DATE_FORMAT,
+  CarbonCommonConstants.CARBON_DATE_DEFAULT_FORMAT));
+
+  public RangeIntervalPartitioner(PartitionInfo partitionInfo) {
+List values = partitionInfo.getRangeIntervalInfo();
+partitionColumnDataType = 
partitionInfo.getColumnSchemaList().get(0).getDataType();
+numPartitions = values.size() - 1;
+// -1 is just for instead of null
+boundsList.add(-1L);
+for (int i = 0; i < numPartitions; i++) {
+  boundsList.add(PartitionUtil.getDataBasedOnDataType(values.get(i), 
partitionColumnDataType,
+  timestampFormatter, dateFormatter));
+}
+comparator = new RangeIntervalComparator();
+intervalType = values.get(numPartitions);
+  }
+
+  @Override public int numPartitions() {
+String numParStr = CarbonProperties.getInstance().getProperty(
+CarbonCommonConstants.MAX_PARTITION_FOR_RANGE_INTERVAL,
+CarbonCommonConstants.DEFAULT_MAX_PARTITION_FOR_RANGE_INTERVAL);
+return Integer.parseInt(numParStr);
+  }
+
+  @Override public int getPartition(Object key) {
+int partitionIndex = -1;
+Object lastBound = boundsList.get(boundsList.size() - 1);
+if (key == null) {
+  return 0;
+} else {
+  for (int i = 1; i <= numPartitions; i++) {
+if (comparator.compareTo(key, boundsList.get(i))) {
+  return i;
+}
+  }
+  switch (intervalType.toLowerCase()) {
+case "year":
+  partitionIndex = getDynamicPartitionForYear(key, lastBound);
+  break;
+case "month":
+  partitionIndex = getDynamicPartitionForMonth(key, lastBound);
+  break;
+case "week":
+  partitionIndex = getDynamicPartitionForWeek(key, lastBound);
+  break;
+case "day":
+  partitionIndex = getDynamicPartitionForDay(key, lastBound);
+  break;
+case "hour":
+  partitionIndex = getDynamicPartitionForHour(key, lastBound);
+  break;
+default:
+  partitionIndex = -1;
+  }
+  // if value's partitionId is great than max partition, return 0 for 
it.
+  if (partitionIndex >= numPartitions()) {
+return 0;
+  }
+  return partitionIndex;
+}
+  }
+
+  public int getDynamicPartitionForYear(Object key, Object lastBound) {
+int

[GitHub] carbondata pull request #1105: [WIP] Implement range interval partition

2017-07-09 Thread lionelcao

Github user lionelcao commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/1105#discussion_r126328234
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/scan/partition/RangeIntervalPartitioner.java
 ---
@@ -0,0 +1,206 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.core.scan.partition;
+
+import java.io.Serializable;
+import java.text.SimpleDateFormat;
+import java.util.ArrayList;
+import java.util.Calendar;
+import java.util.List;
+
+import org.apache.carbondata.core.constants.CarbonCommonConstants;
+import org.apache.carbondata.core.metadata.datatype.DataType;
+import org.apache.carbondata.core.metadata.schema.PartitionInfo;
+import org.apache.carbondata.core.util.CarbonProperties;
+
+/**
+ * Range Interval Partitioner
+ */
+
+public class RangeIntervalPartitioner implements Partitioner {
+
+  private int numPartitions;
+  private RangeIntervalComparator comparator;
+  private List boundsList = new ArrayList<>();
+  private String intervalType;
+  private DataType partitionColumnDataType;
+
+  private SimpleDateFormat timestampFormatter = new 
SimpleDateFormat(CarbonProperties.getInstance()
+  .getProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT,
+  CarbonCommonConstants.CARBON_TIMESTAMP_DEFAULT_FORMAT));
+
+  private SimpleDateFormat dateFormatter = new 
SimpleDateFormat(CarbonProperties.getInstance()
+  .getProperty(CarbonCommonConstants.CARBON_DATE_FORMAT,
+  CarbonCommonConstants.CARBON_DATE_DEFAULT_FORMAT));
+
+  public RangeIntervalPartitioner(PartitionInfo partitionInfo) {
+List values = partitionInfo.getRangeIntervalInfo();
+partitionColumnDataType = 
partitionInfo.getColumnSchemaList().get(0).getDataType();
+numPartitions = values.size() - 1;
+// -1 is just for instead of null
+boundsList.add(-1L);
+for (int i = 0; i < numPartitions; i++) {
+  boundsList.add(PartitionUtil.getDataBasedOnDataType(values.get(i), 
partitionColumnDataType,
+  timestampFormatter, dateFormatter));
+}
+comparator = new RangeIntervalComparator();
+intervalType = values.get(numPartitions);
+  }
+
+  @Override public int numPartitions() {
+String numParStr = CarbonProperties.getInstance().getProperty(
+CarbonCommonConstants.MAX_PARTITION_FOR_RANGE_INTERVAL,
+CarbonCommonConstants.DEFAULT_MAX_PARTITION_FOR_RANGE_INTERVAL);
+return Integer.parseInt(numParStr);
+  }
+
+  @Override public int getPartition(Object key) {
+int partitionIndex = -1;
+Object lastBound = boundsList.get(boundsList.size() - 1);
+if (key == null) {
+  return 0;
+} else {
+  for (int i = 1; i <= numPartitions; i++) {
+if (comparator.compareTo(key, boundsList.get(i))) {
+  return i;
+}
+  }
+  switch (intervalType.toLowerCase()) {
+case "year":
+  partitionIndex = getDynamicPartitionForYear(key, lastBound);
+  break;
+case "month":
+  partitionIndex = getDynamicPartitionForMonth(key, lastBound);
+  break;
+case "week":
+  partitionIndex = getDynamicPartitionForWeek(key, lastBound);
+  break;
+case "day":
+  partitionIndex = getDynamicPartitionForDay(key, lastBound);
+  break;
+case "hour":
+  partitionIndex = getDynamicPartitionForHour(key, lastBound);
+  break;
+default:
+  partitionIndex = -1;
+  }
+  // if value's partitionId is great than max partition, return 0 for 
it.
+  if (partitionIndex >= numPartitions()) {
+return 0;
+  }
+  return partitionIndex;
+}
+  }
+
+  public int getDynamicPartitionForYear(Object key, Object lastBound) {
+int

[GitHub] carbondata pull request #1105: [WIP] Implement range interval partition

2017-07-09 Thread lionelcao

Github user lionelcao commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/1105#discussion_r126328284
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/scan/filter/partition/PartitionFilterUtil.java
 ---
@@ -264,4 +265,81 @@ public static BitSet 
getPartitionMapForRangeFilter(PartitionInfo partitionInfo,
 return partitionMap;
   }
 
+  /**
+   * get partition map of range interval filter on range interval 
partition table
+   * @param partitionInfo
+   * @param partitioner
+   * @param filterValue
+   * @param isGreaterThan
+   * @param isEqualTo
+   * @return
+   */
+  public static BitSet getPartitionMapForRangeIntervalFilter(PartitionInfo 
partitionInfo,
+  RangeIntervalPartitioner partitioner, Object filterValue,
+  boolean isGreaterThan, boolean isEqualTo,
+  DateFormat timestampFormatter, DateFormat dateFormatter) {
+
+List values = partitionInfo.getRangeInfo();
+DataType partitionColumnDataType = 
partitionInfo.getColumnSchemaList().get(0).getDataType();
+
+Comparator comparator =
+
PartitionFilterUtil.getComparatorByDataType(partitionColumnDataType);
+
+BitSet partitionMap = PartitionUtil
+.generateBitSetBySize(partitioner.numPartitions(), false);
+
+int numPartitions = values.size() - 1;
+int result = 0;
+// the partition index of filter value
+int partitionIndex = 0;
+// find the partition of filter value
+for (; partitionIndex < numPartitions; partitionIndex++) {
+  result = comparator.compare(filterValue, 
PartitionUtil.getDataBasedOnDataType(
+  values.get(partitionIndex), partitionColumnDataType, 
timestampFormatter, dateFormatter));
+  if (result <= 0) {
+break;
+  }
+}
+if (partitionIndex == numPartitions) {
+  // filter value is not in given partition, should get partitionId 
dynamically
+  int partitionId = partitioner.getPartition(filterValue);
--- End diff --

this should be partition index


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] carbondata pull request #1105: [WIP] Implement range interval partition

2017-07-09 Thread lionelcao

Github user lionelcao commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/1105#discussion_r126328012
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/scan/partition/RangeIntervalPartitioner.java
 ---
@@ -0,0 +1,206 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.core.scan.partition;
+
+import java.io.Serializable;
+import java.text.SimpleDateFormat;
+import java.util.ArrayList;
+import java.util.Calendar;
+import java.util.List;
+
+import org.apache.carbondata.core.constants.CarbonCommonConstants;
+import org.apache.carbondata.core.metadata.datatype.DataType;
+import org.apache.carbondata.core.metadata.schema.PartitionInfo;
+import org.apache.carbondata.core.util.CarbonProperties;
+
+/**
+ * Range Interval Partitioner
+ */
+
+public class RangeIntervalPartitioner implements Partitioner {
+
+  private int numPartitions;
+  private RangeIntervalComparator comparator;
+  private List boundsList = new ArrayList<>();
+  private String intervalType;
+  private DataType partitionColumnDataType;
+
+  private SimpleDateFormat timestampFormatter = new 
SimpleDateFormat(CarbonProperties.getInstance()
+  .getProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT,
+  CarbonCommonConstants.CARBON_TIMESTAMP_DEFAULT_FORMAT));
+
+  private SimpleDateFormat dateFormatter = new 
SimpleDateFormat(CarbonProperties.getInstance()
+  .getProperty(CarbonCommonConstants.CARBON_DATE_FORMAT,
+  CarbonCommonConstants.CARBON_DATE_DEFAULT_FORMAT));
+
+  public RangeIntervalPartitioner(PartitionInfo partitionInfo) {
+List values = partitionInfo.getRangeIntervalInfo();
+partitionColumnDataType = 
partitionInfo.getColumnSchemaList().get(0).getDataType();
+numPartitions = values.size() - 1;
+// -1 is just for instead of null
+boundsList.add(-1L);
+for (int i = 0; i < numPartitions; i++) {
+  boundsList.add(PartitionUtil.getDataBasedOnDataType(values.get(i), 
partitionColumnDataType,
+  timestampFormatter, dateFormatter));
+}
+comparator = new RangeIntervalComparator();
+intervalType = values.get(numPartitions);
+  }
+
+  @Override public int numPartitions() {
+String numParStr = CarbonProperties.getInstance().getProperty(
+CarbonCommonConstants.MAX_PARTITION_FOR_RANGE_INTERVAL,
+CarbonCommonConstants.DEFAULT_MAX_PARTITION_FOR_RANGE_INTERVAL);
+return Integer.parseInt(numParStr);
+  }
+
+  @Override public int getPartition(Object key) {
+int partitionIndex = -1;
+Object lastBound = boundsList.get(boundsList.size() - 1);
+if (key == null) {
+  return 0;
+} else {
+  for (int i = 1; i <= numPartitions; i++) {
+if (comparator.compareTo(key, boundsList.get(i))) {
+  return i;
+}
+  }
+  switch (intervalType.toLowerCase()) {
+case "year":
+  partitionIndex = getDynamicPartitionForYear(key, lastBound);
+  break;
+case "month":
+  partitionIndex = getDynamicPartitionForMonth(key, lastBound);
+  break;
+case "week":
+  partitionIndex = getDynamicPartitionForWeek(key, lastBound);
+  break;
+case "day":
+  partitionIndex = getDynamicPartitionForDay(key, lastBound);
+  break;
+case "hour":
+  partitionIndex = getDynamicPartitionForHour(key, lastBound);
+  break;
+default:
+  partitionIndex = -1;
+  }
+  // if value's partitionId is great than max partition, return 0 for 
it.
+  if (partitionIndex >= numPartitions()) {
+return 0;
+  }
+  return partitionIndex;
+}
+  }
+
+  public int getDynamicPartitionForYear(Object key, Object lastBound) {
+int

[GitHub] carbondata pull request #1105: [WIP] Implement range interval partition

2017-07-09 Thread lionelcao

Github user lionelcao commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/1105#discussion_r126328001
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/scan/partition/RangeIntervalPartitioner.java
 ---
@@ -0,0 +1,206 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.core.scan.partition;
+
+import java.io.Serializable;
+import java.text.SimpleDateFormat;
+import java.util.ArrayList;
+import java.util.Calendar;
+import java.util.List;
+
+import org.apache.carbondata.core.constants.CarbonCommonConstants;
+import org.apache.carbondata.core.metadata.datatype.DataType;
+import org.apache.carbondata.core.metadata.schema.PartitionInfo;
+import org.apache.carbondata.core.util.CarbonProperties;
+
+/**
+ * Range Interval Partitioner
+ */
+
+public class RangeIntervalPartitioner implements Partitioner {
+
+  private int numPartitions;
+  private RangeIntervalComparator comparator;
+  private List boundsList = new ArrayList<>();
+  private String intervalType;
+  private DataType partitionColumnDataType;
+
+  private SimpleDateFormat timestampFormatter = new 
SimpleDateFormat(CarbonProperties.getInstance()
+  .getProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT,
+  CarbonCommonConstants.CARBON_TIMESTAMP_DEFAULT_FORMAT));
+
+  private SimpleDateFormat dateFormatter = new 
SimpleDateFormat(CarbonProperties.getInstance()
+  .getProperty(CarbonCommonConstants.CARBON_DATE_FORMAT,
+  CarbonCommonConstants.CARBON_DATE_DEFAULT_FORMAT));
+
+  public RangeIntervalPartitioner(PartitionInfo partitionInfo) {
+List values = partitionInfo.getRangeIntervalInfo();
+partitionColumnDataType = 
partitionInfo.getColumnSchemaList().get(0).getDataType();
+numPartitions = values.size() - 1;
+// -1 is just for instead of null
+boundsList.add(-1L);
+for (int i = 0; i < numPartitions; i++) {
+  boundsList.add(PartitionUtil.getDataBasedOnDataType(values.get(i), 
partitionColumnDataType,
+  timestampFormatter, dateFormatter));
+}
+comparator = new RangeIntervalComparator();
+intervalType = values.get(numPartitions);
+  }
+
+  @Override public int numPartitions() {
+String numParStr = CarbonProperties.getInstance().getProperty(
+CarbonCommonConstants.MAX_PARTITION_FOR_RANGE_INTERVAL,
+CarbonCommonConstants.DEFAULT_MAX_PARTITION_FOR_RANGE_INTERVAL);
+return Integer.parseInt(numParStr);
+  }
+
+  @Override public int getPartition(Object key) {
+int partitionIndex = -1;
+Object lastBound = boundsList.get(boundsList.size() - 1);
+if (key == null) {
+  return 0;
+} else {
+  for (int i = 1; i <= numPartitions; i++) {
+if (comparator.compareTo(key, boundsList.get(i))) {
+  return i;
+}
+  }
+  switch (intervalType.toLowerCase()) {
+case "year":
+  partitionIndex = getDynamicPartitionForYear(key, lastBound);
+  break;
+case "month":
+  partitionIndex = getDynamicPartitionForMonth(key, lastBound);
+  break;
+case "week":
+  partitionIndex = getDynamicPartitionForWeek(key, lastBound);
+  break;
+case "day":
+  partitionIndex = getDynamicPartitionForDay(key, lastBound);
+  break;
+case "hour":
+  partitionIndex = getDynamicPartitionForHour(key, lastBound);
+  break;
+default:
+  partitionIndex = -1;
+  }
+  // if value's partitionId is great than max partition, return 0 for 
it.
+  if (partitionIndex >= numPartitions()) {
+return 0;
+  }
+  return partitionIndex;
+}
+  }
+
+  public int getDynamicPartitionForYear(Object key, Object lastBound) {
+int

[GitHub] carbondata pull request #1105: [WIP] Implement range interval partition

2017-07-09 Thread lionelcao

Github user lionelcao commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/1105#discussion_r126327993
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/scan/partition/RangeIntervalPartitioner.java
 ---
@@ -0,0 +1,206 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.core.scan.partition;
+
+import java.io.Serializable;
+import java.text.SimpleDateFormat;
+import java.util.ArrayList;
+import java.util.Calendar;
+import java.util.List;
+
+import org.apache.carbondata.core.constants.CarbonCommonConstants;
+import org.apache.carbondata.core.metadata.datatype.DataType;
+import org.apache.carbondata.core.metadata.schema.PartitionInfo;
+import org.apache.carbondata.core.util.CarbonProperties;
+
+/**
+ * Range Interval Partitioner
+ */
+
+public class RangeIntervalPartitioner implements Partitioner {
+
+  private int numPartitions;
+  private RangeIntervalComparator comparator;
+  private List boundsList = new ArrayList<>();
+  private String intervalType;
+  private DataType partitionColumnDataType;
+
+  private SimpleDateFormat timestampFormatter = new 
SimpleDateFormat(CarbonProperties.getInstance()
+  .getProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT,
+  CarbonCommonConstants.CARBON_TIMESTAMP_DEFAULT_FORMAT));
+
+  private SimpleDateFormat dateFormatter = new 
SimpleDateFormat(CarbonProperties.getInstance()
+  .getProperty(CarbonCommonConstants.CARBON_DATE_FORMAT,
+  CarbonCommonConstants.CARBON_DATE_DEFAULT_FORMAT));
+
+  public RangeIntervalPartitioner(PartitionInfo partitionInfo) {
+List values = partitionInfo.getRangeIntervalInfo();
+partitionColumnDataType = 
partitionInfo.getColumnSchemaList().get(0).getDataType();
+numPartitions = values.size() - 1;
+// -1 is just for instead of null
+boundsList.add(-1L);
+for (int i = 0; i < numPartitions; i++) {
+  boundsList.add(PartitionUtil.getDataBasedOnDataType(values.get(i), 
partitionColumnDataType,
+  timestampFormatter, dateFormatter));
+}
+comparator = new RangeIntervalComparator();
+intervalType = values.get(numPartitions);
+  }
+
+  @Override public int numPartitions() {
+String numParStr = CarbonProperties.getInstance().getProperty(
+CarbonCommonConstants.MAX_PARTITION_FOR_RANGE_INTERVAL,
+CarbonCommonConstants.DEFAULT_MAX_PARTITION_FOR_RANGE_INTERVAL);
+return Integer.parseInt(numParStr);
+  }
+
+  @Override public int getPartition(Object key) {
+int partitionIndex = -1;
+Object lastBound = boundsList.get(boundsList.size() - 1);
+if (key == null) {
+  return 0;
+} else {
+  for (int i = 1; i <= numPartitions; i++) {
+if (comparator.compareTo(key, boundsList.get(i))) {
+  return i;
+}
+  }
+  switch (intervalType.toLowerCase()) {
+case "year":
+  partitionIndex = getDynamicPartitionForYear(key, lastBound);
+  break;
+case "month":
+  partitionIndex = getDynamicPartitionForMonth(key, lastBound);
+  break;
+case "week":
+  partitionIndex = getDynamicPartitionForWeek(key, lastBound);
+  break;
+case "day":
+  partitionIndex = getDynamicPartitionForDay(key, lastBound);
+  break;
+case "hour":
+  partitionIndex = getDynamicPartitionForHour(key, lastBound);
+  break;
+default:
+  partitionIndex = -1;
+  }
+  // if value's partitionId is great than max partition, return 0 for 
it.
+  if (partitionIndex >= numPartitions()) {
+return 0;
+  }
+  return partitionIndex;
+}
+  }
+
+  public int getDynamicPartitionForYear(Object key, Object lastBound) {
+int

[GitHub] carbondata pull request #1105: [WIP] Implement range interval partition

2017-07-09 Thread lionelcao

Github user lionelcao commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/1105#discussion_r126327975
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/scan/partition/RangeIntervalPartitioner.java
 ---
@@ -0,0 +1,206 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.core.scan.partition;
+
+import java.io.Serializable;
+import java.text.SimpleDateFormat;
+import java.util.ArrayList;
+import java.util.Calendar;
+import java.util.List;
+
+import org.apache.carbondata.core.constants.CarbonCommonConstants;
+import org.apache.carbondata.core.metadata.datatype.DataType;
+import org.apache.carbondata.core.metadata.schema.PartitionInfo;
+import org.apache.carbondata.core.util.CarbonProperties;
+
+/**
+ * Range Interval Partitioner
+ */
+
+public class RangeIntervalPartitioner implements Partitioner {
+
+  private int numPartitions;
+  private RangeIntervalComparator comparator;
+  private List boundsList = new ArrayList<>();
+  private String intervalType;
+  private DataType partitionColumnDataType;
+
+  private SimpleDateFormat timestampFormatter = new 
SimpleDateFormat(CarbonProperties.getInstance()
+  .getProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT,
+  CarbonCommonConstants.CARBON_TIMESTAMP_DEFAULT_FORMAT));
+
+  private SimpleDateFormat dateFormatter = new 
SimpleDateFormat(CarbonProperties.getInstance()
+  .getProperty(CarbonCommonConstants.CARBON_DATE_FORMAT,
+  CarbonCommonConstants.CARBON_DATE_DEFAULT_FORMAT));
+
+  public RangeIntervalPartitioner(PartitionInfo partitionInfo) {
+List values = partitionInfo.getRangeIntervalInfo();
+partitionColumnDataType = 
partitionInfo.getColumnSchemaList().get(0).getDataType();
+numPartitions = values.size() - 1;
+// -1 is just for instead of null
+boundsList.add(-1L);
+for (int i = 0; i < numPartitions; i++) {
+  boundsList.add(PartitionUtil.getDataBasedOnDataType(values.get(i), 
partitionColumnDataType,
+  timestampFormatter, dateFormatter));
+}
+comparator = new RangeIntervalComparator();
+intervalType = values.get(numPartitions);
+  }
+
+  @Override public int numPartitions() {
+String numParStr = CarbonProperties.getInstance().getProperty(
+CarbonCommonConstants.MAX_PARTITION_FOR_RANGE_INTERVAL,
+CarbonCommonConstants.DEFAULT_MAX_PARTITION_FOR_RANGE_INTERVAL);
+return Integer.parseInt(numParStr);
+  }
+
+  @Override public int getPartition(Object key) {
+int partitionIndex = -1;
+Object lastBound = boundsList.get(boundsList.size() - 1);
+if (key == null) {
+  return 0;
+} else {
+  for (int i = 1; i <= numPartitions; i++) {
+if (comparator.compareTo(key, boundsList.get(i))) {
+  return i;
+}
+  }
+  switch (intervalType.toLowerCase()) {
+case "year":
+  partitionIndex = getDynamicPartitionForYear(key, lastBound);
+  break;
+case "month":
+  partitionIndex = getDynamicPartitionForMonth(key, lastBound);
+  break;
+case "week":
+  partitionIndex = getDynamicPartitionForWeek(key, lastBound);
+  break;
+case "day":
+  partitionIndex = getDynamicPartitionForDay(key, lastBound);
+  break;
+case "hour":
+  partitionIndex = getDynamicPartitionForHour(key, lastBound);
+  break;
+default:
+  partitionIndex = -1;
+  }
+  // if value's partitionId is great than max partition, return 0 for 
it.
+  if (partitionIndex >= numPartitions()) {
+return 0;
+  }
+  return partitionIndex;
+}
+  }
+
+  public int getDynamicPartitionForYear(Object key, Object lastBound) {
+int

[GitHub] carbondata pull request #1105: [WIP] Implement range interval partition

2017-07-09 Thread lionelcao

Github user lionelcao commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/1105#discussion_r126327928
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/scan/partition/RangeIntervalPartitioner.java
 ---
@@ -0,0 +1,206 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.core.scan.partition;
+
+import java.io.Serializable;
+import java.text.SimpleDateFormat;
+import java.util.ArrayList;
+import java.util.Calendar;
+import java.util.List;
+
+import org.apache.carbondata.core.constants.CarbonCommonConstants;
+import org.apache.carbondata.core.metadata.datatype.DataType;
+import org.apache.carbondata.core.metadata.schema.PartitionInfo;
+import org.apache.carbondata.core.util.CarbonProperties;
+
+/**
+ * Range Interval Partitioner
+ */
+
+public class RangeIntervalPartitioner implements Partitioner {
+
+  private int numPartitions;
+  private RangeIntervalComparator comparator;
+  private List boundsList = new ArrayList<>();
+  private String intervalType;
+  private DataType partitionColumnDataType;
+
+  private SimpleDateFormat timestampFormatter = new 
SimpleDateFormat(CarbonProperties.getInstance()
+  .getProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT,
+  CarbonCommonConstants.CARBON_TIMESTAMP_DEFAULT_FORMAT));
+
+  private SimpleDateFormat dateFormatter = new 
SimpleDateFormat(CarbonProperties.getInstance()
+  .getProperty(CarbonCommonConstants.CARBON_DATE_FORMAT,
+  CarbonCommonConstants.CARBON_DATE_DEFAULT_FORMAT));
+
+  public RangeIntervalPartitioner(PartitionInfo partitionInfo) {
+List values = partitionInfo.getRangeIntervalInfo();
+partitionColumnDataType = 
partitionInfo.getColumnSchemaList().get(0).getDataType();
+numPartitions = values.size() - 1;
+// -1 is just for instead of null
+boundsList.add(-1L);
+for (int i = 0; i < numPartitions; i++) {
+  boundsList.add(PartitionUtil.getDataBasedOnDataType(values.get(i), 
partitionColumnDataType,
+  timestampFormatter, dateFormatter));
+}
+comparator = new RangeIntervalComparator();
+intervalType = values.get(numPartitions);
+  }
+
+  @Override public int numPartitions() {
+String numParStr = CarbonProperties.getInstance().getProperty(
+CarbonCommonConstants.MAX_PARTITION_FOR_RANGE_INTERVAL,
+CarbonCommonConstants.DEFAULT_MAX_PARTITION_FOR_RANGE_INTERVAL);
+return Integer.parseInt(numParStr);
+  }
+
+  @Override public int getPartition(Object key) {
+int partitionIndex = -1;
+Object lastBound = boundsList.get(boundsList.size() - 1);
+if (key == null) {
+  return 0;
+} else {
+  for (int i = 1; i <= numPartitions; i++) {
+if (comparator.compareTo(key, boundsList.get(i))) {
+  return i;
+}
+  }
+  switch (intervalType.toLowerCase()) {
+case "year":
+  partitionIndex = getDynamicPartitionForYear(key, lastBound);
+  break;
+case "month":
+  partitionIndex = getDynamicPartitionForMonth(key, lastBound);
+  break;
+case "week":
+  partitionIndex = getDynamicPartitionForWeek(key, lastBound);
+  break;
+case "day":
+  partitionIndex = getDynamicPartitionForDay(key, lastBound);
+  break;
+case "hour":
+  partitionIndex = getDynamicPartitionForHour(key, lastBound);
+  break;
+default:
+  partitionIndex = -1;
+  }
+  // if value's partitionId is great than max partition, return 0 for 
it.
+  if (partitionIndex >= numPartitions()) {
+return 0;
+  }
+  return partitionIndex;
+}
+  }
+
+  public int getDynamicPartitionForYear(Object key, Object lastBound) {
+int

[GitHub] carbondata pull request #1105: [WIP] Implement range interval partition

2017-07-09 Thread lionelcao

Github user lionelcao commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/1105#discussion_r126327873
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/scan/filter/partition/PartitionFilterUtil.java
 ---
@@ -264,4 +265,81 @@ public static BitSet 
getPartitionMapForRangeFilter(PartitionInfo partitionInfo,
 return partitionMap;
   }
 
+  /**
+   * get partition map of range interval filter on range interval 
partition table
+   * @param partitionInfo
+   * @param partitioner
+   * @param filterValue
+   * @param isGreaterThan
+   * @param isEqualTo
+   * @return
+   */
+  public static BitSet getPartitionMapForRangeIntervalFilter(PartitionInfo 
partitionInfo,
+  RangeIntervalPartitioner partitioner, Object filterValue,
+  boolean isGreaterThan, boolean isEqualTo,
+  DateFormat timestampFormatter, DateFormat dateFormatter) {
+
+List values = partitionInfo.getRangeInfo();
+DataType partitionColumnDataType = 
partitionInfo.getColumnSchemaList().get(0).getDataType();
+
+Comparator comparator =
+
PartitionFilterUtil.getComparatorByDataType(partitionColumnDataType);
+
+BitSet partitionMap = PartitionUtil
+.generateBitSetBySize(partitioner.numPartitions(), false);
+
+int numPartitions = values.size() - 1;
+int result = 0;
+// the partition index of filter value
+int partitionIndex = 0;
+// find the partition of filter value
+for (; partitionIndex < numPartitions; partitionIndex++) {
+  result = comparator.compare(filterValue, 
PartitionUtil.getDataBasedOnDataType(
+  values.get(partitionIndex), partitionColumnDataType, 
timestampFormatter, dateFormatter));
+  if (result <= 0) {
+break;
+  }
+}
+if (partitionIndex == numPartitions) {
+  // filter value is not in given partition, should get partitionId 
dynamically
+  int partitionId = partitioner.getPartition(filterValue);
+  partitionMap.set(0);
+  if (isGreaterThan) {
+// GreaterThan(>), GreaterThanEqualTo(>=)
+partitionMap.set(partitionId, partitioner.numPartitions());
--- End diff --

this should be partition index


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] carbondata pull request #1105: [WIP] Implement range interval partition

2017-07-09 Thread lionelcao

Github user lionelcao commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/1105#discussion_r126327814
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/scan/filter/partition/PartitionFilterUtil.java
 ---
@@ -264,4 +265,81 @@ public static BitSet 
getPartitionMapForRangeFilter(PartitionInfo partitionInfo,
 return partitionMap;
   }
 
+  /**
+   * get partition map of range interval filter on range interval 
partition table
+   * @param partitionInfo
+   * @param partitioner
+   * @param filterValue
+   * @param isGreaterThan
+   * @param isEqualTo
+   * @return
+   */
+  public static BitSet getPartitionMapForRangeIntervalFilter(PartitionInfo 
partitionInfo,
+  RangeIntervalPartitioner partitioner, Object filterValue,
+  boolean isGreaterThan, boolean isEqualTo,
+  DateFormat timestampFormatter, DateFormat dateFormatter) {
+
+List values = partitionInfo.getRangeInfo();
+DataType partitionColumnDataType = 
partitionInfo.getColumnSchemaList().get(0).getDataType();
+
+Comparator comparator =
+
PartitionFilterUtil.getComparatorByDataType(partitionColumnDataType);
+
+BitSet partitionMap = PartitionUtil
+.generateBitSetBySize(partitioner.numPartitions(), false);
+
+int numPartitions = values.size() - 1;
+int result = 0;
+// the partition index of filter value
+int partitionIndex = 0;
+// find the partition of filter value
+for (; partitionIndex < numPartitions; partitionIndex++) {
+  result = comparator.compare(filterValue, 
PartitionUtil.getDataBasedOnDataType(
+  values.get(partitionIndex), partitionColumnDataType, 
timestampFormatter, dateFormatter));
+  if (result <= 0) {
+break;
+  }
+}
+if (partitionIndex == numPartitions) {
+  // filter value is not in given partition, should get partitionId 
dynamically
+  int partitionId = partitioner.getPartition(filterValue);
--- End diff --

this should be partitionIndex


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] carbondata issue #1139: [CARBONDATA-1267] Add short_int case branch in delta...

2017-07-09 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1139
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2981/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] carbondata issue #1139: [CARBONDATA-1267] Add short_int case branch in delta...

2017-07-09 Thread xuchuanyin

Github user xuchuanyin commented on the issue:

https://github.com/apache/carbondata/pull/1139
  
it seems the error has nothing to do with my modification.

retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] carbondata issue #1149: [WIP]Implement hive based schema storage in carbon

2017-07-09 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1149
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2980/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] carbondata issue #1149: [WIP]Implement hive based schema storage in carbon

2017-07-09 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1149
  
Build Success with Spark 1.6, Please check CI 
http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/391/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] carbondata pull request #1103: [WIP] Implement range interval partition

2017-07-09 Thread chenerlu

Github user chenerlu commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/1103#discussion_r126324104
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/metadata/schema/PartitionInfo.java
 ---
@@ -47,6 +47,12 @@
*/
   private int numPartitions;
 
+  /**
+   * range interval information defined for range interval partition table
+   */
+  private List rangeIntervalInfo;
+
+
--- End diff --

Ok~


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] carbondata issue #1146: [WIP] Change Query related RDD to use TableInfo

2017-07-09 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1146
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2979/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] carbondata issue #1146: [WIP] Change Query related RDD to use TableInfo

2017-07-09 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1146
  
Build Failed with Spark 1.6, Please check CI 
http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/388/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] carbondata issue #1146: [WIP] Change Query related RDD to use TableInfo

2017-07-09 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1146
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2978/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] carbondata issue #1146: [WIP] Change Query related RDD to use TableInfo

2017-07-09 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1146
  
Build Failed with Spark 1.6, Please check CI 
http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/387/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] carbondata issue #1146: [WIP] Change Query related RDD to use TableInfo

2017-07-09 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1146
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2976/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] carbondata issue #1125: [CarbonData-1250] Change default partition id & Add ...

2017-07-09 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1125
  
Build Failed with Spark 1.6, Please check CI 
http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/386/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] carbondata issue #1125: [CarbonData-1250] Change default partition id & Add ...

2017-07-09 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1125
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2975/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (CARBONDATA-1153) Can not add column

2017-07-09 Thread cen yuhai (JIRA)


[ 
https://issues.apache.org/jira/browse/CARBONDATA-1153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16079624#comment-16079624
 ] 

cen yuhai commented on CARBONDATA-1153:
---

I found the root cause, I don't have a carbon.properties
{code}
17/07/09 19:08:13 ERROR HdfsFileLock: Executor task launch worker for task 7 
Incomplete HDFS URI, no host: 
hdfs://mycluster../carbon.store/temp/yuhai_carbon/d91e35aa-5f13-499c-adcb-94fc20dcf8fb.lock
java.io.IOException: Incomplete HDFS URI, no host: 
hdfs://mycluster../carbon.store/temp/yuhai_carbon/d91e35aa-5f13-499c-adcb-94fc20dcf8fb.lock
{code}


> Can not add column
> --
>
> Key: CARBONDATA-1153
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1153
> Project: CarbonData
>  Issue Type: Bug
>  Components: spark-integration
>Affects Versions: 1.2.0
>Reporter: cen yuhai
>
> Sometimes it will throws exception as below. why can't I add column? no one 
> are altering the table... 
> {code}
> scala> carbon.sql("alter table temp.yuhai_carbon add columns(test1 string)")
> 17/06/11 22:09:13 AUDIT 
> [org.apache.spark.sql.execution.command.AlterTableAddColumns(207) -- main]: 
> [sh-hadoop-datanode-250-104.elenet.me][master][Thread-1]Alter table add 
> columns request has been received for temp.yuhai_carbon
> 17/06/11 22:10:22 ERROR [org.apache.spark.scheduler.TaskSetManager(70) -- 
> task-result-getter-3]: Task 0 in stage 0.0 failed 4 times; aborting job
> 17/06/11 22:10:22 ERROR 
> [org.apache.spark.sql.execution.command.AlterTableAddColumns(141) -- main]: 
> main Alter table add columns failed :Job aborted due to stage failure: Task 0 
> in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 
> (TID 3, sh-hadoop-datanode-368.elenet.me, executor 7): 
> java.lang.RuntimeException: Dictionary file test1 is locked for updation. 
> Please try after some time
> at scala.sys.package$.error(package.scala:27)
> at 
> org.apache.carbondata.spark.util.GlobalDictionaryUtil$.loadDefaultDictionaryValueForNewColumn(GlobalDictionaryUtil.scala:857)
> at 
> org.apache.carbondata.spark.rdd.AlterTableAddColumnRDD$$anon$1.(AlterTableAddColumnRDD.scala:83)
> at 
> org.apache.carbondata.spark.rdd.AlterTableAddColumnRDD.compute(AlterTableAddColumnRDD.scala:68)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:331)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:295)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:88)
> at org.apache.spark.scheduler.Task.run(Task.scala:104)
> at 
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:351)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] carbondata issue #984: [CARBONDATA-1008] Make Caron table schema compatible ...

2017-07-09 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/984
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2974/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] carbondata issue #984: [CARBONDATA-1008] Make Caron table schema compatible ...

2017-07-09 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/984
  
Build Success with Spark 1.6, Please check CI 
http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/385/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] carbondata issue #984: [CARBONDATA-1008] Make Caron table schema compatible ...

2017-07-09 Thread cenyuhai

Github user cenyuhai commented on the issue:

https://github.com/apache/carbondata/pull/984
  
@chenliang613 can you review this pr?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] carbondata issue #1139: [CARBONDATA-1267] Add short_int case branch in delta...

2017-07-09 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1139
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2973/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] carbondata issue #1139: [CARBONDATA-1267] Add short_int case branch in delta...

2017-07-09 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1139
  
Build Success with Spark 1.6, Please check CI 
http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/384/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] carbondata issue #1139: [CARBONDATA-1267] Add short_int case branch in delta...

2017-07-09 Thread xuchuanyin

Github user xuchuanyin commented on the issue:

https://github.com/apache/carbondata/pull/1139
  
retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] carbondata issue #1139: [CARBONDATA-1267] Add short_int case branch in delta...

2017-07-09 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1139
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2971/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] carbondata issue #1139: [CARBONDATA-1267] Add short_int case branch in delta...

2017-07-09 Thread xuchuanyin

Github user xuchuanyin commented on the issue:

https://github.com/apache/carbondata/pull/1139
  
retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] carbondata issue #1115: [CARBONDATA-1247]Block pruning not working for date ...

2017-07-09 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1115
  
Build Failed with Spark 1.6, Please check CI 
http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/382/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] carbondata issue #1115: [CARBONDATA-1247]Block pruning not working for date ...

2017-07-09 Thread BJangir

Github user BJangir commented on the issue:

https://github.com/apache/carbondata/pull/1115
  
please retest


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] carbondata issue #984: [CARBONDATA-1008] Make Caron table schema compatible ...

2017-07-09 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/984
  
Build Failed with Spark 1.6, Please check CI 
http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/381/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] carbondata issue #984: [CARBONDATA-1008] Make Caron table schema compatible ...

2017-07-09 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/984
  
Build Success with Spark 1.6, Please check CI 
http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/380/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] carbondata issue #984: [CARBONDATA-1008] Make Caron table schema compatible ...

2017-07-09 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/984
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2969/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] carbondata issue #984: [CARBONDATA-1008] Make Caron table schema compatible ...

2017-07-09 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/984
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2968/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] carbondata issue #1142: [CARBONDATA-1271] Enhanced Performance for Hive Inte...

2017-07-09 Thread cenyuhai

Github user cenyuhai commented on the issue:

https://github.com/apache/carbondata/pull/1142
  
Can you provide performance benchmark?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] carbondata issue #984: [CARBONDATA-1008] Make Caron table schema compatible ...

2017-07-09 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/984
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2967/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] carbondata issue #984: [CARBONDATA-1008] Make Caron table schema compatible ...

2017-07-09 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/984
  
Build Success with Spark 1.6, Please check CI 
http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/379/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] carbondata issue #984: [CARBONDATA-1008] Make Caron table schema compatible ...

2017-07-09 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/984
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2966/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] carbondata issue #984: [CARBONDATA-1008] Make Caron table schema compatible ...

2017-07-09 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/984
  
Build Success with Spark 1.6, Please check CI 
http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/378/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

59 matches

Mail list logo