[GitHub] incubator-carbondata issue #310: [CARBONDATA-401] One Pass Load
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/310 Build Success with Spark 1.5.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/142/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #425: [CARBONDATA-429][WIP]reduce the no of of io...
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/425 Build Failed with Spark 1.5.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/141/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #426: [MINOR-FIX]change the declared package of t...
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/426 Build Success with Spark 1.5.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/140/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Updated] (CARBONDATA-431) Analysis compression for numeric datatype compared with Parquet/ORC
[ https://issues.apache.org/jira/browse/CARBONDATA-431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacky Li updated CARBONDATA-431: Fix Version/s: 1.0.0-incubating > Analysis compression for numeric datatype compared with Parquet/ORC > --- > > Key: CARBONDATA-431 > URL: https://issues.apache.org/jira/browse/CARBONDATA-431 > Project: CarbonData > Issue Type: Sub-task >Reporter: suo tong >Assignee: Ashok Kumar > Fix For: 1.0.0-incubating > > Time Spent: 2h 50m > Remaining Estimate: 0h > > For the data type, carbon's string type has better compression ratio, but for > numeric type, orc has the best compression. we should analysis numeric > datatype for carbon to get better compression ratio > DataType TextParquet Orc Carbon > decimal 16G |11G | 6G|13G > int 5G | 1G |1G |3G > String 24G |22G |11G |3G （no > dictionary） ---high cardinality > String30G|4G |4G |1G -- > Dictionary encode1G -- Dictionary encode without inverted index > 3G -- No dictionary encode ---low cardinality -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (CARBONDATA-431) Analysis compression for numeric datatype compared with Parquet/ORC
[ https://issues.apache.org/jira/browse/CARBONDATA-431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacky Li resolved CARBONDATA-431. - Resolution: Fixed Assignee: Ashok Kumar (was: Raghunandan S) > Analysis compression for numeric datatype compared with Parquet/ORC > --- > > Key: CARBONDATA-431 > URL: https://issues.apache.org/jira/browse/CARBONDATA-431 > Project: CarbonData > Issue Type: Sub-task >Reporter: suo tong >Assignee: Ashok Kumar > Time Spent: 2h 50m > Remaining Estimate: 0h > > For the data type, carbon's string type has better compression ratio, but for > numeric type, orc has the best compression. we should analysis numeric > datatype for carbon to get better compression ratio > DataType TextParquet Orc Carbon > decimal 16G |11G | 6G|13G > int 5G | 1G |1G |3G > String 24G |22G |11G |3G （no > dictionary） ---high cardinality > String30G|4G |4G |1G -- > Dictionary encode1G -- Dictionary encode without inverted index > 3G -- No dictionary encode ---low cardinality -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-carbondata issue #426: [MINOR-FIX]change the declared package of t...
Github user jackylk commented on the issue: https://github.com/apache/incubator-carbondata/pull/426 add to whitelist --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #388: [CARBONDATA-431]bigdecimal compressi...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-carbondata/pull/388 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Resolved] (CARBONDATA-528) to support octal escape delimiter char
[ https://issues.apache.org/jira/browse/CARBONDATA-528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacky Li resolved CARBONDATA-528. - Resolution: Fixed Assignee: zhaowei > to support octal escape delimiter char > --- > > Key: CARBONDATA-528 > URL: https://issues.apache.org/jira/browse/CARBONDATA-528 > Project: CarbonData > Issue Type: Improvement >Affects Versions: 0.2.0-incubating >Reporter: zhaowei >Assignee: zhaowei >Priority: Minor > Fix For: 1.0.0-incubating > > Time Spent: 1h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-carbondata pull request #424: [CARBONDATA-528]support octal escape...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-carbondata/pull/424 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #424: [CARBONDATA-528]support octal escape delimi...
Github user jackylk commented on the issue: https://github.com/apache/incubator-carbondata/pull/424 LGTM. I checked java library but could not find one either, I think we can live with it now and refactory later. Thanks for working for this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #388: [CARBONDATA-431]bigdecimal compression.
Github user jackylk commented on the issue: https://github.com/apache/incubator-carbondata/pull/388 LGTM Thanks for working for this --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #412: [CARBONDATA-519]Added vector reader ...
Github user piaoyats commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/412#discussion_r92108917
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/CarbonLateDecodeStrategy.scala
---
@@ -87,19 +90,17 @@ private[sql] class CarbonLateDecodeStrategy extends
SparkStrategy {
private[this] def toCatalystRDD(
relation: LogicalRelation,
output: Seq[Attribute],
- rdd: RDD[Row],
+ rdd: RDD[InternalRow],
needDecode: ArrayBuffer[AttributeReference]):
RDD[InternalRow] = {
-val newRdd = if (needDecode.size > 0) {
+if (needDecode.size > 0) {
+ rdd.asInstanceOf[CarbonScanRDD].setVectorReaderSupport(false)
getDecoderRDD(relation, needDecode, rdd, output)
--- End diff --
hi, i want to know what will happen if setVectorReaderSupport(true) when
needDecode.size > 0
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #388: [CARBONDATA-431]bigdecimal compression.
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/388 Build Success with Spark 1.5.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/139/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #388: [CARBONDATA-431]bigdecimal compression.
Github user ashokblend commented on the issue: https://github.com/apache/incubator-carbondata/pull/388 review comments are handled, please check --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #424: [CARBONDATA-528]support octal escape delimi...
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/424 Build Success with Spark 1.5.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/138/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #273: [CARBONDATA-332]Prohibit to use rese...
Github user lion-x closed the pull request at: https://github.com/apache/incubator-carbondata/pull/273 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #388: [CARBONDATA-431]bigdecimal compression.
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/388 Build Success with Spark 1.5.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/137/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #424: [CARBONDATA-528]support octal escape delimi...
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/424 Build Failed with Spark 1.5.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/136/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #194: [CARBONDATA-270] Double data type va...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-carbondata/pull/194 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Resolved] (CARBONDATA-270) [Filter Optimization] double data type value comparison optimization
[ https://issues.apache.org/jira/browse/CARBONDATA-270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravindra Pesala resolved CARBONDATA-270. Resolution: Fixed Fix Version/s: 1.0.0-incubating > [Filter Optimization] double data type value comparison optimization > > > Key: CARBONDATA-270 > URL: https://issues.apache.org/jira/browse/CARBONDATA-270 > Project: CarbonData > Issue Type: Improvement > Components: core >Reporter: Sujith >Assignee: Sujith >Priority: Minor > Fix For: 1.0.0-incubating > > Time Spent: 1.5h > Remaining Estimate: 0h > > EqualsToExpression evaluation for double values first check for the equality > of nan values and then the double value comparison happens, since nan > comparison scenarios are rare we can push the comparison of nan after the > double value comparison. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-carbondata issue #194: [CARBONDATA-270] Double data type value com...
Github user ravipesala commented on the issue: https://github.com/apache/incubator-carbondata/pull/194 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #194: [CARBONDATA-270] Double data type value com...
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/194 Build Success with Spark 1.5.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/135/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #373: [CARBONDATA-474] test case for columnar pac...
Github user zzcclp commented on the issue: https://github.com/apache/incubator-carbondata/pull/373 @anuragknoldus @ravipesala I have created a [pr](https://github.com/apache/incubator-carbondata/pull/426) for wrong declared package --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #194: [CARBONDATA-270] Double data type value com...
Github user ravipesala commented on the issue: https://github.com/apache/incubator-carbondata/pull/194 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #426: [MINOR-FIX]change the declared package of t...
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/426 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #426: [MINOR-FIX]change the declared packa...
GitHub user zzcclp opened a pull request: https://github.com/apache/incubator-carbondata/pull/426 [MINOR-FIX]change the declared package of these four java files The declared package of these four java files must be modidied to "org.apache.carbondata.core.datastorage.store.filesystem": AlluxioCarbonFileTest.java HDFSCarbonFileTest.java LocalCarbonFileTest.java ViewFsCarbonFileTest.java You can merge this pull request into a Git repository by running: $ git pull https://github.com/zzcclp/incubator-carbondata change_declared_package Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-carbondata/pull/426.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #426 commit 59d17168f8f015acd10f7b540b5928a50d6166d9 Author: Zhang Zhichao <441586...@qq.com> Date: 2016-12-13T06:28:25Z The declared package of these four java files must be modidied to "org.apache.carbondata.core.datastorage.store.filesystem" minor fix --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #388: [CARBONDATA-431]bigdecimal compression.
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/388 Build Failed with Spark 1.5.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/134/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #424: [CARBONDATA-528]support octal escape...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/424#discussion_r92103661
--- Diff:
core/src/main/java/org/apache/carbondata/core/util/CarbonUtil.java ---
@@ -599,22 +599,7 @@ public static void writeLevelCardinalityFile(String
loadFolderLoc, String tableN
* @return
*/
public static String unescapeChar(String parseStr) {
-switch (parseStr) {
- case "\\001":
-return "\001";
- case "\\t":
-return "\t";
- case "\\r":
-return "\r";
- case "\\b":
-return "\b";
- case "\\f":
-return "\f";
- case "\\n":
-return "\n";
- default:
-return parseStr;
-}
+return scala.StringContext.treatEscapes(parseStr);
--- End diff --
I found the functionality of `treatEscapes` is as following.
```
/** Expands standard Scala escape sequences in a string.
* Escape sequences are:
* control: `\b`, `\t`, `\n`, `\f`, `\r`
* escape: `\\`, `\"`, `\'`
* octal: `\d` `\dd` `\ddd` where `d` is an octal digit between `0`
and `7`.
*
* @param str A string that may contain escape sequences
* @return The string with all escape sequences expanded.
*/
```
Do you want to support octal string like \1, \2?
Adding this functionality is good, my concern is that can you find a java
function instead of using scala library? Because generaling speaking, core
should not dependent on scala library, they should be pure java implementation.
Another thing is can you add a testcase for this PR?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #424: [CARBONDATA-528]support octal escape delimi...
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/424 Build Success with Spark 1.5.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/133/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #425: [CARBONDATA-429][WIP]reduce the no of of io...
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/425 Build Failed with Spark 1.5.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/132/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #422: [CARBONDATA-526] FilterExpressionProcessorT...
Github user ravipesala commented on the issue: https://github.com/apache/incubator-carbondata/pull/422 add to whitelist --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #425: [CARBONDATA-429][WIP]reduce the no of of io...
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/425 Build Failed with Spark 1.5.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/131/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Resolved] (CARBONDATA-525) Fix timestamp based test cases
[ https://issues.apache.org/jira/browse/CARBONDATA-525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravindra Pesala resolved CARBONDATA-525. Resolution: Fixed Fix Version/s: 1.0.0-incubating > Fix timestamp based test cases > -- > > Key: CARBONDATA-525 > URL: https://issues.apache.org/jira/browse/CARBONDATA-525 > Project: CarbonData > Issue Type: Test >Reporter: Prabhat Kashyap >Priority: Trivial > Fix For: 1.0.0-incubating > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-carbondata pull request #421: [CARBONDATA -525] Timestamp based te...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-carbondata/pull/421 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #421: [CARBONDATA -525] Timestamp based testcases...
Github user ravipesala commented on the issue: https://github.com/apache/incubator-carbondata/pull/421 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #421: [CARBONDATA -525] Timestamp based testcases...
Github user deepakmehra10 commented on the issue: https://github.com/apache/incubator-carbondata/pull/421 Hi @chenliang613 can you please review the changes made. My PR seems to be fine i.e all the commands are running fine but the build gets failed on CI . Can you please check. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #421: [CARBONDATA -525] Timestamp based testcases...
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/421 Build Success with Spark 1.5.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/130/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #421: [CARBONDATA -525] Timestamp based te...
Github user deepakmehra10 commented on a diff in the pull request: https://github.com/apache/incubator-carbondata/pull/421#discussion_r92096304 --- Diff: core/src/test/java/org/apache/carbondata/scan/complextypes/PrimitiveQueryTypeTest.java --- @@ -37,12 +40,14 @@ import org.junit.Test; import static org.junit.Assert.assertEquals; +import static sun.security.krb5.internal.ktab.KeyTabConstants.keySize; --- End diff -- In order to test timestamp based testcases, I have written a function in the test case which needs keysize and keysize is a constant which is why i have imported this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #425: [CARBONDATA-429][WIP]reduce the no of of io...
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/425 Build Failed with Spark 1.5.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/128/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #424: [CARBONDATA-528]support octal escape delimi...
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/424 Build Failed with Spark 1.5.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/129/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #424: [CARBONDATA-528]support octal escape delimi...
Github user chenliang613 commented on the issue: https://github.com/apache/incubator-carbondata/pull/424 add to whitelist --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #421: [CARBONDATA -525] Timestamp based te...
Github user chenliang613 commented on a diff in the pull request: https://github.com/apache/incubator-carbondata/pull/421#discussion_r92093995 --- Diff: core/src/test/java/org/apache/carbondata/scan/complextypes/PrimitiveQueryTypeTest.java --- @@ -37,12 +40,14 @@ import org.junit.Test; import static org.junit.Assert.assertEquals; +import static sun.security.krb5.internal.ktab.KeyTabConstants.keySize; --- End diff -- Why you imported this constant? you can select to define a constant. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #421: [CARBONDATA -525] Timestamp based te...
Github user kumarvishal09 commented on a diff in the pull request: https://github.com/apache/incubator-carbondata/pull/421#discussion_r92093904 --- Diff: core/src/test/java/org/apache/carbondata/scan/complextypes/PrimitiveQueryTypeTest.java --- @@ -37,12 +40,14 @@ import org.junit.Test; import static org.junit.Assert.assertEquals; +import static sun.security.krb5.internal.ktab.KeyTabConstants.keySize; --- End diff -- Why this import is required?? You can create a constants in your test case --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #421: [CARBONDATA -525] Timestamp based testcases...
Github user chenliang613 commented on the issue: https://github.com/apache/incubator-carbondata/pull/421 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #425: [CARBONDATA-429][WIP]reduce the no of of io...
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/425 Build Failed with Spark 1.5.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/127/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #425: [CARBONDATA-429][WIP]reduce the no of of io...
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/425 Build Failed with Spark 1.5.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/126/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #425: [CARBONDATA-429][WIP]reduce the no o...
GitHub user ashokblend opened a pull request: https://github.com/apache/incubator-carbondata/pull/425 [CARBONDATA-429][WIP]reduce the no of of io operation being done for dictionary. **Problem** Every single query is triggered by user, carbon does an io operation for all dictionary column to check if its dictionary meta is modified. **Solution** This PR will try to reduce the no of IO operation done for above mentioned problem. Idea behind solution is, check for modification of dictionary meta only if table has been loaded with new segment. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ashokblend/incubator-carbondata dictmetacache Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-carbondata/pull/425.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #425 commit 747d35337f34ef252f2884f0616dd89eb8957d88 Author: ashok.blend Date: 2016-12-13T01:30:10Z reduce the no of of io operation being done for dictionary. Idea is, check is table is modified then only tries to reload the cache. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (CARBONDATA-528) to support octal escape delimiter char
zhaowei created CARBONDATA-528: -- Summary: to support octal escape delimiter char Key: CARBONDATA-528 URL: https://issues.apache.org/jira/browse/CARBONDATA-528 Project: CarbonData Issue Type: Improvement Affects Versions: 0.2.0-incubating Reporter: zhaowei Priority: Minor Fix For: 1.0.0-incubating -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-carbondata pull request #388: [CARBONDATA-431]bigdecimal compressi...
Github user piaoyats commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/388#discussion_r92091360
--- Diff:
core/src/main/java/org/apache/carbondata/core/util/ValueCompressionUtil.java ---
@@ -243,46 +270,64 @@ public static Object
getCompressedValues(COMPRESSION_TYPE compType, double[] val
* @param actualDataType
* @return compressor based on actualdatatype
*/
- public static ValueCompressor getValueCompressor(DataType
actualDataType) {
-switch (actualDataType) {
- case DATA_BIGINT:
-return new BigIntCompressor();
- default:
-return new DoubleCompressor();
+ public static ValueCompressor getValueCompressor(
+ CompressionFinder compressionFinder) {
+if (compressionFinder instanceof BigDecimalCompressionFinder) {
+ return new BigDecimalCompressor();
+} else {
+ switch (compressionFinder.getActualDataType()) {
+case DATA_BIGINT:
+ return new BigIntCompressor();
+default:
+ return new DoubleCompressor();
+ }
}
}
- private static ValueCompressonHolder.UnCompressValue[]
getUncompressedValues(
- COMPRESSION_TYPE[] compType, DataType[] actualDataType, DataType[]
changedDataType) {
-
-ValueCompressonHolder.UnCompressValue[] compressValue =
-new ValueCompressonHolder.UnCompressValue[changedDataType.length];
-for (int i = 0; i < changedDataType.length; i++) {
- switch (compType[i]) {
-case ADAPTIVE:
- compressValue[i] = getUnCompressNone(changedDataType[i],
actualDataType[i]);
- break;
-
-case DELTA_DOUBLE:
- compressValue[i] =
getUnCompressDecimalMaxMin(changedDataType[i], actualDataType[i]);
- break;
-
-case BIGINT:
- compressValue[i] = getUnCompressNonDecimal(changedDataType[i]);
- break;
-
-case BIGDECIMAL:
- compressValue[i] = new
UnCompressByteArray(UnCompressByteArray.ByteArrayType.BIG_DECIMAL);
- break;
+ /**
+ * get uncompressed object
+ * @param compressionFinders : Compression types for measures
+ * @return
+ */
+ private static UnCompressValue[] getUncompressedValues(
+ CompressionFinder[] compressionFinders) {
+UnCompressValue[] unCompressedValues = new
UnCompressValue[compressionFinders.length];
+for (int i=0; i< compressionFinders.length; i++) {
+ unCompressedValues[i] = getUncompressedValue(compressionFinders[i]);
+}
+return unCompressedValues;
+ }
+ /**
+ *
+ * @param compressionFinder bigdecimal compression finder
+ * @return Uncompressed measure object
+ */
+ private static UnCompressValue getUncompressedValue(
+ BigDecimalCompressionFinder compressionFinder) {
+UnCompressValue leftPart =
getUncompressedValue(compressionFinder.getLeftCompType(),
+compressionFinder.getLeftActualDataType(),
compressionFinder.getLeftChangedDataType());
+UnCompressValue rightPart =
getUncompressedValue(compressionFinder.getRightCompType(),
+compressionFinder.getRightActualDataType(),
compressionFinder.getRightChangedDataType());
+return new UnCompressBigDecimal<>(compressionFinder, leftPart,
rightPart);
+ }
-default:
- compressValue[i] =
getUnCompressNonDecimalMaxMin(changedDataType[i]);
- }
+ /**
+ *
+ * @param compressionFinder for measure other then bigdecimal
+ * @return
+ */
+ private static UnCompressValue getUncompressedValue(
+ CompressionFinder compressionFinder) {
+if (compressionFinder instanceof BigDecimalCompressionFinder) {
+ return getUncompressedValue(
--- End diff --
same instanceof here
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #424: support octal escape delimited
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/424 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #424: support octal escape delimited
GitHub user boczhaow opened a pull request: https://github.com/apache/incubator-carbondata/pull/424 support octal escape delimited to support octal escape delimited char You can merge this pull request into a Git repository by running: $ git pull https://github.com/boczhaow/incubator-carbondata delimited Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-carbondata/pull/424.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #424 commit 7216dcac7a048861c254ed8a495938df1d0d3d69 Author: boczhaow Date: 2016-12-13T03:06:54Z support octal escape delimited commit 79fd1f5297ce137e042e98cdc025dab25f938311 Author: boczhaow Date: 2016-12-13T03:13:33Z style --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #388: [CARBONDATA-431]bigdecimal compressi...
Github user piaoyats commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/388#discussion_r92090267
--- Diff:
core/src/main/java/org/apache/carbondata/core/util/ValueCompressionUtil.java ---
@@ -243,46 +270,64 @@ public static Object
getCompressedValues(COMPRESSION_TYPE compType, double[] val
* @param actualDataType
* @return compressor based on actualdatatype
*/
- public static ValueCompressor getValueCompressor(DataType
actualDataType) {
-switch (actualDataType) {
- case DATA_BIGINT:
-return new BigIntCompressor();
- default:
-return new DoubleCompressor();
+ public static ValueCompressor getValueCompressor(
+ CompressionFinder compressionFinder) {
+if (compressionFinder instanceof BigDecimalCompressionFinder) {
--- End diff --
instanceof is a heavy operation, we should avoid to call it.
we can add a function to CompressionFinder to detect if a compression is
BigDecimal or not
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #388: [CARBONDATA-431]bigdecimal compressi...
Github user piaoyats commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/388#discussion_r92089924
--- Diff:
core/src/main/java/org/apache/carbondata/core/util/ValueCompressionUtil.java ---
@@ -196,6 +198,31 @@ private static CompressionFinder
getDoubleCompressorFinder(Object maxValue, Obje
}
}
+ private static CompressionFinder getBigDecimalCompressorFinder(
+ Object maxValue, Object minValue, int decimal, byte
dataTypeSelected) {
--- End diff --
change decimal to mantissa, keep same with others
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #388: [CARBONDATA-431]bigdecimal compressi...
Github user piaoyats commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/388#discussion_r92089699
--- Diff:
core/src/main/java/org/apache/carbondata/core/util/ValueCompressionUtil.java ---
@@ -138,8 +141,7 @@ public static CompressionFinder
getCompressionFinder(Object maxValue, Object min
// ''b' for decimal, 'l' for long, 'n' for double
switch (measureStoreType) {
case 'b':
-return new CompressionFinder(COMPRESSION_TYPE.BIGDECIMAL,
DataType.DATA_BYTE,
-DataType.DATA_BYTE);
+return getBigDecimalCompressorFinder(maxValue, minValue, 0,
dataTypeSelected);
--- End diff --
if mantissa equals to 0 hear for big decimal? if it is, keep passing
mantissa to getBigDecimalCompressorFinder
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #388: [CARBONDATA-431]bigdecimal compressi...
Github user piaoyats commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/388#discussion_r92088924
--- Diff:
core/src/main/java/org/apache/carbondata/core/util/BigDecimalCompressionFinder.java
---
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.carbondata.core.util;
+
+import
org.apache.carbondata.core.util.ValueCompressionUtil.COMPRESSION_TYPE;
+import org.apache.carbondata.core.util.ValueCompressionUtil.DataType;
+
+public class BigDecimalCompressionFinder extends CompressionFinder {
+
+ private COMPRESSION_TYPE leftCompType;
+
+ private COMPRESSION_TYPE rightCompType;
+
+ private DataType leftActualDataType;
+
+ private DataType rightActualDataType;
--- End diff --
what does the ActualDataType and ChangedDataType mean? Does
actualDataType mean original data type and changedDataType is the type we
store? suggest we rename the variables more readable
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #373: [CARBONDATA-474] test case for columnar pac...
Github user zzcclp commented on the issue: https://github.com/apache/incubator-carbondata/pull/373 ping @anuragknoldus @ravipesala --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #388: [CARBONDATA-431]bigdecimal compressi...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/388#discussion_r92079674
--- Diff:
core/src/main/java/org/apache/carbondata/core/compression/BigIntCompressor.java
---
@@ -44,6 +44,11 @@ protected Object compressMaxMin(DataType
changedDataType, CarbonWriteDataHolder
Object max) {
long maxValue = (long) max;
long[] value = dataHolder.getWritableLongValues();
+return compressMaxMin(changedDataType, maxValue, value);
+ }
+
+ protected Object compressMaxMin(DataType changedDataType, long maxValue,
--- End diff --
please add description to these compress functions, make it easier to
understand by reader
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #411: [WIP]Support data type: date and char
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/411 Build Failed with Spark 1.5.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/125/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #388: [CARBONDATA-431]bigdecimal compressi...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/388#discussion_r92078130
--- Diff:
core/src/main/java/org/apache/carbondata/core/compression/BigDecimalCompressor.java
---
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.carbondata.core.compression;
+
+import
org.apache.carbondata.core.datastorage.store.dataholder.CarbonWriteDataHolder;
+import org.apache.carbondata.core.util.BigDecimalCompressionFinder;
+import org.apache.carbondata.core.util.CompressionFinder;
+import org.apache.carbondata.core.util.ValueCompressionUtil.DataType;
+
+/**
+ * Bigdecimal data type compressor
+ *
+ */
+public class BigDecimalCompressor extends BigIntCompressor {
+
+ private boolean readLeft = true;
+
+ @Override
+ public Object getCompressedValues(CompressionFinder compressionFinder,
+ CarbonWriteDataHolder dataHolder, Object maxValue, int decimal) {
+BigDecimalCompressionFinder bigdCompressionFinder =
+(BigDecimalCompressionFinder) compressionFinder;
+Long[] maxValues = (Long[]) maxValue;
+Object leftCompressedValue = getCompressedValues(
+bigdCompressionFinder.getLeftCompType(), dataHolder,
+bigdCompressionFinder.getLeftChangedDataType(), maxValues[0], 0);
+readLeft = false;
+Object rightCompressedValue = getCompressedValues(
+bigdCompressionFinder.getRightCompType(), dataHolder,
+bigdCompressionFinder.getRightChangedDataType(), maxValues[1], 0);
+return new Object[] { leftCompressedValue, rightCompressedValue };
+
--- End diff --
remove empty line
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #423: [CARBONDATA-527] Greater than/less-than/Lik...
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/423 Build Success with Spark 1.5.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/124/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #423: [CARBONDATA-527] Greater than/less-t...
GitHub user sujith71955 opened a pull request: https://github.com/apache/incubator-carbondata/pull/423 [CARBONDATA-527] Greater than/less-than/Like filters optimization for dictionary columns Current design In greater than/less-than/Like filters, system first iterates each row present in the dictionary cache for identifying valid filter actual members by applying the filter expression , once evaluation done system will hold the list of identified valid filter actual member values(String), now in next step again system will look up the dictionary cache in order to identify the dictionary surrogate values of the identified members. this look up is an additional cost to our system even though the look up methodology is an binary search in dictionary cache. Proposed design/solution: Identify the dictionary surrogate values in filter expression evaluation step itself when actual dictionary values will be scanned for identifying valid filter members . Keep a dictionary counter variable which will be increased when system iterates through the dictionary cache in order to retrieve each actual member stored in dictionary cache , after this system will evaluate each row against the filter expression to identify whether its a valid filter member or not, while doing this process itself counter value can be taken as valid selected dictionary value since the actual member values and its dictionary values will be kept in same order in dictionary cache as the iteration order. thus it will eliminate the further dictionary look up step which is required to retrieve the dictionary surrogate value against identified actual valid filter member. this can also increase significantly the filter query performance of such filter queries which require expression evaluation to identify it the filter members by looking up dictionary cache, like greater than/less-than/Like filters . Note : this optimization is applicable for dictionary columns. You can merge this pull request into a Git repository by running: $ git pull https://github.com/sujith71955/incubator-carbondata master_likefilter Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-carbondata/pull/423.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #423 commit b81b4aaa58ca9de79b8b3cdf3191a7fa0f14575a Author: sujith71955 Date: 2016-12-12T18:23:38Z JIRA ID: CARBONDATA-527 Description: Identify the dictionary surrogate values in filter expression evaluation step itself when actual dictionary values will be scanned for identifying valid filter members --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (CARBONDATA-527) Greater than/less-than/Like filters optimization for dictionary columns
Sujith created CARBONDATA-527: - Summary: Greater than/less-than/Like filters optimization for dictionary columns Key: CARBONDATA-527 URL: https://issues.apache.org/jira/browse/CARBONDATA-527 Project: CarbonData Issue Type: Improvement Reporter: Sujith Current design In greater than/less-than/Like filters, system first iterates each row present in the dictionary cache for identifying valid filter actual members by applying the filter expression , once evaluation done system will hold the list of identified valid filter actual member values(String), now in next step again system will look up the dictionary cache in order to identify the dictionary surrogate values of the identified members. this look up is an additional cost to our system even though the look up methodology is an binary search in dictionary cache. Proposed design/solution: Identify the dictionary surrogate values in filter expression evaluation step itself when actual dictionary values will be scanned for identifying valid filter members . Keep a dictionary counter variable which will be increased when system iterates through the dictionary cache in order to retrieve each actual member stored in dictionary cache , after this system will evaluate each row against the filter expression to identify whether its a valid filter member or not, while doing this process itself counter value can be taken as valid selected dictionary value since the actual member values and its dictionary values will be kept in same order in dictionary cache as the iteration order. thus it will eliminate the further dictionary look up step which is required to retrieve the dictionary surrogate value against identified actual valid filter member. this can also increase significantly the filter query performance of such filter queries which require expression evaluation to identify it the filter members by looking up dictionary cache, like greater than/less-than/Like filters . Note : this optimization is applicable for dictionary columns. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-carbondata pull request #412: [CARBONDATA-519]Added vector reader ...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/412#discussion_r91961576
--- Diff:
integration/spark-common/src/main/scala/org/apache/carbondata/spark/rdd/CarbonScanRDD.scala
---
@@ -150,12 +153,25 @@ class CarbonScanRDD[V: ClassTag](
val attemptContext = new TaskAttemptContextImpl(new Configuration(),
attemptId)
val format =
prepareInputFormatForExecutor(attemptContext.getConfiguration)
val inputSplit = split.asInstanceOf[CarbonSparkPartition].split.value
-val reader = format.createRecordReader(inputSplit, attemptContext)
+val model = format.getQueryModel(inputSplit, attemptContext)
+val reader = {
+ if (vectorReader) {
+val carbonRecordReader = createVectorizedCarbonRecordReader(model)
+if (carbonRecordReader == null) {
+ new CarbonRecordReader(model,
format.getReadSupportClass(attemptContext.getConfiguration))
+} else {
+ carbonRecordReader
+}
+ } else {
+new CarbonRecordReader(model,
format.getReadSupportClass(attemptContext.getConfiguration))
--- End diff --
should not new CarbonRecordReader directly, can we choose:
option 1: create two InputFormat, one for batch and another for non-batch
option 2: one InputFormat, create RecordReader according to configuration.
I prefer option 2.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #411: [WIP]Support data type: date and char
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/411 Build Failed with Spark 1.5.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/123/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #422: [CARBONDATA-526] FilterExpressionPro...
GitHub user rahulforallp opened a pull request: https://github.com/apache/incubator-carbondata/pull/422 [CARBONDATA-526] FilterExpressionProcessorTest added You can merge this pull request into a Git repository by running: $ git pull https://github.com/rahulforallp/incubator-carbondata CARBONDATA-526 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-carbondata/pull/422.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #422 commit c3871f7cdaecc33459568ca61f28bf5e7b35875b Author: rahulforallp Date: 2016-12-12T13:18:59Z FilterExpressionProcessorTest added --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #422: [CARBONDATA-526] FilterExpressionProcessorT...
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/422 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Resolved] (CARBONDATA-524) improve integration test case of AllDataTypesTestCase5
[ https://issues.apache.org/jira/browse/CARBONDATA-524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravindra Pesala resolved CARBONDATA-524. Resolution: Fixed Assignee: Rahul Kumar Fix Version/s: 1.0.0-incubating > improve integration test case of AllDataTypesTestCase5 > -- > > Key: CARBONDATA-524 > URL: https://issues.apache.org/jira/browse/CARBONDATA-524 > Project: CarbonData > Issue Type: Improvement >Reporter: Rahul Kumar >Assignee: Rahul Kumar > Fix For: 1.0.0-incubating > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-carbondata pull request #420: [CARBONDATA 524] AllDataypeTestCase5...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-carbondata/pull/420 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #420: [CARBONDATA 524] AllDataypeTestCase5 improv...
Github user ravipesala commented on the issue: https://github.com/apache/incubator-carbondata/pull/420 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #420: [CARBONDATA 524] AllDataypeTestCase5 improv...
Github user ravipesala commented on the issue: https://github.com/apache/incubator-carbondata/pull/420 @rahulforallp the title name must be the jira issue Id, so it has to be CARBONDATA-524 not CARBONDATA 524. Please follow this rule in future PR's --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Resolved] (CARBONDATA-450) Increase Test Coverage for Core.reader module
[ https://issues.apache.org/jira/browse/CARBONDATA-450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravindra Pesala resolved CARBONDATA-450. Resolution: Fixed Fix Version/s: 1.0.0-incubating > Increase Test Coverage for Core.reader module > - > > Key: CARBONDATA-450 > URL: https://issues.apache.org/jira/browse/CARBONDATA-450 > Project: CarbonData > Issue Type: Test >Reporter: SWATI RAO > Fix For: 1.0.0-incubating > > Time Spent: 1h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-carbondata pull request #355: [CARBONDATA-450] added test cases fo...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-carbondata/pull/355 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #355: [CARBONDATA-450] added test cases for core....
Github user ravipesala commented on the issue: https://github.com/apache/incubator-carbondata/pull/355 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Resolved] (CARBONDATA-413) Implement unit test cases for scan.expression package
[ https://issues.apache.org/jira/browse/CARBONDATA-413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravindra Pesala resolved CARBONDATA-413. Resolution: Fixed Fix Version/s: 1.0.0-incubating > Implement unit test cases for scan.expression package > - > > Key: CARBONDATA-413 > URL: https://issues.apache.org/jira/browse/CARBONDATA-413 > Project: CarbonData > Issue Type: Improvement >Reporter: Anurag Srivastava >Priority: Trivial > Fix For: 1.0.0-incubating > > Time Spent: 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-carbondata pull request #350: [CARBONDATA-413] Unit test case for ...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-carbondata/pull/350 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #350: [CARBONDATA-413] Unit test case for express...
Github user ravipesala commented on the issue: https://github.com/apache/incubator-carbondata/pull/350 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Resolved] (CARBONDATA-438) Add unit test for scan.scanner.impl package
[ https://issues.apache.org/jira/browse/CARBONDATA-438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravindra Pesala resolved CARBONDATA-438. Resolution: Fixed Assignee: Kunal Kapoor Fix Version/s: 1.0.0-incubating > Add unit test for scan.scanner.impl package > --- > > Key: CARBONDATA-438 > URL: https://issues.apache.org/jira/browse/CARBONDATA-438 > Project: CarbonData > Issue Type: Test >Reporter: SWATI RAO >Assignee: Kunal Kapoor >Priority: Trivial > Fix For: 1.0.0-incubating > > Time Spent: 1h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-carbondata pull request #342: [CARBONDATA-438] Added unit test for...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-carbondata/pull/342 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #342: [CARBONDATA-438] Added unit test for Filter...
Github user ravipesala commented on the issue: https://github.com/apache/incubator-carbondata/pull/342 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #419: [CARBONDATA-523] Added test cases for carbo...
Github user ravipesala commented on the issue: https://github.com/apache/incubator-carbondata/pull/419 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #420: [CARBONDATA 524] AllDataypeTestCase5 improv...
Github user rahulforallp commented on the issue: https://github.com/apache/incubator-carbondata/pull/420 @ravipesala title renamed --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #420: [Carbondata 524] AllDataypeTestCase5 improv...
Github user ravipesala commented on the issue: https://github.com/apache/incubator-carbondata/pull/420 @rahulforallp Please rename title from [Carbondata 524] to [CARBONDATA-524] --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (CARBONDATA-526) Write Unit TestCase For org.apache.carbondata.scan.filter.resolver
Rahul Kumar created CARBONDATA-526: -- Summary: Write Unit TestCase For org.apache.carbondata.scan.filter.resolver Key: CARBONDATA-526 URL: https://issues.apache.org/jira/browse/CARBONDATA-526 Project: CarbonData Issue Type: Improvement Reporter: Rahul Kumar -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-carbondata issue #420: [Carbondata 524] AllDataypeTestCase5 improv...
Github user rahulforallp commented on the issue: https://github.com/apache/incubator-carbondata/pull/420 this build fail cause has been already resolved in PR: #421 . --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #421: Timestamp based testcases were fixed for CI...
Github user deepakmehra10 commented on the issue: https://github.com/apache/incubator-carbondata/pull/421 @ravipesala Please review the PR on priority . This PR includes fixed test cases which were failing on Continuous Integration. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #421: Timestamp based testcases were fixed for CI...
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/421 Build Success with Spark 1.5.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/122/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #421: Timestamp based testcases were fixed...
GitHub user deepakmehra10 opened a pull request: https://github.com/apache/incubator-carbondata/pull/421 Timestamp based testcases were fixed for CI's You can merge this pull request into a Git repository by running: $ git pull https://github.com/deepakmehra10/incubator-carbondata CARBONDATA-525 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-carbondata/pull/421.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #421 commit cec17c404e239eb6a5cb6821b86313d3b1d73e38 Author: deepakmehra10 Date: 2016-12-12T10:29:27Z timstamp based testcases were fixed for CI's --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #419: [CARBONDATA-523] Added test cases for carbo...
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/419 Build Success with Spark 1.5.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/121/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #419: CARBONDATA-523] Added test cases for carbon...
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/419 Build Failed with Spark 1.5.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/120/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #419: CARBONDATA-523] Added test cases for carbon...
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/419 Build Success with Spark 1.5.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/119/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #420: [Carbondata 524] AllDataypeTestCase5...
GitHub user rahulforallp opened a pull request: https://github.com/apache/incubator-carbondata/pull/420 [Carbondata 524] AllDataypeTestCase5 improved You can merge this pull request into a Git repository by running: $ git pull https://github.com/rahulforallp/incubator-carbondata CARBONDATA-524 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-carbondata/pull/420.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #420 commit 31fbf205ce31a171ae5c875f449f2a04e885ce24 Author: rahulforallp Date: 2016-12-12T08:36:56Z AllDataTypesTestCase5 succeeded --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (CARBONDATA-525) Fix timestamp based test cases
Prabhat Kashyap created CARBONDATA-525: -- Summary: Fix timestamp based test cases Key: CARBONDATA-525 URL: https://issues.apache.org/jira/browse/CARBONDATA-525 Project: CarbonData Issue Type: Test Reporter: Prabhat Kashyap Priority: Trivial -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-carbondata issue #420: [Carbondata 524] AllDataypeTestCase5 improv...
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/420 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #331: [CARBONDATA-396] Implement test cases for d...
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/331 Build Success with Spark 1.5.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/118/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (CARBONDATA-524) improve integration test case of AllDataTypesTestCase5
Rahul Kumar created CARBONDATA-524: -- Summary: improve integration test case of AllDataTypesTestCase5 Key: CARBONDATA-524 URL: https://issues.apache.org/jira/browse/CARBONDATA-524 Project: CarbonData Issue Type: Improvement Reporter: Rahul Kumar -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-carbondata issue #350: [CARBONDATA-413] Unit test case for express...
Github user anuragknoldus commented on the issue: https://github.com/apache/incubator-carbondata/pull/350 @ravipesala , I have renamed title. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #:
Github user allwefantasy commented on the pull request: https://github.com/apache/incubator-carbondata/commit/6c9194d97c54351434866f423ef44907b887ae5a#commitcomment-20151464 In integration/spark/src/main/scala/org/apache/spark/sql/execution/command/carbonTableSchema.scala: In integration/spark/src/main/scala/org/apache/spark/sql/execution/command/carbonTableSchema.scala on line 439: Ok, I will fix this later . --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
