[jira] [Commented] (CARBONDATA-129) Do null check before adding value to CarbonProperties
[ https://issues.apache.org/jira/browse/CARBONDATA-129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15418765#comment-15418765 ] Praveen Adlakha commented on CARBONDATA-129: Hi , I am not sure whether you are talking about null check for CarbonProperties or the value we are adding. Had a look at the code CarbonPropeties is a singelton class so cannot be null for values that we are adding already null check is there however there was no empty check have added it in : https://github.com/apache/incubator-carbondata/pull/73 > Do null check before adding value to CarbonProperties > - > > Key: CARBONDATA-129 > URL: https://issues.apache.org/jira/browse/CARBONDATA-129 > Project: CarbonData > Issue Type: Bug > Components: spark >Reporter: Ashok Kumar >Priority: Critical > > CarbonGlobalDictionaryRDD > We are adding value to CarbonProperties without check if its null. > If added value is null then error will come. Its better to do null check > before adding value to CarbonProperties. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CARBONDATA-129) Do null check before adding value to CarbonProperties
[ https://issues.apache.org/jira/browse/CARBONDATA-129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15418764#comment-15418764 ] ASF GitHub Bot commented on CARBONDATA-129: --- GitHub user PraveenAdlakha opened a pull request: https://github.com/apache/incubator-carbondata/pull/73 CARBONDATA-129 Do null check before adding value to CarbonProperties You can merge this pull request into a Git repository by running: $ git pull https://github.com/PraveenAdlakha/incubator-carbondata 129 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-carbondata/pull/73.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #73 commit 615d718ab93c8d3a181bc03a6daac311c7484d2c Author: Praveen Adlakha Date: 2016-08-12T12:41:38Z CARBONDATA-129 Do null check before adding value to CarbonProperties > Do null check before adding value to CarbonProperties > - > > Key: CARBONDATA-129 > URL: https://issues.apache.org/jira/browse/CARBONDATA-129 > Project: CarbonData > Issue Type: Bug > Components: spark >Reporter: Ashok Kumar >Priority: Critical > > CarbonGlobalDictionaryRDD > We are adding value to CarbonProperties without check if its null. > If added value is null then error will come. Its better to do null check > before adding value to CarbonProperties. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-carbondata pull request #73: CARBONDATA-129 Do null check before a...
GitHub user PraveenAdlakha opened a pull request: https://github.com/apache/incubator-carbondata/pull/73 CARBONDATA-129 Do null check before adding value to CarbonProperties You can merge this pull request into a Git repository by running: $ git pull https://github.com/PraveenAdlakha/incubator-carbondata 129 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-carbondata/pull/73.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #73 commit 615d718ab93c8d3a181bc03a6daac311c7484d2c Author: Praveen Adlakha Date: 2016-08-12T12:41:38Z CARBONDATA-129 Do null check before adding value to CarbonProperties --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (CARBONDATA-154) Block prune can not get the right blocks and query result is wrong
[
https://issues.apache.org/jira/browse/CARBONDATA-154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15418501#comment-15418501
]
ASF GitHub Bot commented on CARBONDATA-154:
---
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/70#discussion_r74554574
--- Diff: core/src/main/java/org/carbondata/scan/filter/FilterUtil.java ---
@@ -711,7 +698,7 @@ public static DimColumnFilterInfo
getFilterListForAllMembersRS(Expression expres
*/
public static void getStartKeyForNoDictionaryDimension(
DimColumnResolvedFilterInfo dimColResolvedFilterInfo,
SegmentProperties segmentProperties,
- SortedMap setOfStartKeyByteArray) {
+ SortedMap setOfStartKeyByteArray, List
startKeyList) {
--- End diff --
No use of `segmentProperties` and `startKeyList`, please remove it.
> Block prune can not get the right blocks and query result is wrong
> --
>
> Key: CARBONDATA-154
> URL: https://issues.apache.org/jira/browse/CARBONDATA-154
> Project: CarbonData
> Issue Type: Bug
>Reporter: zhangshunyu
>Priority: Critical
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] incubator-carbondata pull request #70: [CARBONDATA-154] Fix the bug of block...
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/70#discussion_r74554655
--- Diff: core/src/main/java/org/carbondata/scan/filter/FilterUtil.java ---
@@ -766,7 +753,7 @@ public static void getStartKeyForNoDictionaryDimension(
*/
public static void getEndKeyForNoDictionaryDimension(
DimColumnResolvedFilterInfo dimColResolvedFilterInfo,
SegmentProperties segmentProperties,
- SortedMap setOfEndKeyByteArray) {
+ SortedMap setOfEndKeyByteArray, List
endKeyList) {
--- End diff --
Please remove `segmentProperties` and `endKeyList`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #72: Fixed wrong result and random query f...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-carbondata/pull/72 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (CARBONDATA-154) Block prune can not get the right blocks and query result is wrong
[
https://issues.apache.org/jira/browse/CARBONDATA-154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15418502#comment-15418502
]
ASF GitHub Bot commented on CARBONDATA-154:
---
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/70#discussion_r74554655
--- Diff: core/src/main/java/org/carbondata/scan/filter/FilterUtil.java ---
@@ -766,7 +753,7 @@ public static void getStartKeyForNoDictionaryDimension(
*/
public static void getEndKeyForNoDictionaryDimension(
DimColumnResolvedFilterInfo dimColResolvedFilterInfo,
SegmentProperties segmentProperties,
- SortedMap setOfEndKeyByteArray) {
+ SortedMap setOfEndKeyByteArray, List
endKeyList) {
--- End diff --
Please remove `segmentProperties` and `endKeyList`
> Block prune can not get the right blocks and query result is wrong
> --
>
> Key: CARBONDATA-154
> URL: https://issues.apache.org/jira/browse/CARBONDATA-154
> Project: CarbonData
> Issue Type: Bug
>Reporter: zhangshunyu
>Priority: Critical
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] incubator-carbondata pull request #72: Fixed wrong result and random query f...
GitHub user ravipesala opened a pull request: https://github.com/apache/incubator-carbondata/pull/72 Fixed wrong result and random query fails. Query iteration has some problem and it reads wrong data when the data is huge. And also this PR fixes the join queries with tables other than carbon. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ravipesala/incubator-carbondata critical-bug Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-carbondata/pull/72.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #72 commit 5265b618ed31eeef95de9f140624063b2e3bddb8 Author: ravipesala Date: 2016-08-11T13:14:55Z Fixed query fails on bigdata issue commit e3bd0e2df0aa62e61961e2086adcfe902eb4eb09 Author: ravipesala Date: 2016-08-11T16:29:55Z Fixed query fails on bigdata issue2 commit 65182cc44b9ef109ed72eddcd85ba7c38a8087aa Author: ravipesala Date: 2016-08-11T18:40:01Z Fixed query fails on bigdata issue3 commit 35409fde502f917f4396b6b99c1aa7b67cb83c31 Author: ravipesala Date: 2016-08-12T06:34:01Z Fixed join with carbon and other table --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
Re: load data fail
HiAs we discussed, the error of "Table is locked for updation. Please try after some time " has been solved through setting directory rights.The below is new error, please Ravindra check and provide helps :WARN 12-08 16:29:51,871 - Lost task 1.1 in stage 2.0 (TID 6, hadoop03): java.lang.RuntimeException: Dictionary file name is locked for updation. Please try after some timeat scala.sys.package$.error(package.scala:27)at org.carbondata.spark.rdd.CarbonGlobalDictionaryGenerateRDD$$anon$1.(CarbonGlobalDictionaryRDD.scala:354) at org.carbondata.spark.rdd.CarbonGlobalDictionaryGenerateRDD.compute(CarbonGlobalDictionaryRDD.scala:295) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)at org.apache.spark.scheduler.Task.run(Task.scala:89)at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) -- View this message in context: http://apache-carbondata-mailing-list-archive.1130556.n5.nabble.com/load-data-fail-tp100p104.html Sent from the Apache CarbonData Mailing List archive mailing list archive at Nabble.com.
Fwd: load data fail
-- Forwarded message --
From: Ravindra Pesala
Date: 12 August 2016 at 12:45
Subject: Re: load data fail
To: dev
Hi,
Are you getting this exception continuously for every load? Usually it
occurs when you try to load the data concurrently to the same table. So
please make sure that no other instance of carbon is running and data load
on the same table is not happening.
Check if any locks are created under system temp folder with
//lockfile, if it exists please delete.
Thanks & Regards,
Ravi
On 12 August 2016 at 11:55, 金铸 wrote:
> hi ：
> /usr/hdp/2.4.0.0-169/spark/bin/spark-shell --master yarn-client --jars
> /opt/incubator-carbondata/assembly/target/scala-2.10/carbond
> ata_2.10-0.1.0-incubating-SNAPSHOT-shade-hadoop2.2.0.jar,/
> usr/hdp/2.4.0.0-169/spark/lib/datanucleus-api-jdo-3.2.6.jar,
> /usr/hdp/2.4.0.0-169/spark/lib/datanucleus-rdbms-3.2.9.
> jar,/usr/hdp/2.4.0.0-169/spark/lib/datanucleus-core-3.
> 2.10.jar,/opt//mysql-connector-java-5.1.37.jar
>
> scala>import org.apache.spark.sql.CarbonContext
> scala>import java.io.File
> scala>import org.apache.hadoop.hive.conf.HiveConf
>
>
>
>
>
> scala>val cc = new CarbonContext(sc, "hdfs://hadoop01/data/carbonda
> ta01/store")
>
> scala>cc.setConf("hive.metastore.warehouse.dir", "/apps/hive/warehouse")
> scala>cc.setConf(HiveConf.ConfVars.HIVECHECKFILEFORMAT.varname, "false")
> scala>cc.setConf("carbon.kettle.home","/usr/hdp/2.4.0.0-169/
> spark/carbonlib/carbonplugins")
>
> scala> cc.setConf("carbon.kettle.home","/usr/hdp/2.4.0.0-169/spark/
> carbonlib/carbonplugins")
>
> scala> cc.sql(s"load data local inpath 'hdfs://hadoop01/sample.csv' into
> table t4 options('FILEHEADER'='id,name,city,age')")
> INFO 12-08 14:21:24,461 - main Query [LOAD DATA LOCAL INPATH
> 'HDFS://HADOOP01/SAMPLE.CSV' INTO TABLE T4 OPTIONS('FILEHEADER'='ID,NAME,
> CITY,AGE')]
> INFO 12-08 14:21:39,475 - Table MetaData Unlocked Successfully after data
> load
> java.lang.RuntimeException: Table is locked for updation. Please try after
> some time
> at scala.sys.package$.error(package.scala:27)
> at org.apache.spark.sql.execution.command.LoadTable.run(carbonT
> ableSchema.scala:1049)
> at org.apache.spark.sql.execution.ExecutedCommand.sideEffectRes
> ult$lzycompute(commands.scala:58)
> at org.apache.spark.sql.execution.ExecutedCommand.sideEffectRes
> ult(commands.scala:56)
> at org.apache.spark.sql.execution.ExecutedCommand.doExecute(com
> mands.scala:70)
> at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.
> apply(SparkPlan.scala:132)
> at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.
> apply(SparkPlan.scala:130)
> at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperati
> onScope.scala:150)
>
> thanks a lot
>
>
>
> ---
> Confidentiality Notice: The information contained in this e-mail and any
> accompanying attachment(s)
> is intended only for the use of the intended recipient and may be
> confidential and/or privileged of
> Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader
> of this communication is
> not the intended recipient, unauthorized use, forwarding, printing,
> storing, disclosure or copying
> is strictly prohibited, and may be unlawful.If you have received this
> communication in error,please
> immediately notify the sender by return e-mail, and delete the original
> message and all copies from
> your system. Thank you.
>
> ---
>
--
Thanks & Regards,
Ravi
--
Thanks & Regards,
Ravi
[GitHub] incubator-carbondata pull request #70: [CARBONDATA-154] Fix the bug of block...
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/70#discussion_r74554574
--- Diff: core/src/main/java/org/carbondata/scan/filter/FilterUtil.java ---
@@ -711,7 +698,7 @@ public static DimColumnFilterInfo
getFilterListForAllMembersRS(Expression expres
*/
public static void getStartKeyForNoDictionaryDimension(
DimColumnResolvedFilterInfo dimColResolvedFilterInfo,
SegmentProperties segmentProperties,
- SortedMap setOfStartKeyByteArray) {
+ SortedMap setOfStartKeyByteArray, List
startKeyList) {
--- End diff --
No use of `segmentProperties` and `startKeyList`, please remove it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #70: [CARBONDATA-154] Fix the bug of block...
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/70#discussion_r74554497
--- Diff: core/src/main/java/org/carbondata/scan/filter/FilterUtil.java ---
@@ -669,27 +669,14 @@ public static DimColumnFilterInfo
getFilterListForAllMembersRS(Expression expres
* @param segmentProperties
* @return long[] start key
*/
- public static long[] getStartKey(DimColumnResolvedFilterInfo
dimColResolvedFilterInfo,
- SegmentProperties segmentProperties, long[] startKey) {
-Map> dimensionFilter =
-dimColResolvedFilterInfo.getDimensionResolvedFilterInstance();
-for (Map.Entry> entry :
dimensionFilter.entrySet()) {
- List values = entry.getValue();
- if (null == values ||
!entry.getKey().hasEncoding(Encoding.DICTIONARY)) {
-continue;
- }
- boolean isExcludePresent = false;
- for (DimColumnFilterInfo info : values) {
-if (!info.isIncludeFilter()) {
- isExcludePresent = true;
-}
- }
- if (isExcludePresent) {
-continue;
- }
- getStartKeyBasedOnFilterResoverInfo(dimensionFilter, startKey);
+ public static void getStartKey(Map> dimensionFilter,
+ AbsoluteTableIdentifier tableIdentifier, long[] startKey,
SegmentProperties segmentProperties,
--- End diff --
no use of `tableIdentifier` and `segmentProperties`, please remove from
static method.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (CARBONDATA-154) Block prune can not get the right blocks and query result is wrong
[
https://issues.apache.org/jira/browse/CARBONDATA-154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15418499#comment-15418499
]
ASF GitHub Bot commented on CARBONDATA-154:
---
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/70#discussion_r74554497
--- Diff: core/src/main/java/org/carbondata/scan/filter/FilterUtil.java ---
@@ -669,27 +669,14 @@ public static DimColumnFilterInfo
getFilterListForAllMembersRS(Expression expres
* @param segmentProperties
* @return long[] start key
*/
- public static long[] getStartKey(DimColumnResolvedFilterInfo
dimColResolvedFilterInfo,
- SegmentProperties segmentProperties, long[] startKey) {
-Map> dimensionFilter =
-dimColResolvedFilterInfo.getDimensionResolvedFilterInstance();
-for (Map.Entry> entry :
dimensionFilter.entrySet()) {
- List values = entry.getValue();
- if (null == values ||
!entry.getKey().hasEncoding(Encoding.DICTIONARY)) {
-continue;
- }
- boolean isExcludePresent = false;
- for (DimColumnFilterInfo info : values) {
-if (!info.isIncludeFilter()) {
- isExcludePresent = true;
-}
- }
- if (isExcludePresent) {
-continue;
- }
- getStartKeyBasedOnFilterResoverInfo(dimensionFilter, startKey);
+ public static void getStartKey(Map> dimensionFilter,
+ AbsoluteTableIdentifier tableIdentifier, long[] startKey,
SegmentProperties segmentProperties,
--- End diff --
no use of `tableIdentifier` and `segmentProperties`, please remove from
static method.
> Block prune can not get the right blocks and query result is wrong
> --
>
> Key: CARBONDATA-154
> URL: https://issues.apache.org/jira/browse/CARBONDATA-154
> Project: CarbonData
> Issue Type: Bug
>Reporter: zhangshunyu
>Priority: Critical
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
