[jira] [Created] (CARBONDATA-248) There was no header in driver statistics table and scan block time was always zero
Akash R Nilugal created CARBONDATA-248: -- Summary: There was no header in driver statistics table and scan block time was always zero Key: CARBONDATA-248 URL: https://issues.apache.org/jira/browse/CARBONDATA-248 Project: CarbonData Issue Type: Bug Components: core Reporter: Akash R Nilugal Priority: Minor Column header of total query cost was missing in driver statistics table while recording statistics and scan block time was always zero while recording query statistics. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (CARBONDATA-236) maven compile warning
[
https://issues.apache.org/jira/browse/CARBONDATA-236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ChenLiang resolved CARBONDATA-236.
--
Resolution: Fixed
> maven compile warning
> --
>
> Key: CARBONDATA-236
> URL: https://issues.apache.org/jira/browse/CARBONDATA-236
> Project: CarbonData
> Issue Type: Wish
>Reporter: shijinkui
>Assignee: shijinkui
>
> 1. maven-shade-plugin omit version
> 2. [WARNING] 'dependencies.dependency.(groupId:artifactId:type:classifier)'
> must be unique: org.apache.carbondata:carbondata-format:jar -> duplicate
> declaration of version ${project.version} @ line 99, column 17
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CARBONDATA-236) maven compile warning
[
https://issues.apache.org/jira/browse/CARBONDATA-236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ChenLiang updated CARBONDATA-236:
-
Assignee: shijinkui
> maven compile warning
> --
>
> Key: CARBONDATA-236
> URL: https://issues.apache.org/jira/browse/CARBONDATA-236
> Project: CarbonData
> Issue Type: Wish
>Reporter: shijinkui
>Assignee: shijinkui
>
> 1. maven-shade-plugin omit version
> 2. [WARNING] 'dependencies.dependency.(groupId:artifactId:type:classifier)'
> must be unique: org.apache.carbondata:carbondata-format:jar -> duplicate
> declaration of version ${project.version} @ line 99, column 17
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (CARBONDATA-231) Rename repeared table names in same test file and add drop tables.
[ https://issues.apache.org/jira/browse/CARBONDATA-231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ChenLiang resolved CARBONDATA-231. -- Resolution: Fixed > Rename repeared table names in same test file and add drop tables. > -- > > Key: CARBONDATA-231 > URL: https://issues.apache.org/jira/browse/CARBONDATA-231 > Project: CarbonData > Issue Type: Improvement >Affects Versions: 0.2.0-incubating, 0.1.1-incubating >Reporter: zhangshunyu >Assignee: zhangshunyu > Fix For: 0.2.0-incubating, 0.1.1-incubating > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CARBONDATA-231) Rename repeared table names in same test file and add drop tables.
[ https://issues.apache.org/jira/browse/CARBONDATA-231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ChenLiang updated CARBONDATA-231: - Affects Version/s: 0.2.0-incubating > Rename repeared table names in same test file and add drop tables. > -- > > Key: CARBONDATA-231 > URL: https://issues.apache.org/jira/browse/CARBONDATA-231 > Project: CarbonData > Issue Type: Improvement >Affects Versions: 0.2.0-incubating, 0.1.1-incubating >Reporter: zhangshunyu >Assignee: zhangshunyu > Fix For: 0.2.0-incubating, 0.1.1-incubating > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CARBONDATA-231) Rename repeared table names in same test file and add drop tables.
[ https://issues.apache.org/jira/browse/CARBONDATA-231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ChenLiang updated CARBONDATA-231: - Assignee: zhangshunyu > Rename repeared table names in same test file and add drop tables. > -- > > Key: CARBONDATA-231 > URL: https://issues.apache.org/jira/browse/CARBONDATA-231 > Project: CarbonData > Issue Type: Improvement >Affects Versions: 0.2.0-incubating, 0.1.1-incubating >Reporter: zhangshunyu >Assignee: zhangshunyu > Fix For: 0.2.0-incubating, 0.1.1-incubating > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CARBONDATA-231) Rename repeared table names in same test file and add drop tables.
[ https://issues.apache.org/jira/browse/CARBONDATA-231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ChenLiang updated CARBONDATA-231: - Fix Version/s: 0.1.1-incubating 0.2.0-incubating > Rename repeared table names in same test file and add drop tables. > -- > > Key: CARBONDATA-231 > URL: https://issues.apache.org/jira/browse/CARBONDATA-231 > Project: CarbonData > Issue Type: Improvement >Affects Versions: 0.2.0-incubating, 0.1.1-incubating >Reporter: zhangshunyu >Assignee: zhangshunyu > Fix For: 0.2.0-incubating, 0.1.1-incubating > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CARBONDATA-231) Rename repeared table names in same test file and add drop tables.
[ https://issues.apache.org/jira/browse/CARBONDATA-231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ChenLiang updated CARBONDATA-231: - Affects Version/s: 0.1.1-incubating > Rename repeared table names in same test file and add drop tables. > -- > > Key: CARBONDATA-231 > URL: https://issues.apache.org/jira/browse/CARBONDATA-231 > Project: CarbonData > Issue Type: Improvement >Affects Versions: 0.2.0-incubating, 0.1.1-incubating >Reporter: zhangshunyu >Assignee: zhangshunyu > Fix For: 0.2.0-incubating, 0.1.1-incubating > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (CARBONDATA-229) Array Index of bound exception thrown from dictionary look up while writing sort index file
[ https://issues.apache.org/jira/browse/CARBONDATA-229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ChenLiang resolved CARBONDATA-229. -- Resolution: Fixed > Array Index of bound exception thrown from dictionary look up while writing > sort index file > --- > > Key: CARBONDATA-229 > URL: https://issues.apache.org/jira/browse/CARBONDATA-229 > Project: CarbonData > Issue Type: Bug >Reporter: Manish Gupta >Assignee: Manish Gupta > > Whenever we load dictionary data into memory, then in case of populating > reverse dictionary object sometimes a chunk which has no value is also > getting added to the dictionary chunk list. This is happening because the > logic for dictionary chunk distribution in case of forward dictionary is not > implemented for reverse dictionary and 0 size dictionary chunks are not > getting removed while adding to the list of dictionary chunks. > java.lang.IndexOutOfBoundsException: Index: 0, Size: 0 > at java.util.ArrayList.rangeCheck(ArrayList.java:653) > at java.util.ArrayList.get(ArrayList.java:429) > at > org.apache.carbondata.core.cache.dictionary.DictionaryChunksWrapper.next(DictionaryChunksWrapper.java:92) > at > org.apache.carbondata.core.writer.sortindex.CarbonDictionarySortInfoPreparator.prepareDictionarySortModels(CarbonDictionarySortInfoPreparator.java:120) > at > org.apache.carbondata.core.writer.sortindex.CarbonDictionarySortInfoPreparator.getDictionarySortInfo(CarbonDictionarySortInfoPreparator.java:51) > at > org.apache.carbondata.spark.tasks.SortIndexWriterTask.execute(SortIndexWriterTask.scala:44) > at > org.apache.carbondata.spark.rdd.CarbonGlobalDictionaryGenerateRDD$$anon$1.(CarbonGlobalDictionaryRDD.scala:387) > at > org.apache.carbondata.spark.rdd.CarbonGlobalDictionaryGenerateRDD.compute(CarbonGlobalDictionaryRDD.scala:294) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (CARBONDATA-228) Describe formatted command shows unsupported features
[ https://issues.apache.org/jira/browse/CARBONDATA-228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ChenLiang resolved CARBONDATA-228. -- Resolution: Fixed > Describe formatted command shows unsupported features > - > > Key: CARBONDATA-228 > URL: https://issues.apache.org/jira/browse/CARBONDATA-228 > Project: CarbonData > Issue Type: Bug >Reporter: Manohar Vanam >Assignee: Manohar Vanam > > Remove unsupported features from Describe formatted command description -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (CARBONDATA-227) In block distribution parralelism is decided initially and not re initialized after requesting new executors
[ https://issues.apache.org/jira/browse/CARBONDATA-227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ChenLiang resolved CARBONDATA-227. -- Resolution: Fixed > In block distribution parralelism is decided initially and not re initialized > after requesting new executors > > > Key: CARBONDATA-227 > URL: https://issues.apache.org/jira/browse/CARBONDATA-227 > Project: CarbonData > Issue Type: Bug >Reporter: Mohammad Shahid Khan >Assignee: Mohammad Shahid Khan > > In block distribution parralelism is decided initially and not re initialized > after requesting new executors. > Due to this task per node initialization is getting wrong. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (CARBONDATA-224) Fixed data mismatch issue in case of Dictionary Exclude column for Numeric data type
[ https://issues.apache.org/jira/browse/CARBONDATA-224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ChenLiang resolved CARBONDATA-224. -- Resolution: Fixed > Fixed data mismatch issue in case of Dictionary Exclude column for Numeric > data type > > > Key: CARBONDATA-224 > URL: https://issues.apache.org/jira/browse/CARBONDATA-224 > Project: CarbonData > Issue Type: Bug >Reporter: kumar vishal >Assignee: kumar vishal > > Problem: In case of greater than query on dictionary exclude column of for > numeric data type > This is because data is sorted based on string because of this if data is > 1,10,2,3 , data will be sorted like 1,10,2,3 but if we search greater than 3 > then while applying min max will return false as last value is 3 > Solution:we need to sort based on actual data type for this we should have > chain comparator based on data type while loading the data, currently > disabling DictionaryExclude column for numeric data type and will throw > exception. Will raise jira issue to for sorting the based on actual data type -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CARBONDATA-245) Actual Exception is getting lost in case of data dictionary file generation.
[
https://issues.apache.org/jira/browse/CARBONDATA-245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15498890#comment-15498890
]
ASF GitHub Bot commented on CARBONDATA-245:
---
Github user vinodkc commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/160#discussion_r79282613
--- Diff:
core/src/main/java/org/apache/carbondata/core/writer/CarbonDictionaryWriterImpl.java
---
@@ -199,7 +199,10 @@ public CarbonDictionaryWriterImpl(String hdfsStorePath,
*/
@Override public void close() throws IOException {
if (null != dictionaryThriftWriter) {
- writeDictionaryFile();
+ // if stream is open then only need to write dictionary file.
+ if(dictionaryThriftWriter.isOpen()){
--- End diff --
Combine both if conditions
> Actual Exception is getting lost in case of data dictionary file generation.
>
>
> Key: CARBONDATA-245
> URL: https://issues.apache.org/jira/browse/CARBONDATA-245
> Project: CarbonData
> Issue Type: Bug
> Components: data-load
>Reporter: ravikiran
>Priority: Minor
>
> Actual Exception is getting lost in case of data dictionary file
> generation.getting the null pointer exception as in the finally block we are
> trying to write data to a null stream.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CARBONDATA-245) Actual Exception is getting lost in case of data dictionary file generation.
[
https://issues.apache.org/jira/browse/CARBONDATA-245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15498891#comment-15498891
]
ASF GitHub Bot commented on CARBONDATA-245:
---
Github user vinodkc commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/160#discussion_r79282603
--- Diff:
core/src/main/java/org/apache/carbondata/core/writer/ThriftWriter.java ---
@@ -80,6 +80,17 @@ public void open() throws IOException {
}
/**
+ * This will check whether stream and binary out is open or not.
+ * @return
+ */
+ public boolean isOpen() {
+if (null != binaryOut && null != dataOutputStream) {
--- End diff --
need to set binaryOut and dataOutputStream to null in close() method
> Actual Exception is getting lost in case of data dictionary file generation.
>
>
> Key: CARBONDATA-245
> URL: https://issues.apache.org/jira/browse/CARBONDATA-245
> Project: CarbonData
> Issue Type: Bug
> Components: data-load
>Reporter: ravikiran
>Priority: Minor
>
> Actual Exception is getting lost in case of data dictionary file
> generation.getting the null pointer exception as in the finally block we are
> trying to write data to a null stream.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CARBONDATA-246) compaction is wrong in case if last segment is not assigned to an executor.
[
https://issues.apache.org/jira/browse/CARBONDATA-246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15498922#comment-15498922
]
ASF GitHub Bot commented on CARBONDATA-246:
---
Github user manishgupta88 commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/161#discussion_r79282917
--- Diff:
integration/spark/src/main/scala/org/apache/carbondata/spark/rdd/Compactor.scala
---
@@ -69,7 +69,9 @@ object Compactor {
schemaName,
factTableName,
validSegments,
-
carbonTable.getAbsoluteTableIdentifier.getCarbonTableIdentifier.getTableId
+
carbonTable.getAbsoluteTableIdentifier.getCarbonTableIdentifier.getTableId,
+ colCardinality = Array[Int](0),
--- End diff --
Instead of creating an empty list and reassigning it you can create a
reference of Array[Int] type like
var colCardinality: Array[Int] = null
> compaction is wrong in case if last segment is not assigned to an executor.
>
>
> Key: CARBONDATA-246
> URL: https://issues.apache.org/jira/browse/CARBONDATA-246
> Project: CarbonData
> Issue Type: Bug
>Reporter: ravikiran
>
> if during compaction of 4 loads, for any executor if only first 3 loads task
> is assigned then the col cardinality calculation based on the last segment
> info will become wrong.
> in this case the cardinality will go wrong for that executor.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (CARBONDATA-249) As LongType.simpleString in spark is "bigint", Carbon will convert Long to BigInt
Naresh P R created CARBONDATA-249: - Summary: As LongType.simpleString in spark is "bigint", Carbon will convert Long to BigInt Key: CARBONDATA-249 URL: https://issues.apache.org/jira/browse/CARBONDATA-249 Project: CarbonData Issue Type: Improvement Reporter: Naresh P R Priority: Minor Describe command will show DataType.simpleString in datatype column, for LongType Spark DataType, simpleString is "bigint". We are internally using long as name for bigint, which needs to be changed to bigint in Carbon DataTypes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CARBONDATA-246) compaction is wrong in case if last segment is not assigned to an executor.
[
https://issues.apache.org/jira/browse/CARBONDATA-246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15499015#comment-15499015
]
ASF GitHub Bot commented on CARBONDATA-246:
---
Github user ravikiran23 commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/161#discussion_r79283767
--- Diff:
integration/spark/src/main/scala/org/apache/carbondata/spark/rdd/Compactor.scala
---
@@ -69,7 +69,9 @@ object Compactor {
schemaName,
factTableName,
validSegments,
-
carbonTable.getAbsoluteTableIdentifier.getCarbonTableIdentifier.getTableId
+
carbonTable.getAbsoluteTableIdentifier.getCarbonTableIdentifier.getTableId,
+ colCardinality = Array[Int](0),
--- End diff --
fixed. using the null.
> compaction is wrong in case if last segment is not assigned to an executor.
>
>
> Key: CARBONDATA-246
> URL: https://issues.apache.org/jira/browse/CARBONDATA-246
> Project: CarbonData
> Issue Type: Bug
>Reporter: ravikiran
>
> if during compaction of 4 loads, for any executor if only first 3 loads task
> is assigned then the col cardinality calculation based on the last segment
> info will become wrong.
> in this case the cardinality will go wrong for that executor.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CARBONDATA-239) Failure of one compaction in queue should not affect the others.
[
https://issues.apache.org/jira/browse/CARBONDATA-239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15499019#comment-15499019
]
ASF GitHub Bot commented on CARBONDATA-239:
---
Github user ravikiran23 commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/155#discussion_r79283801
--- Diff:
integration/spark/src/main/scala/org/apache/carbondata/spark/rdd/CarbonDataRDDFactory.scala
---
@@ -590,16 +605,25 @@ object CarbonDataRDDFactory extends Logging {
executor, sqlContext, kettleHomePath, storeLocation
)
}
+catch {
+ case e: Exception =>
+logger.error("Exception in compaction thread for table
" + tableForCompaction
+ .carbonTable.getDatabaseName + "." +
+ tableForCompaction.carbonTableIdentifier
+ .getTableName)
+ // not handling the exception. only logging as this is
not the table triggered
+ // by user.
+}
finally {
- // delete the compaction required file
+ // delete the compaction required file in case of
failure or success also.
--- End diff --
fixed
> Failure of one compaction in queue should not affect the others.
>
>
> Key: CARBONDATA-239
> URL: https://issues.apache.org/jira/browse/CARBONDATA-239
> Project: CarbonData
> Issue Type: Bug
>Reporter: ravikiran
>
> Failure of one compaction in queue should not affect the others.
> If a compaction is triggered by the user on table1 , and other requests will
> go to queue. and if the compaction is failed for table1 then the requests in
> queue should continue and at the end the beeline will show the failure
> message to the user.
> if any compaction gets failed for a table which is other than the user
> requested table then the error in the beeline should not appear.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CARBONDATA-250) Throw exception and fail the data load if provided MAXCOLUMNS value is not proper
[ https://issues.apache.org/jira/browse/CARBONDATA-250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manohar Vanam updated CARBONDATA-250: - Description: If the provided MAXCOLUMNS value in load query is not proper, then throw exception and fail the data load. Impact Area : Load query with maxcolumns option value was: If the provided MAXCOLUMNS value in load query is not proper, then throw exception and fail the data load. Impact Area : Load with maxcolumns option value > Throw exception and fail the data load if provided MAXCOLUMNS value is not > proper > - > > Key: CARBONDATA-250 > URL: https://issues.apache.org/jira/browse/CARBONDATA-250 > Project: CarbonData > Issue Type: Bug >Reporter: Manohar Vanam >Assignee: Manohar Vanam > > If the provided MAXCOLUMNS value in load query is not proper, then throw > exception and fail the data load. > Impact Area : Load query with maxcolumns option value -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CARBONDATA-250) Throw exception and fail the data load if provided MAXCOLUMNS value is not proper
Manohar Vanam created CARBONDATA-250: Summary: Throw exception and fail the data load if provided MAXCOLUMNS value is not proper Key: CARBONDATA-250 URL: https://issues.apache.org/jira/browse/CARBONDATA-250 Project: CarbonData Issue Type: Bug Reporter: Manohar Vanam Assignee: Manohar Vanam If the provided MAXCOLUMNS value in load query is not proper, then throw exception and fail the data load. Impact Area : Load with maxcolumns option value -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CARBONDATA-250) Throw exception and fail the data load if provided MAXCOLUMNS value is not proper
[ https://issues.apache.org/jira/browse/CARBONDATA-250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15499083#comment-15499083 ] ASF GitHub Bot commented on CARBONDATA-250: --- GitHub user ManoharVanam opened a pull request: https://github.com/apache/incubator-carbondata/pull/167 [CARBONDATA-250] Throw exception and fail the data load if provided MAXCOLUMNS value is not proper [Problem] : If user executes data load query with improper maxcolumns value, data load is showing success instead of error in the console [Solution] : Throw exception and fail the data load if provided MAXCOLUMNS value is not proper. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ManoharVanam/incubator-carbondata MAXCOLUMN Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-carbondata/pull/167.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #167 commit 650f194050835a28b8b450d12df5f0ba98a7790b Author: Manohar Date: 2016-09-17T13:47:58Z Throwing exception in case of improper MAXCOULUMN value > Throw exception and fail the data load if provided MAXCOLUMNS value is not > proper > - > > Key: CARBONDATA-250 > URL: https://issues.apache.org/jira/browse/CARBONDATA-250 > Project: CarbonData > Issue Type: Bug >Reporter: Manohar Vanam >Assignee: Manohar Vanam > > If the provided MAXCOLUMNS value in load query is not proper, then throw > exception and fail the data load. > Impact Area : Load query with maxcolumns option value -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CARBONDATA-245) Actual Exception is getting lost in case of data dictionary file generation.
[
https://issues.apache.org/jira/browse/CARBONDATA-245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15499119#comment-15499119
]
ASF GitHub Bot commented on CARBONDATA-245:
---
Github user ravikiran23 commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/160#discussion_r79284253
--- Diff:
core/src/main/java/org/apache/carbondata/core/writer/CarbonDictionaryWriterImpl.java
---
@@ -199,7 +199,10 @@ public CarbonDictionaryWriterImpl(String hdfsStorePath,
*/
@Override public void close() throws IOException {
if (null != dictionaryThriftWriter) {
- writeDictionaryFile();
+ // if stream is open then only need to write dictionary file.
+ if(dictionaryThriftWriter.isOpen()){
--- End diff --
fixed
> Actual Exception is getting lost in case of data dictionary file generation.
>
>
> Key: CARBONDATA-245
> URL: https://issues.apache.org/jira/browse/CARBONDATA-245
> Project: CarbonData
> Issue Type: Bug
> Components: data-load
>Reporter: ravikiran
>Priority: Minor
>
> Actual Exception is getting lost in case of data dictionary file
> generation.getting the null pointer exception as in the finally block we are
> trying to write data to a null stream.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CARBONDATA-245) Actual Exception is getting lost in case of data dictionary file generation.
[
https://issues.apache.org/jira/browse/CARBONDATA-245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15499118#comment-15499118
]
ASF GitHub Bot commented on CARBONDATA-245:
---
Github user ravikiran23 commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/160#discussion_r79284972
--- Diff:
core/src/main/java/org/apache/carbondata/core/writer/ThriftWriter.java ---
@@ -80,6 +80,17 @@ public void open() throws IOException {
}
/**
+ * This will check whether stream and binary out is open or not.
+ * @return
+ */
+ public boolean isOpen() {
+if (null != binaryOut && null != dataOutputStream) {
--- End diff --
in close we are using util to close the stream so i think no need to set
null explicitly.
> Actual Exception is getting lost in case of data dictionary file generation.
>
>
> Key: CARBONDATA-245
> URL: https://issues.apache.org/jira/browse/CARBONDATA-245
> Project: CarbonData
> Issue Type: Bug
> Components: data-load
>Reporter: ravikiran
>Priority: Minor
>
> Actual Exception is getting lost in case of data dictionary file
> generation.getting the null pointer exception as in the finally block we are
> trying to write data to a null stream.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (CARBONDATA-251) making the auto compaction as blocking call.
ravikiran created CARBONDATA-251: Summary: making the auto compaction as blocking call. Key: CARBONDATA-251 URL: https://issues.apache.org/jira/browse/CARBONDATA-251 Project: CarbonData Issue Type: Bug Reporter: ravikiran making the auto compaction as blocking call. disabling the system level compaction lock feature. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CARBONDATA-251) making the auto compaction as blocking call.
[ https://issues.apache.org/jira/browse/CARBONDATA-251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15499264#comment-15499264 ] ASF GitHub Bot commented on CARBONDATA-251: --- GitHub user ravikiran23 opened a pull request: https://github.com/apache/incubator-carbondata/pull/168 [CARBONDATA-251] making the auto compaction as blocking call. Problem : the behaviour of auto compaction with spark-submit and thrift server is different. auto compaction will not work in spark-submit as the compaction is background process. Solution : Make the auto compaction feature as blocking call. disable the system level compaction lock feature by default. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ravikiran23/incubator-carbondata RemovalOfSystemCompactionLocks Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-carbondata/pull/168.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #168 commit c5c991e8bbe3a1918fcd1ecfe09d8a1b928cfe4b Author: ravikiran Date: 2016-09-17T09:48:22Z disabling the system compaction lock feature. and making the load ddl to wait for compaction to finish in the auto compaction case. > making the auto compaction as blocking call. > > > Key: CARBONDATA-251 > URL: https://issues.apache.org/jira/browse/CARBONDATA-251 > Project: CarbonData > Issue Type: Bug >Reporter: ravikiran > > making the auto compaction as blocking call. > disabling the system level compaction lock feature. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CARBONDATA-249) As LongType.simpleString in spark is "bigint", Carbon will convert Long to BigInt
[ https://issues.apache.org/jira/browse/CARBONDATA-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15499273#comment-15499273 ] ASF GitHub Bot commented on CARBONDATA-249: --- Github user asfgit closed the pull request at: https://github.com/apache/incubator-carbondata/pull/166 > As LongType.simpleString in spark is "bigint", Carbon will convert Long to > BigInt > - > > Key: CARBONDATA-249 > URL: https://issues.apache.org/jira/browse/CARBONDATA-249 > Project: CarbonData > Issue Type: Improvement >Reporter: Naresh P R >Priority: Minor > > Describe command will show DataType.simpleString in datatype column, > for LongType Spark DataType, simpleString is "bigint". > We are internally using long as name for bigint, which needs to be changed to > bigint in Carbon DataTypes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CARBONDATA-249) As LongType.simpleString in spark is "bigint", Carbon will convert Long to BigInt
[ https://issues.apache.org/jira/browse/CARBONDATA-249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Venkata Ramana G updated CARBONDATA-249: Assignee: Naresh P R > As LongType.simpleString in spark is "bigint", Carbon will convert Long to > BigInt > - > > Key: CARBONDATA-249 > URL: https://issues.apache.org/jira/browse/CARBONDATA-249 > Project: CarbonData > Issue Type: Improvement >Reporter: Naresh P R >Assignee: Naresh P R >Priority: Minor > Fix For: 0.2.0-incubating > > > Describe command will show DataType.simpleString in datatype column, > for LongType Spark DataType, simpleString is "bigint". > We are internally using long as name for bigint, which needs to be changed to > bigint in Carbon DataTypes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (CARBONDATA-249) As LongType.simpleString in spark is "bigint", Carbon will convert Long to BigInt
[ https://issues.apache.org/jira/browse/CARBONDATA-249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Venkata Ramana G resolved CARBONDATA-249. - Resolution: Fixed Fix Version/s: 0.2.0-incubating > As LongType.simpleString in spark is "bigint", Carbon will convert Long to > BigInt > - > > Key: CARBONDATA-249 > URL: https://issues.apache.org/jira/browse/CARBONDATA-249 > Project: CarbonData > Issue Type: Improvement >Reporter: Naresh P R >Assignee: Naresh P R >Priority: Minor > Fix For: 0.2.0-incubating > > > Describe command will show DataType.simpleString in datatype column, > for LongType Spark DataType, simpleString is "bigint". > We are internally using long as name for bigint, which needs to be changed to > bigint in Carbon DataTypes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CARBONDATA-251) making the auto compaction as blocking call.
[
https://issues.apache.org/jira/browse/CARBONDATA-251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15499290#comment-15499290
]
ASF GitHub Bot commented on CARBONDATA-251:
---
Github user gvramana commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/168#discussion_r79287134
--- Diff:
integration/spark/src/main/scala/org/apache/carbondata/spark/rdd/CarbonDataRDDFactory.scala
---
@@ -1030,28 +1021,36 @@ object CarbonDataRDDFactory extends Logging {
logWarning("Cannot write load metadata file as data load failed")
throw new Exception(errorMessage)
} else {
-val metadataDetails = status(0)._2
-if (!isAgg) {
- val status = CarbonLoaderUtil
-.recordLoadMetadata(currentLoadCount,
- metadataDetails,
- carbonLoadModel,
- loadStatus,
- loadStartTime
-)
- if (!status) {
-val errorMessage = "Dataload failed due to failure in table
status updation."
-logger.audit("Data load is failed for " +
-
s"${carbonLoadModel.getDatabaseName}.${carbonLoadModel.getTableName}")
-logger.error("Dataload failed due to failure in table status
updation.")
-throw new Exception(errorMessage)
+ val metadataDetails = status(0)._2
+ if (!isAgg) {
+val status = CarbonLoaderUtil
+ .recordLoadMetadata(currentLoadCount,
+metadataDetails,
+carbonLoadModel,
+loadStatus,
+loadStartTime
+ )
+if (!status) {
+ val errorMessage = "Dataload failed due to failure in table
status updation."
+ logger.audit("Data load is failed for " +
+
s"${carbonLoadModel.getDatabaseName}.${carbonLoadModel.getTableName}")
+ logger.error("Dataload failed due to failure in table status
updation.")
+ throw new Exception(errorMessage)
+}
+ } else if (!carbonLoadModel.isRetentionRequest) {
+// TODO : Handle it
+logInfo("Database updated**")
}
-} else if (!carbonLoadModel.isRetentionRequest) {
- // TODO : Handle it
- logInfo("Database updated**")
+ logger.audit("Data load is successful for " +
+
s"${carbonLoadModel.getDatabaseName}.${carbonLoadModel.getTableName}")
+try {
+ // compaction handling
+ handleSegmentMerging(tableCreationTime)
+}
+catch {
+ case e: Exception =>
+throw new Exception("Dataload is success. Compaction is
failed. Please check logs.")
--- End diff --
Add Auto compaction into error message.
> making the auto compaction as blocking call.
>
>
> Key: CARBONDATA-251
> URL: https://issues.apache.org/jira/browse/CARBONDATA-251
> Project: CarbonData
> Issue Type: Bug
>Reporter: ravikiran
>
> making the auto compaction as blocking call.
> disabling the system level compaction lock feature.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CARBONDATA-251) making the auto compaction as blocking call.
[
https://issues.apache.org/jira/browse/CARBONDATA-251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15499291#comment-15499291
]
ASF GitHub Bot commented on CARBONDATA-251:
---
Github user gvramana commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/168#discussion_r79287073
--- Diff:
integration/spark/src/main/scala/org/apache/carbondata/spark/rdd/CarbonDataRDDFactory.scala
---
@@ -623,14 +623,7 @@ object CarbonDataRDDFactory extends Logging {
}
}
}
- if(compactionModel.isDDLTrigger) {
-// making this an blocking call for DDL
-compactionThread.run()
- }
- else {
-// non blocking call in case of auto compaction.
-compactionThread.start()
- }
+compactionThread.run()
--- End diff --
Add a comment, why run is called instead of start, future will make it
concurrent.
> making the auto compaction as blocking call.
>
>
> Key: CARBONDATA-251
> URL: https://issues.apache.org/jira/browse/CARBONDATA-251
> Project: CarbonData
> Issue Type: Bug
>Reporter: ravikiran
>
> making the auto compaction as blocking call.
> disabling the system level compaction lock feature.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CARBONDATA-251) making the auto compaction as blocking call.
[ https://issues.apache.org/jira/browse/CARBONDATA-251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15499292#comment-15499292 ] ASF GitHub Bot commented on CARBONDATA-251: --- Github user gvramana commented on a diff in the pull request: https://github.com/apache/incubator-carbondata/pull/168#discussion_r79287056 --- Diff: core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java --- @@ -920,7 +920,7 @@ * Default value of Property for enabling system level compaction lock.1 compaction can run * at once. */ - public static String DEFAULT_ENABLE_CONCURRENT_COMPACTION = "false"; + public static String DEFAULT_ENABLE_CONCURRENT_COMPACTION = "true"; --- End diff -- Add comment that this property "carbon.concurrent.compaction" is @Depricated > making the auto compaction as blocking call. > > > Key: CARBONDATA-251 > URL: https://issues.apache.org/jira/browse/CARBONDATA-251 > Project: CarbonData > Issue Type: Bug >Reporter: ravikiran > > making the auto compaction as blocking call. > disabling the system level compaction lock feature. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CARBONDATA-251) making the auto compaction as blocking call.
[ https://issues.apache.org/jira/browse/CARBONDATA-251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15499296#comment-15499296 ] ASF GitHub Bot commented on CARBONDATA-251: --- Github user ravikiran23 commented on a diff in the pull request: https://github.com/apache/incubator-carbondata/pull/168#discussion_r79287261 --- Diff: core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java --- @@ -920,7 +920,7 @@ * Default value of Property for enabling system level compaction lock.1 compaction can run * at once. */ - public static String DEFAULT_ENABLE_CONCURRENT_COMPACTION = "false"; + public static String DEFAULT_ENABLE_CONCURRENT_COMPACTION = "true"; --- End diff -- fixed > making the auto compaction as blocking call. > > > Key: CARBONDATA-251 > URL: https://issues.apache.org/jira/browse/CARBONDATA-251 > Project: CarbonData > Issue Type: Bug >Reporter: ravikiran > > making the auto compaction as blocking call. > disabling the system level compaction lock feature. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CARBONDATA-251) making the auto compaction as blocking call.
[
https://issues.apache.org/jira/browse/CARBONDATA-251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15499298#comment-15499298
]
ASF GitHub Bot commented on CARBONDATA-251:
---
Github user ravikiran23 commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/168#discussion_r79287281
--- Diff:
integration/spark/src/main/scala/org/apache/carbondata/spark/rdd/CarbonDataRDDFactory.scala
---
@@ -623,14 +623,7 @@ object CarbonDataRDDFactory extends Logging {
}
}
}
- if(compactionModel.isDDLTrigger) {
-// making this an blocking call for DDL
-compactionThread.run()
- }
- else {
-// non blocking call in case of auto compaction.
-compactionThread.start()
- }
+compactionThread.run()
--- End diff --
fixed
> making the auto compaction as blocking call.
>
>
> Key: CARBONDATA-251
> URL: https://issues.apache.org/jira/browse/CARBONDATA-251
> Project: CarbonData
> Issue Type: Bug
>Reporter: ravikiran
>
> making the auto compaction as blocking call.
> disabling the system level compaction lock feature.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CARBONDATA-251) making the auto compaction as blocking call.
[
https://issues.apache.org/jira/browse/CARBONDATA-251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15499300#comment-15499300
]
ASF GitHub Bot commented on CARBONDATA-251:
---
Github user ravikiran23 commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/168#discussion_r79287305
--- Diff:
integration/spark/src/main/scala/org/apache/carbondata/spark/rdd/CarbonDataRDDFactory.scala
---
@@ -1030,28 +1021,36 @@ object CarbonDataRDDFactory extends Logging {
logWarning("Cannot write load metadata file as data load failed")
throw new Exception(errorMessage)
} else {
-val metadataDetails = status(0)._2
-if (!isAgg) {
- val status = CarbonLoaderUtil
-.recordLoadMetadata(currentLoadCount,
- metadataDetails,
- carbonLoadModel,
- loadStatus,
- loadStartTime
-)
- if (!status) {
-val errorMessage = "Dataload failed due to failure in table
status updation."
-logger.audit("Data load is failed for " +
-
s"${carbonLoadModel.getDatabaseName}.${carbonLoadModel.getTableName}")
-logger.error("Dataload failed due to failure in table status
updation.")
-throw new Exception(errorMessage)
+ val metadataDetails = status(0)._2
+ if (!isAgg) {
+val status = CarbonLoaderUtil
+ .recordLoadMetadata(currentLoadCount,
+metadataDetails,
+carbonLoadModel,
+loadStatus,
+loadStartTime
+ )
+if (!status) {
+ val errorMessage = "Dataload failed due to failure in table
status updation."
+ logger.audit("Data load is failed for " +
+
s"${carbonLoadModel.getDatabaseName}.${carbonLoadModel.getTableName}")
+ logger.error("Dataload failed due to failure in table status
updation.")
+ throw new Exception(errorMessage)
+}
+ } else if (!carbonLoadModel.isRetentionRequest) {
+// TODO : Handle it
+logInfo("Database updated**")
}
-} else if (!carbonLoadModel.isRetentionRequest) {
- // TODO : Handle it
- logInfo("Database updated**")
+ logger.audit("Data load is successful for " +
+
s"${carbonLoadModel.getDatabaseName}.${carbonLoadModel.getTableName}")
+try {
+ // compaction handling
+ handleSegmentMerging(tableCreationTime)
+}
+catch {
+ case e: Exception =>
+throw new Exception("Dataload is success. Compaction is
failed. Please check logs.")
--- End diff --
fixed
> making the auto compaction as blocking call.
>
>
> Key: CARBONDATA-251
> URL: https://issues.apache.org/jira/browse/CARBONDATA-251
> Project: CarbonData
> Issue Type: Bug
>Reporter: ravikiran
>
> making the auto compaction as blocking call.
> disabling the system level compaction lock feature.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (CARBONDATA-250) Throw exception and fail the data load if provided MAXCOLUMNS value is not proper
[ https://issues.apache.org/jira/browse/CARBONDATA-250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Venkata Ramana G resolved CARBONDATA-250. - Resolution: Fixed Fix Version/s: 0.2.0-incubating > Throw exception and fail the data load if provided MAXCOLUMNS value is not > proper > - > > Key: CARBONDATA-250 > URL: https://issues.apache.org/jira/browse/CARBONDATA-250 > Project: CarbonData > Issue Type: Bug >Reporter: Manohar Vanam >Assignee: Manohar Vanam > Fix For: 0.2.0-incubating > > > If the provided MAXCOLUMNS value in load query is not proper, then throw > exception and fail the data load. > Impact Area : Load query with maxcolumns option value -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CARBONDATA-250) Throw exception and fail the data load if provided MAXCOLUMNS value is not proper
[ https://issues.apache.org/jira/browse/CARBONDATA-250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15499310#comment-15499310 ] ASF GitHub Bot commented on CARBONDATA-250: --- Github user asfgit closed the pull request at: https://github.com/apache/incubator-carbondata/pull/167 > Throw exception and fail the data load if provided MAXCOLUMNS value is not > proper > - > > Key: CARBONDATA-250 > URL: https://issues.apache.org/jira/browse/CARBONDATA-250 > Project: CarbonData > Issue Type: Bug >Reporter: Manohar Vanam >Assignee: Manohar Vanam > Fix For: 0.2.0-incubating > > > If the provided MAXCOLUMNS value in load query is not proper, then throw > exception and fail the data load. > Impact Area : Load query with maxcolumns option value -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CARBONDATA-241) OOM error during query execution in long run
[
https://issues.apache.org/jira/browse/CARBONDATA-241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15499422#comment-15499422
]
ASF GitHub Bot commented on CARBONDATA-241:
---
Github user gvramana commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/158#discussion_r79288466
--- Diff:
core/src/main/java/org/apache/carbondata/core/carbon/datastore/BlockIndexStore.java
---
@@ -260,11 +295,29 @@ public void removeTableBlocks(List
removeTableBlocksInfos,
}
Map map =
tableBlocksMap.get(absoluteTableIdentifier);
// if there is no loaded blocks then return
-if (null == map) {
+if (null == map || map.isEmpty()) {
+ return;
+}
+Map> segmentIdToBlockInfoMap =
+segmentIdToBlockListMap.get(absoluteTableIdentifier);
+if (null == segmentIdToBlockInfoMap ||
segmentIdToBlockInfoMap.isEmpty()) {
return;
}
-for (TableBlockInfo blockInfos : removeTableBlocksInfos) {
- map.remove(blockInfos);
+synchronized (lockObject) {
+ for (String segmentId : segmentsToBeRemoved) {
+List tableBlockInfoList =
segmentIdToBlockInfoMap.get(segmentId);
+if (null == tableBlockInfoList) {
+ continue;
+}
+Iterator tableBlockInfoIterator =
tableBlockInfoList.iterator();
+while (tableBlockInfoIterator.hasNext()) {
+ TableBlockInfo info = tableBlockInfoIterator.next();
+ AbstractIndex remove = map.remove(info);
+ if (null != remove) {
--- End diff --
tableBlockInfoIterator.remove needs to called irrespective of null != remove
> OOM error during query execution in long run
>
>
> Key: CARBONDATA-241
> URL: https://issues.apache.org/jira/browse/CARBONDATA-241
> Project: CarbonData
> Issue Type: Bug
>Reporter: kumar vishal
>Assignee: kumar vishal
>
> **Problem:** During long run query execution is taking more time and it is
> throwing out of memory issue.
> **Reason**: In compaction we are compacting segments and each segment
> metadata is loaded in memory. So after compaction compacted segments are
> invalid but its meta data is not removed from memory because of this
> duplicate metadata is pile up and it is taking more memory and after few days
> query exeution is throwing OOM
> **Solution**: Need to remove invalid blocks from memory
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CARBONDATA-246) compaction is wrong in case if last segment is not assigned to an executor.
[ https://issues.apache.org/jira/browse/CARBONDATA-246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15499475#comment-15499475 ] ASF GitHub Bot commented on CARBONDATA-246: --- Github user gvramana commented on a diff in the pull request: https://github.com/apache/incubator-carbondata/pull/161#discussion_r79289428 --- Diff: integration/spark/src/main/scala/org/apache/carbondata/spark/rdd/CarbonMergerRDD.scala --- @@ -259,6 +253,9 @@ class CarbonMergerRDD[K, V]( ) ) + // keep on assigning till last one is reached. + blocksOfLastSegment = blocksOfOneSegment.asJava --- End diff -- add null check and size check > compaction is wrong in case if last segment is not assigned to an executor. > > > Key: CARBONDATA-246 > URL: https://issues.apache.org/jira/browse/CARBONDATA-246 > Project: CarbonData > Issue Type: Bug >Reporter: ravikiran > > if during compaction of 4 loads, for any executor if only first 3 loads task > is assigned then the col cardinality calculation based on the last segment > info will become wrong. > in this case the cardinality will go wrong for that executor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CARBONDATA-246) compaction is wrong in case if last segment is not assigned to an executor.
[ https://issues.apache.org/jira/browse/CARBONDATA-246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15499476#comment-15499476 ] ASF GitHub Bot commented on CARBONDATA-246: --- Github user gvramana commented on a diff in the pull request: https://github.com/apache/incubator-carbondata/pull/161#discussion_r79289289 --- Diff: integration/spark/src/main/scala/org/apache/carbondata/spark/rdd/CarbonMergerRDD.scala --- @@ -102,6 +102,11 @@ class CarbonMergerRDD[K, V]( var dataloadStatus = CarbonCommonConstants.STORE_LOADSTATUS_FAILURE val carbonSparkPartition = theSplit.asInstanceOf[CarbonSparkPartition] +// get destination segment properties as sent from driver which is of last segment. + +val segmentProperties = new SegmentProperties(carbonMergerMapping.columnSchemaList.asJava, --- End diff -- rename it to maxSegmentcolumnSchemaList and maxSegmentColCardinality. Write comment where it is declared > compaction is wrong in case if last segment is not assigned to an executor. > > > Key: CARBONDATA-246 > URL: https://issues.apache.org/jira/browse/CARBONDATA-246 > Project: CarbonData > Issue Type: Bug >Reporter: ravikiran > > if during compaction of 4 loads, for any executor if only first 3 loads task > is assigned then the col cardinality calculation based on the last segment > info will become wrong. > in this case the cardinality will go wrong for that executor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CARBONDATA-246) compaction is wrong in case if last segment is not assigned to an executor.
[ https://issues.apache.org/jira/browse/CARBONDATA-246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15499477#comment-15499477 ] ASF GitHub Bot commented on CARBONDATA-246: --- Github user gvramana commented on a diff in the pull request: https://github.com/apache/incubator-carbondata/pull/161#discussion_r79289444 --- Diff: integration/spark/src/main/scala/org/apache/carbondata/spark/rdd/CarbonMergerRDD.scala --- @@ -280,6 +277,25 @@ class CarbonMergerRDD[K, V]( taskInfoList.add(new TableTaskInfo(entry._1, entry._2).asInstanceOf[Distributable]) ) } + +// prepare the details required to extract the segment properties using last segment. + --- End diff -- Add empty and null check. > compaction is wrong in case if last segment is not assigned to an executor. > > > Key: CARBONDATA-246 > URL: https://issues.apache.org/jira/browse/CARBONDATA-246 > Project: CarbonData > Issue Type: Bug >Reporter: ravikiran > > if during compaction of 4 loads, for any executor if only first 3 loads task > is assigned then the col cardinality calculation based on the last segment > info will become wrong. > in this case the cardinality will go wrong for that executor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CARBONDATA-246) compaction is wrong in case if last segment is not assigned to an executor.
[ https://issues.apache.org/jira/browse/CARBONDATA-246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15499527#comment-15499527 ] ASF GitHub Bot commented on CARBONDATA-246: --- Github user ravikiran23 commented on a diff in the pull request: https://github.com/apache/incubator-carbondata/pull/161#discussion_r79290197 --- Diff: integration/spark/src/main/scala/org/apache/carbondata/spark/rdd/CarbonMergerRDD.scala --- @@ -102,6 +102,11 @@ class CarbonMergerRDD[K, V]( var dataloadStatus = CarbonCommonConstants.STORE_LOADSTATUS_FAILURE val carbonSparkPartition = theSplit.asInstanceOf[CarbonSparkPartition] +// get destination segment properties as sent from driver which is of last segment. + +val segmentProperties = new SegmentProperties(carbonMergerMapping.columnSchemaList.asJava, --- End diff -- fixed > compaction is wrong in case if last segment is not assigned to an executor. > > > Key: CARBONDATA-246 > URL: https://issues.apache.org/jira/browse/CARBONDATA-246 > Project: CarbonData > Issue Type: Bug >Reporter: ravikiran > > if during compaction of 4 loads, for any executor if only first 3 loads task > is assigned then the col cardinality calculation based on the last segment > info will become wrong. > in this case the cardinality will go wrong for that executor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CARBONDATA-246) compaction is wrong in case if last segment is not assigned to an executor.
[ https://issues.apache.org/jira/browse/CARBONDATA-246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15499534#comment-15499534 ] ASF GitHub Bot commented on CARBONDATA-246: --- Github user ravikiran23 commented on a diff in the pull request: https://github.com/apache/incubator-carbondata/pull/161#discussion_r79290299 --- Diff: integration/spark/src/main/scala/org/apache/carbondata/spark/rdd/CarbonMergerRDD.scala --- @@ -259,6 +253,9 @@ class CarbonMergerRDD[K, V]( ) ) + // keep on assigning till last one is reached. + blocksOfLastSegment = blocksOfOneSegment.asJava --- End diff -- fixed > compaction is wrong in case if last segment is not assigned to an executor. > > > Key: CARBONDATA-246 > URL: https://issues.apache.org/jira/browse/CARBONDATA-246 > Project: CarbonData > Issue Type: Bug >Reporter: ravikiran > > if during compaction of 4 loads, for any executor if only first 3 loads task > is assigned then the col cardinality calculation based on the last segment > info will become wrong. > in this case the cardinality will go wrong for that executor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CARBONDATA-246) compaction is wrong in case if last segment is not assigned to an executor.
[ https://issues.apache.org/jira/browse/CARBONDATA-246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15499535#comment-15499535 ] ASF GitHub Bot commented on CARBONDATA-246: --- Github user ravikiran23 commented on a diff in the pull request: https://github.com/apache/incubator-carbondata/pull/161#discussion_r79290314 --- Diff: integration/spark/src/main/scala/org/apache/carbondata/spark/rdd/CarbonMergerRDD.scala --- @@ -280,6 +277,25 @@ class CarbonMergerRDD[K, V]( taskInfoList.add(new TableTaskInfo(entry._1, entry._2).asInstanceOf[Distributable]) ) } + +// prepare the details required to extract the segment properties using last segment. + --- End diff -- fixed > compaction is wrong in case if last segment is not assigned to an executor. > > > Key: CARBONDATA-246 > URL: https://issues.apache.org/jira/browse/CARBONDATA-246 > Project: CarbonData > Issue Type: Bug >Reporter: ravikiran > > if during compaction of 4 loads, for any executor if only first 3 loads task > is assigned then the col cardinality calculation based on the last segment > info will become wrong. > in this case the cardinality will go wrong for that executor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CARBONDATA-241) OOM error during query execution in long run
[
https://issues.apache.org/jira/browse/CARBONDATA-241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15499568#comment-15499568
]
ASF GitHub Bot commented on CARBONDATA-241:
---
Github user gvramana commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/158#discussion_r79290374
--- Diff:
integration/spark/src/main/scala/org/apache/carbondata/spark/rdd/CarbonScanRDD.scala
---
@@ -102,7 +115,7 @@ class CarbonScanRDD[V: ClassTag](
val splits = carbonInputFormat.getSplits(job)
if (!splits.isEmpty) {
val carbonInputSplits =
splits.asScala.map(_.asInstanceOf[CarbonInputSplit])
-
+
queryModel.setInvalidSegmentIds(validAndInvalidSegments.getInvalidSegments)
--- End diff --
move this to common getSplits, other wise validAndInvalidSegments can be
null, if parallel deletion happens.
> OOM error during query execution in long run
>
>
> Key: CARBONDATA-241
> URL: https://issues.apache.org/jira/browse/CARBONDATA-241
> Project: CarbonData
> Issue Type: Bug
>Reporter: kumar vishal
>Assignee: kumar vishal
>
> **Problem:** During long run query execution is taking more time and it is
> throwing out of memory issue.
> **Reason**: In compaction we are compacting segments and each segment
> metadata is loaded in memory. So after compaction compacted segments are
> invalid but its meta data is not removed from memory because of this
> duplicate metadata is pile up and it is taking more memory and after few days
> query exeution is throwing OOM
> **Solution**: Need to remove invalid blocks from memory
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CARBONDATA-241) OOM error during query execution in long run
[
https://issues.apache.org/jira/browse/CARBONDATA-241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15499570#comment-15499570
]
ASF GitHub Bot commented on CARBONDATA-241:
---
Github user gvramana commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/158#discussion_r79290267
--- Diff:
hadoop/src/main/java/org/apache/carbondata/hadoop/CarbonInputFormat.java ---
@@ -706,8 +725,9 @@ private String getUpdateExtension() {
/**
* @return updateExtension
*/
- private String[] getValidSegments(JobContext job) throws IOException {
-String segmentString =
job.getConfiguration().get(INPUT_SEGMENT_NUMBERS, "");
+ private String[] getSegmentsFromConfiguration(JobContext job, String
segmentType)
+ throws IOException {
+String segmentString = job.getConfiguration().get(segmentType, "");
--- End diff --
change signature to previous one
> OOM error during query execution in long run
>
>
> Key: CARBONDATA-241
> URL: https://issues.apache.org/jira/browse/CARBONDATA-241
> Project: CarbonData
> Issue Type: Bug
>Reporter: kumar vishal
>Assignee: kumar vishal
>
> **Problem:** During long run query execution is taking more time and it is
> throwing out of memory issue.
> **Reason**: In compaction we are compacting segments and each segment
> metadata is loaded in memory. So after compaction compacted segments are
> invalid but its meta data is not removed from memory because of this
> duplicate metadata is pile up and it is taking more memory and after few days
> query exeution is throwing OOM
> **Solution**: Need to remove invalid blocks from memory
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CARBONDATA-241) OOM error during query execution in long run
[
https://issues.apache.org/jira/browse/CARBONDATA-241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15499569#comment-15499569
]
ASF GitHub Bot commented on CARBONDATA-241:
---
Github user gvramana commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/158#discussion_r79290196
--- Diff:
hadoop/src/test/java/org/apache/carbondata/hadoop/ft/CarbonInputMapperTest.java
---
@@ -129,6 +132,37 @@ private int countTheColumns(String outPath) throws
Exception {
return 0;
}
+ private void runJob(String outPath, CarbonProjection projection,
Expression filter)
--- End diff --
move back the code to original place
> OOM error during query execution in long run
>
>
> Key: CARBONDATA-241
> URL: https://issues.apache.org/jira/browse/CARBONDATA-241
> Project: CarbonData
> Issue Type: Bug
>Reporter: kumar vishal
>Assignee: kumar vishal
>
> **Problem:** During long run query execution is taking more time and it is
> throwing out of memory issue.
> **Reason**: In compaction we are compacting segments and each segment
> metadata is loaded in memory. So after compaction compacted segments are
> invalid but its meta data is not removed from memory because of this
> duplicate metadata is pile up and it is taking more memory and after few days
> query exeution is throwing OOM
> **Solution**: Need to remove invalid blocks from memory
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (CARBONDATA-234) wrong message is printed in the logs each time when the compaction is done.
[ https://issues.apache.org/jira/browse/CARBONDATA-234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Venkata Ramana G resolved CARBONDATA-234. - Resolution: Fixed Assignee: ravikiran Fix Version/s: 0.2.0-incubating > wrong message is printed in the logs each time when the compaction is done. > --- > > Key: CARBONDATA-234 > URL: https://issues.apache.org/jira/browse/CARBONDATA-234 > Project: CarbonData > Issue Type: Bug >Reporter: ravikiran >Assignee: ravikiran > Fix For: 0.2.0-incubating > > > wrong message is printed in the logs each time when the compaction is done. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CARBONDATA-234) wrong message is printed in the logs each time when the compaction is done.
[ https://issues.apache.org/jira/browse/CARBONDATA-234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15499603#comment-15499603 ] ASF GitHub Bot commented on CARBONDATA-234: --- Github user asfgit closed the pull request at: https://github.com/apache/incubator-carbondata/pull/149 > wrong message is printed in the logs each time when the compaction is done. > --- > > Key: CARBONDATA-234 > URL: https://issues.apache.org/jira/browse/CARBONDATA-234 > Project: CarbonData > Issue Type: Bug >Reporter: ravikiran > Fix For: 0.2.0-incubating > > > wrong message is printed in the logs each time when the compaction is done. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CARBONDATA-244) Load and delete segment by id queries giving inconsistent results when we execute parallely
[
https://issues.apache.org/jira/browse/CARBONDATA-244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15499620#comment-15499620
]
ASF GitHub Bot commented on CARBONDATA-244:
---
Github user akashrn5 commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/156#discussion_r79291255
--- Diff:
integration/spark/src/main/scala/org/apache/carbondata/spark/rdd/CarbonDataRDDFactory.scala
---
@@ -,25 +1131,32 @@ object CarbonDataRDDFactory extends Logging {
val table =
org.apache.carbondata.core.carbon.metadata.CarbonMetadata.getInstance
.getCarbonTable(carbonLoadModel.getDatabaseName + "_" +
carbonLoadModel.getTableName)
val metaDataPath: String = table.getMetaDataFilepath
-val carbonLock = CarbonLockFactory
+val carbonCleanFilesLock = CarbonLockFactory
.getCarbonLockObj(table.getAbsoluteTableIdentifier.getCarbonTableIdentifier,
-LockUsage.METADATA_LOCK
+LockUsage.CLEAN_FILES_LOCK
)
try {
- if (carbonLock.lockWithRetries()) {
+ if (carbonCleanFilesLock.lockWithRetries()) {
+logger.info("Clean files lock has been successfully acquired.")
deleteLoadsAndUpdateMetadata(carbonLoadModel,
table,
partitioner,
hdfsStoreLocation,
isForceDeletion = true)
}
+ else {
+val errorMsg = "Clean files request is failed for " +
carbonLoadModel.getDatabaseName +
--- End diff --
"Clean file request is running in the background, please try after some
time". u[date the comment since already we know the operation
> Load and delete segment by id queries giving inconsistent results when we
> execute parallely
> ---
>
> Key: CARBONDATA-244
> URL: https://issues.apache.org/jira/browse/CARBONDATA-244
> Project: CarbonData
> Issue Type: Bug
>Reporter: Manohar Vanam
>Assignee: Manohar Vanam
>
> Delete segment by id behavior is inconsistent when we Execute load and
> delete segment by id queries parallely,
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CARBONDATA-244) Load and delete segment by id queries giving inconsistent results when we execute parallely
[ https://issues.apache.org/jira/browse/CARBONDATA-244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15499624#comment-15499624 ] ASF GitHub Bot commented on CARBONDATA-244: --- Github user sujith71955 commented on a diff in the pull request: https://github.com/apache/incubator-carbondata/pull/156#discussion_r79291307 --- Diff: processing/src/main/java/org/apache/carbondata/lcm/locks/CarbonLockUtil.java --- @@ -0,0 +1,64 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.carbondata.lcm.locks; + +import org.apache.carbondata.common.logging.LogService; +import org.apache.carbondata.common.logging.LogServiceFactory; + +/** + * This is a singleton class for initialization of zookeeper client. --- End diff -- provide proper comment > Load and delete segment by id queries giving inconsistent results when we > execute parallely > --- > > Key: CARBONDATA-244 > URL: https://issues.apache.org/jira/browse/CARBONDATA-244 > Project: CarbonData > Issue Type: Bug >Reporter: Manohar Vanam >Assignee: Manohar Vanam > > Delete segment by id behavior is inconsistent when we Execute load and > delete segment by id queries parallely, -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CARBONDATA-244) Load and delete segment by id queries giving inconsistent results when we execute parallely
[ https://issues.apache.org/jira/browse/CARBONDATA-244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15499627#comment-15499627 ] ASF GitHub Bot commented on CARBONDATA-244: --- Github user sujith71955 commented on a diff in the pull request: https://github.com/apache/incubator-carbondata/pull/156#discussion_r79291346 --- Diff: processing/src/main/java/org/apache/carbondata/lcm/locks/LockUsage.java --- @@ -28,5 +28,7 @@ public static String COMPACTION_LOCK = "compaction.lock"; public static String SYSTEMLEVEL_COMPACTION_LOCK = "system_level_compaction.lock"; public static String TABLE_STATUS_LOCK = "tablestatus.lock"; + public static String DELETE_SEGMENT_LOCK = "delete_segment.lock"; --- End diff -- make it final/use enum > Load and delete segment by id queries giving inconsistent results when we > execute parallely > --- > > Key: CARBONDATA-244 > URL: https://issues.apache.org/jira/browse/CARBONDATA-244 > Project: CarbonData > Issue Type: Bug >Reporter: Manohar Vanam >Assignee: Manohar Vanam > > Delete segment by id behavior is inconsistent when we Execute load and > delete segment by id queries parallely, -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CARBONDATA-244) Load and delete segment by id queries giving inconsistent results when we execute parallely
[
https://issues.apache.org/jira/browse/CARBONDATA-244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15499631#comment-15499631
]
ASF GitHub Bot commented on CARBONDATA-244:
---
Github user sujith71955 commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/156#discussion_r79291381
--- Diff:
processing/src/main/java/org/apache/carbondata/lcm/status/SegmentStatusManager.java
---
@@ -287,12 +307,18 @@ private Integer compareDateValues(Long loadValue,
Long userValue) {
}
} else {
-LOG.error("Unable to acquire the metadata lock");
+String errorMsg = "Delete segment by id is failed for " +
tableDetails
++ ". Not able to acquire the delete segment lock due to other
operation running "
--- End diff --
update message, provide operation name
> Load and delete segment by id queries giving inconsistent results when we
> execute parallely
> ---
>
> Key: CARBONDATA-244
> URL: https://issues.apache.org/jira/browse/CARBONDATA-244
> Project: CarbonData
> Issue Type: Bug
>Reporter: Manohar Vanam
>Assignee: Manohar Vanam
>
> Delete segment by id behavior is inconsistent when we Execute load and
> delete segment by id queries parallely,
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CARBONDATA-247) Higher MAXCOLUMNS value in load DML options is leading to out of memory error
[ https://issues.apache.org/jira/browse/CARBONDATA-247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15499635#comment-15499635 ] ASF GitHub Bot commented on CARBONDATA-247: --- Github user asfgit closed the pull request at: https://github.com/apache/incubator-carbondata/pull/162 > Higher MAXCOLUMNS value in load DML options is leading to out of memory error > - > > Key: CARBONDATA-247 > URL: https://issues.apache.org/jira/browse/CARBONDATA-247 > Project: CarbonData > Issue Type: Bug >Reporter: dhatchayani >Priority: Minor > > When a higher value lets say Integer max value is configured for maxcolumns > option in load DML and executor memory is less, then in that case > UnivocityCsvParser throws an out of memory error when it tries to create an > array of size of maxColumns option value. > java.lang.OutOfMemoryError: Java heap space > at > com.univocity.parsers.common.ParserOutput.(ParserOutput.java:86) > at > com.univocity.parsers.common.AbstractParser.(AbstractParser.java:66) > at com.univocity.parsers.csv.CsvParser.(CsvParser.java:50) > at > org.apache.carbondata.processing.csvreaderstep.UnivocityCsvParser.initialize(UnivocityCsvParser.java:114) > at > org.apache.carbondata.processing.csvreaderstep.CsvInput.doProcessUnivocity(CsvInput.java:427) > at > org.apache.carbondata.processing.csvreaderstep.CsvInput.access$100(CsvInput.java:60) > at > org.apache.carbondata.processing.csvreaderstep.CsvInput$1.call(CsvInput.java:389) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (CARBONDATA-247) Higher MAXCOLUMNS value in load DML options is leading to out of memory error
[ https://issues.apache.org/jira/browse/CARBONDATA-247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Venkata Ramana G resolved CARBONDATA-247. - Resolution: Fixed Fix Version/s: 0.2.0-incubating > Higher MAXCOLUMNS value in load DML options is leading to out of memory error > - > > Key: CARBONDATA-247 > URL: https://issues.apache.org/jira/browse/CARBONDATA-247 > Project: CarbonData > Issue Type: Bug >Reporter: dhatchayani >Priority: Minor > Fix For: 0.2.0-incubating > > > When a higher value lets say Integer max value is configured for maxcolumns > option in load DML and executor memory is less, then in that case > UnivocityCsvParser throws an out of memory error when it tries to create an > array of size of maxColumns option value. > java.lang.OutOfMemoryError: Java heap space > at > com.univocity.parsers.common.ParserOutput.(ParserOutput.java:86) > at > com.univocity.parsers.common.AbstractParser.(AbstractParser.java:66) > at com.univocity.parsers.csv.CsvParser.(CsvParser.java:50) > at > org.apache.carbondata.processing.csvreaderstep.UnivocityCsvParser.initialize(UnivocityCsvParser.java:114) > at > org.apache.carbondata.processing.csvreaderstep.CsvInput.doProcessUnivocity(CsvInput.java:427) > at > org.apache.carbondata.processing.csvreaderstep.CsvInput.access$100(CsvInput.java:60) > at > org.apache.carbondata.processing.csvreaderstep.CsvInput$1.call(CsvInput.java:389) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CARBONDATA-247) Higher MAXCOLUMNS value in load DML options is leading to out of memory error
[ https://issues.apache.org/jira/browse/CARBONDATA-247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Venkata Ramana G updated CARBONDATA-247: Component/s: data-load > Higher MAXCOLUMNS value in load DML options is leading to out of memory error > - > > Key: CARBONDATA-247 > URL: https://issues.apache.org/jira/browse/CARBONDATA-247 > Project: CarbonData > Issue Type: Bug > Components: data-load >Reporter: dhatchayani >Priority: Minor > Fix For: 0.2.0-incubating > > > When a higher value lets say Integer max value is configured for maxcolumns > option in load DML and executor memory is less, then in that case > UnivocityCsvParser throws an out of memory error when it tries to create an > array of size of maxColumns option value. > java.lang.OutOfMemoryError: Java heap space > at > com.univocity.parsers.common.ParserOutput.(ParserOutput.java:86) > at > com.univocity.parsers.common.AbstractParser.(AbstractParser.java:66) > at com.univocity.parsers.csv.CsvParser.(CsvParser.java:50) > at > org.apache.carbondata.processing.csvreaderstep.UnivocityCsvParser.initialize(UnivocityCsvParser.java:114) > at > org.apache.carbondata.processing.csvreaderstep.CsvInput.doProcessUnivocity(CsvInput.java:427) > at > org.apache.carbondata.processing.csvreaderstep.CsvInput.access$100(CsvInput.java:60) > at > org.apache.carbondata.processing.csvreaderstep.CsvInput$1.call(CsvInput.java:389) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CARBONDATA-252) Filter result is not proper when Double data type values with 0.0 and -0.0 will be used
Sujith created CARBONDATA-252: - Summary: Filter result is not proper when Double data type values with 0.0 and -0.0 will be used Key: CARBONDATA-252 URL: https://issues.apache.org/jira/browse/CARBONDATA-252 Project: CarbonData Issue Type: Bug Reporter: Sujith Assignee: Sujith Priority: Minor Filter result is not proper when Double data type values with 0.0 and -0.0 will be used in filter model -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CARBONDATA-239) Failure of one compaction in queue should not affect the others.
[ https://issues.apache.org/jira/browse/CARBONDATA-239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15499655#comment-15499655 ] ASF GitHub Bot commented on CARBONDATA-239: --- Github user asfgit closed the pull request at: https://github.com/apache/incubator-carbondata/pull/155 > Failure of one compaction in queue should not affect the others. > > > Key: CARBONDATA-239 > URL: https://issues.apache.org/jira/browse/CARBONDATA-239 > Project: CarbonData > Issue Type: Bug >Reporter: ravikiran > > Failure of one compaction in queue should not affect the others. > If a compaction is triggered by the user on table1 , and other requests will > go to queue. and if the compaction is failed for table1 then the requests in > queue should continue and at the end the beeline will show the failure > message to the user. > if any compaction gets failed for a table which is other than the user > requested table then the error in the beeline should not appear. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (CARBONDATA-239) Failure of one compaction in queue should not affect the others.
[ https://issues.apache.org/jira/browse/CARBONDATA-239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Venkata Ramana G resolved CARBONDATA-239. - Resolution: Fixed Assignee: ravikiran Fix Version/s: 0.2.0-incubating > Failure of one compaction in queue should not affect the others. > > > Key: CARBONDATA-239 > URL: https://issues.apache.org/jira/browse/CARBONDATA-239 > Project: CarbonData > Issue Type: Bug >Reporter: ravikiran >Assignee: ravikiran > Fix For: 0.2.0-incubating > > > Failure of one compaction in queue should not affect the others. > If a compaction is triggered by the user on table1 , and other requests will > go to queue. and if the compaction is failed for table1 then the requests in > queue should continue and at the end the beeline will show the failure > message to the user. > if any compaction gets failed for a table which is other than the user > requested table then the error in the beeline should not appear. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CARBONDATA-244) Load and delete segment by id queries giving inconsistent results when we execute parallely
[
https://issues.apache.org/jira/browse/CARBONDATA-244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15499665#comment-15499665
]
ASF GitHub Bot commented on CARBONDATA-244:
---
Github user sujith71955 commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/156#discussion_r79291710
--- Diff:
processing/src/main/java/org/apache/carbondata/lcm/status/SegmentStatusManager.java
---
@@ -287,12 +307,18 @@ private Integer compareDateValues(Long loadValue,
Long userValue) {
}
} else {
-LOG.error("Unable to acquire the metadata lock");
+String errorMsg = "Delete segment by id is failed for " +
tableDetails
++ ". Not able to acquire the delete segment lock due to other
operation running "
--- End diff --
LGTM
> Load and delete segment by id queries giving inconsistent results when we
> execute parallely
> ---
>
> Key: CARBONDATA-244
> URL: https://issues.apache.org/jira/browse/CARBONDATA-244
> Project: CarbonData
> Issue Type: Bug
>Reporter: Manohar Vanam
>Assignee: Manohar Vanam
>
> Delete segment by id behavior is inconsistent when we Execute load and
> delete segment by id queries parallely,
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CARBONDATA-250) Throw exception and fail the data load if provided MAXCOLUMNS value is not proper
[ https://issues.apache.org/jira/browse/CARBONDATA-250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15499670#comment-15499670 ] ASF GitHub Bot commented on CARBONDATA-250: --- GitHub user sujith71955 opened a pull request: https://github.com/apache/incubator-carbondata/pull/169 [CARBONDATA-250] Filter result is not proper when Double data type values with 0.0 and -0.0 will be used. [**Problem**] Equals filter query where -0.0 is used in filter model was not giving proper result since the system was comparing double value 0.0 and -0.0 as different and resultset was getting changed. eg: select log(c4_double,1) from Test_Boundary1 where log(c4_double,1)= -0.0 [**Solution**] System has to compare double values for its equality and also it will preserve the -0.0 and 0.0 equality as same == operator ,also preserve NaN equality check as per java.lang.Double.equals() You can merge this pull request into a Git repository by running: $ git pull https://github.com/sujith71955/incubator-carbondata master_DoubleTypeEquals Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-carbondata/pull/169.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #169 commit 6b7c39ccb73d84cf2d5943e0542077d26c1a5e87 Author: Venkata Ramana G Date: 2016-09-17T16:38:11Z [CARBONDATA-250] Filter result is not proper when Double data type values with 0.0 and -0.0 will be used. [Description] This method will compare double values for its equality and also it will preserve the -0.0 and 0.0 equality as per == ,also preserve NaN equality check as per java.lang.Double.equals() > Throw exception and fail the data load if provided MAXCOLUMNS value is not > proper > - > > Key: CARBONDATA-250 > URL: https://issues.apache.org/jira/browse/CARBONDATA-250 > Project: CarbonData > Issue Type: Bug >Reporter: Manohar Vanam >Assignee: Manohar Vanam > Fix For: 0.2.0-incubating > > > If the provided MAXCOLUMNS value in load query is not proper, then throw > exception and fail the data load. > Impact Area : Load query with maxcolumns option value -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CARBONDATA-241) OOM error during query execution in long run
[ https://issues.apache.org/jira/browse/CARBONDATA-241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15499676#comment-15499676 ] ASF GitHub Bot commented on CARBONDATA-241: --- Github user asfgit closed the pull request at: https://github.com/apache/incubator-carbondata/pull/158 > OOM error during query execution in long run > > > Key: CARBONDATA-241 > URL: https://issues.apache.org/jira/browse/CARBONDATA-241 > Project: CarbonData > Issue Type: Bug >Reporter: kumar vishal >Assignee: kumar vishal > > **Problem:** During long run query execution is taking more time and it is > throwing out of memory issue. > **Reason**: In compaction we are compacting segments and each segment > metadata is loaded in memory. So after compaction compacted segments are > invalid but its meta data is not removed from memory because of this > duplicate metadata is pile up and it is taking more memory and after few days > query exeution is throwing OOM > **Solution**: Need to remove invalid blocks from memory > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CARBONDATA-241) OOM error during query execution in long run
[ https://issues.apache.org/jira/browse/CARBONDATA-241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15499681#comment-15499681 ] Venkata Ramana G commented on CARBONDATA-241: - [~kumarvishal09] Merged PR to 0.2.0. please raise PR for 0.1.0 > OOM error during query execution in long run > > > Key: CARBONDATA-241 > URL: https://issues.apache.org/jira/browse/CARBONDATA-241 > Project: CarbonData > Issue Type: Bug >Reporter: kumar vishal >Assignee: kumar vishal > > **Problem:** During long run query execution is taking more time and it is > throwing out of memory issue. > **Reason**: In compaction we are compacting segments and each segment > metadata is loaded in memory. So after compaction compacted segments are > invalid but its meta data is not removed from memory because of this > duplicate metadata is pile up and it is taking more memory and after few days > query exeution is throwing OOM > **Solution**: Need to remove invalid blocks from memory > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CARBONDATA-245) Actual Exception is getting lost in case of data dictionary file generation.
[ https://issues.apache.org/jira/browse/CARBONDATA-245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15499701#comment-15499701 ] ASF GitHub Bot commented on CARBONDATA-245: --- Github user asfgit closed the pull request at: https://github.com/apache/incubator-carbondata/pull/160 > Actual Exception is getting lost in case of data dictionary file generation. > > > Key: CARBONDATA-245 > URL: https://issues.apache.org/jira/browse/CARBONDATA-245 > Project: CarbonData > Issue Type: Bug > Components: data-load >Reporter: ravikiran >Priority: Minor > > Actual Exception is getting lost in case of data dictionary file > generation.getting the null pointer exception as in the finally block we are > trying to write data to a null stream. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (CARBONDATA-245) Actual Exception is getting lost in case of data dictionary file generation.
[ https://issues.apache.org/jira/browse/CARBONDATA-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Venkata Ramana G resolved CARBONDATA-245. - Resolution: Fixed Assignee: ravikiran Fix Version/s: 0.2.0-incubating > Actual Exception is getting lost in case of data dictionary file generation. > > > Key: CARBONDATA-245 > URL: https://issues.apache.org/jira/browse/CARBONDATA-245 > Project: CarbonData > Issue Type: Bug > Components: data-load >Reporter: ravikiran >Assignee: ravikiran >Priority: Minor > Fix For: 0.2.0-incubating > > > Actual Exception is getting lost in case of data dictionary file > generation.getting the null pointer exception as in the finally block we are > trying to write data to a null stream. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CARBONDATA-253) Duplicate block loading when distribution is based on blocklet
kumar vishal created CARBONDATA-253: --- Summary: Duplicate block loading when distribution is based on blocklet Key: CARBONDATA-253 URL: https://issues.apache.org/jira/browse/CARBONDATA-253 Project: CarbonData Issue Type: Bug Reporter: kumar vishal Assignee: kumar vishal In case of query execution when distribution is based on blocklet same blocks are getting loaded multiple times this is because hash code and equals method contract is not same -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CARBONDATA-253) Duplicate block loading when distribution is based on blocklet
[ https://issues.apache.org/jira/browse/CARBONDATA-253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15499715#comment-15499715 ] ASF GitHub Bot commented on CARBONDATA-253: --- GitHub user kumarvishal09 opened a pull request: https://github.com/apache/incubator-carbondata/pull/170 [CARBONDATA-253]OOM issue if distribution is based on blocklet duing query execution Problem:In case of query execution when distribution is based on blocklet same blocks are getting loaded multiple times this is because hash code and equals method contract is not same, this is can cause OOM issue if distribution is based on blocklet Solution: As same class will be used to identify unique blocks while distribution and while loading so creating a wrapper class and implementing hash code and equals method based on filepath, offset and length, this will remove duplicate blocks and only one block's metadata will be loaded in memory You can merge this pull request into a Git repository by running: $ git pull https://github.com/kumarvishal09/incubator-carbondata equalsAndHashCodeIssue Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-carbondata/pull/170.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #170 commit 49a76db0ff8e63fc95a309984d215086d888fc7b Author: kumarvishal Date: 2016-09-17T13:25:36Z equalsAndHashCodeIssue > Duplicate block loading when distribution is based on blocklet > -- > > Key: CARBONDATA-253 > URL: https://issues.apache.org/jira/browse/CARBONDATA-253 > Project: CarbonData > Issue Type: Bug >Reporter: kumar vishal >Assignee: kumar vishal > > In case of query execution when distribution is based on blocklet same blocks > are getting loaded multiple times this is because hash code and equals method > contract is not same -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CARBONDATA-244) Load and delete segment by id queries giving inconsistent results when we execute parallely
[ https://issues.apache.org/jira/browse/CARBONDATA-244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15499717#comment-15499717 ] ASF GitHub Bot commented on CARBONDATA-244: --- Github user asfgit closed the pull request at: https://github.com/apache/incubator-carbondata/pull/156 > Load and delete segment by id queries giving inconsistent results when we > execute parallely > --- > > Key: CARBONDATA-244 > URL: https://issues.apache.org/jira/browse/CARBONDATA-244 > Project: CarbonData > Issue Type: Bug >Reporter: Manohar Vanam >Assignee: Manohar Vanam > > Delete segment by id behavior is inconsistent when we Execute load and > delete segment by id queries parallely, -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CARBONDATA-245) Actual Exception is getting lost in case of data dictionary file generation.
[ https://issues.apache.org/jira/browse/CARBONDATA-245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15499721#comment-15499721 ] Venkata Ramana G commented on CARBONDATA-245: - Merged PR in 0.2.0 version, Please raise PR to 0.1.0 version > Actual Exception is getting lost in case of data dictionary file generation. > > > Key: CARBONDATA-245 > URL: https://issues.apache.org/jira/browse/CARBONDATA-245 > Project: CarbonData > Issue Type: Bug > Components: data-load >Reporter: ravikiran >Assignee: ravikiran >Priority: Minor > Fix For: 0.2.0-incubating > > > Actual Exception is getting lost in case of data dictionary file > generation.getting the null pointer exception as in the finally block we are > trying to write data to a null stream. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CARBONDATA-246) compaction is wrong in case if last segment is not assigned to an executor.
[ https://issues.apache.org/jira/browse/CARBONDATA-246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15499731#comment-15499731 ] ASF GitHub Bot commented on CARBONDATA-246: --- Github user asfgit closed the pull request at: https://github.com/apache/incubator-carbondata/pull/161 > compaction is wrong in case if last segment is not assigned to an executor. > > > Key: CARBONDATA-246 > URL: https://issues.apache.org/jira/browse/CARBONDATA-246 > Project: CarbonData > Issue Type: Bug >Reporter: ravikiran > > if during compaction of 4 loads, for any executor if only first 3 loads task > is assigned then the col cardinality calculation based on the last segment > info will become wrong. > in this case the cardinality will go wrong for that executor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CARBONDATA-246) compaction is wrong in case if last segment is not assigned to an executor.
[ https://issues.apache.org/jira/browse/CARBONDATA-246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Venkata Ramana G updated CARBONDATA-246: Affects Version/s: 0.1.0-incubating > compaction is wrong in case if last segment is not assigned to an executor. > > > Key: CARBONDATA-246 > URL: https://issues.apache.org/jira/browse/CARBONDATA-246 > Project: CarbonData > Issue Type: Bug >Affects Versions: 0.1.0-incubating >Reporter: ravikiran > > if during compaction of 4 loads, for any executor if only first 3 loads task > is assigned then the col cardinality calculation based on the last segment > info will become wrong. > in this case the cardinality will go wrong for that executor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CARBONDATA-246) compaction is wrong in case if last segment is not assigned to an executor.
[ https://issues.apache.org/jira/browse/CARBONDATA-246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15499735#comment-15499735 ] Venkata Ramana G commented on CARBONDATA-246: - Merged PR to 0.2.0. Please raise PR to 0.1.0 > compaction is wrong in case if last segment is not assigned to an executor. > > > Key: CARBONDATA-246 > URL: https://issues.apache.org/jira/browse/CARBONDATA-246 > Project: CarbonData > Issue Type: Bug >Affects Versions: 0.1.0-incubating >Reporter: ravikiran > > if during compaction of 4 loads, for any executor if only first 3 loads task > is assigned then the col cardinality calculation based on the last segment > info will become wrong. > in this case the cardinality will go wrong for that executor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CARBONDATA-251) making the auto compaction as blocking call.
[ https://issues.apache.org/jira/browse/CARBONDATA-251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15499745#comment-15499745 ] ASF GitHub Bot commented on CARBONDATA-251: --- Github user asfgit closed the pull request at: https://github.com/apache/incubator-carbondata/pull/168 > making the auto compaction as blocking call. > > > Key: CARBONDATA-251 > URL: https://issues.apache.org/jira/browse/CARBONDATA-251 > Project: CarbonData > Issue Type: Bug >Reporter: ravikiran > > making the auto compaction as blocking call. > disabling the system level compaction lock feature. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (CARBONDATA-246) compaction is wrong in case if last segment is not assigned to an executor.
[ https://issues.apache.org/jira/browse/CARBONDATA-246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Venkata Ramana G reopened CARBONDATA-246: - > compaction is wrong in case if last segment is not assigned to an executor. > > > Key: CARBONDATA-246 > URL: https://issues.apache.org/jira/browse/CARBONDATA-246 > Project: CarbonData > Issue Type: Bug >Affects Versions: 0.1.0-incubating >Reporter: ravikiran >Assignee: ravikiran > Fix For: 0.2.0-incubating > > > if during compaction of 4 loads, for any executor if only first 3 loads task > is assigned then the col cardinality calculation based on the last segment > info will become wrong. > in this case the cardinality will go wrong for that executor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (CARBONDATA-246) compaction is wrong in case if last segment is not assigned to an executor.
[ https://issues.apache.org/jira/browse/CARBONDATA-246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Venkata Ramana G resolved CARBONDATA-246. - Resolution: Fixed Assignee: ravikiran Fix Version/s: 0.2.0-incubating > compaction is wrong in case if last segment is not assigned to an executor. > > > Key: CARBONDATA-246 > URL: https://issues.apache.org/jira/browse/CARBONDATA-246 > Project: CarbonData > Issue Type: Bug >Affects Versions: 0.1.0-incubating >Reporter: ravikiran >Assignee: ravikiran > Fix For: 0.2.0-incubating > > > if during compaction of 4 loads, for any executor if only first 3 loads task > is assigned then the col cardinality calculation based on the last segment > info will become wrong. > in this case the cardinality will go wrong for that executor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (CARBONDATA-251) making the auto compaction as blocking call.
[ https://issues.apache.org/jira/browse/CARBONDATA-251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Venkata Ramana G resolved CARBONDATA-251. - Resolution: Fixed Assignee: ravikiran Fix Version/s: 0.2.0-incubating > making the auto compaction as blocking call. > > > Key: CARBONDATA-251 > URL: https://issues.apache.org/jira/browse/CARBONDATA-251 > Project: CarbonData > Issue Type: Bug >Reporter: ravikiran >Assignee: ravikiran > Fix For: 0.2.0-incubating > > > making the auto compaction as blocking call. > disabling the system level compaction lock feature. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CARBONDATA-248) There was no header in driver statistics table and scan block time was always zero
[ https://issues.apache.org/jira/browse/CARBONDATA-248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15499755#comment-15499755 ] ASF GitHub Bot commented on CARBONDATA-248: --- Github user asfgit closed the pull request at: https://github.com/apache/incubator-carbondata/pull/163 > There was no header in driver statistics table and scan block time was always > zero > -- > > Key: CARBONDATA-248 > URL: https://issues.apache.org/jira/browse/CARBONDATA-248 > Project: CarbonData > Issue Type: Bug > Components: core >Reporter: Akash R Nilugal >Priority: Minor > Fix For: 0.2.0-incubating > > > Column header of total query cost was missing in driver statistics table > while recording statistics and scan block time was always zero while > recording query statistics. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (CARBONDATA-248) There was no header in driver statistics table and scan block time was always zero
[ https://issues.apache.org/jira/browse/CARBONDATA-248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Venkata Ramana G resolved CARBONDATA-248. - Resolution: Fixed Fix Version/s: 0.2.0-incubating > There was no header in driver statistics table and scan block time was always > zero > -- > > Key: CARBONDATA-248 > URL: https://issues.apache.org/jira/browse/CARBONDATA-248 > Project: CarbonData > Issue Type: Bug > Components: core >Reporter: Akash R Nilugal >Priority: Minor > Fix For: 0.2.0-incubating > > > Column header of total query cost was missing in driver statistics table > while recording statistics and scan block time was always zero while > recording query statistics. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CARBONDATA-253) Duplicate block loading when distribution is based on blocklet
[ https://issues.apache.org/jira/browse/CARBONDATA-253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15499810#comment-15499810 ] ASF GitHub Bot commented on CARBONDATA-253: --- Github user asfgit closed the pull request at: https://github.com/apache/incubator-carbondata/pull/170 > Duplicate block loading when distribution is based on blocklet > -- > > Key: CARBONDATA-253 > URL: https://issues.apache.org/jira/browse/CARBONDATA-253 > Project: CarbonData > Issue Type: Bug >Reporter: kumar vishal >Assignee: kumar vishal > > In case of query execution when distribution is based on blocklet same blocks > are getting loaded multiple times this is because hash code and equals method > contract is not same -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (CARBONDATA-253) Duplicate block loading when distribution is based on blocklet
[ https://issues.apache.org/jira/browse/CARBONDATA-253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Venkata Ramana G resolved CARBONDATA-253. - Resolution: Fixed Fix Version/s: 0.2.0-incubating > Duplicate block loading when distribution is based on blocklet > -- > > Key: CARBONDATA-253 > URL: https://issues.apache.org/jira/browse/CARBONDATA-253 > Project: CarbonData > Issue Type: Bug >Reporter: kumar vishal >Assignee: kumar vishal > Fix For: 0.2.0-incubating > > > In case of query execution when distribution is based on blocklet same blocks > are getting loaded multiple times this is because hash code and equals method > contract is not same -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CARBONDATA-250) Throw exception and fail the data load if provided MAXCOLUMNS value is not proper
[ https://issues.apache.org/jira/browse/CARBONDATA-250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15499849#comment-15499849 ] ASF GitHub Bot commented on CARBONDATA-250: --- Github user asfgit closed the pull request at: https://github.com/apache/incubator-carbondata/pull/169 > Throw exception and fail the data load if provided MAXCOLUMNS value is not > proper > - > > Key: CARBONDATA-250 > URL: https://issues.apache.org/jira/browse/CARBONDATA-250 > Project: CarbonData > Issue Type: Bug >Reporter: Manohar Vanam >Assignee: Manohar Vanam > Fix For: 0.2.0-incubating > > > If the provided MAXCOLUMNS value in load query is not proper, then throw > exception and fail the data load. > Impact Area : Load query with maxcolumns option value -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (CARBONDATA-252) Filter result is not proper when Double data type values with 0.0 and -0.0 will be used
[ https://issues.apache.org/jira/browse/CARBONDATA-252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Venkata Ramana G resolved CARBONDATA-252. - Resolution: Fixed Fix Version/s: 0.2.0-incubating > Filter result is not proper when Double data type values with 0.0 and -0.0 > will be used > --- > > Key: CARBONDATA-252 > URL: https://issues.apache.org/jira/browse/CARBONDATA-252 > Project: CarbonData > Issue Type: Bug >Reporter: Sujith >Assignee: Sujith >Priority: Minor > Fix For: 0.2.0-incubating > > > Filter result is not proper when Double data type values with 0.0 and -0.0 > will be used in filter model -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CARBONDATA-254) Code Inspection Optiminization
zhangshunyu created CARBONDATA-254: -- Summary: Code Inspection Optiminization Key: CARBONDATA-254 URL: https://issues.apache.org/jira/browse/CARBONDATA-254 Project: CarbonData Issue Type: Improvement Reporter: zhangshunyu Code Inspection Optiminization -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CARBONDATA-255) keyword SEGMENT should be used instead of LOAD In data management dml because LOAD is not supported now
zhangshunyu created CARBONDATA-255: -- Summary: keyword SEGMENT should be used instead of LOAD In data management dml because LOAD is not supported now Key: CARBONDATA-255 URL: https://issues.apache.org/jira/browse/CARBONDATA-255 Project: CarbonData Issue Type: Bug Affects Versions: 0.1.0-incubating Reporter: zhangshunyu Fix For: 0.2.0-incubating keyword SEGMENT should be used instead of LOAD In data management dml because LOAD is not supported now -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CARBONDATA-254) Code Inspection Optiminization
[ https://issues.apache.org/jira/browse/CARBONDATA-254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhangshunyu updated CARBONDATA-254: --- Affects Version/s: 0.1.0-incubating Fix Version/s: 0.2.0-incubating > Code Inspection Optiminization > -- > > Key: CARBONDATA-254 > URL: https://issues.apache.org/jira/browse/CARBONDATA-254 > Project: CarbonData > Issue Type: Improvement >Affects Versions: 0.1.0-incubating >Reporter: zhangshunyu > Fix For: 0.2.0-incubating > > > Code Inspection Optiminization -- This message was sent by Atlassian JIRA (v6.3.4#6332)
