[jira] [Commented] (CARBONDATA-311) Log the data size of blocklet during data load.
[
https://issues.apache.org/jira/browse/CARBONDATA-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15584415#comment-15584415
]
ASF GitHub Bot commented on CARBONDATA-311:
---
Github user Vimal-Das commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/231#discussion_r83780338
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/store/writer/CarbonFactDataWriterImplForIntIndexAndAggBlock.java
---
@@ -250,6 +254,7 @@ public NodeHolder
buildDataNodeHolder(IndexStorage[] keyStorageArray, byt
}
long blockletDataSize =
holder.getKeyArray().length + holder.getDataArray().length +
indexBlockSize;
+LOGGER.info("A new blocklet is added, its data size is: " +
blockletDataSize + " Byte");
--- End diff --
Move this line after call to writeDataToFile, because if writeDataToFile
fails, we should not log that a blocklet is added.
> Log the data size of blocklet during data load.
> ---
>
> Key: CARBONDATA-311
> URL: https://issues.apache.org/jira/browse/CARBONDATA-311
> Project: CarbonData
> Issue Type: Improvement
>Affects Versions: 0.1.1-incubating
>Reporter: zhangshunyu
>Assignee: zhangshunyu
>Priority: Minor
> Fix For: 0.2.0-incubating
>
>
> Log the data size of blocklet during data load.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (CARBONDATA-317) CSV having only space char is throwing NullPointerException
[ https://issues.apache.org/jira/browse/CARBONDATA-317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Venkata Ramana G resolved CARBONDATA-317. - Resolution: Fixed Fix Version/s: 0.2.0-incubating > CSV having only space char is throwing NullPointerException > --- > > Key: CARBONDATA-317 > URL: https://issues.apache.org/jira/browse/CARBONDATA-317 > Project: CarbonData > Issue Type: Bug >Affects Versions: 0.1.1-incubating >Reporter: Mohammad Shahid Khan >Assignee: Mohammad Shahid Khan >Priority: Minor > Fix For: 0.2.0-incubating > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CARBONDATA-317) CSV having only space char is throwing NullPointerException
[ https://issues.apache.org/jira/browse/CARBONDATA-317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Venkata Ramana G updated CARBONDATA-317: Affects Version/s: 0.1.0-incubating > CSV having only space char is throwing NullPointerException > --- > > Key: CARBONDATA-317 > URL: https://issues.apache.org/jira/browse/CARBONDATA-317 > Project: CarbonData > Issue Type: Bug >Affects Versions: 0.1.1-incubating >Reporter: Mohammad Shahid Khan >Assignee: Mohammad Shahid Khan >Priority: Minor > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CARBONDATA-317) CSV having only space char is throwing NullPointerException
[ https://issues.apache.org/jira/browse/CARBONDATA-317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15583127#comment-15583127 ] ASF GitHub Bot commented on CARBONDATA-317: --- Github user asfgit closed the pull request at: https://github.com/apache/incubator-carbondata/pull/237 > CSV having only space char is throwing NullPointerException > --- > > Key: CARBONDATA-317 > URL: https://issues.apache.org/jira/browse/CARBONDATA-317 > Project: CarbonData > Issue Type: Bug >Affects Versions: 0.1.1-incubating >Reporter: Mohammad Shahid Khan >Assignee: Mohammad Shahid Khan >Priority: Minor > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CARBONDATA-317) CSV having only space char is throwing NullPointerException
[ https://issues.apache.org/jira/browse/CARBONDATA-317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Venkata Ramana G updated CARBONDATA-317: Affects Version/s: (was: 0.1.0-incubating) 0.1.1-incubating > CSV having only space char is throwing NullPointerException > --- > > Key: CARBONDATA-317 > URL: https://issues.apache.org/jira/browse/CARBONDATA-317 > Project: CarbonData > Issue Type: Bug >Affects Versions: 0.1.1-incubating >Reporter: Mohammad Shahid Khan >Assignee: Mohammad Shahid Khan >Priority: Minor > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CARBONDATA-319) Bad Records logging for column LONG data type is not proper
[ https://issues.apache.org/jira/browse/CARBONDATA-319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Venkata Ramana G updated CARBONDATA-319: Priority: Trivial (was: Minor) > Bad Records logging for column LONG data type is not proper > --- > > Key: CARBONDATA-319 > URL: https://issues.apache.org/jira/browse/CARBONDATA-319 > Project: CarbonData > Issue Type: Bug >Reporter: Sujith >Assignee: Sujith >Priority: Trivial > Fix For: 0.2.0-incubating > > > Bad Records logging for column data type is not proper in case of long, > carbon system while creating the table metadata uses BIGINT instead of LONG , > internally it converts the bigint to long type and processes , while > processing the data if any long type data is having issue it will be logged > into the bad record with data type Long which is not proper since as per the > metadata the datatype of column is BIGINT. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CARBONDATA-319) Bad Records logging for column LONG data type is not proper
[ https://issues.apache.org/jira/browse/CARBONDATA-319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Venkata Ramana G updated CARBONDATA-319: Summary: Bad Records logging for column LONG data type is not proper (was: Bad Records logging for column data type is not proper) > Bad Records logging for column LONG data type is not proper > --- > > Key: CARBONDATA-319 > URL: https://issues.apache.org/jira/browse/CARBONDATA-319 > Project: CarbonData > Issue Type: Bug >Reporter: Sujith >Assignee: Sujith >Priority: Minor > Fix For: 0.2.0-incubating > > > Bad Records logging for column data type is not proper in case of long, > carbon system while creating the table metadata uses BIGINT instead of LONG , > internally it converts the bigint to long type and processes , while > processing the data if any long type data is having issue it will be logged > into the bad record with data type Long which is not proper since as per the > metadata the datatype of column is BIGINT. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (CARBONDATA-319) Bad Records logging for column data type is not proper
[ https://issues.apache.org/jira/browse/CARBONDATA-319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Venkata Ramana G resolved CARBONDATA-319. - Resolution: Fixed Fix Version/s: 0.2.0-incubating > Bad Records logging for column data type is not proper > -- > > Key: CARBONDATA-319 > URL: https://issues.apache.org/jira/browse/CARBONDATA-319 > Project: CarbonData > Issue Type: Bug >Reporter: Sujith >Assignee: Sujith >Priority: Minor > Fix For: 0.2.0-incubating > > > Bad Records logging for column data type is not proper in case of long, > carbon system while creating the table metadata uses BIGINT instead of LONG , > internally it converts the bigint to long type and processes , while > processing the data if any long type data is having issue it will be logged > into the bad record with data type Long which is not proper since as per the > metadata the datatype of column is BIGINT. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CARBONDATA-319) Bad Records logging for column data type is not proper
[ https://issues.apache.org/jira/browse/CARBONDATA-319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15583089#comment-15583089 ] ASF GitHub Bot commented on CARBONDATA-319: --- Github user asfgit closed the pull request at: https://github.com/apache/incubator-carbondata/pull/241 > Bad Records logging for column data type is not proper > -- > > Key: CARBONDATA-319 > URL: https://issues.apache.org/jira/browse/CARBONDATA-319 > Project: CarbonData > Issue Type: Bug >Reporter: Sujith >Assignee: Sujith >Priority: Minor > > Bad Records logging for column data type is not proper in case of long, > carbon system while creating the table metadata uses BIGINT instead of LONG , > internally it converts the bigint to long type and processes , while > processing the data if any long type data is having issue it will be logged > into the bad record with data type Long which is not proper since as per the > metadata the datatype of column is BIGINT. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (CARBONDATA-320) problem when dropped a table during all data nodes are down.
[ https://issues.apache.org/jira/browse/CARBONDATA-320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Venkata Ramana G resolved CARBONDATA-320. - Resolution: Fixed > problem when dropped a table during all data nodes are down. > - > > Key: CARBONDATA-320 > URL: https://issues.apache.org/jira/browse/CARBONDATA-320 > Project: CarbonData > Issue Type: Bug > Components: sql >Affects Versions: 0.1.0-incubating >Reporter: ravikiran >Assignee: ravikiran >Priority: Minor > Fix For: 0.2.0-incubating > > > when dropped a table during all data nodes are down. sometimes the drop is > failing and the files will remain in hdfs. but from hive the table will be > removed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CARBONDATA-320) problem when dropped a table during all data nodes are down.
[ https://issues.apache.org/jira/browse/CARBONDATA-320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15583071#comment-15583071 ] ASF GitHub Bot commented on CARBONDATA-320: --- Github user asfgit closed the pull request at: https://github.com/apache/incubator-carbondata/pull/245 > problem when dropped a table during all data nodes are down. > - > > Key: CARBONDATA-320 > URL: https://issues.apache.org/jira/browse/CARBONDATA-320 > Project: CarbonData > Issue Type: Bug > Components: sql >Affects Versions: 0.1.0-incubating >Reporter: ravikiran >Assignee: ravikiran >Priority: Minor > Fix For: 0.2.0-incubating > > > when dropped a table during all data nodes are down. sometimes the drop is > failing and the files will remain in hdfs. but from hive the table will be > removed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CARBONDATA-301) 6. Add SortProcessorStep which sorts the data as per dimension order and write the sorted files to temp location.
[ https://issues.apache.org/jira/browse/CARBONDATA-301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15582964#comment-15582964 ] ASF GitHub Bot commented on CARBONDATA-301: --- GitHub user ravipesala opened a pull request: https://github.com/apache/incubator-carbondata/pull/247 [CARBONDATA-301] Added Sort processor step for data loading. Add SortProcessorStep which sorts the data as per dimension order and write the sorted files to temp location and merge sort it. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ravipesala/incubator-carbondata sort-step Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-carbondata/pull/247.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #247 commit f3d5137b23e2dd0336a5d39bbd79f3c0996bc402 Author: ravipesala Date: 2016-10-17T18:06:37Z Added Sort processor step for dataloading. > 6. Add SortProcessorStep which sorts the data as per dimension order and > write the sorted files to temp location. > - > > Key: CARBONDATA-301 > URL: https://issues.apache.org/jira/browse/CARBONDATA-301 > Project: CarbonData > Issue Type: Sub-task >Reporter: Ravindra Pesala > Fix For: 0.2.0-incubating > > > Add SortProcessorStep which sorts the data as per dimension order and write > the sorted files to temp location. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CARBONDATA-304) Load data failure when set table_blocksize=2048
[
https://issues.apache.org/jira/browse/CARBONDATA-304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Venkata Ramana G updated CARBONDATA-304:
Affects Version/s: 0.1.0-incubating
> Load data failure when set table_blocksize=2048
> ---
>
> Key: CARBONDATA-304
> URL: https://issues.apache.org/jira/browse/CARBONDATA-304
> Project: CarbonData
> Issue Type: Bug
>Affects Versions: 0.1.0-incubating, 0.1.1-incubating
>Reporter: Gin-zhj
>Assignee: Gin-zhj
>Priority: Minor
> Fix For: 0.2.0-incubating
>
>
> First ,create a table with table_blocksize=2048
> CREATE TABLE IF NOT EXISTS t3 (ID Int, date Timestamp, country String, name
> String, phonetype String, serialname String, salary Int) STORED BY
> 'carbondata' TBLPROPERTIES('table_blocksize'='2048');
> Then load data, failure and catch exception:
> org.apache.carbondata.processing.store.writer.exception.CarbonDataWriterException:
> Problem while copying file from local store to carbon store
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (CARBONDATA-304) Load data failure when set table_blocksize=2048
[
https://issues.apache.org/jira/browse/CARBONDATA-304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Venkata Ramana G resolved CARBONDATA-304.
-
Resolution: Fixed
Fix Version/s: 0.2.0-incubating
> Load data failure when set table_blocksize=2048
> ---
>
> Key: CARBONDATA-304
> URL: https://issues.apache.org/jira/browse/CARBONDATA-304
> Project: CarbonData
> Issue Type: Bug
>Affects Versions: 0.1.0-incubating, 0.1.1-incubating
>Reporter: Gin-zhj
>Assignee: Gin-zhj
>Priority: Minor
> Fix For: 0.2.0-incubating
>
>
> First ,create a table with table_blocksize=2048
> CREATE TABLE IF NOT EXISTS t3 (ID Int, date Timestamp, country String, name
> String, phonetype String, serialname String, salary Int) STORED BY
> 'carbondata' TBLPROPERTIES('table_blocksize'='2048');
> Then load data, failure and catch exception:
> org.apache.carbondata.processing.store.writer.exception.CarbonDataWriterException:
> Problem while copying file from local store to carbon store
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CARBONDATA-304) Load data failure when set table_blocksize=2048
[
https://issues.apache.org/jira/browse/CARBONDATA-304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Venkata Ramana G updated CARBONDATA-304:
Affects Version/s: 0.1.1-incubating
> Load data failure when set table_blocksize=2048
> ---
>
> Key: CARBONDATA-304
> URL: https://issues.apache.org/jira/browse/CARBONDATA-304
> Project: CarbonData
> Issue Type: Bug
>Affects Versions: 0.1.0-incubating, 0.1.1-incubating
>Reporter: Gin-zhj
>Assignee: Gin-zhj
>Priority: Minor
> Fix For: 0.2.0-incubating
>
>
> First ,create a table with table_blocksize=2048
> CREATE TABLE IF NOT EXISTS t3 (ID Int, date Timestamp, country String, name
> String, phonetype String, serialname String, salary Int) STORED BY
> 'carbondata' TBLPROPERTIES('table_blocksize'='2048');
> Then load data, failure and catch exception:
> org.apache.carbondata.processing.store.writer.exception.CarbonDataWriterException:
> Problem while copying file from local store to carbon store
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CARBONDATA-304) Load data failure when set table_blocksize=2048
[
https://issues.apache.org/jira/browse/CARBONDATA-304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Venkata Ramana G updated CARBONDATA-304:
Priority: Minor (was: Major)
> Load data failure when set table_blocksize=2048
> ---
>
> Key: CARBONDATA-304
> URL: https://issues.apache.org/jira/browse/CARBONDATA-304
> Project: CarbonData
> Issue Type: Bug
>Reporter: Gin-zhj
>Assignee: Gin-zhj
>Priority: Minor
>
> First ,create a table with table_blocksize=2048
> CREATE TABLE IF NOT EXISTS t3 (ID Int, date Timestamp, country String, name
> String, phonetype String, serialname String, salary Int) STORED BY
> 'carbondata' TBLPROPERTIES('table_blocksize'='2048');
> Then load data, failure and catch exception:
> org.apache.carbondata.processing.store.writer.exception.CarbonDataWriterException:
> Problem while copying file from local store to carbon store
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CARBONDATA-304) Load data failure when set table_blocksize=2048
[
https://issues.apache.org/jira/browse/CARBONDATA-304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15582841#comment-15582841
]
ASF GitHub Bot commented on CARBONDATA-304:
---
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-carbondata/pull/227
> Load data failure when set table_blocksize=2048
> ---
>
> Key: CARBONDATA-304
> URL: https://issues.apache.org/jira/browse/CARBONDATA-304
> Project: CarbonData
> Issue Type: Bug
>Reporter: Gin-zhj
>Assignee: Gin-zhj
>
> First ,create a table with table_blocksize=2048
> CREATE TABLE IF NOT EXISTS t3 (ID Int, date Timestamp, country String, name
> String, phonetype String, serialname String, salary Int) STORED BY
> 'carbondata' TBLPROPERTIES('table_blocksize'='2048');
> Then load data, failure and catch exception:
> org.apache.carbondata.processing.store.writer.exception.CarbonDataWriterException:
> Problem while copying file from local store to carbon store
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CARBONDATA-321) Query is failing on the change of blocklet size
[ https://issues.apache.org/jira/browse/CARBONDATA-321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15581933#comment-15581933 ] ASF GitHub Bot commented on CARBONDATA-321: --- GitHub user sujith71955 opened a pull request: https://github.com/apache/incubator-carbondata/pull/246 [CARBONDATA-321] If user changes the blocklet size the queries will b… **Problem:** If user changes the blocklet size (50) the queries will be failed. Currently byte size calculation in number compressor is int array length * bit length=x bytesize= x/8+x%8; when cardinality is 50 then byte size will be 3 but to calculate the actual int array size of index reverse based on the reverse calculation it is coming 4 int arrayLength=(3*8)/6 **Solution:** We need to round off bit length to nearest value which should be divisible by 8 You can merge this pull request into a Git repository by running: $ git pull https://github.com/sujith71955/incubator-carbondata master_BlockletIssue Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-carbondata/pull/246.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #246 commit e1d91197b4173625a1cb8c469cec95290bb93185 Author: sujith71955 Date: 2016-10-17T11:12:16Z [CARBONDATA-321] If user changes the blocklet size the queries will be failed. this is happening since the inverted index compression is not been handled properly > Query is failing on the change of blocklet size > - > > Key: CARBONDATA-321 > URL: https://issues.apache.org/jira/browse/CARBONDATA-321 > Project: CarbonData > Issue Type: Bug >Reporter: Sujith >Assignee: Sujith > > If user changes the blocklet size the queries will be failed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CARBONDATA-321) Query is failing on the change of blocklet size
Sujith created CARBONDATA-321: - Summary: Query is failing on the change of blocklet size Key: CARBONDATA-321 URL: https://issues.apache.org/jira/browse/CARBONDATA-321 Project: CarbonData Issue Type: Bug Reporter: Sujith Assignee: Sujith If user changes the blocklet size the queries will be failed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CARBONDATA-320) problem when dropped a table during all data nodes are down.
[ https://issues.apache.org/jira/browse/CARBONDATA-320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15581781#comment-15581781 ] ASF GitHub Bot commented on CARBONDATA-320: --- GitHub user ravikiran23 opened a pull request: https://github.com/apache/incubator-carbondata/pull/245 [CARBONDATA-320] problem during drop of table when all datanodes are down. Problem : when all the data nodes are down and user executes drop table. then drop table will some times fail to delete the files from hdfs. but it will remove the hive entry. Solution : Removed the unnecessary checks present in the drop table . so that the deletion of the files wont be skipped. changed the lock type of the drop to new lock type. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ravikiran23/incubator-carbondata dropTableHandling Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-carbondata/pull/245.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #245 commit cf31b3e6a0695fe1a32fc035819a522b3f86c779 Author: ravikiran Date: 2016-10-17T09:49:19Z [CARBONDATA-320] if the datanodes are down and drop is executed then randomly the drop is failing and the files of table will remain in file system. > problem when dropped a table during all data nodes are down. > - > > Key: CARBONDATA-320 > URL: https://issues.apache.org/jira/browse/CARBONDATA-320 > Project: CarbonData > Issue Type: Bug > Components: sql >Affects Versions: 0.1.0-incubating >Reporter: ravikiran >Assignee: ravikiran >Priority: Minor > Fix For: 0.2.0-incubating > > > when dropped a table during all data nodes are down. sometimes the drop is > failing and the files will remain in hdfs. but from hive the table will be > removed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CARBONDATA-320) problem when dropped a table during all data nodes are down.
ravikiran created CARBONDATA-320: Summary: problem when dropped a table during all data nodes are down. Key: CARBONDATA-320 URL: https://issues.apache.org/jira/browse/CARBONDATA-320 Project: CarbonData Issue Type: Bug Components: sql Affects Versions: 0.1.0-incubating Reporter: ravikiran Assignee: ravikiran Priority: Minor Fix For: 0.2.0-incubating when dropped a table during all data nodes are down. sometimes the drop is failing and the files will remain in hdfs. but from hive the table will be removed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
