[jira] [Commented] (CARBONDATA-4240) Properties present in https://github.com/apache/carbondata/blob/master/core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java which ar
[
https://issues.apache.org/jira/browse/CARBONDATA-4240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17403711#comment-17403711
]
Pratyaksh Sharma commented on CARBONDATA-4240:
--
There are few properties which are not getting used anywhere -
{color:#172b4d}spark.carbon.datamanagement.driver{color}
{color:#172b4d}carbon.lease.recovery.retry.count{color}
{color:#172b4d}carbon.lease.recovery.retry.interval{color}
{color:#172b4d}{color:#080808}carbon.number.of.cores.while.altPartition{color}{color}
Also there are few properties which are getting only for testing purpose-
{color:#172b4d}{color:#067d17}carbon.merge.index.in.segment{color}{color}
{color:#172b4d}{color:#067d17}sort.inmemory.size.inmb{color}{color}
> Properties present in
> https://github.com/apache/carbondata/blob/master/core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java
> which are not present in open source doc
> ---
>
> Key: CARBONDATA-4240
> URL: https://issues.apache.org/jira/browse/CARBONDATA-4240
> Project: CarbonData
> Issue Type: Bug
> Components: docs
>Affects Versions: 2.2.0
> Environment: Open source docs
>Reporter: Chetan Bhat
>Priority: Minor
>
> Properties present in
> https://github.com/apache/carbondata/blob/master/core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java
> which are not present in open source doc as mentioned below. These
> properties need to be updated in open source doc.
> carbon.storelocation
> carbon.blocklet.size
> carbon.properties.filepath
> carbon.date.format
> carbon.complex.delimiter.level.1
> carbon.complex.delimiter.level.2
> carbon.complex.delimiter.level.3
> carbon.complex.delimiter.level.4
> carbon.lock.class
> carbon.local.dictionary.enable
> carbon.local.dictionary.decoder.fallback
> spark.deploy.zookeeper.url
> carbon.data.file.version
> spark.carbon.hive.schema.store
> spark.carbon.datamanagement.driver
> spark.carbon.sessionstate.classname
> spark.carbon.sqlastbuilder.classname
> carbon.lease.recovery.retry.count
> carbon.lease.recovery.retry.interval
> carbon.index.schema.storage
> carbon.merge.index.in.segment
> carbon.number.of.cores.while.altPartition
> carbon.minor.compaction.size
> enable.unsafe.columnpage
> carbon.lucene.compression.mode
> sort.inmemory.size.inmb
> is.driver.instance
> carbon.input.metrics.update.interval
> carbon.use.bitset.pipe.line
> is.internal.load.call
> carbon.lucene.index.stop.words
> carbon.load.dateformat.setlenient.enable
> carbon.infilter.subquery.pushdown.enable
> broadcast.record.size
> carbon.indexserver.tempfolder.deletetime
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (CARBONDATA-4240) Properties present in https://github.com/apache/carbondata/blob/master/core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java whi
[
https://issues.apache.org/jira/browse/CARBONDATA-4240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17403711#comment-17403711
]
Pratyaksh Sharma edited comment on CARBONDATA-4240 at 8/24/21, 9:51 AM:
There are few properties which are not getting used anywhere -
{color:#172b4d}spark.carbon.datamanagement.driver{color}
{color:#172b4d}carbon.lease.recovery.retry.count{color}
{color:#172b4d}carbon.lease.recovery.retry.interval{color}
{color:#172b4d}carbon.number.of.cores.while.altPartition{color}
Also there are few properties which are getting used only for testing purpose-
carbon.merge.index.in.segment
sort.inmemory.size.inmb
was (Author: pratyaksh):
There are few properties which are not getting used anywhere -
{color:#172b4d}spark.carbon.datamanagement.driver{color}
{color:#172b4d}carbon.lease.recovery.retry.count{color}
{color:#172b4d}carbon.lease.recovery.retry.interval{color}
{color:#172b4d}{color:#080808}carbon.number.of.cores.while.altPartition{color}{color}
Also there are few properties which are getting only for testing purpose-
{color:#172b4d}{color:#067d17}carbon.merge.index.in.segment{color}{color}
{color:#172b4d}{color:#067d17}sort.inmemory.size.inmb{color}{color}
> Properties present in
> https://github.com/apache/carbondata/blob/master/core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java
> which are not present in open source doc
> ---
>
> Key: CARBONDATA-4240
> URL: https://issues.apache.org/jira/browse/CARBONDATA-4240
> Project: CarbonData
> Issue Type: Bug
> Components: docs
>Affects Versions: 2.2.0
> Environment: Open source docs
>Reporter: Chetan Bhat
>Priority: Minor
>
> Properties present in
> https://github.com/apache/carbondata/blob/master/core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java
> which are not present in open source doc as mentioned below. These
> properties need to be updated in open source doc.
> carbon.storelocation
> carbon.blocklet.size
> carbon.properties.filepath
> carbon.date.format
> carbon.complex.delimiter.level.1
> carbon.complex.delimiter.level.2
> carbon.complex.delimiter.level.3
> carbon.complex.delimiter.level.4
> carbon.lock.class
> carbon.local.dictionary.enable
> carbon.local.dictionary.decoder.fallback
> spark.deploy.zookeeper.url
> carbon.data.file.version
> spark.carbon.hive.schema.store
> spark.carbon.datamanagement.driver
> spark.carbon.sessionstate.classname
> spark.carbon.sqlastbuilder.classname
> carbon.lease.recovery.retry.count
> carbon.lease.recovery.retry.interval
> carbon.index.schema.storage
> carbon.merge.index.in.segment
> carbon.number.of.cores.while.altPartition
> carbon.minor.compaction.size
> enable.unsafe.columnpage
> carbon.lucene.compression.mode
> sort.inmemory.size.inmb
> is.driver.instance
> carbon.input.metrics.update.interval
> carbon.use.bitset.pipe.line
> is.internal.load.call
> carbon.lucene.index.stop.words
> carbon.load.dateformat.setlenient.enable
> carbon.infilter.subquery.pushdown.enable
> broadcast.record.size
> carbon.indexserver.tempfolder.deletetime
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (CARBONDATA-4119) User Input for GeoID column not validated.
[ https://issues.apache.org/jira/browse/CARBONDATA-4119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Indhumathi resolved CARBONDATA-4119. Fix Version/s: 2.3.0 Resolution: Fixed > User Input for GeoID column not validated. > -- > > Key: CARBONDATA-4119 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4119 > Project: CarbonData > Issue Type: Bug > Components: data-load >Affects Versions: 2.1.0 >Reporter: PURUJIT CHAUGULE >Priority: Minor > Fix For: 2.3.0 > > Time Spent: 2h 20m > Remaining Estimate: 0h > > * User Input for geoId column can be paired to multiple pairs of source > columns values (correct internally calculated geoID values are different for > such above source columns values). > * The advantage of using geoID is not applicable when taking user input for > GeoId column is not validated and user input values may differ from actual > internally calculated values. GeoID value is only generated internally if > user does not input the geoID column. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CARBONDATA-4238) Documentation Issue in Github Docs Link https://github.com/apache/carbondata/blob/master/docs/ddl-of-carbondata.md#add-columns
[ https://issues.apache.org/jira/browse/CARBONDATA-4238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Indhumathi resolved CARBONDATA-4238. Fix Version/s: 2.3.0 Resolution: Fixed > Documentation Issue in Github Docs Link > https://github.com/apache/carbondata/blob/master/docs/ddl-of-carbondata.md#add-columns > -- > > Key: CARBONDATA-4238 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4238 > Project: CarbonData > Issue Type: Bug > Components: docs >Affects Versions: 2.2.0 >Reporter: PURUJIT CHAUGULE >Priority: Minor > Fix For: 2.3.0 > > Attachments: Alter Add Complex.png, Alter Add > Complex_Error_message.png > > > [https://github.com/apache/carbondata/blob/master/docs/ddl-of-carbondata.md#add-columns] > * Example provided for Adding of only single-level Complex datatype > columns(only array and struct) has double level array column which is not > supported and needs to be changed to single level array column. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CARBONDATA-4236) Documentation correctness and link issues in https://github.com/apache/carbondata/blob/master/docs/
[ https://issues.apache.org/jira/browse/CARBONDATA-4236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Indhumathi resolved CARBONDATA-4236. Fix Version/s: 2.3.0 Resolution: Fixed > Documentation correctness and link issues in > https://github.com/apache/carbondata/blob/master/docs/ > --- > > Key: CARBONDATA-4236 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4236 > Project: CarbonData > Issue Type: Bug > Components: docs >Affects Versions: 2.2.0 > Environment: docs with content and examples verified on Spark 2.4.5 > and Spark 3.1.1 compatible carbon. >Reporter: Chetan Bhat >Priority: Minor > Fix For: 2.3.0 > > > In the documentation link > https://github.com/apache/carbondata/blob/master/docs/ > Issue 1 :- > In link -> > https://github.com/apache/carbondata/blob/master/docs/configuration-parameters.md > the "See detail" links does not open the target > "http://spark.apache.org/docs/latest/rdd-programming-guide.html#rdd-persistence"; > In link --> > https://github.com/apache/carbondata/blob/master/docs/documentation.md the > link "Apache CarbonData wiki" when clicked tries to open link > "https://cwiki.apache.org/confluence/display/CARBONDATA/CarbonData+Home"; the > target page cant be opened. Similarly the other links in the "External > Resources" section cant be opened due to the same error. > In link > https://github.com/apache/carbondata/blob/master/docs/faq.md#what-are-bad-records > the link "https://thrift.apache.org/docs/install"; when clicked does not open > the target page. > In link > https://github.com/apache/carbondata/blob/master/docs/quick-start-guide.md > when the "Spark website" link is clicked > https://spark.apache.org/downloads.html page is not opened. Also in same page > when the "Apache Spark Documentation" link is clicked the > "http://spark.apache.org/docs/latest/"; page is not opened. > In the link > https://github.com/apache/carbondata/blob/master/docs/release-guide.md > "Product Release Policy link" , "release signing guidelines" , "Apache Nexus > repository" and "repository.apache.org" when clicked the target pages are not > opening. > Issue 2:- > In link --> > https://github.com/apache/carbondata/blob/master/docs/configuration-parameters.md > the "To configure Ranges-based Compaction" to be changed to "To configure > Range-based Compaction" > Issue 3:- > In link --> > https://github.com/apache/carbondata/blob/master/docs/configuration-parameters.md > the "Making this true degrade the LOAD performance" to be changed to "Making > this true degrades the LOAD performance" > Issue 4 :- > In link --> > https://github.com/apache/carbondata/blob/master/docs/configuration-parameters.md > the "user an either set to true" to be changed to "user can either set to > true" -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CARBONDATA-4237) documentation issues in github master docs.
[
https://issues.apache.org/jira/browse/CARBONDATA-4237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Indhumathi resolved CARBONDATA-4237.
Fix Version/s: 2.3.0
Resolution: Fixed
> documentation issues in github master docs.
> ---
>
> Key: CARBONDATA-4237
> URL: https://issues.apache.org/jira/browse/CARBONDATA-4237
> Project: CarbonData
> Issue Type: Bug
> Components: docs
>Affects Versions: 2.2.0
> Environment: Contents verified on Spark 2.4.5 and Spark 3.1.1
>Reporter: PRIYESH RANJAN
>Priority: Minor
> Fix For: 2.3.0
>
>
> +Modification 1 :+
> [https://github.com/apache/carbondata/blob/master/docs/streaming-guide.md]
> Streaming table don't support alter table operation(alter add columns, drop
> column, rename column, change datatypes and rename table name) so In
> Constraint section of this doc ,it can be added.
>
> 0: jdbc:hive2://100-112-148-186:22550/> alter table uniqdata_alter add
> columns(id2 int);
> Error: org.apache.hive.service.cli.HiveSQLException: Error running query:
> org.apache.carbondata.common.exceptions.sql.MalformedCarbonCommandException:
> Alter table add column is not allowed for streaming table
> 0: jdbc:hive2://100-112-148-186:22550/> alter table uniqdata_alter drop
> columns(integer_column1);
> Error: org.apache.hive.service.cli.HiveSQLException: Error running query:
> org.apache.carbondata.common.exceptions.sql.MalformedCarbonCommandException:
> Alter table drop column is not allowed for streaming table.
> 0: jdbc:hive2://100-112-148-186:22550/> ALTER TABLE uniqdata_alter rename TO
> uniqdata_alterTable ;
> Error: org.apache.hive.service.cli.HiveSQLException: Error running query:
> org.apache.carbondata.common.exceptions.sql.MalformedCarbonCommandException:
> Alter rename table is not allowed for streaming table.
>
> +Modification 2 :+
> [https://github.com/apache/carbondata/blob/master/docs/file-structure-of-carbondata.md]
> Since Metadata folder contain segment, tablestatus and schema folder so
> dictionary file related content inside metadata folder can be removed from
> doc.
> eg : Metadata directory stores schema files, tablestatus and *dictionary
> files (including .dict, .dictmeta and .sortindex).* These line from doc can
> be modifed as Metadata directory stores schema files, tablestatus and
> segments details.
>
> +Modification 3 :+
> [https://github.com/apache/carbondata/blob/master/docs/sdk-guide.md]
> In the Quick Example section of following doc, it still converting date
> datatype to Integer value and timestamp datatype to long value whereas now
> they accept value as date and timestamp value respectively.
>
> {{while (reader.hasNext()) {
> Object[] row = (Object[]) reader.readNextRow();
> System.out.println(String.format("%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t",
> i, row[0], row[1], row[2], row[3], row[4], row[5],
> +*new Date((day * ((int) row[6]))), new Timestamp((long) row[7] / 1000)*+,
> row[8]
> ));
> {{can be modified to}}
> while (reader.hasNext()) {
> Object[] row = (Object[]) reader.readNextRow();
>
> System.out.println(String.format("%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t",
> i, row[0], row[1], row[2], row[3], row[4], row[5], +*row[6], row[7]*+,
> row[8], row[9]
> ));\{{}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CARBONDATA-4275) alter drop column fails for all primitive datatype except integer on carbon table with spark 3.1.1
PRIYESH RANJAN created CARBONDATA-4275: -- Summary: alter drop column fails for all primitive datatype except integer on carbon table with spark 3.1.1 Key: CARBONDATA-4275 URL: https://issues.apache.org/jira/browse/CARBONDATA-4275 Project: CarbonData Issue Type: Bug Components: data-query Affects Versions: 2.2.0 Environment: Spark 3.1.1 Reporter: PRIYESH RANJAN Steps:- *Issue : alter drop column fails for all primitive datatype except integer on carbon table with spark 3.1.1* CREATE TABLE uniqdata_alter(CUST_ID int ,CUST_NAME string,ACTIVE_EMUI_VERSION string, DOB timestamp, DOJ timestamp, BIGINT_COLUMN1 bigint,BIGINT_COLUMN2 bigint,DECIMAL_COLUMN1 decimal(30,10), DECIMAL_COLUMN2 decimal(36,10),Double_COLUMN1 double, Double_COLUMN2 double, INTEGER_COLUMN1 int) stored as carbondata ; *Scenario 1*: alter table uniqdata_alter drop columns(Double_COLUMN1); *Scenario 2 :* alter table uniqdata_alter drop columns(cust_name); *Scenario 3:* alter table uniqdata_alter drop columns(DECIMAL_COLUMN2); *Scenario 4:* alter table uniqdata_alter drop columns(BIGINT_COLUMN2); *Scenario 5:* alter table uniqdata_alter drop columns(DOB); *+error message for all above scenario:+* 0: jdbc:hive2://10.21.19.14:23040/default> alter table uniqdata_alter drop columns(Double_COLUMN1); Error: org.apache.hive.service.cli.HiveSQLException: Error running query: org.apache.carbondata.spark.exception.ProcessMetaDataException: operation failed for default.uniqdata_alter: Alter table drop column operation failed: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to alter table. The following columns have types incompatible with the existing columns in their respective positions : col at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.org$apache$spark$sql$hive$thriftserver$SparkExecuteStatementOperation$$execute(SparkExecuteStatementOperation.scala:361) at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.$anonfun$run$2(SparkExecuteStatementOperation.scala:263) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at org.apache.spark.sql.hive.thriftserver.SparkOperation.withLocalProperties(SparkOperation.scala:78) at org.apache.spark.sql.hive.thriftserver.SparkOperation.withLocalProperties$(SparkOperation.scala:62) at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.withLocalProperties(SparkExecuteStatementOperation.scala:43) at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.run(SparkExecuteStatementOperation.scala:263) at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.run(SparkExecuteStatementOperation.scala:258) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746) at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2.run(SparkExecuteStatementOperation.scala:272) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.carbondata.spark.exception.ProcessMetaDataException: operation failed for default.uniqdata_alter: Alter table drop column operation failed: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to alter table. The following columns have types incompatible with the existing columns in their respective positions : col at org.apache.spark.sql.execution.command.MetadataProcessOperation.throwMetadataException(package.scala:69) at org.apache.spark.sql.execution.command.MetadataProcessOperation.throwMetadataException$(package.scala:68) at org.apache.spark.sql.execution.command.MetadataCommand.throwMetadataException(package.scala:134) at org.apache.spark.sql.execution.command.schema.CarbonAlterTableDropColumnCommand.processMetadata(CarbonAlterTableDropColumnCommand.scala:216) at org.apache.spark.sql.execution.command.MetadataCommand.$anonfun$run$1(package.scala:137) at org.apache.spark.sql.execution.command.Auditable.runWithAudit(package.scala:118) at org.apache.spark.sql.execution.command.Auditable.runWithAudit$(package.scala:114) at org.apache.spark.sql.execution.command.MetadataCommand.runWithAudit(package.scala:134) at org.apache.spark.sql.execution.command.MetadataCommand.run(package.scala:137) at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70) at org.apache.spark.sql.execution.command.Ex
