Failed to create table using the Thrift Server

[f...@neusoft.com]

Hi community:
In our project, we used spark-cli to operate carbondata,it is ok.
But we failed to create table using the Thrift Server when we are using the 
existing JDBC process provided by spark.
So I do need your help, please!

hadoop:CHD3.0.0-cdh6.3.2.
hive:2.1.1-cdh6.3.2
Spark : 2.4.7

carbondata :
mvn -DskipTests -Pspark-2.4 -Pwindows clean package
- 
assembly\target\scala-2.11\apache-carbondata-2.1.0-bin-spark2.4.5-hadoop2.7.2.jar
- integration\hive\target\carbondata-hive-2.1.0.jar

We use carbondata as mentioned here 
(Option1).

== SQL script==
create table cb_test_env6(id int, env_name string) STORED AS carbondata
-^^^
error:
21/03/16 16:24:01 INFO thriftserver.SparkExecuteStatementOperation: Running 
query 'create table cb_test_env6(id int, env_name string) STORED AS carbondata' 
with 0a5535f1-2fdb-4577-a4cc-7dfd1f09fb6c
21/03/16 16:24:01 ERROR thriftserver.SparkExecuteStatementOperation: Error 
executing query, currentState RUNNING,
org.apache.spark.sql.catalyst.parser.ParseException:
Operation not allowed: STORED AS with file format 'carbondata'(line 1, pos 61)
at 
org.apache.spark.sql.catalyst.parser.ParserUtils$.operationNotAllowed(ParserUtils.scala:41)
at 
org.apache.spark.sql.execution.SparkSqlAstBuilder$$anonfun$visitGenericFileFormat$1.apply(SparkSqlParser.scala:1280)
at 
org.apache.spark.sql.execution.SparkSqlAstBuilder$$anonfun$visitGenericFileFormat$1.apply(SparkSqlParser.scala:1271)
at 
org.apache.spark.sql.catalyst.parser.ParserUtils$.withOrigin(ParserUtils.scala:108)
at 
org.apache.spark.sql.execution.SparkSqlAstBuilder.visitGenericFileFormat(SparkSqlParser.scala:1271)
at 
org.apache.spark.sql.execution.SparkSqlAstBuilder$$anonfun$visitCreateFileFormat$1.apply(SparkSqlParser.scala:1249)
at 
org.apache.spark.sql.execution.SparkSqlAstBuilder$$anonfun$visitCreateFileFormat$1.apply(SparkSqlParser.scala:1243)
at 
org.apache.spark.sql.catalyst.parser.ParserUtils$.withOrigin(ParserUtils.scala:108)
at 
org.apache.spark.sql.execution.SparkSqlAstBuilder.visitCreateFileFormat(SparkSqlParser.scala:1242)
at 
org.apache.spark.sql.execution.SparkSqlAstBuilder$$anonfun$visitCreateHiveTable$1$$anonfun$34.apply(SparkSqlParser.scala:1146)
at 
org.apache.spark.sql.execution.SparkSqlAstBuilder$$anonfun$visitCreateHiveTable$1$$anonfun$34.apply(SparkSqlParser.scala:1146)
at scala.Option.map(Option.scala:146)
at 
org.apache.spark.sql.execution.SparkSqlAstBuilder$$anonfun$visitCreateHiveTable$1.apply(SparkSqlParser.scala:1146)
at 
org.apache.spark.sql.execution.SparkSqlAstBuilder$$anonfun$visitCreateHiveTable$1.apply(SparkSqlParser.scala:1113)
at 
org.apache.spark.sql.catalyst.parser.ParserUtils$.withOrigin(ParserUtils.scala:108)
at 
org.apache.spark.sql.execution.SparkSqlAstBuilder.visitCreateHiveTable(SparkSqlParser.scala:1113)
at 
org.apache.spark.sql.execution.SparkSqlAstBuilder.visitCreateHiveTable(SparkSqlParser.scala:55)
at 
org.apache.spark.sql.catalyst.parser.SqlBaseParser$CreateHiveTableContext.accept(SqlBaseParser.java:1215)
at 
org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visit(AbstractParseTreeVisitor.java:18)
at 
org.apache.spark.sql.catalyst.parser.AstBuilder$$anonfun$visitSingleStatement$1.apply(AstBuilder.scala:72)
at 
org.apache.spark.sql.catalyst.parser.AstBuilder$$anonfun$visitSingleStatement$1.apply(AstBuilder.scala:72)
at 
org.apache.spark.sql.catalyst.parser.ParserUtils$.withOrigin(ParserUtils.scala:108)
at 
org.apache.spark.sql.catalyst.parser.AstBuilder.visitSingleStatement(AstBuilder.scala:71)
at 
org.apache.spark.sql.catalyst.parser.AbstractSqlParser$$anonfun$parsePlan$1.apply(ParseDriver.scala:70)
at 
org.apache.spark.sql.catalyst.parser.AbstractSqlParser$$anonfun$parsePlan$1.apply(ParseDriver.scala:69)
at 
org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:100)
at 
org.apache.spark.sql.execution.SparkSqlParser.parse(SparkSqlParser.scala:48)
at 
org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parsePlan(ParseDriver.scala:69)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:643)
at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:694)
at 
org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.org$apache$spark$sql$hive$thriftserver$SparkExecuteStatementOperation$$execute(SparkExecuteStatementOperation.scala:232)
at 
org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$1$$anon$2.run(SparkExecuteStatementOperation.scala:175)
at 
org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$1$$anon$2.run(SparkExec

Re: Re: How to update based on the value of the source table

[f...@neusoft.com]

Thank you very much for solving my problems.

The table A may be a wide table.   100~200 colums.

Will this feature be supported in later versions?

Regards


---
Confidentiality Notice: The information contained in this e-mail and any 
accompanying attachment(s)
is intended only for the use of the intended recipient and may be confidential 
and/or privileged of
Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader of 
this communication
is not the intended recipient,unauthorized use,forwarding, printing, storing, 
disclosure or copying
is strictly prohibited, and may be unlawful.If you have received this 
communication in error,please
immediately notify the sender by return e-mail, and delete the original message 
and all copies from
your system. Thank you.
---

Re: Re: How to update based on the value of the source table

[f...@neusoft.com]

hi，

table A:
id,column_name1, column_name2,column_name3
1,03OP20111684,,
2,01OP20111607,30,
3,01OP20111649,90,0020018393
4,01OP20111688,40,0020013010

table B ,:
id,column_name1, column_name2,column_name3
1,,10,
2,,40,
3,01OP20111638,,0020029804

For example: the first and second records only update column [column_name2], 
and the third record updates columns [column_name1,column_name3].
so ,I expect the updated result:
table A:
id,column_name1, column_name2,column_name3
1,03OP20111684,10,
2,01OP20111607,40,
3,01OP20111649,90,0020018393
4,01OP20111688,40,0020013010

Help me how to write the SQL for update ?

Thanks.

neusoft TSD 刘feng


From: akashrn5
Date: 2020-10-09 12:56
To: dev@carbondata.apache.org
Subject: Re: How to update based on the value of the source table
Hi,

I checked our test cases, we have a similar test case and works fine.
You can refer "update carbon table[select from source table with where and
exist]" in
UpdateCarbonTableTestCase.scala,

In that test case, you can have a query like below

*sql("""update iud.dest11 d set (d.c3, d.c5 ) = (select s.c33,s.c55 from
iud.source2 s where d.c1 = s.c11)""").show()*

It works fine. I didn't get the actual question, where you said

*when B.column_name1 == '' ,  no update operation is performed.
when B.column_name1 !='',  update operation is performed.*

can you please tell me what value are you trying to update?
If you can give the complete test case, it will be useful to check.

Also, please join the slack channel for detailed discussion.

https://join.slack.com/t/carbondataworkspace/shared_invite/zt-g8sv1g92-pr3GTvjrW5H9DVvNl6H2dg

Thanks,

Regards,
Akash R Nilugal



--
Sent from: 
http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
---
Confidentiality Notice: The information contained in this e-mail and any 
accompanying attachment(s)
is intended only for the use of the intended recipient and may be confidential 
and/or privileged of
Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader of 
this communication
is not the intended recipient,unauthorized use,forwarding, printing, storing, 
disclosure or copying
is strictly prohibited, and may be unlawful.If you have received this 
communication in error,please
immediately notify the sender by return e-mail, and delete the original message 
and all copies from
your system. Thank you.
---

How to update based on the value of the source table

[f...@neusoft.com]

hi dev:
 updating the CarbonData Table :
 UPDATE A SET (column_name1, column_name2,column_name3) =(select column_name1, 
column_name2,column_name3 from B   WHERE A.id = B.id  );

table  B is a mirror table of table A .
The fields in B are updated to table A
I want to do batch updates:
Whether to support updating based on the value of table B [column_name1, 
column_name2, column_name3] in B,
for example, when B.column_name1 == '' ,  no update operation is performed.
  when B.column_name1 !='',  update operation is performed.

neusoft TSD salute

---
Confidentiality Notice: The information contained in this e-mail and any 
accompanying attachment(s)
is intended only for the use of the intended recipient and may be confidential 
and/or privileged of
Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader of 
this communication
is not the intended recipient,unauthorized use,forwarding, printing, storing, 
disclosure or copying
is strictly prohibited, and may be unlawful.If you have received this 
communication in error,please
immediately notify the sender by return e-mail, and delete the original message 
and all copies from
your system. Thank you.
---

hello

[f...@neusoft.com]

TSD salute
---
Confidentiality Notice: The information contained in this e-mail and any 
accompanying attachment(s)
is intended only for the use of the intended recipient and may be confidential 
and/or privileged of
Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader of 
this communication
is not the intended recipient,unauthorized use,forwarding, printing, storing, 
disclosure or copying
is strictly prohibited, and may be unlawful.If you have received this 
communication in error,please
immediately notify the sender by return e-mail, and delete the original message 
and all copies from
your system. Thank you.
---