[ https://issues.apache.org/jira/browse/KYLIN-3857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16789136#comment-16789136 ]
XiaoXiang Yu edited comment on KYLIN-3857 at 3/11/19 4:59 AM: -------------------------------------------------------------- Logs in My DEV env (with quote turn off): KYLIN [ INFO ] 03-11 12:53:39.114 org.apache.kylin.common.util.BufferedLogger.log(BufferedLogger.java:38) from Scheduler 1705744661 Job 6d64fe74-ea50-76a6-bbd1-f85210984416-129 > Create and distribute table, cmd: KYLIN [ INFO ] 03-11 12:53:39.115 org.apache.kylin.common.util.BufferedLogger.log(BufferedLogger.java:38) from Scheduler 1705744661 Job 6d64fe74-ea50-76a6-bbd1-f85210984416-129 > hive -e "USE lacus; DROP TABLE IF EXISTS kylin_intermediate_samplecube_e1c8bdcc_d55b_3807_25e2_b8d2b6fccabe; CREATE EXTERNAL TABLE IF NOT EXISTS kylin_intermediate_samplecube_e1c8bdcc_d55b_3807_25e2_b8d2b6fccabe ( KYLIN_SALES_LSTG_FORMAT_NAME string ,KYLIN_SALES_LSTG_SITE_ID int ,KYLIN_SALES_SELLER_ID bigint ,KYLIN_ACCOUNT_ACCOUNT_COUNTRY string ,KYLIN_SALES_PRICE decimal(19,4) ,KYLIN_SALES_ITEM_COUNT bigint ) STORED AS SEQUENCEFILE LOCATION 'hdfs://sandbox.hortonworks.com:8020/kylin/lacus/lacus-kylin_metadata/kylin-6d64fe74-ea50-76a6-bbd1-f85210984416/kylin_intermediate_samplecube_e1c8bdcc_d55b_3807_25e2_b8d2b6fccabe'; ALTER TABLE kylin_intermediate_samplecube_e1c8bdcc_d55b_3807_25e2_b8d2b6fccabe SET TBLPROPERTIES('auto.purge'='true'); INSERT OVERWRITE TABLE kylin_intermediate_samplecube_e1c8bdcc_d55b_3807_25e2_b8d2b6fccabe SELECT KYLIN_SALES.LSTG_FORMAT_NAME as KYLIN_SALES_LSTG_FORMAT_NAME ,KYLIN_SALES.LSTG_SITE_ID as KYLIN_SALES_LSTG_SITE_ID ,KYLIN_SALES.SELLER_ID as KYLIN_SALES_SELLER_ID ,KYLIN_ACCOUNT.ACCOUNT_COUNTRY as KYLIN_ACCOUNT_ACCOUNT_COUNTRY ,KYLIN_SALES.PRICE as KYLIN_SALES_PRICE ,KYLIN_SALES.ITEM_COUNT as KYLIN_SALES_ITEM_COUNT FROM LACUS.KYLIN_SALES as KYLIN_SALES LEFT JOIN LACUS.KYLIN_ACCOUNT as KYLIN_ACCOUNT ON KYLIN_SALES.SELLER_ID = KYLIN_ACCOUNT.ACCOUNT_ID LEFT JOIN LACUS.KYLIN_COUNTRY as KYLIN_COUNTRY ON KYLIN_ACCOUNT.ACCOUNT_COUNTRY = KYLIN_COUNTRY.COUNTRY WHERE 1=1; " --hiveconf mapreduce.map.output.compress.codec=org.apache.hadoop.io.compress.SnappyCodec --hiveconf dfs.replication=2 --hiveconf hive.exec.compress.output=true KYLIN [ INFO ] 03-11 12:53:39.115 org.apache.kylin.common.util.SSHClient.execCommand(SSHClient.java:262) from Scheduler 1705744661 Job 6d64fe74-ea50-76a6-bbd1-f85210984416-129 > [root@sandbox] Execute command: hive -e "USE lacus; DROP TABLE IF EXISTS kylin_intermediate_samplecube_e1c8bdcc_d55b_3807_25e2_b8d2b6fccabe; CREATE EXTERNAL TABLE IF NOT EXISTS kylin_intermediate_samplecube_e1c8bdcc_d55b_3807_25e2_b8d2b6fccabe ( KYLIN_SALES_LSTG_FORMAT_NAME string ,KYLIN_SALES_LSTG_SITE_ID int ,KYLIN_SALES_SELLER_ID bigint ,KYLIN_ACCOUNT_ACCOUNT_COUNTRY string ,KYLIN_SALES_PRICE decimal(19,4) ,KYLIN_SALES_ITEM_COUNT bigint ) STORED AS SEQUENCEFILE LOCATION 'hdfs://sandbox.hortonworks.com:8020/kylin/lacus/lacus-kylin_metadata/kylin-6d64fe74-ea50-76a6-bbd1-f85210984416/kylin_intermediate_samplecube_e1c8bdcc_d55b_3807_25e2_b8d2b6fccabe'; ALTER TABLE kylin_intermediate_samplecube_e1c8bdcc_d55b_3807_25e2_b8d2b6fccabe SET TBLPROPERTIES('auto.purge'='true'); INSERT OVERWRITE TABLE kylin_intermediate_samplecube_e1c8bdcc_d55b_3807_25e2_b8d2b6fccabe SELECT KYLIN_SALES.LSTG_FORMAT_NAME as KYLIN_SALES_LSTG_FORMAT_NAME ,KYLIN_SALES.LSTG_SITE_ID as KYLIN_SALES_LSTG_SITE_ID ,KYLIN_SALES.SELLER_ID as KYLIN_SALES_SELLER_ID ,KYLIN_ACCOUNT.ACCOUNT_COUNTRY as KYLIN_ACCOUNT_ACCOUNT_COUNTRY ,KYLIN_SALES.PRICE as KYLIN_SALES_PRICE ,KYLIN_SALES.ITEM_COUNT as KYLIN_SALES_ITEM_COUNT FROM LACUS.KYLIN_SALES as KYLIN_SALES LEFT JOIN LACUS.KYLIN_ACCOUNT as KYLIN_ACCOUNT ON KYLIN_SALES.SELLER_ID = KYLIN_ACCOUNT.ACCOUNT_ID LEFT JOIN LACUS.KYLIN_COUNTRY as KYLIN_COUNTRY ON KYLIN_ACCOUNT.ACCOUNT_COUNTRY = KYLIN_COUNTRY.COUNTRY WHERE 1=1; " --hiveconf mapreduce.map.output.compress.codec=org.apache.hadoop.io.compress.SnappyCodec --hiveconf dfs.replication=2 --hiveconf hive.exec.compress.output=true KYLIN [ INFO ] 03-11 12:53:42.237 org.apache.kylin.common.util.BufferedLogger.log(BufferedLogger.java:38) from Scheduler 1705744661 Job 6d64fe74-ea50-76a6-bbd1-f85210984416-129 > WARNING: Use "yarn jar" to launch YARN applications. KYLIN [ INFO ] 03-11 12:53:43.241 org.apache.kylin.common.util.BufferedLogger.log(BufferedLogger.java:38) from Scheduler 1705744661 Job 6d64fe74-ea50-76a6-bbd1-f85210984416-129 > Logging initialized using configuration in file:/etc/hive/2.4.0.0-169/0/hive-log4j.properties KYLIN [ INFO ] 03-11 12:56:07.045 org.apache.kylin.common.util.SSHClient.execCommand(SSHClient.java:262) from Scheduler 1705744661 Job 6d64fe74-ea50-76a6-bbd1-f85210984416-221 > [root@sandbox] Execute command: hive -e "USE lacus; DROP TABLE IF EXISTS lacus.kylin_intermediate_samplecube_e1c8bdcc_d55b_3807_25e2_b8d2b6fccabe; " --hiveconf mapreduce.map.output.compress.codec=org.apache.hadoop.io.compress.SnappyCodec --hiveconf dfs.replication=2 --hiveconf hive.exec.compress.output=true was (Author: hit_lacus): Screenshots in My DEV env (with quote turn off): > add parameter to change sql quoting character for sqoop > ------------------------------------------------------- > > Key: KYLIN-3857 > URL: https://issues.apache.org/jira/browse/KYLIN-3857 > Project: Kylin > Issue Type: Bug > Reporter: Davide Malagoli > Assignee: XiaoXiang Yu > Priority: Major > Attachments: image-2019-03-11-11-46-05-946.png, > image-2019-03-11-11-46-14-305.png, image-2019-03-11-11-46-20-664.png, > image-2019-03-11-11-46-31-966.png, sqoop-error.log, sqoop-script.sh > > > I've set up Kylin to access a SQL server using JDBC and do the ETL step as > described in > [http://kylin.apache.org/docs/tutorial/setup_jdbc_datasource.html] > but when the "Sqoop To Flat Hive Table" runs it terminates with an error > shown in the attached log. > It seems that the root cause is the presence of the '`' character in the > query, which has no meaning for sql server. > > My problem seems to related to this piece of code > > public class FlatTableSqlQuoteUtils { > public static final String QUOTE = "`"; > /** > * Quote identifier by default quote ` > * @param identifier > * @return > */ > public static String quoteIdentifier(String identifier){ > return QUOTE + identifier + QUOTE; > } > > The solution may be to add a parameter to change this character used for > quoting. > > Just to be sure I made a copy of the original "sqoop" command to "sqoop.orig" > and replaced the original with a little script of mine (attached). > This little script removes the "`" character from the original command and > the call "sqoop.orig" with the cleaned arguments. > It works, but it is still an ugly workaround. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)