[jira] [Created] (KYLIN-3792) Reassign's inconsistent state need automation heal
XiaoXiang Yu created KYLIN-3792: --- Summary: Reassign's inconsistent state need automation heal Key: KYLIN-3792 URL: https://issues.apache.org/jira/browse/KYLIN-3792 Project: Kylin Issue Type: Improvement Components: Real-time Streaming Reporter: XiaoXiang Yu Assignee: XiaoXiang Yu Realtime streaming has a reassign/rebalance operation which may cause some inconsistent state which is difficult to recover. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3792) Reassign's inconsistent state need automation heal
[ https://issues.apache.org/jira/browse/KYLIN-3792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3792: External issue URL: (was: https://github.com/hit-lacus/hit-lacus.github.io/issues/23) > Reassign's inconsistent state need automation heal > -- > > Key: KYLIN-3792 > URL: https://issues.apache.org/jira/browse/KYLIN-3792 > Project: Kylin > Issue Type: Improvement > Components: Real-time Streaming >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Original Estimate: 96h > Remaining Estimate: 96h > > Realtime streaming has a reassign/rebalance operation which may cause some > inconsistent state which is difficult to recover. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3792) Reassign's inconsistent state need automation heal
[ https://issues.apache.org/jira/browse/KYLIN-3792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3792: External issue URL: https://github.com/hit-lacus/hit-lacus.github.io/issues/23 Remaining Estimate: 96h Original Estimate: 96h > Reassign's inconsistent state need automation heal > -- > > Key: KYLIN-3792 > URL: https://issues.apache.org/jira/browse/KYLIN-3792 > Project: Kylin > Issue Type: Improvement > Components: Real-time Streaming >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Original Estimate: 96h > Remaining Estimate: 96h > > Realtime streaming has a reassign/rebalance operation which may cause some > inconsistent state which is difficult to recover. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3792) Reassign's inconsistent state need automation heal
[ https://issues.apache.org/jira/browse/KYLIN-3792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3792: Labels: reassign (was: ) > Reassign's inconsistent state need automation heal > -- > > Key: KYLIN-3792 > URL: https://issues.apache.org/jira/browse/KYLIN-3792 > Project: Kylin > Issue Type: Improvement > Components: Real-time Streaming >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Labels: reassign > Original Estimate: 96h > Remaining Estimate: 96h > > Realtime streaming has a reassign/rebalance operation which may cause some > inconsistent state which is difficult to recover. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3792) Reassign's inconsistent state need automation recovery
[ https://issues.apache.org/jira/browse/KYLIN-3792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3792: Summary: Reassign's inconsistent state need automation recovery (was: Reassign's inconsistent state need automation heal) > Reassign's inconsistent state need automation recovery > -- > > Key: KYLIN-3792 > URL: https://issues.apache.org/jira/browse/KYLIN-3792 > Project: Kylin > Issue Type: Improvement > Components: Real-time Streaming >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Labels: reassign > Original Estimate: 96h > Remaining Estimate: 96h > > Realtime streaming has a reassign/rebalance operation which may cause some > inconsistent state which is difficult to recover. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3044) Support SQL Server as data source
[ https://issues.apache.org/jira/browse/KYLIN-3044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16755607#comment-16755607 ] XiaoXiang Yu commented on KYLIN-3044: - !image-2019-01-30-10-46-44-236.png! > Support SQL Server as data source > - > > Key: KYLIN-3044 > URL: https://issues.apache.org/jira/browse/KYLIN-3044 > Project: Kylin > Issue Type: New Feature >Reporter: Kaige Liu >Assignee: Kaige Liu >Priority: Major > Fix For: v2.3.0 > > Attachments: KYLIN-3044-sqlserver-as-datasource.patch > > > [KYLIN-1351|https://issues.apache.org/jira/browse/KYLIN-1351] has added > Vertica as data source. Base on the work of KYLIN-1351, I'd like to enable > SQL Server as data source of kylin. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3798) SQL Server cannot list databases
XiaoXiang Yu created KYLIN-3798: --- Summary: SQL Server cannot list databases Key: KYLIN-3798 URL: https://issues.apache.org/jira/browse/KYLIN-3798 Project: Kylin Issue Type: Bug Components: RDBMS Source Reporter: XiaoXiang Yu Assignee: XiaoXiang Yu Attachments: image-2019-01-30-11-14-11-939.png, image-2019-01-30-11-14-38-581.png, image-2019-01-30-11-15-07-865.png !image-2019-01-30-11-14-11-939.png! !image-2019-01-30-11-15-07-865.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3798) SQL Server cannot list databases
[ https://issues.apache.org/jira/browse/KYLIN-3798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3798: Attachment: image-2019-01-30-11-16-41-320.png > SQL Server cannot list databases > > > Key: KYLIN-3798 > URL: https://issues.apache.org/jira/browse/KYLIN-3798 > Project: Kylin > Issue Type: Bug > Components: RDBMS Source >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Attachments: image-2019-01-30-11-14-11-939.png, > image-2019-01-30-11-14-38-581.png, image-2019-01-30-11-15-07-865.png, > image-2019-01-30-11-16-16-975.png, image-2019-01-30-11-16-41-320.png > > > > !image-2019-01-30-11-14-11-939.png! > > !image-2019-01-30-11-15-07-865.png! > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3798) SQL Server cannot list databases
[ https://issues.apache.org/jira/browse/KYLIN-3798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3798: Description: !image-2019-01-30-11-14-11-939.png! !image-2019-01-30-11-16-41-320.png! was: !image-2019-01-30-11-14-11-939.png! !image-2019-01-30-11-15-07-865.png! > SQL Server cannot list databases > > > Key: KYLIN-3798 > URL: https://issues.apache.org/jira/browse/KYLIN-3798 > Project: Kylin > Issue Type: Bug > Components: RDBMS Source >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Attachments: image-2019-01-30-11-14-11-939.png, > image-2019-01-30-11-14-38-581.png, image-2019-01-30-11-15-07-865.png, > image-2019-01-30-11-16-16-975.png, image-2019-01-30-11-16-41-320.png > > > > !image-2019-01-30-11-14-11-939.png! > > > !image-2019-01-30-11-16-41-320.png! > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3798) SQL Server cannot list databases
[ https://issues.apache.org/jira/browse/KYLIN-3798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3798: Attachment: image-2019-01-30-11-16-16-975.png > SQL Server cannot list databases > > > Key: KYLIN-3798 > URL: https://issues.apache.org/jira/browse/KYLIN-3798 > Project: Kylin > Issue Type: Bug > Components: RDBMS Source >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Attachments: image-2019-01-30-11-14-11-939.png, > image-2019-01-30-11-14-38-581.png, image-2019-01-30-11-15-07-865.png, > image-2019-01-30-11-16-16-975.png, image-2019-01-30-11-16-41-320.png > > > > !image-2019-01-30-11-14-11-939.png! > > !image-2019-01-30-11-15-07-865.png! > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3798) SQL Server cannot list databases
[ https://issues.apache.org/jira/browse/KYLIN-3798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3798: Attachment: (was: image-2019-01-30-11-15-07-865.png) > SQL Server cannot list databases > > > Key: KYLIN-3798 > URL: https://issues.apache.org/jira/browse/KYLIN-3798 > Project: Kylin > Issue Type: Bug > Components: RDBMS Source >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Attachments: image-2019-01-30-11-14-11-939.png, > image-2019-01-30-11-16-16-975.png, image-2019-01-30-11-16-41-320.png > > > User report sqlserver as data source didn't show databases. Following is his > email: > > {quote}Hi,{quote} > {quote} I followed all the steps in this url > [http://kylin.apache.org/docs/tutorial/setup_jdbc_datasource.html] , but when > i click on Load table button or Load table from tree i don't seem to have any > tables loaded from the SQL Server data source.There are no errors/exceptions > in the logs too. > > kylin.properties: > {quote} > {quote}kylin.source.default=8 > kylin.source.jdbc.connection-url=jdbc:sqlserver://hostname:1433;database=sample > kylin.source.jdbc.driver=com.microsoft.sqlserver.jdbc.SQLServerDriver > kylin.source.jdbc.dialect=mssql > kylin.source.jdbc.user=user > kylin.source.jdbc.pass=pass > kylin.source.jdbc.sqoop-home=sqoophome > kylin.source.jdbc.filed-delimiter=| > kylin.source.jdbc.sqoop-mapper-num=4{quote} > I reproduce this bug in my dev env. And I find this question was caused by > SQLServerJdbcMetadata. I have modified SQLServerJdbcMetadata and databases > could be showed. > > !image-2019-01-30-11-14-11-939.png! > > > !image-2019-01-30-11-16-41-320.png! > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3798) SQL Server cannot list databases
[ https://issues.apache.org/jira/browse/KYLIN-3798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3798: Attachment: (was: image-2019-01-30-11-14-38-581.png) > SQL Server cannot list databases > > > Key: KYLIN-3798 > URL: https://issues.apache.org/jira/browse/KYLIN-3798 > Project: Kylin > Issue Type: Bug > Components: RDBMS Source >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Attachments: image-2019-01-30-11-14-11-939.png, > image-2019-01-30-11-16-16-975.png, image-2019-01-30-11-16-41-320.png > > > User report sqlserver as data source didn't show databases. Following is his > email: > > {quote}Hi,{quote} > {quote} I followed all the steps in this url > [http://kylin.apache.org/docs/tutorial/setup_jdbc_datasource.html] , but when > i click on Load table button or Load table from tree i don't seem to have any > tables loaded from the SQL Server data source.There are no errors/exceptions > in the logs too. > > kylin.properties: > {quote} > {quote}kylin.source.default=8 > kylin.source.jdbc.connection-url=jdbc:sqlserver://hostname:1433;database=sample > kylin.source.jdbc.driver=com.microsoft.sqlserver.jdbc.SQLServerDriver > kylin.source.jdbc.dialect=mssql > kylin.source.jdbc.user=user > kylin.source.jdbc.pass=pass > kylin.source.jdbc.sqoop-home=sqoophome > kylin.source.jdbc.filed-delimiter=| > kylin.source.jdbc.sqoop-mapper-num=4{quote} > I reproduce this bug in my dev env. And I find this question was caused by > SQLServerJdbcMetadata. I have modified SQLServerJdbcMetadata and databases > could be showed. > > !image-2019-01-30-11-14-11-939.png! > > > !image-2019-01-30-11-16-41-320.png! > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3798) SQL Server cannot list databases
[ https://issues.apache.org/jira/browse/KYLIN-3798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3798: Description: User report sqlserver as data source didn't show databases. Following is his email: {quote}Hi, {quote} {quote} I followed all the steps in this url [http://kylin.apache.org/docs/tutorial/setup_jdbc_datasource.html] , but when i click on Load table button or Load table from tree i don't seem to have any tables loaded from the SQL Server data source.There are no errors/exceptions in the logs too. kylin.properties: {quote} {quote}kylin.source.default=8 kylin.source.jdbc.connection-url=jdbc:sqlserver://hostname:1433;database=sample kylin.source.jdbc.driver=com.microsoft.sqlserver.jdbc.SQLServerDriver kylin.source.jdbc.dialect=mssql kylin.source.jdbc.user=user kylin.source.jdbc.pass=pass kylin.source.jdbc.sqoop-home=sqoophome kylin.source.jdbc.filed-delimiter=| kylin.source.jdbc.sqoop-mapper-num=4 {quote} I reproduce this bug in my dev env. And I find this question was caused by SQLServerJdbcMetadata. I have modified SQLServerJdbcMetadata and databases could be showed. My env: SqlServer version is 2017, and JDBC Driver version is {quote} com.microsoft.sqlserver mssql-jdbc 6.4.0.jre8 {quote} !image-2019-01-30-11-14-11-939.png! !image-2019-01-30-11-16-41-320.png! was: User report sqlserver as data source didn't show databases. Following is his email: {quote}Hi,{quote} {quote} I followed all the steps in this url [http://kylin.apache.org/docs/tutorial/setup_jdbc_datasource.html] , but when i click on Load table button or Load table from tree i don't seem to have any tables loaded from the SQL Server data source.There are no errors/exceptions in the logs too. kylin.properties: {quote} {quote}kylin.source.default=8 kylin.source.jdbc.connection-url=jdbc:sqlserver://hostname:1433;database=sample kylin.source.jdbc.driver=com.microsoft.sqlserver.jdbc.SQLServerDriver kylin.source.jdbc.dialect=mssql kylin.source.jdbc.user=user kylin.source.jdbc.pass=pass kylin.source.jdbc.sqoop-home=sqoophome kylin.source.jdbc.filed-delimiter=| kylin.source.jdbc.sqoop-mapper-num=4{quote} I reproduce this bug in my dev env. And I find this question was caused by SQLServerJdbcMetadata. I have modified SQLServerJdbcMetadata and databases could be showed. !image-2019-01-30-11-14-11-939.png! !image-2019-01-30-11-16-41-320.png! > SQL Server cannot list databases > > > Key: KYLIN-3798 > URL: https://issues.apache.org/jira/browse/KYLIN-3798 > Project: Kylin > Issue Type: Bug > Components: RDBMS Source >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Attachments: image-2019-01-30-11-14-11-939.png, > image-2019-01-30-11-16-16-975.png, image-2019-01-30-11-16-41-320.png > > > User report sqlserver as data source didn't show databases. Following is his > email: > > {quote}Hi, > {quote} > {quote} I followed all the steps in this url > [http://kylin.apache.org/docs/tutorial/setup_jdbc_datasource.html] , but when > i click on Load table button or Load table from tree i don't seem to have any > tables loaded from the SQL Server data source.There are no errors/exceptions > in the logs too. > > kylin.properties: > > {quote} > {quote}kylin.source.default=8 > > kylin.source.jdbc.connection-url=jdbc:sqlserver://hostname:1433;database=sample > kylin.source.jdbc.driver=com.microsoft.sqlserver.jdbc.SQLServerDriver > kylin.source.jdbc.dialect=mssql > kylin.source.jdbc.user=user > kylin.source.jdbc.pass=pass > kylin.source.jdbc.sqoop-home=sqoophome > kylin.source.jdbc.filed-delimiter=| > kylin.source.jdbc.sqoop-mapper-num=4 > {quote} > I reproduce this bug in my dev env. And I find this question was caused by > SQLServerJdbcMetadata. I have modified SQLServerJdbcMetadata and databases > could be showed. > > My env: SqlServer version is 2017, and JDBC Driver version is > {quote} > com.microsoft.sqlserver > mssql-jdbc > 6.4.0.jre8 > > {quote} > > > !image-2019-01-30-11-14-11-939.png! > > > !image-2019-01-30-11-16-41-320.png! > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3798) SQL Server cannot list databases
[ https://issues.apache.org/jira/browse/KYLIN-3798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16755634#comment-16755634 ] XiaoXiang Yu commented on KYLIN-3798: - h2. Test Result !image-2019-01-30-11-25-47-946.png! > SQL Server cannot list databases > > > Key: KYLIN-3798 > URL: https://issues.apache.org/jira/browse/KYLIN-3798 > Project: Kylin > Issue Type: Bug > Components: RDBMS Source >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Attachments: image-2019-01-30-11-14-11-939.png, > image-2019-01-30-11-16-16-975.png, image-2019-01-30-11-16-41-320.png, > image-2019-01-30-11-25-47-946.png > > > User report sqlserver as data source didn't show databases. Following is his > email: > > {quote}Hi, > {quote} > {quote} I followed all the steps in this url > [http://kylin.apache.org/docs/tutorial/setup_jdbc_datasource.html] , but when > i click on Load table button or Load table from tree i don't seem to have any > tables loaded from the SQL Server data source.There are no errors/exceptions > in the logs too. > > kylin.properties: > > {quote} > {quote}kylin.source.default=8 > > kylin.source.jdbc.connection-url=jdbc:sqlserver://hostname:1433;database=sample > kylin.source.jdbc.driver=com.microsoft.sqlserver.jdbc.SQLServerDriver > kylin.source.jdbc.dialect=mssql > kylin.source.jdbc.user=user > kylin.source.jdbc.pass=pass > kylin.source.jdbc.sqoop-home=sqoophome > kylin.source.jdbc.filed-delimiter=| > kylin.source.jdbc.sqoop-mapper-num=4 > {quote} > I reproduce this bug in my dev env. And I find this question was caused by > SQLServerJdbcMetadata. I have modified SQLServerJdbcMetadata and databases > could be showed. > > My env: SqlServer version is 2017, and JDBC Driver version is > {quote} > com.microsoft.sqlserver > mssql-jdbc > 6.4.0.jre8 > > {quote} > > > !image-2019-01-30-11-14-11-939.png! > > > !image-2019-01-30-11-16-41-320.png! > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3798) SQL Server cannot list databases
[ https://issues.apache.org/jira/browse/KYLIN-3798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3798: Attachment: image-2019-01-30-11-25-47-946.png > SQL Server cannot list databases > > > Key: KYLIN-3798 > URL: https://issues.apache.org/jira/browse/KYLIN-3798 > Project: Kylin > Issue Type: Bug > Components: RDBMS Source >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Attachments: image-2019-01-30-11-14-11-939.png, > image-2019-01-30-11-16-16-975.png, image-2019-01-30-11-16-41-320.png, > image-2019-01-30-11-25-47-946.png > > > User report sqlserver as data source didn't show databases. Following is his > email: > > {quote}Hi, > {quote} > {quote} I followed all the steps in this url > [http://kylin.apache.org/docs/tutorial/setup_jdbc_datasource.html] , but when > i click on Load table button or Load table from tree i don't seem to have any > tables loaded from the SQL Server data source.There are no errors/exceptions > in the logs too. > > kylin.properties: > > {quote} > {quote}kylin.source.default=8 > > kylin.source.jdbc.connection-url=jdbc:sqlserver://hostname:1433;database=sample > kylin.source.jdbc.driver=com.microsoft.sqlserver.jdbc.SQLServerDriver > kylin.source.jdbc.dialect=mssql > kylin.source.jdbc.user=user > kylin.source.jdbc.pass=pass > kylin.source.jdbc.sqoop-home=sqoophome > kylin.source.jdbc.filed-delimiter=| > kylin.source.jdbc.sqoop-mapper-num=4 > {quote} > I reproduce this bug in my dev env. And I find this question was caused by > SQLServerJdbcMetadata. I have modified SQLServerJdbcMetadata and databases > could be showed. > > My env: SqlServer version is 2017, and JDBC Driver version is > {quote} > com.microsoft.sqlserver > mssql-jdbc > 6.4.0.jre8 > > {quote} > > > !image-2019-01-30-11-14-11-939.png! > > > !image-2019-01-30-11-16-41-320.png! > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3798) SQL Server cannot list databases
[ https://issues.apache.org/jira/browse/KYLIN-3798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3798: Description: User report sqlserver as data source didn't show databases. Following is his email: {quote}Hi,{quote} {quote} I followed all the steps in this url [http://kylin.apache.org/docs/tutorial/setup_jdbc_datasource.html] , but when i click on Load table button or Load table from tree i don't seem to have any tables loaded from the SQL Server data source.There are no errors/exceptions in the logs too. kylin.properties: {quote} {quote}kylin.source.default=8 kylin.source.jdbc.connection-url=jdbc:sqlserver://hostname:1433;database=sample kylin.source.jdbc.driver=com.microsoft.sqlserver.jdbc.SQLServerDriver kylin.source.jdbc.dialect=mssql kylin.source.jdbc.user=user kylin.source.jdbc.pass=pass kylin.source.jdbc.sqoop-home=sqoophome kylin.source.jdbc.filed-delimiter=| kylin.source.jdbc.sqoop-mapper-num=4{quote} I reproduce this bug in my dev env. And I find this question was caused by SQLServerJdbcMetadata. I have modified SQLServerJdbcMetadata and databases could be showed. !image-2019-01-30-11-14-11-939.png! !image-2019-01-30-11-16-41-320.png! was: !image-2019-01-30-11-14-11-939.png! !image-2019-01-30-11-16-41-320.png! > SQL Server cannot list databases > > > Key: KYLIN-3798 > URL: https://issues.apache.org/jira/browse/KYLIN-3798 > Project: Kylin > Issue Type: Bug > Components: RDBMS Source >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Attachments: image-2019-01-30-11-14-11-939.png, > image-2019-01-30-11-16-16-975.png, image-2019-01-30-11-16-41-320.png > > > User report sqlserver as data source didn't show databases. Following is his > email: > > {quote}Hi,{quote} > {quote} I followed all the steps in this url > [http://kylin.apache.org/docs/tutorial/setup_jdbc_datasource.html] , but when > i click on Load table button or Load table from tree i don't seem to have any > tables loaded from the SQL Server data source.There are no errors/exceptions > in the logs too. > > kylin.properties: > {quote} > {quote}kylin.source.default=8 > kylin.source.jdbc.connection-url=jdbc:sqlserver://hostname:1433;database=sample > kylin.source.jdbc.driver=com.microsoft.sqlserver.jdbc.SQLServerDriver > kylin.source.jdbc.dialect=mssql > kylin.source.jdbc.user=user > kylin.source.jdbc.pass=pass > kylin.source.jdbc.sqoop-home=sqoophome > kylin.source.jdbc.filed-delimiter=| > kylin.source.jdbc.sqoop-mapper-num=4{quote} > I reproduce this bug in my dev env. And I find this question was caused by > SQLServerJdbcMetadata. I have modified SQLServerJdbcMetadata and databases > could be showed. > > !image-2019-01-30-11-14-11-939.png! > > > !image-2019-01-30-11-16-41-320.png! > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3791) Map return by Maps.transformValues is a immutable view
[ https://issues.apache.org/jira/browse/KYLIN-3791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu resolved KYLIN-3791. - Resolution: Fixed > Map return by Maps.transformValues is a immutable view > -- > > Key: KYLIN-3791 > URL: https://issues.apache.org/jira/browse/KYLIN-3791 > Project: Kylin > Issue Type: Improvement > Components: Real-time Streaming >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Attachments: image-2019-01-26-20-53-10-085.png, > image-2019-01-26-20-54-12-483.png > > > !image-2019-01-26-20-53-10-085.png! > > > !image-2019-01-26-20-54-12-483.png! > > After restart my receiver, I found a lot of error logs which is confused. It > is a bug caused by misuse of Guava. The bug can only be triggered when > loading segment cache. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3759) Streaming ClassNotFoundExeception when lambda is enable in MR job
[ https://issues.apache.org/jira/browse/KYLIN-3759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu resolved KYLIN-3759. - Resolution: Fixed > Streaming ClassNotFoundExeception when lambda is enable in MR job > - > > Key: KYLIN-3759 > URL: https://issues.apache.org/jira/browse/KYLIN-3759 > Project: Kylin > Issue Type: Improvement > Components: Real-time Streaming >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Attachments: image-2019-01-23-16-09-29-317.png > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3784) Hadoop Common ReflectionUtils.printThreadInfo signature change
[ https://issues.apache.org/jira/browse/KYLIN-3784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu resolved KYLIN-3784. - Resolution: Fixed > Hadoop Common ReflectionUtils.printThreadInfo signature change > -- > > Key: KYLIN-3784 > URL: https://issues.apache.org/jira/browse/KYLIN-3784 > Project: Kylin > Issue Type: Improvement > Components: Real-time Streaming >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Attachments: image-2019-01-24-09-59-00-421.png > > > !image-2019-01-24-09-59-00-421.png! > > > Can not build a jar because Hadoop jar version compatibility. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KYLIN-3857) add parameter to change sql quoting character for sqoop
[ https://issues.apache.org/jira/browse/KYLIN-3857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu reassigned KYLIN-3857: --- Assignee: XiaoXiang Yu > add parameter to change sql quoting character for sqoop > --- > > Key: KYLIN-3857 > URL: https://issues.apache.org/jira/browse/KYLIN-3857 > Project: Kylin > Issue Type: Bug >Reporter: Davide Malagoli >Assignee: XiaoXiang Yu >Priority: Major > Attachments: sqoop-error.log, sqoop-script.sh > > > I've set up Kylin to access a SQL server using JDBC and do the ETL step as > described in > [http://kylin.apache.org/docs/tutorial/setup_jdbc_datasource.html] > but when the "Sqoop To Flat Hive Table" runs it terminates with an error > shown in the attached log. > It seems that the root cause is the presence of the '`' character in the > query, which has no meaning for sql server. > > My problem seems to related to this piece of code > > public class FlatTableSqlQuoteUtils { > public static final String QUOTE = "`"; > /** > * Quote identifier by default quote ` > * @param identifier > * @return > */ > public static String quoteIdentifier(String identifier){ > return QUOTE + identifier + QUOTE; > } > > The solution may be to add a parameter to change this character used for > quoting. > > Just to be sure I made a copy of the original "sqoop" command to "sqoop.orig" > and replaced the original with a little script of mine (attached). > This little script removes the "`" character from the original command and > the call "sqoop.orig" with the cleaned arguments. > It works, but it is still an ugly workaround. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3857) add parameter to change sql quoting character for sqoop
[ https://issues.apache.org/jira/browse/KYLIN-3857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3857: Attachment: image-2019-03-11-11-46-20-664.png > add parameter to change sql quoting character for sqoop > --- > > Key: KYLIN-3857 > URL: https://issues.apache.org/jira/browse/KYLIN-3857 > Project: Kylin > Issue Type: Bug >Reporter: Davide Malagoli >Assignee: XiaoXiang Yu >Priority: Major > Attachments: image-2019-03-11-11-46-05-946.png, > image-2019-03-11-11-46-14-305.png, image-2019-03-11-11-46-20-664.png, > image-2019-03-11-11-46-31-966.png, sqoop-error.log, sqoop-script.sh > > > I've set up Kylin to access a SQL server using JDBC and do the ETL step as > described in > [http://kylin.apache.org/docs/tutorial/setup_jdbc_datasource.html] > but when the "Sqoop To Flat Hive Table" runs it terminates with an error > shown in the attached log. > It seems that the root cause is the presence of the '`' character in the > query, which has no meaning for sql server. > > My problem seems to related to this piece of code > > public class FlatTableSqlQuoteUtils { > public static final String QUOTE = "`"; > /** > * Quote identifier by default quote ` > * @param identifier > * @return > */ > public static String quoteIdentifier(String identifier){ > return QUOTE + identifier + QUOTE; > } > > The solution may be to add a parameter to change this character used for > quoting. > > Just to be sure I made a copy of the original "sqoop" command to "sqoop.orig" > and replaced the original with a little script of mine (attached). > This little script removes the "`" character from the original command and > the call "sqoop.orig" with the cleaned arguments. > It works, but it is still an ugly workaround. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3857) add parameter to change sql quoting character for sqoop
[ https://issues.apache.org/jira/browse/KYLIN-3857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3857: Attachment: image-2019-03-11-11-46-05-946.png > add parameter to change sql quoting character for sqoop > --- > > Key: KYLIN-3857 > URL: https://issues.apache.org/jira/browse/KYLIN-3857 > Project: Kylin > Issue Type: Bug >Reporter: Davide Malagoli >Assignee: XiaoXiang Yu >Priority: Major > Attachments: image-2019-03-11-11-46-05-946.png, > image-2019-03-11-11-46-14-305.png, image-2019-03-11-11-46-20-664.png, > image-2019-03-11-11-46-31-966.png, sqoop-error.log, sqoop-script.sh > > > I've set up Kylin to access a SQL server using JDBC and do the ETL step as > described in > [http://kylin.apache.org/docs/tutorial/setup_jdbc_datasource.html] > but when the "Sqoop To Flat Hive Table" runs it terminates with an error > shown in the attached log. > It seems that the root cause is the presence of the '`' character in the > query, which has no meaning for sql server. > > My problem seems to related to this piece of code > > public class FlatTableSqlQuoteUtils { > public static final String QUOTE = "`"; > /** > * Quote identifier by default quote ` > * @param identifier > * @return > */ > public static String quoteIdentifier(String identifier){ > return QUOTE + identifier + QUOTE; > } > > The solution may be to add a parameter to change this character used for > quoting. > > Just to be sure I made a copy of the original "sqoop" command to "sqoop.orig" > and replaced the original with a little script of mine (attached). > This little script removes the "`" character from the original command and > the call "sqoop.orig" with the cleaned arguments. > It works, but it is still an ugly workaround. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3857) add parameter to change sql quoting character for sqoop
[ https://issues.apache.org/jira/browse/KYLIN-3857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3857: Attachment: image-2019-03-11-11-46-31-966.png > add parameter to change sql quoting character for sqoop > --- > > Key: KYLIN-3857 > URL: https://issues.apache.org/jira/browse/KYLIN-3857 > Project: Kylin > Issue Type: Bug >Reporter: Davide Malagoli >Assignee: XiaoXiang Yu >Priority: Major > Attachments: image-2019-03-11-11-46-05-946.png, > image-2019-03-11-11-46-14-305.png, image-2019-03-11-11-46-20-664.png, > image-2019-03-11-11-46-31-966.png, sqoop-error.log, sqoop-script.sh > > > I've set up Kylin to access a SQL server using JDBC and do the ETL step as > described in > [http://kylin.apache.org/docs/tutorial/setup_jdbc_datasource.html] > but when the "Sqoop To Flat Hive Table" runs it terminates with an error > shown in the attached log. > It seems that the root cause is the presence of the '`' character in the > query, which has no meaning for sql server. > > My problem seems to related to this piece of code > > public class FlatTableSqlQuoteUtils { > public static final String QUOTE = "`"; > /** > * Quote identifier by default quote ` > * @param identifier > * @return > */ > public static String quoteIdentifier(String identifier){ > return QUOTE + identifier + QUOTE; > } > > The solution may be to add a parameter to change this character used for > quoting. > > Just to be sure I made a copy of the original "sqoop" command to "sqoop.orig" > and replaced the original with a little script of mine (attached). > This little script removes the "`" character from the original command and > the call "sqoop.orig" with the cleaned arguments. > It works, but it is still an ugly workaround. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3857) add parameter to change sql quoting character for sqoop
[ https://issues.apache.org/jira/browse/KYLIN-3857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16789133#comment-16789133 ] XiaoXiang Yu commented on KYLIN-3857: - Screenshots in My DEV env (with quote turn on): !image-2019-03-11-11-46-14-305.png! !image-2019-03-11-11-46-20-664.png! !image-2019-03-11-11-46-31-966.png! > add parameter to change sql quoting character for sqoop > --- > > Key: KYLIN-3857 > URL: https://issues.apache.org/jira/browse/KYLIN-3857 > Project: Kylin > Issue Type: Bug >Reporter: Davide Malagoli >Assignee: XiaoXiang Yu >Priority: Major > Attachments: image-2019-03-11-11-46-05-946.png, > image-2019-03-11-11-46-14-305.png, image-2019-03-11-11-46-20-664.png, > image-2019-03-11-11-46-31-966.png, sqoop-error.log, sqoop-script.sh > > > I've set up Kylin to access a SQL server using JDBC and do the ETL step as > described in > [http://kylin.apache.org/docs/tutorial/setup_jdbc_datasource.html] > but when the "Sqoop To Flat Hive Table" runs it terminates with an error > shown in the attached log. > It seems that the root cause is the presence of the '`' character in the > query, which has no meaning for sql server. > > My problem seems to related to this piece of code > > public class FlatTableSqlQuoteUtils { > public static final String QUOTE = "`"; > /** > * Quote identifier by default quote ` > * @param identifier > * @return > */ > public static String quoteIdentifier(String identifier){ > return QUOTE + identifier + QUOTE; > } > > The solution may be to add a parameter to change this character used for > quoting. > > Just to be sure I made a copy of the original "sqoop" command to "sqoop.orig" > and replaced the original with a little script of mine (attached). > This little script removes the "`" character from the original command and > the call "sqoop.orig" with the cleaned arguments. > It works, but it is still an ugly workaround. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3857) add parameter to change sql quoting character for sqoop
[ https://issues.apache.org/jira/browse/KYLIN-3857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3857: Attachment: image-2019-03-11-11-46-14-305.png > add parameter to change sql quoting character for sqoop > --- > > Key: KYLIN-3857 > URL: https://issues.apache.org/jira/browse/KYLIN-3857 > Project: Kylin > Issue Type: Bug >Reporter: Davide Malagoli >Assignee: XiaoXiang Yu >Priority: Major > Attachments: image-2019-03-11-11-46-05-946.png, > image-2019-03-11-11-46-14-305.png, image-2019-03-11-11-46-20-664.png, > image-2019-03-11-11-46-31-966.png, sqoop-error.log, sqoop-script.sh > > > I've set up Kylin to access a SQL server using JDBC and do the ETL step as > described in > [http://kylin.apache.org/docs/tutorial/setup_jdbc_datasource.html] > but when the "Sqoop To Flat Hive Table" runs it terminates with an error > shown in the attached log. > It seems that the root cause is the presence of the '`' character in the > query, which has no meaning for sql server. > > My problem seems to related to this piece of code > > public class FlatTableSqlQuoteUtils { > public static final String QUOTE = "`"; > /** > * Quote identifier by default quote ` > * @param identifier > * @return > */ > public static String quoteIdentifier(String identifier){ > return QUOTE + identifier + QUOTE; > } > > The solution may be to add a parameter to change this character used for > quoting. > > Just to be sure I made a copy of the original "sqoop" command to "sqoop.orig" > and replaced the original with a little script of mine (attached). > This little script removes the "`" character from the original command and > the call "sqoop.orig" with the cleaned arguments. > It works, but it is still an ugly workaround. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3857) add parameter to change sql quoting character for sqoop
[ https://issues.apache.org/jira/browse/KYLIN-3857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16789136#comment-16789136 ] XiaoXiang Yu commented on KYLIN-3857: - Screenshots in My DEV env (with quote turn off): > add parameter to change sql quoting character for sqoop > --- > > Key: KYLIN-3857 > URL: https://issues.apache.org/jira/browse/KYLIN-3857 > Project: Kylin > Issue Type: Bug >Reporter: Davide Malagoli >Assignee: XiaoXiang Yu >Priority: Major > Attachments: image-2019-03-11-11-46-05-946.png, > image-2019-03-11-11-46-14-305.png, image-2019-03-11-11-46-20-664.png, > image-2019-03-11-11-46-31-966.png, sqoop-error.log, sqoop-script.sh > > > I've set up Kylin to access a SQL server using JDBC and do the ETL step as > described in > [http://kylin.apache.org/docs/tutorial/setup_jdbc_datasource.html] > but when the "Sqoop To Flat Hive Table" runs it terminates with an error > shown in the attached log. > It seems that the root cause is the presence of the '`' character in the > query, which has no meaning for sql server. > > My problem seems to related to this piece of code > > public class FlatTableSqlQuoteUtils { > public static final String QUOTE = "`"; > /** > * Quote identifier by default quote ` > * @param identifier > * @return > */ > public static String quoteIdentifier(String identifier){ > return QUOTE + identifier + QUOTE; > } > > The solution may be to add a parameter to change this character used for > quoting. > > Just to be sure I made a copy of the original "sqoop" command to "sqoop.orig" > and replaced the original with a little script of mine (attached). > This little script removes the "`" character from the original command and > the call "sqoop.orig" with the cleaned arguments. > It works, but it is still an ugly workaround. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3857) add parameter to change sql quoting character for sqoop
[ https://issues.apache.org/jira/browse/KYLIN-3857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16789136#comment-16789136 ] XiaoXiang Yu edited comment on KYLIN-3857 at 3/11/19 4:59 AM: -- Logs in My DEV env (with quote turn off): KYLIN [ INFO ] 03-11 12:53:39.114 org.apache.kylin.common.util.BufferedLogger.log(BufferedLogger.java:38) from Scheduler 1705744661 Job 6d64fe74-ea50-76a6-bbd1-f85210984416-129 > Create and distribute table, cmd: KYLIN [ INFO ] 03-11 12:53:39.115 org.apache.kylin.common.util.BufferedLogger.log(BufferedLogger.java:38) from Scheduler 1705744661 Job 6d64fe74-ea50-76a6-bbd1-f85210984416-129 > hive -e "USE lacus; DROP TABLE IF EXISTS kylin_intermediate_samplecube_e1c8bdcc_d55b_3807_25e2_b8d2b6fccabe; CREATE EXTERNAL TABLE IF NOT EXISTS kylin_intermediate_samplecube_e1c8bdcc_d55b_3807_25e2_b8d2b6fccabe ( KYLIN_SALES_LSTG_FORMAT_NAME string ,KYLIN_SALES_LSTG_SITE_ID int ,KYLIN_SALES_SELLER_ID bigint ,KYLIN_ACCOUNT_ACCOUNT_COUNTRY string ,KYLIN_SALES_PRICE decimal(19,4) ,KYLIN_SALES_ITEM_COUNT bigint ) STORED AS SEQUENCEFILE LOCATION 'hdfs://sandbox.hortonworks.com:8020/kylin/lacus/lacus-kylin_metadata/kylin-6d64fe74-ea50-76a6-bbd1-f85210984416/kylin_intermediate_samplecube_e1c8bdcc_d55b_3807_25e2_b8d2b6fccabe'; ALTER TABLE kylin_intermediate_samplecube_e1c8bdcc_d55b_3807_25e2_b8d2b6fccabe SET TBLPROPERTIES('auto.purge'='true'); INSERT OVERWRITE TABLE kylin_intermediate_samplecube_e1c8bdcc_d55b_3807_25e2_b8d2b6fccabe SELECT KYLIN_SALES.LSTG_FORMAT_NAME as KYLIN_SALES_LSTG_FORMAT_NAME ,KYLIN_SALES.LSTG_SITE_ID as KYLIN_SALES_LSTG_SITE_ID ,KYLIN_SALES.SELLER_ID as KYLIN_SALES_SELLER_ID ,KYLIN_ACCOUNT.ACCOUNT_COUNTRY as KYLIN_ACCOUNT_ACCOUNT_COUNTRY ,KYLIN_SALES.PRICE as KYLIN_SALES_PRICE ,KYLIN_SALES.ITEM_COUNT as KYLIN_SALES_ITEM_COUNT FROM LACUS.KYLIN_SALES as KYLIN_SALES LEFT JOIN LACUS.KYLIN_ACCOUNT as KYLIN_ACCOUNT ON KYLIN_SALES.SELLER_ID = KYLIN_ACCOUNT.ACCOUNT_ID LEFT JOIN LACUS.KYLIN_COUNTRY as KYLIN_COUNTRY ON KYLIN_ACCOUNT.ACCOUNT_COUNTRY = KYLIN_COUNTRY.COUNTRY WHERE 1=1; " --hiveconf mapreduce.map.output.compress.codec=org.apache.hadoop.io.compress.SnappyCodec --hiveconf dfs.replication=2 --hiveconf hive.exec.compress.output=true KYLIN [ INFO ] 03-11 12:53:39.115 org.apache.kylin.common.util.SSHClient.execCommand(SSHClient.java:262) from Scheduler 1705744661 Job 6d64fe74-ea50-76a6-bbd1-f85210984416-129 > [root@sandbox] Execute command: hive -e "USE lacus; DROP TABLE IF EXISTS kylin_intermediate_samplecube_e1c8bdcc_d55b_3807_25e2_b8d2b6fccabe; CREATE EXTERNAL TABLE IF NOT EXISTS kylin_intermediate_samplecube_e1c8bdcc_d55b_3807_25e2_b8d2b6fccabe ( KYLIN_SALES_LSTG_FORMAT_NAME string ,KYLIN_SALES_LSTG_SITE_ID int ,KYLIN_SALES_SELLER_ID bigint ,KYLIN_ACCOUNT_ACCOUNT_COUNTRY string ,KYLIN_SALES_PRICE decimal(19,4) ,KYLIN_SALES_ITEM_COUNT bigint ) STORED AS SEQUENCEFILE LOCATION 'hdfs://sandbox.hortonworks.com:8020/kylin/lacus/lacus-kylin_metadata/kylin-6d64fe74-ea50-76a6-bbd1-f85210984416/kylin_intermediate_samplecube_e1c8bdcc_d55b_3807_25e2_b8d2b6fccabe'; ALTER TABLE kylin_intermediate_samplecube_e1c8bdcc_d55b_3807_25e2_b8d2b6fccabe SET TBLPROPERTIES('auto.purge'='true'); INSERT OVERWRITE TABLE kylin_intermediate_samplecube_e1c8bdcc_d55b_3807_25e2_b8d2b6fccabe SELECT KYLIN_SALES.LSTG_FORMAT_NAME as KYLIN_SALES_LSTG_FORMAT_NAME ,KYLIN_SALES.LSTG_SITE_ID as KYLIN_SALES_LSTG_SITE_ID ,KYLIN_SALES.SELLER_ID as KYLIN_SALES_SELLER_ID ,KYLIN_ACCOUNT.ACCOUNT_COUNTRY as KYLIN_ACCOUNT_ACCOUNT_COUNTRY ,KYLIN_SALES.PRICE as KYLIN_SALES_PRICE ,KYLIN_SALES.ITEM_COUNT as KYLIN_SALES_ITEM_COUNT FROM LACUS.KYLIN_SALES as KYLIN_SALES LEFT JOIN LACUS.KYLIN_ACCOUNT as KYLIN_ACCOUNT ON KYLIN_SALES.SELLER_ID = KYLIN_ACCOUNT.ACCOUNT_ID LEFT JOIN LACUS.KYLIN_COUNTRY as KYLIN_COUNTRY ON KYLIN_ACCOUNT.ACCOUNT_COUNTRY = KYLIN_COUNTRY.COUNTRY WHERE 1=1; " --hiveconf mapreduce.map.output.compress.codec=org.apache.hadoop.io.compress.SnappyCodec --hiveconf dfs.replication=2 --hiveconf hive.exec.compress.output=true KYLIN [ INFO ] 03-11 12:53:42.237 org.apache.kylin.common.util.BufferedLogger.log(BufferedLogger.java:38) from Scheduler 1705744661 Job 6d64fe74-ea50-76a6-bbd1-f85210984416-129 > WARNING: Use "yarn jar" to launch YARN applications. KYLIN [ INFO ] 03-11 12:53:43.241 org.apache.kylin.common.util.BufferedLogger.log(BufferedLogger.java:38) from Scheduler 1705744661 Job 6d64fe74-ea50-76a6-bbd1-f85210984416-129 > Logging initialized using configuration in file:/etc/hive/2.4.0.0-169/0/hive-log4j.properties KYLIN [ INFO ] 03-11 12:56:07.045 org.apache.kylin.common.util.SSHClient.execCommand(SSHClient.java:262) from Scheduler 1705744661 Job 6d64fe74-ea50-76a6-bbd1-f85210984416-221 > [root@sandbox] Execute command: hive -e "USE lacus; DROP TABLE IF EXISTS lacus.kylin_intermediate_samplecube_e1c8bdcc_d55b_3807_25e2_b8d2b6fccabe; " --hiveconf mapred
[jira] [Comment Edited] (KYLIN-3857) add parameter to change sql quoting character for sqoop
[ https://issues.apache.org/jira/browse/KYLIN-3857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16789133#comment-16789133 ] XiaoXiang Yu edited comment on KYLIN-3857 at 3/11/19 5:00 AM: -- Use kylin.source.hive.enable.quote=false in kylin.properties to turn off quote. Screenshots in My DEV env (with quote turn on): !image-2019-03-11-11-46-14-305.png! !image-2019-03-11-11-46-20-664.png! !image-2019-03-11-11-46-31-966.png! was (Author: hit_lacus): Screenshots in My DEV env (with quote turn on): !image-2019-03-11-11-46-14-305.png! !image-2019-03-11-11-46-20-664.png! !image-2019-03-11-11-46-31-966.png! > add parameter to change sql quoting character for sqoop > --- > > Key: KYLIN-3857 > URL: https://issues.apache.org/jira/browse/KYLIN-3857 > Project: Kylin > Issue Type: Bug >Reporter: Davide Malagoli >Assignee: XiaoXiang Yu >Priority: Major > Attachments: image-2019-03-11-11-46-05-946.png, > image-2019-03-11-11-46-14-305.png, image-2019-03-11-11-46-20-664.png, > image-2019-03-11-11-46-31-966.png, sqoop-error.log, sqoop-script.sh > > > I've set up Kylin to access a SQL server using JDBC and do the ETL step as > described in > [http://kylin.apache.org/docs/tutorial/setup_jdbc_datasource.html] > but when the "Sqoop To Flat Hive Table" runs it terminates with an error > shown in the attached log. > It seems that the root cause is the presence of the '`' character in the > query, which has no meaning for sql server. > > My problem seems to related to this piece of code > > public class FlatTableSqlQuoteUtils { > public static final String QUOTE = "`"; > /** > * Quote identifier by default quote ` > * @param identifier > * @return > */ > public static String quoteIdentifier(String identifier){ > return QUOTE + identifier + QUOTE; > } > > The solution may be to add a parameter to change this character used for > quoting. > > Just to be sure I made a copy of the original "sqoop" command to "sqoop.orig" > and replaced the original with a little script of mine (attached). > This little script removes the "`" character from the original command and > the call "sqoop.orig" with the cleaned arguments. > It works, but it is still an ugly workaround. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3857) add parameter to change sql quoting character for sqoop
[ https://issues.apache.org/jira/browse/KYLIN-3857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16789160#comment-16789160 ] XiaoXiang Yu commented on KYLIN-3857: - Import Microsoft SQL Server as source by Data Source SDK can be refer to: https://github.com/hit-lacus/hit-lacus.github.io/issues/32#issuecomment-469134060 > add parameter to change sql quoting character for sqoop > --- > > Key: KYLIN-3857 > URL: https://issues.apache.org/jira/browse/KYLIN-3857 > Project: Kylin > Issue Type: Bug >Reporter: Davide Malagoli >Assignee: XiaoXiang Yu >Priority: Major > Attachments: image-2019-03-11-11-46-05-946.png, > image-2019-03-11-11-46-14-305.png, image-2019-03-11-11-46-20-664.png, > image-2019-03-11-11-46-31-966.png, sqoop-error.log, sqoop-script.sh > > > I've set up Kylin to access a SQL server using JDBC and do the ETL step as > described in > [http://kylin.apache.org/docs/tutorial/setup_jdbc_datasource.html] > but when the "Sqoop To Flat Hive Table" runs it terminates with an error > shown in the attached log. > It seems that the root cause is the presence of the '`' character in the > query, which has no meaning for sql server. > > My problem seems to related to this piece of code > > public class FlatTableSqlQuoteUtils { > public static final String QUOTE = "`"; > /** > * Quote identifier by default quote ` > * @param identifier > * @return > */ > public static String quoteIdentifier(String identifier){ > return QUOTE + identifier + QUOTE; > } > > The solution may be to add a parameter to change this character used for > quoting. > > Just to be sure I made a copy of the original "sqoop" command to "sqoop.orig" > and replaced the original with a little script of mine (attached). > This little script removes the "`" character from the original command and > the call "sqoop.orig" with the cleaned arguments. > It works, but it is still an ugly workaround. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3857) add parameter to change sql quoting character for sqoop
[ https://issues.apache.org/jira/browse/KYLIN-3857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16789133#comment-16789133 ] XiaoXiang Yu edited comment on KYLIN-3857 at 3/11/19 5:06 AM: -- Use kylin.source.hive.enable.quote=false in kylin.properties to turn off quote. (default value is true) Screenshots in My DEV env (with quote turn on): !image-2019-03-11-11-46-14-305.png! !image-2019-03-11-11-46-20-664.png! !image-2019-03-11-11-46-31-966.png! was (Author: hit_lacus): Use kylin.source.hive.enable.quote=false in kylin.properties to turn off quote. Screenshots in My DEV env (with quote turn on): !image-2019-03-11-11-46-14-305.png! !image-2019-03-11-11-46-20-664.png! !image-2019-03-11-11-46-31-966.png! > add parameter to change sql quoting character for sqoop > --- > > Key: KYLIN-3857 > URL: https://issues.apache.org/jira/browse/KYLIN-3857 > Project: Kylin > Issue Type: Bug >Reporter: Davide Malagoli >Assignee: XiaoXiang Yu >Priority: Major > Attachments: image-2019-03-11-11-46-05-946.png, > image-2019-03-11-11-46-14-305.png, image-2019-03-11-11-46-20-664.png, > image-2019-03-11-11-46-31-966.png, sqoop-error.log, sqoop-script.sh > > > I've set up Kylin to access a SQL server using JDBC and do the ETL step as > described in > [http://kylin.apache.org/docs/tutorial/setup_jdbc_datasource.html] > but when the "Sqoop To Flat Hive Table" runs it terminates with an error > shown in the attached log. > It seems that the root cause is the presence of the '`' character in the > query, which has no meaning for sql server. > > My problem seems to related to this piece of code > > public class FlatTableSqlQuoteUtils { > public static final String QUOTE = "`"; > /** > * Quote identifier by default quote ` > * @param identifier > * @return > */ > public static String quoteIdentifier(String identifier){ > return QUOTE + identifier + QUOTE; > } > > The solution may be to add a parameter to change this character used for > quoting. > > Just to be sure I made a copy of the original "sqoop" command to "sqoop.orig" > and replaced the original with a little script of mine (attached). > This little script removes the "`" character from the original command and > the call "sqoop.orig" with the cleaned arguments. > It works, but it is still an ugly workaround. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3857) add parameter to change sql quoting character for sqoop
[ https://issues.apache.org/jira/browse/KYLIN-3857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3857: Fix Version/s: v2.6.2 > add parameter to change sql quoting character for sqoop > --- > > Key: KYLIN-3857 > URL: https://issues.apache.org/jira/browse/KYLIN-3857 > Project: Kylin > Issue Type: Bug >Reporter: Davide Malagoli >Assignee: XiaoXiang Yu >Priority: Major > Fix For: v2.6.2 > > Attachments: image-2019-03-11-11-46-05-946.png, > image-2019-03-11-11-46-14-305.png, image-2019-03-11-11-46-20-664.png, > image-2019-03-11-11-46-31-966.png, sqoop-error.log, sqoop-script.sh > > > I've set up Kylin to access a SQL server using JDBC and do the ETL step as > described in > [http://kylin.apache.org/docs/tutorial/setup_jdbc_datasource.html] > but when the "Sqoop To Flat Hive Table" runs it terminates with an error > shown in the attached log. > It seems that the root cause is the presence of the '`' character in the > query, which has no meaning for sql server. > > My problem seems to related to this piece of code > > public class FlatTableSqlQuoteUtils { > public static final String QUOTE = "`"; > /** > * Quote identifier by default quote ` > * @param identifier > * @return > */ > public static String quoteIdentifier(String identifier){ > return QUOTE + identifier + QUOTE; > } > > The solution may be to add a parameter to change this character used for > quoting. > > Just to be sure I made a copy of the original "sqoop" command to "sqoop.orig" > and replaced the original with a little script of mine (attached). > This little script removes the "`" character from the original command and > the call "sqoop.orig" with the cleaned arguments. > It works, but it is still an ugly workaround. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3905) Enable shrunken dictionary default
[ https://issues.apache.org/jira/browse/KYLIN-3905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3905: Description: When using bitmap measure on a large cardinality column(require global dictionaty), build base cuboid step need frequent cache swap so it cannot finished within a reasonable period. When shrunken dictionary enabled, a new step will be added to build separated dictionary for each `InputSplit`, Mapper of **BuildBaseCuboid** step only has to fetch a smaller dictionary for itself, instead of a larger global dictionary. It will reduce cache swap and make **BuildBaseCuboid** step run as quicker as possible. was: When using bitmap measure on a large cardinality column(require global dictionaty), build base cuboid step need frequent cache swap so it cannot finished within a reasonable period. This patch add a new step to build separated dictionary for each `InputSplit`, Mapper of **BuildBaseCuboid** step only has to fetch a smaller dictionary for itself, instead of a larger global dictionary. It will reduce cache swap and make **BuildBaseCuboid** step run as quicker as possible. > Enable shrunken dictionary default > -- > > Key: KYLIN-3905 > URL: https://issues.apache.org/jira/browse/KYLIN-3905 > Project: Kylin > Issue Type: Improvement >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Minor > > When using bitmap measure on a large cardinality column(require global > dictionaty), build base cuboid step need frequent cache swap so it cannot > finished within a reasonable period. > When shrunken dictionary enabled, a new step will be added to build separated > dictionary for each `InputSplit`, Mapper of **BuildBaseCuboid** step only has > to fetch a smaller dictionary for itself, instead of a larger global > dictionary. It will reduce cache swap and make **BuildBaseCuboid** step run > as quicker as possible. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3905) Enable shrunken dictionary default
XiaoXiang Yu created KYLIN-3905: --- Summary: Enable shrunken dictionary default Key: KYLIN-3905 URL: https://issues.apache.org/jira/browse/KYLIN-3905 Project: Kylin Issue Type: Improvement Reporter: XiaoXiang Yu Assignee: XiaoXiang Yu -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3905) Enable shrunken dictionary default
[ https://issues.apache.org/jira/browse/KYLIN-3905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16800378#comment-16800378 ] XiaoXiang Yu commented on KYLIN-3905: - h2. Comparsion and Summary {quote} * CDH cluster with 56 vcore and 110GB Memory * Fact Table 153326740 rows * Build cube with three bitmap count-distinct measure, one column's cardinality is 55200325{quote} h4. Without ShrunkenDict (Cannot completed) * Build basecuboid cannot completed h4. With ShrunkenDict (Completed) * New added step build ShrunkenDict for each map task [!https://user-images.githubusercontent.com/14030549/54500164-cd2efd00-4954-11e9-85a1-8ae5e67063c7.png|width=355!|https://user-images.githubusercontent.com/14030549/54500164-cd2efd00-4954-11e9-85a1-8ae5e67063c7.png] * MapReduce Job Stats [!https://user-images.githubusercontent.com/14030549/54500186-12532f00-4955-11e9-9d61-202f92ca54e5.png|width=1151!|https://user-images.githubusercontent.com/14030549/54500186-12532f00-4955-11e9-9d61-202f92ca54e5.png] * ShrunkenDict in HDFS [!https://user-images.githubusercontent.com/14030549/54341171-286ea000-4674-11e9-8d99-560e94d37cc4.png!|https://user-images.githubusercontent.com/14030549/54341171-286ea000-4674-11e9-8d99-560e94d37cc4.png] [!https://user-images.githubusercontent.com/14030549/54341626-3a9d0e00-4675-11e9-962c-c6a805f3208f.png!|https://user-images.githubusercontent.com/14030549/54341626-3a9d0e00-4675-11e9-962c-c6a805f3208f.png] > Enable shrunken dictionary default > -- > > Key: KYLIN-3905 > URL: https://issues.apache.org/jira/browse/KYLIN-3905 > Project: Kylin > Issue Type: Improvement >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Minor > > When using bitmap measure on a large cardinality column(require global > dictionaty), build base cuboid step need frequent cache swap so it cannot > finished within a reasonable period. > When shrunken dictionary enabled, a new step will be added to build separated > dictionary for each `InputSplit`, Mapper of **BuildBaseCuboid** step only has > to fetch a smaller dictionary for itself, instead of a larger global > dictionary. It will reduce cache swap and make **BuildBaseCuboid** step run > as quicker as possible. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3905) Enable shrunken dictionary default
[ https://issues.apache.org/jira/browse/KYLIN-3905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3905: Description: When using bitmap measure on a large cardinality column(require global dictionaty), build base cuboid step need frequent cache swap so it cannot finished within a reasonable period. This patch add a new step to build separated dictionary for each `InputSplit`, Mapper of **BuildBaseCuboid** step only has to fetch a smaller dictionary for itself, instead of a larger global dictionary. It will reduce cache swap and make **BuildBaseCuboid** step run as quicker as possible. > Enable shrunken dictionary default > -- > > Key: KYLIN-3905 > URL: https://issues.apache.org/jira/browse/KYLIN-3905 > Project: Kylin > Issue Type: Improvement >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Minor > > When using bitmap measure on a large cardinality column(require global > dictionaty), build base cuboid step need frequent cache swap so it cannot > finished within a reasonable period. > This patch add a new step to build separated dictionary for each > `InputSplit`, Mapper of **BuildBaseCuboid** step only has to fetch a smaller > dictionary for itself, instead of a larger global dictionary. It will reduce > cache swap and make **BuildBaseCuboid** step run as quicker as possible. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3905) Enable shrunken dictionary default
[ https://issues.apache.org/jira/browse/KYLIN-3905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16800378#comment-16800378 ] XiaoXiang Yu edited comment on KYLIN-3905 at 3/25/19 3:28 AM: -- h2. Comparsion and Summary {quote} * CDH cluster with 56 vcore and 110GB Memory * Fact Table 153326740 rows * Build cube with three bitmap count-distinct measure, one column's cardinality is 55200325{quote} h4. Without ShrunkenDict (Cannot completed) * Build basecuboid cannot completed !image-2019-03-25-11-26-59-198.png! h4. With ShrunkenDict (Completed) * New added step build ShrunkenDict for each map task !image-2019-03-25-11-27-26-149.png! * MapReduce Job Stats !image-2019-03-25-11-27-46-175.png![!https://user-images.githubusercontent.com/14030549/54500186-12532f00-4955-11e9-9d61-202f92ca54e5.png|width=1151!|https://user-images.githubusercontent.com/14030549/54500186-12532f00-4955-11e9-9d61-202f92ca54e5.png] !image-2019-03-25-11-28-14-256.png! [!https://user-images.githubusercontent.com/14030549/54341171-286ea000-4674-11e9-8d99-560e94d37cc4.png!|https://user-images.githubusercontent.com/14030549/54341171-286ea000-4674-11e9-8d99-560e94d37cc4.png] [!https://user-images.githubusercontent.com/14030549/54341626-3a9d0e00-4675-11e9-962c-c6a805f3208f.png!|https://user-images.githubusercontent.com/14030549/54341626-3a9d0e00-4675-11e9-962c-c6a805f3208f.png] was (Author: hit_lacus): h2. Comparsion and Summary {quote} * CDH cluster with 56 vcore and 110GB Memory * Fact Table 153326740 rows * Build cube with three bitmap count-distinct measure, one column's cardinality is 55200325{quote} h4. Without ShrunkenDict (Cannot completed) * Build basecuboid cannot completed h4. With ShrunkenDict (Completed) * New added step build ShrunkenDict for each map task [!https://user-images.githubusercontent.com/14030549/54500164-cd2efd00-4954-11e9-85a1-8ae5e67063c7.png|width=355!|https://user-images.githubusercontent.com/14030549/54500164-cd2efd00-4954-11e9-85a1-8ae5e67063c7.png] * MapReduce Job Stats [!https://user-images.githubusercontent.com/14030549/54500186-12532f00-4955-11e9-9d61-202f92ca54e5.png|width=1151!|https://user-images.githubusercontent.com/14030549/54500186-12532f00-4955-11e9-9d61-202f92ca54e5.png] * ShrunkenDict in HDFS [!https://user-images.githubusercontent.com/14030549/54341171-286ea000-4674-11e9-8d99-560e94d37cc4.png!|https://user-images.githubusercontent.com/14030549/54341171-286ea000-4674-11e9-8d99-560e94d37cc4.png] [!https://user-images.githubusercontent.com/14030549/54341626-3a9d0e00-4675-11e9-962c-c6a805f3208f.png!|https://user-images.githubusercontent.com/14030549/54341626-3a9d0e00-4675-11e9-962c-c6a805f3208f.png] > Enable shrunken dictionary default > -- > > Key: KYLIN-3905 > URL: https://issues.apache.org/jira/browse/KYLIN-3905 > Project: Kylin > Issue Type: Improvement >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Minor > Attachments: image-2019-03-25-11-26-59-198.png, > image-2019-03-25-11-27-26-149.png, image-2019-03-25-11-27-46-175.png, > image-2019-03-25-11-28-14-256.png > > > When using bitmap measure on a large cardinality column(require global > dictionaty), build base cuboid step need frequent cache swap so it cannot > finished within a reasonable period. > When shrunken dictionary enabled, a new step will be added to build separated > dictionary for each `InputSplit`, Mapper of **BuildBaseCuboid** step only has > to fetch a smaller dictionary for itself, instead of a larger global > dictionary. It will reduce cache swap and make **BuildBaseCuboid** step run > as quicker as possible. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3905) Enable shrunken dictionary default
[ https://issues.apache.org/jira/browse/KYLIN-3905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16800379#comment-16800379 ] XiaoXiang Yu commented on KYLIN-3905: - h3. Return time statistics of global dictionary(without Shrunken Dictionary) !image-2019-03-25-11-29-19-383.png! > Enable shrunken dictionary default > -- > > Key: KYLIN-3905 > URL: https://issues.apache.org/jira/browse/KYLIN-3905 > Project: Kylin > Issue Type: Improvement >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Minor > Attachments: image-2019-03-25-11-26-59-198.png, > image-2019-03-25-11-27-26-149.png, image-2019-03-25-11-27-46-175.png, > image-2019-03-25-11-28-14-256.png, image-2019-03-25-11-29-19-383.png > > > When using bitmap measure on a large cardinality column(require global > dictionaty), build base cuboid step need frequent cache swap so it cannot > finished within a reasonable period. > When shrunken dictionary enabled, a new step will be added to build separated > dictionary for each `InputSplit`, Mapper of **BuildBaseCuboid** step only has > to fetch a smaller dictionary for itself, instead of a larger global > dictionary. It will reduce cache swap and make **BuildBaseCuboid** step run > as quicker as possible. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3905) Enable shrunken dictionary default
[ https://issues.apache.org/jira/browse/KYLIN-3905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3905: Attachment: image-2019-03-25-11-29-19-383.png > Enable shrunken dictionary default > -- > > Key: KYLIN-3905 > URL: https://issues.apache.org/jira/browse/KYLIN-3905 > Project: Kylin > Issue Type: Improvement >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Minor > Attachments: image-2019-03-25-11-26-59-198.png, > image-2019-03-25-11-27-26-149.png, image-2019-03-25-11-27-46-175.png, > image-2019-03-25-11-28-14-256.png, image-2019-03-25-11-29-19-383.png > > > When using bitmap measure on a large cardinality column(require global > dictionaty), build base cuboid step need frequent cache swap so it cannot > finished within a reasonable period. > When shrunken dictionary enabled, a new step will be added to build separated > dictionary for each `InputSplit`, Mapper of **BuildBaseCuboid** step only has > to fetch a smaller dictionary for itself, instead of a larger global > dictionary. It will reduce cache swap and make **BuildBaseCuboid** step run > as quicker as possible. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3905) Enable shrunken dictionary default
[ https://issues.apache.org/jira/browse/KYLIN-3905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3905: Fix Version/s: v2.6.2 Component/s: Measure - Count Distinct > Enable shrunken dictionary default > -- > > Key: KYLIN-3905 > URL: https://issues.apache.org/jira/browse/KYLIN-3905 > Project: Kylin > Issue Type: Improvement > Components: Measure - Count Distinct >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Minor > Fix For: v2.6.2 > > Attachments: image-2019-03-25-11-26-59-198.png, > image-2019-03-25-11-27-26-149.png, image-2019-03-25-11-27-46-175.png, > image-2019-03-25-11-28-14-256.png, image-2019-03-25-11-29-19-383.png > > > When using bitmap measure on a large cardinality column(require global > dictionaty), build base cuboid step need frequent cache swap so it cannot > finished within a reasonable period. > When shrunken dictionary enabled, a new step will be added to build separated > dictionary for each `InputSplit`, Mapper of **BuildBaseCuboid** step only has > to fetch a smaller dictionary for itself, instead of a larger global > dictionary. It will reduce cache swap and make **BuildBaseCuboid** step run > as quicker as possible. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3905) Enable shrunken dictionary default
[ https://issues.apache.org/jira/browse/KYLIN-3905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16800378#comment-16800378 ] XiaoXiang Yu edited comment on KYLIN-3905 at 3/25/19 3:35 AM: -- h2. Comparsion and Summary {quote} * CDH cluster with 56 vcore and 110GB Memory * Fact Table 153326740 rows * Build cube with three bitmap count-distinct measure, one column's cardinality is 55200325{quote} h4. Without ShrunkenDict (Cannot completed) * Build basecuboid cannot completed !image-2019-03-25-11-26-59-198.png! h4. With ShrunkenDict (Completed) * MapReduce Job Stats [!https://user-images.githubusercontent.com/14030549/54500186-12532f00-4955-11e9-9d61-202f92ca54e5.png|width=1151!|https://user-images.githubusercontent.com/14030549/54500186-12532f00-4955-11e9-9d61-202f92ca54e5.png] !image-2019-03-25-11-28-14-256.png! was (Author: hit_lacus): h2. Comparsion and Summary {quote} * CDH cluster with 56 vcore and 110GB Memory * Fact Table 153326740 rows * Build cube with three bitmap count-distinct measure, one column's cardinality is 55200325{quote} h4. Without ShrunkenDict (Cannot completed) * Build basecuboid cannot completed !image-2019-03-25-11-26-59-198.png! h4. With ShrunkenDict (Completed) * New added step build ShrunkenDict for each map task !image-2019-03-25-11-27-26-149.png! * MapReduce Job Stats !image-2019-03-25-11-27-46-175.png![!https://user-images.githubusercontent.com/14030549/54500186-12532f00-4955-11e9-9d61-202f92ca54e5.png|width=1151!|https://user-images.githubusercontent.com/14030549/54500186-12532f00-4955-11e9-9d61-202f92ca54e5.png] !image-2019-03-25-11-28-14-256.png! [!https://user-images.githubusercontent.com/14030549/54341171-286ea000-4674-11e9-8d99-560e94d37cc4.png!|https://user-images.githubusercontent.com/14030549/54341171-286ea000-4674-11e9-8d99-560e94d37cc4.png] [!https://user-images.githubusercontent.com/14030549/54341626-3a9d0e00-4675-11e9-962c-c6a805f3208f.png!|https://user-images.githubusercontent.com/14030549/54341626-3a9d0e00-4675-11e9-962c-c6a805f3208f.png] > Enable shrunken dictionary default > -- > > Key: KYLIN-3905 > URL: https://issues.apache.org/jira/browse/KYLIN-3905 > Project: Kylin > Issue Type: Improvement > Components: Measure - Count Distinct >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Minor > Fix For: v2.6.2 > > Attachments: image-2019-03-25-11-26-59-198.png, > image-2019-03-25-11-27-26-149.png, image-2019-03-25-11-27-46-175.png, > image-2019-03-25-11-28-14-256.png, image-2019-03-25-11-29-19-383.png > > > When using bitmap measure on a large cardinality column(require global > dictionaty), build base cuboid step need frequent cache swap so it cannot > finished within a reasonable period. > When shrunken dictionary enabled, a new step will be added to build separated > dictionary for each `InputSplit`, Mapper of **BuildBaseCuboid** step only has > to fetch a smaller dictionary for itself, instead of a larger global > dictionary. It will reduce cache swap and make **BuildBaseCuboid** step run > as quicker as possible. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3905) Enable shrunken dictionary default
[ https://issues.apache.org/jira/browse/KYLIN-3905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3905: Description: In dev mail list's discussion, I suggest to enable shrunken dictionary by default, and received some dev's aggrement. When using bitmap measure on a large cardinality column(require global dictionaty), build base cuboid step need frequent cache swap so it cannot finished within a reasonable period. When shrunken dictionary enabled, a new step will be added to build separated dictionary for each `InputSplit`, Mapper of **BuildBaseCuboid** step only has to fetch a smaller dictionary for itself, instead of a larger global dictionary. It will reduce cache swap and make **BuildBaseCuboid** step run as quicker as possible. http://mail-archives.apache.org/mod_mbox/kylin-dev//201903.mbox/%3c62efcb72-b235-4fc3-9add-0fc510d97...@kyligence.io%3e was: In dev mail list's discussion, I suggest to enable http://mail-archives.apache.org/mod_mbox/kylin-dev//201903.mbox/%3c62efcb72-b235-4fc3-9add-0fc510d97...@kyligence.io%3e When using bitmap measure on a large cardinality column(require global dictionaty), build base cuboid step need frequent cache swap so it cannot finished within a reasonable period. When shrunken dictionary enabled, a new step will be added to build separated dictionary for each `InputSplit`, Mapper of **BuildBaseCuboid** step only has to fetch a smaller dictionary for itself, instead of a larger global dictionary. It will reduce cache swap and make **BuildBaseCuboid** step run as quicker as possible. > Enable shrunken dictionary default > -- > > Key: KYLIN-3905 > URL: https://issues.apache.org/jira/browse/KYLIN-3905 > Project: Kylin > Issue Type: Improvement > Components: Measure - Count Distinct >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Minor > Fix For: v2.6.2 > > Attachments: image-2019-03-25-11-26-59-198.png, > image-2019-03-25-11-27-26-149.png, image-2019-03-25-11-27-46-175.png, > image-2019-03-25-11-28-14-256.png, image-2019-03-25-11-29-19-383.png > > > In dev mail list's discussion, I suggest to enable shrunken dictionary by > default, and received some dev's aggrement. > When using bitmap measure on a large cardinality column(require global > dictionaty), build base cuboid step need frequent cache swap so it cannot > finished within a reasonable period. > When shrunken dictionary enabled, a new step will be added to build > separated dictionary for each `InputSplit`, Mapper of **BuildBaseCuboid** > step only has to fetch a smaller dictionary for itself, instead of a larger > global dictionary. It will reduce cache swap and make **BuildBaseCuboid** > step run as quicker as possible. > > http://mail-archives.apache.org/mod_mbox/kylin-dev//201903.mbox/%3c62efcb72-b235-4fc3-9add-0fc510d97...@kyligence.io%3e -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3905) Enable shrunken dictionary default
[ https://issues.apache.org/jira/browse/KYLIN-3905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16800379#comment-16800379 ] XiaoXiang Yu edited comment on KYLIN-3905 at 3/25/19 3:42 AM: -- h3. Return time statistics of global dictionary(without Shrunken Dictionary) In following screenshot, we can find that cache swap occupy most duration when build base cuboid related to Bitmap measure. !image-2019-03-25-11-29-19-383.png! was (Author: hit_lacus): h3. Return time statistics of global dictionary(without Shrunken Dictionary) !image-2019-03-25-11-29-19-383.png! > Enable shrunken dictionary default > -- > > Key: KYLIN-3905 > URL: https://issues.apache.org/jira/browse/KYLIN-3905 > Project: Kylin > Issue Type: Improvement > Components: Measure - Count Distinct >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Minor > Fix For: v2.6.2 > > Attachments: image-2019-03-25-11-26-59-198.png, > image-2019-03-25-11-27-26-149.png, image-2019-03-25-11-27-46-175.png, > image-2019-03-25-11-28-14-256.png, image-2019-03-25-11-29-19-383.png > > > In dev mail list's discussion, I suggest to enable shrunken dictionary by > default, and received some dev's aggrement. > When using bitmap measure on a large cardinality column(require global > dictionaty), build base cuboid step need frequent cache swap so it cannot > finished within a reasonable period. > When shrunken dictionary enabled, a new step will be added to build > separated dictionary for each `InputSplit`, Mapper of **BuildBaseCuboid** > step only has to fetch a smaller dictionary for itself, instead of a larger > global dictionary. It will reduce cache swap and make **BuildBaseCuboid** > step run as quicker as possible. > > http://mail-archives.apache.org/mod_mbox/kylin-dev//201903.mbox/%3c62efcb72-b235-4fc3-9add-0fc510d97...@kyligence.io%3e -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3905) Enable shrunken dictionary default
[ https://issues.apache.org/jira/browse/KYLIN-3905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3905: Description: In dev mail list's discussion, I suggest to enable http://mail-archives.apache.org/mod_mbox/kylin-dev//201903.mbox/%3c62efcb72-b235-4fc3-9add-0fc510d97...@kyligence.io%3e When using bitmap measure on a large cardinality column(require global dictionaty), build base cuboid step need frequent cache swap so it cannot finished within a reasonable period. When shrunken dictionary enabled, a new step will be added to build separated dictionary for each `InputSplit`, Mapper of **BuildBaseCuboid** step only has to fetch a smaller dictionary for itself, instead of a larger global dictionary. It will reduce cache swap and make **BuildBaseCuboid** step run as quicker as possible. was: When using bitmap measure on a large cardinality column(require global dictionaty), build base cuboid step need frequent cache swap so it cannot finished within a reasonable period. When shrunken dictionary enabled, a new step will be added to build separated dictionary for each `InputSplit`, Mapper of **BuildBaseCuboid** step only has to fetch a smaller dictionary for itself, instead of a larger global dictionary. It will reduce cache swap and make **BuildBaseCuboid** step run as quicker as possible. > Enable shrunken dictionary default > -- > > Key: KYLIN-3905 > URL: https://issues.apache.org/jira/browse/KYLIN-3905 > Project: Kylin > Issue Type: Improvement > Components: Measure - Count Distinct >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Minor > Fix For: v2.6.2 > > Attachments: image-2019-03-25-11-26-59-198.png, > image-2019-03-25-11-27-26-149.png, image-2019-03-25-11-27-46-175.png, > image-2019-03-25-11-28-14-256.png, image-2019-03-25-11-29-19-383.png > > > In dev mail list's discussion, I suggest to enable > > http://mail-archives.apache.org/mod_mbox/kylin-dev//201903.mbox/%3c62efcb72-b235-4fc3-9add-0fc510d97...@kyligence.io%3e > > When using bitmap measure on a large cardinality column(require global > dictionaty), build base cuboid step need frequent cache swap so it cannot > finished within a reasonable period. > When shrunken dictionary enabled, a new step will be added to build > separated dictionary for each `InputSplit`, Mapper of **BuildBaseCuboid** > step only has to fetch a smaller dictionary for itself, instead of a larger > global dictionary. It will reduce cache swap and make **BuildBaseCuboid** > step run as quicker as possible. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-3792) Reassign's inconsistent state need automation recovery
[ https://issues.apache.org/jira/browse/KYLIN-3792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu closed KYLIN-3792. --- Resolution: Won't Do > Reassign's inconsistent state need automation recovery > -- > > Key: KYLIN-3792 > URL: https://issues.apache.org/jira/browse/KYLIN-3792 > Project: Kylin > Issue Type: Improvement > Components: Real-time Streaming >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Labels: reassign, won't-fix > Original Estimate: 96h > Remaining Estimate: 96h > > Realtime streaming has a reassign/rebalance operation which may cause some > inconsistent state which is difficult to recover. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3792) Reassign's inconsistent state need automation recovery
[ https://issues.apache.org/jira/browse/KYLIN-3792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3792: Labels: reassign won't-fix (was: reassign) > Reassign's inconsistent state need automation recovery > -- > > Key: KYLIN-3792 > URL: https://issues.apache.org/jira/browse/KYLIN-3792 > Project: Kylin > Issue Type: Improvement > Components: Real-time Streaming >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Labels: reassign, won't-fix > Original Estimate: 96h > Remaining Estimate: 96h > > Realtime streaming has a reassign/rebalance operation which may cause some > inconsistent state which is difficult to recover. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-3711) Clean unused/duplicated code
[ https://issues.apache.org/jira/browse/KYLIN-3711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu closed KYLIN-3711. --- Resolution: Not A Problem > Clean unused/duplicated code > > > Key: KYLIN-3711 > URL: https://issues.apache.org/jira/browse/KYLIN-3711 > Project: Kylin > Issue Type: Improvement >Affects Versions: Future >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Minor > Labels: Refactoring > > I am reading source code and found some unused class, such as > +org.apache.kylin.rest.broadcaster.BroadcasterReceiveServlet+ , this class is > created at 2015 and seems do nothing now(it should be replcaed by > +org.apache.kylin.rest.controller.CacheController+). > > So someone should commented it with "duplicated" or simply delete > it.[BroadcasterReceiveServlet|https://github.com/apache/kylin/blob/kylin-2.5.0/server-base/src/main/java/org/apache/kylin/rest/broadcaster/BroadcasterReceiveServlet.java] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3942) Rea-time OLAP don't support multi-level json event
XiaoXiang Yu created KYLIN-3942: --- Summary: Rea-time OLAP don't support multi-level json event Key: KYLIN-3942 URL: https://issues.apache.org/jira/browse/KYLIN-3942 Project: Kylin Issue Type: Bug Components: Real-time Streaming Affects Versions: v3.0.0-alpha Reporter: XiaoXiang Yu Assignee: XiaoXiang Yu Currently real-time OLAP didn't support multi-level json event. For example,if I have a kafka multi-level json event like this: {quote}{"country":"JAPAN","amount":13.075058425023922,"qty":8,"currency":"USD","order_time":1554801950882,"category":"ELECTRONIC","device":"Andriod","user":\{"gender":"Female","id":"7a0cfa5e-bbaa-79ef-1a38-e06f02c85fcb","first_name":"unknown","age":16}} {quote} Receiver will throw exception like this and discard that event: {quote}2019-04-09 09:46:09,878 ERROR [StreamingV2Cube_channel] kafka.TimedJsonStreamParser:107 : error com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot deserialize instance of `java.lang.String` out of START_OBJECT token at [Source: (String)"\{"country":"US","amount":14.498498222823619,"qty":1,"currency":"USD","order_time":1554803169876,"category":"Other","device":"Other","user":{"gender":"Female","id":"0736b41a-9ae7-9b4a-a124-f74436d3eb41","first_name":"unknown","age":26}}"; line: 1, column: 140] (through reference chain: java.util.HashMap["user"]) at com.fasterxml.jackson.databind.exc.MismatchedInputException.from(MismatchedInputException.java:63) at com.fasterxml.jackson.databind.DeserializationContext.reportInputMismatch(DeserializationContext.java:1342) at com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1138) at com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1092) at com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:63) at com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:10) at com.fasterxml.jackson.databind.deser.std.MapDeserializer._readAndBindStringKeyMap(MapDeserializer.java:527) at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:364) at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:29) at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4001) at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3030) at org.apache.kylin.stream.source.kafka.TimedJsonStreamParser.parse(TimedJsonStreamParser.java:79) at org.apache.kylin.stream.source.kafka.TimedJsonStreamParser.parse(TimedJsonStreamParser.java:54) at org.apache.kylin.stream.source.kafka.consumer.KafkaConnector.nextEvent(KafkaConnector.java:110) at org.apache.kylin.stream.core.consumer.StreamingConsumerChannel.run(StreamingConsumerChannel.java:93) at java.lang.Thread.run(Thread.java:748) {quote} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3944) Streaming Cube UI display empty cube planner info
XiaoXiang Yu created KYLIN-3944: --- Summary: Streaming Cube UI display empty cube planner info Key: KYLIN-3944 URL: https://issues.apache.org/jira/browse/KYLIN-3944 Project: Kylin Issue Type: Bug Components: Website Affects Versions: v2.6.1 Reporter: XiaoXiang Yu Attachments: image-2019-04-10-10-58-17-452.png, image-2019-04-10-10-59-07-532.png In cube design page, I click "Planner" tab, it will show empty table. !image-2019-04-10-10-58-17-452.png! !image-2019-04-10-10-59-07-532.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3944) Streaming Cube UI display empty cube planner info
[ https://issues.apache.org/jira/browse/KYLIN-3944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3944: Description: In cube design page, I click "Planner" tab, it will show "Cuboid Distribution" empty table. !image-2019-04-10-10-58-17-452.png! !image-2019-04-10-10-59-07-532.png! was: In cube design page, I click "Planner" tab, it will show empty table. !image-2019-04-10-10-58-17-452.png! !image-2019-04-10-10-59-07-532.png! > Streaming Cube UI display empty cube planner info > - > > Key: KYLIN-3944 > URL: https://issues.apache.org/jira/browse/KYLIN-3944 > Project: Kylin > Issue Type: Bug > Components: Website >Affects Versions: v2.6.1 >Reporter: XiaoXiang Yu >Priority: Major > Attachments: image-2019-04-10-10-58-17-452.png, > image-2019-04-10-10-59-07-532.png > > > > In cube design page, I click "Planner" tab, it will show "Cuboid > Distribution" empty table. > > > !image-2019-04-10-10-58-17-452.png! > > !image-2019-04-10-10-59-07-532.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3944) Streaming Cube UI display empty cube planner info
[ https://issues.apache.org/jira/browse/KYLIN-3944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3944: Description: In cube design page, I click "Planner" tab, it will show "Cuboid Distribution" empty table. And these empty info won't disappear if I change to another streaming tab. !image-2019-04-10-10-58-17-452.png! !image-2019-04-10-10-59-07-532.png! was: In cube design page, I click "Planner" tab, it will show "Cuboid Distribution" empty table. !image-2019-04-10-10-58-17-452.png! !image-2019-04-10-10-59-07-532.png! > Streaming Cube UI display empty cube planner info > - > > Key: KYLIN-3944 > URL: https://issues.apache.org/jira/browse/KYLIN-3944 > Project: Kylin > Issue Type: Bug > Components: Website >Affects Versions: v2.6.1 >Reporter: XiaoXiang Yu >Priority: Major > Attachments: image-2019-04-10-10-58-17-452.png, > image-2019-04-10-10-59-07-532.png > > > > In cube design page, I click "Planner" tab, it will show "Cuboid > Distribution" empty table. > And these empty info won't disappear if I change to another streaming tab. > > > !image-2019-04-10-10-58-17-452.png! > > !image-2019-04-10-10-59-07-532.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-3944) Streaming Cube UI display empty cube planner info
[ https://issues.apache.org/jira/browse/KYLIN-3944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu closed KYLIN-3944. --- Resolution: Not A Bug It's a problem caused by error packaging. > Streaming Cube UI display empty cube planner info > - > > Key: KYLIN-3944 > URL: https://issues.apache.org/jira/browse/KYLIN-3944 > Project: Kylin > Issue Type: Bug > Components: Website >Affects Versions: v2.6.1 >Reporter: XiaoXiang Yu >Priority: Major > Attachments: image-2019-04-10-10-58-17-452.png, > image-2019-04-10-10-59-07-532.png > > > > In cube design page, I click "Planner" tab, it will show "Cuboid > Distribution" empty table. > And these empty info won't disappear if I change to another streaming tab. > > > !image-2019-04-10-10-58-17-452.png! > > !image-2019-04-10-10-59-07-532.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3959) Realtime OLAP query result should not be cached
XiaoXiang Yu created KYLIN-3959: --- Summary: Realtime OLAP query result should not be cached Key: KYLIN-3959 URL: https://issues.apache.org/jira/browse/KYLIN-3959 Project: Kylin Issue Type: Bug Components: Real-time Streaming Affects Versions: v3.0.0-alpha Reporter: XiaoXiang Yu Assignee: XiaoXiang Yu Fix For: v3.0.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-3609) NPE from QueryMetricsFacade#updateMetricsToReservoir
[ https://issues.apache.org/jira/browse/KYLIN-3609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu closed KYLIN-3609. --- Resolution: Cannot Reproduce > NPE from QueryMetricsFacade#updateMetricsToReservoir > > > Key: KYLIN-3609 > URL: https://issues.apache.org/jira/browse/KYLIN-3609 > Project: Kylin > Issue Type: Bug >Reporter: Ted Yu >Assignee: XiaoXiang Yu >Priority: Minor > > When running test suite, I saw the following in test output: > {code} > 2018-10-02 20:59:20,415 WARN [Query c8f42f2e-8c77-2bfc-97ab-053fbeb7c86e-1] > service.QueryService:423 : Write metric error. > java.lang.NullPointerException > at > org.apache.kylin.rest.metrics.QueryMetricsFacade.updateMetricsToReservoir(QueryMetricsFacade.java:148) > at > org.apache.kylin.rest.metrics.QueryMetricsFacade.updateMetrics(QueryMetricsFacade.java:74) > at > org.apache.kylin.rest.service.QueryService.recordMetric(QueryService.java:505) > at > org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:421) > at > org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:353) > at > org.apache.kylin.rest.controller.QueryController.query(QueryController.java:87) > at > org.apache.kylin.rest.controller.QueryControllerTest.testQueryException(QueryControllerTest.java:63) > {code} > It seems sqlResponse.getResults() returned null. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3959) Realtime OLAP query result should not be cached
[ https://issues.apache.org/jira/browse/KYLIN-3959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu resolved KYLIN-3959. - Resolution: Fixed > Realtime OLAP query result should not be cached > --- > > Key: KYLIN-3959 > URL: https://issues.apache.org/jira/browse/KYLIN-3959 > Project: Kylin > Issue Type: Bug > Components: Real-time Streaming >Affects Versions: v3.0.0-alpha >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Fix For: v3.0.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KYLIN-3942) Rea-time OLAP don't support multi-level json event
[ https://issues.apache.org/jira/browse/KYLIN-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu reassigned KYLIN-3942: --- Assignee: XiaoXiang Yu (was: luguosheng) Fix Version/s: v3.0.0 > Rea-time OLAP don't support multi-level json event > -- > > Key: KYLIN-3942 > URL: https://issues.apache.org/jira/browse/KYLIN-3942 > Project: Kylin > Issue Type: Bug > Components: Real-time Streaming >Affects Versions: v3.0.0-alpha >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Critical > Fix For: v3.0.0 > > > Currently real-time OLAP didn't support multi-level json event. > For example,if I have a kafka multi-level json event like this: > {quote}{"country":"JAPAN","amount":13.075058425023922,"qty":8,"currency":"USD","order_time":1554801950882,"category":"ELECTRONIC","device":"Andriod","user":\{"gender":"Female","id":"7a0cfa5e-bbaa-79ef-1a38-e06f02c85fcb","first_name":"unknown","age":16}} > {quote} > > Receiver will throw exception like this and discard that event: > > {quote}2019-04-09 09:46:09,878 ERROR [StreamingV2Cube_channel] > kafka.TimedJsonStreamParser:107 : error > com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot > deserialize instance of `java.lang.String` out of START_OBJECT token > at [Source: > (String)"\{"country":"US","amount":14.498498222823619,"qty":1,"currency":"USD","order_time":1554803169876,"category":"Other","device":"Other","user":{"gender":"Female","id":"0736b41a-9ae7-9b4a-a124-f74436d3eb41","first_name":"unknown","age":26}}"; > line: 1, column: 140] (through reference chain: java.util.HashMap["user"]) > at > com.fasterxml.jackson.databind.exc.MismatchedInputException.from(MismatchedInputException.java:63) > at > com.fasterxml.jackson.databind.DeserializationContext.reportInputMismatch(DeserializationContext.java:1342) > at > com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1138) > at > com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1092) > at > com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:63) > at > com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:10) > at > com.fasterxml.jackson.databind.deser.std.MapDeserializer._readAndBindStringKeyMap(MapDeserializer.java:527) > at > com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:364) > at > com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:29) > at > com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4001) > at > com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3030) > at > org.apache.kylin.stream.source.kafka.TimedJsonStreamParser.parse(TimedJsonStreamParser.java:79) > at > org.apache.kylin.stream.source.kafka.TimedJsonStreamParser.parse(TimedJsonStreamParser.java:54) > at > org.apache.kylin.stream.source.kafka.consumer.KafkaConnector.nextEvent(KafkaConnector.java:110) > at > org.apache.kylin.stream.core.consumer.StreamingConsumerChannel.run(StreamingConsumerChannel.java:93) > at java.lang.Thread.run(Thread.java:748) > {quote} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3942) Rea-time OLAP don't support multi-level json event
[ https://issues.apache.org/jira/browse/KYLIN-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu resolved KYLIN-3942. - Resolution: Fixed > Rea-time OLAP don't support multi-level json event > -- > > Key: KYLIN-3942 > URL: https://issues.apache.org/jira/browse/KYLIN-3942 > Project: Kylin > Issue Type: Bug > Components: Real-time Streaming >Affects Versions: v3.0.0-alpha >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Critical > Fix For: v3.0.0 > > > Currently real-time OLAP didn't support multi-level json event. > For example,if I have a kafka multi-level json event like this: > {quote}{"country":"JAPAN","amount":13.075058425023922,"qty":8,"currency":"USD","order_time":1554801950882,"category":"ELECTRONIC","device":"Andriod","user":\{"gender":"Female","id":"7a0cfa5e-bbaa-79ef-1a38-e06f02c85fcb","first_name":"unknown","age":16}} > {quote} > > Receiver will throw exception like this and discard that event: > > {quote}2019-04-09 09:46:09,878 ERROR [StreamingV2Cube_channel] > kafka.TimedJsonStreamParser:107 : error > com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot > deserialize instance of `java.lang.String` out of START_OBJECT token > at [Source: > (String)"\{"country":"US","amount":14.498498222823619,"qty":1,"currency":"USD","order_time":1554803169876,"category":"Other","device":"Other","user":{"gender":"Female","id":"0736b41a-9ae7-9b4a-a124-f74436d3eb41","first_name":"unknown","age":26}}"; > line: 1, column: 140] (through reference chain: java.util.HashMap["user"]) > at > com.fasterxml.jackson.databind.exc.MismatchedInputException.from(MismatchedInputException.java:63) > at > com.fasterxml.jackson.databind.DeserializationContext.reportInputMismatch(DeserializationContext.java:1342) > at > com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1138) > at > com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1092) > at > com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:63) > at > com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:10) > at > com.fasterxml.jackson.databind.deser.std.MapDeserializer._readAndBindStringKeyMap(MapDeserializer.java:527) > at > com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:364) > at > com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:29) > at > com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4001) > at > com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3030) > at > org.apache.kylin.stream.source.kafka.TimedJsonStreamParser.parse(TimedJsonStreamParser.java:79) > at > org.apache.kylin.stream.source.kafka.TimedJsonStreamParser.parse(TimedJsonStreamParser.java:54) > at > org.apache.kylin.stream.source.kafka.consumer.KafkaConnector.nextEvent(KafkaConnector.java:110) > at > org.apache.kylin.stream.core.consumer.StreamingConsumerChannel.run(StreamingConsumerChannel.java:93) > at java.lang.Thread.run(Thread.java:748) > {quote} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3962) Support streaming cubing using Spark Streaming or Flink
[ https://issues.apache.org/jira/browse/KYLIN-3962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16820980#comment-16820980 ] XiaoXiang Yu commented on KYLIN-3962: - If I use Flink streaming to ingest streaming message, and write cuboid data(I think this maybe ) to external storage(like hbase or redis), I think it will have some drawback: - Using remote storage instead of local storage is will increase the data prepare delay. - Introduce external dependency like flink cluster. - Too heavy pressure on external storage. - Filter and aggregate will be slower if data in remote. > Support streaming cubing using Spark Streaming or Flink > --- > > Key: KYLIN-3962 > URL: https://issues.apache.org/jira/browse/KYLIN-3962 > Project: Kylin > Issue Type: Improvement >Reporter: Liu Shaohui >Priority: Major > > KYLIN-3654 introduced the Real-time Streaming, but in my opinion, the arch is > a little too complicated to handle. > As streaming frameworks like spark streaming, flink are widely used in many > companies.Can we use the streaming framework to support real time cubing in > Kylin. > This is just a proposal. More discussion and suggestions are welcomed~ > More details of this proposal will be added later. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3962) Support streaming cubing using Spark Streaming or Flink
[ https://issues.apache.org/jira/browse/KYLIN-3962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16820980#comment-16820980 ] XiaoXiang Yu edited comment on KYLIN-3962 at 4/18/19 11:44 AM: --- If I use Flink streaming to ingest streaming message, and write cuboid data(I think this maybe _++_) to external storage(like hbase or redis), I think it will have some drawback: - Using remote storage instead of local storage will increase the data prepare delay. - Introduce external dependency like flink cluster. - Too heavy pressure on external storage(maybe every entered message will cause a r/w to storage layer). - If we decide to use more cuboid, it will cause too much r/w to storage. If we decide to use more cuboid, most query will hit base cuboid, filter and aggregate will be slower if data in remote (we can use memory cache in receiver). was (Author: hit_lacus): If I use Flink streaming to ingest streaming message, and write cuboid data(I think this maybe ) to external storage(like hbase or redis), I think it will have some drawback: - Using remote storage instead of local storage is will increase the data prepare delay. - Introduce external dependency like flink cluster. - Too heavy pressure on external storage. - Filter and aggregate will be slower if data in remote. > Support streaming cubing using Spark Streaming or Flink > --- > > Key: KYLIN-3962 > URL: https://issues.apache.org/jira/browse/KYLIN-3962 > Project: Kylin > Issue Type: Improvement >Reporter: Liu Shaohui >Priority: Major > > KYLIN-3654 introduced the Real-time Streaming, but in my opinion, the arch is > a little too complicated to handle. > As streaming frameworks like spark streaming, flink are widely used in many > companies.Can we use the streaming framework to support real time cubing in > Kylin. > This is just a proposal. More discussion and suggestions are welcomed~ > More details of this proposal will be added later. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3962) Support streaming cubing using Spark Streaming or Flink
[ https://issues.apache.org/jira/browse/KYLIN-3962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16820980#comment-16820980 ] XiaoXiang Yu edited comment on KYLIN-3962 at 4/18/19 11:50 AM: --- If I use Flink streaming to ingest streaming message, and write cuboid data(I think this maybe _++_) to external storage(like hbase or redis), I think it will have some drawback: - Using remote storage instead of local storage will increase the data prepare delay. - Introduce external dependency like flink cluster. - Too heavy pressure on external storage(maybe every entered message will cause a r/w to storage layer). - If we decide to use more cuboid, it will cause too much r/w to storage. If we decide to use less cuboid, most query will hit base cuboid, filter and aggregate will be slower if data in remote (we can use memory cache in receiver). was (Author: hit_lacus): If I use Flink streaming to ingest streaming message, and write cuboid data(I think this maybe _++_) to external storage(like hbase or redis), I think it will have some drawback: - Using remote storage instead of local storage will increase the data prepare delay. - Introduce external dependency like flink cluster. - Too heavy pressure on external storage(maybe every entered message will cause a r/w to storage layer). - If we decide to use more cuboid, it will cause too much r/w to storage. If we decide to use more cuboid, most query will hit base cuboid, filter and aggregate will be slower if data in remote (we can use memory cache in receiver). > Support streaming cubing using Spark Streaming or Flink > --- > > Key: KYLIN-3962 > URL: https://issues.apache.org/jira/browse/KYLIN-3962 > Project: Kylin > Issue Type: Improvement >Reporter: Liu Shaohui >Priority: Major > > KYLIN-3654 introduced the Real-time Streaming, but in my opinion, the arch is > a little too complicated to handle. > As streaming frameworks like spark streaming, flink are widely used in many > companies.Can we use the streaming framework to support real time cubing in > Kylin. > This is just a proposal. More discussion and suggestions are welcomed~ > More details of this proposal will be added later. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3857) Add parameter to change sql quoting character for sqoop
[ https://issues.apache.org/jira/browse/KYLIN-3857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu resolved KYLIN-3857. - Resolution: Fixed Fix Version/s: v3.0.0 > Add parameter to change sql quoting character for sqoop > --- > > Key: KYLIN-3857 > URL: https://issues.apache.org/jira/browse/KYLIN-3857 > Project: Kylin > Issue Type: Bug >Reporter: Davide Malagoli >Assignee: XiaoXiang Yu >Priority: Major > Fix For: v3.0.0, v2.6.2 > > Attachments: image-2019-03-11-11-46-05-946.png, > image-2019-03-11-11-46-14-305.png, image-2019-03-11-11-46-20-664.png, > image-2019-03-11-11-46-31-966.png, sqoop-error.log, sqoop-script.sh > > > I've set up Kylin to access a SQL server using JDBC and do the ETL step as > described in > [http://kylin.apache.org/docs/tutorial/setup_jdbc_datasource.html] > but when the "Sqoop To Flat Hive Table" runs it terminates with an error > shown in the attached log. > It seems that the root cause is the presence of the '`' character in the > query, which has no meaning for sql server. > > My problem seems to related to this piece of code > > public class FlatTableSqlQuoteUtils { > public static final String QUOTE = "`"; > /** > * Quote identifier by default quote ` > * @param identifier > * @return > */ > public static String quoteIdentifier(String identifier){ > return QUOTE + identifier + QUOTE; > } > > The solution may be to add a parameter to change this character used for > quoting. > > Just to be sure I made a copy of the original "sqoop" command to "sqoop.orig" > and replaced the original with a little script of mine (attached). > This little script removes the "`" character from the original command and > the call "sqoop.orig" with the cleaned arguments. > It works, but it is still an ugly workaround. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
XiaoXiang Yu created KYLIN-3970: --- Summary: NRT Streaming cannot work at HDP3.0 Key: KYLIN-3970 URL: https://issues.apache.org/jira/browse/KYLIN-3970 Project: Kylin Issue Type: Bug Components: NRT Streaming Affects Versions: v2.6.1 Reporter: XiaoXiang Yu Assignee: XiaoXiang Yu Fix For: v3.0.0, v2.6.2 We found that when using [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], NRT streaming cannot work in serval step for jar conflict problem. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[ https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3970: Labels: HDP Hadoop3 (was: ) > NRT Streaming cannot work at HDP3.0 > --- > > Key: KYLIN-3970 > URL: https://issues.apache.org/jira/browse/KYLIN-3970 > Project: Kylin > Issue Type: Bug > Components: NRT Streaming >Affects Versions: v2.6.1 >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Labels: HDP, Hadoop3 > Fix For: v3.0.0, v2.6.2 > > > We found that when using > [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], > NRT streaming cannot work in serval step for jar conflict problem. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[ https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3970: Attachment: image-2019-04-23-14-25-51-066.png > NRT Streaming cannot work at HDP3.0 > --- > > Key: KYLIN-3970 > URL: https://issues.apache.org/jira/browse/KYLIN-3970 > Project: Kylin > Issue Type: Bug > Components: NRT Streaming >Affects Versions: v2.6.1 >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Labels: HDP, Hadoop3 > Fix For: v3.0.0, v2.6.2 > > Attachments: image-2019-04-23-14-25-51-066.png > > > We found that when using > [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], > NRT streaming cannot work in serval step for jar conflict problem. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[ https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3970: Description: We found that when using [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], NRT streaming cannot work in serval step for jar conflict problem. !image-2019-04-23-14-25-51-066.png! was:We found that when using [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], NRT streaming cannot work in serval step for jar conflict problem. > NRT Streaming cannot work at HDP3.0 > --- > > Key: KYLIN-3970 > URL: https://issues.apache.org/jira/browse/KYLIN-3970 > Project: Kylin > Issue Type: Bug > Components: NRT Streaming >Affects Versions: v2.6.1 >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Labels: HDP, Hadoop3 > Fix For: v3.0.0, v2.6.2 > > Attachments: image-2019-04-23-14-25-51-066.png > > > We found that when using > [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], > NRT streaming cannot work in serval step for jar conflict problem. > > !image-2019-04-23-14-25-51-066.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[ https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16823735#comment-16823735 ] XiaoXiang Yu commented on KYLIN-3970: - h3. *First problem:* *Kafka jar conflict* org.apache.kylin.engine.mr.exception.MapReduceException: no counters for job job_1555653800318_0004Job Diagnostics:Task failed task_1555653800318_0004_m_00 Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 0 Failure task Diagnostics: Error: org.apache.kafka.clients.consumer.ConsumerConfig.configNames()Ljava/util/Set; at org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:173) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166) at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166) at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) h3. *Second problem: Hive null value* org.apache.kylin.engine.mr.exception.MapReduceException: no counters for job job_1555653800318_0007Job Diagnostics:Task failed task_1555653800318_0007_m_00 Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 0 Failure task Diagnostics: Error: java.lang.IllegalArgumentException: there is no valid date pattern for:\N at org.apache.kylin.common.util.DateFormat.stringToMillis(DateFormat.java:124) at org.apache.kylin.metadata.datatype.DataTypeOrder$DateTimeOrder.toComparable(DataTypeOrder.java:154) at org.apache.kylin.metadata.datatype.DataTypeOrder$DateTimeOrder.toComparable(DataTypeOrder.java:148) at org.apache.kylin.metadata.datatype.DataTypeOrder.compare(DataTypeOrder.java:95) at org.apache.kylin.metadata.datatype.DataTypeOrder.max(DataTypeOrder.java:90) at org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper.doMap(FactDistinctColumnsMapper.java:194) at org.apache.kylin.engine.mr.KylinMapper.map(KylinMapper.java:77) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168) at org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:173) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166) at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166) at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) > NRT Streaming cannot work at HDP3.0 > --- > > Key: KYLIN-3970 > URL: https://issues.apache.org/jira/browse/KYLIN-3970 > Project: Kylin > Issue Type: Bug > Components: NRT Streaming >Affects Versions: v2.6.1 >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Labels: HDP, Hadoop3 > Fix For: v3.0.0, v2.6.2 > > Attachments: image-2019-04-23-14-25-51-066.png > > > We found that when using > [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], > NRT streaming cannot work in serval step for jar conflict problem. > > !image-2019-04-23-14-25-51-066.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[ https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3970: Description: We found that when using [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], NRT streaming cannot work in serval step for jar conflict problem. !image-2019-04-24-15-35-06-712.png! was: We found that when using [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], NRT streaming cannot work in serval step for jar conflict problem. !image-2019-04-23-14-25-51-066.png! > NRT Streaming cannot work at HDP3.0 > --- > > Key: KYLIN-3970 > URL: https://issues.apache.org/jira/browse/KYLIN-3970 > Project: Kylin > Issue Type: Bug > Components: NRT Streaming >Affects Versions: v2.6.1 >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Labels: HDP, Hadoop3 > Fix For: v3.0.0, v2.6.2 > > Attachments: image-2019-04-23-14-25-51-066.png > > > We found that when using > [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], > NRT streaming cannot work in serval step for jar conflict problem. > > !image-2019-04-24-15-35-06-712.png! > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[ https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3970: Attachment: image-2019-04-24-15-38-10-098.png > NRT Streaming cannot work at HDP3.0 > --- > > Key: KYLIN-3970 > URL: https://issues.apache.org/jira/browse/KYLIN-3970 > Project: Kylin > Issue Type: Bug > Components: NRT Streaming >Affects Versions: v2.6.1 >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Labels: HDP, Hadoop3 > Fix For: v3.0.0, v2.6.2 > > Attachments: image-2019-04-23-14-25-51-066.png, > image-2019-04-24-15-38-10-098.png > > > We found that when using > [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], > NRT streaming cannot work in serval step for jar conflict problem. > > !image-2019-04-24-15-35-06-712.png! > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[ https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3970: Description: We found that when using [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], NRT streaming cannot work in serval step for jar conflict problem. h2. *Env* |HDFS|3.1.1| |YARN|3.1.1| |MapReduce2|3.1.1| |Hive|3.1.0| |HBase|2.0.0| |ZooKeeper|3.4.6| |HDP|3.0.1.0| !image-2019-04-24-15-38-10-098.png! was: We found that when using [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], NRT streaming cannot work in serval step for jar conflict problem. !image-2019-04-24-15-35-06-712.png! > NRT Streaming cannot work at HDP3.0 > --- > > Key: KYLIN-3970 > URL: https://issues.apache.org/jira/browse/KYLIN-3970 > Project: Kylin > Issue Type: Bug > Components: NRT Streaming >Affects Versions: v2.6.1 >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Labels: HDP, Hadoop3 > Fix For: v3.0.0, v2.6.2 > > Attachments: image-2019-04-23-14-25-51-066.png, > image-2019-04-24-15-38-10-098.png > > > We found that when using > [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], > NRT streaming cannot work in serval step for jar conflict problem. > h2. *Env* > |HDFS|3.1.1| > |YARN|3.1.1| > |MapReduce2|3.1.1| > |Hive|3.1.0| > |HBase|2.0.0| > |ZooKeeper|3.4.6| > |HDP|3.0.1.0| > > > !image-2019-04-24-15-38-10-098.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[ https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16824880#comment-16824880 ] XiaoXiang Yu commented on KYLIN-3970: - Root Cause HDP contains a low version kafka-client jar, both in Kylin process's classpath and MapReduce's classpath : "/usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar". This is my classpath of Kylin: CLASSPATH = /home/kylin-hdp3/xiaoxiang/apache-kylin-2.6.1-bin-hadoop3/bin/../tomcat/bin/bootstrap.jar:/home/kylin-hdp3/xiaoxiang/apache-kylin-2.6.1-bin-hadoop3/bin/../tomcat/bin/tomcat-juli.jar:/home/kylin-hdp3/xiaoxiang/apache-kylin-2.6.1-bin-hadoop3/bin/../tomcat/lib/*:/home/kylin-hdp3/xiaoxiang/apache-kylin-2.6.1-bin-hadoop3/conf:/home/kylin-hdp3/xiaoxiang/apache-kylin-2.6.1-bin-hadoop3/lib/*:/home/kylin-hdp3/xiaoxiang/apache-kylin-2.6.1-bin-hadoop3/ext/*::/usr/hdp/3.0.1.0-187/hbase/conf:/usr/local/bin/jdk1.8.0_191/lib/tools.jar:/usr/hdp/3.0.1.0-187/hbase:/usr/hdp/3.0.1.0-187/hbase/lib/accessors-smart-1.2.jar:/usr/hdp/3.0.1.0-187/hbase/lib/aopalliance-1.0.jar:/usr/hdp/3.0.1.0-187/hbase/lib/aopalliance-repackaged-2.5.0-b32.jar:/usr/hdp/3.0.1.0-187/hbase/lib/asm-5.0.4.jar:/usr/hdp/3.0.1.0-187/hbase/lib/atlas-plugin-classloader-1.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/audience-annotations-0.5.0.jar:/usr/hdp/3.0.1.0-187/hbase/lib/avro-1.7.7.jar:/usr/hdp/3.0.1.0-187/hbase/lib/aws-java-sdk-bundle-1.11.271.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-beanutils-1.9.3.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-cli-1.2.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-codec-1.10.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-collections-3.2.2.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-compress-1.4.1.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-configuration2-2.1.1.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-crypto-1.0.0.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-csv-1.0.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-daemon-1.0.13.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-io-2.5.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-lang-2.6.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-lang3-3.6.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-logging-1.2.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-math3-3.6.1.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-net-3.6.jar:/usr/hdp/3.0.1.0-187/hbase/lib/curator-client-4.0.0.jar:/usr/hdp/3.0.1.0-187/hbase/lib/curator-framework-4.0.0.jar:/usr/hdp/3.0.1.0-187/hbase/lib/curator-recipes-4.0.0.jar:/usr/hdp/3.0.1.0-187/hbase/lib/disruptor-3.3.6.jar:/usr/hdp/3.0.1.0-187/hbase/lib/dnsjava-2.1.7.jar:/usr/hdp/3.0.1.0-187/hbase/lib/ehcache-3.3.1.jar:/usr/hdp/3.0.1.0-187/hbase/lib/findbugs-annotations-1.3.9-1.jar:/usr/hdp/3.0.1.0-187/hbase/lib/fst-2.50.jar:/usr/hdp/3.0.1.0-187/hbase/lib/geronimo-jcache_1.0_spec-1.0-alpha-1.jar:/usr/hdp/3.0.1.0-187/hbase/lib/gson-2.2.4.jar:/usr/hdp/3.0.1.0-187/hbase/lib/guava-11.0.2.jar:/usr/hdp/3.0.1.0-187/hbase/lib/guice-4.0.jar:/usr/hdp/3.0.1.0-187/hbase/lib/guice-servlet-4.0.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hamcrest-core-1.3.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-annotations-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-annotations-2.0.0.3.0.1.0-187-tests.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-annotations.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-backup-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-backup.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-bridge-shim-1.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-client-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-client.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-common-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-common-2.0.0.3.0.1.0-187-tests.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-common.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-endpoint-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-endpoint.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-examples-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-examples.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-external-blockcache-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-external-blockcache.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-hadoop2-compat-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-hadoop2-compat-2.0.0.3.0.1.0-187-tests.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-hadoop2-compat.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-hadoop-compat-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-hadoop-compat-2.0.0.3.0.1.0-187-tests.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-hadoop-compat.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-http-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-http.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-it-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-it-2.0.0.3.0.1.0-187-tests.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-it.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-mapreduce-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-mapreduce-2.0.0.3.0.1.0-187-tests.jar:/usr
[jira] [Comment Edited] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[ https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16823735#comment-16823735 ] XiaoXiang Yu edited comment on KYLIN-3970 at 4/24/19 7:48 AM: -- *Problem:* *Kafka jar conflict* Caused by: org.apache.kafka.common.config.ConfigException: Missing required configuration "partition.assignment.strategy" which has no default value. at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:124) at org.apache.kafka.common.config.AbstractConfig.(AbstractConfig.java:48) at org.apache.kafka.clients.consumer.ConsumerConfig.(ConsumerConfig.java:194) at org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:430) at org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:413) at org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:400) at org.apache.kylin.source.kafka.util.KafkaClient.getKafkaConsumer(KafkaClient.java:48) at org.apache.kylin.source.kafka.util.KafkaClient.getEarliestOffsets(KafkaClient.java:128) at org.apache.kylin.source.kafka.KafkaSource.enrichSourcePartitionBeforeBuild(KafkaSource.java:100) at org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:232) at org.apache.kylin.rest.service.JobService.submitJob(JobService.java:202) at org.apache.kylin.rest.controller.CubeController.buildInternal(CubeController.java:394) ... 80 more org.apache.kylin.engine.mr.exception.MapReduceException: no counters for job job_1555653800318_0004Job Diagnostics:Task failed task_1555653800318_0004_m_00 Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 0 Failure task Diagnostics: Error: org.apache.kafka.clients.consumer.ConsumerConfig.configNames()Ljava/util/Set; at org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:173) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166) at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166) at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) h3. was (Author: hit_lacus): h3. *First problem:* *Kafka jar conflict* org.apache.kylin.engine.mr.exception.MapReduceException: no counters for job job_1555653800318_0004Job Diagnostics:Task failed task_1555653800318_0004_m_00 Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 0 Failure task Diagnostics: Error: org.apache.kafka.clients.consumer.ConsumerConfig.configNames()Ljava/util/Set; at org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:173) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166) at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166) at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) h3. *Second problem: Hive null value* org.apache.kylin.engine.mr.exception.MapReduceException: no counters for job job_1555653800318_0007Job Diagnostics:Task failed task_1555653800318_0007_m_00 Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 0 Failure task Diagnostics: Error: java.lang.IllegalArgumentException: there is no valid date pattern for:\N at org.apache.kylin.common.util.DateFormat.stringToMillis(DateFormat.java:124) at org.apache.kylin.metadata.datatype.DataTypeOrder$DateTimeOrder.toComparable(DataTypeOrder.java:154) at org.apache.kylin.metadata.datatype.DataTypeOrder$DateTimeOrder.toComparable(DataTypeOrder.java:148) at org.apache.kylin.metadata.datatype.DataTypeOrder.compare(DataTypeOrder.java:95) at org.apache.kylin.metadata.datatype.DataTypeOrder.max(DataTypeOrder.java:90) at org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper.doMap(FactDistinctColumnsMapper.java:194) at org.apache.kylin.engine.mr.KylinMapper.map(KylinMapper.java:77) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799) at org.apache.hadoop.m
[jira] [Updated] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[ https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3970: Fix Version/s: (was: v2.6.2) (was: v3.0.0) > NRT Streaming cannot work at HDP3.0 > --- > > Key: KYLIN-3970 > URL: https://issues.apache.org/jira/browse/KYLIN-3970 > Project: Kylin > Issue Type: Bug > Components: NRT Streaming >Affects Versions: v2.6.1 >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Labels: HDP, Hadoop3 > Attachments: image-2019-04-23-14-25-51-066.png, > image-2019-04-24-15-38-10-098.png > > > We found that when using > [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], > NRT streaming cannot work in serval step for jar conflict problem. > h2. *Env* > |HDFS|3.1.1| > |YARN|3.1.1| > |MapReduce2|3.1.1| > |Hive|3.1.0| > |HBase|2.0.0| > |ZooKeeper|3.4.6| > |HDP|3.0.1.0| > > > !image-2019-04-24-15-38-10-098.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[ https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16824880#comment-16824880 ] XiaoXiang Yu edited comment on KYLIN-3970 at 4/24/19 7:50 AM: -- h2. *Root Cause* HDP contains a low version kafka-client jar, both in Kylin process's classpath and MapReduce's classpath : "/usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar". This is my classpath of Kylin: {code}CLASSPATH = /home/kylin-hdp3/xiaoxiang/apache-kylin-2.6.1-bin-hadoop3/bin/../tomcat/bin/bootstrap.jar:/home/kylin-hdp3/xiaoxiang/apache-kylin-2.6.1-bin-hadoop3/bin/../tomcat/bin/tomcat-juli.jar:/home/kylin-hdp3/xiaoxiang/apache-kylin-2.6.1-bin-hadoop3/bin/../tomcat/lib/*:/home/kylin-hdp3/xiaoxiang/apache-kylin-2.6.1-bin-hadoop3/conf:/home/kylin-hdp3/xiaoxiang/apache-kylin-2.6.1-bin-hadoop3/lib/*:/home/kylin-hdp3/xiaoxiang/apache-kylin-2.6.1-bin-hadoop3/ext/*::/usr/hdp/3.0.1.0-187/hbase/conf:/usr/local/bin/jdk1.8.0_191/lib/tools.jar:/usr/hdp/3.0.1.0-187/hbase:/usr/hdp/3.0.1.0-187/hbase/lib/accessors-smart-1.2.jar:/usr/hdp/3.0.1.0-187/hbase/lib/aopalliance-1.0.jar:/usr/hdp/3.0.1.0-187/hbase/lib/aopalliance-repackaged-2.5.0-b32.jar:/usr/hdp/3.0.1.0-187/hbase/lib/asm-5.0.4.jar:/usr/hdp/3.0.1.0-187/hbase/lib/atlas-plugin-classloader-1.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/audience-annotations-0.5.0.jar:/usr/hdp/3.0.1.0-187/hbase/lib/avro-1.7.7.jar:/usr/hdp/3.0.1.0-187/hbase/lib/aws-java-sdk-bundle-1.11.271.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-beanutils-1.9.3.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-cli-1.2.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-codec-1.10.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-collections-3.2.2.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-compress-1.4.1.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-configuration2-2.1.1.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-crypto-1.0.0.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-csv-1.0.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-daemon-1.0.13.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-io-2.5.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-lang-2.6.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-lang3-3.6.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-logging-1.2.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-math3-3.6.1.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-net-3.6.jar:/usr/hdp/3.0.1.0-187/hbase/lib/curator-client-4.0.0.jar:/usr/hdp/3.0.1.0-187/hbase/lib/curator-framework-4.0.0.jar:/usr/hdp/3.0.1.0-187/hbase/lib/curator-recipes-4.0.0.jar:/usr/hdp/3.0.1.0-187/hbase/lib/disruptor-3.3.6.jar:/usr/hdp/3.0.1.0-187/hbase/lib/dnsjava-2.1.7.jar:/usr/hdp/3.0.1.0-187/hbase/lib/ehcache-3.3.1.jar:/usr/hdp/3.0.1.0-187/hbase/lib/findbugs-annotations-1.3.9-1.jar:/usr/hdp/3.0.1.0-187/hbase/lib/fst-2.50.jar:/usr/hdp/3.0.1.0-187/hbase/lib/geronimo-jcache_1.0_spec-1.0-alpha-1.jar:/usr/hdp/3.0.1.0-187/hbase/lib/gson-2.2.4.jar:/usr/hdp/3.0.1.0-187/hbase/lib/guava-11.0.2.jar:/usr/hdp/3.0.1.0-187/hbase/lib/guice-4.0.jar:/usr/hdp/3.0.1.0-187/hbase/lib/guice-servlet-4.0.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hamcrest-core-1.3.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-annotations-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-annotations-2.0.0.3.0.1.0-187-tests.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-annotations.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-backup-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-backup.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-bridge-shim-1.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-client-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-client.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-common-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-common-2.0.0.3.0.1.0-187-tests.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-common.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-endpoint-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-endpoint.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-examples-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-examples.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-external-blockcache-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-external-blockcache.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-hadoop2-compat-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-hadoop2-compat-2.0.0.3.0.1.0-187-tests.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-hadoop2-compat.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-hadoop-compat-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-hadoop-compat-2.0.0.3.0.1.0-187-tests.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-hadoop-compat.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-http-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-http.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-it-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-it-2.0.0.3.0.1.0-187-tests.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-it.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-mapreduce-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-18
[jira] [Comment Edited] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[ https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16823735#comment-16823735 ] XiaoXiang Yu edited comment on KYLIN-3970 at 4/24/19 7:54 AM: -- *Problem:* *Kafka jar conflict* {code} Caused by: org.apache.kafka.common.config.ConfigException: Missing required configuration "partition.assignment.strategy" which has no default value. at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:124) at org.apache.kafka.common.config.AbstractConfig.(AbstractConfig.java:48) at org.apache.kafka.clients.consumer.ConsumerConfig.(ConsumerConfig.java:194) at org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:430) at org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:413) at org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:400) at org.apache.kylin.source.kafka.util.KafkaClient.getKafkaConsumer(KafkaClient.java:48) at org.apache.kylin.source.kafka.util.KafkaClient.getEarliestOffsets(KafkaClient.java:128) at org.apache.kylin.source.kafka.KafkaSource.enrichSourcePartitionBeforeBuild(KafkaSource.java:100) at org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:232) at org.apache.kylin.rest.service.JobService.submitJob(JobService.java:202) at org.apache.kylin.rest.controller.CubeController.buildInternal(CubeController.java:394) ... 80 more {code} {code} org.apache.kylin.engine.mr.exception.MapReduceException: no counters for job job_1555653800318_0004Job Diagnostics:Task failed task_1555653800318_0004_m_00 Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 0 Failure task Diagnostics: Error: org.apache.kafka.clients.consumer.ConsumerConfig.configNames()Ljava/util/Set; at org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:173) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166) at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166) at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) {code} h3. was (Author: hit_lacus): *Problem:* *Kafka jar conflict* Caused by: org.apache.kafka.common.config.ConfigException: Missing required configuration "partition.assignment.strategy" which has no default value. at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:124) at org.apache.kafka.common.config.AbstractConfig.(AbstractConfig.java:48) at org.apache.kafka.clients.consumer.ConsumerConfig.(ConsumerConfig.java:194) at org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:430) at org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:413) at org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:400) at org.apache.kylin.source.kafka.util.KafkaClient.getKafkaConsumer(KafkaClient.java:48) at org.apache.kylin.source.kafka.util.KafkaClient.getEarliestOffsets(KafkaClient.java:128) at org.apache.kylin.source.kafka.KafkaSource.enrichSourcePartitionBeforeBuild(KafkaSource.java:100) at org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:232) at org.apache.kylin.rest.service.JobService.submitJob(JobService.java:202) at org.apache.kylin.rest.controller.CubeController.buildInternal(CubeController.java:394) ... 80 more org.apache.kylin.engine.mr.exception.MapReduceException: no counters for job job_1555653800318_0004Job Diagnostics:Task failed task_1555653800318_0004_m_00 Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 0 Failure task Diagnostics: Error: org.apache.kafka.clients.consumer.ConsumerConfig.configNames()Ljava/util/Set; at org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:173) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166) at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166) at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) h3. > NRT Streaming cannot work at HDP3.0 > --- > > Key: KYLIN-
[jira] [Comment Edited] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[ https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16823735#comment-16823735 ] XiaoXiang Yu edited comment on KYLIN-3970 at 4/24/19 7:55 AM: -- h2. *Problem:* *Kafka jar conflict* * when click build buttom {code:java} Caused by: org.apache.kafka.common.config.ConfigException: Missing required configuration "partition.assignment.strategy" which has no default value. at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:124) at org.apache.kafka.common.config.AbstractConfig.(AbstractConfig.java:48) at org.apache.kafka.clients.consumer.ConsumerConfig.(ConsumerConfig.java:194) at org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:430) at org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:413) at org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:400) at org.apache.kylin.source.kafka.util.KafkaClient.getKafkaConsumer(KafkaClient.java:48) at org.apache.kylin.source.kafka.util.KafkaClient.getEarliestOffsets(KafkaClient.java:128) at org.apache.kylin.source.kafka.KafkaSource.enrichSourcePartitionBeforeBuild(KafkaSource.java:100) at org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:232) at org.apache.kylin.rest.service.JobService.submitJob(JobService.java:202) at org.apache.kylin.rest.controller.CubeController.buildInternal(CubeController.java:394) ... 80 more {code} * First step of build segment {code:java} org.apache.kylin.engine.mr.exception.MapReduceException: no counters for job job_1555653800318_0004Job Diagnostics:Task failed task_1555653800318_0004_m_00 Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 0 Failure task Diagnostics: Error: org.apache.kafka.clients.consumer.ConsumerConfig.configNames()Ljava/util/Set; at org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:173) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166) at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166) at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) {code} h3. was (Author: hit_lacus): *Problem:* *Kafka jar conflict* {code} Caused by: org.apache.kafka.common.config.ConfigException: Missing required configuration "partition.assignment.strategy" which has no default value. at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:124) at org.apache.kafka.common.config.AbstractConfig.(AbstractConfig.java:48) at org.apache.kafka.clients.consumer.ConsumerConfig.(ConsumerConfig.java:194) at org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:430) at org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:413) at org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:400) at org.apache.kylin.source.kafka.util.KafkaClient.getKafkaConsumer(KafkaClient.java:48) at org.apache.kylin.source.kafka.util.KafkaClient.getEarliestOffsets(KafkaClient.java:128) at org.apache.kylin.source.kafka.KafkaSource.enrichSourcePartitionBeforeBuild(KafkaSource.java:100) at org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:232) at org.apache.kylin.rest.service.JobService.submitJob(JobService.java:202) at org.apache.kylin.rest.controller.CubeController.buildInternal(CubeController.java:394) ... 80 more {code} {code} org.apache.kylin.engine.mr.exception.MapReduceException: no counters for job job_1555653800318_0004Job Diagnostics:Task failed task_1555653800318_0004_m_00 Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 0 Failure task Diagnostics: Error: org.apache.kafka.clients.consumer.ConsumerConfig.configNames()Ljava/util/Set; at org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:173) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166) at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166) at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) {code} h3. > NR
[jira] [Updated] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[ https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3970: Attachment: (was: image-2019-04-23-14-25-51-066.png) > NRT Streaming cannot work at HDP3.0 > --- > > Key: KYLIN-3970 > URL: https://issues.apache.org/jira/browse/KYLIN-3970 > Project: Kylin > Issue Type: Bug > Components: NRT Streaming >Affects Versions: v2.6.1 >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Labels: HDP, Hadoop3 > Attachments: image-2019-04-24-15-38-10-098.png > > > We found that when using > [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], > NRT streaming cannot work in serval step for jar conflict problem. > h2. *Env* > |HDFS|3.1.1| > |YARN|3.1.1| > |MapReduce2|3.1.1| > |Hive|3.1.0| > |HBase|2.0.0| > |ZooKeeper|3.4.6| > |HDP|3.0.1.0| > > > !image-2019-04-24-15-38-10-098.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[ https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16824897#comment-16824897 ] XiaoXiang Yu commented on KYLIN-3970: - h2. *Solution* # remove old version kafka-client.jar # add *mapreduce.job.user.classpath.first=true* in _kylin_job_conf.xml_ > NRT Streaming cannot work at HDP3.0 > --- > > Key: KYLIN-3970 > URL: https://issues.apache.org/jira/browse/KYLIN-3970 > Project: Kylin > Issue Type: Bug > Components: NRT Streaming >Affects Versions: v2.6.1 >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Labels: HDP, Hadoop3 > Attachments: image-2019-04-24-15-38-10-098.png > > > We found that when using > [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], > NRT streaming cannot work in serval step for jar conflict problem. > h2. *Env* > |HDFS|3.1.1| > |YARN|3.1.1| > |MapReduce2|3.1.1| > |Hive|3.1.0| > |HBase|2.0.0| > |ZooKeeper|3.4.6| > |HDP|3.0.1.0| > > > !image-2019-04-24-15-38-10-098.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[ https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16824897#comment-16824897 ] XiaoXiang Yu edited comment on KYLIN-3970 at 4/24/19 8:21 AM: -- h2. *Try* # remove old version kafka-client.jar # add *mapreduce.job.user.classpath.first=true* in _kylin_job_conf.xml_ We will got a NoSuchMethodError caused by Guava {code} 2019-04-24 08:06:04,378 INFO [main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output: DEFAULT.SAMPLE_1.COUNTRY 'US' => reducer 0 2019-04-24 08:06:04,378 INFO [main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output: DEFAULT.SAMPLE_1.CATEGORY 'BOOK' => reducer 1 2019-04-24 08:06:04,379 INFO [main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output: DEFAULT.SAMPLE_1.DEVICE 'Andriod' => reducer 2 2019-04-24 08:06:04,379 INFO [main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output: DEFAULT.SAMPLE_1.USER_GENDER 'Male' => reducer 3 2019-04-24 08:06:04,379 INFO [main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output: DEFAULT.SAMPLE_1.COUNTRY 'KOREA' => reducer 0 2019-04-24 08:06:04,379 INFO [main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output: DEFAULT.SAMPLE_1.CATEGORY 'CLOTH' => reducer 1 2019-04-24 08:06:04,379 INFO [main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output: DEFAULT.SAMPLE_1.DEVICE 'iOS' => reducer 2 2019-04-24 08:06:04,379 INFO [main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output: DEFAULT.SAMPLE_1.USER_GENDER 'Female' => reducer 3 2019-04-24 08:06:04,381 FATAL [Thread-13] org.apache.hadoop.yarn.YarnUncaughtExceptionHandler: Thread Thread[Thread-13,5,main] threw an Error. Shutting down now... java.lang.NoSuchMethodError: com.google.common.hash.Hasher.putString(Ljava/lang/CharSequence;)Lcom/google/common/hash/Hasher; at org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper$CuboidStatCalculator.putRowKeyToHLLNew(FactDistinctColumnsMapper.java:377) at org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper$CuboidStatCalculator.run(FactDistinctColumnsMapper.java:411) at java.lang.Thread.run(Thread.java:748) 2019-04-24 08:06:04,385 INFO [Thread-13] org.apache.hadoop.util.ExitUtil: Exiting with status -1: ExitException {code} was (Author: hit_lacus): h2. *Solution* # remove old version kafka-client.jar # add *mapreduce.job.user.classpath.first=true* in _kylin_job_conf.xml_ > NRT Streaming cannot work at HDP3.0 > --- > > Key: KYLIN-3970 > URL: https://issues.apache.org/jira/browse/KYLIN-3970 > Project: Kylin > Issue Type: Bug > Components: NRT Streaming >Affects Versions: v2.6.1 >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Labels: HDP, Hadoop3 > Attachments: image-2019-04-24-15-38-10-098.png > > > We found that when using > [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], > NRT streaming cannot work in serval step for jar conflict problem. > h2. *Env* > |HDFS|3.1.1| > |YARN|3.1.1| > |MapReduce2|3.1.1| > |Hive|3.1.0| > |HBase|2.0.0| > |ZooKeeper|3.4.6| > |HDP|3.0.1.0| > > > !image-2019-04-24-15-38-10-098.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[ https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16824897#comment-16824897 ] XiaoXiang Yu edited comment on KYLIN-3970 at 4/24/19 8:26 AM: -- h2. *Try* remove old version kafka-client.jar {code} mv /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar.old {code} add *mapreduce.job.user.classpath.first=true* in _kylin_job_conf.xml_ We will got a NoSuchMethodError caused by Guava {code:java} 2019-04-24 08:06:04,378 INFO [main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output: DEFAULT.SAMPLE_1.COUNTRY 'US' => reducer 0 2019-04-24 08:06:04,378 INFO [main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output: DEFAULT.SAMPLE_1.CATEGORY 'BOOK' => reducer 1 2019-04-24 08:06:04,379 INFO [main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output: DEFAULT.SAMPLE_1.DEVICE 'Andriod' => reducer 2 2019-04-24 08:06:04,379 INFO [main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output: DEFAULT.SAMPLE_1.USER_GENDER 'Male' => reducer 3 2019-04-24 08:06:04,379 INFO [main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output: DEFAULT.SAMPLE_1.COUNTRY 'KOREA' => reducer 0 2019-04-24 08:06:04,379 INFO [main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output: DEFAULT.SAMPLE_1.CATEGORY 'CLOTH' => reducer 1 2019-04-24 08:06:04,379 INFO [main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output: DEFAULT.SAMPLE_1.DEVICE 'iOS' => reducer 2 2019-04-24 08:06:04,379 INFO [main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output: DEFAULT.SAMPLE_1.USER_GENDER 'Female' => reducer 3 2019-04-24 08:06:04,381 FATAL [Thread-13] org.apache.hadoop.yarn.YarnUncaughtExceptionHandler: Thread Thread[Thread-13,5,main] threw an Error. Shutting down now... java.lang.NoSuchMethodError: com.google.common.hash.Hasher.putString(Ljava/lang/CharSequence;)Lcom/google/common/hash/Hasher; at org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper$CuboidStatCalculator.putRowKeyToHLLNew(FactDistinctColumnsMapper.java:377) at org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper$CuboidStatCalculator.run(FactDistinctColumnsMapper.java:411) at java.lang.Thread.run(Thread.java:748) 2019-04-24 08:06:04,385 INFO [Thread-13] org.apache.hadoop.util.ExitUtil: Exiting with status -1: ExitException {code} was (Author: hit_lacus): h2. *Try* # remove old version kafka-client.jar # add *mapreduce.job.user.classpath.first=true* in _kylin_job_conf.xml_ We will got a NoSuchMethodError caused by Guava {code} 2019-04-24 08:06:04,378 INFO [main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output: DEFAULT.SAMPLE_1.COUNTRY 'US' => reducer 0 2019-04-24 08:06:04,378 INFO [main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output: DEFAULT.SAMPLE_1.CATEGORY 'BOOK' => reducer 1 2019-04-24 08:06:04,379 INFO [main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output: DEFAULT.SAMPLE_1.DEVICE 'Andriod' => reducer 2 2019-04-24 08:06:04,379 INFO [main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output: DEFAULT.SAMPLE_1.USER_GENDER 'Male' => reducer 3 2019-04-24 08:06:04,379 INFO [main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output: DEFAULT.SAMPLE_1.COUNTRY 'KOREA' => reducer 0 2019-04-24 08:06:04,379 INFO [main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output: DEFAULT.SAMPLE_1.CATEGORY 'CLOTH' => reducer 1 2019-04-24 08:06:04,379 INFO [main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output: DEFAULT.SAMPLE_1.DEVICE 'iOS' => reducer 2 2019-04-24 08:06:04,379 INFO [main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output: DEFAULT.SAMPLE_1.USER_GENDER 'Female' => reducer 3 2019-04-24 08:06:04,381 FATAL [Thread-13] org.apache.hadoop.yarn.YarnUncaughtExceptionHandler: Thread Thread[Thread-13,5,main] threw an Error. Shutting down now... java.lang.NoSuchMethodError: com.google.common.hash.Hasher.putString(Ljava/lang/CharSequence;)Lcom/google/common/hash/Hasher; at org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper$CuboidStatCalculator.putRowKeyToHLLNew(FactDistinctColumnsMapper.java:377) at org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper$CuboidStatCalculator.run(FactDistinctColumnsMapper.java:411) at java.lang.Thread.run(Thread.java:748) 2019-04-24 08:06:04,385 INFO [Thread-13] org.apache.hadoop.util.ExitUtil: Exiting with status -1: ExitException {code} > NRT Streaming cannot work at HDP3.0 > --- > > Key: KYLIN-3970 > URL: https:
[jira] [Comment Edited] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[ https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16824897#comment-16824897 ] XiaoXiang Yu edited comment on KYLIN-3970 at 4/24/19 8:43 AM: -- h2. *Try-1* remove old version kafka-client.jar {code:java} mv /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar.old {code} We want to let mapper use kylin's classpath first, so we add *mapreduce.job.user.classpath.first=true* in _kylin_job_conf.xml_ But this cause more conflict, we will got a NoSuchMethodError caused by Guava. {code:java} 2019-04-24 08:06:04,378 INFO [main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output: DEFAULT.SAMPLE_1.COUNTRY 'US' => reducer 0 2019-04-24 08:06:04,378 INFO [main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output: DEFAULT.SAMPLE_1.CATEGORY 'BOOK' => reducer 1 2019-04-24 08:06:04,379 INFO [main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output: DEFAULT.SAMPLE_1.DEVICE 'Andriod' => reducer 2 2019-04-24 08:06:04,379 INFO [main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output: DEFAULT.SAMPLE_1.USER_GENDER 'Male' => reducer 3 2019-04-24 08:06:04,379 INFO [main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output: DEFAULT.SAMPLE_1.COUNTRY 'KOREA' => reducer 0 2019-04-24 08:06:04,379 INFO [main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output: DEFAULT.SAMPLE_1.CATEGORY 'CLOTH' => reducer 1 2019-04-24 08:06:04,379 INFO [main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output: DEFAULT.SAMPLE_1.DEVICE 'iOS' => reducer 2 2019-04-24 08:06:04,379 INFO [main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output: DEFAULT.SAMPLE_1.USER_GENDER 'Female' => reducer 3 2019-04-24 08:06:04,381 FATAL [Thread-13] org.apache.hadoop.yarn.YarnUncaughtExceptionHandler: Thread Thread[Thread-13,5,main] threw an Error. Shutting down now... java.lang.NoSuchMethodError: com.google.common.hash.Hasher.putString(Ljava/lang/CharSequence;)Lcom/google/common/hash/Hasher; at org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper$CuboidStatCalculator.putRowKeyToHLLNew(FactDistinctColumnsMapper.java:377) at org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper$CuboidStatCalculator.run(FactDistinctColumnsMapper.java:411) at java.lang.Thread.run(Thread.java:748) 2019-04-24 08:06:04,385 INFO [Thread-13] org.apache.hadoop.util.ExitUtil: Exiting with status -1: ExitException {code} was (Author: hit_lacus): h2. *Try* remove old version kafka-client.jar {code} mv /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar.old {code} add *mapreduce.job.user.classpath.first=true* in _kylin_job_conf.xml_ We will got a NoSuchMethodError caused by Guava {code:java} 2019-04-24 08:06:04,378 INFO [main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output: DEFAULT.SAMPLE_1.COUNTRY 'US' => reducer 0 2019-04-24 08:06:04,378 INFO [main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output: DEFAULT.SAMPLE_1.CATEGORY 'BOOK' => reducer 1 2019-04-24 08:06:04,379 INFO [main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output: DEFAULT.SAMPLE_1.DEVICE 'Andriod' => reducer 2 2019-04-24 08:06:04,379 INFO [main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output: DEFAULT.SAMPLE_1.USER_GENDER 'Male' => reducer 3 2019-04-24 08:06:04,379 INFO [main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output: DEFAULT.SAMPLE_1.COUNTRY 'KOREA' => reducer 0 2019-04-24 08:06:04,379 INFO [main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output: DEFAULT.SAMPLE_1.CATEGORY 'CLOTH' => reducer 1 2019-04-24 08:06:04,379 INFO [main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output: DEFAULT.SAMPLE_1.DEVICE 'iOS' => reducer 2 2019-04-24 08:06:04,379 INFO [main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output: DEFAULT.SAMPLE_1.USER_GENDER 'Female' => reducer 3 2019-04-24 08:06:04,381 FATAL [Thread-13] org.apache.hadoop.yarn.YarnUncaughtExceptionHandler: Thread Thread[Thread-13,5,main] threw an Error. Shutting down now... java.lang.NoSuchMethodError: com.google.common.hash.Hasher.putString(Ljava/lang/CharSequence;)Lcom/google/common/hash/Hasher; at org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper$CuboidStatCalculator.putRowKeyToHLLNew(FactDistinctColumnsMapper.java:377) at org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper$CuboidStatCalculator.run(FactDistinctColumnsMapper.java:411) at java.lang.Thread.run(Thread.java:748) 2019-04-2
[jira] [Commented] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[ https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16824923#comment-16824923 ] XiaoXiang Yu commented on KYLIN-3970: - h2. *Try2* # remove old version kafka lib mv /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar.old # remove mapdreduce lib cache rm -rf /data/hadoop/yarn/local/usercache/kylin-hdp3/appcache # remove hadoop fs -copyToLocal /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz . hadoop fs -mv /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz.ori tar zxf mapreduce.tar.gz rm hadoop/share/hadoop/tools/lib/kafka-clients-0.8.2.1.jar tar zcf mapreduce.tar.gz hadoop/ hadoop fs -copyFromLocal mapreduce.tar.gz /hdp/apps/3.0.1.0-187/mapreduce > NRT Streaming cannot work at HDP3.0 > --- > > Key: KYLIN-3970 > URL: https://issues.apache.org/jira/browse/KYLIN-3970 > Project: Kylin > Issue Type: Bug > Components: NRT Streaming >Affects Versions: v2.6.1 >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Labels: HDP, Hadoop3 > Attachments: image-2019-04-24-15-38-10-098.png > > > We found that when using > [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], > NRT streaming cannot work in serval step for jar conflict problem. > h2. *Env* > |HDFS|3.1.1| > |YARN|3.1.1| > |MapReduce2|3.1.1| > |Hive|3.1.0| > |HBase|2.0.0| > |ZooKeeper|3.4.6| > |HDP|3.0.1.0| > > > !image-2019-04-24-15-38-10-098.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[ https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16824923#comment-16824923 ] XiaoXiang Yu edited comment on KYLIN-3970 at 4/24/19 8:53 AM: -- h2. *Try2* {code} ## remove old version kafka lib mv /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar.old ## remove mapdreduce lib cache rm -rf /data/hadoop/yarn/local/usercache/kylin-hdp3/appcache ## remove kafka-clients-0.8.2.1.jar in HDFS hadoop fs -copyToLocal /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz . hadoop fs -mv /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz.ori tar zxf mapreduce.tar.gz rm hadoop/share/hadoop/tools/lib/kafka-clients-0.8.2.1.jar tar zcf mapreduce.tar.gz hadoop/ hadoop fs -copyFromLocal mapreduce.tar.gz /hdp/apps/3.0.1.0-187/mapreduce {code} was (Author: hit_lacus): h2. *Try2* # remove old version kafka lib mv /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar.old # remove mapdreduce lib cache rm -rf /data/hadoop/yarn/local/usercache/kylin-hdp3/appcache # remove hadoop fs -copyToLocal /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz . hadoop fs -mv /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz.ori tar zxf mapreduce.tar.gz rm hadoop/share/hadoop/tools/lib/kafka-clients-0.8.2.1.jar tar zcf mapreduce.tar.gz hadoop/ hadoop fs -copyFromLocal mapreduce.tar.gz /hdp/apps/3.0.1.0-187/mapreduce > NRT Streaming cannot work at HDP3.0 > --- > > Key: KYLIN-3970 > URL: https://issues.apache.org/jira/browse/KYLIN-3970 > Project: Kylin > Issue Type: Bug > Components: NRT Streaming >Affects Versions: v2.6.1 >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Labels: HDP, Hadoop3 > Attachments: image-2019-04-24-15-38-10-098.png > > > We found that when using > [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], > NRT streaming cannot work in serval step for jar conflict problem. > h2. *Env* > |HDFS|3.1.1| > |YARN|3.1.1| > |MapReduce2|3.1.1| > |Hive|3.1.0| > |HBase|2.0.0| > |ZooKeeper|3.4.6| > |HDP|3.0.1.0| > > > !image-2019-04-24-15-38-10-098.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[ https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16824923#comment-16824923 ] XiaoXiang Yu edited comment on KYLIN-3970 at 4/24/19 8:56 AM: -- h2. *Try2* {code:java} ## remove old version kafka lib mv /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar.old ## remove mapdreduce lib cache rm -rf /data/hadoop/yarn/local/usercache/kylin-hdp3/appcache ## remove kafka-clients-0.8.2.1.jar in HDFS hadoop fs -copyToLocal /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz . hadoop fs -mv /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz.ori tar zxf mapreduce.tar.gz rm hadoop/share/hadoop/tools/lib/kafka-clients-0.8.2.1.jar tar zcf mapreduce.tar.gz hadoop/ hadoop fs -copyFromLocal mapreduce.tar.gz /hdp/apps/3.0.1.0-187/mapreduce {code} In my env, this can work. was (Author: hit_lacus): h2. *Try2* {code} ## remove old version kafka lib mv /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar.old ## remove mapdreduce lib cache rm -rf /data/hadoop/yarn/local/usercache/kylin-hdp3/appcache ## remove kafka-clients-0.8.2.1.jar in HDFS hadoop fs -copyToLocal /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz . hadoop fs -mv /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz.ori tar zxf mapreduce.tar.gz rm hadoop/share/hadoop/tools/lib/kafka-clients-0.8.2.1.jar tar zcf mapreduce.tar.gz hadoop/ hadoop fs -copyFromLocal mapreduce.tar.gz /hdp/apps/3.0.1.0-187/mapreduce {code} > NRT Streaming cannot work at HDP3.0 > --- > > Key: KYLIN-3970 > URL: https://issues.apache.org/jira/browse/KYLIN-3970 > Project: Kylin > Issue Type: Bug > Components: NRT Streaming >Affects Versions: v2.6.1 >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Labels: HDP, Hadoop3 > Attachments: image-2019-04-24-15-38-10-098.png > > > We found that when using > [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], > NRT streaming cannot work in serval step for jar conflict problem. > h2. *Env* > |HDFS|3.1.1| > |YARN|3.1.1| > |MapReduce2|3.1.1| > |Hive|3.1.0| > |HBase|2.0.0| > |ZooKeeper|3.4.6| > |HDP|3.0.1.0| > > > !image-2019-04-24-15-38-10-098.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[ https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16824923#comment-16824923 ] XiaoXiang Yu edited comment on KYLIN-3970 at 4/24/19 8:57 AM: -- h2. *Try2* {code:java} ## remove old version kafka lib mv /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar.old ## remove mapdreduce lib cache rm -rf /data/hadoop/yarn/local/usercache/kylin-hdp3/appcache ## remove kafka-clients-0.8.2.1.jar in mapreduce.tar.gz(HDFS) hadoop fs -copyToLocal /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz . hadoop fs -mv /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz.ori tar zxf mapreduce.tar.gz rm hadoop/share/hadoop/tools/lib/kafka-clients-0.8.2.1.jar tar zcf mapreduce.tar.gz hadoop/ hadoop fs -copyFromLocal mapreduce.tar.gz /hdp/apps/3.0.1.0-187/mapreduce {code} In my env, this can work. was (Author: hit_lacus): h2. *Try2* {code:java} ## remove old version kafka lib mv /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar.old ## remove mapdreduce lib cache rm -rf /data/hadoop/yarn/local/usercache/kylin-hdp3/appcache ## remove kafka-clients-0.8.2.1.jar in HDFS hadoop fs -copyToLocal /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz . hadoop fs -mv /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz.ori tar zxf mapreduce.tar.gz rm hadoop/share/hadoop/tools/lib/kafka-clients-0.8.2.1.jar tar zcf mapreduce.tar.gz hadoop/ hadoop fs -copyFromLocal mapreduce.tar.gz /hdp/apps/3.0.1.0-187/mapreduce {code} In my env, this can work. > NRT Streaming cannot work at HDP3.0 > --- > > Key: KYLIN-3970 > URL: https://issues.apache.org/jira/browse/KYLIN-3970 > Project: Kylin > Issue Type: Bug > Components: NRT Streaming >Affects Versions: v2.6.1 >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Labels: HDP, Hadoop3 > Attachments: image-2019-04-24-15-38-10-098.png > > > We found that when using > [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], > NRT streaming cannot work in serval step for jar conflict problem. > h2. *Env* > |HDFS|3.1.1| > |YARN|3.1.1| > |MapReduce2|3.1.1| > |Hive|3.1.0| > |HBase|2.0.0| > |ZooKeeper|3.4.6| > |HDP|3.0.1.0| > > > !image-2019-04-24-15-38-10-098.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[ https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16824923#comment-16824923 ] XiaoXiang Yu edited comment on KYLIN-3970 at 4/24/19 9:00 AM: -- h2. *Try2* {code:java} ## remove old version kafka lib mv /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar.old ## remove mapdreduce lib cache rm -rf /data/hadoop/yarn/local/usercache/kylin-hdp3/appcache ## remove kafka-clients-0.8.2.1.jar in mapreduce.tar.gz(HDFS) hadoop fs -copyToLocal /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz . hadoop fs -mv /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz.ori tar zxf mapreduce.tar.gz rm hadoop/share/hadoop/tools/lib/kafka-clients-0.8.2.1.jar tar zcf mapreduce.tar.gz hadoop/ hadoop fs -copyFromLocal mapreduce.tar.gz /hdp/apps/3.0.1.0-187/mapreduce ## Restart kylin and try again {code} Though it looks so tricky, but this can work without repackage. was (Author: hit_lacus): h2. *Try2* {code:java} ## remove old version kafka lib mv /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar.old ## remove mapdreduce lib cache rm -rf /data/hadoop/yarn/local/usercache/kylin-hdp3/appcache ## remove kafka-clients-0.8.2.1.jar in mapreduce.tar.gz(HDFS) hadoop fs -copyToLocal /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz . hadoop fs -mv /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz.ori tar zxf mapreduce.tar.gz rm hadoop/share/hadoop/tools/lib/kafka-clients-0.8.2.1.jar tar zcf mapreduce.tar.gz hadoop/ hadoop fs -copyFromLocal mapreduce.tar.gz /hdp/apps/3.0.1.0-187/mapreduce {code} In my env, this can work. > NRT Streaming cannot work at HDP3.0 > --- > > Key: KYLIN-3970 > URL: https://issues.apache.org/jira/browse/KYLIN-3970 > Project: Kylin > Issue Type: Bug > Components: NRT Streaming >Affects Versions: v2.6.1 >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Labels: HDP, Hadoop3 > Attachments: image-2019-04-24-15-38-10-098.png > > > We found that when using > [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], > NRT streaming cannot work in serval step for jar conflict problem. > h2. *Env* > |HDFS|3.1.1| > |YARN|3.1.1| > |MapReduce2|3.1.1| > |Hive|3.1.0| > |HBase|2.0.0| > |ZooKeeper|3.4.6| > |HDP|3.0.1.0| > > > !image-2019-04-24-15-38-10-098.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[ https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16824923#comment-16824923 ] XiaoXiang Yu edited comment on KYLIN-3970 at 4/24/19 9:22 AM: -- h2. *Try2* {code:java} ## remove old version kafka lib mv /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar.old ## remove mapdreduce lib cache rm -rf /data/hadoop/yarn/local/usercache/kylin-hdp3/appcache ## remove kafka-clients-0.8.2.1.jar in mapreduce.tar.gz(HDFS) hadoop fs -copyToLocal /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz . hadoop fs -mv /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz.ori tar zxf mapreduce.tar.gz rm hadoop/share/hadoop/tools/lib/kafka-clients-0.8.2.1.jar tar zcf mapreduce.tar.gz hadoop/ hadoop fs -copyFromLocal mapreduce.tar.gz /hdp/apps/3.0.1.0-187/mapreduce ## Restart kylin and try again {code} Though it looks so tricky, but this can work . was (Author: hit_lacus): h2. *Try2* {code:java} ## remove old version kafka lib mv /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar.old ## remove mapdreduce lib cache rm -rf /data/hadoop/yarn/local/usercache/kylin-hdp3/appcache ## remove kafka-clients-0.8.2.1.jar in mapreduce.tar.gz(HDFS) hadoop fs -copyToLocal /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz . hadoop fs -mv /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz.ori tar zxf mapreduce.tar.gz rm hadoop/share/hadoop/tools/lib/kafka-clients-0.8.2.1.jar tar zcf mapreduce.tar.gz hadoop/ hadoop fs -copyFromLocal mapreduce.tar.gz /hdp/apps/3.0.1.0-187/mapreduce ## Restart kylin and try again {code} Though it looks so tricky, but this can work without repackage. > NRT Streaming cannot work at HDP3.0 > --- > > Key: KYLIN-3970 > URL: https://issues.apache.org/jira/browse/KYLIN-3970 > Project: Kylin > Issue Type: Bug > Components: NRT Streaming >Affects Versions: v2.6.1 >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Labels: HDP, Hadoop3 > Attachments: image-2019-04-24-15-38-10-098.png > > > We found that when using > [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], > NRT streaming cannot work in serval step for jar conflict problem. > h2. *Env* > |HDFS|3.1.1| > |YARN|3.1.1| > |MapReduce2|3.1.1| > |Hive|3.1.0| > |HBase|2.0.0| > |ZooKeeper|3.4.6| > |HDP|3.0.1.0| > > > !image-2019-04-24-15-38-10-098.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[ https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16824923#comment-16824923 ] XiaoXiang Yu edited comment on KYLIN-3970 at 4/24/19 9:23 AM: -- h2. *Try-2* {code:java} ## remove old version kafka lib mv /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar.old ## remove mapdreduce lib cache rm -rf /data/hadoop/yarn/local/usercache/kylin-hdp3/appcache ## remove kafka-clients-0.8.2.1.jar in mapreduce.tar.gz(HDFS) hadoop fs -copyToLocal /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz . hadoop fs -mv /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz.ori tar zxf mapreduce.tar.gz rm hadoop/share/hadoop/tools/lib/kafka-clients-0.8.2.1.jar tar zcf mapreduce.tar.gz hadoop/ hadoop fs -copyFromLocal mapreduce.tar.gz /hdp/apps/3.0.1.0-187/mapreduce ## Restart kylin and try again {code} Though it looks so tricky, but this can work . was (Author: hit_lacus): h2. *Try2* {code:java} ## remove old version kafka lib mv /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar.old ## remove mapdreduce lib cache rm -rf /data/hadoop/yarn/local/usercache/kylin-hdp3/appcache ## remove kafka-clients-0.8.2.1.jar in mapreduce.tar.gz(HDFS) hadoop fs -copyToLocal /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz . hadoop fs -mv /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz.ori tar zxf mapreduce.tar.gz rm hadoop/share/hadoop/tools/lib/kafka-clients-0.8.2.1.jar tar zcf mapreduce.tar.gz hadoop/ hadoop fs -copyFromLocal mapreduce.tar.gz /hdp/apps/3.0.1.0-187/mapreduce ## Restart kylin and try again {code} Though it looks so tricky, but this can work . > NRT Streaming cannot work at HDP3.0 > --- > > Key: KYLIN-3970 > URL: https://issues.apache.org/jira/browse/KYLIN-3970 > Project: Kylin > Issue Type: Bug > Components: NRT Streaming >Affects Versions: v2.6.1 >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Labels: HDP, Hadoop3 > Attachments: image-2019-04-24-15-38-10-098.png > > > We found that when using > [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], > NRT streaming cannot work in serval step for jar conflict problem. > h2. *Env* > |HDFS|3.1.1| > |YARN|3.1.1| > |MapReduce2|3.1.1| > |Hive|3.1.0| > |HBase|2.0.0| > |ZooKeeper|3.4.6| > |HDP|3.0.1.0| > > > !image-2019-04-24-15-38-10-098.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[ https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3970: Description: We found that when using [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], NRT streaming cannot work in serval step for jar conflict problem. h3. *Env* |HDFS|3.1.1| |YARN|3.1.1| |MapReduce2|3.1.1| |Hive|3.1.0| |HBase|2.0.0| |ZooKeeper|3.4.6| |HDP|3.0.1.0| h3. Error message !image-2019-04-24-15-38-10-098.png! was: We found that when using [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], NRT streaming cannot work in serval step for jar conflict problem. h2. *Env* |HDFS|3.1.1| |YARN|3.1.1| |MapReduce2|3.1.1| |Hive|3.1.0| |HBase|2.0.0| |ZooKeeper|3.4.6| |HDP|3.0.1.0| !image-2019-04-24-15-38-10-098.png! > NRT Streaming cannot work at HDP3.0 > --- > > Key: KYLIN-3970 > URL: https://issues.apache.org/jira/browse/KYLIN-3970 > Project: Kylin > Issue Type: Bug > Components: NRT Streaming >Affects Versions: v2.6.1 >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Labels: HDP, Hadoop3 > Attachments: image-2019-04-24-15-38-10-098.png > > > We found that when using > [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], > NRT streaming cannot work in serval step for jar conflict problem. > h3. *Env* > |HDFS|3.1.1| > |YARN|3.1.1| > |MapReduce2|3.1.1| > |Hive|3.1.0| > |HBase|2.0.0| > |ZooKeeper|3.4.6| > |HDP|3.0.1.0| > > h3. Error message > > !image-2019-04-24-15-38-10-098.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[ https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3970: Description: We found that when using [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], NRT streaming cannot work in serval step for jar conflict problem. h3. *Env* |HDFS|3.1.1| |YARN|3.1.1| |MapReduce2|3.1.1| |Hive|3.1.0| |HBase|2.0.0| |ZooKeeper|3.4.6| |HDP|3.0.1.0| h3. *Error message* !image-2019-04-24-15-38-10-098.png! was: We found that when using [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], NRT streaming cannot work in serval step for jar conflict problem. h3. *Env* |HDFS|3.1.1| |YARN|3.1.1| |MapReduce2|3.1.1| |Hive|3.1.0| |HBase|2.0.0| |ZooKeeper|3.4.6| |HDP|3.0.1.0| h3. Error message !image-2019-04-24-15-38-10-098.png! > NRT Streaming cannot work at HDP3.0 > --- > > Key: KYLIN-3970 > URL: https://issues.apache.org/jira/browse/KYLIN-3970 > Project: Kylin > Issue Type: Bug > Components: NRT Streaming >Affects Versions: v2.6.1 >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Labels: HDP, Hadoop3 > Attachments: image-2019-04-24-15-38-10-098.png > > > We found that when using > [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], > NRT streaming cannot work in serval step for jar conflict problem. > h3. *Env* > |HDFS|3.1.1| > |YARN|3.1.1| > |MapReduce2|3.1.1| > |Hive|3.1.0| > |HBase|2.0.0| > |ZooKeeper|3.4.6| > |HDP|3.0.1.0| > h3. *Error message* > > !image-2019-04-24-15-38-10-098.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[ https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16824923#comment-16824923 ] XiaoXiang Yu edited comment on KYLIN-3970 at 4/24/19 9:30 AM: -- h2. *Try-2* {code:java} ## remove old version kafka lib mv /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar.old ## remove mapdreduce lib cache rm -rf /data/hadoop/yarn/local/usercache/${LOGIN_NMAE}/appcache ## remove kafka-clients-0.8.2.1.jar in mapreduce.tar.gz(HDFS) hadoop fs -copyToLocal /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz . hadoop fs -mv /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz.ori tar zxf mapreduce.tar.gz rm hadoop/share/hadoop/tools/lib/kafka-clients-0.8.2.1.jar ## repackage mapreduce.tar.gz tar zcf mapreduce.tar.gz hadoop/ ## put it in right place hadoop fs -copyFromLocal mapreduce.tar.gz /hdp/apps/3.0.1.0-187/mapreduce ## Restart kylin and try again sh bin/kylin.sh start{code} Though it looks so tricky, but this can work . was (Author: hit_lacus): h2. *Try-2* {code:java} ## remove old version kafka lib mv /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar.old ## remove mapdreduce lib cache rm -rf /data/hadoop/yarn/local/usercache/kylin-hdp3/appcache ## remove kafka-clients-0.8.2.1.jar in mapreduce.tar.gz(HDFS) hadoop fs -copyToLocal /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz . hadoop fs -mv /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz.ori tar zxf mapreduce.tar.gz rm hadoop/share/hadoop/tools/lib/kafka-clients-0.8.2.1.jar tar zcf mapreduce.tar.gz hadoop/ hadoop fs -copyFromLocal mapreduce.tar.gz /hdp/apps/3.0.1.0-187/mapreduce ## Restart kylin and try again {code} Though it looks so tricky, but this can work . > NRT Streaming cannot work at HDP3.0 > --- > > Key: KYLIN-3970 > URL: https://issues.apache.org/jira/browse/KYLIN-3970 > Project: Kylin > Issue Type: Bug > Components: NRT Streaming >Affects Versions: v2.6.1 >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Labels: HDP, Hadoop3 > Attachments: image-2019-04-24-15-38-10-098.png > > > We found that when using > [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], > NRT streaming cannot work in serval step for jar conflict problem. > h3. *Env* > |HDFS|3.1.1| > |YARN|3.1.1| > |MapReduce2|3.1.1| > |Hive|3.1.0| > |HBase|2.0.0| > |ZooKeeper|3.4.6| > |HDP|3.0.1.0| > h3. *Error message* > > !image-2019-04-24-15-38-10-098.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KYLIN-3996) postgresql can not to be used to construct the flat hive table in NO.1 step in building cube via datasourcedefaultSdk
[ https://issues.apache.org/jira/browse/KYLIN-3996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu reassigned KYLIN-3996: --- Assignee: XiaoXiang Yu > postgresql can not to be used to construct the flat hive table in NO.1 step > in building cube via datasourcedefaultSdk > - > > Key: KYLIN-3996 > URL: https://issues.apache.org/jira/browse/KYLIN-3996 > Project: Kylin > Issue Type: Bug > Components: Driver - JDBC >Affects Versions: v2.6.1 > Environment: kylin3.0.0+hadoop2.7.4+hive2.1.1+hbase1.2.6 >Reporter: haijiandu >Assignee: XiaoXiang Yu >Priority: Critical > Labels: datasource, postgresql, sdk > Attachments: kylin_log_error_1.jpg, kylin_log_error_2.jpg, > kylin_log_error_3.jpg, kylin_log_error_4.jpg, kylin_log_error_5.jpg, > kylin_properties_duhj.jpg, kylin_sql.jpg, > kylin_sqoopToflatHiveTable_Error.jpg, postgresql_xml.jpg, > server_config_kylin_1.jpg, server_config_kylin_1.jpg, > server_config_kylin_2.jpg > > > postgresql can not to be used to construct the flat hive table in NO.1 step > in building cube via datasourcedefaultSdk. whether in kylin.properties or in > postgresql.xml,corresponding configureations have been finished as required > by official specification.There alwarys occurs a problem such as "ERROR > manager.SqlManager: Error executing statement: > org.postgresql.util.PSQLException: > ERROR: syntax error at or near ".", > when kylin is linked to postgresql databese, it seems that the errors happen > in sql brought by kylin. the sql has back quote,as it is "`" ,that causes > query failure brought by table name and columns defined by back quote(`). > as required, in kylin.properties ,its configuration has been as followings > kylin.source.default=16 > kylin.source.jdbc.connection-url=jdbc:postgresql://192.168.75.123:5432/dvdrental > kylin.source.jdbc.driver=org.postgresql.Driver > kylin.source.jdbc.dialect=postgresql > kylin.source.jdbc.adaptor=org.apache.kylin.sdk.datasource.adaptor.DefaultAdaptor > kylin.source.jdbc.user=postgres > kylin.source.jdbc.pass=1234 > kylin.source.jdbc.sqoop-home=/duhj/sqoop-1.4.7.bin__hadoop-2.6.0 > kylin.source.jdbc.filed-delimiter=| > kylin.source.jdbc.sqoop-mapper-num=4 > what is more, in > ~/apache-kylin-3.0/conf/datasource/postgresql.xml,corresponding conf is as > followings: > - > > > > > > > > > > > is there a method to solve it? > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-4001) Allow user-specified time format using real-time
[ https://issues.apache.org/jira/browse/KYLIN-4001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-4001: Fix Version/s: v3.0.0 > Allow user-specified time format using real-time > > > Key: KYLIN-4001 > URL: https://issues.apache.org/jira/browse/KYLIN-4001 > Project: Kylin > Issue Type: Improvement > Components: Real-time Streaming >Reporter: ning guo >Priority: Minor > Fix For: v3.0.0 > > > * I found that real-time only supports millisecond timestamp, does not > support second timestamp and Date type like '2019-01-01 11:11:11'. > * I add a LongTimeParser and a DateTimeParser and page configuration > * You can configure tsParser, tsPattern on the page that creates the > streaming table. > * for date : > {code:java} > { "timestamp":"2019-04-29 11:11:11","gmv":1.1 } > You can specify > tsParser=org.apache.kylin.stream.source.kafka.DateTimeParser > tsPattern=-MM-dd HH:mm:ss{code} > > * for second : > {code:java} > { "timestamp":"1556618887","gmv":1.1 } > You can specify > tsParser=org.apache.kylin.stream.source.kafka.LongTimeParser > tsPattern=S{code} > > * for millisecond : > {code:java} > { "timestamp":"1556618887000","gmv":1.1 } > You can specify > tsParser=org.apache.kylin.stream.source.kafka.LongTimeParser > tsPattern=MS > {code} > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KYLIN-4007) Kylin concurrently builds a global dictionary for too long
[ https://issues.apache.org/jira/browse/KYLIN-4007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu reassigned KYLIN-4007: --- Assignee: XiaoXiang Yu > Kylin concurrently builds a global dictionary for too long > -- > > Key: KYLIN-4007 > URL: https://issues.apache.org/jira/browse/KYLIN-4007 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.6.1 >Reporter: mouxingya >Assignee: XiaoXiang Yu >Priority: Minor > Attachments: WechatIMG20.png > > > When a dimension (user_id) is accurately calculated at the same time, the > original data is in the order of millions, and the rowkey has the encoding > type fixed_lengrh and length 100. It took a few hours or even failed in the > build dimension dictionary step. After investigation, it was found that there > are concurrent processes that want to obtain the lock corresponding to the > cube / dict path, resulting in a deadlock. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-4007) Kylin concurrently builds a global dictionary for too long
[ https://issues.apache.org/jira/browse/KYLIN-4007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16843878#comment-16843878 ] XiaoXiang Yu commented on KYLIN-4007: - Looks like this issue is related to https://issues.apache.org/jira/browse/KYLIN-3633. > Kylin concurrently builds a global dictionary for too long > -- > > Key: KYLIN-4007 > URL: https://issues.apache.org/jira/browse/KYLIN-4007 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.6.1 >Reporter: mouxingya >Assignee: XiaoXiang Yu >Priority: Minor > Attachments: WechatIMG20.png > > > When a dimension (user_id) is accurately calculated at the same time, the > original data is in the order of millions, and the rowkey has the encoding > type fixed_lengrh and length 100. It took a few hours or even failed in the > build dimension dictionary step. After investigation, it was found that there > are concurrent processes that want to obtain the lock corresponding to the > cube / dict path, resulting in a deadlock. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-4007) Kylin concurrently builds a global dictionary for too long
[ https://issues.apache.org/jira/browse/KYLIN-4007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16843878#comment-16843878 ] XiaoXiang Yu edited comment on KYLIN-4007 at 5/20/19 11:28 AM: --- Looks like this issue is related to https://issues.apache.org/jira/browse/KYLIN-3633. But it should be fixed at v2.5.1, maybe it caused by other exception. But without more log/ stack trace, it difficult to detect in which way should we repair. was (Author: hit_lacus): Looks like this issue is related to https://issues.apache.org/jira/browse/KYLIN-3633. > Kylin concurrently builds a global dictionary for too long > -- > > Key: KYLIN-4007 > URL: https://issues.apache.org/jira/browse/KYLIN-4007 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.6.1 >Reporter: mouxingya >Assignee: XiaoXiang Yu >Priority: Minor > Attachments: WechatIMG20.png > > > When a dimension (user_id) is accurately calculated at the same time, the > original data is in the order of millions, and the rowkey has the encoding > type fixed_lengrh and length 100. It took a few hours or even failed in the > build dimension dictionary step. After investigation, it was found that there > are concurrent processes that want to obtain the lock corresponding to the > cube / dict path, resulting in a deadlock. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-4008) Real-time Streaming submit streaming job failed for spark engine
[ https://issues.apache.org/jira/browse/KYLIN-4008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16844623#comment-16844623 ] XiaoXiang Yu commented on KYLIN-4008: - Hi , currently realtime-olap do not support spark build engine. Maybe it will be support later. > Real-time Streaming submit streaming job failed for spark engine > > > Key: KYLIN-4008 > URL: https://issues.apache.org/jira/browse/KYLIN-4008 > Project: Kylin > Issue Type: Bug > Components: Real-time Streaming >Affects Versions: v3.0.0-alpha >Reporter: zengrui >Priority: Major > Attachments: error.bmp > > > Create a Realtime Streaming Cube and the Cube Engine is Spark, when the > coordinator node receive a remoteStoreCompelete request and exist some > segments can build, the streaming job submit failed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-4008) Real-time Streaming submit streaming job failed for spark engine
[ https://issues.apache.org/jira/browse/KYLIN-4008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16844623#comment-16844623 ] XiaoXiang Yu edited comment on KYLIN-4008 at 5/21/19 8:23 AM: -- Hi , currently realtime-olap do not support spark build engine. Maybe it will be support later. So I think this should not be a bug. was (Author: hit_lacus): Hi , currently realtime-olap do not support spark build engine. Maybe it will be support later. > Real-time Streaming submit streaming job failed for spark engine > > > Key: KYLIN-4008 > URL: https://issues.apache.org/jira/browse/KYLIN-4008 > Project: Kylin > Issue Type: Bug > Components: Real-time Streaming >Affects Versions: v3.0.0-alpha >Reporter: zengrui >Priority: Major > Attachments: error.bmp > > > Create a Realtime Streaming Cube and the Cube Engine is Spark, when the > coordinator node receive a remoteStoreCompelete request and exist some > segments can build, the streaming job submit failed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-4010) TimeZone is hard-coded in function makeSegmentName for class CubeSegment
[ https://issues.apache.org/jira/browse/KYLIN-4010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16844627#comment-16844627 ] XiaoXiang Yu commented on KYLIN-4010: - Hi, this seems to be a known issue in https://issues.apache.org/jira/browse/KYLIN-4001, and it should be implemented in next release. > TimeZone is hard-coded in function makeSegmentName for class CubeSegment > > > Key: KYLIN-4010 > URL: https://issues.apache.org/jira/browse/KYLIN-4010 > Project: Kylin > Issue Type: Improvement > Components: Others >Affects Versions: v2.6.2, v3.0.0-alpha >Reporter: zengrui >Priority: Minor > > In Real-Time Streaming Cube when I send some records to kafka topic, the > tmestamp for the record is 2019-01-01 00:00:00.000, but kylin create a > segment named 2018123116_2018123117. > Then I found that TimeZone is hard-coded to "GMT" in function makeSegmentName > for class CubeSegment. I think that it should be config in kylin.properties. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-4008) Real-time Streaming submit streaming job failed for spark engine
[ https://issues.apache.org/jira/browse/KYLIN-4008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-4008: Issue Type: New Feature (was: Bug) > Real-time Streaming submit streaming job failed for spark engine > > > Key: KYLIN-4008 > URL: https://issues.apache.org/jira/browse/KYLIN-4008 > Project: Kylin > Issue Type: New Feature > Components: Real-time Streaming >Affects Versions: v3.0.0-alpha >Reporter: zengrui >Priority: Major > Attachments: error.bmp > > > Create a Realtime Streaming Cube and the Cube Engine is Spark, when the > coordinator node receive a remoteStoreCompelete request and exist some > segments can build, the streaming job submit failed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-4008) Real-time Streaming submit streaming job failed for spark engine
[ https://issues.apache.org/jira/browse/KYLIN-4008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-4008: Priority: Minor (was: Major) > Real-time Streaming submit streaming job failed for spark engine > > > Key: KYLIN-4008 > URL: https://issues.apache.org/jira/browse/KYLIN-4008 > Project: Kylin > Issue Type: New Feature > Components: Real-time Streaming >Affects Versions: v3.0.0-alpha >Reporter: zengrui >Priority: Minor > Attachments: error.bmp > > > Create a Realtime Streaming Cube and the Cube Engine is Spark, when the > coordinator node receive a remoteStoreCompelete request and exist some > segments can build, the streaming job submit failed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-4008) Real-time Streaming submit streaming job failed for spark engine
[ https://issues.apache.org/jira/browse/KYLIN-4008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16844623#comment-16844623 ] XiaoXiang Yu edited comment on KYLIN-4008 at 5/21/19 8:30 AM: -- Hi , currently realtime-olap do not support spark build engine. Maybe it will be support later. So I think this should not be a bug, so I change this issue's type. was (Author: hit_lacus): Hi , currently realtime-olap do not support spark build engine. Maybe it will be support later. So I think this should not be a bug. > Real-time Streaming submit streaming job failed for spark engine > > > Key: KYLIN-4008 > URL: https://issues.apache.org/jira/browse/KYLIN-4008 > Project: Kylin > Issue Type: New Feature > Components: Real-time Streaming >Affects Versions: v3.0.0-alpha >Reporter: zengrui >Priority: Minor > Attachments: error.bmp > > > Create a Realtime Streaming Cube and the Cube Engine is Spark, when the > coordinator node receive a remoteStoreCompelete request and exist some > segments can build, the streaming job submit failed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)