[jira] [Created] (KYLIN-3792) Reassign's inconsistent state need automation heal
XiaoXiang Yu created KYLIN-3792: --- Summary: Reassign's inconsistent state need automation heal Key: KYLIN-3792 URL: https://issues.apache.org/jira/browse/KYLIN-3792 Project: Kylin Issue Type: Improvement Components: Real-time Streaming Reporter: XiaoXiang Yu Assignee: XiaoXiang Yu Realtime streaming has a reassign/rebalance operation which may cause some inconsistent state which is difficult to recover. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3792) Reassign's inconsistent state need automation heal
[ https://issues.apache.org/jira/browse/KYLIN-3792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3792: External issue URL: (was: https://github.com/hit-lacus/hit-lacus.github.io/issues/23) > Reassign's inconsistent state need automation heal > -- > > Key: KYLIN-3792 > URL: https://issues.apache.org/jira/browse/KYLIN-3792 > Project: Kylin > Issue Type: Improvement > Components: Real-time Streaming >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Original Estimate: 96h > Remaining Estimate: 96h > > Realtime streaming has a reassign/rebalance operation which may cause some > inconsistent state which is difficult to recover. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3792) Reassign's inconsistent state need automation heal
[ https://issues.apache.org/jira/browse/KYLIN-3792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3792: External issue URL: https://github.com/hit-lacus/hit-lacus.github.io/issues/23 Remaining Estimate: 96h Original Estimate: 96h > Reassign's inconsistent state need automation heal > -- > > Key: KYLIN-3792 > URL: https://issues.apache.org/jira/browse/KYLIN-3792 > Project: Kylin > Issue Type: Improvement > Components: Real-time Streaming >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Original Estimate: 96h > Remaining Estimate: 96h > > Realtime streaming has a reassign/rebalance operation which may cause some > inconsistent state which is difficult to recover. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3792) Reassign's inconsistent state need automation heal
[ https://issues.apache.org/jira/browse/KYLIN-3792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3792: Labels: reassign (was: ) > Reassign's inconsistent state need automation heal > -- > > Key: KYLIN-3792 > URL: https://issues.apache.org/jira/browse/KYLIN-3792 > Project: Kylin > Issue Type: Improvement > Components: Real-time Streaming >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Labels: reassign > Original Estimate: 96h > Remaining Estimate: 96h > > Realtime streaming has a reassign/rebalance operation which may cause some > inconsistent state which is difficult to recover. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3792) Reassign's inconsistent state need automation recovery
[ https://issues.apache.org/jira/browse/KYLIN-3792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3792: Summary: Reassign's inconsistent state need automation recovery (was: Reassign's inconsistent state need automation heal) > Reassign's inconsistent state need automation recovery > -- > > Key: KYLIN-3792 > URL: https://issues.apache.org/jira/browse/KYLIN-3792 > Project: Kylin > Issue Type: Improvement > Components: Real-time Streaming >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Labels: reassign > Original Estimate: 96h > Remaining Estimate: 96h > > Realtime streaming has a reassign/rebalance operation which may cause some > inconsistent state which is difficult to recover. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3044) Support SQL Server as data source
[ https://issues.apache.org/jira/browse/KYLIN-3044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16755607#comment-16755607 ] XiaoXiang Yu commented on KYLIN-3044: - !image-2019-01-30-10-46-44-236.png! > Support SQL Server as data source > - > > Key: KYLIN-3044 > URL: https://issues.apache.org/jira/browse/KYLIN-3044 > Project: Kylin > Issue Type: New Feature >Reporter: Kaige Liu >Assignee: Kaige Liu >Priority: Major > Fix For: v2.3.0 > > Attachments: KYLIN-3044-sqlserver-as-datasource.patch > > > [KYLIN-1351|https://issues.apache.org/jira/browse/KYLIN-1351] has added > Vertica as data source. Base on the work of KYLIN-1351, I'd like to enable > SQL Server as data source of kylin. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3798) SQL Server cannot list databases
XiaoXiang Yu created KYLIN-3798: --- Summary: SQL Server cannot list databases Key: KYLIN-3798 URL: https://issues.apache.org/jira/browse/KYLIN-3798 Project: Kylin Issue Type: Bug Components: RDBMS Source Reporter: XiaoXiang Yu Assignee: XiaoXiang Yu Attachments: image-2019-01-30-11-14-11-939.png, image-2019-01-30-11-14-38-581.png, image-2019-01-30-11-15-07-865.png !image-2019-01-30-11-14-11-939.png! !image-2019-01-30-11-15-07-865.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3798) SQL Server cannot list databases
[ https://issues.apache.org/jira/browse/KYLIN-3798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3798: Attachment: image-2019-01-30-11-16-41-320.png > SQL Server cannot list databases > > > Key: KYLIN-3798 > URL: https://issues.apache.org/jira/browse/KYLIN-3798 > Project: Kylin > Issue Type: Bug > Components: RDBMS Source >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Attachments: image-2019-01-30-11-14-11-939.png, > image-2019-01-30-11-14-38-581.png, image-2019-01-30-11-15-07-865.png, > image-2019-01-30-11-16-16-975.png, image-2019-01-30-11-16-41-320.png > > > > !image-2019-01-30-11-14-11-939.png! > > !image-2019-01-30-11-15-07-865.png! > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3798) SQL Server cannot list databases
[ https://issues.apache.org/jira/browse/KYLIN-3798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3798: Description: !image-2019-01-30-11-14-11-939.png! !image-2019-01-30-11-16-41-320.png! was: !image-2019-01-30-11-14-11-939.png! !image-2019-01-30-11-15-07-865.png! > SQL Server cannot list databases > > > Key: KYLIN-3798 > URL: https://issues.apache.org/jira/browse/KYLIN-3798 > Project: Kylin > Issue Type: Bug > Components: RDBMS Source >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Attachments: image-2019-01-30-11-14-11-939.png, > image-2019-01-30-11-14-38-581.png, image-2019-01-30-11-15-07-865.png, > image-2019-01-30-11-16-16-975.png, image-2019-01-30-11-16-41-320.png > > > > !image-2019-01-30-11-14-11-939.png! > > > !image-2019-01-30-11-16-41-320.png! > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3798) SQL Server cannot list databases
[ https://issues.apache.org/jira/browse/KYLIN-3798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3798: Attachment: image-2019-01-30-11-16-16-975.png > SQL Server cannot list databases > > > Key: KYLIN-3798 > URL: https://issues.apache.org/jira/browse/KYLIN-3798 > Project: Kylin > Issue Type: Bug > Components: RDBMS Source >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Attachments: image-2019-01-30-11-14-11-939.png, > image-2019-01-30-11-14-38-581.png, image-2019-01-30-11-15-07-865.png, > image-2019-01-30-11-16-16-975.png, image-2019-01-30-11-16-41-320.png > > > > !image-2019-01-30-11-14-11-939.png! > > !image-2019-01-30-11-15-07-865.png! > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3798) SQL Server cannot list databases
[
https://issues.apache.org/jira/browse/KYLIN-3798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
XiaoXiang Yu updated KYLIN-3798:
Attachment: (was: image-2019-01-30-11-15-07-865.png)
> SQL Server cannot list databases
>
>
> Key: KYLIN-3798
> URL: https://issues.apache.org/jira/browse/KYLIN-3798
> Project: Kylin
> Issue Type: Bug
> Components: RDBMS Source
>Reporter: XiaoXiang Yu
>Assignee: XiaoXiang Yu
>Priority: Major
> Attachments: image-2019-01-30-11-14-11-939.png,
> image-2019-01-30-11-16-16-975.png, image-2019-01-30-11-16-41-320.png
>
>
> User report sqlserver as data source didn't show databases. Following is his
> email:
>
> {quote}Hi,{quote}
> {quote} I followed all the steps in this url
> [http://kylin.apache.org/docs/tutorial/setup_jdbc_datasource.html] , but when
> i click on Load table button or Load table from tree i don't seem to have any
> tables loaded from the SQL Server data source.There are no errors/exceptions
> in the logs too.
>
> kylin.properties:
> {quote}
> {quote}kylin.source.default=8
> kylin.source.jdbc.connection-url=jdbc:sqlserver://hostname:1433;database=sample
> kylin.source.jdbc.driver=com.microsoft.sqlserver.jdbc.SQLServerDriver
> kylin.source.jdbc.dialect=mssql
> kylin.source.jdbc.user=user
> kylin.source.jdbc.pass=pass
> kylin.source.jdbc.sqoop-home=sqoophome
> kylin.source.jdbc.filed-delimiter=|
> kylin.source.jdbc.sqoop-mapper-num=4{quote}
> I reproduce this bug in my dev env. And I find this question was caused by
> SQLServerJdbcMetadata. I have modified SQLServerJdbcMetadata and databases
> could be showed.
>
> !image-2019-01-30-11-14-11-939.png!
>
>
> !image-2019-01-30-11-16-41-320.png!
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3798) SQL Server cannot list databases
[
https://issues.apache.org/jira/browse/KYLIN-3798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
XiaoXiang Yu updated KYLIN-3798:
Attachment: (was: image-2019-01-30-11-14-38-581.png)
> SQL Server cannot list databases
>
>
> Key: KYLIN-3798
> URL: https://issues.apache.org/jira/browse/KYLIN-3798
> Project: Kylin
> Issue Type: Bug
> Components: RDBMS Source
>Reporter: XiaoXiang Yu
>Assignee: XiaoXiang Yu
>Priority: Major
> Attachments: image-2019-01-30-11-14-11-939.png,
> image-2019-01-30-11-16-16-975.png, image-2019-01-30-11-16-41-320.png
>
>
> User report sqlserver as data source didn't show databases. Following is his
> email:
>
> {quote}Hi,{quote}
> {quote} I followed all the steps in this url
> [http://kylin.apache.org/docs/tutorial/setup_jdbc_datasource.html] , but when
> i click on Load table button or Load table from tree i don't seem to have any
> tables loaded from the SQL Server data source.There are no errors/exceptions
> in the logs too.
>
> kylin.properties:
> {quote}
> {quote}kylin.source.default=8
> kylin.source.jdbc.connection-url=jdbc:sqlserver://hostname:1433;database=sample
> kylin.source.jdbc.driver=com.microsoft.sqlserver.jdbc.SQLServerDriver
> kylin.source.jdbc.dialect=mssql
> kylin.source.jdbc.user=user
> kylin.source.jdbc.pass=pass
> kylin.source.jdbc.sqoop-home=sqoophome
> kylin.source.jdbc.filed-delimiter=|
> kylin.source.jdbc.sqoop-mapper-num=4{quote}
> I reproduce this bug in my dev env. And I find this question was caused by
> SQLServerJdbcMetadata. I have modified SQLServerJdbcMetadata and databases
> could be showed.
>
> !image-2019-01-30-11-14-11-939.png!
>
>
> !image-2019-01-30-11-16-41-320.png!
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3798) SQL Server cannot list databases
[
https://issues.apache.org/jira/browse/KYLIN-3798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
XiaoXiang Yu updated KYLIN-3798:
Description:
User report sqlserver as data source didn't show databases. Following is his
email:
{quote}Hi,
{quote}
{quote} I followed all the steps in this url
[http://kylin.apache.org/docs/tutorial/setup_jdbc_datasource.html] , but when i
click on Load table button or Load table from tree i don't seem to have any
tables loaded from the SQL Server data source.There are no errors/exceptions in
the logs too.
kylin.properties:
{quote}
{quote}kylin.source.default=8
kylin.source.jdbc.connection-url=jdbc:sqlserver://hostname:1433;database=sample
kylin.source.jdbc.driver=com.microsoft.sqlserver.jdbc.SQLServerDriver
kylin.source.jdbc.dialect=mssql
kylin.source.jdbc.user=user
kylin.source.jdbc.pass=pass
kylin.source.jdbc.sqoop-home=sqoophome
kylin.source.jdbc.filed-delimiter=|
kylin.source.jdbc.sqoop-mapper-num=4
{quote}
I reproduce this bug in my dev env. And I find this question was caused by
SQLServerJdbcMetadata. I have modified SQLServerJdbcMetadata and databases
could be showed.
My env: SqlServer version is 2017, and JDBC Driver version is
{quote}
com.microsoft.sqlserver
mssql-jdbc
6.4.0.jre8
{quote}
!image-2019-01-30-11-14-11-939.png!
!image-2019-01-30-11-16-41-320.png!
was:
User report sqlserver as data source didn't show databases. Following is his
email:
{quote}Hi,{quote}
{quote} I followed all the steps in this url
[http://kylin.apache.org/docs/tutorial/setup_jdbc_datasource.html] , but when i
click on Load table button or Load table from tree i don't seem to have any
tables loaded from the SQL Server data source.There are no errors/exceptions in
the logs too.
kylin.properties:
{quote}
{quote}kylin.source.default=8
kylin.source.jdbc.connection-url=jdbc:sqlserver://hostname:1433;database=sample
kylin.source.jdbc.driver=com.microsoft.sqlserver.jdbc.SQLServerDriver
kylin.source.jdbc.dialect=mssql
kylin.source.jdbc.user=user
kylin.source.jdbc.pass=pass
kylin.source.jdbc.sqoop-home=sqoophome
kylin.source.jdbc.filed-delimiter=|
kylin.source.jdbc.sqoop-mapper-num=4{quote}
I reproduce this bug in my dev env. And I find this question was caused by
SQLServerJdbcMetadata. I have modified SQLServerJdbcMetadata and databases
could be showed.
!image-2019-01-30-11-14-11-939.png!
!image-2019-01-30-11-16-41-320.png!
> SQL Server cannot list databases
>
>
> Key: KYLIN-3798
> URL: https://issues.apache.org/jira/browse/KYLIN-3798
> Project: Kylin
> Issue Type: Bug
> Components: RDBMS Source
>Reporter: XiaoXiang Yu
>Assignee: XiaoXiang Yu
>Priority: Major
> Attachments: image-2019-01-30-11-14-11-939.png,
> image-2019-01-30-11-16-16-975.png, image-2019-01-30-11-16-41-320.png
>
>
> User report sqlserver as data source didn't show databases. Following is his
> email:
>
> {quote}Hi,
> {quote}
> {quote} I followed all the steps in this url
> [http://kylin.apache.org/docs/tutorial/setup_jdbc_datasource.html] , but when
> i click on Load table button or Load table from tree i don't seem to have any
> tables loaded from the SQL Server data source.There are no errors/exceptions
> in the logs too.
>
> kylin.properties:
>
> {quote}
> {quote}kylin.source.default=8
>
> kylin.source.jdbc.connection-url=jdbc:sqlserver://hostname:1433;database=sample
> kylin.source.jdbc.driver=com.microsoft.sqlserver.jdbc.SQLServerDriver
> kylin.source.jdbc.dialect=mssql
> kylin.source.jdbc.user=user
> kylin.source.jdbc.pass=pass
> kylin.source.jdbc.sqoop-home=sqoophome
> kylin.source.jdbc.filed-delimiter=|
> kylin.source.jdbc.sqoop-mapper-num=4
> {quote}
> I reproduce this bug in my dev env. And I find this question was caused by
> SQLServerJdbcMetadata. I have modified SQLServerJdbcMetadata and databases
> could be showed.
>
> My env: SqlServer version is 2017, and JDBC Driver version is
> {quote}
> com.microsoft.sqlserver
> mssql-jdbc
> 6.4.0.jre8
>
> {quote}
>
>
> !image-2019-01-30-11-14-11-939.png!
>
>
> !image-2019-01-30-11-16-41-320.png!
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3798) SQL Server cannot list databases
[
https://issues.apache.org/jira/browse/KYLIN-3798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16755634#comment-16755634
]
XiaoXiang Yu commented on KYLIN-3798:
-
h2. Test Result
!image-2019-01-30-11-25-47-946.png!
> SQL Server cannot list databases
>
>
> Key: KYLIN-3798
> URL: https://issues.apache.org/jira/browse/KYLIN-3798
> Project: Kylin
> Issue Type: Bug
> Components: RDBMS Source
>Reporter: XiaoXiang Yu
>Assignee: XiaoXiang Yu
>Priority: Major
> Attachments: image-2019-01-30-11-14-11-939.png,
> image-2019-01-30-11-16-16-975.png, image-2019-01-30-11-16-41-320.png,
> image-2019-01-30-11-25-47-946.png
>
>
> User report sqlserver as data source didn't show databases. Following is his
> email:
>
> {quote}Hi,
> {quote}
> {quote} I followed all the steps in this url
> [http://kylin.apache.org/docs/tutorial/setup_jdbc_datasource.html] , but when
> i click on Load table button or Load table from tree i don't seem to have any
> tables loaded from the SQL Server data source.There are no errors/exceptions
> in the logs too.
>
> kylin.properties:
>
> {quote}
> {quote}kylin.source.default=8
>
> kylin.source.jdbc.connection-url=jdbc:sqlserver://hostname:1433;database=sample
> kylin.source.jdbc.driver=com.microsoft.sqlserver.jdbc.SQLServerDriver
> kylin.source.jdbc.dialect=mssql
> kylin.source.jdbc.user=user
> kylin.source.jdbc.pass=pass
> kylin.source.jdbc.sqoop-home=sqoophome
> kylin.source.jdbc.filed-delimiter=|
> kylin.source.jdbc.sqoop-mapper-num=4
> {quote}
> I reproduce this bug in my dev env. And I find this question was caused by
> SQLServerJdbcMetadata. I have modified SQLServerJdbcMetadata and databases
> could be showed.
>
> My env: SqlServer version is 2017, and JDBC Driver version is
> {quote}
> com.microsoft.sqlserver
> mssql-jdbc
> 6.4.0.jre8
>
> {quote}
>
>
> !image-2019-01-30-11-14-11-939.png!
>
>
> !image-2019-01-30-11-16-41-320.png!
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3798) SQL Server cannot list databases
[
https://issues.apache.org/jira/browse/KYLIN-3798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
XiaoXiang Yu updated KYLIN-3798:
Attachment: image-2019-01-30-11-25-47-946.png
> SQL Server cannot list databases
>
>
> Key: KYLIN-3798
> URL: https://issues.apache.org/jira/browse/KYLIN-3798
> Project: Kylin
> Issue Type: Bug
> Components: RDBMS Source
>Reporter: XiaoXiang Yu
>Assignee: XiaoXiang Yu
>Priority: Major
> Attachments: image-2019-01-30-11-14-11-939.png,
> image-2019-01-30-11-16-16-975.png, image-2019-01-30-11-16-41-320.png,
> image-2019-01-30-11-25-47-946.png
>
>
> User report sqlserver as data source didn't show databases. Following is his
> email:
>
> {quote}Hi,
> {quote}
> {quote} I followed all the steps in this url
> [http://kylin.apache.org/docs/tutorial/setup_jdbc_datasource.html] , but when
> i click on Load table button or Load table from tree i don't seem to have any
> tables loaded from the SQL Server data source.There are no errors/exceptions
> in the logs too.
>
> kylin.properties:
>
> {quote}
> {quote}kylin.source.default=8
>
> kylin.source.jdbc.connection-url=jdbc:sqlserver://hostname:1433;database=sample
> kylin.source.jdbc.driver=com.microsoft.sqlserver.jdbc.SQLServerDriver
> kylin.source.jdbc.dialect=mssql
> kylin.source.jdbc.user=user
> kylin.source.jdbc.pass=pass
> kylin.source.jdbc.sqoop-home=sqoophome
> kylin.source.jdbc.filed-delimiter=|
> kylin.source.jdbc.sqoop-mapper-num=4
> {quote}
> I reproduce this bug in my dev env. And I find this question was caused by
> SQLServerJdbcMetadata. I have modified SQLServerJdbcMetadata and databases
> could be showed.
>
> My env: SqlServer version is 2017, and JDBC Driver version is
> {quote}
> com.microsoft.sqlserver
> mssql-jdbc
> 6.4.0.jre8
>
> {quote}
>
>
> !image-2019-01-30-11-14-11-939.png!
>
>
> !image-2019-01-30-11-16-41-320.png!
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3798) SQL Server cannot list databases
[
https://issues.apache.org/jira/browse/KYLIN-3798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
XiaoXiang Yu updated KYLIN-3798:
Description:
User report sqlserver as data source didn't show databases. Following is his
email:
{quote}Hi,{quote}
{quote} I followed all the steps in this url
[http://kylin.apache.org/docs/tutorial/setup_jdbc_datasource.html] , but when i
click on Load table button or Load table from tree i don't seem to have any
tables loaded from the SQL Server data source.There are no errors/exceptions in
the logs too.
kylin.properties:
{quote}
{quote}kylin.source.default=8
kylin.source.jdbc.connection-url=jdbc:sqlserver://hostname:1433;database=sample
kylin.source.jdbc.driver=com.microsoft.sqlserver.jdbc.SQLServerDriver
kylin.source.jdbc.dialect=mssql
kylin.source.jdbc.user=user
kylin.source.jdbc.pass=pass
kylin.source.jdbc.sqoop-home=sqoophome
kylin.source.jdbc.filed-delimiter=|
kylin.source.jdbc.sqoop-mapper-num=4{quote}
I reproduce this bug in my dev env. And I find this question was caused by
SQLServerJdbcMetadata. I have modified SQLServerJdbcMetadata and databases
could be showed.
!image-2019-01-30-11-14-11-939.png!
!image-2019-01-30-11-16-41-320.png!
was:
!image-2019-01-30-11-14-11-939.png!
!image-2019-01-30-11-16-41-320.png!
> SQL Server cannot list databases
>
>
> Key: KYLIN-3798
> URL: https://issues.apache.org/jira/browse/KYLIN-3798
> Project: Kylin
> Issue Type: Bug
> Components: RDBMS Source
>Reporter: XiaoXiang Yu
>Assignee: XiaoXiang Yu
>Priority: Major
> Attachments: image-2019-01-30-11-14-11-939.png,
> image-2019-01-30-11-16-16-975.png, image-2019-01-30-11-16-41-320.png
>
>
> User report sqlserver as data source didn't show databases. Following is his
> email:
>
> {quote}Hi,{quote}
> {quote} I followed all the steps in this url
> [http://kylin.apache.org/docs/tutorial/setup_jdbc_datasource.html] , but when
> i click on Load table button or Load table from tree i don't seem to have any
> tables loaded from the SQL Server data source.There are no errors/exceptions
> in the logs too.
>
> kylin.properties:
> {quote}
> {quote}kylin.source.default=8
> kylin.source.jdbc.connection-url=jdbc:sqlserver://hostname:1433;database=sample
> kylin.source.jdbc.driver=com.microsoft.sqlserver.jdbc.SQLServerDriver
> kylin.source.jdbc.dialect=mssql
> kylin.source.jdbc.user=user
> kylin.source.jdbc.pass=pass
> kylin.source.jdbc.sqoop-home=sqoophome
> kylin.source.jdbc.filed-delimiter=|
> kylin.source.jdbc.sqoop-mapper-num=4{quote}
> I reproduce this bug in my dev env. And I find this question was caused by
> SQLServerJdbcMetadata. I have modified SQLServerJdbcMetadata and databases
> could be showed.
>
> !image-2019-01-30-11-14-11-939.png!
>
>
> !image-2019-01-30-11-16-41-320.png!
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (KYLIN-3791) Map return by Maps.transformValues is a immutable view
[ https://issues.apache.org/jira/browse/KYLIN-3791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu resolved KYLIN-3791. - Resolution: Fixed > Map return by Maps.transformValues is a immutable view > -- > > Key: KYLIN-3791 > URL: https://issues.apache.org/jira/browse/KYLIN-3791 > Project: Kylin > Issue Type: Improvement > Components: Real-time Streaming >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Attachments: image-2019-01-26-20-53-10-085.png, > image-2019-01-26-20-54-12-483.png > > > !image-2019-01-26-20-53-10-085.png! > > > !image-2019-01-26-20-54-12-483.png! > > After restart my receiver, I found a lot of error logs which is confused. It > is a bug caused by misuse of Guava. The bug can only be triggered when > loading segment cache. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3759) Streaming ClassNotFoundExeception when lambda is enable in MR job
[ https://issues.apache.org/jira/browse/KYLIN-3759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu resolved KYLIN-3759. - Resolution: Fixed > Streaming ClassNotFoundExeception when lambda is enable in MR job > - > > Key: KYLIN-3759 > URL: https://issues.apache.org/jira/browse/KYLIN-3759 > Project: Kylin > Issue Type: Improvement > Components: Real-time Streaming >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Attachments: image-2019-01-23-16-09-29-317.png > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3784) Hadoop Common ReflectionUtils.printThreadInfo signature change
[ https://issues.apache.org/jira/browse/KYLIN-3784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu resolved KYLIN-3784. - Resolution: Fixed > Hadoop Common ReflectionUtils.printThreadInfo signature change > -- > > Key: KYLIN-3784 > URL: https://issues.apache.org/jira/browse/KYLIN-3784 > Project: Kylin > Issue Type: Improvement > Components: Real-time Streaming >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Attachments: image-2019-01-24-09-59-00-421.png > > > !image-2019-01-24-09-59-00-421.png! > > > Can not build a jar because Hadoop jar version compatibility. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KYLIN-3857) add parameter to change sql quoting character for sqoop
[
https://issues.apache.org/jira/browse/KYLIN-3857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
XiaoXiang Yu reassigned KYLIN-3857:
---
Assignee: XiaoXiang Yu
> add parameter to change sql quoting character for sqoop
> ---
>
> Key: KYLIN-3857
> URL: https://issues.apache.org/jira/browse/KYLIN-3857
> Project: Kylin
> Issue Type: Bug
>Reporter: Davide Malagoli
>Assignee: XiaoXiang Yu
>Priority: Major
> Attachments: sqoop-error.log, sqoop-script.sh
>
>
> I've set up Kylin to access a SQL server using JDBC and do the ETL step as
> described in
> [http://kylin.apache.org/docs/tutorial/setup_jdbc_datasource.html]
> but when the "Sqoop To Flat Hive Table" runs it terminates with an error
> shown in the attached log.
> It seems that the root cause is the presence of the '`' character in the
> query, which has no meaning for sql server.
>
> My problem seems to related to this piece of code
>
> public class FlatTableSqlQuoteUtils {
> public static final String QUOTE = "`";
> /**
> * Quote identifier by default quote `
> * @param identifier
> * @return
> */
> public static String quoteIdentifier(String identifier){
> return QUOTE + identifier + QUOTE;
> }
>
> The solution may be to add a parameter to change this character used for
> quoting.
>
> Just to be sure I made a copy of the original "sqoop" command to "sqoop.orig"
> and replaced the original with a little script of mine (attached).
> This little script removes the "`" character from the original command and
> the call "sqoop.orig" with the cleaned arguments.
> It works, but it is still an ugly workaround.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3857) add parameter to change sql quoting character for sqoop
[
https://issues.apache.org/jira/browse/KYLIN-3857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
XiaoXiang Yu updated KYLIN-3857:
Attachment: image-2019-03-11-11-46-20-664.png
> add parameter to change sql quoting character for sqoop
> ---
>
> Key: KYLIN-3857
> URL: https://issues.apache.org/jira/browse/KYLIN-3857
> Project: Kylin
> Issue Type: Bug
>Reporter: Davide Malagoli
>Assignee: XiaoXiang Yu
>Priority: Major
> Attachments: image-2019-03-11-11-46-05-946.png,
> image-2019-03-11-11-46-14-305.png, image-2019-03-11-11-46-20-664.png,
> image-2019-03-11-11-46-31-966.png, sqoop-error.log, sqoop-script.sh
>
>
> I've set up Kylin to access a SQL server using JDBC and do the ETL step as
> described in
> [http://kylin.apache.org/docs/tutorial/setup_jdbc_datasource.html]
> but when the "Sqoop To Flat Hive Table" runs it terminates with an error
> shown in the attached log.
> It seems that the root cause is the presence of the '`' character in the
> query, which has no meaning for sql server.
>
> My problem seems to related to this piece of code
>
> public class FlatTableSqlQuoteUtils {
> public static final String QUOTE = "`";
> /**
> * Quote identifier by default quote `
> * @param identifier
> * @return
> */
> public static String quoteIdentifier(String identifier){
> return QUOTE + identifier + QUOTE;
> }
>
> The solution may be to add a parameter to change this character used for
> quoting.
>
> Just to be sure I made a copy of the original "sqoop" command to "sqoop.orig"
> and replaced the original with a little script of mine (attached).
> This little script removes the "`" character from the original command and
> the call "sqoop.orig" with the cleaned arguments.
> It works, but it is still an ugly workaround.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3857) add parameter to change sql quoting character for sqoop
[
https://issues.apache.org/jira/browse/KYLIN-3857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
XiaoXiang Yu updated KYLIN-3857:
Attachment: image-2019-03-11-11-46-05-946.png
> add parameter to change sql quoting character for sqoop
> ---
>
> Key: KYLIN-3857
> URL: https://issues.apache.org/jira/browse/KYLIN-3857
> Project: Kylin
> Issue Type: Bug
>Reporter: Davide Malagoli
>Assignee: XiaoXiang Yu
>Priority: Major
> Attachments: image-2019-03-11-11-46-05-946.png,
> image-2019-03-11-11-46-14-305.png, image-2019-03-11-11-46-20-664.png,
> image-2019-03-11-11-46-31-966.png, sqoop-error.log, sqoop-script.sh
>
>
> I've set up Kylin to access a SQL server using JDBC and do the ETL step as
> described in
> [http://kylin.apache.org/docs/tutorial/setup_jdbc_datasource.html]
> but when the "Sqoop To Flat Hive Table" runs it terminates with an error
> shown in the attached log.
> It seems that the root cause is the presence of the '`' character in the
> query, which has no meaning for sql server.
>
> My problem seems to related to this piece of code
>
> public class FlatTableSqlQuoteUtils {
> public static final String QUOTE = "`";
> /**
> * Quote identifier by default quote `
> * @param identifier
> * @return
> */
> public static String quoteIdentifier(String identifier){
> return QUOTE + identifier + QUOTE;
> }
>
> The solution may be to add a parameter to change this character used for
> quoting.
>
> Just to be sure I made a copy of the original "sqoop" command to "sqoop.orig"
> and replaced the original with a little script of mine (attached).
> This little script removes the "`" character from the original command and
> the call "sqoop.orig" with the cleaned arguments.
> It works, but it is still an ugly workaround.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3857) add parameter to change sql quoting character for sqoop
[
https://issues.apache.org/jira/browse/KYLIN-3857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
XiaoXiang Yu updated KYLIN-3857:
Attachment: image-2019-03-11-11-46-31-966.png
> add parameter to change sql quoting character for sqoop
> ---
>
> Key: KYLIN-3857
> URL: https://issues.apache.org/jira/browse/KYLIN-3857
> Project: Kylin
> Issue Type: Bug
>Reporter: Davide Malagoli
>Assignee: XiaoXiang Yu
>Priority: Major
> Attachments: image-2019-03-11-11-46-05-946.png,
> image-2019-03-11-11-46-14-305.png, image-2019-03-11-11-46-20-664.png,
> image-2019-03-11-11-46-31-966.png, sqoop-error.log, sqoop-script.sh
>
>
> I've set up Kylin to access a SQL server using JDBC and do the ETL step as
> described in
> [http://kylin.apache.org/docs/tutorial/setup_jdbc_datasource.html]
> but when the "Sqoop To Flat Hive Table" runs it terminates with an error
> shown in the attached log.
> It seems that the root cause is the presence of the '`' character in the
> query, which has no meaning for sql server.
>
> My problem seems to related to this piece of code
>
> public class FlatTableSqlQuoteUtils {
> public static final String QUOTE = "`";
> /**
> * Quote identifier by default quote `
> * @param identifier
> * @return
> */
> public static String quoteIdentifier(String identifier){
> return QUOTE + identifier + QUOTE;
> }
>
> The solution may be to add a parameter to change this character used for
> quoting.
>
> Just to be sure I made a copy of the original "sqoop" command to "sqoop.orig"
> and replaced the original with a little script of mine (attached).
> This little script removes the "`" character from the original command and
> the call "sqoop.orig" with the cleaned arguments.
> It works, but it is still an ugly workaround.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3857) add parameter to change sql quoting character for sqoop
[
https://issues.apache.org/jira/browse/KYLIN-3857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16789133#comment-16789133
]
XiaoXiang Yu commented on KYLIN-3857:
-
Screenshots in My DEV env (with quote turn on):
!image-2019-03-11-11-46-14-305.png!
!image-2019-03-11-11-46-20-664.png!
!image-2019-03-11-11-46-31-966.png!
> add parameter to change sql quoting character for sqoop
> ---
>
> Key: KYLIN-3857
> URL: https://issues.apache.org/jira/browse/KYLIN-3857
> Project: Kylin
> Issue Type: Bug
>Reporter: Davide Malagoli
>Assignee: XiaoXiang Yu
>Priority: Major
> Attachments: image-2019-03-11-11-46-05-946.png,
> image-2019-03-11-11-46-14-305.png, image-2019-03-11-11-46-20-664.png,
> image-2019-03-11-11-46-31-966.png, sqoop-error.log, sqoop-script.sh
>
>
> I've set up Kylin to access a SQL server using JDBC and do the ETL step as
> described in
> [http://kylin.apache.org/docs/tutorial/setup_jdbc_datasource.html]
> but when the "Sqoop To Flat Hive Table" runs it terminates with an error
> shown in the attached log.
> It seems that the root cause is the presence of the '`' character in the
> query, which has no meaning for sql server.
>
> My problem seems to related to this piece of code
>
> public class FlatTableSqlQuoteUtils {
> public static final String QUOTE = "`";
> /**
> * Quote identifier by default quote `
> * @param identifier
> * @return
> */
> public static String quoteIdentifier(String identifier){
> return QUOTE + identifier + QUOTE;
> }
>
> The solution may be to add a parameter to change this character used for
> quoting.
>
> Just to be sure I made a copy of the original "sqoop" command to "sqoop.orig"
> and replaced the original with a little script of mine (attached).
> This little script removes the "`" character from the original command and
> the call "sqoop.orig" with the cleaned arguments.
> It works, but it is still an ugly workaround.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3857) add parameter to change sql quoting character for sqoop
[
https://issues.apache.org/jira/browse/KYLIN-3857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
XiaoXiang Yu updated KYLIN-3857:
Attachment: image-2019-03-11-11-46-14-305.png
> add parameter to change sql quoting character for sqoop
> ---
>
> Key: KYLIN-3857
> URL: https://issues.apache.org/jira/browse/KYLIN-3857
> Project: Kylin
> Issue Type: Bug
>Reporter: Davide Malagoli
>Assignee: XiaoXiang Yu
>Priority: Major
> Attachments: image-2019-03-11-11-46-05-946.png,
> image-2019-03-11-11-46-14-305.png, image-2019-03-11-11-46-20-664.png,
> image-2019-03-11-11-46-31-966.png, sqoop-error.log, sqoop-script.sh
>
>
> I've set up Kylin to access a SQL server using JDBC and do the ETL step as
> described in
> [http://kylin.apache.org/docs/tutorial/setup_jdbc_datasource.html]
> but when the "Sqoop To Flat Hive Table" runs it terminates with an error
> shown in the attached log.
> It seems that the root cause is the presence of the '`' character in the
> query, which has no meaning for sql server.
>
> My problem seems to related to this piece of code
>
> public class FlatTableSqlQuoteUtils {
> public static final String QUOTE = "`";
> /**
> * Quote identifier by default quote `
> * @param identifier
> * @return
> */
> public static String quoteIdentifier(String identifier){
> return QUOTE + identifier + QUOTE;
> }
>
> The solution may be to add a parameter to change this character used for
> quoting.
>
> Just to be sure I made a copy of the original "sqoop" command to "sqoop.orig"
> and replaced the original with a little script of mine (attached).
> This little script removes the "`" character from the original command and
> the call "sqoop.orig" with the cleaned arguments.
> It works, but it is still an ugly workaround.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3857) add parameter to change sql quoting character for sqoop
[
https://issues.apache.org/jira/browse/KYLIN-3857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16789136#comment-16789136
]
XiaoXiang Yu commented on KYLIN-3857:
-
Screenshots in My DEV env (with quote turn off):
> add parameter to change sql quoting character for sqoop
> ---
>
> Key: KYLIN-3857
> URL: https://issues.apache.org/jira/browse/KYLIN-3857
> Project: Kylin
> Issue Type: Bug
>Reporter: Davide Malagoli
>Assignee: XiaoXiang Yu
>Priority: Major
> Attachments: image-2019-03-11-11-46-05-946.png,
> image-2019-03-11-11-46-14-305.png, image-2019-03-11-11-46-20-664.png,
> image-2019-03-11-11-46-31-966.png, sqoop-error.log, sqoop-script.sh
>
>
> I've set up Kylin to access a SQL server using JDBC and do the ETL step as
> described in
> [http://kylin.apache.org/docs/tutorial/setup_jdbc_datasource.html]
> but when the "Sqoop To Flat Hive Table" runs it terminates with an error
> shown in the attached log.
> It seems that the root cause is the presence of the '`' character in the
> query, which has no meaning for sql server.
>
> My problem seems to related to this piece of code
>
> public class FlatTableSqlQuoteUtils {
> public static final String QUOTE = "`";
> /**
> * Quote identifier by default quote `
> * @param identifier
> * @return
> */
> public static String quoteIdentifier(String identifier){
> return QUOTE + identifier + QUOTE;
> }
>
> The solution may be to add a parameter to change this character used for
> quoting.
>
> Just to be sure I made a copy of the original "sqoop" command to "sqoop.orig"
> and replaced the original with a little script of mine (attached).
> This little script removes the "`" character from the original command and
> the call "sqoop.orig" with the cleaned arguments.
> It works, but it is still an ugly workaround.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3857) add parameter to change sql quoting character for sqoop
[
https://issues.apache.org/jira/browse/KYLIN-3857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16789136#comment-16789136
]
XiaoXiang Yu edited comment on KYLIN-3857 at 3/11/19 4:59 AM:
--
Logs in My DEV env (with quote turn off):
KYLIN [ INFO ] 03-11 12:53:39.114
org.apache.kylin.common.util.BufferedLogger.log(BufferedLogger.java:38) from
Scheduler 1705744661 Job 6d64fe74-ea50-76a6-bbd1-f85210984416-129
> Create and distribute table, cmd:
KYLIN [ INFO ] 03-11 12:53:39.115
org.apache.kylin.common.util.BufferedLogger.log(BufferedLogger.java:38) from
Scheduler 1705744661 Job 6d64fe74-ea50-76a6-bbd1-f85210984416-129
> hive -e "USE lacus;
DROP TABLE IF EXISTS
kylin_intermediate_samplecube_e1c8bdcc_d55b_3807_25e2_b8d2b6fccabe;
CREATE EXTERNAL TABLE IF NOT EXISTS
kylin_intermediate_samplecube_e1c8bdcc_d55b_3807_25e2_b8d2b6fccabe
(
KYLIN_SALES_LSTG_FORMAT_NAME string
,KYLIN_SALES_LSTG_SITE_ID int
,KYLIN_SALES_SELLER_ID bigint
,KYLIN_ACCOUNT_ACCOUNT_COUNTRY string
,KYLIN_SALES_PRICE decimal(19,4)
,KYLIN_SALES_ITEM_COUNT bigint
)
STORED AS SEQUENCEFILE
LOCATION
'hdfs://sandbox.hortonworks.com:8020/kylin/lacus/lacus-kylin_metadata/kylin-6d64fe74-ea50-76a6-bbd1-f85210984416/kylin_intermediate_samplecube_e1c8bdcc_d55b_3807_25e2_b8d2b6fccabe';
ALTER TABLE kylin_intermediate_samplecube_e1c8bdcc_d55b_3807_25e2_b8d2b6fccabe
SET TBLPROPERTIES('auto.purge'='true');
INSERT OVERWRITE TABLE
kylin_intermediate_samplecube_e1c8bdcc_d55b_3807_25e2_b8d2b6fccabe SELECT
KYLIN_SALES.LSTG_FORMAT_NAME as KYLIN_SALES_LSTG_FORMAT_NAME
,KYLIN_SALES.LSTG_SITE_ID as KYLIN_SALES_LSTG_SITE_ID
,KYLIN_SALES.SELLER_ID as KYLIN_SALES_SELLER_ID
,KYLIN_ACCOUNT.ACCOUNT_COUNTRY as KYLIN_ACCOUNT_ACCOUNT_COUNTRY
,KYLIN_SALES.PRICE as KYLIN_SALES_PRICE
,KYLIN_SALES.ITEM_COUNT as KYLIN_SALES_ITEM_COUNT
FROM LACUS.KYLIN_SALES as KYLIN_SALES
LEFT JOIN LACUS.KYLIN_ACCOUNT as KYLIN_ACCOUNT
ON KYLIN_SALES.SELLER_ID = KYLIN_ACCOUNT.ACCOUNT_ID
LEFT JOIN LACUS.KYLIN_COUNTRY as KYLIN_COUNTRY
ON KYLIN_ACCOUNT.ACCOUNT_COUNTRY = KYLIN_COUNTRY.COUNTRY
WHERE 1=1;
" --hiveconf
mapreduce.map.output.compress.codec=org.apache.hadoop.io.compress.SnappyCodec
--hiveconf dfs.replication=2 --hiveconf hive.exec.compress.output=true
KYLIN [ INFO ] 03-11 12:53:39.115
org.apache.kylin.common.util.SSHClient.execCommand(SSHClient.java:262) from
Scheduler 1705744661 Job 6d64fe74-ea50-76a6-bbd1-f85210984416-129
> [root@sandbox] Execute command: hive -e "USE lacus;
DROP TABLE IF EXISTS
kylin_intermediate_samplecube_e1c8bdcc_d55b_3807_25e2_b8d2b6fccabe;
CREATE EXTERNAL TABLE IF NOT EXISTS
kylin_intermediate_samplecube_e1c8bdcc_d55b_3807_25e2_b8d2b6fccabe
(
KYLIN_SALES_LSTG_FORMAT_NAME string
,KYLIN_SALES_LSTG_SITE_ID int
,KYLIN_SALES_SELLER_ID bigint
,KYLIN_ACCOUNT_ACCOUNT_COUNTRY string
,KYLIN_SALES_PRICE decimal(19,4)
,KYLIN_SALES_ITEM_COUNT bigint
)
STORED AS SEQUENCEFILE
LOCATION
'hdfs://sandbox.hortonworks.com:8020/kylin/lacus/lacus-kylin_metadata/kylin-6d64fe74-ea50-76a6-bbd1-f85210984416/kylin_intermediate_samplecube_e1c8bdcc_d55b_3807_25e2_b8d2b6fccabe';
ALTER TABLE kylin_intermediate_samplecube_e1c8bdcc_d55b_3807_25e2_b8d2b6fccabe
SET TBLPROPERTIES('auto.purge'='true');
INSERT OVERWRITE TABLE
kylin_intermediate_samplecube_e1c8bdcc_d55b_3807_25e2_b8d2b6fccabe SELECT
KYLIN_SALES.LSTG_FORMAT_NAME as KYLIN_SALES_LSTG_FORMAT_NAME
,KYLIN_SALES.LSTG_SITE_ID as KYLIN_SALES_LSTG_SITE_ID
,KYLIN_SALES.SELLER_ID as KYLIN_SALES_SELLER_ID
,KYLIN_ACCOUNT.ACCOUNT_COUNTRY as KYLIN_ACCOUNT_ACCOUNT_COUNTRY
,KYLIN_SALES.PRICE as KYLIN_SALES_PRICE
,KYLIN_SALES.ITEM_COUNT as KYLIN_SALES_ITEM_COUNT
FROM LACUS.KYLIN_SALES as KYLIN_SALES
LEFT JOIN LACUS.KYLIN_ACCOUNT as KYLIN_ACCOUNT
ON KYLIN_SALES.SELLER_ID = KYLIN_ACCOUNT.ACCOUNT_ID
LEFT JOIN LACUS.KYLIN_COUNTRY as KYLIN_COUNTRY
ON KYLIN_ACCOUNT.ACCOUNT_COUNTRY = KYLIN_COUNTRY.COUNTRY
WHERE 1=1;
" --hiveconf
mapreduce.map.output.compress.codec=org.apache.hadoop.io.compress.SnappyCodec
--hiveconf dfs.replication=2 --hiveconf hive.exec.compress.output=true
KYLIN [ INFO ] 03-11 12:53:42.237
org.apache.kylin.common.util.BufferedLogger.log(BufferedLogger.java:38) from
Scheduler 1705744661 Job 6d64fe74-ea50-76a6-bbd1-f85210984416-129
> WARNING: Use "yarn jar" to launch YARN applications.
KYLIN [ INFO ] 03-11 12:53:43.241
org.apache.kylin.common.util.BufferedLogger.log(BufferedLogger.java:38) from
Scheduler 1705744661 Job 6d64fe74-ea50-76a6-bbd1-f85210984416-129
>
Logging initialized using configuration in
file:/etc/hive/2.4.0.0-169/0/hive-log4j.properties
KYLIN [ INFO ] 03-11 12:56:07.045
org.apache.kylin.common.util.SSHClient.execCommand(SSHClient.java:262) from
Scheduler 1705744661 Job 6d64fe74-ea50-76a6-bbd1-f85210984416-221
> [root@sandbox] Execute command: hive -e "USE lacus;
DROP TABLE IF EXISTS
lacus.kylin_intermediate_samplecube_e1c8bdcc_d55b_3807_25e2_b8d2b6fccabe;
" --hiveconf
mapred
[jira] [Comment Edited] (KYLIN-3857) add parameter to change sql quoting character for sqoop
[
https://issues.apache.org/jira/browse/KYLIN-3857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16789133#comment-16789133
]
XiaoXiang Yu edited comment on KYLIN-3857 at 3/11/19 5:00 AM:
--
Use kylin.source.hive.enable.quote=false in kylin.properties to turn off quote.
Screenshots in My DEV env (with quote turn on):
!image-2019-03-11-11-46-14-305.png!
!image-2019-03-11-11-46-20-664.png!
!image-2019-03-11-11-46-31-966.png!
was (Author: hit_lacus):
Screenshots in My DEV env (with quote turn on):
!image-2019-03-11-11-46-14-305.png!
!image-2019-03-11-11-46-20-664.png!
!image-2019-03-11-11-46-31-966.png!
> add parameter to change sql quoting character for sqoop
> ---
>
> Key: KYLIN-3857
> URL: https://issues.apache.org/jira/browse/KYLIN-3857
> Project: Kylin
> Issue Type: Bug
>Reporter: Davide Malagoli
>Assignee: XiaoXiang Yu
>Priority: Major
> Attachments: image-2019-03-11-11-46-05-946.png,
> image-2019-03-11-11-46-14-305.png, image-2019-03-11-11-46-20-664.png,
> image-2019-03-11-11-46-31-966.png, sqoop-error.log, sqoop-script.sh
>
>
> I've set up Kylin to access a SQL server using JDBC and do the ETL step as
> described in
> [http://kylin.apache.org/docs/tutorial/setup_jdbc_datasource.html]
> but when the "Sqoop To Flat Hive Table" runs it terminates with an error
> shown in the attached log.
> It seems that the root cause is the presence of the '`' character in the
> query, which has no meaning for sql server.
>
> My problem seems to related to this piece of code
>
> public class FlatTableSqlQuoteUtils {
> public static final String QUOTE = "`";
> /**
> * Quote identifier by default quote `
> * @param identifier
> * @return
> */
> public static String quoteIdentifier(String identifier){
> return QUOTE + identifier + QUOTE;
> }
>
> The solution may be to add a parameter to change this character used for
> quoting.
>
> Just to be sure I made a copy of the original "sqoop" command to "sqoop.orig"
> and replaced the original with a little script of mine (attached).
> This little script removes the "`" character from the original command and
> the call "sqoop.orig" with the cleaned arguments.
> It works, but it is still an ugly workaround.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3857) add parameter to change sql quoting character for sqoop
[
https://issues.apache.org/jira/browse/KYLIN-3857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16789160#comment-16789160
]
XiaoXiang Yu commented on KYLIN-3857:
-
Import Microsoft SQL Server as source by Data Source SDK can be refer to:
https://github.com/hit-lacus/hit-lacus.github.io/issues/32#issuecomment-469134060
> add parameter to change sql quoting character for sqoop
> ---
>
> Key: KYLIN-3857
> URL: https://issues.apache.org/jira/browse/KYLIN-3857
> Project: Kylin
> Issue Type: Bug
>Reporter: Davide Malagoli
>Assignee: XiaoXiang Yu
>Priority: Major
> Attachments: image-2019-03-11-11-46-05-946.png,
> image-2019-03-11-11-46-14-305.png, image-2019-03-11-11-46-20-664.png,
> image-2019-03-11-11-46-31-966.png, sqoop-error.log, sqoop-script.sh
>
>
> I've set up Kylin to access a SQL server using JDBC and do the ETL step as
> described in
> [http://kylin.apache.org/docs/tutorial/setup_jdbc_datasource.html]
> but when the "Sqoop To Flat Hive Table" runs it terminates with an error
> shown in the attached log.
> It seems that the root cause is the presence of the '`' character in the
> query, which has no meaning for sql server.
>
> My problem seems to related to this piece of code
>
> public class FlatTableSqlQuoteUtils {
> public static final String QUOTE = "`";
> /**
> * Quote identifier by default quote `
> * @param identifier
> * @return
> */
> public static String quoteIdentifier(String identifier){
> return QUOTE + identifier + QUOTE;
> }
>
> The solution may be to add a parameter to change this character used for
> quoting.
>
> Just to be sure I made a copy of the original "sqoop" command to "sqoop.orig"
> and replaced the original with a little script of mine (attached).
> This little script removes the "`" character from the original command and
> the call "sqoop.orig" with the cleaned arguments.
> It works, but it is still an ugly workaround.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3857) add parameter to change sql quoting character for sqoop
[
https://issues.apache.org/jira/browse/KYLIN-3857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16789133#comment-16789133
]
XiaoXiang Yu edited comment on KYLIN-3857 at 3/11/19 5:06 AM:
--
Use kylin.source.hive.enable.quote=false in kylin.properties to turn off quote.
(default value is true)
Screenshots in My DEV env (with quote turn on):
!image-2019-03-11-11-46-14-305.png!
!image-2019-03-11-11-46-20-664.png!
!image-2019-03-11-11-46-31-966.png!
was (Author: hit_lacus):
Use kylin.source.hive.enable.quote=false in kylin.properties to turn off quote.
Screenshots in My DEV env (with quote turn on):
!image-2019-03-11-11-46-14-305.png!
!image-2019-03-11-11-46-20-664.png!
!image-2019-03-11-11-46-31-966.png!
> add parameter to change sql quoting character for sqoop
> ---
>
> Key: KYLIN-3857
> URL: https://issues.apache.org/jira/browse/KYLIN-3857
> Project: Kylin
> Issue Type: Bug
>Reporter: Davide Malagoli
>Assignee: XiaoXiang Yu
>Priority: Major
> Attachments: image-2019-03-11-11-46-05-946.png,
> image-2019-03-11-11-46-14-305.png, image-2019-03-11-11-46-20-664.png,
> image-2019-03-11-11-46-31-966.png, sqoop-error.log, sqoop-script.sh
>
>
> I've set up Kylin to access a SQL server using JDBC and do the ETL step as
> described in
> [http://kylin.apache.org/docs/tutorial/setup_jdbc_datasource.html]
> but when the "Sqoop To Flat Hive Table" runs it terminates with an error
> shown in the attached log.
> It seems that the root cause is the presence of the '`' character in the
> query, which has no meaning for sql server.
>
> My problem seems to related to this piece of code
>
> public class FlatTableSqlQuoteUtils {
> public static final String QUOTE = "`";
> /**
> * Quote identifier by default quote `
> * @param identifier
> * @return
> */
> public static String quoteIdentifier(String identifier){
> return QUOTE + identifier + QUOTE;
> }
>
> The solution may be to add a parameter to change this character used for
> quoting.
>
> Just to be sure I made a copy of the original "sqoop" command to "sqoop.orig"
> and replaced the original with a little script of mine (attached).
> This little script removes the "`" character from the original command and
> the call "sqoop.orig" with the cleaned arguments.
> It works, but it is still an ugly workaround.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3857) add parameter to change sql quoting character for sqoop
[
https://issues.apache.org/jira/browse/KYLIN-3857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
XiaoXiang Yu updated KYLIN-3857:
Fix Version/s: v2.6.2
> add parameter to change sql quoting character for sqoop
> ---
>
> Key: KYLIN-3857
> URL: https://issues.apache.org/jira/browse/KYLIN-3857
> Project: Kylin
> Issue Type: Bug
>Reporter: Davide Malagoli
>Assignee: XiaoXiang Yu
>Priority: Major
> Fix For: v2.6.2
>
> Attachments: image-2019-03-11-11-46-05-946.png,
> image-2019-03-11-11-46-14-305.png, image-2019-03-11-11-46-20-664.png,
> image-2019-03-11-11-46-31-966.png, sqoop-error.log, sqoop-script.sh
>
>
> I've set up Kylin to access a SQL server using JDBC and do the ETL step as
> described in
> [http://kylin.apache.org/docs/tutorial/setup_jdbc_datasource.html]
> but when the "Sqoop To Flat Hive Table" runs it terminates with an error
> shown in the attached log.
> It seems that the root cause is the presence of the '`' character in the
> query, which has no meaning for sql server.
>
> My problem seems to related to this piece of code
>
> public class FlatTableSqlQuoteUtils {
> public static final String QUOTE = "`";
> /**
> * Quote identifier by default quote `
> * @param identifier
> * @return
> */
> public static String quoteIdentifier(String identifier){
> return QUOTE + identifier + QUOTE;
> }
>
> The solution may be to add a parameter to change this character used for
> quoting.
>
> Just to be sure I made a copy of the original "sqoop" command to "sqoop.orig"
> and replaced the original with a little script of mine (attached).
> This little script removes the "`" character from the original command and
> the call "sqoop.orig" with the cleaned arguments.
> It works, but it is still an ugly workaround.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3905) Enable shrunken dictionary default
[ https://issues.apache.org/jira/browse/KYLIN-3905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3905: Description: When using bitmap measure on a large cardinality column(require global dictionaty), build base cuboid step need frequent cache swap so it cannot finished within a reasonable period. When shrunken dictionary enabled, a new step will be added to build separated dictionary for each `InputSplit`, Mapper of **BuildBaseCuboid** step only has to fetch a smaller dictionary for itself, instead of a larger global dictionary. It will reduce cache swap and make **BuildBaseCuboid** step run as quicker as possible. was: When using bitmap measure on a large cardinality column(require global dictionaty), build base cuboid step need frequent cache swap so it cannot finished within a reasonable period. This patch add a new step to build separated dictionary for each `InputSplit`, Mapper of **BuildBaseCuboid** step only has to fetch a smaller dictionary for itself, instead of a larger global dictionary. It will reduce cache swap and make **BuildBaseCuboid** step run as quicker as possible. > Enable shrunken dictionary default > -- > > Key: KYLIN-3905 > URL: https://issues.apache.org/jira/browse/KYLIN-3905 > Project: Kylin > Issue Type: Improvement >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Minor > > When using bitmap measure on a large cardinality column(require global > dictionaty), build base cuboid step need frequent cache swap so it cannot > finished within a reasonable period. > When shrunken dictionary enabled, a new step will be added to build separated > dictionary for each `InputSplit`, Mapper of **BuildBaseCuboid** step only has > to fetch a smaller dictionary for itself, instead of a larger global > dictionary. It will reduce cache swap and make **BuildBaseCuboid** step run > as quicker as possible. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3905) Enable shrunken dictionary default
XiaoXiang Yu created KYLIN-3905: --- Summary: Enable shrunken dictionary default Key: KYLIN-3905 URL: https://issues.apache.org/jira/browse/KYLIN-3905 Project: Kylin Issue Type: Improvement Reporter: XiaoXiang Yu Assignee: XiaoXiang Yu -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3905) Enable shrunken dictionary default
[
https://issues.apache.org/jira/browse/KYLIN-3905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16800378#comment-16800378
]
XiaoXiang Yu commented on KYLIN-3905:
-
h2. Comparsion and Summary
{quote} * CDH cluster with 56 vcore and 110GB Memory
* Fact Table 153326740 rows
* Build cube with three bitmap count-distinct measure, one column's
cardinality is 55200325{quote}
h4. Without ShrunkenDict (Cannot completed)
* Build basecuboid cannot completed
h4. With ShrunkenDict (Completed)
* New added step build ShrunkenDict for each map task
[!https://user-images.githubusercontent.com/14030549/54500164-cd2efd00-4954-11e9-85a1-8ae5e67063c7.png|width=355!|https://user-images.githubusercontent.com/14030549/54500164-cd2efd00-4954-11e9-85a1-8ae5e67063c7.png]
* MapReduce Job Stats
[!https://user-images.githubusercontent.com/14030549/54500186-12532f00-4955-11e9-9d61-202f92ca54e5.png|width=1151!|https://user-images.githubusercontent.com/14030549/54500186-12532f00-4955-11e9-9d61-202f92ca54e5.png]
* ShrunkenDict in HDFS
[!https://user-images.githubusercontent.com/14030549/54341171-286ea000-4674-11e9-8d99-560e94d37cc4.png!|https://user-images.githubusercontent.com/14030549/54341171-286ea000-4674-11e9-8d99-560e94d37cc4.png]
[!https://user-images.githubusercontent.com/14030549/54341626-3a9d0e00-4675-11e9-962c-c6a805f3208f.png!|https://user-images.githubusercontent.com/14030549/54341626-3a9d0e00-4675-11e9-962c-c6a805f3208f.png]
> Enable shrunken dictionary default
> --
>
> Key: KYLIN-3905
> URL: https://issues.apache.org/jira/browse/KYLIN-3905
> Project: Kylin
> Issue Type: Improvement
>Reporter: XiaoXiang Yu
>Assignee: XiaoXiang Yu
>Priority: Minor
>
> When using bitmap measure on a large cardinality column(require global
> dictionaty), build base cuboid step need frequent cache swap so it cannot
> finished within a reasonable period.
> When shrunken dictionary enabled, a new step will be added to build separated
> dictionary for each `InputSplit`, Mapper of **BuildBaseCuboid** step only has
> to fetch a smaller dictionary for itself, instead of a larger global
> dictionary. It will reduce cache swap and make **BuildBaseCuboid** step run
> as quicker as possible.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3905) Enable shrunken dictionary default
[ https://issues.apache.org/jira/browse/KYLIN-3905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3905: Description: When using bitmap measure on a large cardinality column(require global dictionaty), build base cuboid step need frequent cache swap so it cannot finished within a reasonable period. This patch add a new step to build separated dictionary for each `InputSplit`, Mapper of **BuildBaseCuboid** step only has to fetch a smaller dictionary for itself, instead of a larger global dictionary. It will reduce cache swap and make **BuildBaseCuboid** step run as quicker as possible. > Enable shrunken dictionary default > -- > > Key: KYLIN-3905 > URL: https://issues.apache.org/jira/browse/KYLIN-3905 > Project: Kylin > Issue Type: Improvement >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Minor > > When using bitmap measure on a large cardinality column(require global > dictionaty), build base cuboid step need frequent cache swap so it cannot > finished within a reasonable period. > This patch add a new step to build separated dictionary for each > `InputSplit`, Mapper of **BuildBaseCuboid** step only has to fetch a smaller > dictionary for itself, instead of a larger global dictionary. It will reduce > cache swap and make **BuildBaseCuboid** step run as quicker as possible. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3905) Enable shrunken dictionary default
[
https://issues.apache.org/jira/browse/KYLIN-3905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16800378#comment-16800378
]
XiaoXiang Yu edited comment on KYLIN-3905 at 3/25/19 3:28 AM:
--
h2. Comparsion and Summary
{quote} * CDH cluster with 56 vcore and 110GB Memory
* Fact Table 153326740 rows
* Build cube with three bitmap count-distinct measure, one column's
cardinality is 55200325{quote}
h4. Without ShrunkenDict (Cannot completed)
* Build basecuboid cannot completed
!image-2019-03-25-11-26-59-198.png!
h4. With ShrunkenDict (Completed)
* New added step build ShrunkenDict for each map task
!image-2019-03-25-11-27-26-149.png!
* MapReduce Job Stats
!image-2019-03-25-11-27-46-175.png![!https://user-images.githubusercontent.com/14030549/54500186-12532f00-4955-11e9-9d61-202f92ca54e5.png|width=1151!|https://user-images.githubusercontent.com/14030549/54500186-12532f00-4955-11e9-9d61-202f92ca54e5.png]
!image-2019-03-25-11-28-14-256.png!
[!https://user-images.githubusercontent.com/14030549/54341171-286ea000-4674-11e9-8d99-560e94d37cc4.png!|https://user-images.githubusercontent.com/14030549/54341171-286ea000-4674-11e9-8d99-560e94d37cc4.png]
[!https://user-images.githubusercontent.com/14030549/54341626-3a9d0e00-4675-11e9-962c-c6a805f3208f.png!|https://user-images.githubusercontent.com/14030549/54341626-3a9d0e00-4675-11e9-962c-c6a805f3208f.png]
was (Author: hit_lacus):
h2. Comparsion and Summary
{quote} * CDH cluster with 56 vcore and 110GB Memory
* Fact Table 153326740 rows
* Build cube with three bitmap count-distinct measure, one column's
cardinality is 55200325{quote}
h4. Without ShrunkenDict (Cannot completed)
* Build basecuboid cannot completed
h4. With ShrunkenDict (Completed)
* New added step build ShrunkenDict for each map task
[!https://user-images.githubusercontent.com/14030549/54500164-cd2efd00-4954-11e9-85a1-8ae5e67063c7.png|width=355!|https://user-images.githubusercontent.com/14030549/54500164-cd2efd00-4954-11e9-85a1-8ae5e67063c7.png]
* MapReduce Job Stats
[!https://user-images.githubusercontent.com/14030549/54500186-12532f00-4955-11e9-9d61-202f92ca54e5.png|width=1151!|https://user-images.githubusercontent.com/14030549/54500186-12532f00-4955-11e9-9d61-202f92ca54e5.png]
* ShrunkenDict in HDFS
[!https://user-images.githubusercontent.com/14030549/54341171-286ea000-4674-11e9-8d99-560e94d37cc4.png!|https://user-images.githubusercontent.com/14030549/54341171-286ea000-4674-11e9-8d99-560e94d37cc4.png]
[!https://user-images.githubusercontent.com/14030549/54341626-3a9d0e00-4675-11e9-962c-c6a805f3208f.png!|https://user-images.githubusercontent.com/14030549/54341626-3a9d0e00-4675-11e9-962c-c6a805f3208f.png]
> Enable shrunken dictionary default
> --
>
> Key: KYLIN-3905
> URL: https://issues.apache.org/jira/browse/KYLIN-3905
> Project: Kylin
> Issue Type: Improvement
>Reporter: XiaoXiang Yu
>Assignee: XiaoXiang Yu
>Priority: Minor
> Attachments: image-2019-03-25-11-26-59-198.png,
> image-2019-03-25-11-27-26-149.png, image-2019-03-25-11-27-46-175.png,
> image-2019-03-25-11-28-14-256.png
>
>
> When using bitmap measure on a large cardinality column(require global
> dictionaty), build base cuboid step need frequent cache swap so it cannot
> finished within a reasonable period.
> When shrunken dictionary enabled, a new step will be added to build separated
> dictionary for each `InputSplit`, Mapper of **BuildBaseCuboid** step only has
> to fetch a smaller dictionary for itself, instead of a larger global
> dictionary. It will reduce cache swap and make **BuildBaseCuboid** step run
> as quicker as possible.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3905) Enable shrunken dictionary default
[ https://issues.apache.org/jira/browse/KYLIN-3905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16800379#comment-16800379 ] XiaoXiang Yu commented on KYLIN-3905: - h3. Return time statistics of global dictionary(without Shrunken Dictionary) !image-2019-03-25-11-29-19-383.png! > Enable shrunken dictionary default > -- > > Key: KYLIN-3905 > URL: https://issues.apache.org/jira/browse/KYLIN-3905 > Project: Kylin > Issue Type: Improvement >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Minor > Attachments: image-2019-03-25-11-26-59-198.png, > image-2019-03-25-11-27-26-149.png, image-2019-03-25-11-27-46-175.png, > image-2019-03-25-11-28-14-256.png, image-2019-03-25-11-29-19-383.png > > > When using bitmap measure on a large cardinality column(require global > dictionaty), build base cuboid step need frequent cache swap so it cannot > finished within a reasonable period. > When shrunken dictionary enabled, a new step will be added to build separated > dictionary for each `InputSplit`, Mapper of **BuildBaseCuboid** step only has > to fetch a smaller dictionary for itself, instead of a larger global > dictionary. It will reduce cache swap and make **BuildBaseCuboid** step run > as quicker as possible. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3905) Enable shrunken dictionary default
[ https://issues.apache.org/jira/browse/KYLIN-3905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3905: Attachment: image-2019-03-25-11-29-19-383.png > Enable shrunken dictionary default > -- > > Key: KYLIN-3905 > URL: https://issues.apache.org/jira/browse/KYLIN-3905 > Project: Kylin > Issue Type: Improvement >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Minor > Attachments: image-2019-03-25-11-26-59-198.png, > image-2019-03-25-11-27-26-149.png, image-2019-03-25-11-27-46-175.png, > image-2019-03-25-11-28-14-256.png, image-2019-03-25-11-29-19-383.png > > > When using bitmap measure on a large cardinality column(require global > dictionaty), build base cuboid step need frequent cache swap so it cannot > finished within a reasonable period. > When shrunken dictionary enabled, a new step will be added to build separated > dictionary for each `InputSplit`, Mapper of **BuildBaseCuboid** step only has > to fetch a smaller dictionary for itself, instead of a larger global > dictionary. It will reduce cache swap and make **BuildBaseCuboid** step run > as quicker as possible. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3905) Enable shrunken dictionary default
[ https://issues.apache.org/jira/browse/KYLIN-3905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3905: Fix Version/s: v2.6.2 Component/s: Measure - Count Distinct > Enable shrunken dictionary default > -- > > Key: KYLIN-3905 > URL: https://issues.apache.org/jira/browse/KYLIN-3905 > Project: Kylin > Issue Type: Improvement > Components: Measure - Count Distinct >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Minor > Fix For: v2.6.2 > > Attachments: image-2019-03-25-11-26-59-198.png, > image-2019-03-25-11-27-26-149.png, image-2019-03-25-11-27-46-175.png, > image-2019-03-25-11-28-14-256.png, image-2019-03-25-11-29-19-383.png > > > When using bitmap measure on a large cardinality column(require global > dictionaty), build base cuboid step need frequent cache swap so it cannot > finished within a reasonable period. > When shrunken dictionary enabled, a new step will be added to build separated > dictionary for each `InputSplit`, Mapper of **BuildBaseCuboid** step only has > to fetch a smaller dictionary for itself, instead of a larger global > dictionary. It will reduce cache swap and make **BuildBaseCuboid** step run > as quicker as possible. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3905) Enable shrunken dictionary default
[
https://issues.apache.org/jira/browse/KYLIN-3905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16800378#comment-16800378
]
XiaoXiang Yu edited comment on KYLIN-3905 at 3/25/19 3:35 AM:
--
h2. Comparsion and Summary
{quote} * CDH cluster with 56 vcore and 110GB Memory
* Fact Table 153326740 rows
* Build cube with three bitmap count-distinct measure, one column's
cardinality is 55200325{quote}
h4. Without ShrunkenDict (Cannot completed)
* Build basecuboid cannot completed
!image-2019-03-25-11-26-59-198.png!
h4. With ShrunkenDict (Completed)
* MapReduce Job Stats
[!https://user-images.githubusercontent.com/14030549/54500186-12532f00-4955-11e9-9d61-202f92ca54e5.png|width=1151!|https://user-images.githubusercontent.com/14030549/54500186-12532f00-4955-11e9-9d61-202f92ca54e5.png]
!image-2019-03-25-11-28-14-256.png!
was (Author: hit_lacus):
h2. Comparsion and Summary
{quote} * CDH cluster with 56 vcore and 110GB Memory
* Fact Table 153326740 rows
* Build cube with three bitmap count-distinct measure, one column's
cardinality is 55200325{quote}
h4. Without ShrunkenDict (Cannot completed)
* Build basecuboid cannot completed
!image-2019-03-25-11-26-59-198.png!
h4. With ShrunkenDict (Completed)
* New added step build ShrunkenDict for each map task
!image-2019-03-25-11-27-26-149.png!
* MapReduce Job Stats
!image-2019-03-25-11-27-46-175.png![!https://user-images.githubusercontent.com/14030549/54500186-12532f00-4955-11e9-9d61-202f92ca54e5.png|width=1151!|https://user-images.githubusercontent.com/14030549/54500186-12532f00-4955-11e9-9d61-202f92ca54e5.png]
!image-2019-03-25-11-28-14-256.png!
[!https://user-images.githubusercontent.com/14030549/54341171-286ea000-4674-11e9-8d99-560e94d37cc4.png!|https://user-images.githubusercontent.com/14030549/54341171-286ea000-4674-11e9-8d99-560e94d37cc4.png]
[!https://user-images.githubusercontent.com/14030549/54341626-3a9d0e00-4675-11e9-962c-c6a805f3208f.png!|https://user-images.githubusercontent.com/14030549/54341626-3a9d0e00-4675-11e9-962c-c6a805f3208f.png]
> Enable shrunken dictionary default
> --
>
> Key: KYLIN-3905
> URL: https://issues.apache.org/jira/browse/KYLIN-3905
> Project: Kylin
> Issue Type: Improvement
> Components: Measure - Count Distinct
>Reporter: XiaoXiang Yu
>Assignee: XiaoXiang Yu
>Priority: Minor
> Fix For: v2.6.2
>
> Attachments: image-2019-03-25-11-26-59-198.png,
> image-2019-03-25-11-27-26-149.png, image-2019-03-25-11-27-46-175.png,
> image-2019-03-25-11-28-14-256.png, image-2019-03-25-11-29-19-383.png
>
>
> When using bitmap measure on a large cardinality column(require global
> dictionaty), build base cuboid step need frequent cache swap so it cannot
> finished within a reasonable period.
> When shrunken dictionary enabled, a new step will be added to build separated
> dictionary for each `InputSplit`, Mapper of **BuildBaseCuboid** step only has
> to fetch a smaller dictionary for itself, instead of a larger global
> dictionary. It will reduce cache swap and make **BuildBaseCuboid** step run
> as quicker as possible.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3905) Enable shrunken dictionary default
[ https://issues.apache.org/jira/browse/KYLIN-3905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3905: Description: In dev mail list's discussion, I suggest to enable shrunken dictionary by default, and received some dev's aggrement. When using bitmap measure on a large cardinality column(require global dictionaty), build base cuboid step need frequent cache swap so it cannot finished within a reasonable period. When shrunken dictionary enabled, a new step will be added to build separated dictionary for each `InputSplit`, Mapper of **BuildBaseCuboid** step only has to fetch a smaller dictionary for itself, instead of a larger global dictionary. It will reduce cache swap and make **BuildBaseCuboid** step run as quicker as possible. http://mail-archives.apache.org/mod_mbox/kylin-dev//201903.mbox/%3c62efcb72-b235-4fc3-9add-0fc510d97...@kyligence.io%3e was: In dev mail list's discussion, I suggest to enable http://mail-archives.apache.org/mod_mbox/kylin-dev//201903.mbox/%3c62efcb72-b235-4fc3-9add-0fc510d97...@kyligence.io%3e When using bitmap measure on a large cardinality column(require global dictionaty), build base cuboid step need frequent cache swap so it cannot finished within a reasonable period. When shrunken dictionary enabled, a new step will be added to build separated dictionary for each `InputSplit`, Mapper of **BuildBaseCuboid** step only has to fetch a smaller dictionary for itself, instead of a larger global dictionary. It will reduce cache swap and make **BuildBaseCuboid** step run as quicker as possible. > Enable shrunken dictionary default > -- > > Key: KYLIN-3905 > URL: https://issues.apache.org/jira/browse/KYLIN-3905 > Project: Kylin > Issue Type: Improvement > Components: Measure - Count Distinct >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Minor > Fix For: v2.6.2 > > Attachments: image-2019-03-25-11-26-59-198.png, > image-2019-03-25-11-27-26-149.png, image-2019-03-25-11-27-46-175.png, > image-2019-03-25-11-28-14-256.png, image-2019-03-25-11-29-19-383.png > > > In dev mail list's discussion, I suggest to enable shrunken dictionary by > default, and received some dev's aggrement. > When using bitmap measure on a large cardinality column(require global > dictionaty), build base cuboid step need frequent cache swap so it cannot > finished within a reasonable period. > When shrunken dictionary enabled, a new step will be added to build > separated dictionary for each `InputSplit`, Mapper of **BuildBaseCuboid** > step only has to fetch a smaller dictionary for itself, instead of a larger > global dictionary. It will reduce cache swap and make **BuildBaseCuboid** > step run as quicker as possible. > > http://mail-archives.apache.org/mod_mbox/kylin-dev//201903.mbox/%3c62efcb72-b235-4fc3-9add-0fc510d97...@kyligence.io%3e -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3905) Enable shrunken dictionary default
[ https://issues.apache.org/jira/browse/KYLIN-3905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16800379#comment-16800379 ] XiaoXiang Yu edited comment on KYLIN-3905 at 3/25/19 3:42 AM: -- h3. Return time statistics of global dictionary(without Shrunken Dictionary) In following screenshot, we can find that cache swap occupy most duration when build base cuboid related to Bitmap measure. !image-2019-03-25-11-29-19-383.png! was (Author: hit_lacus): h3. Return time statistics of global dictionary(without Shrunken Dictionary) !image-2019-03-25-11-29-19-383.png! > Enable shrunken dictionary default > -- > > Key: KYLIN-3905 > URL: https://issues.apache.org/jira/browse/KYLIN-3905 > Project: Kylin > Issue Type: Improvement > Components: Measure - Count Distinct >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Minor > Fix For: v2.6.2 > > Attachments: image-2019-03-25-11-26-59-198.png, > image-2019-03-25-11-27-26-149.png, image-2019-03-25-11-27-46-175.png, > image-2019-03-25-11-28-14-256.png, image-2019-03-25-11-29-19-383.png > > > In dev mail list's discussion, I suggest to enable shrunken dictionary by > default, and received some dev's aggrement. > When using bitmap measure on a large cardinality column(require global > dictionaty), build base cuboid step need frequent cache swap so it cannot > finished within a reasonable period. > When shrunken dictionary enabled, a new step will be added to build > separated dictionary for each `InputSplit`, Mapper of **BuildBaseCuboid** > step only has to fetch a smaller dictionary for itself, instead of a larger > global dictionary. It will reduce cache swap and make **BuildBaseCuboid** > step run as quicker as possible. > > http://mail-archives.apache.org/mod_mbox/kylin-dev//201903.mbox/%3c62efcb72-b235-4fc3-9add-0fc510d97...@kyligence.io%3e -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3905) Enable shrunken dictionary default
[ https://issues.apache.org/jira/browse/KYLIN-3905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3905: Description: In dev mail list's discussion, I suggest to enable http://mail-archives.apache.org/mod_mbox/kylin-dev//201903.mbox/%3c62efcb72-b235-4fc3-9add-0fc510d97...@kyligence.io%3e When using bitmap measure on a large cardinality column(require global dictionaty), build base cuboid step need frequent cache swap so it cannot finished within a reasonable period. When shrunken dictionary enabled, a new step will be added to build separated dictionary for each `InputSplit`, Mapper of **BuildBaseCuboid** step only has to fetch a smaller dictionary for itself, instead of a larger global dictionary. It will reduce cache swap and make **BuildBaseCuboid** step run as quicker as possible. was: When using bitmap measure on a large cardinality column(require global dictionaty), build base cuboid step need frequent cache swap so it cannot finished within a reasonable period. When shrunken dictionary enabled, a new step will be added to build separated dictionary for each `InputSplit`, Mapper of **BuildBaseCuboid** step only has to fetch a smaller dictionary for itself, instead of a larger global dictionary. It will reduce cache swap and make **BuildBaseCuboid** step run as quicker as possible. > Enable shrunken dictionary default > -- > > Key: KYLIN-3905 > URL: https://issues.apache.org/jira/browse/KYLIN-3905 > Project: Kylin > Issue Type: Improvement > Components: Measure - Count Distinct >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Minor > Fix For: v2.6.2 > > Attachments: image-2019-03-25-11-26-59-198.png, > image-2019-03-25-11-27-26-149.png, image-2019-03-25-11-27-46-175.png, > image-2019-03-25-11-28-14-256.png, image-2019-03-25-11-29-19-383.png > > > In dev mail list's discussion, I suggest to enable > > http://mail-archives.apache.org/mod_mbox/kylin-dev//201903.mbox/%3c62efcb72-b235-4fc3-9add-0fc510d97...@kyligence.io%3e > > When using bitmap measure on a large cardinality column(require global > dictionaty), build base cuboid step need frequent cache swap so it cannot > finished within a reasonable period. > When shrunken dictionary enabled, a new step will be added to build > separated dictionary for each `InputSplit`, Mapper of **BuildBaseCuboid** > step only has to fetch a smaller dictionary for itself, instead of a larger > global dictionary. It will reduce cache swap and make **BuildBaseCuboid** > step run as quicker as possible. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-3792) Reassign's inconsistent state need automation recovery
[ https://issues.apache.org/jira/browse/KYLIN-3792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu closed KYLIN-3792. --- Resolution: Won't Do > Reassign's inconsistent state need automation recovery > -- > > Key: KYLIN-3792 > URL: https://issues.apache.org/jira/browse/KYLIN-3792 > Project: Kylin > Issue Type: Improvement > Components: Real-time Streaming >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Labels: reassign, won't-fix > Original Estimate: 96h > Remaining Estimate: 96h > > Realtime streaming has a reassign/rebalance operation which may cause some > inconsistent state which is difficult to recover. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3792) Reassign's inconsistent state need automation recovery
[ https://issues.apache.org/jira/browse/KYLIN-3792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3792: Labels: reassign won't-fix (was: reassign) > Reassign's inconsistent state need automation recovery > -- > > Key: KYLIN-3792 > URL: https://issues.apache.org/jira/browse/KYLIN-3792 > Project: Kylin > Issue Type: Improvement > Components: Real-time Streaming >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Labels: reassign, won't-fix > Original Estimate: 96h > Remaining Estimate: 96h > > Realtime streaming has a reassign/rebalance operation which may cause some > inconsistent state which is difficult to recover. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-3711) Clean unused/duplicated code
[ https://issues.apache.org/jira/browse/KYLIN-3711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu closed KYLIN-3711. --- Resolution: Not A Problem > Clean unused/duplicated code > > > Key: KYLIN-3711 > URL: https://issues.apache.org/jira/browse/KYLIN-3711 > Project: Kylin > Issue Type: Improvement >Affects Versions: Future >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Minor > Labels: Refactoring > > I am reading source code and found some unused class, such as > +org.apache.kylin.rest.broadcaster.BroadcasterReceiveServlet+ , this class is > created at 2015 and seems do nothing now(it should be replcaed by > +org.apache.kylin.rest.controller.CacheController+). > > So someone should commented it with "duplicated" or simply delete > it.[BroadcasterReceiveServlet|https://github.com/apache/kylin/blob/kylin-2.5.0/server-base/src/main/java/org/apache/kylin/rest/broadcaster/BroadcasterReceiveServlet.java] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3942) Rea-time OLAP don't support multi-level json event
XiaoXiang Yu created KYLIN-3942:
---
Summary: Rea-time OLAP don't support multi-level json event
Key: KYLIN-3942
URL: https://issues.apache.org/jira/browse/KYLIN-3942
Project: Kylin
Issue Type: Bug
Components: Real-time Streaming
Affects Versions: v3.0.0-alpha
Reporter: XiaoXiang Yu
Assignee: XiaoXiang Yu
Currently real-time OLAP didn't support multi-level json event.
For example,if I have a kafka multi-level json event like this:
{quote}{"country":"JAPAN","amount":13.075058425023922,"qty":8,"currency":"USD","order_time":1554801950882,"category":"ELECTRONIC","device":"Andriod","user":\{"gender":"Female","id":"7a0cfa5e-bbaa-79ef-1a38-e06f02c85fcb","first_name":"unknown","age":16}}
{quote}
Receiver will throw exception like this and discard that event:
{quote}2019-04-09 09:46:09,878 ERROR [StreamingV2Cube_channel]
kafka.TimedJsonStreamParser:107 : error
com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot deserialize
instance of `java.lang.String` out of START_OBJECT token
at [Source:
(String)"\{"country":"US","amount":14.498498222823619,"qty":1,"currency":"USD","order_time":1554803169876,"category":"Other","device":"Other","user":{"gender":"Female","id":"0736b41a-9ae7-9b4a-a124-f74436d3eb41","first_name":"unknown","age":26}}";
line: 1, column: 140] (through reference chain: java.util.HashMap["user"])
at
com.fasterxml.jackson.databind.exc.MismatchedInputException.from(MismatchedInputException.java:63)
at
com.fasterxml.jackson.databind.DeserializationContext.reportInputMismatch(DeserializationContext.java:1342)
at
com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1138)
at
com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1092)
at
com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:63)
at
com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:10)
at
com.fasterxml.jackson.databind.deser.std.MapDeserializer._readAndBindStringKeyMap(MapDeserializer.java:527)
at
com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:364)
at
com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:29)
at
com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4001)
at
com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3030)
at
org.apache.kylin.stream.source.kafka.TimedJsonStreamParser.parse(TimedJsonStreamParser.java:79)
at
org.apache.kylin.stream.source.kafka.TimedJsonStreamParser.parse(TimedJsonStreamParser.java:54)
at
org.apache.kylin.stream.source.kafka.consumer.KafkaConnector.nextEvent(KafkaConnector.java:110)
at
org.apache.kylin.stream.core.consumer.StreamingConsumerChannel.run(StreamingConsumerChannel.java:93)
at java.lang.Thread.run(Thread.java:748)
{quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3944) Streaming Cube UI display empty cube planner info
XiaoXiang Yu created KYLIN-3944: --- Summary: Streaming Cube UI display empty cube planner info Key: KYLIN-3944 URL: https://issues.apache.org/jira/browse/KYLIN-3944 Project: Kylin Issue Type: Bug Components: Website Affects Versions: v2.6.1 Reporter: XiaoXiang Yu Attachments: image-2019-04-10-10-58-17-452.png, image-2019-04-10-10-59-07-532.png In cube design page, I click "Planner" tab, it will show empty table. !image-2019-04-10-10-58-17-452.png! !image-2019-04-10-10-59-07-532.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3944) Streaming Cube UI display empty cube planner info
[ https://issues.apache.org/jira/browse/KYLIN-3944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3944: Description: In cube design page, I click "Planner" tab, it will show "Cuboid Distribution" empty table. !image-2019-04-10-10-58-17-452.png! !image-2019-04-10-10-59-07-532.png! was: In cube design page, I click "Planner" tab, it will show empty table. !image-2019-04-10-10-58-17-452.png! !image-2019-04-10-10-59-07-532.png! > Streaming Cube UI display empty cube planner info > - > > Key: KYLIN-3944 > URL: https://issues.apache.org/jira/browse/KYLIN-3944 > Project: Kylin > Issue Type: Bug > Components: Website >Affects Versions: v2.6.1 >Reporter: XiaoXiang Yu >Priority: Major > Attachments: image-2019-04-10-10-58-17-452.png, > image-2019-04-10-10-59-07-532.png > > > > In cube design page, I click "Planner" tab, it will show "Cuboid > Distribution" empty table. > > > !image-2019-04-10-10-58-17-452.png! > > !image-2019-04-10-10-59-07-532.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3944) Streaming Cube UI display empty cube planner info
[ https://issues.apache.org/jira/browse/KYLIN-3944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3944: Description: In cube design page, I click "Planner" tab, it will show "Cuboid Distribution" empty table. And these empty info won't disappear if I change to another streaming tab. !image-2019-04-10-10-58-17-452.png! !image-2019-04-10-10-59-07-532.png! was: In cube design page, I click "Planner" tab, it will show "Cuboid Distribution" empty table. !image-2019-04-10-10-58-17-452.png! !image-2019-04-10-10-59-07-532.png! > Streaming Cube UI display empty cube planner info > - > > Key: KYLIN-3944 > URL: https://issues.apache.org/jira/browse/KYLIN-3944 > Project: Kylin > Issue Type: Bug > Components: Website >Affects Versions: v2.6.1 >Reporter: XiaoXiang Yu >Priority: Major > Attachments: image-2019-04-10-10-58-17-452.png, > image-2019-04-10-10-59-07-532.png > > > > In cube design page, I click "Planner" tab, it will show "Cuboid > Distribution" empty table. > And these empty info won't disappear if I change to another streaming tab. > > > !image-2019-04-10-10-58-17-452.png! > > !image-2019-04-10-10-59-07-532.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-3944) Streaming Cube UI display empty cube planner info
[ https://issues.apache.org/jira/browse/KYLIN-3944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu closed KYLIN-3944. --- Resolution: Not A Bug It's a problem caused by error packaging. > Streaming Cube UI display empty cube planner info > - > > Key: KYLIN-3944 > URL: https://issues.apache.org/jira/browse/KYLIN-3944 > Project: Kylin > Issue Type: Bug > Components: Website >Affects Versions: v2.6.1 >Reporter: XiaoXiang Yu >Priority: Major > Attachments: image-2019-04-10-10-58-17-452.png, > image-2019-04-10-10-59-07-532.png > > > > In cube design page, I click "Planner" tab, it will show "Cuboid > Distribution" empty table. > And these empty info won't disappear if I change to another streaming tab. > > > !image-2019-04-10-10-58-17-452.png! > > !image-2019-04-10-10-59-07-532.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3959) Realtime OLAP query result should not be cached
XiaoXiang Yu created KYLIN-3959: --- Summary: Realtime OLAP query result should not be cached Key: KYLIN-3959 URL: https://issues.apache.org/jira/browse/KYLIN-3959 Project: Kylin Issue Type: Bug Components: Real-time Streaming Affects Versions: v3.0.0-alpha Reporter: XiaoXiang Yu Assignee: XiaoXiang Yu Fix For: v3.0.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-3609) NPE from QueryMetricsFacade#updateMetricsToReservoir
[
https://issues.apache.org/jira/browse/KYLIN-3609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
XiaoXiang Yu closed KYLIN-3609.
---
Resolution: Cannot Reproduce
> NPE from QueryMetricsFacade#updateMetricsToReservoir
>
>
> Key: KYLIN-3609
> URL: https://issues.apache.org/jira/browse/KYLIN-3609
> Project: Kylin
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: XiaoXiang Yu
>Priority: Minor
>
> When running test suite, I saw the following in test output:
> {code}
> 2018-10-02 20:59:20,415 WARN [Query c8f42f2e-8c77-2bfc-97ab-053fbeb7c86e-1]
> service.QueryService:423 : Write metric error.
> java.lang.NullPointerException
> at
> org.apache.kylin.rest.metrics.QueryMetricsFacade.updateMetricsToReservoir(QueryMetricsFacade.java:148)
> at
> org.apache.kylin.rest.metrics.QueryMetricsFacade.updateMetrics(QueryMetricsFacade.java:74)
> at
> org.apache.kylin.rest.service.QueryService.recordMetric(QueryService.java:505)
> at
> org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:421)
> at
> org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:353)
> at
> org.apache.kylin.rest.controller.QueryController.query(QueryController.java:87)
> at
> org.apache.kylin.rest.controller.QueryControllerTest.testQueryException(QueryControllerTest.java:63)
> {code}
> It seems sqlResponse.getResults() returned null.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (KYLIN-3959) Realtime OLAP query result should not be cached
[ https://issues.apache.org/jira/browse/KYLIN-3959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu resolved KYLIN-3959. - Resolution: Fixed > Realtime OLAP query result should not be cached > --- > > Key: KYLIN-3959 > URL: https://issues.apache.org/jira/browse/KYLIN-3959 > Project: Kylin > Issue Type: Bug > Components: Real-time Streaming >Affects Versions: v3.0.0-alpha >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Fix For: v3.0.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KYLIN-3942) Rea-time OLAP don't support multi-level json event
[
https://issues.apache.org/jira/browse/KYLIN-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
XiaoXiang Yu reassigned KYLIN-3942:
---
Assignee: XiaoXiang Yu (was: luguosheng)
Fix Version/s: v3.0.0
> Rea-time OLAP don't support multi-level json event
> --
>
> Key: KYLIN-3942
> URL: https://issues.apache.org/jira/browse/KYLIN-3942
> Project: Kylin
> Issue Type: Bug
> Components: Real-time Streaming
>Affects Versions: v3.0.0-alpha
>Reporter: XiaoXiang Yu
>Assignee: XiaoXiang Yu
>Priority: Critical
> Fix For: v3.0.0
>
>
> Currently real-time OLAP didn't support multi-level json event.
> For example,if I have a kafka multi-level json event like this:
> {quote}{"country":"JAPAN","amount":13.075058425023922,"qty":8,"currency":"USD","order_time":1554801950882,"category":"ELECTRONIC","device":"Andriod","user":\{"gender":"Female","id":"7a0cfa5e-bbaa-79ef-1a38-e06f02c85fcb","first_name":"unknown","age":16}}
> {quote}
>
> Receiver will throw exception like this and discard that event:
>
> {quote}2019-04-09 09:46:09,878 ERROR [StreamingV2Cube_channel]
> kafka.TimedJsonStreamParser:107 : error
> com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot
> deserialize instance of `java.lang.String` out of START_OBJECT token
> at [Source:
> (String)"\{"country":"US","amount":14.498498222823619,"qty":1,"currency":"USD","order_time":1554803169876,"category":"Other","device":"Other","user":{"gender":"Female","id":"0736b41a-9ae7-9b4a-a124-f74436d3eb41","first_name":"unknown","age":26}}";
> line: 1, column: 140] (through reference chain: java.util.HashMap["user"])
> at
> com.fasterxml.jackson.databind.exc.MismatchedInputException.from(MismatchedInputException.java:63)
> at
> com.fasterxml.jackson.databind.DeserializationContext.reportInputMismatch(DeserializationContext.java:1342)
> at
> com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1138)
> at
> com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1092)
> at
> com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:63)
> at
> com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:10)
> at
> com.fasterxml.jackson.databind.deser.std.MapDeserializer._readAndBindStringKeyMap(MapDeserializer.java:527)
> at
> com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:364)
> at
> com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:29)
> at
> com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4001)
> at
> com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3030)
> at
> org.apache.kylin.stream.source.kafka.TimedJsonStreamParser.parse(TimedJsonStreamParser.java:79)
> at
> org.apache.kylin.stream.source.kafka.TimedJsonStreamParser.parse(TimedJsonStreamParser.java:54)
> at
> org.apache.kylin.stream.source.kafka.consumer.KafkaConnector.nextEvent(KafkaConnector.java:110)
> at
> org.apache.kylin.stream.core.consumer.StreamingConsumerChannel.run(StreamingConsumerChannel.java:93)
> at java.lang.Thread.run(Thread.java:748)
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (KYLIN-3942) Rea-time OLAP don't support multi-level json event
[
https://issues.apache.org/jira/browse/KYLIN-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
XiaoXiang Yu resolved KYLIN-3942.
-
Resolution: Fixed
> Rea-time OLAP don't support multi-level json event
> --
>
> Key: KYLIN-3942
> URL: https://issues.apache.org/jira/browse/KYLIN-3942
> Project: Kylin
> Issue Type: Bug
> Components: Real-time Streaming
>Affects Versions: v3.0.0-alpha
>Reporter: XiaoXiang Yu
>Assignee: XiaoXiang Yu
>Priority: Critical
> Fix For: v3.0.0
>
>
> Currently real-time OLAP didn't support multi-level json event.
> For example,if I have a kafka multi-level json event like this:
> {quote}{"country":"JAPAN","amount":13.075058425023922,"qty":8,"currency":"USD","order_time":1554801950882,"category":"ELECTRONIC","device":"Andriod","user":\{"gender":"Female","id":"7a0cfa5e-bbaa-79ef-1a38-e06f02c85fcb","first_name":"unknown","age":16}}
> {quote}
>
> Receiver will throw exception like this and discard that event:
>
> {quote}2019-04-09 09:46:09,878 ERROR [StreamingV2Cube_channel]
> kafka.TimedJsonStreamParser:107 : error
> com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot
> deserialize instance of `java.lang.String` out of START_OBJECT token
> at [Source:
> (String)"\{"country":"US","amount":14.498498222823619,"qty":1,"currency":"USD","order_time":1554803169876,"category":"Other","device":"Other","user":{"gender":"Female","id":"0736b41a-9ae7-9b4a-a124-f74436d3eb41","first_name":"unknown","age":26}}";
> line: 1, column: 140] (through reference chain: java.util.HashMap["user"])
> at
> com.fasterxml.jackson.databind.exc.MismatchedInputException.from(MismatchedInputException.java:63)
> at
> com.fasterxml.jackson.databind.DeserializationContext.reportInputMismatch(DeserializationContext.java:1342)
> at
> com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1138)
> at
> com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1092)
> at
> com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:63)
> at
> com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:10)
> at
> com.fasterxml.jackson.databind.deser.std.MapDeserializer._readAndBindStringKeyMap(MapDeserializer.java:527)
> at
> com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:364)
> at
> com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:29)
> at
> com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4001)
> at
> com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3030)
> at
> org.apache.kylin.stream.source.kafka.TimedJsonStreamParser.parse(TimedJsonStreamParser.java:79)
> at
> org.apache.kylin.stream.source.kafka.TimedJsonStreamParser.parse(TimedJsonStreamParser.java:54)
> at
> org.apache.kylin.stream.source.kafka.consumer.KafkaConnector.nextEvent(KafkaConnector.java:110)
> at
> org.apache.kylin.stream.core.consumer.StreamingConsumerChannel.run(StreamingConsumerChannel.java:93)
> at java.lang.Thread.run(Thread.java:748)
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3962) Support streaming cubing using Spark Streaming or Flink
[ https://issues.apache.org/jira/browse/KYLIN-3962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16820980#comment-16820980 ] XiaoXiang Yu commented on KYLIN-3962: - If I use Flink streaming to ingest streaming message, and write cuboid data(I think this maybe ) to external storage(like hbase or redis), I think it will have some drawback: - Using remote storage instead of local storage is will increase the data prepare delay. - Introduce external dependency like flink cluster. - Too heavy pressure on external storage. - Filter and aggregate will be slower if data in remote. > Support streaming cubing using Spark Streaming or Flink > --- > > Key: KYLIN-3962 > URL: https://issues.apache.org/jira/browse/KYLIN-3962 > Project: Kylin > Issue Type: Improvement >Reporter: Liu Shaohui >Priority: Major > > KYLIN-3654 introduced the Real-time Streaming, but in my opinion, the arch is > a little too complicated to handle. > As streaming frameworks like spark streaming, flink are widely used in many > companies.Can we use the streaming framework to support real time cubing in > Kylin. > This is just a proposal. More discussion and suggestions are welcomed~ > More details of this proposal will be added later. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3962) Support streaming cubing using Spark Streaming or Flink
[ https://issues.apache.org/jira/browse/KYLIN-3962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16820980#comment-16820980 ] XiaoXiang Yu edited comment on KYLIN-3962 at 4/18/19 11:44 AM: --- If I use Flink streaming to ingest streaming message, and write cuboid data(I think this maybe _++_) to external storage(like hbase or redis), I think it will have some drawback: - Using remote storage instead of local storage will increase the data prepare delay. - Introduce external dependency like flink cluster. - Too heavy pressure on external storage(maybe every entered message will cause a r/w to storage layer). - If we decide to use more cuboid, it will cause too much r/w to storage. If we decide to use more cuboid, most query will hit base cuboid, filter and aggregate will be slower if data in remote (we can use memory cache in receiver). was (Author: hit_lacus): If I use Flink streaming to ingest streaming message, and write cuboid data(I think this maybe ) to external storage(like hbase or redis), I think it will have some drawback: - Using remote storage instead of local storage is will increase the data prepare delay. - Introduce external dependency like flink cluster. - Too heavy pressure on external storage. - Filter and aggregate will be slower if data in remote. > Support streaming cubing using Spark Streaming or Flink > --- > > Key: KYLIN-3962 > URL: https://issues.apache.org/jira/browse/KYLIN-3962 > Project: Kylin > Issue Type: Improvement >Reporter: Liu Shaohui >Priority: Major > > KYLIN-3654 introduced the Real-time Streaming, but in my opinion, the arch is > a little too complicated to handle. > As streaming frameworks like spark streaming, flink are widely used in many > companies.Can we use the streaming framework to support real time cubing in > Kylin. > This is just a proposal. More discussion and suggestions are welcomed~ > More details of this proposal will be added later. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3962) Support streaming cubing using Spark Streaming or Flink
[ https://issues.apache.org/jira/browse/KYLIN-3962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16820980#comment-16820980 ] XiaoXiang Yu edited comment on KYLIN-3962 at 4/18/19 11:50 AM: --- If I use Flink streaming to ingest streaming message, and write cuboid data(I think this maybe _++_) to external storage(like hbase or redis), I think it will have some drawback: - Using remote storage instead of local storage will increase the data prepare delay. - Introduce external dependency like flink cluster. - Too heavy pressure on external storage(maybe every entered message will cause a r/w to storage layer). - If we decide to use more cuboid, it will cause too much r/w to storage. If we decide to use less cuboid, most query will hit base cuboid, filter and aggregate will be slower if data in remote (we can use memory cache in receiver). was (Author: hit_lacus): If I use Flink streaming to ingest streaming message, and write cuboid data(I think this maybe _++_) to external storage(like hbase or redis), I think it will have some drawback: - Using remote storage instead of local storage will increase the data prepare delay. - Introduce external dependency like flink cluster. - Too heavy pressure on external storage(maybe every entered message will cause a r/w to storage layer). - If we decide to use more cuboid, it will cause too much r/w to storage. If we decide to use more cuboid, most query will hit base cuboid, filter and aggregate will be slower if data in remote (we can use memory cache in receiver). > Support streaming cubing using Spark Streaming or Flink > --- > > Key: KYLIN-3962 > URL: https://issues.apache.org/jira/browse/KYLIN-3962 > Project: Kylin > Issue Type: Improvement >Reporter: Liu Shaohui >Priority: Major > > KYLIN-3654 introduced the Real-time Streaming, but in my opinion, the arch is > a little too complicated to handle. > As streaming frameworks like spark streaming, flink are widely used in many > companies.Can we use the streaming framework to support real time cubing in > Kylin. > This is just a proposal. More discussion and suggestions are welcomed~ > More details of this proposal will be added later. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3857) Add parameter to change sql quoting character for sqoop
[
https://issues.apache.org/jira/browse/KYLIN-3857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
XiaoXiang Yu resolved KYLIN-3857.
-
Resolution: Fixed
Fix Version/s: v3.0.0
> Add parameter to change sql quoting character for sqoop
> ---
>
> Key: KYLIN-3857
> URL: https://issues.apache.org/jira/browse/KYLIN-3857
> Project: Kylin
> Issue Type: Bug
>Reporter: Davide Malagoli
>Assignee: XiaoXiang Yu
>Priority: Major
> Fix For: v3.0.0, v2.6.2
>
> Attachments: image-2019-03-11-11-46-05-946.png,
> image-2019-03-11-11-46-14-305.png, image-2019-03-11-11-46-20-664.png,
> image-2019-03-11-11-46-31-966.png, sqoop-error.log, sqoop-script.sh
>
>
> I've set up Kylin to access a SQL server using JDBC and do the ETL step as
> described in
> [http://kylin.apache.org/docs/tutorial/setup_jdbc_datasource.html]
> but when the "Sqoop To Flat Hive Table" runs it terminates with an error
> shown in the attached log.
> It seems that the root cause is the presence of the '`' character in the
> query, which has no meaning for sql server.
>
> My problem seems to related to this piece of code
>
> public class FlatTableSqlQuoteUtils {
> public static final String QUOTE = "`";
> /**
> * Quote identifier by default quote `
> * @param identifier
> * @return
> */
> public static String quoteIdentifier(String identifier){
> return QUOTE + identifier + QUOTE;
> }
>
> The solution may be to add a parameter to change this character used for
> quoting.
>
> Just to be sure I made a copy of the original "sqoop" command to "sqoop.orig"
> and replaced the original with a little script of mine (attached).
> This little script removes the "`" character from the original command and
> the call "sqoop.orig" with the cleaned arguments.
> It works, but it is still an ugly workaround.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
XiaoXiang Yu created KYLIN-3970: --- Summary: NRT Streaming cannot work at HDP3.0 Key: KYLIN-3970 URL: https://issues.apache.org/jira/browse/KYLIN-3970 Project: Kylin Issue Type: Bug Components: NRT Streaming Affects Versions: v2.6.1 Reporter: XiaoXiang Yu Assignee: XiaoXiang Yu Fix For: v3.0.0, v2.6.2 We found that when using [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], NRT streaming cannot work in serval step for jar conflict problem. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[ https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3970: Labels: HDP Hadoop3 (was: ) > NRT Streaming cannot work at HDP3.0 > --- > > Key: KYLIN-3970 > URL: https://issues.apache.org/jira/browse/KYLIN-3970 > Project: Kylin > Issue Type: Bug > Components: NRT Streaming >Affects Versions: v2.6.1 >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Labels: HDP, Hadoop3 > Fix For: v3.0.0, v2.6.2 > > > We found that when using > [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], > NRT streaming cannot work in serval step for jar conflict problem. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[ https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3970: Attachment: image-2019-04-23-14-25-51-066.png > NRT Streaming cannot work at HDP3.0 > --- > > Key: KYLIN-3970 > URL: https://issues.apache.org/jira/browse/KYLIN-3970 > Project: Kylin > Issue Type: Bug > Components: NRT Streaming >Affects Versions: v2.6.1 >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Labels: HDP, Hadoop3 > Fix For: v3.0.0, v2.6.2 > > Attachments: image-2019-04-23-14-25-51-066.png > > > We found that when using > [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], > NRT streaming cannot work in serval step for jar conflict problem. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[ https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3970: Description: We found that when using [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], NRT streaming cannot work in serval step for jar conflict problem. !image-2019-04-23-14-25-51-066.png! was:We found that when using [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], NRT streaming cannot work in serval step for jar conflict problem. > NRT Streaming cannot work at HDP3.0 > --- > > Key: KYLIN-3970 > URL: https://issues.apache.org/jira/browse/KYLIN-3970 > Project: Kylin > Issue Type: Bug > Components: NRT Streaming >Affects Versions: v2.6.1 >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Labels: HDP, Hadoop3 > Fix For: v3.0.0, v2.6.2 > > Attachments: image-2019-04-23-14-25-51-066.png > > > We found that when using > [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], > NRT streaming cannot work in serval step for jar conflict problem. > > !image-2019-04-23-14-25-51-066.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[ https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16823735#comment-16823735 ] XiaoXiang Yu commented on KYLIN-3970: - h3. *First problem:* *Kafka jar conflict* org.apache.kylin.engine.mr.exception.MapReduceException: no counters for job job_1555653800318_0004Job Diagnostics:Task failed task_1555653800318_0004_m_00 Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 0 Failure task Diagnostics: Error: org.apache.kafka.clients.consumer.ConsumerConfig.configNames()Ljava/util/Set; at org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:173) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166) at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166) at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) h3. *Second problem: Hive null value* org.apache.kylin.engine.mr.exception.MapReduceException: no counters for job job_1555653800318_0007Job Diagnostics:Task failed task_1555653800318_0007_m_00 Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 0 Failure task Diagnostics: Error: java.lang.IllegalArgumentException: there is no valid date pattern for:\N at org.apache.kylin.common.util.DateFormat.stringToMillis(DateFormat.java:124) at org.apache.kylin.metadata.datatype.DataTypeOrder$DateTimeOrder.toComparable(DataTypeOrder.java:154) at org.apache.kylin.metadata.datatype.DataTypeOrder$DateTimeOrder.toComparable(DataTypeOrder.java:148) at org.apache.kylin.metadata.datatype.DataTypeOrder.compare(DataTypeOrder.java:95) at org.apache.kylin.metadata.datatype.DataTypeOrder.max(DataTypeOrder.java:90) at org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper.doMap(FactDistinctColumnsMapper.java:194) at org.apache.kylin.engine.mr.KylinMapper.map(KylinMapper.java:77) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168) at org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:173) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166) at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166) at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) > NRT Streaming cannot work at HDP3.0 > --- > > Key: KYLIN-3970 > URL: https://issues.apache.org/jira/browse/KYLIN-3970 > Project: Kylin > Issue Type: Bug > Components: NRT Streaming >Affects Versions: v2.6.1 >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Labels: HDP, Hadoop3 > Fix For: v3.0.0, v2.6.2 > > Attachments: image-2019-04-23-14-25-51-066.png > > > We found that when using > [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], > NRT streaming cannot work in serval step for jar conflict problem. > > !image-2019-04-23-14-25-51-066.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[ https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3970: Description: We found that when using [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], NRT streaming cannot work in serval step for jar conflict problem. !image-2019-04-24-15-35-06-712.png! was: We found that when using [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], NRT streaming cannot work in serval step for jar conflict problem. !image-2019-04-23-14-25-51-066.png! > NRT Streaming cannot work at HDP3.0 > --- > > Key: KYLIN-3970 > URL: https://issues.apache.org/jira/browse/KYLIN-3970 > Project: Kylin > Issue Type: Bug > Components: NRT Streaming >Affects Versions: v2.6.1 >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Labels: HDP, Hadoop3 > Fix For: v3.0.0, v2.6.2 > > Attachments: image-2019-04-23-14-25-51-066.png > > > We found that when using > [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], > NRT streaming cannot work in serval step for jar conflict problem. > > !image-2019-04-24-15-35-06-712.png! > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[ https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3970: Attachment: image-2019-04-24-15-38-10-098.png > NRT Streaming cannot work at HDP3.0 > --- > > Key: KYLIN-3970 > URL: https://issues.apache.org/jira/browse/KYLIN-3970 > Project: Kylin > Issue Type: Bug > Components: NRT Streaming >Affects Versions: v2.6.1 >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Labels: HDP, Hadoop3 > Fix For: v3.0.0, v2.6.2 > > Attachments: image-2019-04-23-14-25-51-066.png, > image-2019-04-24-15-38-10-098.png > > > We found that when using > [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], > NRT streaming cannot work in serval step for jar conflict problem. > > !image-2019-04-24-15-35-06-712.png! > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[ https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3970: Description: We found that when using [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], NRT streaming cannot work in serval step for jar conflict problem. h2. *Env* |HDFS|3.1.1| |YARN|3.1.1| |MapReduce2|3.1.1| |Hive|3.1.0| |HBase|2.0.0| |ZooKeeper|3.4.6| |HDP|3.0.1.0| !image-2019-04-24-15-38-10-098.png! was: We found that when using [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], NRT streaming cannot work in serval step for jar conflict problem. !image-2019-04-24-15-35-06-712.png! > NRT Streaming cannot work at HDP3.0 > --- > > Key: KYLIN-3970 > URL: https://issues.apache.org/jira/browse/KYLIN-3970 > Project: Kylin > Issue Type: Bug > Components: NRT Streaming >Affects Versions: v2.6.1 >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Labels: HDP, Hadoop3 > Fix For: v3.0.0, v2.6.2 > > Attachments: image-2019-04-23-14-25-51-066.png, > image-2019-04-24-15-38-10-098.png > > > We found that when using > [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], > NRT streaming cannot work in serval step for jar conflict problem. > h2. *Env* > |HDFS|3.1.1| > |YARN|3.1.1| > |MapReduce2|3.1.1| > |Hive|3.1.0| > |HBase|2.0.0| > |ZooKeeper|3.4.6| > |HDP|3.0.1.0| > > > !image-2019-04-24-15-38-10-098.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[ https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16824880#comment-16824880 ] XiaoXiang Yu commented on KYLIN-3970: - Root Cause HDP contains a low version kafka-client jar, both in Kylin process's classpath and MapReduce's classpath : "/usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar". This is my classpath of Kylin: CLASSPATH = /home/kylin-hdp3/xiaoxiang/apache-kylin-2.6.1-bin-hadoop3/bin/../tomcat/bin/bootstrap.jar:/home/kylin-hdp3/xiaoxiang/apache-kylin-2.6.1-bin-hadoop3/bin/../tomcat/bin/tomcat-juli.jar:/home/kylin-hdp3/xiaoxiang/apache-kylin-2.6.1-bin-hadoop3/bin/../tomcat/lib/*:/home/kylin-hdp3/xiaoxiang/apache-kylin-2.6.1-bin-hadoop3/conf:/home/kylin-hdp3/xiaoxiang/apache-kylin-2.6.1-bin-hadoop3/lib/*:/home/kylin-hdp3/xiaoxiang/apache-kylin-2.6.1-bin-hadoop3/ext/*::/usr/hdp/3.0.1.0-187/hbase/conf:/usr/local/bin/jdk1.8.0_191/lib/tools.jar:/usr/hdp/3.0.1.0-187/hbase:/usr/hdp/3.0.1.0-187/hbase/lib/accessors-smart-1.2.jar:/usr/hdp/3.0.1.0-187/hbase/lib/aopalliance-1.0.jar:/usr/hdp/3.0.1.0-187/hbase/lib/aopalliance-repackaged-2.5.0-b32.jar:/usr/hdp/3.0.1.0-187/hbase/lib/asm-5.0.4.jar:/usr/hdp/3.0.1.0-187/hbase/lib/atlas-plugin-classloader-1.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/audience-annotations-0.5.0.jar:/usr/hdp/3.0.1.0-187/hbase/lib/avro-1.7.7.jar:/usr/hdp/3.0.1.0-187/hbase/lib/aws-java-sdk-bundle-1.11.271.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-beanutils-1.9.3.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-cli-1.2.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-codec-1.10.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-collections-3.2.2.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-compress-1.4.1.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-configuration2-2.1.1.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-crypto-1.0.0.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-csv-1.0.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-daemon-1.0.13.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-io-2.5.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-lang-2.6.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-lang3-3.6.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-logging-1.2.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-math3-3.6.1.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-net-3.6.jar:/usr/hdp/3.0.1.0-187/hbase/lib/curator-client-4.0.0.jar:/usr/hdp/3.0.1.0-187/hbase/lib/curator-framework-4.0.0.jar:/usr/hdp/3.0.1.0-187/hbase/lib/curator-recipes-4.0.0.jar:/usr/hdp/3.0.1.0-187/hbase/lib/disruptor-3.3.6.jar:/usr/hdp/3.0.1.0-187/hbase/lib/dnsjava-2.1.7.jar:/usr/hdp/3.0.1.0-187/hbase/lib/ehcache-3.3.1.jar:/usr/hdp/3.0.1.0-187/hbase/lib/findbugs-annotations-1.3.9-1.jar:/usr/hdp/3.0.1.0-187/hbase/lib/fst-2.50.jar:/usr/hdp/3.0.1.0-187/hbase/lib/geronimo-jcache_1.0_spec-1.0-alpha-1.jar:/usr/hdp/3.0.1.0-187/hbase/lib/gson-2.2.4.jar:/usr/hdp/3.0.1.0-187/hbase/lib/guava-11.0.2.jar:/usr/hdp/3.0.1.0-187/hbase/lib/guice-4.0.jar:/usr/hdp/3.0.1.0-187/hbase/lib/guice-servlet-4.0.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hamcrest-core-1.3.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-annotations-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-annotations-2.0.0.3.0.1.0-187-tests.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-annotations.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-backup-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-backup.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-bridge-shim-1.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-client-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-client.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-common-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-common-2.0.0.3.0.1.0-187-tests.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-common.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-endpoint-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-endpoint.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-examples-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-examples.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-external-blockcache-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-external-blockcache.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-hadoop2-compat-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-hadoop2-compat-2.0.0.3.0.1.0-187-tests.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-hadoop2-compat.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-hadoop-compat-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-hadoop-compat-2.0.0.3.0.1.0-187-tests.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-hadoop-compat.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-http-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-http.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-it-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-it-2.0.0.3.0.1.0-187-tests.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-it.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-mapreduce-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-mapreduce-2.0.0.3.0.1.0-187-tests.jar:/usr
[jira] [Comment Edited] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[ https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16823735#comment-16823735 ] XiaoXiang Yu edited comment on KYLIN-3970 at 4/24/19 7:48 AM: -- *Problem:* *Kafka jar conflict* Caused by: org.apache.kafka.common.config.ConfigException: Missing required configuration "partition.assignment.strategy" which has no default value. at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:124) at org.apache.kafka.common.config.AbstractConfig.(AbstractConfig.java:48) at org.apache.kafka.clients.consumer.ConsumerConfig.(ConsumerConfig.java:194) at org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:430) at org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:413) at org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:400) at org.apache.kylin.source.kafka.util.KafkaClient.getKafkaConsumer(KafkaClient.java:48) at org.apache.kylin.source.kafka.util.KafkaClient.getEarliestOffsets(KafkaClient.java:128) at org.apache.kylin.source.kafka.KafkaSource.enrichSourcePartitionBeforeBuild(KafkaSource.java:100) at org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:232) at org.apache.kylin.rest.service.JobService.submitJob(JobService.java:202) at org.apache.kylin.rest.controller.CubeController.buildInternal(CubeController.java:394) ... 80 more org.apache.kylin.engine.mr.exception.MapReduceException: no counters for job job_1555653800318_0004Job Diagnostics:Task failed task_1555653800318_0004_m_00 Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 0 Failure task Diagnostics: Error: org.apache.kafka.clients.consumer.ConsumerConfig.configNames()Ljava/util/Set; at org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:173) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166) at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166) at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) h3. was (Author: hit_lacus): h3. *First problem:* *Kafka jar conflict* org.apache.kylin.engine.mr.exception.MapReduceException: no counters for job job_1555653800318_0004Job Diagnostics:Task failed task_1555653800318_0004_m_00 Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 0 Failure task Diagnostics: Error: org.apache.kafka.clients.consumer.ConsumerConfig.configNames()Ljava/util/Set; at org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:173) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166) at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166) at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) h3. *Second problem: Hive null value* org.apache.kylin.engine.mr.exception.MapReduceException: no counters for job job_1555653800318_0007Job Diagnostics:Task failed task_1555653800318_0007_m_00 Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 0 Failure task Diagnostics: Error: java.lang.IllegalArgumentException: there is no valid date pattern for:\N at org.apache.kylin.common.util.DateFormat.stringToMillis(DateFormat.java:124) at org.apache.kylin.metadata.datatype.DataTypeOrder$DateTimeOrder.toComparable(DataTypeOrder.java:154) at org.apache.kylin.metadata.datatype.DataTypeOrder$DateTimeOrder.toComparable(DataTypeOrder.java:148) at org.apache.kylin.metadata.datatype.DataTypeOrder.compare(DataTypeOrder.java:95) at org.apache.kylin.metadata.datatype.DataTypeOrder.max(DataTypeOrder.java:90) at org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper.doMap(FactDistinctColumnsMapper.java:194) at org.apache.kylin.engine.mr.KylinMapper.map(KylinMapper.java:77) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799) at org.apache.hadoop.m
[jira] [Updated] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[ https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3970: Fix Version/s: (was: v2.6.2) (was: v3.0.0) > NRT Streaming cannot work at HDP3.0 > --- > > Key: KYLIN-3970 > URL: https://issues.apache.org/jira/browse/KYLIN-3970 > Project: Kylin > Issue Type: Bug > Components: NRT Streaming >Affects Versions: v2.6.1 >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Labels: HDP, Hadoop3 > Attachments: image-2019-04-23-14-25-51-066.png, > image-2019-04-24-15-38-10-098.png > > > We found that when using > [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], > NRT streaming cannot work in serval step for jar conflict problem. > h2. *Env* > |HDFS|3.1.1| > |YARN|3.1.1| > |MapReduce2|3.1.1| > |Hive|3.1.0| > |HBase|2.0.0| > |ZooKeeper|3.4.6| > |HDP|3.0.1.0| > > > !image-2019-04-24-15-38-10-098.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[
https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16824880#comment-16824880
]
XiaoXiang Yu edited comment on KYLIN-3970 at 4/24/19 7:50 AM:
--
h2. *Root Cause*
HDP contains a low version kafka-client jar, both in Kylin process's classpath
and MapReduce's classpath :
"/usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar".
This is my classpath of Kylin:
{code}CLASSPATH =
/home/kylin-hdp3/xiaoxiang/apache-kylin-2.6.1-bin-hadoop3/bin/../tomcat/bin/bootstrap.jar:/home/kylin-hdp3/xiaoxiang/apache-kylin-2.6.1-bin-hadoop3/bin/../tomcat/bin/tomcat-juli.jar:/home/kylin-hdp3/xiaoxiang/apache-kylin-2.6.1-bin-hadoop3/bin/../tomcat/lib/*:/home/kylin-hdp3/xiaoxiang/apache-kylin-2.6.1-bin-hadoop3/conf:/home/kylin-hdp3/xiaoxiang/apache-kylin-2.6.1-bin-hadoop3/lib/*:/home/kylin-hdp3/xiaoxiang/apache-kylin-2.6.1-bin-hadoop3/ext/*::/usr/hdp/3.0.1.0-187/hbase/conf:/usr/local/bin/jdk1.8.0_191/lib/tools.jar:/usr/hdp/3.0.1.0-187/hbase:/usr/hdp/3.0.1.0-187/hbase/lib/accessors-smart-1.2.jar:/usr/hdp/3.0.1.0-187/hbase/lib/aopalliance-1.0.jar:/usr/hdp/3.0.1.0-187/hbase/lib/aopalliance-repackaged-2.5.0-b32.jar:/usr/hdp/3.0.1.0-187/hbase/lib/asm-5.0.4.jar:/usr/hdp/3.0.1.0-187/hbase/lib/atlas-plugin-classloader-1.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/audience-annotations-0.5.0.jar:/usr/hdp/3.0.1.0-187/hbase/lib/avro-1.7.7.jar:/usr/hdp/3.0.1.0-187/hbase/lib/aws-java-sdk-bundle-1.11.271.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-beanutils-1.9.3.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-cli-1.2.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-codec-1.10.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-collections-3.2.2.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-compress-1.4.1.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-configuration2-2.1.1.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-crypto-1.0.0.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-csv-1.0.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-daemon-1.0.13.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-io-2.5.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-lang-2.6.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-lang3-3.6.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-logging-1.2.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-math3-3.6.1.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-net-3.6.jar:/usr/hdp/3.0.1.0-187/hbase/lib/curator-client-4.0.0.jar:/usr/hdp/3.0.1.0-187/hbase/lib/curator-framework-4.0.0.jar:/usr/hdp/3.0.1.0-187/hbase/lib/curator-recipes-4.0.0.jar:/usr/hdp/3.0.1.0-187/hbase/lib/disruptor-3.3.6.jar:/usr/hdp/3.0.1.0-187/hbase/lib/dnsjava-2.1.7.jar:/usr/hdp/3.0.1.0-187/hbase/lib/ehcache-3.3.1.jar:/usr/hdp/3.0.1.0-187/hbase/lib/findbugs-annotations-1.3.9-1.jar:/usr/hdp/3.0.1.0-187/hbase/lib/fst-2.50.jar:/usr/hdp/3.0.1.0-187/hbase/lib/geronimo-jcache_1.0_spec-1.0-alpha-1.jar:/usr/hdp/3.0.1.0-187/hbase/lib/gson-2.2.4.jar:/usr/hdp/3.0.1.0-187/hbase/lib/guava-11.0.2.jar:/usr/hdp/3.0.1.0-187/hbase/lib/guice-4.0.jar:/usr/hdp/3.0.1.0-187/hbase/lib/guice-servlet-4.0.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hamcrest-core-1.3.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-annotations-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-annotations-2.0.0.3.0.1.0-187-tests.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-annotations.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-backup-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-backup.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-bridge-shim-1.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-client-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-client.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-common-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-common-2.0.0.3.0.1.0-187-tests.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-common.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-endpoint-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-endpoint.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-examples-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-examples.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-external-blockcache-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-external-blockcache.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-hadoop2-compat-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-hadoop2-compat-2.0.0.3.0.1.0-187-tests.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-hadoop2-compat.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-hadoop-compat-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-hadoop-compat-2.0.0.3.0.1.0-187-tests.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-hadoop-compat.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-http-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-http.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-it-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-it-2.0.0.3.0.1.0-187-tests.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-it.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-mapreduce-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-18
[jira] [Comment Edited] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[
https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16823735#comment-16823735
]
XiaoXiang Yu edited comment on KYLIN-3970 at 4/24/19 7:54 AM:
--
*Problem:* *Kafka jar conflict*
{code}
Caused by: org.apache.kafka.common.config.ConfigException: Missing required
configuration "partition.assignment.strategy" which has no default value.
at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:124)
at org.apache.kafka.common.config.AbstractConfig.(AbstractConfig.java:48)
at
org.apache.kafka.clients.consumer.ConsumerConfig.(ConsumerConfig.java:194)
at
org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:430)
at
org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:413)
at
org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:400)
at
org.apache.kylin.source.kafka.util.KafkaClient.getKafkaConsumer(KafkaClient.java:48)
at
org.apache.kylin.source.kafka.util.KafkaClient.getEarliestOffsets(KafkaClient.java:128)
at
org.apache.kylin.source.kafka.KafkaSource.enrichSourcePartitionBeforeBuild(KafkaSource.java:100)
at
org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:232)
at org.apache.kylin.rest.service.JobService.submitJob(JobService.java:202)
at
org.apache.kylin.rest.controller.CubeController.buildInternal(CubeController.java:394)
... 80 more
{code}
{code}
org.apache.kylin.engine.mr.exception.MapReduceException: no counters for job
job_1555653800318_0004Job Diagnostics:Task failed
task_1555653800318_0004_m_00
Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0
killedReduces: 0
Failure task Diagnostics:
Error:
org.apache.kafka.clients.consumer.ConsumerConfig.configNames()Ljava/util/Set;
at
org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:173)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166)
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}
h3.
was (Author: hit_lacus):
*Problem:* *Kafka jar conflict*
Caused by: org.apache.kafka.common.config.ConfigException: Missing required
configuration "partition.assignment.strategy" which has no default value.
at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:124)
at org.apache.kafka.common.config.AbstractConfig.(AbstractConfig.java:48)
at
org.apache.kafka.clients.consumer.ConsumerConfig.(ConsumerConfig.java:194)
at
org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:430)
at
org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:413)
at
org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:400)
at
org.apache.kylin.source.kafka.util.KafkaClient.getKafkaConsumer(KafkaClient.java:48)
at
org.apache.kylin.source.kafka.util.KafkaClient.getEarliestOffsets(KafkaClient.java:128)
at
org.apache.kylin.source.kafka.KafkaSource.enrichSourcePartitionBeforeBuild(KafkaSource.java:100)
at
org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:232)
at org.apache.kylin.rest.service.JobService.submitJob(JobService.java:202)
at
org.apache.kylin.rest.controller.CubeController.buildInternal(CubeController.java:394)
... 80 more
org.apache.kylin.engine.mr.exception.MapReduceException: no counters for job
job_1555653800318_0004Job Diagnostics:Task failed
task_1555653800318_0004_m_00
Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0
killedReduces: 0
Failure task Diagnostics:
Error:
org.apache.kafka.clients.consumer.ConsumerConfig.configNames()Ljava/util/Set;
at
org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:173)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166)
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
h3.
> NRT Streaming cannot work at HDP3.0
> ---
>
> Key: KYLIN-
[jira] [Comment Edited] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[
https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16823735#comment-16823735
]
XiaoXiang Yu edited comment on KYLIN-3970 at 4/24/19 7:55 AM:
--
h2. *Problem:* *Kafka jar conflict*
* when click build buttom
{code:java}
Caused by: org.apache.kafka.common.config.ConfigException: Missing required
configuration "partition.assignment.strategy" which has no default value.
at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:124)
at org.apache.kafka.common.config.AbstractConfig.(AbstractConfig.java:48)
at
org.apache.kafka.clients.consumer.ConsumerConfig.(ConsumerConfig.java:194)
at
org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:430)
at
org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:413)
at
org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:400)
at
org.apache.kylin.source.kafka.util.KafkaClient.getKafkaConsumer(KafkaClient.java:48)
at
org.apache.kylin.source.kafka.util.KafkaClient.getEarliestOffsets(KafkaClient.java:128)
at
org.apache.kylin.source.kafka.KafkaSource.enrichSourcePartitionBeforeBuild(KafkaSource.java:100)
at
org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:232)
at org.apache.kylin.rest.service.JobService.submitJob(JobService.java:202)
at
org.apache.kylin.rest.controller.CubeController.buildInternal(CubeController.java:394)
... 80 more
{code}
* First step of build segment
{code:java}
org.apache.kylin.engine.mr.exception.MapReduceException: no counters for job
job_1555653800318_0004Job Diagnostics:Task failed
task_1555653800318_0004_m_00
Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0
killedReduces: 0
Failure task Diagnostics:
Error:
org.apache.kafka.clients.consumer.ConsumerConfig.configNames()Ljava/util/Set;
at
org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:173)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166)
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}
h3.
was (Author: hit_lacus):
*Problem:* *Kafka jar conflict*
{code}
Caused by: org.apache.kafka.common.config.ConfigException: Missing required
configuration "partition.assignment.strategy" which has no default value.
at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:124)
at org.apache.kafka.common.config.AbstractConfig.(AbstractConfig.java:48)
at
org.apache.kafka.clients.consumer.ConsumerConfig.(ConsumerConfig.java:194)
at
org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:430)
at
org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:413)
at
org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:400)
at
org.apache.kylin.source.kafka.util.KafkaClient.getKafkaConsumer(KafkaClient.java:48)
at
org.apache.kylin.source.kafka.util.KafkaClient.getEarliestOffsets(KafkaClient.java:128)
at
org.apache.kylin.source.kafka.KafkaSource.enrichSourcePartitionBeforeBuild(KafkaSource.java:100)
at
org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:232)
at org.apache.kylin.rest.service.JobService.submitJob(JobService.java:202)
at
org.apache.kylin.rest.controller.CubeController.buildInternal(CubeController.java:394)
... 80 more
{code}
{code}
org.apache.kylin.engine.mr.exception.MapReduceException: no counters for job
job_1555653800318_0004Job Diagnostics:Task failed
task_1555653800318_0004_m_00
Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0
killedReduces: 0
Failure task Diagnostics:
Error:
org.apache.kafka.clients.consumer.ConsumerConfig.configNames()Ljava/util/Set;
at
org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:173)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166)
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}
h3.
> NR
[jira] [Updated] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[ https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3970: Attachment: (was: image-2019-04-23-14-25-51-066.png) > NRT Streaming cannot work at HDP3.0 > --- > > Key: KYLIN-3970 > URL: https://issues.apache.org/jira/browse/KYLIN-3970 > Project: Kylin > Issue Type: Bug > Components: NRT Streaming >Affects Versions: v2.6.1 >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Labels: HDP, Hadoop3 > Attachments: image-2019-04-24-15-38-10-098.png > > > We found that when using > [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], > NRT streaming cannot work in serval step for jar conflict problem. > h2. *Env* > |HDFS|3.1.1| > |YARN|3.1.1| > |MapReduce2|3.1.1| > |Hive|3.1.0| > |HBase|2.0.0| > |ZooKeeper|3.4.6| > |HDP|3.0.1.0| > > > !image-2019-04-24-15-38-10-098.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[ https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16824897#comment-16824897 ] XiaoXiang Yu commented on KYLIN-3970: - h2. *Solution* # remove old version kafka-client.jar # add *mapreduce.job.user.classpath.first=true* in _kylin_job_conf.xml_ > NRT Streaming cannot work at HDP3.0 > --- > > Key: KYLIN-3970 > URL: https://issues.apache.org/jira/browse/KYLIN-3970 > Project: Kylin > Issue Type: Bug > Components: NRT Streaming >Affects Versions: v2.6.1 >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Labels: HDP, Hadoop3 > Attachments: image-2019-04-24-15-38-10-098.png > > > We found that when using > [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], > NRT streaming cannot work in serval step for jar conflict problem. > h2. *Env* > |HDFS|3.1.1| > |YARN|3.1.1| > |MapReduce2|3.1.1| > |Hive|3.1.0| > |HBase|2.0.0| > |ZooKeeper|3.4.6| > |HDP|3.0.1.0| > > > !image-2019-04-24-15-38-10-098.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[
https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16824897#comment-16824897
]
XiaoXiang Yu edited comment on KYLIN-3970 at 4/24/19 8:21 AM:
--
h2. *Try*
# remove old version kafka-client.jar
# add *mapreduce.job.user.classpath.first=true* in _kylin_job_conf.xml_
We will got a NoSuchMethodError caused by Guava
{code}
2019-04-24 08:06:04,378 INFO [main]
org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output:
DEFAULT.SAMPLE_1.COUNTRY 'US' => reducer 0 2019-04-24 08:06:04,378 INFO [main]
org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output:
DEFAULT.SAMPLE_1.CATEGORY 'BOOK' => reducer 1 2019-04-24 08:06:04,379 INFO
[main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample
output: DEFAULT.SAMPLE_1.DEVICE 'Andriod' => reducer 2 2019-04-24 08:06:04,379
INFO [main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample
output: DEFAULT.SAMPLE_1.USER_GENDER 'Male' => reducer 3 2019-04-24
08:06:04,379 INFO [main]
org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output:
DEFAULT.SAMPLE_1.COUNTRY 'KOREA' => reducer 0 2019-04-24 08:06:04,379 INFO
[main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample
output: DEFAULT.SAMPLE_1.CATEGORY 'CLOTH' => reducer 1 2019-04-24 08:06:04,379
INFO [main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample
output: DEFAULT.SAMPLE_1.DEVICE 'iOS' => reducer 2 2019-04-24 08:06:04,379 INFO
[main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample
output: DEFAULT.SAMPLE_1.USER_GENDER 'Female' => reducer 3 2019-04-24
08:06:04,381 FATAL [Thread-13]
org.apache.hadoop.yarn.YarnUncaughtExceptionHandler: Thread
Thread[Thread-13,5,main] threw an Error. Shutting down now...
java.lang.NoSuchMethodError:
com.google.common.hash.Hasher.putString(Ljava/lang/CharSequence;)Lcom/google/common/hash/Hasher;
at
org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper$CuboidStatCalculator.putRowKeyToHLLNew(FactDistinctColumnsMapper.java:377)
at
org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper$CuboidStatCalculator.run(FactDistinctColumnsMapper.java:411)
at java.lang.Thread.run(Thread.java:748) 2019-04-24 08:06:04,385 INFO
[Thread-13] org.apache.hadoop.util.ExitUtil: Exiting with status -1:
ExitException
{code}
was (Author: hit_lacus):
h2. *Solution*
# remove old version kafka-client.jar
# add *mapreduce.job.user.classpath.first=true* in _kylin_job_conf.xml_
> NRT Streaming cannot work at HDP3.0
> ---
>
> Key: KYLIN-3970
> URL: https://issues.apache.org/jira/browse/KYLIN-3970
> Project: Kylin
> Issue Type: Bug
> Components: NRT Streaming
>Affects Versions: v2.6.1
>Reporter: XiaoXiang Yu
>Assignee: XiaoXiang Yu
>Priority: Major
> Labels: HDP, Hadoop3
> Attachments: image-2019-04-24-15-38-10-098.png
>
>
> We found that when using
> [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz],
> NRT streaming cannot work in serval step for jar conflict problem.
> h2. *Env*
> |HDFS|3.1.1|
> |YARN|3.1.1|
> |MapReduce2|3.1.1|
> |Hive|3.1.0|
> |HBase|2.0.0|
> |ZooKeeper|3.4.6|
> |HDP|3.0.1.0|
>
>
> !image-2019-04-24-15-38-10-098.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[
https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16824897#comment-16824897
]
XiaoXiang Yu edited comment on KYLIN-3970 at 4/24/19 8:26 AM:
--
h2. *Try*
remove old version kafka-client.jar
{code}
mv /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar
/usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar.old
{code}
add *mapreduce.job.user.classpath.first=true* in _kylin_job_conf.xml_
We will got a NoSuchMethodError caused by Guava
{code:java}
2019-04-24 08:06:04,378 INFO [main]
org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output:
DEFAULT.SAMPLE_1.COUNTRY 'US' => reducer 0 2019-04-24 08:06:04,378 INFO [main]
org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output:
DEFAULT.SAMPLE_1.CATEGORY 'BOOK' => reducer 1 2019-04-24 08:06:04,379 INFO
[main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample
output: DEFAULT.SAMPLE_1.DEVICE 'Andriod' => reducer 2 2019-04-24 08:06:04,379
INFO [main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample
output: DEFAULT.SAMPLE_1.USER_GENDER 'Male' => reducer 3 2019-04-24
08:06:04,379 INFO [main]
org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output:
DEFAULT.SAMPLE_1.COUNTRY 'KOREA' => reducer 0 2019-04-24 08:06:04,379 INFO
[main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample
output: DEFAULT.SAMPLE_1.CATEGORY 'CLOTH' => reducer 1 2019-04-24 08:06:04,379
INFO [main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample
output: DEFAULT.SAMPLE_1.DEVICE 'iOS' => reducer 2 2019-04-24 08:06:04,379 INFO
[main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample
output: DEFAULT.SAMPLE_1.USER_GENDER 'Female' => reducer 3 2019-04-24
08:06:04,381 FATAL [Thread-13]
org.apache.hadoop.yarn.YarnUncaughtExceptionHandler: Thread
Thread[Thread-13,5,main] threw an Error. Shutting down now...
java.lang.NoSuchMethodError:
com.google.common.hash.Hasher.putString(Ljava/lang/CharSequence;)Lcom/google/common/hash/Hasher;
at
org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper$CuboidStatCalculator.putRowKeyToHLLNew(FactDistinctColumnsMapper.java:377)
at
org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper$CuboidStatCalculator.run(FactDistinctColumnsMapper.java:411)
at java.lang.Thread.run(Thread.java:748) 2019-04-24 08:06:04,385 INFO
[Thread-13] org.apache.hadoop.util.ExitUtil: Exiting with status -1:
ExitException
{code}
was (Author: hit_lacus):
h2. *Try*
# remove old version kafka-client.jar
# add *mapreduce.job.user.classpath.first=true* in _kylin_job_conf.xml_
We will got a NoSuchMethodError caused by Guava
{code}
2019-04-24 08:06:04,378 INFO [main]
org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output:
DEFAULT.SAMPLE_1.COUNTRY 'US' => reducer 0 2019-04-24 08:06:04,378 INFO [main]
org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output:
DEFAULT.SAMPLE_1.CATEGORY 'BOOK' => reducer 1 2019-04-24 08:06:04,379 INFO
[main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample
output: DEFAULT.SAMPLE_1.DEVICE 'Andriod' => reducer 2 2019-04-24 08:06:04,379
INFO [main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample
output: DEFAULT.SAMPLE_1.USER_GENDER 'Male' => reducer 3 2019-04-24
08:06:04,379 INFO [main]
org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output:
DEFAULT.SAMPLE_1.COUNTRY 'KOREA' => reducer 0 2019-04-24 08:06:04,379 INFO
[main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample
output: DEFAULT.SAMPLE_1.CATEGORY 'CLOTH' => reducer 1 2019-04-24 08:06:04,379
INFO [main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample
output: DEFAULT.SAMPLE_1.DEVICE 'iOS' => reducer 2 2019-04-24 08:06:04,379 INFO
[main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample
output: DEFAULT.SAMPLE_1.USER_GENDER 'Female' => reducer 3 2019-04-24
08:06:04,381 FATAL [Thread-13]
org.apache.hadoop.yarn.YarnUncaughtExceptionHandler: Thread
Thread[Thread-13,5,main] threw an Error. Shutting down now...
java.lang.NoSuchMethodError:
com.google.common.hash.Hasher.putString(Ljava/lang/CharSequence;)Lcom/google/common/hash/Hasher;
at
org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper$CuboidStatCalculator.putRowKeyToHLLNew(FactDistinctColumnsMapper.java:377)
at
org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper$CuboidStatCalculator.run(FactDistinctColumnsMapper.java:411)
at java.lang.Thread.run(Thread.java:748) 2019-04-24 08:06:04,385 INFO
[Thread-13] org.apache.hadoop.util.ExitUtil: Exiting with status -1:
ExitException
{code}
> NRT Streaming cannot work at HDP3.0
> ---
>
> Key: KYLIN-3970
> URL: https:
[jira] [Comment Edited] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[
https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16824897#comment-16824897
]
XiaoXiang Yu edited comment on KYLIN-3970 at 4/24/19 8:43 AM:
--
h2. *Try-1*
remove old version kafka-client.jar
{code:java}
mv /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar
/usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar.old
{code}
We want to let mapper use kylin's classpath first, so we add
*mapreduce.job.user.classpath.first=true* in _kylin_job_conf.xml_
But this cause more conflict, we will got a NoSuchMethodError caused by Guava.
{code:java}
2019-04-24 08:06:04,378 INFO [main]
org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output:
DEFAULT.SAMPLE_1.COUNTRY 'US' => reducer 0 2019-04-24 08:06:04,378 INFO [main]
org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output:
DEFAULT.SAMPLE_1.CATEGORY 'BOOK' => reducer 1 2019-04-24 08:06:04,379 INFO
[main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample
output: DEFAULT.SAMPLE_1.DEVICE 'Andriod' => reducer 2 2019-04-24 08:06:04,379
INFO [main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample
output: DEFAULT.SAMPLE_1.USER_GENDER 'Male' => reducer 3 2019-04-24
08:06:04,379 INFO [main]
org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output:
DEFAULT.SAMPLE_1.COUNTRY 'KOREA' => reducer 0 2019-04-24 08:06:04,379 INFO
[main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample
output: DEFAULT.SAMPLE_1.CATEGORY 'CLOTH' => reducer 1 2019-04-24 08:06:04,379
INFO [main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample
output: DEFAULT.SAMPLE_1.DEVICE 'iOS' => reducer 2 2019-04-24 08:06:04,379 INFO
[main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample
output: DEFAULT.SAMPLE_1.USER_GENDER 'Female' => reducer 3 2019-04-24
08:06:04,381 FATAL [Thread-13]
org.apache.hadoop.yarn.YarnUncaughtExceptionHandler: Thread
Thread[Thread-13,5,main] threw an Error. Shutting down now...
java.lang.NoSuchMethodError:
com.google.common.hash.Hasher.putString(Ljava/lang/CharSequence;)Lcom/google/common/hash/Hasher;
at
org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper$CuboidStatCalculator.putRowKeyToHLLNew(FactDistinctColumnsMapper.java:377)
at
org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper$CuboidStatCalculator.run(FactDistinctColumnsMapper.java:411)
at java.lang.Thread.run(Thread.java:748) 2019-04-24 08:06:04,385 INFO
[Thread-13] org.apache.hadoop.util.ExitUtil: Exiting with status -1:
ExitException
{code}
was (Author: hit_lacus):
h2. *Try*
remove old version kafka-client.jar
{code}
mv /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar
/usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar.old
{code}
add *mapreduce.job.user.classpath.first=true* in _kylin_job_conf.xml_
We will got a NoSuchMethodError caused by Guava
{code:java}
2019-04-24 08:06:04,378 INFO [main]
org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output:
DEFAULT.SAMPLE_1.COUNTRY 'US' => reducer 0 2019-04-24 08:06:04,378 INFO [main]
org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output:
DEFAULT.SAMPLE_1.CATEGORY 'BOOK' => reducer 1 2019-04-24 08:06:04,379 INFO
[main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample
output: DEFAULT.SAMPLE_1.DEVICE 'Andriod' => reducer 2 2019-04-24 08:06:04,379
INFO [main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample
output: DEFAULT.SAMPLE_1.USER_GENDER 'Male' => reducer 3 2019-04-24
08:06:04,379 INFO [main]
org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output:
DEFAULT.SAMPLE_1.COUNTRY 'KOREA' => reducer 0 2019-04-24 08:06:04,379 INFO
[main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample
output: DEFAULT.SAMPLE_1.CATEGORY 'CLOTH' => reducer 1 2019-04-24 08:06:04,379
INFO [main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample
output: DEFAULT.SAMPLE_1.DEVICE 'iOS' => reducer 2 2019-04-24 08:06:04,379 INFO
[main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample
output: DEFAULT.SAMPLE_1.USER_GENDER 'Female' => reducer 3 2019-04-24
08:06:04,381 FATAL [Thread-13]
org.apache.hadoop.yarn.YarnUncaughtExceptionHandler: Thread
Thread[Thread-13,5,main] threw an Error. Shutting down now...
java.lang.NoSuchMethodError:
com.google.common.hash.Hasher.putString(Ljava/lang/CharSequence;)Lcom/google/common/hash/Hasher;
at
org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper$CuboidStatCalculator.putRowKeyToHLLNew(FactDistinctColumnsMapper.java:377)
at
org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper$CuboidStatCalculator.run(FactDistinctColumnsMapper.java:411)
at java.lang.Thread.run(Thread.java:748) 2019-04-2
[jira] [Commented] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[ https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16824923#comment-16824923 ] XiaoXiang Yu commented on KYLIN-3970: - h2. *Try2* # remove old version kafka lib mv /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar.old # remove mapdreduce lib cache rm -rf /data/hadoop/yarn/local/usercache/kylin-hdp3/appcache # remove hadoop fs -copyToLocal /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz . hadoop fs -mv /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz.ori tar zxf mapreduce.tar.gz rm hadoop/share/hadoop/tools/lib/kafka-clients-0.8.2.1.jar tar zcf mapreduce.tar.gz hadoop/ hadoop fs -copyFromLocal mapreduce.tar.gz /hdp/apps/3.0.1.0-187/mapreduce > NRT Streaming cannot work at HDP3.0 > --- > > Key: KYLIN-3970 > URL: https://issues.apache.org/jira/browse/KYLIN-3970 > Project: Kylin > Issue Type: Bug > Components: NRT Streaming >Affects Versions: v2.6.1 >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Labels: HDP, Hadoop3 > Attachments: image-2019-04-24-15-38-10-098.png > > > We found that when using > [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], > NRT streaming cannot work in serval step for jar conflict problem. > h2. *Env* > |HDFS|3.1.1| > |YARN|3.1.1| > |MapReduce2|3.1.1| > |Hive|3.1.0| > |HBase|2.0.0| > |ZooKeeper|3.4.6| > |HDP|3.0.1.0| > > > !image-2019-04-24-15-38-10-098.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[
https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16824923#comment-16824923
]
XiaoXiang Yu edited comment on KYLIN-3970 at 4/24/19 8:53 AM:
--
h2. *Try2*
{code}
## remove old version kafka lib
mv /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar
/usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar.old
## remove mapdreduce lib cache
rm -rf /data/hadoop/yarn/local/usercache/kylin-hdp3/appcache
## remove kafka-clients-0.8.2.1.jar in HDFS
hadoop fs -copyToLocal /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz .
hadoop fs -mv /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz
/hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz.ori
tar zxf mapreduce.tar.gz
rm hadoop/share/hadoop/tools/lib/kafka-clients-0.8.2.1.jar
tar zcf mapreduce.tar.gz hadoop/
hadoop fs -copyFromLocal mapreduce.tar.gz /hdp/apps/3.0.1.0-187/mapreduce
{code}
was (Author: hit_lacus):
h2. *Try2*
# remove old version kafka lib
mv /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar
/usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar.old
# remove mapdreduce lib cache
rm -rf /data/hadoop/yarn/local/usercache/kylin-hdp3/appcache
# remove
hadoop fs -copyToLocal /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz .
hadoop fs -mv /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz
/hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz.ori
tar zxf mapreduce.tar.gz
rm hadoop/share/hadoop/tools/lib/kafka-clients-0.8.2.1.jar
tar zcf mapreduce.tar.gz hadoop/
hadoop fs -copyFromLocal mapreduce.tar.gz /hdp/apps/3.0.1.0-187/mapreduce
> NRT Streaming cannot work at HDP3.0
> ---
>
> Key: KYLIN-3970
> URL: https://issues.apache.org/jira/browse/KYLIN-3970
> Project: Kylin
> Issue Type: Bug
> Components: NRT Streaming
>Affects Versions: v2.6.1
>Reporter: XiaoXiang Yu
>Assignee: XiaoXiang Yu
>Priority: Major
> Labels: HDP, Hadoop3
> Attachments: image-2019-04-24-15-38-10-098.png
>
>
> We found that when using
> [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz],
> NRT streaming cannot work in serval step for jar conflict problem.
> h2. *Env*
> |HDFS|3.1.1|
> |YARN|3.1.1|
> |MapReduce2|3.1.1|
> |Hive|3.1.0|
> |HBase|2.0.0|
> |ZooKeeper|3.4.6|
> |HDP|3.0.1.0|
>
>
> !image-2019-04-24-15-38-10-098.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[
https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16824923#comment-16824923
]
XiaoXiang Yu edited comment on KYLIN-3970 at 4/24/19 8:56 AM:
--
h2. *Try2*
{code:java}
## remove old version kafka lib
mv /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar
/usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar.old
## remove mapdreduce lib cache
rm -rf /data/hadoop/yarn/local/usercache/kylin-hdp3/appcache
## remove kafka-clients-0.8.2.1.jar in HDFS
hadoop fs -copyToLocal /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz .
hadoop fs -mv /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz
/hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz.ori
tar zxf mapreduce.tar.gz
rm hadoop/share/hadoop/tools/lib/kafka-clients-0.8.2.1.jar
tar zcf mapreduce.tar.gz hadoop/
hadoop fs -copyFromLocal mapreduce.tar.gz /hdp/apps/3.0.1.0-187/mapreduce
{code}
In my env, this can work.
was (Author: hit_lacus):
h2. *Try2*
{code}
## remove old version kafka lib
mv /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar
/usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar.old
## remove mapdreduce lib cache
rm -rf /data/hadoop/yarn/local/usercache/kylin-hdp3/appcache
## remove kafka-clients-0.8.2.1.jar in HDFS
hadoop fs -copyToLocal /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz .
hadoop fs -mv /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz
/hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz.ori
tar zxf mapreduce.tar.gz
rm hadoop/share/hadoop/tools/lib/kafka-clients-0.8.2.1.jar
tar zcf mapreduce.tar.gz hadoop/
hadoop fs -copyFromLocal mapreduce.tar.gz /hdp/apps/3.0.1.0-187/mapreduce
{code}
> NRT Streaming cannot work at HDP3.0
> ---
>
> Key: KYLIN-3970
> URL: https://issues.apache.org/jira/browse/KYLIN-3970
> Project: Kylin
> Issue Type: Bug
> Components: NRT Streaming
>Affects Versions: v2.6.1
>Reporter: XiaoXiang Yu
>Assignee: XiaoXiang Yu
>Priority: Major
> Labels: HDP, Hadoop3
> Attachments: image-2019-04-24-15-38-10-098.png
>
>
> We found that when using
> [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz],
> NRT streaming cannot work in serval step for jar conflict problem.
> h2. *Env*
> |HDFS|3.1.1|
> |YARN|3.1.1|
> |MapReduce2|3.1.1|
> |Hive|3.1.0|
> |HBase|2.0.0|
> |ZooKeeper|3.4.6|
> |HDP|3.0.1.0|
>
>
> !image-2019-04-24-15-38-10-098.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[
https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16824923#comment-16824923
]
XiaoXiang Yu edited comment on KYLIN-3970 at 4/24/19 8:57 AM:
--
h2. *Try2*
{code:java}
## remove old version kafka lib
mv /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar
/usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar.old
## remove mapdreduce lib cache
rm -rf /data/hadoop/yarn/local/usercache/kylin-hdp3/appcache
## remove kafka-clients-0.8.2.1.jar in mapreduce.tar.gz(HDFS)
hadoop fs -copyToLocal /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz .
hadoop fs -mv /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz
/hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz.ori
tar zxf mapreduce.tar.gz
rm hadoop/share/hadoop/tools/lib/kafka-clients-0.8.2.1.jar
tar zcf mapreduce.tar.gz hadoop/
hadoop fs -copyFromLocal mapreduce.tar.gz /hdp/apps/3.0.1.0-187/mapreduce
{code}
In my env, this can work.
was (Author: hit_lacus):
h2. *Try2*
{code:java}
## remove old version kafka lib
mv /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar
/usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar.old
## remove mapdreduce lib cache
rm -rf /data/hadoop/yarn/local/usercache/kylin-hdp3/appcache
## remove kafka-clients-0.8.2.1.jar in HDFS
hadoop fs -copyToLocal /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz .
hadoop fs -mv /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz
/hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz.ori
tar zxf mapreduce.tar.gz
rm hadoop/share/hadoop/tools/lib/kafka-clients-0.8.2.1.jar
tar zcf mapreduce.tar.gz hadoop/
hadoop fs -copyFromLocal mapreduce.tar.gz /hdp/apps/3.0.1.0-187/mapreduce
{code}
In my env, this can work.
> NRT Streaming cannot work at HDP3.0
> ---
>
> Key: KYLIN-3970
> URL: https://issues.apache.org/jira/browse/KYLIN-3970
> Project: Kylin
> Issue Type: Bug
> Components: NRT Streaming
>Affects Versions: v2.6.1
>Reporter: XiaoXiang Yu
>Assignee: XiaoXiang Yu
>Priority: Major
> Labels: HDP, Hadoop3
> Attachments: image-2019-04-24-15-38-10-098.png
>
>
> We found that when using
> [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz],
> NRT streaming cannot work in serval step for jar conflict problem.
> h2. *Env*
> |HDFS|3.1.1|
> |YARN|3.1.1|
> |MapReduce2|3.1.1|
> |Hive|3.1.0|
> |HBase|2.0.0|
> |ZooKeeper|3.4.6|
> |HDP|3.0.1.0|
>
>
> !image-2019-04-24-15-38-10-098.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[
https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16824923#comment-16824923
]
XiaoXiang Yu edited comment on KYLIN-3970 at 4/24/19 9:00 AM:
--
h2. *Try2*
{code:java}
## remove old version kafka lib
mv /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar
/usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar.old
## remove mapdreduce lib cache
rm -rf /data/hadoop/yarn/local/usercache/kylin-hdp3/appcache
## remove kafka-clients-0.8.2.1.jar in mapreduce.tar.gz(HDFS)
hadoop fs -copyToLocal /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz .
hadoop fs -mv /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz
/hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz.ori
tar zxf mapreduce.tar.gz
rm hadoop/share/hadoop/tools/lib/kafka-clients-0.8.2.1.jar
tar zcf mapreduce.tar.gz hadoop/
hadoop fs -copyFromLocal mapreduce.tar.gz /hdp/apps/3.0.1.0-187/mapreduce
## Restart kylin and try again
{code}
Though it looks so tricky, but this can work without repackage.
was (Author: hit_lacus):
h2. *Try2*
{code:java}
## remove old version kafka lib
mv /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar
/usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar.old
## remove mapdreduce lib cache
rm -rf /data/hadoop/yarn/local/usercache/kylin-hdp3/appcache
## remove kafka-clients-0.8.2.1.jar in mapreduce.tar.gz(HDFS)
hadoop fs -copyToLocal /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz .
hadoop fs -mv /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz
/hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz.ori
tar zxf mapreduce.tar.gz
rm hadoop/share/hadoop/tools/lib/kafka-clients-0.8.2.1.jar
tar zcf mapreduce.tar.gz hadoop/
hadoop fs -copyFromLocal mapreduce.tar.gz /hdp/apps/3.0.1.0-187/mapreduce
{code}
In my env, this can work.
> NRT Streaming cannot work at HDP3.0
> ---
>
> Key: KYLIN-3970
> URL: https://issues.apache.org/jira/browse/KYLIN-3970
> Project: Kylin
> Issue Type: Bug
> Components: NRT Streaming
>Affects Versions: v2.6.1
>Reporter: XiaoXiang Yu
>Assignee: XiaoXiang Yu
>Priority: Major
> Labels: HDP, Hadoop3
> Attachments: image-2019-04-24-15-38-10-098.png
>
>
> We found that when using
> [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz],
> NRT streaming cannot work in serval step for jar conflict problem.
> h2. *Env*
> |HDFS|3.1.1|
> |YARN|3.1.1|
> |MapReduce2|3.1.1|
> |Hive|3.1.0|
> |HBase|2.0.0|
> |ZooKeeper|3.4.6|
> |HDP|3.0.1.0|
>
>
> !image-2019-04-24-15-38-10-098.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[
https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16824923#comment-16824923
]
XiaoXiang Yu edited comment on KYLIN-3970 at 4/24/19 9:22 AM:
--
h2. *Try2*
{code:java}
## remove old version kafka lib
mv /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar
/usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar.old
## remove mapdreduce lib cache
rm -rf /data/hadoop/yarn/local/usercache/kylin-hdp3/appcache
## remove kafka-clients-0.8.2.1.jar in mapreduce.tar.gz(HDFS)
hadoop fs -copyToLocal /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz .
hadoop fs -mv /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz
/hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz.ori
tar zxf mapreduce.tar.gz
rm hadoop/share/hadoop/tools/lib/kafka-clients-0.8.2.1.jar
tar zcf mapreduce.tar.gz hadoop/
hadoop fs -copyFromLocal mapreduce.tar.gz /hdp/apps/3.0.1.0-187/mapreduce
## Restart kylin and try again
{code}
Though it looks so tricky, but this can work .
was (Author: hit_lacus):
h2. *Try2*
{code:java}
## remove old version kafka lib
mv /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar
/usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar.old
## remove mapdreduce lib cache
rm -rf /data/hadoop/yarn/local/usercache/kylin-hdp3/appcache
## remove kafka-clients-0.8.2.1.jar in mapreduce.tar.gz(HDFS)
hadoop fs -copyToLocal /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz .
hadoop fs -mv /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz
/hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz.ori
tar zxf mapreduce.tar.gz
rm hadoop/share/hadoop/tools/lib/kafka-clients-0.8.2.1.jar
tar zcf mapreduce.tar.gz hadoop/
hadoop fs -copyFromLocal mapreduce.tar.gz /hdp/apps/3.0.1.0-187/mapreduce
## Restart kylin and try again
{code}
Though it looks so tricky, but this can work without repackage.
> NRT Streaming cannot work at HDP3.0
> ---
>
> Key: KYLIN-3970
> URL: https://issues.apache.org/jira/browse/KYLIN-3970
> Project: Kylin
> Issue Type: Bug
> Components: NRT Streaming
>Affects Versions: v2.6.1
>Reporter: XiaoXiang Yu
>Assignee: XiaoXiang Yu
>Priority: Major
> Labels: HDP, Hadoop3
> Attachments: image-2019-04-24-15-38-10-098.png
>
>
> We found that when using
> [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz],
> NRT streaming cannot work in serval step for jar conflict problem.
> h2. *Env*
> |HDFS|3.1.1|
> |YARN|3.1.1|
> |MapReduce2|3.1.1|
> |Hive|3.1.0|
> |HBase|2.0.0|
> |ZooKeeper|3.4.6|
> |HDP|3.0.1.0|
>
>
> !image-2019-04-24-15-38-10-098.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[
https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16824923#comment-16824923
]
XiaoXiang Yu edited comment on KYLIN-3970 at 4/24/19 9:23 AM:
--
h2. *Try-2*
{code:java}
## remove old version kafka lib
mv /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar
/usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar.old
## remove mapdreduce lib cache
rm -rf /data/hadoop/yarn/local/usercache/kylin-hdp3/appcache
## remove kafka-clients-0.8.2.1.jar in mapreduce.tar.gz(HDFS)
hadoop fs -copyToLocal /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz .
hadoop fs -mv /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz
/hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz.ori
tar zxf mapreduce.tar.gz
rm hadoop/share/hadoop/tools/lib/kafka-clients-0.8.2.1.jar
tar zcf mapreduce.tar.gz hadoop/
hadoop fs -copyFromLocal mapreduce.tar.gz /hdp/apps/3.0.1.0-187/mapreduce
## Restart kylin and try again
{code}
Though it looks so tricky, but this can work .
was (Author: hit_lacus):
h2. *Try2*
{code:java}
## remove old version kafka lib
mv /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar
/usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar.old
## remove mapdreduce lib cache
rm -rf /data/hadoop/yarn/local/usercache/kylin-hdp3/appcache
## remove kafka-clients-0.8.2.1.jar in mapreduce.tar.gz(HDFS)
hadoop fs -copyToLocal /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz .
hadoop fs -mv /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz
/hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz.ori
tar zxf mapreduce.tar.gz
rm hadoop/share/hadoop/tools/lib/kafka-clients-0.8.2.1.jar
tar zcf mapreduce.tar.gz hadoop/
hadoop fs -copyFromLocal mapreduce.tar.gz /hdp/apps/3.0.1.0-187/mapreduce
## Restart kylin and try again
{code}
Though it looks so tricky, but this can work .
> NRT Streaming cannot work at HDP3.0
> ---
>
> Key: KYLIN-3970
> URL: https://issues.apache.org/jira/browse/KYLIN-3970
> Project: Kylin
> Issue Type: Bug
> Components: NRT Streaming
>Affects Versions: v2.6.1
>Reporter: XiaoXiang Yu
>Assignee: XiaoXiang Yu
>Priority: Major
> Labels: HDP, Hadoop3
> Attachments: image-2019-04-24-15-38-10-098.png
>
>
> We found that when using
> [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz],
> NRT streaming cannot work in serval step for jar conflict problem.
> h2. *Env*
> |HDFS|3.1.1|
> |YARN|3.1.1|
> |MapReduce2|3.1.1|
> |Hive|3.1.0|
> |HBase|2.0.0|
> |ZooKeeper|3.4.6|
> |HDP|3.0.1.0|
>
>
> !image-2019-04-24-15-38-10-098.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[ https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3970: Description: We found that when using [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], NRT streaming cannot work in serval step for jar conflict problem. h3. *Env* |HDFS|3.1.1| |YARN|3.1.1| |MapReduce2|3.1.1| |Hive|3.1.0| |HBase|2.0.0| |ZooKeeper|3.4.6| |HDP|3.0.1.0| h3. Error message !image-2019-04-24-15-38-10-098.png! was: We found that when using [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], NRT streaming cannot work in serval step for jar conflict problem. h2. *Env* |HDFS|3.1.1| |YARN|3.1.1| |MapReduce2|3.1.1| |Hive|3.1.0| |HBase|2.0.0| |ZooKeeper|3.4.6| |HDP|3.0.1.0| !image-2019-04-24-15-38-10-098.png! > NRT Streaming cannot work at HDP3.0 > --- > > Key: KYLIN-3970 > URL: https://issues.apache.org/jira/browse/KYLIN-3970 > Project: Kylin > Issue Type: Bug > Components: NRT Streaming >Affects Versions: v2.6.1 >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Labels: HDP, Hadoop3 > Attachments: image-2019-04-24-15-38-10-098.png > > > We found that when using > [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], > NRT streaming cannot work in serval step for jar conflict problem. > h3. *Env* > |HDFS|3.1.1| > |YARN|3.1.1| > |MapReduce2|3.1.1| > |Hive|3.1.0| > |HBase|2.0.0| > |ZooKeeper|3.4.6| > |HDP|3.0.1.0| > > h3. Error message > > !image-2019-04-24-15-38-10-098.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[ https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3970: Description: We found that when using [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], NRT streaming cannot work in serval step for jar conflict problem. h3. *Env* |HDFS|3.1.1| |YARN|3.1.1| |MapReduce2|3.1.1| |Hive|3.1.0| |HBase|2.0.0| |ZooKeeper|3.4.6| |HDP|3.0.1.0| h3. *Error message* !image-2019-04-24-15-38-10-098.png! was: We found that when using [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], NRT streaming cannot work in serval step for jar conflict problem. h3. *Env* |HDFS|3.1.1| |YARN|3.1.1| |MapReduce2|3.1.1| |Hive|3.1.0| |HBase|2.0.0| |ZooKeeper|3.4.6| |HDP|3.0.1.0| h3. Error message !image-2019-04-24-15-38-10-098.png! > NRT Streaming cannot work at HDP3.0 > --- > > Key: KYLIN-3970 > URL: https://issues.apache.org/jira/browse/KYLIN-3970 > Project: Kylin > Issue Type: Bug > Components: NRT Streaming >Affects Versions: v2.6.1 >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Labels: HDP, Hadoop3 > Attachments: image-2019-04-24-15-38-10-098.png > > > We found that when using > [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], > NRT streaming cannot work in serval step for jar conflict problem. > h3. *Env* > |HDFS|3.1.1| > |YARN|3.1.1| > |MapReduce2|3.1.1| > |Hive|3.1.0| > |HBase|2.0.0| > |ZooKeeper|3.4.6| > |HDP|3.0.1.0| > h3. *Error message* > > !image-2019-04-24-15-38-10-098.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[
https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16824923#comment-16824923
]
XiaoXiang Yu edited comment on KYLIN-3970 at 4/24/19 9:30 AM:
--
h2. *Try-2*
{code:java}
## remove old version kafka lib
mv /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar
/usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar.old
## remove mapdreduce lib cache
rm -rf /data/hadoop/yarn/local/usercache/${LOGIN_NMAE}/appcache
## remove kafka-clients-0.8.2.1.jar in mapreduce.tar.gz(HDFS)
hadoop fs -copyToLocal /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz .
hadoop fs -mv /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz
/hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz.ori
tar zxf mapreduce.tar.gz
rm hadoop/share/hadoop/tools/lib/kafka-clients-0.8.2.1.jar
## repackage mapreduce.tar.gz
tar zcf mapreduce.tar.gz hadoop/
## put it in right place
hadoop fs -copyFromLocal mapreduce.tar.gz /hdp/apps/3.0.1.0-187/mapreduce
## Restart kylin and try again
sh bin/kylin.sh start{code}
Though it looks so tricky, but this can work .
was (Author: hit_lacus):
h2. *Try-2*
{code:java}
## remove old version kafka lib
mv /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar
/usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar.old
## remove mapdreduce lib cache
rm -rf /data/hadoop/yarn/local/usercache/kylin-hdp3/appcache
## remove kafka-clients-0.8.2.1.jar in mapreduce.tar.gz(HDFS)
hadoop fs -copyToLocal /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz .
hadoop fs -mv /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz
/hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz.ori
tar zxf mapreduce.tar.gz
rm hadoop/share/hadoop/tools/lib/kafka-clients-0.8.2.1.jar
tar zcf mapreduce.tar.gz hadoop/
hadoop fs -copyFromLocal mapreduce.tar.gz /hdp/apps/3.0.1.0-187/mapreduce
## Restart kylin and try again
{code}
Though it looks so tricky, but this can work .
> NRT Streaming cannot work at HDP3.0
> ---
>
> Key: KYLIN-3970
> URL: https://issues.apache.org/jira/browse/KYLIN-3970
> Project: Kylin
> Issue Type: Bug
> Components: NRT Streaming
>Affects Versions: v2.6.1
>Reporter: XiaoXiang Yu
>Assignee: XiaoXiang Yu
>Priority: Major
> Labels: HDP, Hadoop3
> Attachments: image-2019-04-24-15-38-10-098.png
>
>
> We found that when using
> [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz],
> NRT streaming cannot work in serval step for jar conflict problem.
> h3. *Env*
> |HDFS|3.1.1|
> |YARN|3.1.1|
> |MapReduce2|3.1.1|
> |Hive|3.1.0|
> |HBase|2.0.0|
> |ZooKeeper|3.4.6|
> |HDP|3.0.1.0|
> h3. *Error message*
>
> !image-2019-04-24-15-38-10-098.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (KYLIN-3996) postgresql can not to be used to construct the flat hive table in NO.1 step in building cube via datasourcedefaultSdk
[ https://issues.apache.org/jira/browse/KYLIN-3996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu reassigned KYLIN-3996: --- Assignee: XiaoXiang Yu > postgresql can not to be used to construct the flat hive table in NO.1 step > in building cube via datasourcedefaultSdk > - > > Key: KYLIN-3996 > URL: https://issues.apache.org/jira/browse/KYLIN-3996 > Project: Kylin > Issue Type: Bug > Components: Driver - JDBC >Affects Versions: v2.6.1 > Environment: kylin3.0.0+hadoop2.7.4+hive2.1.1+hbase1.2.6 >Reporter: haijiandu >Assignee: XiaoXiang Yu >Priority: Critical > Labels: datasource, postgresql, sdk > Attachments: kylin_log_error_1.jpg, kylin_log_error_2.jpg, > kylin_log_error_3.jpg, kylin_log_error_4.jpg, kylin_log_error_5.jpg, > kylin_properties_duhj.jpg, kylin_sql.jpg, > kylin_sqoopToflatHiveTable_Error.jpg, postgresql_xml.jpg, > server_config_kylin_1.jpg, server_config_kylin_1.jpg, > server_config_kylin_2.jpg > > > postgresql can not to be used to construct the flat hive table in NO.1 step > in building cube via datasourcedefaultSdk. whether in kylin.properties or in > postgresql.xml，corresponding configureations have been finished as required > by official specification.There alwarys occurs a problem such as "ERROR > manager.SqlManager: Error executing statement: > org.postgresql.util.PSQLException: > ERROR: syntax error at or near ".", > when kylin is linked to postgresql databese, it seems that the errors happen > in sql brought by kylin. the sql has back quote,as it is "`" ,that causes > query failure brought by table name and columns defined by back quote(`). > as required, in kylin.properties ,its configuration has been as followings > kylin.source.default=16 > kylin.source.jdbc.connection-url=jdbc:postgresql://192.168.75.123:5432/dvdrental > kylin.source.jdbc.driver=org.postgresql.Driver > kylin.source.jdbc.dialect=postgresql > kylin.source.jdbc.adaptor=org.apache.kylin.sdk.datasource.adaptor.DefaultAdaptor > kylin.source.jdbc.user=postgres > kylin.source.jdbc.pass=1234 > kylin.source.jdbc.sqoop-home=/duhj/sqoop-1.4.7.bin__hadoop-2.6.0 > kylin.source.jdbc.filed-delimiter=| > kylin.source.jdbc.sqoop-mapper-num=4 > what is more, in > ~/apache-kylin-3.0/conf/datasource/postgresql.xml，corresponding conf is as > followings: > - > > > > > > > > > > > is there a method to solve it? > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-4001) Allow user-specified time format using real-time
[
https://issues.apache.org/jira/browse/KYLIN-4001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
XiaoXiang Yu updated KYLIN-4001:
Fix Version/s: v3.0.0
> Allow user-specified time format using real-time
>
>
> Key: KYLIN-4001
> URL: https://issues.apache.org/jira/browse/KYLIN-4001
> Project: Kylin
> Issue Type: Improvement
> Components: Real-time Streaming
>Reporter: ning guo
>Priority: Minor
> Fix For: v3.0.0
>
>
> * I found that real-time only supports millisecond timestamp, does not
> support second timestamp and Date type like '2019-01-01 11:11:11'.
> * I add a LongTimeParser and a DateTimeParser and page configuration
> * You can configure tsParser, tsPattern on the page that creates the
> streaming table.
> * for date :
> {code:java}
> { "timestamp":"2019-04-29 11:11:11","gmv":1.1 }
> You can specify
> tsParser=org.apache.kylin.stream.source.kafka.DateTimeParser
> tsPattern=-MM-dd HH:mm:ss{code}
>
> * for second :
> {code:java}
> { "timestamp":"1556618887","gmv":1.1 }
> You can specify
> tsParser=org.apache.kylin.stream.source.kafka.LongTimeParser
> tsPattern=S{code}
>
> * for millisecond :
> {code:java}
> { "timestamp":"1556618887000","gmv":1.1 }
> You can specify
> tsParser=org.apache.kylin.stream.source.kafka.LongTimeParser
> tsPattern=MS
> {code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (KYLIN-4007) Kylin concurrently builds a global dictionary for too long
[ https://issues.apache.org/jira/browse/KYLIN-4007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu reassigned KYLIN-4007: --- Assignee: XiaoXiang Yu > Kylin concurrently builds a global dictionary for too long > -- > > Key: KYLIN-4007 > URL: https://issues.apache.org/jira/browse/KYLIN-4007 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.6.1 >Reporter: mouxingya >Assignee: XiaoXiang Yu >Priority: Minor > Attachments: WechatIMG20.png > > > When a dimension (user_id) is accurately calculated at the same time, the > original data is in the order of millions, and the rowkey has the encoding > type fixed_lengrh and length 100. It took a few hours or even failed in the > build dimension dictionary step. After investigation, it was found that there > are concurrent processes that want to obtain the lock corresponding to the > cube / dict path, resulting in a deadlock. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-4007) Kylin concurrently builds a global dictionary for too long
[ https://issues.apache.org/jira/browse/KYLIN-4007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16843878#comment-16843878 ] XiaoXiang Yu commented on KYLIN-4007: - Looks like this issue is related to https://issues.apache.org/jira/browse/KYLIN-3633. > Kylin concurrently builds a global dictionary for too long > -- > > Key: KYLIN-4007 > URL: https://issues.apache.org/jira/browse/KYLIN-4007 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.6.1 >Reporter: mouxingya >Assignee: XiaoXiang Yu >Priority: Minor > Attachments: WechatIMG20.png > > > When a dimension (user_id) is accurately calculated at the same time, the > original data is in the order of millions, and the rowkey has the encoding > type fixed_lengrh and length 100. It took a few hours or even failed in the > build dimension dictionary step. After investigation, it was found that there > are concurrent processes that want to obtain the lock corresponding to the > cube / dict path, resulting in a deadlock. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-4007) Kylin concurrently builds a global dictionary for too long
[ https://issues.apache.org/jira/browse/KYLIN-4007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16843878#comment-16843878 ] XiaoXiang Yu edited comment on KYLIN-4007 at 5/20/19 11:28 AM: --- Looks like this issue is related to https://issues.apache.org/jira/browse/KYLIN-3633. But it should be fixed at v2.5.1, maybe it caused by other exception. But without more log/ stack trace, it difficult to detect in which way should we repair. was (Author: hit_lacus): Looks like this issue is related to https://issues.apache.org/jira/browse/KYLIN-3633. > Kylin concurrently builds a global dictionary for too long > -- > > Key: KYLIN-4007 > URL: https://issues.apache.org/jira/browse/KYLIN-4007 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.6.1 >Reporter: mouxingya >Assignee: XiaoXiang Yu >Priority: Minor > Attachments: WechatIMG20.png > > > When a dimension (user_id) is accurately calculated at the same time, the > original data is in the order of millions, and the rowkey has the encoding > type fixed_lengrh and length 100. It took a few hours or even failed in the > build dimension dictionary step. After investigation, it was found that there > are concurrent processes that want to obtain the lock corresponding to the > cube / dict path, resulting in a deadlock. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-4008) Real-time Streaming submit streaming job failed for spark engine
[ https://issues.apache.org/jira/browse/KYLIN-4008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16844623#comment-16844623 ] XiaoXiang Yu commented on KYLIN-4008: - Hi , currently realtime-olap do not support spark build engine. Maybe it will be support later. > Real-time Streaming submit streaming job failed for spark engine > > > Key: KYLIN-4008 > URL: https://issues.apache.org/jira/browse/KYLIN-4008 > Project: Kylin > Issue Type: Bug > Components: Real-time Streaming >Affects Versions: v3.0.0-alpha >Reporter: zengrui >Priority: Major > Attachments: error.bmp > > > Create a Realtime Streaming Cube and the Cube Engine is Spark, when the > coordinator node receive a remoteStoreCompelete request and exist some > segments can build, the streaming job submit failed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-4008) Real-time Streaming submit streaming job failed for spark engine
[ https://issues.apache.org/jira/browse/KYLIN-4008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16844623#comment-16844623 ] XiaoXiang Yu edited comment on KYLIN-4008 at 5/21/19 8:23 AM: -- Hi , currently realtime-olap do not support spark build engine. Maybe it will be support later. So I think this should not be a bug. was (Author: hit_lacus): Hi , currently realtime-olap do not support spark build engine. Maybe it will be support later. > Real-time Streaming submit streaming job failed for spark engine > > > Key: KYLIN-4008 > URL: https://issues.apache.org/jira/browse/KYLIN-4008 > Project: Kylin > Issue Type: Bug > Components: Real-time Streaming >Affects Versions: v3.0.0-alpha >Reporter: zengrui >Priority: Major > Attachments: error.bmp > > > Create a Realtime Streaming Cube and the Cube Engine is Spark, when the > coordinator node receive a remoteStoreCompelete request and exist some > segments can build, the streaming job submit failed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-4010) TimeZone is hard-coded in function makeSegmentName for class CubeSegment
[ https://issues.apache.org/jira/browse/KYLIN-4010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16844627#comment-16844627 ] XiaoXiang Yu commented on KYLIN-4010: - Hi, this seems to be a known issue in https://issues.apache.org/jira/browse/KYLIN-4001, and it should be implemented in next release. > TimeZone is hard-coded in function makeSegmentName for class CubeSegment > > > Key: KYLIN-4010 > URL: https://issues.apache.org/jira/browse/KYLIN-4010 > Project: Kylin > Issue Type: Improvement > Components: Others >Affects Versions: v2.6.2, v3.0.0-alpha >Reporter: zengrui >Priority: Minor > > In Real-Time Streaming Cube when I send some records to kafka topic, the > tmestamp for the record is 2019-01-01 00:00:00.000, but kylin create a > segment named 2018123116_2018123117. > Then I found that TimeZone is hard-coded to "GMT" in function makeSegmentName > for class CubeSegment. I think that it should be config in kylin.properties. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-4008) Real-time Streaming submit streaming job failed for spark engine
[ https://issues.apache.org/jira/browse/KYLIN-4008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-4008: Issue Type: New Feature (was: Bug) > Real-time Streaming submit streaming job failed for spark engine > > > Key: KYLIN-4008 > URL: https://issues.apache.org/jira/browse/KYLIN-4008 > Project: Kylin > Issue Type: New Feature > Components: Real-time Streaming >Affects Versions: v3.0.0-alpha >Reporter: zengrui >Priority: Major > Attachments: error.bmp > > > Create a Realtime Streaming Cube and the Cube Engine is Spark, when the > coordinator node receive a remoteStoreCompelete request and exist some > segments can build, the streaming job submit failed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-4008) Real-time Streaming submit streaming job failed for spark engine
[ https://issues.apache.org/jira/browse/KYLIN-4008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-4008: Priority: Minor (was: Major) > Real-time Streaming submit streaming job failed for spark engine > > > Key: KYLIN-4008 > URL: https://issues.apache.org/jira/browse/KYLIN-4008 > Project: Kylin > Issue Type: New Feature > Components: Real-time Streaming >Affects Versions: v3.0.0-alpha >Reporter: zengrui >Priority: Minor > Attachments: error.bmp > > > Create a Realtime Streaming Cube and the Cube Engine is Spark, when the > coordinator node receive a remoteStoreCompelete request and exist some > segments can build, the streaming job submit failed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-4008) Real-time Streaming submit streaming job failed for spark engine
[ https://issues.apache.org/jira/browse/KYLIN-4008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16844623#comment-16844623 ] XiaoXiang Yu edited comment on KYLIN-4008 at 5/21/19 8:30 AM: -- Hi , currently realtime-olap do not support spark build engine. Maybe it will be support later. So I think this should not be a bug, so I change this issue's type. was (Author: hit_lacus): Hi , currently realtime-olap do not support spark build engine. Maybe it will be support later. So I think this should not be a bug. > Real-time Streaming submit streaming job failed for spark engine > > > Key: KYLIN-4008 > URL: https://issues.apache.org/jira/browse/KYLIN-4008 > Project: Kylin > Issue Type: New Feature > Components: Real-time Streaming >Affects Versions: v3.0.0-alpha >Reporter: zengrui >Priority: Minor > Attachments: error.bmp > > > Create a Realtime Streaming Cube and the Cube Engine is Spark, when the > coordinator node receive a remoteStoreCompelete request and exist some > segments can build, the streaming job submit failed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
