[jira] [Updated] (KYLIN-4080) Project schema update event causes error reload NEW DataModelDesc
[ https://issues.apache.org/jira/browse/KYLIN-4080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Li updated KYLIN-4080: --- Summary: Project schema update event causes error reload NEW DataModelDesc (was: Project schema update event casues error reload NEW DataModelDesc) > Project schema update event causes error reload NEW DataModelDesc > - > > Key: KYLIN-4080 > URL: https://issues.apache.org/jira/browse/KYLIN-4080 > Project: Kylin > Issue Type: Bug > Components: Metadata >Affects Versions: v2.5.2 >Reporter: Yuzhang QIU >Assignee: Yuzhang QIU >Priority: Blocker > > Hi, dear Kylin dev team: >When create new DataModelDesc, DataModelManager.createDataModelDese:246 > will temporarily add the new model name into selected project(project1) > cache, but won't persist it. The TEMPORARY ADD operation will make the model > reloading successful, rather than throw "No project found for model ..." > exception(at ProjectManager:391). >However, If there have another threads are processing "Broadcasting > update project_schema, project1", it will clean up cache of project1 and > reload it, which will reset the "TEMPORARY ADD" operation. Meanwhile, the > model saving thread has persisted the DataModelDesc and start to reload it, > but will find there have "No project for this model". > The new model can't be created again because the conflict timestamp and > can't be reloaded into cache because the abrove problem. >How do you think about this?? > >Best regards > >yuzhang -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KYLIN-3552) Provide a sdk to support different data sources
[ https://issues.apache.org/jira/browse/KYLIN-3552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Li reassigned KYLIN-3552: -- Assignee: youcheng.zhang (was: Dong Li) > Provide a sdk to support different data sources > --- > > Key: KYLIN-3552 > URL: https://issues.apache.org/jira/browse/KYLIN-3552 > Project: Kylin > Issue Type: New Feature > Components: Query Engine, RDBMS Source >Reporter: youcheng.zhang >Assignee: youcheng.zhang >Priority: Minor > Fix For: v2.6.0 > > > Kylin already supports JDBC source, but it takes much efforts to develop an > implementation to a new source engine, like supporting metadata sync, cube > build and query pushdown. It’s mainly because the SQL dialects and jdbc > implementations between source engines are quite different. > We propose to develop a data source sdk, which provides APIs to help > developers handle these dialect differences and easily implement an adaptor > for one source engine. With this SDK, users can achieve followings from a > JDBC source: > # synchronize metadata and data from jdbc source. > # Build cube from jdbc source. > # query pushdown to jdbc source engine when cube is unmatched. > The SDK will define some interfaces of adaptors. To implement a new source, > developers only need to implement an adaptor for the source. > To keep compatibly, we will keep the existing JDBC source module, and > implement a new JDBC source module as the caller of the SDK. Besides, we will > also provide a pushdown implementation class as the caller of SDK for > pushdown scenarios. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KYLIN-3552) Provide a sdk to support different data sources
[ https://issues.apache.org/jira/browse/KYLIN-3552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Li reassigned KYLIN-3552: -- Assignee: Dong Li > Provide a sdk to support different data sources > --- > > Key: KYLIN-3552 > URL: https://issues.apache.org/jira/browse/KYLIN-3552 > Project: Kylin > Issue Type: New Feature > Components: Query Engine, RDBMS Source >Reporter: youcheng.zhang >Assignee: Dong Li >Priority: Minor > Fix For: v2.6.0 > > > Kylin already supports JDBC source, but it takes much efforts to develop an > implementation to a new source engine, like supporting metadata sync, cube > build and query pushdown. It’s mainly because the SQL dialects and jdbc > implementations between source engines are quite different. > We propose to develop a data source sdk, which provides APIs to help > developers handle these dialect differences and easily implement an adaptor > for one source engine. With this SDK, users can achieve followings from a > JDBC source: > # synchronize metadata and data from jdbc source. > # Build cube from jdbc source. > # query pushdown to jdbc source engine when cube is unmatched. > The SDK will define some interfaces of adaptors. To implement a new source, > developers only need to implement an adaptor for the source. > To keep compatibly, we will keep the existing JDBC source module, and > implement a new JDBC source module as the caller of the SDK. Besides, we will > also provide a pushdown implementation class as the caller of SDK for > pushdown scenarios. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3552) Provide a sdk to support different data sources
[ https://issues.apache.org/jira/browse/KYLIN-3552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Li updated KYLIN-3552: --- Description: Kylin already supports JDBC source, but it takes much efforts to develop an implementation to a new source engine, like supporting metadata sync, cube build and query pushdown. It’s mainly because the SQL dialects and jdbc implementations between source engines are quite different. We propose to develop a data source sdk, which provides APIs to help developers handle these dialect differences and easily implement an adaptor for one source engine. With this SDK, users can achieve followings from a JDBC source: # synchronize metadata and data from jdbc source. # Build cube from jdbc source. # query pushdown to jdbc source engine when cube is unmatched. The SDK will define some interfaces of adaptors. To implement a new source, developers only need to implement an adaptor for the source. To keep compatibly, we will keep the existing JDBC source module, and implement a new JDBC source module as the caller of the SDK. Besides, we will also provide a pushdown implementation class as the caller of SDK for pushdown scenarios. was: Kylin already supports JDBC source, but it takes much efforts to develop an implementation to a new source engine, like supporting metadata sync, cube build and query pushdown. It’s mainly because the SQL dialects and jdbc implementations between source engines are quite different. We propose to develop a data source sdk, which provides APIs to help developers handle these dialect differences and easily implement an adaptor for one source engine. With this SDK, users can achieve followings from a JDBC source: # synchronize metadata and data from jdbc source. # Build cube from jdbc source. # query pushdown to jdbc source engine when cube is unmatched. The SDK will define some interfaces of adaptors. To implement a new source, developers only need to implement an adaptor for the source. To keep compatibly, we will keep the existing JDBC sources, and implement a new JDBC source as the caller of the SDK. Besides, we will also provide a pushdown implementation class as the caller of SDK for pushdown scenarios. > Provide a sdk to support different data sources > --- > > Key: KYLIN-3552 > URL: https://issues.apache.org/jira/browse/KYLIN-3552 > Project: Kylin > Issue Type: New Feature > Components: Query Engine, RDBMS Source >Reporter: youcheng.zhang >Priority: Minor > Fix For: v2.6.0 > > > Kylin already supports JDBC source, but it takes much efforts to develop an > implementation to a new source engine, like supporting metadata sync, cube > build and query pushdown. It’s mainly because the SQL dialects and jdbc > implementations between source engines are quite different. > We propose to develop a data source sdk, which provides APIs to help > developers handle these dialect differences and easily implement an adaptor > for one source engine. With this SDK, users can achieve followings from a > JDBC source: > # synchronize metadata and data from jdbc source. > # Build cube from jdbc source. > # query pushdown to jdbc source engine when cube is unmatched. > The SDK will define some interfaces of adaptors. To implement a new source, > developers only need to implement an adaptor for the source. > To keep compatibly, we will keep the existing JDBC source module, and > implement a new JDBC source module as the caller of the SDK. Besides, we will > also provide a pushdown implementation class as the caller of SDK for > pushdown scenarios. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3503) Missing java.util.logging.config.file when starting kylin instance
[ https://issues.apache.org/jira/browse/KYLIN-3503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16590063#comment-16590063 ] Dong Li commented on KYLIN-3503: Do you mean catalina logs should go to tomcat/logs? It should be better. +1 for the fix > Missing java.util.logging.config.file when starting kylin instance > -- > > Key: KYLIN-3503 > URL: https://issues.apache.org/jira/browse/KYLIN-3503 > Project: Kylin > Issue Type: Improvement > Components: Web >Reporter: Zhong Yanghong >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3254) Cannot use Kylin Cubes with Excel
[ https://issues.apache.org/jira/browse/KYLIN-3254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16521862#comment-16521862 ] Dong Li commented on KYLIN-3254: Hi Jean, According to http://kylin.apache.org/docs23/tutorial/powerbi.html, you only need ODBC Driver to connect Apache Kylin from Excel, and PowerBI is not requred. And Excel 2013 Professional should be supported. Could you share more information or screenshots for the failure? Thanks! > Cannot use Kylin Cubes with Excel > - > > Key: KYLIN-3254 > URL: https://issues.apache.org/jira/browse/KYLIN-3254 > Project: Kylin > Issue Type: Bug > Components: Driver - ODBC >Affects Versions: v2.2.0 >Reporter: Jean-Luc BELLIER >Assignee: Dong Li >Priority: Critical > > Hello > I downloaded the ODBC driver to use Kylin with Excel. My OS is Windows 8.1 > 64-bits > I first clicked on the KylinODBCDriver (x64).exe, but it says that the > program could not be executed on my PC. > So i ran the 32-bit version. I succeeded in configuring the Kylin ODBC > connection. > > Now I would like to use my Kylin cubes with Excel. So I downloaded PowerBI as > mentioned on Kylin site, but the installation program tells me that my > version of Excel was not compatible, because the driver requires Excel2010 > SP1 or Excel 2013. > My version of Excel is 2013 Professional version. > > Are there any limitations ? How can I visualize my Kylin cubes with Excel in > my case ? > > Thank you in advance for your help. > > Best regards, > Jean-Luc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3354) KeywordDefaultDirtyHack cannot handle double-quoted defaultCatalog identifier
[
https://issues.apache.org/jira/browse/KYLIN-3354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Li resolved KYLIN-3354.
Resolution: Fixed
Fix Version/s: v2.4.0
> KeywordDefaultDirtyHack cannot handle double-quoted defaultCatalog identifier
> -
>
> Key: KYLIN-3354
> URL: https://issues.apache.org/jira/browse/KYLIN-3354
> Project: Kylin
> Issue Type: Improvement
>Reporter: Dong Li
>Assignee: Dong Li
>Priority: Major
> Fix For: v2.4.0
>
>
> for example, following SQL cannot be escaped by KeywordDefaultDirtyHack:
> {quote}
> select count(*) from "defaultCatalog"."DEFAULT"."TABLE"
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (KYLIN-3363) Wrong partition condition appended in JDBC Source
[ https://issues.apache.org/jira/browse/KYLIN-3363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Li resolved KYLIN-3363. Resolution: Fixed Fix Version/s: v2.4.0 > Wrong partition condition appended in JDBC Source > - > > Key: KYLIN-3363 > URL: https://issues.apache.org/jira/browse/KYLIN-3363 > Project: Kylin > Issue Type: Bug > Components: RDBMS Source >Affects Versions: v2.3.1 >Reporter: Dong Li >Assignee: Dong Li >Priority: Minor > Fix For: v2.4.0 > > > create a cube, and select the partition column from lookup table, then select > shard by column from fact table. > then an unexecutable sql will be created to build the flatten table with > sqoop: > select ... from [FACT] where [LOOKUP].[PARTITION_COL] > ... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3345) Use Apache Parent POM 19
[ https://issues.apache.org/jira/browse/KYLIN-3345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Li resolved KYLIN-3345. Resolution: Fixed Fix Version/s: v2.4.0 > Use Apache Parent POM 19 > > > Key: KYLIN-3345 > URL: https://issues.apache.org/jira/browse/KYLIN-3345 > Project: Kylin > Issue Type: Improvement > Components: Tools, Build and Test >Reporter: Ted Yu >Assignee: Dong Li >Priority: Major > Fix For: v2.4.0 > > > Kylin is still using Apache Parent POM 16. Apache Parent POM 19 is out. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3380) Allow to configure sqoop for jdbc source with a kylin_sqoop_conf.xml like hive
[ https://issues.apache.org/jira/browse/KYLIN-3380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Li resolved KYLIN-3380. Resolution: Fixed Fix Version/s: v2.4.0 > Allow to configure sqoop for jdbc source with a kylin_sqoop_conf.xml like hive > -- > > Key: KYLIN-3380 > URL: https://issues.apache.org/jira/browse/KYLIN-3380 > Project: Kylin > Issue Type: Improvement > Components: RDBMS Source >Affects Versions: v2.3.1 >Reporter: Dong Li >Assignee: Dong Li >Priority: Minor > Fix For: v2.4.0 > > > In hive source, users can config hive with a kylin_hive_conf.xml under conf > directory. > But for jdbc source, where we use sqoop to make flatten table, users cannot > customize the configuration according to the real environment. To fix this, > we can have a kylin_sqoop_conf.xml file under conf directory, then users can > specify some configurations for sqoop. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3380) Allow to configure sqoop for jdbc source with a kylin_sqoop_conf.xml like hive
Dong Li created KYLIN-3380: -- Summary: Allow to configure sqoop for jdbc source with a kylin_sqoop_conf.xml like hive Key: KYLIN-3380 URL: https://issues.apache.org/jira/browse/KYLIN-3380 Project: Kylin Issue Type: Improvement Components: RDBMS Source Affects Versions: v2.3.1 Reporter: Dong Li In hive source, users can config hive with a kylin_hive_conf.xml under conf directory. But for jdbc source, where we use sqoop to make flatten table, users cannot customize the configuration according to the real environment. To fix this, we can have a kylin_sqoop_conf.xml file under conf directory, then users can specify some configurations for sqoop. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KYLIN-3363) Wrong partition condition appended in JDBC Source
[ https://issues.apache.org/jira/browse/KYLIN-3363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Li reassigned KYLIN-3363: -- Assignee: Dong Li > Wrong partition condition appended in JDBC Source > - > > Key: KYLIN-3363 > URL: https://issues.apache.org/jira/browse/KYLIN-3363 > Project: Kylin > Issue Type: Bug > Components: RDBMS Source >Affects Versions: v2.3.1 >Reporter: Dong Li >Assignee: Dong Li >Priority: Minor > > create a cube, and select the partition column from lookup table, then select > shard by column from fact table. > then an unexecutable sql will be created to build the flatten table with > sqoop: > select ... from [FACT] where [LOOKUP].[PARTITION_COL] > ... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3363) Wrong partition condition appended in JDBC Source
[ https://issues.apache.org/jira/browse/KYLIN-3363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Li updated KYLIN-3363: --- Issue Type: Bug (was: Improvement) > Wrong partition condition appended in JDBC Source > - > > Key: KYLIN-3363 > URL: https://issues.apache.org/jira/browse/KYLIN-3363 > Project: Kylin > Issue Type: Bug > Components: RDBMS Source >Affects Versions: v2.3.1 >Reporter: Dong Li >Priority: Minor > > create a cube, and select the partition column from lookup table, then select > shard by column from fact table. > then an unexecutable sql will be created to build the flatten table with > sqoop: > select ... from [FACT] where [LOOKUP].[PARTITION_COL] > ... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3363) Wrong partition condition appended in JDBC Source
[ https://issues.apache.org/jira/browse/KYLIN-3363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Li updated KYLIN-3363: --- Affects Version/s: v2.3.1 > Wrong partition condition appended in JDBC Source > - > > Key: KYLIN-3363 > URL: https://issues.apache.org/jira/browse/KYLIN-3363 > Project: Kylin > Issue Type: Bug > Components: RDBMS Source >Affects Versions: v2.3.1 >Reporter: Dong Li >Priority: Minor > > create a cube, and select the partition column from lookup table, then select > shard by column from fact table. > then an unexecutable sql will be created to build the flatten table with > sqoop: > select ... from [FACT] where [LOOKUP].[PARTITION_COL] > ... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3351) Cube Planner not working in apche kylin 2.3.0(open Source)
[ https://issues.apache.org/jira/browse/KYLIN-3351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16460653#comment-16460653 ] Dong Li commented on KYLIN-3351: Please refer to http://kylin.apache.org/community/ You need to subscribe to the user mailing list firstly, and then send your questions to u...@kylin.apache.org. To learn more about mailing list, please refer to http://www.apache.org/foundation/mailinglists.html > Cube Planner not working in apche kylin 2.3.0(open Source) > -- > > Key: KYLIN-3351 > URL: https://issues.apache.org/jira/browse/KYLIN-3351 > Project: Kylin > Issue Type: Task >Reporter: praveenece >Priority: Major > > Hi Team > i want test Cube planner in apache-kylin(2.3.0),So i created cube with > segment, and i hit query to cube many times more than thousands but the cube > planner there is no change in cuboid level like color changing and not get > COUNT in exactly row count and other items.can u please guide me. > Note: > Could u tell me? > Cube Planner Working only old Cube (like 3 month before) or new Cube also. > Configuration(kylin.Properties) > kylin.cube.cubeplanner.enabled=true > Kylin Version=2.3.0(SINGLE NODE) > MyCube creation =10days before > I referred: http://kylin.apache.org/docs23/howto/howto_use_cube_planner.html > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3297) sql在解析>大值 and <小值的时候，kylin出现内存溢出。
[ https://issues.apache.org/jira/browse/KYLIN-3297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Li updated KYLIN-3297: --- Component/s: (was: RDBMS Source) Query Engine > sql在解析>大值 and <小值的时候，kylin出现内存溢出。 > - > > Key: KYLIN-3297 > URL: https://issues.apache.org/jira/browse/KYLIN-3297 > Project: Kylin > Issue Type: Bug > Components: Query Engine >Affects Versions: v2.2.0 > Environment: oracle linux 6.8 > cup 8C > memory 48G > apache kylin v2.2.0 >Reporter: 陈家宇 >Priority: Major > > select f.plant_code,f.report_business_area,f.orderby,sum(f.a)a, > case when sum(f.a)=0 then 0 else sum(f.qty)/sum(f.a) end zt from ( > select h.business_area_desc,h.plant_code, > case when p.report_business_area like '%切大片%' then '切大片' > else p.report_business_area end report_business_area, > case when p.report_business_area like '%切大片%' then '00' > else p.orderby end orderby, > sum(p.quantity) qty,sum(a)a from pkn_kpi p > inner join hcm_area h on p.plant_id=h.plant_id > where 1=1 > and h.business_area_desc='上海' > and h.plant_code='S1' > and p.calendar_day>='2018-03-19' > and p.calendar_day<='2018-03-18' > and trim(p.report_business_area) is not null > group by h.business_area_desc,h.plant_code,p.report_business_area,p.orderby > ) f > group by f.plant_code,f.report_business_area,f.orderby > order by orderby > 以上的calendar_day条件，顺序输入反了，变成calendar_day>大值 and > calendar_day<小值，出现了kylin内存溢出，并且崩溃。正常这种情况是没有数据输出才对。 > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3354) KeywordDefaultDirtyHack cannot handle double-quoted defaultCatalog identifier
[
https://issues.apache.org/jira/browse/KYLIN-3354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459489#comment-16459489
]
Dong Li commented on KYLIN-3354:
Hi Alberto,
Actually it's no duplicated.
This JIRA is to enhance KeywordDefaultDirtyHack class to remove
"defaultCatalog" from all SQL statements to compromise with some BI
applications, including query pushdown and accessing cube.
For KYLIN-3339, it's only about query pushdown.
> KeywordDefaultDirtyHack cannot handle double-quoted defaultCatalog identifier
> -
>
> Key: KYLIN-3354
> URL: https://issues.apache.org/jira/browse/KYLIN-3354
> Project: Kylin
> Issue Type: Improvement
>Reporter: Dong Li
>Assignee: Dong Li
>Priority: Major
>
> for example, following SQL cannot be escaped by KeywordDefaultDirtyHack:
> {quote}
> select count(*) from "defaultCatalog"."DEFAULT"."TABLE"
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3339) Decoupling SQL keywords set with specify implementation in query pushdown
[ https://issues.apache.org/jira/browse/KYLIN-3339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Li updated KYLIN-3339: --- Summary: Decoupling SQL keywords set with specify implementation in query pushdown (was: Decoupling SQL keywords set with specify implementation.) > Decoupling SQL keywords set with specify implementation in query pushdown > - > > Key: KYLIN-3339 > URL: https://issues.apache.org/jira/browse/KYLIN-3339 > Project: Kylin > Issue Type: Improvement >Reporter: jiatao.tao >Assignee: jiatao.tao >Priority: Trivial > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KYLIN-3354) KeywordDefaultDirtyHack cannot handle double-quoted defaultCatalog identifier
[
https://issues.apache.org/jira/browse/KYLIN-3354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Li reassigned KYLIN-3354:
--
Assignee: Dong Li
> KeywordDefaultDirtyHack cannot handle double-quoted defaultCatalog identifier
> -
>
> Key: KYLIN-3354
> URL: https://issues.apache.org/jira/browse/KYLIN-3354
> Project: Kylin
> Issue Type: Improvement
>Reporter: Dong Li
>Assignee: Dong Li
>Priority: Major
>
> for example, following SQL cannot be escaped by KeywordDefaultDirtyHack:
> {quote}
> select count(*) from "defaultCatalog"."DEFAULT"."TABLE"
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3354) KeywordDefaultDirtyHack cannot handle double-quoted defaultCatalog identifier
[ https://issues.apache.org/jira/browse/KYLIN-3354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Li updated KYLIN-3354: --- Description: for example, following SQL cannot be escaped by KeywordDefaultDirtyHack: ``` select count(*) from "defaultCatalog"."DEFAULT"."TABLE" ``` was: for example, following SQL cannot be escaped by KeywordDefaultDirtyHack: select count(*) from "defaultCatalog"."DEFAULT"."TABLE" > KeywordDefaultDirtyHack cannot handle double-quoted defaultCatalog identifier > - > > Key: KYLIN-3354 > URL: https://issues.apache.org/jira/browse/KYLIN-3354 > Project: Kylin > Issue Type: Improvement >Reporter: Dong Li >Priority: Major > > for example, following SQL cannot be escaped by KeywordDefaultDirtyHack: > ``` > select count(*) from "defaultCatalog"."DEFAULT"."TABLE" > ``` -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3354) KeywordDefaultDirtyHack cannot handle double-quoted defaultCatalog identifier
Dong Li created KYLIN-3354: -- Summary: KeywordDefaultDirtyHack cannot handle double-quoted defaultCatalog identifier Key: KYLIN-3354 URL: https://issues.apache.org/jira/browse/KYLIN-3354 Project: Kylin Issue Type: Improvement Reporter: Dong Li for example, following SQL cannot be escaped by KeywordDefaultDirtyHack: select count(*) from "defaultCatalog"."DEFAULT"."TABLE" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KYLIN-3345) Use Apache Parent POM 19
[ https://issues.apache.org/jira/browse/KYLIN-3345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Li reassigned KYLIN-3345: -- Assignee: Dong Li > Use Apache Parent POM 19 > > > Key: KYLIN-3345 > URL: https://issues.apache.org/jira/browse/KYLIN-3345 > Project: Kylin > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Dong Li >Priority: Major > > Kylin is still using Apache Parent POM 16. Apache Parent POM 19 is out. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3327) Upgrade surefire version to 2.21.0
[ https://issues.apache.org/jira/browse/KYLIN-3327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Li resolved KYLIN-3327. Resolution: Fixed Fix Version/s: v2.4.0 > Upgrade surefire version to 2.21.0 > -- > > Key: KYLIN-3327 > URL: https://issues.apache.org/jira/browse/KYLIN-3327 > Project: Kylin > Issue Type: Task >Reporter: Ted Yu >Assignee: Dong Li >Priority: Major > Fix For: v2.4.0 > > > Currently we are using 2.19.1. > We should upgrade to 2.21.0 which contains SUREFIRE-1422. > This would make unit testing more robust on Apache Infra. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3323) Allow debugTomcat add more arguments
[ https://issues.apache.org/jira/browse/KYLIN-3323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16421938#comment-16421938 ] Dong Li commented on KYLIN-3323: Hi Yongjie, if you want to set system properties in IDE for debugging, maybe you can directly use JVM opts, such as "-Dkylin.job.use-remote-cli=true", then this patch seems not very valuable. > Allow debugTomcat add more arguments > > > Key: KYLIN-3323 > URL: https://issues.apache.org/jira/browse/KYLIN-3323 > Project: Kylin > Issue Type: Improvement >Reporter: yongjie zhao >Assignee: yongjie zhao >Priority: Minor > Attachments: Screen Shot 2018-03-28 at 3.05.00 PM.png > > > On current, debugTomcat only accepts port argument. I must edit > kylin.property for execute remote command, such as > kylin.job.use-remote-cli=true. This issue can be passed arguments for > debugTomcat -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KYLIN-3327) Upgrade surefire version to 2.21.0
[ https://issues.apache.org/jira/browse/KYLIN-3327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Li reassigned KYLIN-3327: -- Assignee: Dong Li > Upgrade surefire version to 2.21.0 > -- > > Key: KYLIN-3327 > URL: https://issues.apache.org/jira/browse/KYLIN-3327 > Project: Kylin > Issue Type: Task >Reporter: Ted Yu >Assignee: Dong Li >Priority: Major > > Currently we are using 2.19.1. > We should upgrade to 2.21.0 which contains SUREFIRE-1422. > This would make unit testing more robust on Apache Infra. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3315) allow each project to set its own source in project level override configuration
Dong Li created KYLIN-3315: -- Summary: allow each project to set its own source in project level override configuration Key: KYLIN-3315 URL: https://issues.apache.org/jira/browse/KYLIN-3315 Project: Kylin Issue Type: Improvement Components: Metadata Reporter: Dong Li Currently, all projects connect to the same source which is set in kylin.properties with kylin.source.default property. It's better to allow each project to set its own source in project level override configuration. As the result, we can have project A connects to JDBC, and project B connects to Hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3288) "Sqoop To Flat Hive Table" step should specify "mapreduce.queue.name"
[ https://issues.apache.org/jira/browse/KYLIN-3288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Li resolved KYLIN-3288. Resolution: Fixed > "Sqoop To Flat Hive Table" step should specify "mapreduce.queue.name" > - > > Key: KYLIN-3288 > URL: https://issues.apache.org/jira/browse/KYLIN-3288 > Project: Kylin > Issue Type: Bug > Components: RDBMS Source >Reporter: Shaoxiong Zhan >Assignee: Shaoxiong Zhan >Priority: Major > Fix For: v2.3.1 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3270) Add integration test for Optimize Job
[ https://issues.apache.org/jira/browse/KYLIN-3270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407467#comment-16407467 ] Dong Li commented on KYLIN-3270: I did a "git am" on master branch, but not tried on 2.3.x branch. > Add integration test for Optimize Job > - > > Key: KYLIN-3270 > URL: https://issues.apache.org/jira/browse/KYLIN-3270 > Project: Kylin > Issue Type: Sub-task > Components: Tools, Build and Test >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Critical > Fix For: v2.3.1 > > Attachments: APACHE-KYLIN-3270.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3257) Useless call in FuzzyValueCombination
[
https://issues.apache.org/jira/browse/KYLIN-3257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Li resolved KYLIN-3257.
Resolution: Fixed
Fix Version/s: v2.4.0
> Useless call in FuzzyValueCombination
> -
>
> Key: KYLIN-3257
> URL: https://issues.apache.org/jira/browse/KYLIN-3257
> Project: Kylin
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Wenzheng Liu
>Priority: Minor
> Fix For: v2.4.0
>
>
> {code}
> public static List> calculate(Map
> fuzzyValues, long cap) {
> Collections.emptyMap();
> {code}
> The empty map has no effect to method execution.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3257) Useless call in FuzzyValueCombination
[
https://issues.apache.org/jira/browse/KYLIN-3257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400340#comment-16400340
]
Dong Li commented on KYLIN-3257:
Thanks Wenzheng. Please go ahead.
> Useless call in FuzzyValueCombination
> -
>
> Key: KYLIN-3257
> URL: https://issues.apache.org/jira/browse/KYLIN-3257
> Project: Kylin
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Wenzheng Liu
>Priority: Minor
>
> {code}
> public static List> calculate(Map
> fuzzyValues, long cap) {
> Collections.emptyMap();
> {code}
> The empty map has no effect to method execution.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (KYLIN-3257) Useless call in FuzzyValueCombination
[
https://issues.apache.org/jira/browse/KYLIN-3257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Li reassigned KYLIN-3257:
--
Assignee: Wenzheng Liu
> Useless call in FuzzyValueCombination
> -
>
> Key: KYLIN-3257
> URL: https://issues.apache.org/jira/browse/KYLIN-3257
> Project: Kylin
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Wenzheng Liu
>Priority: Minor
>
> {code}
> public static List> calculate(Map
> fuzzyValues, long cap) {
> Collections.emptyMap();
> {code}
> The empty map has no effect to method execution.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3270) Add integration test for Optimize Job
[ https://issues.apache.org/jira/browse/KYLIN-3270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16398177#comment-16398177 ] Dong Li commented on KYLIN-3270: +1 LGTM But the patch cannot be applied because a rebase is needed... [~yaho] could you do the rebase and update the patch? > Add integration test for Optimize Job > - > > Key: KYLIN-3270 > URL: https://issues.apache.org/jira/browse/KYLIN-3270 > Project: Kylin > Issue Type: Sub-task > Components: Tools, Build and Test >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Critical > Fix For: v2.3.1 > > Attachments: APACHE-KYLIN-3270.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-2913) Enable job retry for configurable exceptions
[ https://issues.apache.org/jira/browse/KYLIN-2913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Li resolved KYLIN-2913. Resolution: Fixed this issue has been fixed at *https://issues.apache.org/jira/browse/KYLIN-3263* > Enable job retry for configurable exceptions > > > Key: KYLIN-2913 > URL: https://issues.apache.org/jira/browse/KYLIN-2913 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Affects Versions: v2.1.0 >Reporter: Wang, Gang >Assignee: Wang, Gang >Priority: Minor > Fix For: v2.3.0 > > Attachments: > 0001-KYLIN-2913-Enable-job-retry-for-configurable-excepti.patch > > > In our production environment, we always get some certain exceptions from > Hadoop or HBase, like > "org.apache.kylin.job.exception.NoEnoughReplicationException", > "java.util.ConcurrentModificationException", which results in job failure. > While, these exceptions can be handled by retry actually. So, it will be much > more convenient if we are able to make job retry on some configurable > exceptions. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-2913) Enable job retry for configurable exceptions
[ https://issues.apache.org/jira/browse/KYLIN-2913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16374047#comment-16374047 ] Dong Li edited comment on KYLIN-2913 at 2/23/18 7:55 AM: - Found a critical bug on this patch, if user set job retry number as 1, and kylin.job.retry-exception-classes as empty as default, then all job steps will run 2 times even if the step succeeded, and all Merge job will fail. [~gwang3] please have a look. thanks! was (Author: lidong_sjtu): Found a critical bug on this patch, if user set job retry number as 1, and kylin.job.retry-exception-classes as empty as default, then all job steps will run 2 times, and Merge job will fail. [~gwang3] please have a look. thanks! > Enable job retry for configurable exceptions > > > Key: KYLIN-2913 > URL: https://issues.apache.org/jira/browse/KYLIN-2913 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Affects Versions: v2.1.0 >Reporter: Wang, Gang >Assignee: Wang, Gang >Priority: Minor > Fix For: v2.3.0 > > Attachments: > 0001-KYLIN-2913-Enable-job-retry-for-configurable-excepti.patch > > > In our production environment, we always get some certain exceptions from > Hadoop or HBase, like > "org.apache.kylin.job.exception.NoEnoughReplicationException", > "java.util.ConcurrentModificationException", which results in job failure. > While, these exceptions can be handled by retry actually. So, it will be much > more convenient if we are able to make job retry on some configurable > exceptions. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3135) Fix regular expression bug in SQL comments
[
https://issues.apache.org/jira/browse/KYLIN-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Li resolved KYLIN-3135.
Resolution: Fixed
Patch merged to master and 2.3.x branches. Thanks!
> Fix regular expression bug in SQL comments
> --
>
> Key: KYLIN-3135
> URL: https://issues.apache.org/jira/browse/KYLIN-3135
> Project: Kylin
> Issue Type: Bug
>Reporter: hahayuan
>Assignee: hahayuan
>Priority: Major
> Fix For: v2.3.0
>
> Attachments: 0001-KYLIN-3135.patch,
> 0001-kylin-3135-revert-first-regex-too.patch,
> 0002-KYLIN-3135-fix-regEx.patch, multi_line_comments.PNG,
> one_line_comments.PNG
>
>
> Hi,all.
> Recently,I was testing query function of kylin,
> sometimes I just comment with /**/ instead of delete the sql,cause I need to
> query and compare again.
> And I was confused that the results says it was "No Support Sql",but it can
> query success without comments.
> For example,
> {code:java}
> /*
> select count(*) from kylin_sales;
> */
> select * from kylin_sales;
> {code}
> So I view the code and find the commentPatterns of /\**/ was
> {code:java}
> /\\*[^\\*/]*
> {code}
> ,clearly it was wrong.
> The regular expression of [abc] means any character in abc,such as a or b.
> So the [^\\*/] means that * or / can't appear,
> But under this circumstances the */ need to be as a string not separated
> character.
> the */ can't appear not * or / can't appear.
> I rewrite the regular expression,
> {code:java}
> /\\*[\\s\\S]*?\\*/
> {code}
> if you think it's necessary to change the old code,please review and replace
> it.
> Thank for you time.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (KYLIN-3224) data can't show when use kylin pushdown model
[ https://issues.apache.org/jira/browse/KYLIN-3224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Li resolved KYLIN-3224. Resolution: Fixed Thanks! PR merged to master and 2.3.x branches. > data can't show when use kylin pushdown model > -- > > Key: KYLIN-3224 > URL: https://issues.apache.org/jira/browse/KYLIN-3224 > Project: Kylin > Issue Type: Bug > Components: Query Engine, Web >Affects Versions: v2.2.0, v2.3.0 >Reporter: peng.jianhua >Assignee: peng.jianhua >Priority: Major > Fix For: v2.3.0 > > Attachments: 0001-KYLIN-3224.patch, 01.PNG, 02.PNG, 03.PNG, 04.PNG, > KYLIN-3224.patch, KYLIN-3224.png > > > select * from kylin_sales > use pushdown model,and the result shows like 01.png -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Reopened] (KYLIN-3135) Fix regular expression bug in SQL comments
[
https://issues.apache.org/jira/browse/KYLIN-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Li reopened KYLIN-3135:
> Fix regular expression bug in SQL comments
> --
>
> Key: KYLIN-3135
> URL: https://issues.apache.org/jira/browse/KYLIN-3135
> Project: Kylin
> Issue Type: Bug
>Reporter: hahayuan
>Assignee: hahayuan
>Priority: Major
> Fix For: v2.3.0
>
> Attachments: 0001-KYLIN-3135.patch,
> 0001-kylin-3135-revert-first-regex-too.patch,
> 0002-KYLIN-3135-fix-regEx.patch, multi_line_comments.PNG,
> one_line_comments.PNG
>
>
> Hi,all.
> Recently,I was testing query function of kylin,
> sometimes I just comment with /**/ instead of delete the sql,cause I need to
> query and compare again.
> And I was confused that the results says it was "No Support Sql",but it can
> query success without comments.
> For example,
> {code:java}
> /*
> select count(*) from kylin_sales;
> */
> select * from kylin_sales;
> {code}
> So I view the code and find the commentPatterns of /\**/ was
> {code:java}
> /\\*[^\\*/]*
> {code}
> ,clearly it was wrong.
> The regular expression of [abc] means any character in abc,such as a or b.
> So the [^\\*/] means that * or / can't appear,
> But under this circumstances the */ need to be as a string not separated
> character.
> the */ can't appear not * or / can't appear.
> I rewrite the regular expression,
> {code:java}
> /\\*[\\s\\S]*?\\*/
> {code}
> if you think it's necessary to change the old code,please review and replace
> it.
> Thank for you time.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-2978) add SQL's execution plan feature
[ https://issues.apache.org/jira/browse/KYLIN-2978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Li updated KYLIN-2978: --- Component/s: (was: Driver - JDBC) Query Engine > add SQL's execution plan feature > > > Key: KYLIN-2978 > URL: https://issues.apache.org/jira/browse/KYLIN-2978 > Project: Kylin > Issue Type: New Feature > Components: Query Engine >Affects Versions: v2.1.0 >Reporter: cnwangdp >Priority: Minor > Labels: features > Fix For: Future > > > SQL's execution plan is import to performance analysis; -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-2253) sql union did not remove duplicated records (distinct)
[ https://issues.apache.org/jira/browse/KYLIN-2253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Li updated KYLIN-2253: --- Component/s: (was: Driver - JDBC) Query Engine > sql union did not remove duplicated records (distinct) > --- > > Key: KYLIN-2253 > URL: https://issues.apache.org/jira/browse/KYLIN-2253 > Project: Kylin > Issue Type: Bug > Components: Query Engine >Affects Versions: v1.5.4.1 > Environment: apache-kylin-1.6.0-hbase1.x-bin.tar.gz >Reporter: zhou degao >Priority: Critical > > sql like following: > select "LOOKUP_TIME_BY_DAY"."THE_YEAR" as "c0", > "LOOKUP_TIME_BY_DAY"."MONTH_OF_YEAR" as "c1" from "VCGBI_FACT_SALE" as > "VCGBI_FACT_SALE" join "LOOKUP_TIME_BY_DAY" as "LOOKUP_TIME_BY_DAY" on > "VCGBI_FACT_SALE"."DEAL_TIME" = "LOOKUP_TIME_BY_DAY"."THE_DATE" group by > "LOOKUP_TIME_BY_DAY"."THE_YEAR", "LOOKUP_TIME_BY_DAY"."MONTH_OF_YEAR" > union > select "LOOKUP_TIME_BY_DAY"."THE_YEAR" as "c0", > "LOOKUP_TIME_BY_DAY"."MONTH_OF_YEAR" as "c1" from > "VCGBI_FACT_PERSON_MISC_DATA" as "VCGBI_FACT_PERSON_MISC_DATA" join > "LOOKUP_TIME_BY_DAY" as "LOOKUP_TIME_BY_DAY" on > "VCGBI_FACT_PERSON_MISC_DATA"."TIME" = "LOOKUP_TIME_BY_DAY"."THE_DATE" group > by "LOOKUP_TIME_BY_DAY"."THE_YEAR", "LOOKUP_TIME_BY_DAY"."MONTH_OF_YEAR" > order by 1 ASC, 2 ASC > I got result like following: > 2016 1 > 2016 1 > 2016 2 > 2016 2 > 2016 3 > 2016 3 > 2016 4 > 2016 4 > 2016 5 > 2016 5 > 2016 6 > 2016 6 > 2016 7 > 2016 7 > 2016 8 > 2016 8 > 2016 9 > 2016 9 > 2016 10 > 2016 11 > 2016 12 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-3225) java.lang.ClassNotFoundException: org.apache.kylin.jdbc.Driver
[ https://issues.apache.org/jira/browse/KYLIN-3225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Li closed KYLIN-3225. -- Resolution: Won't Fix Hi Bowen, Please make sure kylin-jdbc dependency is included in your classpath. For example, try to modify the scope of kylin-jdbc from TEST to COMPILE in your pom.xml file. And, Jira is for issues/features reporting. For questions, please discuss in mailing list: [http://kylin.apache.org/community/] Thanks! > java.lang.ClassNotFoundException: org.apache.kylin.jdbc.Driver > -- > > Key: KYLIN-3225 > URL: https://issues.apache.org/jira/browse/KYLIN-3225 > Project: Kylin > Issue Type: Bug > Components: Driver - JDBC >Affects Versions: v2.2.0 >Reporter: Bowen Yang >Assignee: Dong Li >Priority: Major > Attachments: KylinClient.java, > WeChatWorkScreenshot_9a071c7d-18b3-445a-8f99-1bed111250b3.png, > WeChatWorkScreenshot_de991bc7-7635-49ab-a53b-08899cd7ef4b.png, pom.xml > > > Hi guys, > I am using jdbc to connect kylin which version is 2.2.0 to analyze something, > but I got "java.lang.ClassNotFoundException: org.apache.kylin.jdbc.Driver" > exception. > I tried 1.5.3 and 1.6.0 version, but this exception still happened. > Is it a bug in the kylin jdbc driver? The attachments are my pom, java code > and screenshots, please help to see this issue. Thanks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-657) JDBC Driver not register into DriverManager
[ https://issues.apache.org/jira/browse/KYLIN-657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Li closed KYLIN-657. - Assignee: Xiaoyu Wang (was: liyang) > JDBC Driver not register into DriverManager > --- > > Key: KYLIN-657 > URL: https://issues.apache.org/jira/browse/KYLIN-657 > Project: Kylin > Issue Type: Bug > Components: Driver - JDBC >Affects Versions: v0.6.5, v0.7.1 >Reporter: Xiaoyu Wang >Assignee: Xiaoyu Wang >Priority: Major > Fix For: v0.7.1 > > > The Driver not register into DriverManager > when use spring to manage the datasource pool with > org.apache.commons.dbcp.BasicDataSource > got exception: > Exception in thread "main" java.sql.SQLException: No suitable driver found > for jdbc:kylin://x.x.x.x/default > at java.sql.DriverManager.getConnection(DriverManager.java:596) > at java.sql.DriverManager.getConnection(DriverManager.java:187) > Pull Request: > https://github.com/KylinOLAP/Kylin/pull/452 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-643) JDBC couldn't connect to Kylin: "java.sql.SQLException: Authentication Failed"
[ https://issues.apache.org/jira/browse/KYLIN-643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Li closed KYLIN-643. - > JDBC couldn't connect to Kylin: "java.sql.SQLException: Authentication Failed" > -- > > Key: KYLIN-643 > URL: https://issues.apache.org/jira/browse/KYLIN-643 > Project: Kylin > Issue Type: Bug > Components: Driver - JDBC >Affects Versions: v0.7.1 >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI >Priority: Major > Fix For: v0.7.1 > > > Get authentication error at client side (I'm using eclipse as client): > java.sql.SQLException: Authentication Failed. > at org.apache.kylin.jdbc.Driver$1.onConnectionInit(Driver.java:116) > at > net.hydromatic.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:137) > at > org.eclipse.datatools.connectivity.drivers.jdbc.JDBCConnection.createConnection(JDBCConnection.java:328) > at > org.eclipse.datatools.connectivity.DriverConnectionBase.internalCreateConnection(DriverConnectionBase.java:105) > at > org.eclipse.datatools.connectivity.DriverConnectionBase.open(DriverConnectionBase.java:54) > at > org.eclipse.datatools.connectivity.drivers.jdbc.JDBCConnection.open(JDBCConnection.java:96) > at > org.eclipse.datatools.connectivity.drivers.jdbc.JDBCConnectionFactory.createConnection(JDBCConnectionFactory.java:53) > at > org.eclipse.datatools.connectivity.internal.ConnectionFactoryProvider.createConnection(ConnectionFactoryProvider.java:83) > at > org.eclipse.datatools.connectivity.internal.ConnectionProfile.createConnection(ConnectionProfile.java:359) > at > org.eclipse.datatools.connectivity.ui.PingJob.createTestConnection(PingJob.java:76) > at org.eclipse.datatools.connectivity.ui.PingJob.run(PingJob.java:59) > at org.eclipse.core.internal.jobs.Worker.run(Worker.java:54) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-595) Kylin JDBC driver should not assume Kylin server listen on either 80 or 443
[ https://issues.apache.org/jira/browse/KYLIN-595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Li closed KYLIN-595. - > Kylin JDBC driver should not assume Kylin server listen on either 80 or 443 > --- > > Key: KYLIN-595 > URL: https://issues.apache.org/jira/browse/KYLIN-595 > Project: Kylin > Issue Type: Bug > Components: Driver - JDBC >Affects Versions: v0.6.5 >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI >Priority: Major > Fix For: v0.7.1 > > > Kylin JDBC driver assumes the server is working on 80 or 443 port; When user > create a JDBC connection, it will check whether user specify the "ssl" > property with default value false; if false, it will connect to server with > http on port 80; if ssl=true, it will use https to connect the server on 443; > Kylin should not make such assumption. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-2880) Enhance BadQueryDetector to include query id
[ https://issues.apache.org/jira/browse/KYLIN-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Li resolved KYLIN-2880. Resolution: Fixed Fix Version/s: v2.3.0 Thanks Yanghong. Patch merged, see commit https://github.com/apache/kylin/commit/6a2ee80c9ec19be1d7e72ed8b587fbc48a195867 > Enhance BadQueryDetector to include query id > > > Key: KYLIN-2880 > URL: https://issues.apache.org/jira/browse/KYLIN-2880 > Project: Kylin > Issue Type: Improvement >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Major > Fix For: v2.3.0 > > Attachments: APACHE-KYLIN-2880.patch > > > To better correlate bad queries with the bottom physical execution > information, it's better to let BadQueryDetector include query id -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-2908) Add one option for migration tool to indicate whether to migrate segment data
[ https://issues.apache.org/jira/browse/KYLIN-2908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Li resolved KYLIN-2908. Resolution: Fixed Fix Version/s: v2.3.0 thanks yanghong! patch merged to master branch. https://github.com/apache/kylin/commit/c704f7cda79921c5a22645492a52c94734847541 > Add one option for migration tool to indicate whether to migrate segment data > - > > Key: KYLIN-2908 > URL: https://issues.apache.org/jira/browse/KYLIN-2908 > Project: Kylin > Issue Type: Improvement > Components: Tools, Build and Test >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Major > Fix For: v2.3.0 > > Attachments: APACHE-KYLIN-2908.patch > > > Current migration tool will migrate segment data to the destination. However, > in some cases, we only want to migrate cube schema without segment data. For > example, when we do some compatibility testing for upgrade. This can also > avoid two env refer the same htable, which may leading to some issue when dst > env do some fresh to delete the htable which src env still refers. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3204) Potentially unclosed resources in JdbcExplorer#evalQueryMetadata
[
https://issues.apache.org/jira/browse/KYLIN-3204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Li resolved KYLIN-3204.
Resolution: Fixed
Fix Version/s: v2.3.0
Thanks Kaige! Patch merged with a minor refine commit:
[https://github.com/apache/kylin/commit/fee5730258a5082999bcf8d3141aa4cb179b693c]
> Potentially unclosed resources in JdbcExplorer#evalQueryMetadata
>
>
> Key: KYLIN-3204
> URL: https://issues.apache.org/jira/browse/KYLIN-3204
> Project: Kylin
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Kaige Liu
>Priority: Major
> Fix For: v2.3.0
>
> Attachments: KYLIN-3204-fix-potentially-unclosed-resources.patch
>
>
> {code}
> Connection con = SqlUtil.getConnection(dbconf);
> DatabaseMetaData dbmd = con.getMetaData();
> ResultSet rs = dbmd.getColumns(null, tmpDatabase, tmpView, null);
> {code}
> con and rs should be closed upon return even if there is exception.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (KYLIN-2909) Refine Email Template for notification by freemarker
[ https://issues.apache.org/jira/browse/KYLIN-2909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Li resolved KYLIN-2909. Resolution: Fixed Thanks Yanghong! PR has been merged to master branch, with an minor refactor commit appended: [https://github.com/apache/kylin/commit/74e3f614d6a9cc2807865b8fb210836880b8da85] > Refine Email Template for notification by freemarker > > > Key: KYLIN-2909 > URL: https://issues.apache.org/jira/browse/KYLIN-2909 > Project: Kylin > Issue Type: Improvement >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Major > Fix For: v2.3.0 > > Attachments: JOB-DISCARDED.png, JOB-SUCCEED.png, JOB_ERROR.png > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-2893) Missing zero check for totalHitFrequency in CuboidStats ctor
[
https://issues.apache.org/jira/browse/KYLIN-2893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16340739#comment-16340739
]
Dong Li commented on KYLIN-2893:
Hi [~yaho], could you help review this patch? thanks!
> Missing zero check for totalHitFrequency in CuboidStats ctor
>
>
> Key: KYLIN-2893
> URL: https://issues.apache.org/jira/browse/KYLIN-2893
> Project: Kylin
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Kaige Liu
>Priority: Minor
> Attachments: KYLIN-2893.master.001.patch
>
>
> {code}
> if (hitFrequencyMap.get(cuboid) != null) {
> tmpCuboidHitProbabilityMap.put(cuboid, unitUncertainProb
> + (1 - WEIGHT_FOR_UN_QUERY) *
> hitFrequencyMap.get(cuboid) / totalHitFrequency);
> {code}
> We should check that totalHitFrequency is not zero before performing division.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (KYLIN-3196) Replace StringUtils.containsOnly with Regex
[
https://issues.apache.org/jira/browse/KYLIN-3196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Li resolved KYLIN-3196.
Resolution: Fixed
thanks kaige! patch merged to master branch.
> Replace StringUtils.containsOnly with Regex
> ---
>
> Key: KYLIN-3196
> URL: https://issues.apache.org/jira/browse/KYLIN-3196
> Project: Kylin
> Issue Type: Improvement
> Components: REST Service
>Reporter: Kaige Liu
>Assignee: Kaige Liu
>Priority: Minor
> Fix For: v2.3.0
>
> Attachments: KYLIN-3196-replace-containsOnly-1.patch
>
>
> Notice that we use StringUtils.contains to validate project/cube/model names.
> It's not high efficiency and elegant.
>
> I did a small test:
> {code:java}
> public class TempTest {
> Pattern r = Pattern.compile("^[a-zA-Z0-9_]*$");
> @Test
> public void test() {
> final char[] VALID_MODELNAME =
> "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890_".toCharArray();
> String s1 = "abc@";
> System.out.println("Call StringUtils.containsOnly 100 times");
> long start = System.nanoTime();
> for(int i =0; i<100; ++i) {
> StringUtils.containsOnly(s1, VALID_MODELNAME);
> }
> long end = System.nanoTime();
> System.out.println(end - start);
> System.out.println("Call Regex match 100 times");
> start = System.nanoTime();
> for(int i =0; i<100; ++i) {
> containsByRegex(s1);
> }
> end = System.nanoTime();
> System.out.println(end - start);
> }
> private boolean containsByRegex(final String s) {
> Matcher matcher = r.matcher(s);
> return matcher.find();
> }
> }{code}
> The result shows:
> {code:java}
> Call StringUtils.containsOnly 100 times
> 4740997
> Call Regex match 100 times
> 753182
> {code}
>
> Conclusion:
> Regex is better than StringUtils.containsOnly
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (KYLIN-3193) Prevent users cloning models across projects
[ https://issues.apache.org/jira/browse/KYLIN-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Li resolved KYLIN-3193. Resolution: Fixed > Prevent users cloning models across projects > > > Key: KYLIN-3193 > URL: https://issues.apache.org/jira/browse/KYLIN-3193 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.2.0 >Reporter: Kaige Liu >Assignee: Kaige Liu >Priority: Minor > Fix For: v2.3.0 > > Attachments: KYLIN-3193-forbit-across-prj-2.patch > > > Nowadays, data sources and tables are separated by projects in Kylin. So > cloning models across projects will leads to tables not found. Should prevent > users performing the action. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3193) Prevent users cloning models across projects
[ https://issues.apache.org/jira/browse/KYLIN-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16338929#comment-16338929 ] Dong Li commented on KYLIN-3193: thanks kaige! patch merged to master branch. > Prevent users cloning models across projects > > > Key: KYLIN-3193 > URL: https://issues.apache.org/jira/browse/KYLIN-3193 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.2.0 >Reporter: Kaige Liu >Assignee: Kaige Liu >Priority: Minor > Fix For: v2.3.0 > > Attachments: KYLIN-3193-forbit-across-prj-2.patch > > > Nowadays, data sources and tables are separated by projects in Kylin. So > cloning models across projects will leads to tables not found. Should prevent > users performing the action. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-2849) duplicate segment，cannot be deleted and data cannot be refreshed and merged
[
https://issues.apache.org/jira/browse/KYLIN-2849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Li resolved KYLIN-2849.
Resolution: Duplicate
Fix Version/s: v2.3.0
Yes. this issue has been resolved in KYLIN-3004
> duplicate segment，cannot be deleted and data cannot be refreshed and merged
> ---
>
> Key: KYLIN-2849
> URL: https://issues.apache.org/jira/browse/KYLIN-2849
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine, Metadata, REST Service
>Affects Versions: v2.0.0
> Environment: hadoop：hadoop-2.6.0-cdh5.8.2
> hive ：2.1.0
> hbase：0.98
>Reporter: 翟玉勇
>Assignee: Dong Li
>Priority: Major
> Labels: scope
> Fix For: v2.3.0
>
> Attachments: kylin-1.png, kylin-2.png
>
>
> cube duplicate segments。
> cannot be deleted and data cannot be refreshed and merged
> {code}
> try
> curl -X DELETE
> "http://127.0.0.1:7070/kylin/api/cubes/Remain_Cube_2/segs/2017082200_2017082300;
> -H "Authorization: Basic QURNSU46S1lMSU4=" -H "Content-Type:
> application/json;charset=UTF-8"
> Cannot delete segment '2017082200_2017082300' as it is neither the
> first nor the last
> segment.","stacktrace":"org.apache.kylin.rest.exception.InternalErrorException:
> Cannot delete segment '2017082200_2017082300' as it is neither the
> first nor the last segment
> {code}
> 暂时解决办法：
> {code}
> public CubeInstance deleteSegment(CubeInstance cube, String segmentName)
> throws IOException {
> if (!segmentName.equals(cube.getSegments().get(0).getName()) &&
> !segmentName.equals(cube.getSegments().get(cube.getSegments().size() -
> 1).getName())) {
> //throw new IllegalArgumentException("Cannot delete segment '" +
> segmentName + "' as it is neither the first nor the last segment.");
> }
> CubeSegment toDelete = null;
> for (CubeSegment seg : cube.getSegments()) {
> if (seg.getName().equals(segmentName)) {
> toDelete = seg;
> }
> }
> if (toDelete == null) {
> throw new IllegalArgumentException("Cannot find segment '" +
> segmentName + "'");
> }
> if (toDelete.getStatus() != SegmentStatusEnum.READY) {
> //throw new IllegalArgumentException("Cannot delete segment '" +
> segmentName + "' as its status is not READY. Discard the on-going job for
> it.");
> }
> CubeUpdate update = new CubeUpdate(cube);
> update.setToRemoveSegs(new CubeSegment[] { toDelete });
> return CubeManager.getInstance(getConfig()).updateCube(update);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (KYLIN-2919) Release Kylin 2.2
[ https://issues.apache.org/jira/browse/KYLIN-2919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Li resolved KYLIN-2919. Resolution: Fixed Fix Version/s: v2.2.0 > Release Kylin 2.2 > - > > Key: KYLIN-2919 > URL: https://issues.apache.org/jira/browse/KYLIN-2919 > Project: Kylin > Issue Type: Task >Reporter: Dong Li >Assignee: Dong Li >Priority: Major > Fix For: v2.2.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-2919) Release Kylin 2.2
[ https://issues.apache.org/jira/browse/KYLIN-2919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Li closed KYLIN-2919. -- > Release Kylin 2.2 > - > > Key: KYLIN-2919 > URL: https://issues.apache.org/jira/browse/KYLIN-2919 > Project: Kylin > Issue Type: Task >Reporter: Dong Li >Assignee: Dong Li >Priority: Major > Fix For: v2.2.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3098) add a new config kylin.query.max-return-rows for the maximum row count returned to user
[ https://issues.apache.org/jira/browse/KYLIN-3098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Li resolved KYLIN-3098. Resolution: Fixed Fix Version/s: v2.3.0 Merged to master branch, thanks Yanghong! > add a new config kylin.query.max-return-rows for the maximum row count > returned to user > --- > > Key: KYLIN-3098 > URL: https://issues.apache.org/jira/browse/KYLIN-3098 > Project: Kylin > Issue Type: Improvement >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Major > Fix For: v2.3.0 > > Attachments: APACHE-KYLIN-3098.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-2894) Change the query cache expiration strategy by signature checking
[
https://issues.apache.org/jira/browse/KYLIN-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16333518#comment-16333518
]
Dong Li commented on KYLIN-2894:
Hi Yanghong, I see the PR is already submitted. Could you explain more about
the signature here? Thanks!
> Change the query cache expiration strategy by signature checking
>
>
> Key: KYLIN-2894
> URL: https://issues.apache.org/jira/browse/KYLIN-2894
> Project: Kylin
> Issue Type: Sub-task
> Components: Query Engine
>Reporter: Zhong Yanghong
>Assignee: Zhong Yanghong
>Priority: Major
>
> Currently to invalid query cache, {{CacheService}} will either invoke
> {{cleanDataCache}} or {{cleanAllDataCache}}. Both methods will clear all of
> the query cache, which is very inefficient. In eBay PROD environment, there's
> around 400 cubing jobs per day, which means the query cache will be cleared
> very 4 minutes. Then we introduced a signature based cache invalidation
> strategy. The basic idea is as follows:
> * Add a signature for {{SQLResponse}}, here we choose the cube last build time
> * When fetch {{SQLResponse}} for cache, first check whether the signature is
> consistent. If not, this cached value is overdue and will be invalidate.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (KYLIN-2881) Improve hbase coprocessor exception handling at kylin server side
[
https://issues.apache.org/jira/browse/KYLIN-2881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Li resolved KYLIN-2881.
Resolution: Fixed
patch merged to master branch, thanks yanghong!
> Improve hbase coprocessor exception handling at kylin server side
> --
>
> Key: KYLIN-2881
> URL: https://issues.apache.org/jira/browse/KYLIN-2881
> Project: Kylin
> Issue Type: Improvement
> Components: Query Engine
>Reporter: Zhong Yanghong
>Assignee: Zhong Yanghong
>Priority: Major
> Fix For: v2.3.0
>
> Attachments: APACHE-KYLIN-2881.patch
>
>
> A query may need a bunch of hconnections to visit hbase. Anyone of them
> fails, the whole query should be interrupted, and all of the related
> hconnections should be released for future other queries despite that
> currently we cannot interrupt the threads at hbase coprocessor side.
> To achieve this goal, one property is added to QueryContext to keep
> {{Throwable}}. If any sub-step during the query detects this {{Throwable}},
> it should stop. In {{ExpectedSizeIterator}}, there are two parts where this
> detection is needed.
> * before putting data to the queue
> * during iteration
> If exception happens during a query, it should stop its related threads.
> Thus, {{QueryStopListener}} is added for this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (KYLIN-1872) Make query visible and interruptible, improve server's stablility
[ https://issues.apache.org/jira/browse/KYLIN-1872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Li resolved KYLIN-1872. Resolution: Fixed Fix Version/s: v2.3.0 patch merged to master branch, thanks Gang! > Make query visible and interruptible, improve server's stablility > - > > Key: KYLIN-1872 > URL: https://issues.apache.org/jira/browse/KYLIN-1872 > Project: Kylin > Issue Type: Improvement > Components: Query Engine >Reporter: Ma Gang >Assignee: Ma Gang >Priority: Major > Fix For: v2.3.0 > > Attachments: APACHE-KYLIN-1872.patch, > query_visible_interruptable-1.4rc.patch, > query_visible_interruptable-master.patch > > > Problem: > 1. Large query result will break kylin server, for example: select * from > fact_table. Even when properties "kylin.query.scan.threshold" and > "kylin.query.mem.budget" are set properly, OOM still happens, because the > hbase rpc thread is not interrupted, the result will continually go to kylin > server. And server will run OOM quickly when there are multiple such queries. > 2. Tow many slow queries will occupy all tomcat threads, and make server > unresponsed. > 3. There's no corelation id for a specified query, so it is hard to find the > rpc log for a specified query, if there are too many queries running > concurrently. > Solution: > 1. Interrupt the rpc thread and main query thread when return result size > larger than the config limit size. > 2. Make query visible. Admin can view all running queries, and detail of each > query. >Split the query into following steps: >1) sql parse >2) cube plan >3) query cache >4) multiple cube segment query > a. for each segment request, have muliple endpoint range request. > b. for each endpoint range request, have multiple coprocessor request. > c. for each coprocessor request, have multiple region server rpc. >Admin can view the startTime/endTime of each step, and the thread stack > trace if the step is running. > 3. Add query id as corelation id in the rpc log. > 4. Admin can interrupt a running query, to release the thread, memory, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3153) Create a document for system cube creation
[ https://issues.apache.org/jira/browse/KYLIN-3153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Li resolved KYLIN-3153. Resolution: Fixed Thanks Yanghong. This patch has been merged. > Create a document for system cube creation > -- > > Key: KYLIN-3153 > URL: https://issues.apache.org/jira/browse/KYLIN-3153 > Project: Kylin > Issue Type: Sub-task >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong > Fix For: v2.3.0 > > Attachments: APACHE-KYLIN-3153.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (KYLIN-3155) Create a document for how to use dashboard
[ https://issues.apache.org/jira/browse/KYLIN-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Li resolved KYLIN-3155. Resolution: Fixed Thanks Yanghong. This patch has been merged. > Create a document for how to use dashboard > -- > > Key: KYLIN-3155 > URL: https://issues.apache.org/jira/browse/KYLIN-3155 > Project: Kylin > Issue Type: Sub-task >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong > Fix For: v2.3.0 > > Attachments: APACHE-KYLIN-3155.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (KYLIN-3154) Create a document for cube planner
[ https://issues.apache.org/jira/browse/KYLIN-3154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Li resolved KYLIN-3154. Resolution: Fixed Thanks Yanghong. This patch has been merged. > Create a document for cube planner > -- > > Key: KYLIN-3154 > URL: https://issues.apache.org/jira/browse/KYLIN-3154 > Project: Kylin > Issue Type: Sub-task >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong > Fix For: v2.3.0 > > Attachments: APACHE-KYLIN-3154.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2909) Refine Email Template for notification by freemarker
[ https://issues.apache.org/jira/browse/KYLIN-2909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16314483#comment-16314483 ] Dong Li commented on KYLIN-2909: Hi [~yaho], got some comments in this PR, please have a check. Thanks! > Refine Email Template for notification by freemarker > > > Key: KYLIN-2909 > URL: https://issues.apache.org/jira/browse/KYLIN-2909 > Project: Kylin > Issue Type: Improvement >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong > Attachments: JOB-DISCARDED.png, JOB-SUCCEED.png, JOB_ERROR.png > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (KYLIN-2909) Refine Email Template for notification by freemarker
[ https://issues.apache.org/jira/browse/KYLIN-2909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16309178#comment-16309178 ] Dong Li edited comment on KYLIN-2909 at 1/3/18 5:36 AM: Hi [~yaho], I saw all these templates are related to jobs. How about put these templates and freemarker dependency to core-job module rather than core-common? was (Author: lidong_sjtu): Hi [~yaho], I saw all these templates are related to jobs. How about put these templates and freemarker to core-job module rather than core-common? > Refine Email Template for notification by freemarker > > > Key: KYLIN-2909 > URL: https://issues.apache.org/jira/browse/KYLIN-2909 > Project: Kylin > Issue Type: Improvement >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong > Attachments: JOB-DISCARDED.png, JOB-SUCCEED.png, JOB_ERROR.png > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2909) Refine Email Template for notification by freemarker
[ https://issues.apache.org/jira/browse/KYLIN-2909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16309178#comment-16309178 ] Dong Li commented on KYLIN-2909: Hi [~yaho], I saw all these templates are related to jobs. How about put these templates and freemarker to core-job module rather than core-common? > Refine Email Template for notification by freemarker > > > Key: KYLIN-2909 > URL: https://issues.apache.org/jira/browse/KYLIN-2909 > Project: Kylin > Issue Type: Improvement >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong > Attachments: JOB-DISCARDED.png, JOB-SUCCEED.png, JOB_ERROR.png > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (KYLIN-3143) Wrong use of Preconditions.checkNotNull() in ManagedUser#removeAuthoritie
[
https://issues.apache.org/jira/browse/KYLIN-3143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Li resolved KYLIN-3143.
Resolution: Fixed
Fix Version/s: v2.3.0
merged to master branch.
> Wrong use of Preconditions.checkNotNull() in ManagedUser#removeAuthoritie
> -
>
> Key: KYLIN-3143
> URL: https://issues.apache.org/jira/browse/KYLIN-3143
> Project: Kylin
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Chao Long
>Priority: Minor
> Fix For: v2.3.0
>
> Attachments:
> KYLIN-3143-THE-RIGHT-WAY-TO-USE-PRECONDITIONS-CHECKOUTNOTNULL.patch
>
>
> {code}
> public void removeAuthoritie(String auth) {
> Preconditions.checkNotNull(this.authorities == null);
> {code}
> I think the intention was to check this.authorities, not {{this.authorities
> == null}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (KYLIN-3143) Wrong use of Preconditions.checkNotNull() in ManagedUser#removeAuthoritie
[
https://issues.apache.org/jira/browse/KYLIN-3143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Li reassigned KYLIN-3143:
--
Assignee: Chao Long
> Wrong use of Preconditions.checkNotNull() in ManagedUser#removeAuthoritie
> -
>
> Key: KYLIN-3143
> URL: https://issues.apache.org/jira/browse/KYLIN-3143
> Project: Kylin
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Chao Long
>Priority: Minor
>
> {code}
> public void removeAuthoritie(String auth) {
> Preconditions.checkNotNull(this.authorities == null);
> {code}
> I think the intention was to check this.authorities, not {{this.authorities
> == null}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Resolved] (KYLIN-2913) Enable job retry for configurable exceptions
[ https://issues.apache.org/jira/browse/KYLIN-2913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Li resolved KYLIN-2913. Resolution: Fixed Thanks Gang, merged to master branch. > Enable job retry for configurable exceptions > > > Key: KYLIN-2913 > URL: https://issues.apache.org/jira/browse/KYLIN-2913 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Affects Versions: v2.1.0 >Reporter: Wang, Gang >Assignee: Wang, Gang >Priority: Minor > Fix For: v2.3.0 > > Attachments: > 0001-KYLIN-2913-Enable-job-retry-for-configurable-excepti.patch > > > In our production environment, we always get some certain exceptions from > Hadoop or HBase, like > "org.apache.kylin.job.exception.NoEnoughReplicationException", > "java.util.ConcurrentModificationException", which results in job failure. > While, these exceptions can be handled by retry actually. So, it will be much > more convenient if we are able to make job retry on some configurable > exceptions. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (KYLIN-2956) building trie dictionary blocked on value of length over 4095
[
https://issues.apache.org/jira/browse/KYLIN-2956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Li resolved KYLIN-2956.
Resolution: Fixed
Fix Version/s: v2.3.0
Thanks Gang, merged to master branch.
> building trie dictionary blocked on value of length over 4095
> --
>
> Key: KYLIN-2956
> URL: https://issues.apache.org/jira/browse/KYLIN-2956
> Project: Kylin
> Issue Type: Bug
> Components: General
>Reporter: Wang, Gang
>Assignee: Wang, Gang
> Fix For: v2.3.0
>
> Attachments:
> 0001-KYLIN-2956-building-trie-dictionary-blocked-on-value.patch
>
>
> In the new release, Kylin will check the value length when building trie
> dictionary, in class TrieDictionaryBuilder method buildTrieBytes, through
> method:
> private void positiveShortPreCheck(int i, String fieldName) {
> if (!BytesUtil.isPositiveShort(i)) {
> throw new IllegalStateException(fieldName + " is not positive short,
> usually caused by too long dict value.");
> }
> }
> public static boolean isPositiveShort(int i) {
> return (i & 0x7000) == 0;
> }
> And 0x7000 in binary: 0111 , so the
> value length should be less than 0001 0001 ,
> values 4095 in decimalism.
> I wonder why is 0x7000, should 0x8000 ( 1000
> ), support max length: 0111
> (32767)
> be what you want?
> Or 32767 may be too large, I prefer use 0xE000, 0xE000 (
> 1110 ), support max length: 0001
> (8191)
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Resolved] (KYLIN-2866) Enlarge the reducer number for hyperloglog statistics calculation at step FactDistinctColumnsJob
[ https://issues.apache.org/jira/browse/KYLIN-2866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Li resolved KYLIN-2866. Resolution: Fixed Thanks Yanghong, patch has been merged to master branch. > Enlarge the reducer number for hyperloglog statistics calculation at step > FactDistinctColumnsJob > > > Key: KYLIN-2866 > URL: https://issues.apache.org/jira/browse/KYLIN-2866 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong > Fix For: v2.3.0 > > Attachments: APACHE-KYLIN-2866-refined.patch, APACHE-KYLIN-2866.patch > > > Currently only one reducer is assigned for hll stats calculation, which may > become the bottleneck for slow down this step. Since the stats for different > cuboids will not influence each other, it's better to divide the cuboid set > into several and assign a reduce for each subset. > The strategy of this patch is to assign 100 cuboids into a subset. And > there's a upper limit of reducers for hll stats calculation. Currently it's > 50. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (KYLIN-2902) Introduce project-level query number control
[ https://issues.apache.org/jira/browse/KYLIN-2902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Li resolved KYLIN-2902. Resolution: Fixed merged to master branch, with a minor refine commit at https://github.com/apache/kylin/commit/8690fd2d671ca7fa49576cc84eda41c3f24e539f, please have a check. thanks! > Introduce project-level query number control > > > Key: KYLIN-2902 > URL: https://issues.apache.org/jira/browse/KYLIN-2902 > Project: Kylin > Issue Type: New Feature >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong > Fix For: v2.3.0 > > Attachments: APACHE-KYLIN-2902-refined.patch, APACHE-KYLIN-2902.patch > > > One kylin server may contain many projects. To avoid some projects occupying > too much resources, like http connection, project-level query number limit is > useful. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (KYLIN-2736) Use multiple threads to calculate HyperLogLogPlusCounter in FactDistinctColumnsMapper
[ https://issues.apache.org/jira/browse/KYLIN-2736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Li resolved KYLIN-2736. Resolution: Fixed merged to master branch, with a minor refine commit at https://github.com/apache/kylin/commit/79374047d3bcd8468bee9ac56f68f4f191edad7b > Use multiple threads to calculate HyperLogLogPlusCounter in > FactDistinctColumnsMapper > - > > Key: KYLIN-2736 > URL: https://issues.apache.org/jira/browse/KYLIN-2736 > Project: Kylin > Issue Type: Improvement >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong > Fix For: v2.3.0 > > Attachments: APACHE-KYLIN-2736.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (KYLIN-3031) KeywordDefaultDirtyHack should ignore case of default like other database does
[ https://issues.apache.org/jira/browse/KYLIN-3031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Li reassigned KYLIN-3031: -- Assignee: Zhong Yanghong (was: liyang) > KeywordDefaultDirtyHack should ignore case of default like other database does > -- > > Key: KYLIN-3031 > URL: https://issues.apache.org/jira/browse/KYLIN-3031 > Project: Kylin > Issue Type: Bug >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong > Fix For: v2.3.0 > > Attachments: APACHE-KYLIN-3031.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (KYLIN-3004) Delete cube segment validation
[
https://issues.apache.org/jira/browse/KYLIN-3004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Li resolved KYLIN-3004.
Resolution: Fixed
Fix Version/s: v2.3.0
Thanks Julian. Patch merged to master branch, see commit
https://github.com/apache/kylin/commit/07142889d4423b3211be625eceb8fd093fd34a22
I also append a code review commit, please have a look:
https://github.com/apache/kylin/commit/b55f4967067f0ecc056d58c9ba081e496f0d2b9e
> Delete cube segment validation
> --
>
> Key: KYLIN-3004
> URL: https://issues.apache.org/jira/browse/KYLIN-3004
> Project: Kylin
> Issue Type: Bug
> Components: REST Service
>Affects Versions: v2.1.0
>Reporter: Pan, Julian
>Assignee: Pan, Julian
> Fix For: v2.3.0
>
> Attachments: KYLIN_3004.patch, KYLIN_3004.patch
>
>
> In version 2.1, kylin allow hole between segment.
> But currently, there is validation for delete segment.
> Here is the code in CubeService:
> if (!segmentName.equals(cube.getSegments().get(0).getName())
> &&
> !segmentName.equals(cube.getSegments().get(cube.getSegments().size() -
> 1).getName())) {
> throw new
> BadRequestException(String.format(msg.getDELETE_NOT_FIRST_LAST_SEG(),
> segmentName));
> }
> Does it make sense? Should we remove the limit?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Resolved] (KYLIN-3001) Cache key issue
[
https://issues.apache.org/jira/browse/KYLIN-3001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Li resolved KYLIN-3001.
Resolution: Fixed
Fix Version/s: v2.3.0
merged to master branch
https://github.com/apache/kylin/commit/c9bcc5b88475adef6cbd74f149128702997c776b
> Cache key issue
>
>
> Key: KYLIN-3001
> URL: https://issues.apache.org/jira/browse/KYLIN-3001
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Affects Versions: v2.1.0
>Reporter: Pan, Julian
>Assignee: liyang
> Fix For: v2.3.0
>
> Attachments: KYLIN_3001_Cache_key_issue.patch
>
>
> Currently the cacheKey generate by SQLRequest getCacheKey:
> cacheKey = Lists.newArrayList(sql.replaceAll("\\s+", ""), project, offset,
> limit, acceptPartial,
> backdoorToggles);
> Which will remove all the space and break line.
> If I have a sql:
> select a, sum(c) from table
> --where a > 0 group by a;
> and another sql:
> select a, sum(c) from table
> --where a > 0
> group by a;
> The two sql will share same cache key.
> But the first sql should be break. If you query second sql, and then query
> the first again. It will return result.
> And another issue is if remove all space, which will mix a as b and aasb.
> I think the better way is just replace multiple space to one space
> Should I create a patch? Or just provide the fix:
> cacheKey = Lists.newArrayList(sql.replaceAll("[ ]", " "), project, offset,
> limit, acceptPartial,
> backdoorToggles);
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Resolved] (KYLIN-2905) Refine the process of submitting a job
[ https://issues.apache.org/jira/browse/KYLIN-2905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Li resolved KYLIN-2905. Resolution: Fixed Fix Version/s: v2.3.0 merged to master branch https://github.com/apache/kylin/commit/58005727ea7f0f9a35a83c737b1a6b9c6b342f6f > Refine the process of submitting a job > -- > > Key: KYLIN-2905 > URL: https://issues.apache.org/jira/browse/KYLIN-2905 > Project: Kylin > Issue Type: Improvement > Components: Metadata >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong > Fix For: v2.3.0 > > Attachments: APACHE-KYLIN-2905.patch > > > In cases hbase is very busy, it cost some time to put all of the job related > metadata into hbase. The job related metadata includes *job info* and *job > output info*. Currently, kylin put *job info* into hbase first, then *job > output info*. If server is down and only part of *job output info* has been > put into hbase during this period, kylin will fail to load job when restarts. > To solve this issue, it's better to adjust the order by adding *job output > info* first, then *job info*. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (KYLIN-3082) Close of GTBuilder should be placed in finally block in InMemCubeBuilder
[
https://issues.apache.org/jira/browse/KYLIN-3082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Li resolved KYLIN-3082.
Resolution: Fixed
Fix Version/s: v2.3.0
> Close of GTBuilder should be placed in finally block in InMemCubeBuilder
>
>
> Key: KYLIN-3082
> URL: https://issues.apache.org/jira/browse/KYLIN-3082
> Project: Kylin
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Dong Li
>Priority: Minor
> Fix For: v2.3.0
>
>
> {code}
> baseBuilder.write(r);
> count++;
> }
> aggregationScanner.close();
> baseBuilder.close();
> {code}
> Since write() call may produce IOE, the close() call should be placed in
> finally block.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Resolved] (KYLIN-2935) Improve the way to deploy coprocessor
[ https://issues.apache.org/jira/browse/KYLIN-2935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Li resolved KYLIN-2935. Resolution: Fixed Fix Version/s: v2.3.0 merged to master branch: https://github.com/apache/kylin/commit/7ca66d436ff5ec97dd80741dbaa9d73812743648 > Improve the way to deploy coprocessor > - > > Key: KYLIN-2935 > URL: https://issues.apache.org/jira/browse/KYLIN-2935 > Project: Kylin > Issue Type: Improvement >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong > Fix For: v2.3.0 > > Attachments: APACHE-KYLIN-2935.patch > > > There are two improvements for deploying coprocessor: > # Make thread number configurable > # Filter tables by git commit -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (KYLIN-1778) Timeout handling for Query on WebUI
[ https://issues.apache.org/jira/browse/KYLIN-1778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Li closed KYLIN-1778. -- Resolution: Duplicate Fix Version/s: v2.3.0 > Timeout handling for Query on WebUI > --- > > Key: KYLIN-1778 > URL: https://issues.apache.org/jira/browse/KYLIN-1778 > Project: Kylin > Issue Type: Bug > Components: Web >Affects Versions: v1.5.2 >Reporter: Dong Li >Assignee: Zhong,Jason >Priority: Minor > Fix For: v2.3.0 > > > In this js file: webapp/app/js/services/query.js, we find the timeout is hard > coded as 300s for query. > When user's query exceed this limitation, the query directly failed with no > messages, but may found it succeeded in logs several minutes later. > Solution options: > 1. Make the timeout limitation configurable > 2. Prompt timeout message to user -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (KYLIN-3082) Close of GTBuilder should be placed in finally block in InMemCubeBuilder
[
https://issues.apache.org/jira/browse/KYLIN-3082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Li reassigned KYLIN-3082:
--
Assignee: Dong Li
> Close of GTBuilder should be placed in finally block in InMemCubeBuilder
>
>
> Key: KYLIN-3082
> URL: https://issues.apache.org/jira/browse/KYLIN-3082
> Project: Kylin
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Dong Li
>Priority: Minor
>
> {code}
> baseBuilder.write(r);
> count++;
> }
> aggregationScanner.close();
> baseBuilder.close();
> {code}
> Since write() call may produce IOE, the close() call should be placed in
> finally block.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KYLIN-2956) building trie dictionary blocked on value of length over 4095
[
https://issues.apache.org/jira/browse/KYLIN-2956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16298651#comment-16298651
]
Dong Li commented on KYLIN-2956:
Thanks Gang!
thought that the mask should be 0x8000, right?
and it will be nice to have some UTs to cover this method.
> building trie dictionary blocked on value of length over 4095
> --
>
> Key: KYLIN-2956
> URL: https://issues.apache.org/jira/browse/KYLIN-2956
> Project: Kylin
> Issue Type: Bug
> Components: General
>Reporter: Wang, Gang
>Assignee: Wang, Gang
> Attachments:
> 0001-KYLIN-2956-building-trie-dictionary-blocked-on-value.patch
>
>
> In the new release, Kylin will check the value length when building trie
> dictionary, in class TrieDictionaryBuilder method buildTrieBytes, through
> method:
> private void positiveShortPreCheck(int i, String fieldName) {
> if (!BytesUtil.isPositiveShort(i)) {
> throw new IllegalStateException(fieldName + " is not positive short,
> usually caused by too long dict value.");
> }
> }
> public static boolean isPositiveShort(int i) {
> return (i & 0x7000) == 0;
> }
> And 0x7000 in binary: 0111 , so the
> value length should be less than 0001 0001 ,
> values 4095 in decimalism.
> I wonder why is 0x7000, should 0x8000 ( 1000
> ), support max length: 0111
> (32767)
> be what you want?
> Or 32767 may be too large, I prefer use 0xE000, 0xE000 (
> 1110 ), support max length: 0001
> (8191)
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KYLIN-2913) Enable job retry for configurable exceptions
[ https://issues.apache.org/jira/browse/KYLIN-2913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16298648#comment-16298648 ] Dong Li commented on KYLIN-2913: if the value of "kylin.job.retry-exception-classes" became empty, please keep current behavior as default. > Enable job retry for configurable exceptions > > > Key: KYLIN-2913 > URL: https://issues.apache.org/jira/browse/KYLIN-2913 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Affects Versions: v2.1.0 >Reporter: Wang, Gang >Assignee: Wang, Gang >Priority: Minor > Fix For: v2.3.0 > > Attachments: > 0001-KYLIN-2913-Enable-job-retry-for-configurable-excepti.patch > > > In our production environment, we always get some certain exceptions from > Hadoop or HBase, like > "org.apache.kylin.job.exception.NoEnoughReplicationException", > "java.util.ConcurrentModificationException", which results in job failure. > While, these exceptions can be handled by retry actually. So, it will be much > more convenient if we are able to make job retry on some configurable > exceptions. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (KYLIN-3111) Close of Admin instance should be placed in finally block
[
https://issues.apache.org/jira/browse/KYLIN-3111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Li resolved KYLIN-3111.
Resolution: Fixed
Fix Version/s: v2.3.0
Merged to master branch, thanks Chao!
> Close of Admin instance should be placed in finally block
> -
>
> Key: KYLIN-3111
> URL: https://issues.apache.org/jira/browse/KYLIN-3111
> Project: Kylin
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Chao Long
> Fix For: v2.3.0
>
> Attachments:
> KYLIN-3111-Place-close-of-admin-instance-into-finally-block.patch
>
>
> Looking at the code in DeployCoprocessorCLI.java and
> HtableAlterMetadataCLI.java , I see that close of Admin instance is without
> finally block:
> {code}
> hbaseAdmin.disableTable(table.getTableName());
> table.setValue(metadataKey, metadataValue);
> hbaseAdmin.modifyTable(table.getTableName(), table);
> hbaseAdmin.enableTable(table.getTableName());
> hbaseAdmin.close();
> {code}
> If any exception is thrown in the operations prior to the close(), the
> close() would be skipped, leading to resource leak.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KYLIN-3111) Close of Admin instance should be placed in finally block
[
https://issues.apache.org/jira/browse/KYLIN-3111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16294038#comment-16294038
]
Dong Li commented on KYLIN-3111:
Thanks [~Wayne0101] for your contribution!
Two suggestions for this patch:
1. Commit message should follow the same patterns, for example: "KYLIN-3111
X". Please refer to other commits and update your message.
2. Your author information(email, name) is not included in this patch. Please
update your git config so that your contribution can be recognized directly.
> Close of Admin instance should be placed in finally block
> -
>
> Key: KYLIN-3111
> URL: https://issues.apache.org/jira/browse/KYLIN-3111
> Project: Kylin
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Chao Long
> Attachments: KYLIN-3111.patch
>
>
> Looking at the code in DeployCoprocessorCLI.java and
> HtableAlterMetadataCLI.java , I see that close of Admin instance is without
> finally block:
> {code}
> hbaseAdmin.disableTable(table.getTableName());
> table.setValue(metadataKey, metadataValue);
> hbaseAdmin.modifyTable(table.getTableName(), table);
> hbaseAdmin.enableTable(table.getTableName());
> hbaseAdmin.close();
> {code}
> If any exception is thrown in the operations prior to the close(), the
> close() would be skipped, leading to resource leak.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (KYLIN-3111) Close of Admin instance should be placed in finally block
[
https://issues.apache.org/jira/browse/KYLIN-3111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Li reassigned KYLIN-3111:
--
Assignee: Dong Li
> Close of Admin instance should be placed in finally block
> -
>
> Key: KYLIN-3111
> URL: https://issues.apache.org/jira/browse/KYLIN-3111
> Project: Kylin
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Dong Li
> Attachments: KYLIN-3111.patch
>
>
> Looking at the code in DeployCoprocessorCLI.java and
> HtableAlterMetadataCLI.java , I see that close of Admin instance is without
> finally block:
> {code}
> hbaseAdmin.disableTable(table.getTableName());
> table.setValue(metadataKey, metadataValue);
> hbaseAdmin.modifyTable(table.getTableName(), table);
> hbaseAdmin.enableTable(table.getTableName());
> hbaseAdmin.close();
> {code}
> If any exception is thrown in the operations prior to the close(), the
> close() would be skipped, leading to resource leak.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (KYLIN-3111) Close of Admin instance should be placed in finally block
[
https://issues.apache.org/jira/browse/KYLIN-3111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Li reassigned KYLIN-3111:
--
Assignee: Chao Long (was: Dong Li)
> Close of Admin instance should be placed in finally block
> -
>
> Key: KYLIN-3111
> URL: https://issues.apache.org/jira/browse/KYLIN-3111
> Project: Kylin
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Chao Long
> Attachments: KYLIN-3111.patch
>
>
> Looking at the code in DeployCoprocessorCLI.java and
> HtableAlterMetadataCLI.java , I see that close of Admin instance is without
> finally block:
> {code}
> hbaseAdmin.disableTable(table.getTableName());
> table.setValue(metadataKey, metadataValue);
> hbaseAdmin.modifyTable(table.getTableName(), table);
> hbaseAdmin.enableTable(table.getTableName());
> hbaseAdmin.close();
> {code}
> If any exception is thrown in the operations prior to the close(), the
> close() would be skipped, leading to resource leak.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (KYLIN-2866) Enlarge the reducer number for hyperloglog statistics calculation at step FactDistinctColumnsJob
[ https://issues.apache.org/jira/browse/KYLIN-2866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16289234#comment-16289234 ] Dong Li edited comment on KYLIN-2866 at 12/13/17 1:11 PM: -- Hi [~yaho], in this patch, I found several duplicated code between org.apache.kylin.engine.mr.steps.SaveStatisticsStep#doWork and org.apache.kylin.engine.mr.common.CubeStatsReader, which will make the code more complex. Could you make some refine? Besides, in org.apache.kylin.engine.mr.common.MapReduceUtil#getHLLShardBase, which seems to calculate reducer number. What does the "HLLShardBase" mean here? was (Author: lidong_sjtu): Hi [~yaho], in this patch, I found several duplicated code between org.apache.kylin.engine.mr.steps.SaveStatisticsStep#doWork and org.apache.kylin.engine.mr.common.CubeStatsReader, which make code complex. Could you make some refine? Besides, in org.apache.kylin.engine.mr.common.MapReduceUtil#getHLLShardBase, which seems to calculate reducer number. What does the "HLLShardBase" mean here? > Enlarge the reducer number for hyperloglog statistics calculation at step > FactDistinctColumnsJob > > > Key: KYLIN-2866 > URL: https://issues.apache.org/jira/browse/KYLIN-2866 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong > Fix For: v2.3.0 > > Attachments: APACHE-KYLIN-2866.patch > > > Currently only one reducer is assigned for hll stats calculation, which may > become the bottleneck for slow down this step. Since the stats for different > cuboids will not influence each other, it's better to divide the cuboid set > into several and assign a reduce for each subset. > The strategy of this patch is to assign 100 cuboids into a subset. And > there's a upper limit of reducers for hll stats calculation. Currently it's > 50. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2866) Enlarge the reducer number for hyperloglog statistics calculation at step FactDistinctColumnsJob
[ https://issues.apache.org/jira/browse/KYLIN-2866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16289234#comment-16289234 ] Dong Li commented on KYLIN-2866: Hi [~yaho], in this patch, I found several duplicated code between org.apache.kylin.engine.mr.steps.SaveStatisticsStep#doWork and org.apache.kylin.engine.mr.common.CubeStatsReader, which make code complex. Could you make some refine? Besides, in org.apache.kylin.engine.mr.common.MapReduceUtil#getHLLShardBase, which seems to calculate reducer number. What does the "HLLShardBase" mean here? > Enlarge the reducer number for hyperloglog statistics calculation at step > FactDistinctColumnsJob > > > Key: KYLIN-2866 > URL: https://issues.apache.org/jira/browse/KYLIN-2866 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong > Fix For: v2.3.0 > > Attachments: APACHE-KYLIN-2866.patch > > > Currently only one reducer is assigned for hll stats calculation, which may > become the bottleneck for slow down this step. Since the stats for different > cuboids will not influence each other, it's better to divide the cuboid set > into several and assign a reduce for each subset. > The strategy of this patch is to assign 100 cuboids into a subset. And > there's a upper limit of reducers for hll stats calculation. Currently it's > 50. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (KYLIN-2867) split large fuzzy Key set
[ https://issues.apache.org/jira/browse/KYLIN-2867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Li resolved KYLIN-2867. Resolution: Fixed Fix Version/s: v2.3.0 merged to master branch. > split large fuzzy Key set > - > > Key: KYLIN-2867 > URL: https://issues.apache.org/jira/browse/KYLIN-2867 > Project: Kylin > Issue Type: Improvement > Components: Query Engine >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong > Fix For: v2.3.0 > > Attachments: APACHE-KYLIN-2867.patch > > > Too many fuzzy keys in one scan may downgrade the performance. Currently if > the number of fuzzy keys exceeds a threshold, like 200, the scan will not use > fuzzy key. However, this may also downgrade the performance. > There are many cases in eBay that the total number of fuzzy keys is around > 1000. To deal with these cases, it's better to split the large fuzzy key set, > like 5*200. Then for each subset, assign a scan. This strategy will increase > the number of scan to reduce the number of fuzzy keys in each scan to improve > the total performance. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2881) Improve hbase coprocessor exception handling at kylin server side
[
https://issues.apache.org/jira/browse/KYLIN-2881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16288953#comment-16288953
]
Dong Li commented on KYLIN-2881:
Hi [~yaho], I'm reviewing the patch, and got a question.
In class KylinTestBase, why added a function call of initExecQueryUsingKylin()?
As this is not needed before your patch.
> Improve hbase coprocessor exception handling at kylin server side
> --
>
> Key: KYLIN-2881
> URL: https://issues.apache.org/jira/browse/KYLIN-2881
> Project: Kylin
> Issue Type: Improvement
> Components: Query Engine
>Reporter: Zhong Yanghong
>Assignee: Zhong Yanghong
> Fix For: v2.3.0
>
> Attachments: APACHE-KYLIN-2881.patch
>
>
> A query may need a bunch of hconnections to visit hbase. Anyone of them
> fails, the whole query should be interrupted, and all of the related
> hconnections should be released for future other queries despite that
> currently we cannot interrupt the threads at hbase coprocessor side.
> To achieve this goal, one property is added to QueryContext to keep
> {{Throwable}}. If any sub-step during the query detects this {{Throwable}},
> it should stop. In {{ExpectedSizeIterator}}, there are two parts where this
> detection is needed.
> * before putting data to the queue
> * during iteration
> If exception happens during a query, it should stop its related threads.
> Thus, {{QueryStopListener}} is added for this.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KYLIN-3098) add a new config kylin.query.max-return-rows for the maximum row count returned to user
[ https://issues.apache.org/jira/browse/KYLIN-3098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16288835#comment-16288835 ] Dong Li commented on KYLIN-3098: Hi [~yaho], the patch cannot be applied on latest master branch, could you rebase master and resubmit the patch? > add a new config kylin.query.max-return-rows for the maximum row count > returned to user > --- > > Key: KYLIN-3098 > URL: https://issues.apache.org/jira/browse/KYLIN-3098 > Project: Kylin > Issue Type: Improvement >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong > Attachments: APACHE-KYLIN-3098.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (KYLIN-3043) Don't need create materialized view for lookup tables without snapshot
[ https://issues.apache.org/jira/browse/KYLIN-3043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Li resolved KYLIN-3043. Resolution: Fixed merged to https://github.com/apache/kylin/commit/17838a19afb3891d6de7b5ac28dd61757a9e00f0 > Don't need create materialized view for lookup tables without snapshot > -- > > Key: KYLIN-3043 > URL: https://issues.apache.org/jira/browse/KYLIN-3043 > Project: Kylin > Issue Type: Improvement >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong > Fix For: v2.3.0 > > Attachments: APACHE-KYLIN-3043.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (KYLIN-3039) Unclosed hbaseAdmin in ITAclTableMigrationToolTest
[
https://issues.apache.org/jira/browse/KYLIN-3039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Li resolved KYLIN-3039.
Resolution: Fixed
Fix Version/s: v2.3.0
Thanks Shengping, Merged to master branch
> Unclosed hbaseAdmin in ITAclTableMigrationToolTest
> --
>
> Key: KYLIN-3039
> URL: https://issues.apache.org/jira/browse/KYLIN-3039
> Project: Kylin
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Shengping
>Priority: Minor
> Fix For: v2.3.0
>
> Attachments: KYLIN-3039.patch
>
>
> Here is related code:
> {code}
> Admin hbaseAdmin = new HBaseAdmin(conf);
> creatTable(hbaseAdmin, conf, aclTable, new String[] {
> AclConstant.ACL_INFO_FAMILY, AclConstant.ACL_ACES_FAMILY });
> creatTable(hbaseAdmin, conf, userTable, new String[] {
> AclConstant.USER_AUTHORITY_FAMILY });
> {code}
> hbaseAdmin should be closed upon return.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Resolved] (KYLIN-2975) Unclosed Statement in test
[
https://issues.apache.org/jira/browse/KYLIN-2975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Li resolved KYLIN-2975.
Resolution: Fixed
Fix Version/s: v2.3.0
Thanks Shengping. Patch merged to master branch.
> Unclosed Statement in test
> --
>
> Key: KYLIN-2975
> URL: https://issues.apache.org/jira/browse/KYLIN-2975
> Project: Kylin
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Shengping
>Priority: Minor
> Fix For: v2.3.0
>
> Attachments: KYLIN-2975.patch
>
>
> Some Statement resource is not closed in tests.
> e.g. from QuerACLTestUtil.java :
> {code}
> Statement statement = conn.createStatement();
> return statement.executeQuery(sql);
> {code}
> The statement should be closed after use.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KYLIN-2975) Unclosed Statement in test
[
https://issues.apache.org/jira/browse/KYLIN-2975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Li updated KYLIN-2975:
---
Attachment: (was: KYLIN-2975.patch)
> Unclosed Statement in test
> --
>
> Key: KYLIN-2975
> URL: https://issues.apache.org/jira/browse/KYLIN-2975
> Project: Kylin
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Shengping
>Priority: Minor
>
> Some Statement resource is not closed in tests.
> e.g. from QuerACLTestUtil.java :
> {code}
> Statement statement = conn.createStatement();
> return statement.executeQuery(sql);
> {code}
> The statement should be closed after use.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Resolved] (KYLIN-2969) java.io.IOException: java.lang.IllegalArgumentException: Column 4 value '20102' met dictionary error: Value : 20102 not exists
[
https://issues.apache.org/jira/browse/KYLIN-2969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Li resolved KYLIN-2969.
Resolution: Fixed
Fix Version/s: v2.3.0
Merged in master branch.
> java.io.IOException: java.lang.IllegalArgumentException: Column 4 value
> '20102' met dictionary error: Value : 20102 not exists
> --
>
> Key: KYLIN-2969
> URL: https://issues.apache.org/jira/browse/KYLIN-2969
> Project: Kylin
> Issue Type: Bug
>Reporter: Zhong Yanghong
>Assignee: Zhong Yanghong
>Priority: Major
> Fix For: v2.3.0
>
> Attachments: APACHE-KYLIN-2969.patch
>
>
> It's a bug related to the {{NumberDictionary}} compatibility issue. In
> {{Number2BytesConverter}}, if the method {{getCodec}} invoked two times in
> one thread, like getCodec(19), getCodec(16). For the second call, it will
> return {{NumberBytesCodec(19)}} rather than {{NumberBytesCodec(16)}}, which
> is not corrrect
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Resolved] (KYLIN-2971) the "Realization Names" print in logQuery is not correct
[
https://issues.apache.org/jira/browse/KYLIN-2971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Li resolved KYLIN-2971.
Resolution: Fixed
Fix Version/s: v2.3.0
> the "Realization Names" print in logQuery is not correct
>
>
> Key: KYLIN-2971
> URL: https://issues.apache.org/jira/browse/KYLIN-2971
> Project: Kylin
> Issue Type: Bug
>Reporter: Zhong Yanghong
>Assignee: Zhong Yanghong
>Priority: Major
> Fix For: v2.3.0
>
> Attachments: APACHE-KYLIN-2971.patch
>
>
> OLAPContext.clearThreadLocalContexts() should be called before a query
> starts. And we should consider query hit cache.
> Example,
> # query Q1 hit project P1 and cube C1;
> # then query Q2 hit project P2 and cube C2;
> # then Q1 comes again, it will show project P1 and {color:#f79232}cube
> C2{color}. However, it should be {color:#f79232}cube C1{color}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Resolved] (KYLIN-2987) Skip moving to Trash when drop an intermediate hive table or redistribute a hive table
[ https://issues.apache.org/jira/browse/KYLIN-2987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Li resolved KYLIN-2987. Resolution: Fixed Fix Version/s: v2.3.0 Merged in master branch. > Skip moving to Trash when drop an intermediate hive table or redistribute a > hive table > -- > > Key: KYLIN-2987 > URL: https://issues.apache.org/jira/browse/KYLIN-2987 > Project: Kylin > Issue Type: Improvement >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong > Fix For: v2.3.0 > > Attachments: APACHE-KYLIN-2987.patch > > > At kylin side, we can add auto.purge=true when creating intermediate table. > However, to make ‘auto.purge’ effective for “insert overwrite table”, we > still need one patch for hive. > https://issues.apache.org/jira/browse/HIVE-15880 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2971) the "Realization Names" print in logQuery is not correct
[
https://issues.apache.org/jira/browse/KYLIN-2971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16237978#comment-16237978
]
Dong Li commented on KYLIN-2971:
LGTM, merged in master branch.
> the "Realization Names" print in logQuery is not correct
>
>
> Key: KYLIN-2971
> URL: https://issues.apache.org/jira/browse/KYLIN-2971
> Project: Kylin
> Issue Type: Bug
>Reporter: Zhong Yanghong
>Assignee: Zhong Yanghong
>Priority: Major
> Attachments: APACHE-KYLIN-2971.patch
>
>
> OLAPContext.clearThreadLocalContexts() should be called before a query
> starts. And we should consider query hit cache.
> Example,
> # query Q1 hit project P1 and cube C1;
> # then query Q2 hit project P2 and cube C2;
> # then Q1 comes again, it will show project P1 and {color:#f79232}cube
> C2{color}. However, it should be {color:#f79232}cube C1{color}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KYLIN-2725) Introduce a tool for creating system cubes relating to query & job metrics
[ https://issues.apache.org/jira/browse/KYLIN-2725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16237200#comment-16237200 ] Dong Li commented on KYLIN-2725: [~yaho] Hi Yanghong, please select a 'fix version' when you resolve issues. At this moment, the next release is v2.3.0, you can select v2.3.0. > Introduce a tool for creating system cubes relating to query & job metrics > -- > > Key: KYLIN-2725 > URL: https://issues.apache.org/jira/browse/KYLIN-2725 > Project: Kylin > Issue Type: Sub-task >Affects Versions: v2.2.0 >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Major > > System cubes are the basis of dashboard & cube planner. Dashboard & cube > planner can query on system cubes to fetch required data. Currently, we > design five system cubes. The details are as follows: > * *METRICS_QUERY* > ** Dimension > USER, HOST, PROJECT, REALIZATION, REALIZATION_TYPE, QUERY_TYPE, > EXCEPTION, *TIME_DIMENSIONS* > ** Measure > COUNT, > MIN & MAX & SUM & PERCENTILE of QUERY_TIME_COST, > MAX & SUM of CALCITE_COUNT_RETURN, > MAX & SUM of STORAGE_COUNT_RETURN, > MAX & SUM of CALCITE_COUNT_AGGREGATE_FILTER, > COUNT_DISTINCT(hll) of QUERY_HASH_CODE > * *METRICS_QUERY_CUBE* > ** Dimension > CUBE_NAME, SEGMENT_NAME, CUBOID_SOURCE, CUBOID_TARGET, IF_MATCH, > IF_SUCCESS, *TIME_DIMENSIONS* > ** Measure > COUNT, > SUM of WEIGHT_PER_HIT, > MAX & SUM of STORAGE_CALL_COUNT, > MAX & SUM of STORAGE_CALL_TIME_SUM, > MAX & SUM of STORAGE_CALL_TIME_MAX, > MAX & SUM of STORAGE_COUNT_SKIP, > MAX & SUM of STORAGE_COUNT_SCAN, > MAX & SUM of STORAGE_COUNT_RETURN, > MAX & SUM of STORAGE_COUNT_AGGREGATE_FILTER > MAX & SUM of STORAGE_COUNT_AGGREGATE > * *METRICS_QUERY_RPC* > ** Dimension > HOST, PROJECT, REALIZATION, RPC_SERVER, EXCEPTION, *TIME_DIMENSIONS* > ** Measure > COUNT, > MAX & SUM & PERCENTILE of CALL_TIME, > MAX & SUM of COUNT_RETURN, > MAX & SUM of COUNT_SCAN, > MAX & SUM of COUNT_SKIP, > MAX & SUM of COUNT_AGGREGATE_FILTER, > MAX & SUM of COUNT_AGGREGATE > * *METRICS_JOB* > ** Dimension > USER, PROJECT, CUBE_NAME, JOB_TYPE, CUBING_TYPE, *TIME_DIMENSIONS* > ** Measure > COUNT, > MIN & MAX & SUM & PERCENTILE of DURATION, > MIN & MAX & SUM of TABLE_SIZE, > MIN & MAX & SUM of CUBE_SIZE, > MIN & MAX & SUM of PER_BYTES_TIME_COST, > MIN & MAX & SUM of WAIT_RESOURCE_TIME, > MAX & SUM of STEP_DURATION_DISTINCT_COLUMNS, > MAX & SUM of STEP_DURATION_DICTIONARY, > MAX & SUM of STEP_DURATION_INMEM_CUBING, > MAX & SUM of STEP_DURATION_HFILE_CONVERT > * *METRICS_JOB_EXCEPTION* > ** Dimension > USER, PROJECT, CUBE_NAME, JOB_TYPE, CUBING_TYPE, EXCEPTION, > *TIME_DIMENSIONS* > ** Measure > COUNT > * *TIME_DIMENSIONS* > KYEAR_BEGIN_DATE, KMONTH_BEGIN_DATE, KWEEK_BEGIN_DATE, KDAY_DATE, > KTIME_HOUR, KTIME_MINUTE -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2703) kylin supports managing ACL through apache ranger.
[ https://issues.apache.org/jira/browse/KYLIN-2703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Li updated KYLIN-2703: --- Summary: kylin supports managing ACL through apache ranger. (was: kylin supports managing access rights for project and cube through apache ranger.) > kylin supports managing ACL through apache ranger. > -- > > Key: KYLIN-2703 > URL: https://issues.apache.org/jira/browse/KYLIN-2703 > Project: Kylin > Issue Type: New Feature > Components: General >Reporter: peng.jianhua >Assignee: peng.jianhua > Labels: newbie, patch, scope > Fix For: v2.2.0 > > Attachments: > 0001-KYLIN-2703-kylin-supports-managing-access-rights-for.patch, > KylinAuditLog.jpg, KylinPlugins.jpg, KylinPolicies.jpg, > KylinServiceEntry.jpg, NewKylinPolicy.jpg, NewKylinService.jpg, > Ranger-PMS-hope.png > > > Ranger is a framework to enable, monitor and manage comprehensive data > security across the Hadoop platform. Apache Ranger has the following goals: > 1. Centralized security administration to manage all security related tasks > in a central UI or using REST APIs. > 2. Fine grained authorization to do a specific action and/or operation with > Hadoop component/tool and managed through a central administration tool > 3. Standardize authorization method across all Hadoop components. > 4. Enhanced support for different authorization methods - Role based access > control, attribute based access control etc. > 5. Centralize auditing of user access and administrative actions (security > related) within all the components of Hadoop. > Ranger has supported enable, monitor and manage following components: > 1. HDFS > 2. HIVE > 3. HBASE > 4. KNOX > 5. YARN > 6. STORM > 7. SOLR > 8. KAFKA > 9. ATLAS > In order to improve the flexibility of kylin privilege control and enhance > value of kylin in the Apache Hadoop ecosystem, like hdfs, yarn, hive, hbase, > Kylin should also support that using Ranger to control access rights for > project and cube. > Specific implementation plan is as following: > On the ranger website, administrators can configure policies to control user > access to projects and cube permissions. > Kylin provides an abstract class and authorization interfaces for use by the > ranger plugin. kylin instantiates ranger plugin’s implementation class when > starting(this class extends the abstract class provided by kylin). > Ranger plugin periodically polls ranger admin, updates the policy to the > local, and updates project and cube access rights based on policy information. > In the Kylin side： > 1. Kylin provides an abstract class that enables the ranger plugin's > implementation class to extend. > 2. Add configuration item. 1) ranger authorization switch, 2) ranger plugin > implementation class's name. > 3. Instantiate the ranger plugin implementation class when starting kylin. > 4. kylin provides authorization interfaces for ranger plugin calls. > 5. According to the ranger authorization configuration item, hide kylin's > authorization management page. > 6. Using ranger manager access rights of the kylin does not affect kylin's > existing permissions functions and logic. > In the Ranger side： > 1. Ranger plugin will periodically polls ranger admin, updates the policy to > the local. > 2. The ranger plugin invoking the authorization interfaces provided by kylin > to updates the project and cube access rights based on the policy information. > reference link:https://issues.apache.org/jira/browse/RANGER-1672 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
