关于hadoop3文件访问的bug
在使用kylin3访问hive3的表时,一直会报 duplicate key的错误,但我看了下,表里没有重复key,后来发现是因为hive3有版本备份的功能,每个表会存多个备份文件,kylin3无法只读取最近的一个版本,而是全都读取了,所以会一直出现重复key的问题, 请问这是bug吗,可以用什么方法规避吗?
[VOTE] Release apache-kylin-4.0.0-alpha (RC1)
Hi all, I have created a build for Apache Kylin 4.0.0-alpha, release candidate 1. Please note, this release is built on kylin-on-parquet-v2 branch. Changes highlights: [KYLIN-4213] - The new build engine with Spark-SQL [KYLIN-4450] - Add the feature that adjusting spark driver memory adaptively [KYLIN-4458] - FilePruner prune shards [KYLIN-4475] - Support intersect count for Kylin on Parquet [KYLIN-4462] - Support Count Distinct,TopN and Percentile by kylin on Parquet [KYLIN-4713] - Support use diff spark schedule pool for diff query [KYLIN-4468] - Support Percentile by kylin on Parquet [KYLIN-4662] - Migrate from third-party Spark to offical Apache Spark [KYLIN-4701] - Upgrade front-end from HBase Storage to Parquet Storage [KYLIN-4644] - New tool to clean up intermediate files for Kylin 4.0 [KYLIN-4744] - Add tracking URL for build spark job on yarn Thanks to everyone who has contributed to this release. Here are the release notes:https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316121&version=12348093 The commit to being voted upon:https://github.com/apache/kylin/commit/a285f9a5b84affc36c5466ce5a1b2fcdb4348b37 Its hash is a285f9a5b84affc36c5466ce5a1b2fcdb4348b37. The artifacts to be voted on, including the source package and two pre-compiled binary packages are located here:https://dist.apache.org/repos/dist/dev/kylin/apache-kylin-4.0.0-alpha-rc1/ The hash of the artifacts are as follows: apache-kylin-4.0.0-alpha-source-release.zip.sha256 f98da070a9839251c8cf3806c274c1191bb11d0b251288b4d2586e034f6ac291 apache-kylin-4.0.0-alpha-bin-hadoop2.tar.gz.sha256 8075af2608b62177f04bc5a528194c555959775ed69ae80e5ccaf9a37ec1bf74 apache-kylin-4.0.0-alpha-bin-cdh57.tar.gz.sha256 f98da070a9839251c8cf3806c274c1191bb11d0b251288b4d2586e034f6ac291 A staged Maven repository is available for review at:https://repository.apache.org/content/repositories/orgapachekylin-1081/ Release artifacts are signed with the following key:https://people.apache.org/keys/committer/nic.asc Please vote on releasing this package as Apache Kylin 4.0.0-alpha. The vote is open for the next 72 hours and passes if a majority of at least three +1 binding votes are cast. [ ] +1 Release this package as Apache Kylin 4.0.0-alpha [ ] 0 I don't feel strongly about it, but I'm okay with the release [ ] -1 Do not release this package because... Here is my vote: +1 (binding) - Best regards, Ni Chunen / George
??????[VOTE] Release apache-kylin-4.0.0-alpha (RC1)
+1 from my side. (non-binding) Best regards, Zhichao Zhang -- -- ??: "dev" https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316121&version=12348093 The commit to being voted upon:https://github.com/apache/kylin/commit/a285f9a5b84affc36c5466ce5a1b2fcdb4348b37 Its hash is a285f9a5b84affc36c5466ce5a1b2fcdb4348b37. The artifacts to be voted on, including the source package and two pre-compiled binary packages are located here:https://dist.apache.org/repos/dist/dev/kylin/apache-kylin-4.0.0-alpha-rc1/ The hash of the artifacts are as follows: apache-kylin-4.0.0-alpha-source-release.zip.sha256 f98da070a9839251c8cf3806c274c1191bb11d0b251288b4d2586e034f6ac291 apache-kylin-4.0.0-alpha-bin-hadoop2.tar.gz.sha256 8075af2608b62177f04bc5a528194c555959775ed69ae80e5ccaf9a37ec1bf74 apache-kylin-4.0.0-alpha-bin-cdh57.tar.gz.sha256 f98da070a9839251c8cf3806c274c1191bb11d0b251288b4d2586e034f6ac291 A staged Maven repository is available for review at:https://repository.apache.org/content/repositories/orgapachekylin-1081/ Release artifacts are signed with the following key:https://people.apache.org/keys/committer/nic.asc Please vote on releasing this package as Apache Kylin 4.0.0-alpha. The vote is open for the next 72 hours and passes if a majority of at least three +1 binding votes are cast. [ ] +1 Release this package as Apache Kylin 4.0.0-alpha [ ] 0 I don't feel strongly about it, but I'm okay with the release [ ] -1 Do not release this package because... Here is my vote: +1 (binding) - Best regards, Ni Chunen / George
[jira] [Created] (KYLIN-4751) Throws NPE when run test case TestTopNUDAF
Zhichao Zhang created KYLIN-4751: - Summary: Throws NPE when run test case TestTopNUDAF Key: KYLIN-4751 URL: https://issues.apache.org/jira/browse/KYLIN-4751 Project: Kylin Issue Type: Bug Components: Spark Engine Affects Versions: v4.0.0-alpha Reporter: Zhichao Zhang Assignee: Zhichao Zhang Fix For: v4.0.0-beta When run test case 'TestTopNUDAF' it will throw NPE. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re:回复：[VOTE] Release apache-kylin-4.0.0-alpha (RC1)
+1 mvn test passed on my machine; happy path passed on CDH 5.7. -- Best wishes to you ! From ：Xiaoxiang Yu 在 2020-09-09 08:21:57，"恩爸" <441586...@qq.com> 写道： >+1 from my side. (non-binding) > > > > > > > >Best regards, >Zhichao Zhang > > > > > > > > > >-- 原始邮件 -- >发件人: > "dev" > 发送时间: 2020年9月8日(星期二) 晚上11:49 >收件人: "dev" >主题: [VOTE] Release apache-kylin-4.0.0-alpha (RC1) > > > >Hi all, > >I have created a build for Apache Kylin 4.0.0-alpha, release candidate >1. Please note, this release is built on kylin-on-parquet-v2 branch. > > >Changes highlights: >[KYLIN-4213] - The new build engine with Spark-SQL >[KYLIN-4450] - Add the feature that adjusting spark driver memory adaptively >[KYLIN-4458] - FilePruner prune shards >[KYLIN-4475] - Support intersect count for Kylin on Parquet >[KYLIN-4462] - Support Count Distinct,TopN and Percentile by kylin on Parquet >[KYLIN-4713] - Support use diff spark schedule pool for diff query >[KYLIN-4468] - Support Percentile by kylin on Parquet >[KYLIN-4662] - Migrate from third-party Spark to offical Apache Spark >[KYLIN-4701] - Upgrade front-end from HBase Storage to Parquet Storage >[KYLIN-4644] - New tool to clean up intermediate files for Kylin 4.0 >[KYLIN-4744] - Add tracking URL for build spark job on yarn > >Thanks to everyone who has contributed to this release. > >Here are the release >notes:https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316121&version=12348093 > > >The commit to being voted >upon:https://github.com/apache/kylin/commit/a285f9a5b84affc36c5466ce5a1b2fcdb4348b37 > >Its hash is a285f9a5b84affc36c5466ce5a1b2fcdb4348b37. > >The artifacts to be voted on, including the source package and two >pre-compiled binary packages are located >here:https://dist.apache.org/repos/dist/dev/kylin/apache-kylin-4.0.0-alpha-rc1/ > >The hash of the artifacts are as follows: >apache-kylin-4.0.0-alpha-source-release.zip.sha256 >f98da070a9839251c8cf3806c274c1191bb11d0b251288b4d2586e034f6ac291 >apache-kylin-4.0.0-alpha-bin-hadoop2.tar.gz.sha256 >8075af2608b62177f04bc5a528194c555959775ed69ae80e5ccaf9a37ec1bf74 >apache-kylin-4.0.0-alpha-bin-cdh57.tar.gz.sha256 >f98da070a9839251c8cf3806c274c1191bb11d0b251288b4d2586e034f6ac291 > > >A staged Maven repository is available for review >at:https://repository.apache.org/content/repositories/orgapachekylin-1081/ > >Release artifacts are signed with the following >key:https://people.apache.org/keys/committer/nic.asc > > >Please vote on releasing this package as Apache Kylin 4.0.0-alpha. > > >The vote is open for the next 72 hours and passes if a majority of at >least three +1 binding votes are cast. > > >[ ] +1 Release this package as Apache Kylin 4.0.0-alpha >[ ] 0 I don't feel strongly about it, but I'm okay with the release >[ ] -1 Do not release this package because... > > >Here is my vote: > >+1 (binding) > >- > >Best regards, >Ni Chunen / George
Re: 回复：[VOTE] Release apache-kylin-4.0.0-alpha (RC1)
+1 (non-binding) mvn test passed on my machine; happy path passed on HDP2.4 && CDH 5.7. --- Best wishes, Rupeng Wang 在 2020/9/9 11:32，“Xiaoxiang Yu” 写入: +1 mvn test passed on my machine; happy path passed on CDH 5.7. -- Best wishes to you ! From ：Xiaoxiang Yu 在 2020-09-09 08:21:57，"恩爸" <441586...@qq.com> 写道： >+1 from my side. (non-binding) > >Best regards, >Zhichao Zhang >-- 原始邮件 -- >发件人: "dev" 发送时间: 2020年9月8日(星期二) 晚上11:49 >收件人: "dev" >主题: [VOTE] Release apache-kylin-4.0.0-alpha (RC1) > > > >Hi all, > >I have created a build for Apache Kylin 4.0.0-alpha, release candidate >1. Please note, this release is built on kylin-on-parquet-v2 branch. > > >Changes highlights: >[KYLIN-4213] - The new build engine with Spark-SQL >[KYLIN-4450] - Add the feature that adjusting spark driver memory adaptively >[KYLIN-4458] - FilePruner prune shards >[KYLIN-4475] - Support intersect count for Kylin on Parquet >[KYLIN-4462] - Support Count Distinct,TopN and Percentile by kylin on Parquet >[KYLIN-4713] - Support use diff spark schedule pool for diff query >[KYLIN-4468] - Support Percentile by kylin on Parquet >[KYLIN-4662] - Migrate from third-party Spark to offical Apache Spark >[KYLIN-4701] - Upgrade front-end from HBase Storage to Parquet Storage >[KYLIN-4644] - New tool to clean up intermediate files for Kylin 4.0 >[KYLIN-4744] - Add tracking URL for build spark job on yarn > >Thanks to everyone who has contributed to this release. > >Here are the release >notes:https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316121&version=12348093 > > >The commit to being voted >upon:https://github.com/apache/kylin/commit/a285f9a5b84affc36c5466ce5a1b2fcdb4348b37 > >Its hash is a285f9a5b84affc36c5466ce5a1b2fcdb4348b37. > >The artifacts to be voted on, including the source package and two >pre-compiled binary packages are located >here:https://dist.apache.org/repos/dist/dev/kylin/apache-kylin-4.0.0-alpha-rc1/ > >The hash of the artifacts are as follows: >apache-kylin-4.0.0-alpha-source-release.zip.sha256 >f98da070a9839251c8cf3806c274c1191bb11d0b251288b4d2586e034f6ac291 >apache-kylin-4.0.0-alpha-bin-hadoop2.tar.gz.sha256 >8075af2608b62177f04bc5a528194c555959775ed69ae80e5ccaf9a37ec1bf74 >apache-kylin-4.0.0-alpha-bin-cdh57.tar.gz.sha256 >f98da070a9839251c8cf3806c274c1191bb11d0b251288b4d2586e034f6ac291 > > >A staged Maven repository is available for review >at:https://repository.apache.org/content/repositories/orgapachekylin-1081/ > >Release artifacts are signed with the following >key:https://people.apache.org/keys/committer/nic.asc > > >Please vote on releasing this package as Apache Kylin 4.0.0-alpha. > > >The vote is open for the next 72 hours and passes if a majority of at >least three +1 binding votes are cast. > > >[ ] +1 Release this package as Apache Kylin 4.0.0-alpha >[ ] 0 I don't feel strongly about it, but I'm okay with the release >[ ] -1 Do not release this package because... > > >Here is my vote: > >+1 (binding) > >- > >Best regards, >Ni Chunen / George
[jira] [Created] (KYLIN-4752) Refine server mode checking
Zhong Yanghong created KYLIN-4752: - Summary: Refine server mode checking Key: KYLIN-4752 URL: https://issues.apache.org/jira/browse/KYLIN-4752 Project: Kylin Issue Type: Improvement Reporter: Zhong Yanghong Assignee: Zhong Yanghong -- This message was sent by Atlassian Jira (v8.3.4#803005)
