[kylin] branch add-registry created (now 8915b12)
This is an automated email from the ASF dual-hosted git repository. nic pushed a change to branch add-registry in repository https://gitbox.apache.org/repos/asf/kylin.git. at 8915b12 Fix some link (#1789) No new revisions were added by this update.
[kylin] branch v3.1.3-release created (now c8aadf3)
This is an automated email from the ASF dual-hosted git repository. nic pushed a change to branch v3.1.3-release in repository https://gitbox.apache.org/repos/asf/kylin.git. at c8aadf3 Modify shaded-guava version for release kylin-3.1.3 This branch includes the following new commits: new c8aadf3 Modify shaded-guava version for release kylin-3.1.3 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[kylin] 01/01: Modify shaded-guava version for release kylin-3.1.3
This is an automated email from the ASF dual-hosted git repository.
nic pushed a commit to branch v3.1.3-release
in repository https://gitbox.apache.org/repos/asf/kylin.git
commit c8aadf333a2760a72722990f56f3e186beff5e01
Author: nichunen
AuthorDate: Wed Dec 29 08:49:00 2021 +
Modify shaded-guava version for release kylin-3.1.3
---
pom.xml | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/pom.xml b/pom.xml
index dd4375d..068d6de 100644
--- a/pom.xml
+++ b/pom.xml
@@ -248,7 +248,7 @@
org.apache.kylin
kylin-shaded-guava
-${project.version}
+3.1.0
org.apache.kylin
@@ -1362,7 +1362,7 @@
-external
+
core-common
core-metadata
core-dictionary
[kylin] 01/01: Modify shaded-guava version from release kylin3.1.3
This is an automated email from the ASF dual-hosted git repository.
nic pushed a commit to branch v3.1.3-release
in repository https://gitbox.apache.org/repos/asf/kylin.git
commit 432041723ae5e3072fe6c3ce73f0c76ad62984e4
Author: nichunen
AuthorDate: Mon Dec 27 07:11:01 2021 +
Modify shaded-guava version from release kylin3.1.3
---
pom.xml | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/pom.xml b/pom.xml
index dd4375d..068d6de 100644
--- a/pom.xml
+++ b/pom.xml
@@ -248,7 +248,7 @@
org.apache.kylin
kylin-shaded-guava
-${project.version}
+3.1.0
org.apache.kylin
@@ -1362,7 +1362,7 @@
-external
+
core-common
core-metadata
core-dictionary
[kylin] branch v3.1.3-release updated (cbf982d -> 4320417)
This is an automated email from the ASF dual-hosted git repository. nic pushed a change to branch v3.1.3-release in repository https://gitbox.apache.org/repos/asf/kylin.git. discard cbf982d Modify shaded-guava version from release kylin3.1.3 new 4320417 Modify shaded-guava version from release kylin3.1.3 This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (cbf982d) \ N -- N -- N refs/heads/v3.1.3-release (4320417) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[kylin] 01/01: Modify shaded-guava version from release kylin3.1.3
This is an automated email from the ASF dual-hosted git repository.
nic pushed a commit to branch v3.1.3-release
in repository https://gitbox.apache.org/repos/asf/kylin.git
commit cbf982d44779bd2b319af93f7164a588f6363274
Author: nichunen
AuthorDate: Mon Dec 27 07:11:01 2021 +
Modify shaded-guava version from release kylin3.1.3
---
pom.xml | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/pom.xml b/pom.xml
index dd4375d..7a07bdd 100644
--- a/pom.xml
+++ b/pom.xml
@@ -248,7 +248,7 @@
org.apache.kylin
kylin-shaded-guava
-${project.version}
+3.1.0
org.apache.kylin
@@ -1362,7 +1362,7 @@
-external
+
core-common
core-metadata
core-dictionary
[kylin] branch v3.1.3-release created (now cbf982d)
This is an automated email from the ASF dual-hosted git repository. nic pushed a change to branch v3.1.3-release in repository https://gitbox.apache.org/repos/asf/kylin.git. at cbf982d Modify shaded-guava version from release kylin3.1.3 This branch includes the following new commits: new cbf982d Modify shaded-guava version from release kylin3.1.3 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[kylin] branch v4.0.1-release created (now b5eec8c)
This is an automated email from the ASF dual-hosted git repository. nic pushed a change to branch v4.0.1-release in repository https://gitbox.apache.org/repos/asf/kylin.git. at b5eec8c Modify shaded-guava version for release kylin4.0.1 This branch includes the following new commits: new b5eec8c Modify shaded-guava version for release kylin4.0.1 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[kylin] 01/01: Modify shaded-guava version for release kylin4.0.1
This is an automated email from the ASF dual-hosted git repository.
nic pushed a commit to branch v4.0.1-release
in repository https://gitbox.apache.org/repos/asf/kylin.git
commit b5eec8c7c155295177bc21a716def6bdb314989a
Author: nichunen
AuthorDate: Mon Dec 27 06:12:48 2021 +
Modify shaded-guava version for release kylin4.0.1
---
pom.xml | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/pom.xml b/pom.xml
index 8b5ae85..ba02797 100644
--- a/pom.xml
+++ b/pom.xml
@@ -257,7 +257,7 @@
org.apache.kylin
kylin-shaded-guava
-${project.version}
+3.1.0
org.apache.kylin
@@ -1527,7 +1527,7 @@
-external
+
core-common
core-metadata
[kylin] branch v4.0.1-release created (now adffcff)
This is an automated email from the ASF dual-hosted git repository. nic pushed a change to branch v4.0.1-release in repository https://gitbox.apache.org/repos/asf/kylin.git. at adffcff Modify shaded-guava for release kylin4.0.1 This branch includes the following new commits: new adffcff Modify shaded-guava for release kylin4.0.1 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[kylin] 01/01: Modify shaded-guava for release kylin4.0.1
This is an automated email from the ASF dual-hosted git repository.
nic pushed a commit to branch v4.0.1-release
in repository https://gitbox.apache.org/repos/asf/kylin.git
commit adffcff244511862af7d0b6f651032e5ae5c0ec9
Author: nichunen
AuthorDate: Fri Nov 19 14:14:46 2021 +
Modify shaded-guava for release kylin4.0.1
---
pom.xml | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/pom.xml b/pom.xml
index 8b5ae85..2e13541 100644
--- a/pom.xml
+++ b/pom.xml
@@ -257,7 +257,7 @@
org.apache.kylin
kylin-shaded-guava
-${project.version}
+3.1.0
org.apache.kylin
@@ -1527,7 +1527,7 @@
-external
+
core-common
core-metadata
[kylin] 01/01: Modify shaded-guava fro release kylin3.1.3
This is an automated email from the ASF dual-hosted git repository.
nic pushed a commit to branch v3.1.3-release
in repository https://gitbox.apache.org/repos/asf/kylin.git
commit eb2ef1a78918d6e11f77001e15a03b771075f561
Author: nichunen
AuthorDate: Fri Nov 19 12:42:56 2021 +
Modify shaded-guava fro release kylin3.1.3
---
pom.xml | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/pom.xml b/pom.xml
index dd4375d..472e67b 100644
--- a/pom.xml
+++ b/pom.xml
@@ -248,7 +248,7 @@
org.apache.kylin
kylin-shaded-guava
-${project.version}
+3.1.0
org.apache.kylin
@@ -1362,7 +1362,7 @@
-external
+
core-common
core-metadata
core-dictionary
[kylin] branch v3.1.3-release created (now eb2ef1a)
This is an automated email from the ASF dual-hosted git repository. nic pushed a change to branch v3.1.3-release in repository https://gitbox.apache.org/repos/asf/kylin.git. at eb2ef1a Modify shaded-guava fro release kylin3.1.3 This branch includes the following new commits: new eb2ef1a Modify shaded-guava fro release kylin3.1.3 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[kylin] 01/01: Modify shaded-guava for release kylin3.1.3
This is an automated email from the ASF dual-hosted git repository.
nic pushed a commit to branch v3.1.3-release
in repository https://gitbox.apache.org/repos/asf/kylin.git
commit 2d5c96e95b14869c56763a475dd6b3c249aeaceb
Author: nichunen
AuthorDate: Fri Nov 19 08:10:20 2021 +
Modify shaded-guava for release kylin3.1.3
---
pom.xml | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/pom.xml b/pom.xml
index dd4375d..472e67b 100644
--- a/pom.xml
+++ b/pom.xml
@@ -248,7 +248,7 @@
org.apache.kylin
kylin-shaded-guava
-${project.version}
+3.1.0
org.apache.kylin
@@ -1362,7 +1362,7 @@
-external
+
core-common
core-metadata
core-dictionary
[kylin] branch v3.1.3-release created (now 2d5c96e)
This is an automated email from the ASF dual-hosted git repository. nic pushed a change to branch v3.1.3-release in repository https://gitbox.apache.org/repos/asf/kylin.git. at 2d5c96e Modify shaded-guava for release kylin3.1.3 This branch includes the following new commits: new 2d5c96e Modify shaded-guava for release kylin3.1.3 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[kylin] branch v4.0.1-release created (now 5304912)
This is an automated email from the ASF dual-hosted git repository. nic pushed a change to branch v4.0.1-release in repository https://gitbox.apache.org/repos/asf/kylin.git. at 5304912 Modify shaded-guava for release kylin4.0.1 This branch includes the following new commits: new 5304912 Modify shaded-guava for release kylin4.0.1 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[kylin] 01/01: Modify shaded-guava for release kylin4.0.1
This is an automated email from the ASF dual-hosted git repository.
nic pushed a commit to branch v4.0.1-release
in repository https://gitbox.apache.org/repos/asf/kylin.git
commit 530491272c7d5abfffad533d1672bc661e42f946
Author: nichunen
AuthorDate: Fri Nov 19 07:20:18 2021 +
Modify shaded-guava for release kylin4.0.1
---
pom.xml | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/pom.xml b/pom.xml
index 8b5ae85..7db2a5d 100644
--- a/pom.xml
+++ b/pom.xml
@@ -257,7 +257,7 @@
org.apache.kylin
kylin-shaded-guava
-${project.version}
+3.1.0
org.apache.kylin
@@ -1527,7 +1527,7 @@
-external
+
core-common
core-metadata
[kylin] 01/01: Modify shaded-guava version for release kylin4.0.1
This is an automated email from the ASF dual-hosted git repository.
nic pushed a commit to branch v4.0.1-release
in repository https://gitbox.apache.org/repos/asf/kylin.git
commit 9545166c001ea66a7632410c98cd234df6ca63d1
Author: nichunen
AuthorDate: Thu Nov 18 10:10:22 2021 +
Modify shaded-guava version for release kylin4.0.1
---
pom.xml | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/pom.xml b/pom.xml
index 8b5ae85..321d4fe 100644
--- a/pom.xml
+++ b/pom.xml
@@ -257,7 +257,7 @@
org.apache.kylin
kylin-shaded-guava
-${project.version}
+3.1.0
org.apache.kylin
@@ -1527,7 +1527,7 @@
-external
+
core-common
core-metadata
[kylin] branch v4.0.1-release created (now 9545166)
This is an automated email from the ASF dual-hosted git repository. nic pushed a change to branch v4.0.1-release in repository https://gitbox.apache.org/repos/asf/kylin.git. at 9545166 Modify shaded-guava version for release kylin4.0.1 This branch includes the following new commits: new 9545166 Modify shaded-guava version for release kylin4.0.1 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[kylin] 01/01: Modify shaded-guava for release kylin3.1.3
This is an automated email from the ASF dual-hosted git repository.
nic pushed a commit to branch v3.1.3-release
in repository https://gitbox.apache.org/repos/asf/kylin.git
commit 5e53149dc115ccbe25897f650e55651c22ac07a4
Author: nichunen
AuthorDate: Thu Nov 18 08:31:08 2021 +
Modify shaded-guava for release kylin3.1.3
---
pom.xml | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/pom.xml b/pom.xml
index dd4375d..f7d0998 100644
--- a/pom.xml
+++ b/pom.xml
@@ -248,7 +248,7 @@
org.apache.kylin
kylin-shaded-guava
-${project.version}
+3.1.0
org.apache.kylin
@@ -1362,7 +1362,7 @@
-external
+
core-common
core-metadata
core-dictionary
[kylin] branch v3.1.3-release created (now 5e53149)
This is an automated email from the ASF dual-hosted git repository. nic pushed a change to branch v3.1.3-release in repository https://gitbox.apache.org/repos/asf/kylin.git. at 5e53149 Modify shaded-guava for release kylin3.1.3 This branch includes the following new commits: new 5e53149 Modify shaded-guava for release kylin3.1.3 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[kylin] branch v4.0.1-release created (now 12485b6)
This is an automated email from the ASF dual-hosted git repository. nic pushed a change to branch v4.0.1-release in repository https://gitbox.apache.org/repos/asf/kylin.git. at 12485b6 modify shaded-guava for release This branch includes the following new commits: new 12485b6 modify shaded-guava for release The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[kylin] 01/01: modify shaded-guava for release
This is an automated email from the ASF dual-hosted git repository.
nic pushed a commit to branch v4.0.1-release
in repository https://gitbox.apache.org/repos/asf/kylin.git
commit 12485b630f6ed40733ad6cee763f7ff50d79da37
Author: nichunen
AuthorDate: Wed Nov 17 10:46:23 2021 +
modify shaded-guava for release
---
pom.xml | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/pom.xml b/pom.xml
index 8b5ae85..7db2a5d 100644
--- a/pom.xml
+++ b/pom.xml
@@ -257,7 +257,7 @@
org.apache.kylin
kylin-shaded-guava
-${project.version}
+3.1.0
org.apache.kylin
@@ -1527,7 +1527,7 @@
-external
+
core-common
core-metadata
[kylin] branch v3.1.3-release created (now 8a174c8)
This is an automated email from the ASF dual-hosted git repository. nic pushed a change to branch v3.1.3-release in repository https://gitbox.apache.org/repos/asf/kylin.git. at 8a174c8 Modify shaded-guava version for release v3.1.3 This branch includes the following new commits: new 8a174c8 Modify shaded-guava version for release v3.1.3 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[kylin] 01/01: Modify shaded-guava version for release v3.1.3
This is an automated email from the ASF dual-hosted git repository.
nic pushed a commit to branch v3.1.3-release
in repository https://gitbox.apache.org/repos/asf/kylin.git
commit 8a174c8c9ca02dfd310e8b0bc6a085127027d2f0
Author: nichunen
AuthorDate: Wed Nov 17 08:54:27 2021 +
Modify shaded-guava version for release v3.1.3
---
pom.xml | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/pom.xml b/pom.xml
index dd4375d..d16728f 100644
--- a/pom.xml
+++ b/pom.xml
@@ -248,7 +248,7 @@
org.apache.kylin
kylin-shaded-guava
-${project.version}
+3.1.0
org.apache.kylin
@@ -1362,7 +1362,7 @@
-external
+
core-common
core-metadata
core-dictionary
[kylin] branch v3.1.3-release created (now 6b7f97a)
This is an automated email from the ASF dual-hosted git repository. nic pushed a change to branch v3.1.3-release in repository https://gitbox.apache.org/repos/asf/kylin.git. at 6b7f97a Modify shade-guava version info This branch includes the following new commits: new 6b7f97a Modify shade-guava version info The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[kylin] 01/01: Modify shade-guava version info
This is an automated email from the ASF dual-hosted git repository.
nic pushed a commit to branch v3.1.3-release
in repository https://gitbox.apache.org/repos/asf/kylin.git
commit 6b7f97acecd2c193d2e9fd9348508f6613ce3348
Author: nichunen
AuthorDate: Wed Nov 17 06:52:42 2021 +
Modify shade-guava version info
---
pom.xml | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/pom.xml b/pom.xml
index 8b5ae85..2e13541 100644
--- a/pom.xml
+++ b/pom.xml
@@ -257,7 +257,7 @@
org.apache.kylin
kylin-shaded-guava
-${project.version}
+3.1.0
org.apache.kylin
@@ -1527,7 +1527,7 @@
-external
+
core-common
core-metadata
[kylin] 01/01: Modify shade-guava version for release
This is an automated email from the ASF dual-hosted git repository.
nic pushed a commit to branch v3.1.3-release
in repository https://gitbox.apache.org/repos/asf/kylin.git
commit 170174f92994d74ec86553bf08ae8edf86347ff3
Author: nichunen
AuthorDate: Fri Nov 5 06:12:40 2021 +
Modify shade-guava version for release
---
pom.xml | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/pom.xml b/pom.xml
index dd4375d..2769e62 100644
--- a/pom.xml
+++ b/pom.xml
@@ -248,7 +248,7 @@
org.apache.kylin
kylin-shaded-guava
-${project.version}
+3.1.0
org.apache.kylin
@@ -1362,7 +1362,6 @@
-external
core-common
core-metadata
core-dictionary
[kylin] branch v3.1.3-release created (now 170174f)
This is an automated email from the ASF dual-hosted git repository. nic pushed a change to branch v3.1.3-release in repository https://gitbox.apache.org/repos/asf/kylin.git. at 170174f Modify shade-guava version for release This branch includes the following new commits: new 170174f Modify shade-guava version for release The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[kylin] branch v4.0.1-release updated: fix shade guava for release
This is an automated email from the ASF dual-hosted git repository.
nic pushed a commit to branch v4.0.1-release
in repository https://gitbox.apache.org/repos/asf/kylin.git
The following commit(s) were added to refs/heads/v4.0.1-release by this push:
new 49fa74f fix shade guava for release
49fa74f is described below
commit 49fa74fe2dd56f2a585956f7d0e26ea308a41d53
Author: nichunen
AuthorDate: Tue Nov 2 08:32:18 2021 +
fix shade guava for release
---
pom.xml | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/pom.xml b/pom.xml
index 1b794fd..9124cf6 100644
--- a/pom.xml
+++ b/pom.xml
@@ -257,7 +257,7 @@
org.apache.kylin
kylin-shaded-guava
-${project.version}
+3.1.0
org.apache.kylin
@@ -1527,7 +1527,7 @@
-external
+
core-common
core-metadata
[kylin] branch v4.0.1-release created (now 5187aff)
This is an automated email from the ASF dual-hosted git repository. nic pushed a change to branch v4.0.1-release in repository https://gitbox.apache.org/repos/asf/kylin.git. at 5187aff KYLIN-5069 Refactor hive and hadoop dependency of kylin4 No new revisions were added by this update.
[kylin] branch v4.0.0-release created (now a7b7edb)
This is an automated email from the ASF dual-hosted git repository. nic pushed a change to branch v4.0.0-release in repository https://gitbox.apache.org/repos/asf/kylin.git. at a7b7edb Add option to remove stale files under job_tmp for StorageCleanupJob (#1732) No new revisions were added by this update.
[kylin] branch kylin-on-parquet-v2 updated (0186251 -> dc2fc39)
This is an automated email from the ASF dual-hosted git repository.
nic pushed a change to branch kylin-on-parquet-v2
in repository https://gitbox.apache.org/repos/asf/kylin.git.
omit 0186251 [maven-release-plugin] prepare release mvn -DskipTests
-Papache-release -Darguments="-Dgpg.passphrase=${GPG_PASSPHRASE} -DskipTests"
release:perform
This update removed existing revisions from the reference, leaving the
reference pointing at a previous point in the repository history.
* -- * -- N refs/heads/kylin-on-parquet-v2 (dc2fc39)
\
O -- O -- O (0186251)
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
build-engine/pom.xml| 2 +-
cache/pom.xml | 2 +-
core-common/pom.xml | 2 +-
core-cube/pom.xml | 2 +-
core-dictionary/pom.xml | 2 +-
core-job/pom.xml| 2 +-
core-metadata/pom.xml | 2 +-
core-metrics/pom.xml| 2 +-
core-storage/pom.xml| 2 +-
datasource-sdk/pom.xml | 2 +-
external/pom.xml| 5 +++--
jdbc/pom.xml| 2 +-
kylin-spark-project/kylin-spark-classloader/pom.xml | 4 ++--
kylin-spark-project/kylin-spark-common/pom.xml | 4 ++--
kylin-spark-project/kylin-spark-engine/pom.xml | 2 +-
kylin-spark-project/kylin-spark-metadata/pom.xml| 2 +-
kylin-spark-project/kylin-spark-query/pom.xml | 2 +-
kylin-spark-project/kylin-spark-test/pom.xml| 2 +-
kylin-spark-project/pom.xml | 2 +-
metastore-hbase/pom.xml | 2 +-
metrics-reporter-hive/pom.xml | 2 +-
metrics-reporter-kafka/pom.xml | 2 +-
parquet-assembly/pom.xml| 2 +-
pom.xml | 4 ++--
query/pom.xml | 2 +-
server-base/pom.xml | 6 +++---
server/pom.xml | 4 ++--
source-hive/pom.xml | 2 +-
tool-assembly/pom.xml | 2 +-
tool/pom.xml| 2 +-
30 files changed, 38 insertions(+), 37 deletions(-)
[kylin] branch v3.1.1-release updated (d86ec25 -> eef2f48)
This is an automated email from the ASF dual-hosted git repository. nic pushed a change to branch v3.1.1-release in repository https://gitbox.apache.org/repos/asf/kylin.git. from d86ec25 KYLIN-4770: Move Kylin service on k8s from background to foreground add 44df5db KYLIN-4689 BuildJobSubmitter should obey MaxJobLimit add 53cbf18 KYLIN-4689 Purge inconsistent lock before check segmentBuildJobCheckList add eef2f48 KYLIN-4591 Release shaded-guava separately No new revisions were added by this update. Summary of changes: .../org/apache/kylin/job/impl/threadpool/FetcherRunner.java | 2 +- external/pom.xml | 12 ++-- pom.xml | 4 ++-- .../stream/coordinator/coordinate/BuildJobSubmitter.java | 11 +++ 4 files changed, 20 insertions(+), 9 deletions(-)
[kylin] branch v3.1.1-release created (now d86ec25)
This is an automated email from the ASF dual-hosted git repository. nic pushed a change to branch v3.1.1-release in repository https://gitbox.apache.org/repos/asf/kylin.git. at d86ec25 KYLIN-4770: Move Kylin service on k8s from background to foreground No new revisions were added by this update.
[kylin] branch v4.0.0-alpha-release created (now 3405ab2)
This is an automated email from the ASF dual-hosted git repository. nic pushed a change to branch v4.0.0-alpha-release in repository https://gitbox.apache.org/repos/asf/kylin.git. at 3405ab2 KYLIN-4719 Refine kylin-defaults.properties for parquet Storage No new revisions were added by this update.
[kylin] 01/02: KYLIN-4597 Fix NPE when download diagnosis info for a job
This is an automated email from the ASF dual-hosted git repository.
nic pushed a commit to branch v3.1.0-release
in repository https://gitbox.apache.org/repos/asf/kylin.git
commit 5a815c40847240af4bb035c1b05d4567c0b13abb
Author: Zhichao Zhang <441586...@qq.com>
AuthorDate: Mon Jun 22 23:41:08 2020 +0800
KYLIN-4597 Fix NPE when download diagnosis info for a job
Problems:
When download diagnosis info for a job, it throws NPE.
Solutions:
Don't replace '-' to '' for job id, which will change the value of job id
and can't find the job.
---
.../src/main/java/org/apache/kylin/common/util/CliCommandExecutor.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/core-common/src/main/java/org/apache/kylin/common/util/CliCommandExecutor.java
b/core-common/src/main/java/org/apache/kylin/common/util/CliCommandExecutor.java
index 74ea1f9..bb08b4c 100644
---
a/core-common/src/main/java/org/apache/kylin/common/util/CliCommandExecutor.java
+++
b/core-common/src/main/java/org/apache/kylin/common/util/CliCommandExecutor.java
@@ -163,7 +163,7 @@ public class CliCommandExecutor {
}
}
-public static final String COMMAND_BLOCK_LIST = "[
&`>|{}()$;\\-#~!+*]+";
+public static final String COMMAND_BLOCK_LIST = "[
&`>|{}()$;\\#~!+*]+";
public static final String COMMAND_WHITE_LIST = "[^\\w%,@/:=?.\"\\[\\]]";
public static final String HIVE_BLOCK_LIST = "[ <>()$;\\-#!+*\"'/=%@]+";
[kylin] branch v3.1.0-release created (now 14c2ad3)
This is an automated email from the ASF dual-hosted git repository. nic pushed a change to branch v3.1.0-release in repository https://gitbox.apache.org/repos/asf/kylin.git. at 14c2ad3 KYLIN-4598 Missing dependency when run kylin.sh org.apache.kylin.* This branch includes the following new commits: new 5a815c4 KYLIN-4597 Fix NPE when download diagnosis info for a job new 14c2ad3 KYLIN-4598 Missing dependency when run kylin.sh org.apache.kylin.* The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[kylin] 02/02: KYLIN-4598 Missing dependency when run kylin.sh org.apache.kylin.*
This is an automated email from the ASF dual-hosted git repository.
nic pushed a commit to branch v3.1.0-release
in repository https://gitbox.apache.org/repos/asf/kylin.git
commit 14c2ad3e401649dd7d628755917a295942f6c69a
Author: Zhichao Zhang <441586...@qq.com>
AuthorDate: Mon Jun 22 23:47:55 2020 +0800
KYLIN-4598 Missing dependency when run kylin.sh org.apache.kylin.*
Problems:
It throws 'NoClassDefFoundError:
org/apache/kafka/clients/producer/Producer' when run kylin.sh
org.apache.kylin.source.kafka.util.KafkaSampleProducer.
Solutions:
Add 'export kafka_dependency' before export HBASE_CLASSPATH in kylin.sh
---
build/bin/kylin.sh | 6 ++
1 file changed, 6 insertions(+)
diff --git a/build/bin/kylin.sh b/build/bin/kylin.sh
index 59d6de7..d56976b 100755
--- a/build/bin/kylin.sh
+++ b/build/bin/kylin.sh
@@ -40,10 +40,16 @@ function retrieveDependency() {
source ${dir}/cached-hive-dependency.sh
source ${dir}/cached-hbase-dependency.sh
source ${dir}/cached-hadoop-conf-dir.sh
+source ${dir}/cached-kafka-dependency.sh
+source ${dir}/cached-spark-dependency.sh
+source ${dir}/cached-flink-dependency.sh
else
source ${dir}/find-hive-dependency.sh
source ${dir}/find-hbase-dependency.sh
source ${dir}/find-hadoop-conf-dir.sh
+source ${dir}/find-kafka-dependency.sh
+source ${dir}/find-spark-dependency.sh
+source ${dir}/find-flink-dependency.sh
fi
#retrive $KYLIN_EXTRA_START_OPTS
[kylin] 01/02: KYLIN-4597 Fix NPE when download diagnosis info for a job
This is an automated email from the ASF dual-hosted git repository.
nic pushed a commit to branch v3.1.0-release
in repository https://gitbox.apache.org/repos/asf/kylin.git
commit 6d5f5b0aba1207e2a187bd4e9d2428ad5e09ca51
Author: Zhichao Zhang <441586...@qq.com>
AuthorDate: Mon Jun 22 23:41:08 2020 +0800
KYLIN-4597 Fix NPE when download diagnosis info for a job
Problems:
When download diagnosis info for a job, it throws NPE.
Solutions:
Don't replace '-' to '' for job id, which will change the value of job id
and can't find the job.
---
.../src/main/java/org/apache/kylin/common/util/CliCommandExecutor.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/core-common/src/main/java/org/apache/kylin/common/util/CliCommandExecutor.java
b/core-common/src/main/java/org/apache/kylin/common/util/CliCommandExecutor.java
index 74ea1f9..bb08b4c 100644
---
a/core-common/src/main/java/org/apache/kylin/common/util/CliCommandExecutor.java
+++
b/core-common/src/main/java/org/apache/kylin/common/util/CliCommandExecutor.java
@@ -163,7 +163,7 @@ public class CliCommandExecutor {
}
}
-public static final String COMMAND_BLOCK_LIST = "[
&`>|{}()$;\\-#~!+*]+";
+public static final String COMMAND_BLOCK_LIST = "[
&`>|{}()$;\\#~!+*]+";
public static final String COMMAND_WHITE_LIST = "[^\\w%,@/:=?.\"\\[\\]]";
public static final String HIVE_BLOCK_LIST = "[ <>()$;\\-#!+*\"'/=%@]+";
[kylin] 02/02: KYLIN-4598 Missing dependency when run kylin.sh org.apache.kylin.*
This is an automated email from the ASF dual-hosted git repository.
nic pushed a commit to branch v3.1.0-release
in repository https://gitbox.apache.org/repos/asf/kylin.git
commit 30ff1ec11201ca1fedf0f177bdfa3fe780ae90cc
Author: Zhichao Zhang <441586...@qq.com>
AuthorDate: Mon Jun 22 23:47:55 2020 +0800
KYLIN-4598 Missing dependency when run kylin.sh org.apache.kylin.*
Problems:
It throws 'NoClassDefFoundError:
org/apache/kafka/clients/producer/Producer' when run kylin.sh
org.apache.kylin.source.kafka.util.KafkaSampleProducer.
Solutions:
Add 'export kafka_dependency' before export HBASE_CLASSPATH in kylin.sh
---
build/bin/kylin.sh | 6 ++
1 file changed, 6 insertions(+)
diff --git a/build/bin/kylin.sh b/build/bin/kylin.sh
index 59d6de7..d56976b 100755
--- a/build/bin/kylin.sh
+++ b/build/bin/kylin.sh
@@ -40,10 +40,16 @@ function retrieveDependency() {
source ${dir}/cached-hive-dependency.sh
source ${dir}/cached-hbase-dependency.sh
source ${dir}/cached-hadoop-conf-dir.sh
+source ${dir}/cached-kafka-dependency.sh
+source ${dir}/cached-spark-dependency.sh
+source ${dir}/cached-flink-dependency.sh
else
source ${dir}/find-hive-dependency.sh
source ${dir}/find-hbase-dependency.sh
source ${dir}/find-hadoop-conf-dir.sh
+source ${dir}/find-kafka-dependency.sh
+source ${dir}/find-spark-dependency.sh
+source ${dir}/find-flink-dependency.sh
fi
#retrive $KYLIN_EXTRA_START_OPTS
[kylin] branch v3.1.0-release updated (fb842bc -> 30ff1ec)
This is an automated email from the ASF dual-hosted git repository. nic pushed a change to branch v3.1.0-release in repository https://gitbox.apache.org/repos/asf/kylin.git. discard fb842bc [maven-release-plugin] prepare release kylin-3.1.0 discard 4512b89 KYLIN-4598 Missing dependency when run kylin.sh org.apache.kylin.* discard da44e57 KYLIN-4597 Fix NPE when download diagnosis info for a job new 6d5f5b0 KYLIN-4597 Fix NPE when download diagnosis info for a job new 30ff1ec KYLIN-4598 Missing dependency when run kylin.sh org.apache.kylin.* This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (fb842bc) \ N -- N -- N refs/heads/v3.1.0-release (30ff1ec) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: assembly/pom.xml | 2 +- cache/pom.xml | 5 +++-- core-common/pom.xml| 2 +- core-cube/pom.xml | 2 +- core-dictionary/pom.xml| 2 +- core-job/pom.xml | 2 +- core-metadata/pom.xml | 2 +- core-metrics/pom.xml | 2 +- core-storage/pom.xml | 2 +- cube-migration/pom.xml | 2 +- datasource-sdk/pom.xml | 6 -- engine-flink/pom.xml | 6 -- engine-mr/pom.xml | 2 +- engine-spark/pom.xml | 2 +- jdbc/pom.xml | 2 +- kylin-it/pom.xml | 4 ++-- metrics-reporter-hive/pom.xml | 2 +- metrics-reporter-kafka/pom.xml | 2 +- pom.xml| 11 ++- query/pom.xml | 2 +- server-base/pom.xml| 2 +- server/pom.xml | 2 +- source-hive/pom.xml| 2 +- source-jdbc/pom.xml| 5 +++-- source-kafka/pom.xml | 2 +- storage-hbase/pom.xml | 8 +--- storage-stream/pom.xml | 2 +- stream-coordinator/pom.xml | 5 +++-- stream-core/pom.xml| 2 +- stream-receiver/pom.xml| 14 +- stream-source-kafka/pom.xml| 2 +- tomcat-ext/pom.xml | 2 +- tool-assembly/pom.xml | 2 +- tool/pom.xml | 2 +- 34 files changed, 64 insertions(+), 50 deletions(-)
[kylin] 01/02: KYLIN-4597 Fix NPE when download diagnosis info for a job
This is an automated email from the ASF dual-hosted git repository.
nic pushed a commit to branch v3.1.0-release
in repository https://gitbox.apache.org/repos/asf/kylin.git
commit da44e571607e28c7c247e0255f711fe42586b294
Author: Zhichao Zhang <441586...@qq.com>
AuthorDate: Mon Jun 22 23:41:08 2020 +0800
KYLIN-4597 Fix NPE when download diagnosis info for a job
Problems:
When download diagnosis info for a job, it throws NPE.
Solutions:
Don't replace '-' to '' for job id, which will change the value of job id
and can't find the job.
---
.../src/main/java/org/apache/kylin/common/util/CliCommandExecutor.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/core-common/src/main/java/org/apache/kylin/common/util/CliCommandExecutor.java
b/core-common/src/main/java/org/apache/kylin/common/util/CliCommandExecutor.java
index 74ea1f9..bb08b4c 100644
---
a/core-common/src/main/java/org/apache/kylin/common/util/CliCommandExecutor.java
+++
b/core-common/src/main/java/org/apache/kylin/common/util/CliCommandExecutor.java
@@ -163,7 +163,7 @@ public class CliCommandExecutor {
}
}
-public static final String COMMAND_BLOCK_LIST = "[
&`>|{}()$;\\-#~!+*]+";
+public static final String COMMAND_BLOCK_LIST = "[
&`>|{}()$;\\#~!+*]+";
public static final String COMMAND_WHITE_LIST = "[^\\w%,@/:=?.\"\\[\\]]";
public static final String HIVE_BLOCK_LIST = "[ <>()$;\\-#!+*\"'/=%@]+";
[kylin] 02/02: KYLIN-4598 Missing dependency when run kylin.sh org.apache.kylin.*
This is an automated email from the ASF dual-hosted git repository.
nic pushed a commit to branch v3.1.0-release
in repository https://gitbox.apache.org/repos/asf/kylin.git
commit 4512b8948db90c43c348e1f3d61326587aeb9961
Author: Zhichao Zhang <441586...@qq.com>
AuthorDate: Mon Jun 22 23:47:55 2020 +0800
KYLIN-4598 Missing dependency when run kylin.sh org.apache.kylin.*
Problems:
It throws 'NoClassDefFoundError:
org/apache/kafka/clients/producer/Producer' when run kylin.sh
org.apache.kylin.source.kafka.util.KafkaSampleProducer.
Solutions:
Add 'export kafka_dependency' before export HBASE_CLASSPATH in kylin.sh
---
build/bin/kylin.sh | 6 ++
1 file changed, 6 insertions(+)
diff --git a/build/bin/kylin.sh b/build/bin/kylin.sh
index 59d6de7..d56976b 100755
--- a/build/bin/kylin.sh
+++ b/build/bin/kylin.sh
@@ -40,10 +40,16 @@ function retrieveDependency() {
source ${dir}/cached-hive-dependency.sh
source ${dir}/cached-hbase-dependency.sh
source ${dir}/cached-hadoop-conf-dir.sh
+source ${dir}/cached-kafka-dependency.sh
+source ${dir}/cached-spark-dependency.sh
+source ${dir}/cached-flink-dependency.sh
else
source ${dir}/find-hive-dependency.sh
source ${dir}/find-hbase-dependency.sh
source ${dir}/find-hadoop-conf-dir.sh
+source ${dir}/find-kafka-dependency.sh
+source ${dir}/find-spark-dependency.sh
+source ${dir}/find-flink-dependency.sh
fi
#retrive $KYLIN_EXTRA_START_OPTS
[kylin] branch v3.1.0-release updated (c02bb74 -> 4512b89)
This is an automated email from the ASF dual-hosted git repository. nic pushed a change to branch v3.1.0-release in repository https://gitbox.apache.org/repos/asf/kylin.git. omit c02bb74 KYLIN-4598 Missing dependency when run kylin.sh org.apache.kylin.* omit 9e700ab KYLIN-4597 Fix NPE when download diagnosis info for a job add 8a79f60 KYLIN-4591 Release shaded-guava separately new da44e57 KYLIN-4597 Fix NPE when download diagnosis info for a job new 4512b89 KYLIN-4598 Missing dependency when run kylin.sh org.apache.kylin.* This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (c02bb74) \ N -- N -- N refs/heads/v3.1.0-release (4512b89) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: external/pom.xml | 16 external/shaded-guava/pom.xml | 9 ++--- pom.xml | 5 +++-- 3 files changed, 9 insertions(+), 21 deletions(-)
[kylin] branch v3.1.0-release created (now c02bb74)
This is an automated email from the ASF dual-hosted git repository. nic pushed a change to branch v3.1.0-release in repository https://gitbox.apache.org/repos/asf/kylin.git. at c02bb74 KYLIN-4598 Missing dependency when run kylin.sh org.apache.kylin.* No new revisions were added by this update.
[kylin] branch v3.1.0-release created (now 8a79f60)
This is an automated email from the ASF dual-hosted git repository. nic pushed a change to branch v3.1.0-release in repository https://gitbox.apache.org/repos/asf/kylin.git. at 8a79f60 KYLIN-4591 Release shaded-guava separately No new revisions were added by this update.
[kylin] 01/01: [maven-release-plugin] prepare for next development iteration
This is an automated email from the ASF dual-hosted git repository. nic pushed a commit to branch v3.1.0-release in repository https://gitbox.apache.org/repos/asf/kylin.git commit ec079287c345d491a954dbe4cf7a0dbf3a306e17 Author: nichunen AuthorDate: Sun Jun 21 07:49:40 2020 + [maven-release-plugin] prepare for next development iteration --- assembly/pom.xml | 2 +- cache/pom.xml | 5 ++--- core-common/pom.xml| 2 +- core-cube/pom.xml | 2 +- core-dictionary/pom.xml| 2 +- core-job/pom.xml | 2 +- core-metadata/pom.xml | 2 +- core-metrics/pom.xml | 2 +- core-storage/pom.xml | 2 +- cube-migration/pom.xml | 2 +- datasource-sdk/pom.xml | 6 ++ engine-flink/pom.xml | 6 ++ engine-mr/pom.xml | 2 +- engine-spark/pom.xml | 2 +- jdbc/pom.xml | 2 +- kylin-it/pom.xml | 4 ++-- metrics-reporter-hive/pom.xml | 2 +- metrics-reporter-kafka/pom.xml | 2 +- pom.xml| 9 - query/pom.xml | 2 +- server-base/pom.xml| 2 +- server/pom.xml | 2 +- source-hive/pom.xml| 2 +- source-jdbc/pom.xml| 5 ++--- source-kafka/pom.xml | 2 +- storage-hbase/pom.xml | 8 +++- storage-stream/pom.xml | 2 +- stream-coordinator/pom.xml | 5 ++--- stream-core/pom.xml| 2 +- stream-receiver/pom.xml| 14 +- stream-source-kafka/pom.xml| 2 +- tomcat-ext/pom.xml | 2 +- tool-assembly/pom.xml | 2 +- tool/pom.xml | 2 +- 34 files changed, 49 insertions(+), 63 deletions(-) diff --git a/assembly/pom.xml b/assembly/pom.xml index 4d58c91..42e1ff8 100644 --- a/assembly/pom.xml +++ b/assembly/pom.xml @@ -26,7 +26,7 @@ kylin org.apache.kylin -3.1.0-SNAPSHOT +3.1.1-SNAPSHOT diff --git a/cache/pom.xml b/cache/pom.xml index 57052f8..def1f99 100644 --- a/cache/pom.xml +++ b/cache/pom.xml @@ -15,8 +15,7 @@ limitations under the License. --> -http://www.w3.org/2001/XMLSchema-instance; xmlns="http://maven.apache.org/POM/4.0.0; - xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd;> +http://maven.apache.org/POM/4.0.0; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd;> 4.0.0 kylin-cache @@ -27,7 +26,7 @@ org.apache.kylin kylin -3.1.0-SNAPSHOT +3.1.1-SNAPSHOT diff --git a/core-common/pom.xml b/core-common/pom.xml index 407ab9c..5ec2838 100644 --- a/core-common/pom.xml +++ b/core-common/pom.xml @@ -28,7 +28,7 @@ org.apache.kylin kylin -3.1.0-SNAPSHOT +3.1.1-SNAPSHOT diff --git a/core-cube/pom.xml b/core-cube/pom.xml index 812bfbf..7828ccd 100644 --- a/core-cube/pom.xml +++ b/core-cube/pom.xml @@ -28,7 +28,7 @@ org.apache.kylin kylin -3.1.0-SNAPSHOT +3.1.1-SNAPSHOT diff --git a/core-dictionary/pom.xml b/core-dictionary/pom.xml index 9645ac9..a079566 100644 --- a/core-dictionary/pom.xml +++ b/core-dictionary/pom.xml @@ -28,7 +28,7 @@ org.apache.kylin kylin -3.1.0-SNAPSHOT +3.1.1-SNAPSHOT diff --git a/core-job/pom.xml b/core-job/pom.xml index d920b0e..cf2e4c9 100644 --- a/core-job/pom.xml +++ b/core-job/pom.xml @@ -28,7 +28,7 @@ org.apache.kylin kylin -3.1.0-SNAPSHOT +3.1.1-SNAPSHOT diff --git a/core-metadata/pom.xml b/core-metadata/pom.xml index a76e268..3e05f0f 100644 --- a/core-metadata/pom.xml +++ b/core-metadata/pom.xml @@ -28,7 +28,7 @@ org.apache.kylin kylin -3.1.0-SNAPSHOT +3.1.1-SNAPSHOT diff --git a/core-metrics/pom.xml b/core-metrics/pom.xml index eca9e5e..70cf077 100644 --- a/core-metrics/pom.xml +++ b/core-metrics/pom.xml @@ -28,7 +28,7 @@ kylin org.apache.kylin -3.1.0-SNAPSHOT +3.1.1-SNAPSHOT diff --git a/core-storage/pom.xml b/core-storage/pom.xml index e79b313..ebbadda 100644 --- a/core-storage/pom.xml +++ b/core-storage/pom.xml @@ -28,7 +28,7 @@ org.apache.kylin kylin -3.1.0-SNAPSHOT +3.1.1-SNAPSHOT diff --git a/cube-migration/pom.xml b/cube-migration/pom.xml index 4f75de5..8c24745 100755 --- a/cube-migration/pom.xml +++ b/cube-migration/pom.xml @@ -28,7 +28,7 @@ org.apache.kylin kylin -3.1.0-SNAPSHOT +3.1.1-SNAPSHOT diff --git a/datasource-sdk/pom.xml b
[kylin] branch v3.1.0-release created (now ec07928)
This is an automated email from the ASF dual-hosted git repository. nic pushed a change to branch v3.1.0-release in repository https://gitbox.apache.org/repos/asf/kylin.git. at ec07928 [maven-release-plugin] prepare for next development iteration This branch includes the following new commits: new ec07928 [maven-release-plugin] prepare for next development iteration The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[kylin] branch v3.1.0-release updated (77cbfc4 -> 414b788)
This is an automated email from the ASF dual-hosted git repository. nic pushed a change to branch v3.1.0-release in repository https://gitbox.apache.org/repos/asf/kylin.git. from 77cbfc4 KYLIN-3832 Disable all convert and quote behavior for postgresql add 8d0fc5d KYLIN-4371 Fix ut add 996d249 KYLIN-4580 Add 90, 360 days auto-merge thresholds for system cubes add 414b788 KYLIN-4591 Release shaded-guava separately No new revisions were added by this update. Summary of changes: external/pom.xml | 16 external/shaded-guava/pom.xml| 9 ++--- pom.xml | 3 +-- .../kylin/tool/metrics/systemcube/CubeDescCreator.java | 2 +- .../kylin/tool/metrics/systemcube/SCCreatorTest.java | 1 + tool/src/test/resources/SCSinkTools.json | 2 +- 6 files changed, 10 insertions(+), 23 deletions(-)
[kylin] 08/10: Package kafka and curator
This is an automated email from the ASF dual-hosted git repository.
nic pushed a commit to branch master-hadoop3
in repository https://gitbox.apache.org/repos/asf/kylin.git
commit b5710095462938416c99d0d659c71cee025633f3
Author: nichunen
AuthorDate: Tue Jul 23 18:09:44 2019 +0800
Package kafka and curator
---
pom.xml | 21 ++---
1 file changed, 18 insertions(+), 3 deletions(-)
diff --git a/pom.xml b/pom.xml
index 017859f..1d8d9b6 100644
--- a/pom.xml
+++ b/pom.xml
@@ -555,19 +555,34 @@
org.apache.curator
curator-framework
${curator.version}
-provided
+
+
+org.apache.zookeeper
+zookeeper
+
+
org.apache.curator
curator-recipes
${curator.version}
-provided
+
+
+org.apache.zookeeper
+zookeeper
+
+
org.apache.curator
curator-client
${curator.version}
-provided
+
+
+org.apache.zookeeper
+zookeeper
+
+
com.google.code.findbugs
[kylin] 09/10: KYLIN-3973: InvalidProtocolBufferException: Protocol message was too large.
This is an automated email from the ASF dual-hosted git repository.
nic pushed a commit to branch master-hadoop3
in repository https://gitbox.apache.org/repos/asf/kylin.git
commit bef71cfc9bb56e03c478104fbd46e876f1d43bf4
Author: Zhou Kang
AuthorDate: Thu Sep 12 20:04:44 2019 +0800
KYLIN-3973: InvalidProtocolBufferException: Protocol message was too large.
---
.../hbase/cube/v2/coprocessor/endpoint/generated/CubeVisitProtos.java | 2 ++
1 file changed, 2 insertions(+)
diff --git
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/cube/v2/coprocessor/endpoint/generated/CubeVisitProtos.java
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/cube/v2/coprocessor/endpoint/generated/CubeVisitProtos.java
index 2da6020..576e5df 100644
---
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/cube/v2/coprocessor/endpoint/generated/CubeVisitProtos.java
+++
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/cube/v2/coprocessor/endpoint/generated/CubeVisitProtos.java
@@ -4,6 +4,7 @@
package org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated;
public final class CubeVisitProtos {
+ private static final int DEFAULT_SIZE_LIMIT = Integer.MAX_VALUE;
private CubeVisitProtos() {}
public static void registerAllExtensions(
com.google.protobuf.ExtensionRegistry registry) {
@@ -2296,6 +2297,7 @@ public final class CubeVisitProtos {
com.google.protobuf.ExtensionRegistryLite extensionRegistry)
throws com.google.protobuf.InvalidProtocolBufferException {
initFields();
+ input.setSizeLimit(DEFAULT_SIZE_LIMIT);
int mutable_bitField0_ = 0;
com.google.protobuf.UnknownFieldSet.Builder unknownFields =
com.google.protobuf.UnknownFieldSet.newBuilder();
[kylin] 03/10: Fix dependency issues for branch of hadoop3
This is an automated email from the ASF dual-hosted git repository.
nic pushed a commit to branch master-hadoop3
in repository https://gitbox.apache.org/repos/asf/kylin.git
commit bc02c2052d94033f46c14bad1778948bf6b60f02
Author: nic
AuthorDate: Fri Jun 19 17:52:33 2020 +0800
Fix dependency issues for branch of hadoop3
---
core-common/pom.xml| 4 +++
.../mr/common/DefaultSslProtocolSocketFactory.java | 0
pom.xml| 7 +++-
.../apache/kylin/rest/job/KylinHealthCheckJob.java | 11 ---
.../kylin/rest/job/StorageCleanJobHbaseUtil.java | 2 +-
.../org/apache/kylin/rest/service/CubeService.java | 5 +--
.../kylin/storage/hbase/steps/CreateHTableJob.java | 3 +-
.../kylin/storage/hbase/steps/CubeHFileJob.java| 9 ++---
.../kylin/storage/hbase/steps/CubeHTableUtil.java | 4 ++-
.../kylin/storage/hbase/steps/HBaseFlinkSteps.java | 1 +
.../kylin/storage/hbase/steps/HBaseSparkSteps.java | 1 +
.../storage/hbase/steps/HFileOutputFormat3.java| 38 --
.../storage/hbase/util/DeployCoprocessorCLI.java | 8 -
.../kylin/storage/hbase/util/StorageCleanUtil.java | 13
stream-receiver/pom.xml| 4 +++
.../kylin/tool/extractor/HBaseUsageExtractor.java | 2 +-
.../migration/CubeMigrationCrossClusterCLI.java| 7 ++--
.../kylin/tool/migration/DstClusterUtil.java | 2 +-
18 files changed, 72 insertions(+), 49 deletions(-)
diff --git a/core-common/pom.xml b/core-common/pom.xml
index 6266558..407ab9c 100644
--- a/core-common/pom.xml
+++ b/core-common/pom.xml
@@ -114,5 +114,9 @@
curator-test
test
+
+org.apache.kylin
+kylin-shaded-guava
+
diff --git
a/engine-mr/src/main/java/org/apache/kylin/engine/mr/common/DefaultSslProtocolSocketFactory.java
b/engine-mr/src/main/java/org/apache/kylin/engine/mr/common/DefaultSslProtocolSocketFactory.java
deleted file mode 100644
index e69de29..000
diff --git a/pom.xml b/pom.xml
index 30a4f8f..017859f 100644
--- a/pom.xml
+++ b/pom.xml
@@ -1069,6 +1069,11 @@
org.eclipse.jetty
+jetty-runner
+${jetty.version}
+
+
+org.eclipse.jetty
jetty-util
${jetty.version}
test
@@ -1243,7 +1248,7 @@
org.apache.maven.plugins
maven-shade-plugin
-3.0.0
+3.1.0
org.apache.maven.plugins
diff --git
a/server-base/src/main/java/org/apache/kylin/rest/job/KylinHealthCheckJob.java
b/server-base/src/main/java/org/apache/kylin/rest/job/KylinHealthCheckJob.java
index 21fa784..bb3614a 100644
---
a/server-base/src/main/java/org/apache/kylin/rest/job/KylinHealthCheckJob.java
+++
b/server-base/src/main/java/org/apache/kylin/rest/job/KylinHealthCheckJob.java
@@ -29,8 +29,9 @@ import org.apache.commons.cli.OptionBuilder;
import org.apache.commons.cli.Options;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
-import org.apache.hadoop.hbase.HBaseConfiguration;
-import org.apache.hadoop.hbase.client.HBaseAdmin;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Admin;
+import org.apache.hadoop.hbase.client.Connection;
import org.apache.kylin.common.KylinConfig;
import org.apache.kylin.common.util.AbstractApplication;
import org.apache.kylin.common.util.BufferedLogger;
@@ -49,6 +50,7 @@ import org.apache.kylin.job.execution.CheckpointExecutable;
import org.apache.kylin.job.execution.ExecutableState;
import org.apache.kylin.metadata.model.DataModelManager;
import org.apache.kylin.metadata.model.SegmentStatusEnum;
+import org.apache.kylin.storage.hbase.HBaseConnection;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -199,13 +201,14 @@ public class KylinHealthCheckJob extends
AbstractApplication {
private void checkHBaseTables(List cubes) throws IOException
{
reporter.log("## Checking HBase Table of segments");
-HBaseAdmin hbaseAdmin = new HBaseAdmin(HBaseConfiguration.create());
+Connection conn =
HBaseConnection.get(KylinConfig.getInstanceFromEnv().getStorageUrl());
+Admin hbaseAdmin = conn.getAdmin();
try {
for (CubeInstance cube : cubes) {
for (CubeSegment segment : cube.getSegments()) {
if (segment.getStatus() != SegmentStatusEnum.NEW) {
String tableName =
segment.getStorageLocationIdentifier();
-if ((!hbaseAdmin.tableExists(tableName)) ||
(!hbaseAdmin.isTableEnabled(tableName))) {
+if
((!hbaseAdmin.tableExists(TableName.valueOf(tableName)) ||
(!hbaseAdmin.isTa
[kylin] 07/10: KYLIN-4086 Upgrade avatica to 1.12
This is an automated email from the ASF dual-hosted git repository.
nic pushed a commit to branch master-hadoop3
in repository https://gitbox.apache.org/repos/asf/kylin.git
commit d30a34ae1b7a4f31f52c17b05a4eaf4c31ae9c5b
Author: nichunen
AuthorDate: Mon Jul 22 23:21:50 2019 +0800
KYLIN-4086 Upgrade avatica to 1.12
---
build/script/elimate-jar-conflict.sh | 12 ++--
1 file changed, 6 insertions(+), 6 deletions(-)
diff --git a/build/script/elimate-jar-conflict.sh
b/build/script/elimate-jar-conflict.sh
index d02a874..84bec69 100644
--- a/build/script/elimate-jar-conflict.sh
+++ b/build/script/elimate-jar-conflict.sh
@@ -6,13 +6,13 @@ unzip kylin.war && rm -f kylin.war
cd WEB-INF/lib
#remove slf4j-api-1.7.21.jar to solve slf4j conflict
rm -f slf4j-api-1.7.21.jar
-mkdir modify_avatica_jar && mv avatica-1.10.0.jar modify_avatica_jar
+mkdir modify_avatica_jar && mv avatica-1.12.0.jar modify_avatica_jar
cd modify_avatica_jar
-#remove org/slf4j in avatica-1.10.0.jar and repackage it to solve slf4j
conflict
-unzip avatica-1.10.0.jar && rm -f avatica-1.10.0.jar
-rm -rf org/slf4j && jar -cf avatica-1.10.0.jar ./
-rm -rf `ls | egrep -v avatica-1.10.0.jar`
-mv avatica-1.10.0.jar ..
+#remove org/slf4j in avatica-1.12.0.jar and repackage it to solve slf4j
conflict
+unzip avatica-1.12.0.jar && rm -f avatica-1.12.0.jar
+rm -rf org/slf4j && jar -cf avatica-1.12.0.jar ./
+rm -rf `ls | egrep -v avatica-1.12.0.jar`
+mv avatica-1.12.0.jar ..
cd .. && rm -rf modify_avatica_jar
cd ${current_dir}/build/tomcat/webapps
#repackage kylin.war
[kylin] 02/10: Fix compile error
This is an automated email from the ASF dual-hosted git repository.
nic pushed a commit to branch master-hadoop3
in repository https://gitbox.apache.org/repos/asf/kylin.git
commit e2f22eb694e57d3b4214d8ac20dd30a830978b93
Author: nichunen
AuthorDate: Fri Jun 19 09:38:48 2020 +0800
Fix compile error
---
core-common/pom.xml | 4
kylin-it/pom.xml| 4
pom.xml | 58 +++--
3 files changed, 56 insertions(+), 10 deletions(-)
diff --git a/core-common/pom.xml b/core-common/pom.xml
index 407ab9c..6266558 100644
--- a/core-common/pom.xml
+++ b/core-common/pom.xml
@@ -114,9 +114,5 @@
curator-test
test
-
-org.apache.kylin
-kylin-shaded-guava
-
diff --git a/kylin-it/pom.xml b/kylin-it/pom.xml
index 878da15..92614fa 100644
--- a/kylin-it/pom.xml
+++ b/kylin-it/pom.xml
@@ -77,10 +77,6 @@
org.apache.kylin
-kylin-shaded-guava
-
-
-org.apache.kylin
kylin-core-storage
diff --git a/pom.xml b/pom.xml
index 0e3aec8..30a4f8f 100644
--- a/pom.xml
+++ b/pom.xml
@@ -115,7 +115,7 @@
3.1.1
3.2.2
2.5.0
-
+3.1
2.10.0
@@ -248,6 +248,11 @@
org.apache.kylin
+kylin-shaded-guava
+${project.version}
+
+
+org.apache.kylin
kylin-core-common
${project.version}
@@ -308,6 +313,11 @@
org.apache.kylin
+kylin-engine-flink
+${project.version}
+
+
+org.apache.kylin
kylin-source-hive
${project.version}
@@ -353,6 +363,11 @@
org.apache.kylin
+kylin-cube-migration
+${project.version}
+
+
+org.apache.kylin
kylin-tool
${project.version}
@@ -434,6 +449,24 @@
${project.version}
+
+org.apache.kylin
+kylin-datasource-sdk
+${project.version}
+
+
+org.apache.kylin
+kylin-datasource-sdk
+${project.version}
+framework
+
+
+org.apache.kylin
+kylin-assembly
+${project.version}
+test-jar
+
+
org.apache.hadoop
@@ -508,6 +541,11 @@
true
+org.apache.hadoop
+hadoop-distcp
+${hadoop2.version}
+
+
org.apache.zookeeper
zookeeper
${zookeeper.version}
@@ -562,6 +600,12 @@
provided
+commons-httpclient
+commons-httpclient
+${commons-httpclient.version}
+provided
+
+
org.apache.commons
commons-math3
${commons-math3.version}
@@ -1000,6 +1044,11 @@
spring-security-saml2-core
${spring.framework.security.extensions.version}
+
+org.springframework.security
+spring-security-cas
+${spring.framework.security.version}
+
org.eclipse.jetty
@@ -1015,6 +1064,11 @@
org.eclipse.jetty
+jetty-servlet
+${jetty.version}
+
+
+org.eclipse.jetty
jetty-util
${jetty.version}
test
@@ -1366,7 +1420,7 @@
net.alchim31.maven
scala-maven-plugin
-
+3.4.1
[kylin] 05/10: KYLIN-4072 find-hbase-dependency.sh don't work at CDH 6.x
This is an automated email from the ASF dual-hosted git repository.
nic pushed a commit to branch master-hadoop3
in repository https://gitbox.apache.org/repos/asf/kylin.git
commit 675273eb7c99eb488a66d0f2a1b64a11fe9ece63
Author: XiaoxiangYu
AuthorDate: Wed Jul 17 10:30:22 2019 +0800
KYLIN-4072 find-hbase-dependency.sh don't work at CDH 6.x
---
build/bin/find-hbase-dependency.sh | 14 --
build/bin/find-hive-dependency.sh | 2 +-
2 files changed, 13 insertions(+), 3 deletions(-)
diff --git a/build/bin/find-hbase-dependency.sh
b/build/bin/find-hbase-dependency.sh
index b4631ba..848ac2a 100755
--- a/build/bin/find-hbase-dependency.sh
+++ b/build/bin/find-hbase-dependency.sh
@@ -23,6 +23,9 @@ echo Retrieving hbase dependency...
hbase_classpath=`hbase classpath`
+hadoop version | head -1 | grep cdh6
+is_cdh6=$?
+
# special handling for Amazon EMR, to prevent re-init of hbase-setenv
is_aws=`uname -r | grep amzn`
if [ -n "$is_aws" ] && [ -d "/usr/lib/oozie/lib" ]; then
@@ -42,10 +45,17 @@ done
if [ -z "$hbase_common_path" ]
then
-quit "hbase-common lib not found"
+if [[ $is_cdh6 -ne 0 ]]; then
+ quit "hbase-common lib not found"
+fi
+fi
+
+if [[ $is_cdh6 -eq 0 ]]; then
+hbase_dependency=${hbase_classpath}
+else
+hbase_dependency=${hbase_common_path}
fi
-hbase_dependency=${hbase_common_path}
verbose "hbase dependency: $hbase_dependency"
export hbase_dependency
echo "export HBASE_ENV_INIT=$HBASE_ENV_INIT
diff --git a/build/bin/find-hive-dependency.sh
b/build/bin/find-hive-dependency.sh
index 22ee8f4..02da707 100755
--- a/build/bin/find-hive-dependency.sh
+++ b/build/bin/find-hive-dependency.sh
@@ -37,7 +37,7 @@ else
hive_env=`hive ${hive_conf_properties} -e set 2>&1 | grep 'env:CLASSPATH'`
fi
-if [ -z $hive_env ]
+if [ -z "${hive_env}" ]
then
hive_permission=`hive ${hive_conf_properties} -e set 2>&1 | grep 'No valid
credentials provided'`
if [ -n "$hive_permission" ]
[kylin] 04/10: KYLIN-3537
This is an automated email from the ASF dual-hosted git repository.
nic pushed a commit to branch master-hadoop3
in repository https://gitbox.apache.org/repos/asf/kylin.git
commit eb41f3cdf53055963dca8f90c847deba6a3f3ca6
Author: alexandr.sidorchuk
AuthorDate: Thu Feb 21 15:26:55 2019 +0300
KYLIN-3537
Use Spark to build Cube on Yarn faild at Setp8 on HDP3.0
add HBase dependicies to avoid java.lang.NoClassDefFoundError
Signed-off-by: shaofengshi
---
.../apache/kylin/storage/hbase/steps/HBaseSparkSteps.java | 14 ++
1 file changed, 14 insertions(+)
diff --git
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HBaseSparkSteps.java
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HBaseSparkSteps.java
index c593f2a..5074a0d 100644
---
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HBaseSparkSteps.java
+++
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HBaseSparkSteps.java
@@ -74,6 +74,20 @@ public class HBaseSparkSteps extends HBaseJobSteps {
StringUtil.appendWithSeparator(jars,
ClassUtil.findContainingJar("org.apache.hadoop.hbase.regionserver.MetricsRegionServerSourceFactoryImpl",
null));//hbase-hadoop2-compat-1.1.1.jar
+//KYLIN-3537
+StringUtil.appendWithSeparator(jars,
+
ClassUtil.findContainingJar("org.apache.hadoop.hbase.io.hfile.HFileWriterImpl",
null));//hbase-server.jar
+StringUtil.appendWithSeparator(jars,
+
ClassUtil.findContainingJar("org.apache.hbase.thirdparty.com.google.common.cache.CacheLoader",
null));//hbase-shaded-miscellaneous.jar
+StringUtil.appendWithSeparator(jars,
+
ClassUtil.findContainingJar("org.apache.hadoop.hbase.metrics.MetricRegistry",
null));//hbase-metrics-api.jar
+StringUtil.appendWithSeparator(jars,
+
ClassUtil.findContainingJar("org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl",
null));//hbase-metrics.jar
+StringUtil.appendWithSeparator(jars,
+
ClassUtil.findContainingJar("org.apache.hbase.thirdparty.com.google.protobuf.Message",
null));//hbase-shaded-protobuf.jar
+StringUtil.appendWithSeparator(jars,
+
ClassUtil.findContainingJar("org.apache.hadoop.hbase.shaded.protobuf.generated.HFileProtos",
null));//hbase-protocol-shaded.jar
+
StringUtil.appendWithSeparator(jars,
seg.getConfig().getSparkAdditionalJars());
sparkExecutable.setJars(jars.toString());
[kylin] branch master-hadoop3 created (now adef1ae)
This is an automated email from the ASF dual-hosted git repository. nic pushed a change to branch master-hadoop3 in repository https://gitbox.apache.org/repos/asf/kylin.git. at adef1ae Add keys This branch includes the following new commits: new f092b20 KYLIN-2565 upgrade to hadoop 3.0 hbase 2.0, pass UT new e2f22eb Fix compile error new bc02c20 Fix dependency issues for branch of hadoop3 new eb41f3c KYLIN-3537 new 675273e KYLIN-4072 find-hbase-dependency.sh don't work at CDH 6.x new 8cfe05d remove htable toUpperCase, for HBase tableName is case sensitive new d30a34a KYLIN-4086 Upgrade avatica to 1.12 new b571009 Package kafka and curator new bef71cf KYLIN-3973: InvalidProtocolBufferException: Protocol message was too large. new adef1ae Add keys The 10 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[kylin] 10/10: Add keys
This is an automated email from the ASF dual-hosted git repository. nic pushed a commit to branch master-hadoop3 in repository https://gitbox.apache.org/repos/asf/kylin.git commit adef1ae806b9517f3b866b9b3d38f1b5b9375131 Author: nichunen AuthorDate: Mon Feb 10 13:11:15 2020 + Add keys --- KEYS | 57 ++ storage-hbase/pom.xml | 4 ++ .../kylin/storage/hbase/steps/CubeHFileJob.java| 1 - 3 files changed, 61 insertions(+), 1 deletion(-) diff --git a/KEYS b/KEYS index 8c0fa0b..ea1ffd5 100644 --- a/KEYS +++ b/KEYS @@ -478,3 +478,60 @@ mUdVyu44Yf7c3XnrZL0F7mwzFmpgeTWGOTzqfO5YSeHr5x8gizsBVADaoB1uPjPz kzyYjjy3XzKvLlVIHuqksvDyibrlU1StUmNS64lfugMY1RfLERUpRdV3 =DvSV -END PGP PUBLIC KEY BLOCK- +pub 4096R/479A9809 2020-02-09 +uid Chunen Ni +sig 3479A9809 2020-02-09 Chunen Ni +sub 4096R/FB0A0AE0 2020-02-09 +sig 479A9809 2020-02-09 Chunen Ni + +-BEGIN PGP PUBLIC KEY BLOCK- +Version: GnuPG v2.0.14 (GNU/Linux) + +mQINBF4/tlsBEAC3fFwqUc93Ce4cPI9qYOYbyooRlCnMce4maU855/1b2Q3LNmQY +lYbkahVpXcOCB4UUg+O5m1EZUqgKIYwf+Bet+va8efgf+I6xwNcWai1p9He/A52B +ZJIcjHpHJr2vRU5sCTM0Ksxv5IZMrBaAcFQ10KQft32ic+iAGsvRHrMHMkLYgjc4 +oc97w6cTywMj32ZX5aylVzezJWCSvEm8Ur+hNvMA14shpenbW+1xfXaiz0IAb0yc +KU4Idk/H1tuLXbULz04t6Drwe4btOYMcuqXhSzzCZg2cVCGXNbnoacsTCipTEAdb +rxEjCZ77a/lK5BtjD6bKEYgZaHvUyw/HeDFcKuk5GiPwnfuRcuyrv7nIS9oXBFMz ++ZqyD5adpDFdNnfRE9X5Qvtqczv4YcOM3O6jbXF/UoupgJexE9jrUAlXPJ8YDPXJ +4C9o67FYx1uKWHt6OQJQK3uWieZ9TV/seFZwm4Nbx4W13DifnL0AVW2sE/eNsDzN +IjzSZqbpbvJyZBPNWJaCi0RitMfyjdQnPyfEyqUnbd8Qt0XNq9zyLA8tOm8rTj3T +Re0AfjnhbpKRteBKgcv+HrGrLnGqy1P37RD0wb/3g0WqQHURgXjZXXcdH+eRcJ+h +Npot/5C1ipOnXCZ4vJ7MFjOhYJ2LjABwI1LKRYlcSktwwyHte6fyG+qykwARAQAB +tBpDaHVuZW4gTmkgPG5pY0BhcGFjaGUub3JnPokCOAQTAQIAIgUCXj+2WwIbAwYL +CQgHAwIGFQgCCQoLBBYCAwECHgECF4AACgkQyS5HqEeamAkMxg//Tb119mu0jvKx +pAlT2HwozQZV+Gabs23qyVZktuUs2mYUoAtZrRiJ9jo7QCWgSc3JcrAzdPzjc6nf +THpO1zz+AftzESdlve9yofG78zRbvoYYrf+hgjuFrBDK/A/pPiYHyt5nsZklbCwZ +AZ8DPY8YVbJXaH4Wci5smEtHKIViXN60y4Yi9qlXaFkEyM3eRK2lPPjiIVtsf+H8 +bWkObtQALk0rnlNEZCch5ZT8dNpIuaUrzi9kOcN7N2xQtlFluerdyKniir4Y1Gij +QPyn1VxFyQ9eXPjREyjPwybancbqtINXtrJuhjA8IwPtg2fnKiqxofUUcwV3uPEa +bximeiIG434ez5ArMfZhB06NxCTPv4C1kzm+GjyocUCzKg+rRjOuuQLe/PlS4L+W +MsBZfSoBgH3Q9OgiRXXbLfp3NN4bo+oIBAxfZYS2+8tR7ozQO/h4N7Dakbw9M7Kw +OCCIEO1LvB2qjQ0e7RMrHsTGMctxwWW8ATb5OJmLDfs675IeI/oAylgTcfPZOWwo +hm8Wj1H/bA+CkK6YWZlrWwqqN/lVn+83rlxeZ/TD+K8IVaHQ1f+nobLoZzeScv1c +wrT1nfqKqO5gvr7O2bBNvMw1FSOX7DTn0ggGNQy/sJqf7gevcq8PtL9QiOl9gSM3 +046tN9Ij0l2PtV4l93tjcEWR6qrljiS5Ag0EXj+2WwEQAMrwsuaaoUA+KaMl2LQS +TKXICTc2+8nS2MQrZ/Y4GJQed+RQM9wmf6ozgGg/cVooAflcTwGuBzIutCLcXHb4 +bEzQ7ylvm3BoazjrlA0+rDFWKvLADDjzg56h0DndkTPRE0WR6rEs1VACCpWmTdc6 +wFkdT3QoRGr+Db1djZVTroUWpPQCc7BYPFofJd7hZQn8AClMsH8SNQzGslhY7PhK +pX8pIaeIm/UyM8TpmSCGa8K6x0WoSARmv4HwkuKPafu6pEbZo8XrHytVnrSCUsrG +Y/W+xIoj9HDLREa37T/8FEIVmV19hfvSvnck2I5Pq7/BMICJE/wwIoEhehy9lG7+ +SBpcQ9dJ4OQ9ttkv7+6LoDjGOEU3pQM3uIuxlLT1zvRvb9xCic2oqtd61apzR4so +4iwu0ozPClD7GJhdAYXGYdOuT+Y3FvEOZAtIhvgWhTpom2R8gvf6U70PW9clrk6K +G9/K21WYX7OFe0c+6rhVD0SOd7mR5QfDpXP9fxLBe8k7WjUp+FImxmb7zbVhsIAs +DXqg7I/RZiAxwYXJu0aLW3ey5mpD64Y0BMsjFSxuygt9URdQ0fCfIVkcS1cy3FfU +UI0wEvSZAcUlwH14dUGzehbMEfnF2nOvecFk5BH5BFTQydX26JkAmsFgGTKGiA3E +HDLR+V2yTXZ6IowJi87G4qrxABEBAAGJAh8EGAECAAkFAl4/tlsCGwwACgkQyS5H +qEeamAlZew//Z1osjgOo6ysONEQyzdiMP4UTVn6Un0q1X7FOQq03Ods3rZuTn7SB +DF/qnVn0iPuk5fmplq5SWP0YSBRl+ooVyiY7i5alK1mCVNUQln8bebNJdR4q2p5v +zEWVN9T2qzpnw0AdG1wlAURC39+2Zt1s/uDyHvkoBQ7K7aqjdNvdsZf1NFRdHcYz +tNi2dOwDdTftS8pwER3Nf8rytpuqfwyBRnRnv0Tuc0uCkPAgCvTNjofY8QM/BluF +oERlWTr9MLltxlm9EztPBiEdoezpQHR3Mla4rH1R98iX8WpoXJrhUNwcF513cezU +cdKQVWDlfA9ZuqvS9apzS+lyX7E/lrO8bhlDoJyclE14kHHLQv3vK0UwSQt/q9tF +DxU9A8tXXx3YnRFVffifanxbpLGwWzViD0vHNbb4yJGaAJGlrGBSx44+xGn1jd9K +7v2DxUj7dDIlC422feV90AtUziaOuisCXbEcmy6+cmjF17qJFFLqZeZvUKumYylq +dQXQoPX2Wn4+S2dnD/BX1aE1267UGdK4Dx07aDHqdjFeFWDWPPrYt0/Gm0yF913U +I81JaVK7up++EnPj2G9yx4PrZvODdpFgHkGFXLRgPPkO4aEUnEgX0lMRS/Je/9rr +1tJNGDK1fK5w+i57Yxo8d1PnVWBeZ1uIVoqhfe7Cx08f8PcPijKxKKY= +=IfNu +-END PGP PUBLIC KEY BLOCK- diff --git a/storage-hbase/pom.xml b/storage-hbase/pom.xml index fab0108..ab073a0 100644 --- a/storage-hbase/pom.xml +++ b/storage-hbase/pom.xml @@ -46,6 +46,10 @@ org.apache.kylin kylin-cache + +org.apache.kylin +kylin-engine-mr + org.apache.kylin diff --git a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/CubeHFileJob.java b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/CubeHFileJob.java index 32f0ce8..28752ca 100644 --- a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/CubeHFileJob.java +++ b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/CubeHFileJob.java @@ -22,7 +22,6 @@ import static org.apache.hadoop.hbase.HBaseConfiguration.merge; import
[kylin] 06/10: remove htable toUpperCase, for HBase tableName is case sensitive
This is an automated email from the ASF dual-hosted git repository.
nic pushed a commit to branch master-hadoop3
in repository https://gitbox.apache.org/repos/asf/kylin.git
commit 8cfe05d141288c2d75815348de5df2c652dcc41b
Author: liuzx32
AuthorDate: Wed Jun 26 16:27:26 2019 +0800
remove htable toUpperCase, for HBase tableName is case sensitive
---
.../main/java/org/apache/kylin/storage/hbase/steps/CubeHFileJob.java| 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/CubeHFileJob.java
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/CubeHFileJob.java
index 86189e5..32f0ce8 100644
---
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/CubeHFileJob.java
+++
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/CubeHFileJob.java
@@ -105,7 +105,7 @@ public class CubeHFileJob extends AbstractHadoopJob {
Configuration hbaseConf = HBaseConfiguration.create(getConf());
-String hTableName =
getOptionValue(OPTION_HTABLE_NAME).toUpperCase(Locale.ROOT);
+String hTableName = getOptionValue(OPTION_HTABLE_NAME);
connection = ConnectionFactory.createConnection(hbaseConf);
Table table = connection.getTable(TableName.valueOf(hTableName));
RegionLocator regionLocator =
connection.getRegionLocator(TableName.valueOf(hTableName));
[kylin] branch v3.1.0-release created (now 77cbfc4)
This is an automated email from the ASF dual-hosted git repository. nic pushed a change to branch v3.1.0-release in repository https://gitbox.apache.org/repos/asf/kylin.git. at 77cbfc4 KYLIN-3832 Disable all convert and quote behavior for postgresql No new revisions were added by this update.
[kylin] branch master updated: KYLIN-4480 runtime non-equi join
This is an automated email from the ASF dual-hosted git repository.
nic pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kylin.git
The following commit(s) were added to refs/heads/master by this push:
new 1fce51e KYLIN-4480 runtime non-equi join
1fce51e is described below
commit 1fce51edf5cfb687b77b2a3f41ca3ce382efd923
Author: Xian Li
AuthorDate: Wed Apr 29 15:40:54 2020 +0800
KYLIN-4480 runtime non-equi join
---
.../org/apache/kylin/query/ITKylinQueryTest.java | 5 +
.../query/sql_non_equal_join/query_00.sql | 11 +
.../query/sql_non_equal_join/query_01.sql | 12 ++
.../query/sql_non_equal_join/query_02.sql | 17 ++
pom.xml| 2 +-
.../apache/kylin/query/optrule/OLAPJoinRule.java | 10 +-
.../kylin/query/relnode/OLAPNonEquiJoinRel.java| 224 +
7 files changed, 277 insertions(+), 4 deletions(-)
diff --git
a/kylin-it/src/test/java/org/apache/kylin/query/ITKylinQueryTest.java
b/kylin-it/src/test/java/org/apache/kylin/query/ITKylinQueryTest.java
index 07edde2..7ec3cff 100644
--- a/kylin-it/src/test/java/org/apache/kylin/query/ITKylinQueryTest.java
+++ b/kylin-it/src/test/java/org/apache/kylin/query/ITKylinQueryTest.java
@@ -240,6 +240,11 @@ public class ITKylinQueryTest extends KylinTestBase {
}
@Test
+public void testNonEqualJoin() throws Exception {
+execAndCompQuery(getQueryFolderPrefix() +
"src/test/resources/query/sql_non_equal_join", null, true);
+}
+
+@Test
public void testUnionQuery() throws Exception {
execAndCompQuery(getQueryFolderPrefix() +
"src/test/resources/query/sql_union", null, true);
}
diff --git a/kylin-it/src/test/resources/query/sql_non_equal_join/query_00.sql
b/kylin-it/src/test/resources/query/sql_non_equal_join/query_00.sql
new file mode 100644
index 000..45b27e2
--- /dev/null
+++ b/kylin-it/src/test/resources/query/sql_non_equal_join/query_00.sql
@@ -0,0 +1,11 @@
+-- non-equal join
+SELECT * FROM
+(
+SELECT CAL_DT, ORDER_ID FROM TEST_KYLIN_FACT GROUP BY CAL_DT, ORDER_ID
+) FACT
+LEFT JOIN
+(
+SELECT CAL_DT as DT_CAL_DT, WEEK_BEG_DT FROM EDW.TEST_CAL_DT
+) DT
+ON FACT.CAL_DT = DT.DT_CAL_DT AND WEEK_BEG_DT = DATE'2013-03-24'
+
diff --git a/kylin-it/src/test/resources/query/sql_non_equal_join/query_01.sql
b/kylin-it/src/test/resources/query/sql_non_equal_join/query_01.sql
new file mode 100644
index 000..c843695
--- /dev/null
+++ b/kylin-it/src/test/resources/query/sql_non_equal_join/query_01.sql
@@ -0,0 +1,12 @@
+-- non equal join with equal join
+SELECT ITEM_CNT, FACT.CAL_DT, ORDER_ID, WEEK_BEG_DT FROM
+(
+select sum(ITEM_COUNT) as ITEM_CNT, CAL_DT, TEST_ORDER.ORDER_ID
+FROM TEST_KYLIN_FACT as TEST_KYLIN_FACT
+INNER JOIN TEST_ORDER as TEST_ORDER
+ON TEST_KYLIN_FACT.ORDER_ID = TEST_ORDER.ORDER_ID
+GROUP BY CAL_DT, TEST_ORDER.ORDER_ID
+) FACT
+LEFT JOIN EDW.TEST_CAL_DT DT
+ON FACT.CAL_DT = DT.CAL_DT AND WEEK_BEG_DT = DATE'2013-03-24'
+
diff --git a/kylin-it/src/test/resources/query/sql_non_equal_join/query_02.sql
b/kylin-it/src/test/resources/query/sql_non_equal_join/query_02.sql
new file mode 100644
index 000..035459f
--- /dev/null
+++ b/kylin-it/src/test/resources/query/sql_non_equal_join/query_02.sql
@@ -0,0 +1,17 @@
+-- non equal join with complex join conditions
+SELECT ITEM_CNT, FACT.CAL_DT, ORDER_ID, WEEK_BEG_DT FROM
+(
+SELECT SUM(ITEM_COUNT) AS ITEM_CNT, CAL_DT, TEST_ORDER.ORDER_ID
+FROM TEST_KYLIN_FACT AS TEST_KYLIN_FACT
+INNER JOIN TEST_ORDER AS TEST_ORDER
+ON TEST_KYLIN_FACT.ORDER_ID = TEST_ORDER.ORDER_ID
+GROUP BY CAL_DT, TEST_ORDER.ORDER_ID
+) FACT
+LEFT JOIN EDW.TEST_CAL_DT DT
+ON FACT.CAL_DT = DT.CAL_DT
+AND WEEK_BEG_DT = DATE'2013-03-24'
+OR (WEEK_BEG_DT < DATE'2013-03-24'
+AND (CASE WHEN ITEM_CNT > 100 THEN ORDER_ID > 1 ELSE DT.CAL_DT <
DATE'2013-04-24' END)
+AND SUBSTRING(CAST(ORDER_ID AS VARCHAR), 1, 2) <> '14')
+
+
diff --git a/pom.xml b/pom.xml
index bd916b0..94406e5 100644
--- a/pom.xml
+++ b/pom.xml
@@ -95,7 +95,7 @@
0.9.10
-1.16.0-kylin-r3
+1.16.0-kylin-r4
1.12.0
diff --git
a/query/src/main/java/org/apache/kylin/query/optrule/OLAPJoinRule.java
b/query/src/main/java/org/apache/kylin/query/optrule/OLAPJoinRule.java
index d87a0c4..ff6351b 100644
--- a/query/src/main/java/org/apache/kylin/query/optrule/OLAPJoinRule.java
+++ b/query/src/main/java/org/apache/kylin/query/optrule/OLAPJoinRule.java
@@ -29,6 +29,7 @@ import org.apache.calcite.rel.core.JoinRelType;
import org.apache.calcite.rel.logical.LogicalJoin;
import org.apache.kylin.query.relnode.OLAPFilterRel;
import org.apache.kylin.query.relnode.OLAPJoinRel;
+import org.apache.kylin.query.relnode.OLAPNonEquiJoinRel;
import org.apache.kylin.query.relnode.OLAPRel;
/**
@@ -53,9 +54,12 @@ public class OLAPJoinRule extends ConverterRule {
final JoinInfo info = Join
[kylin] branch master updated: KYLIN-4539 Spark build failed due to com.codahale.metrics.json.MetricsModule loaded from kylin job jar
This is an automated email from the ASF dual-hosted git repository.
nic pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kylin.git
The following commit(s) were added to refs/heads/master by this push:
new c844743 KYLIN-4539 Spark build failed due to
com.codahale.metrics.json.MetricsModule loaded from kylin job jar
c844743 is described below
commit c844743acbf6ffbab46937bded3ac4e0fafa5130
Author: Guangxu Cheng
AuthorDate: Thu May 28 11:36:21 2020 +0800
KYLIN-4539 Spark build failed due to
com.codahale.metrics.json.MetricsModule loaded from kylin job jar
---
assembly/pom.xml | 4
1 file changed, 4 insertions(+)
diff --git a/assembly/pom.xml b/assembly/pom.xml
index 52d4409..4d58c91 100644
--- a/assembly/pom.xml
+++ b/assembly/pom.xml
@@ -232,6 +232,10 @@
org.apache.commons.jocl
${shadeBase}.org.apache.commons.jocl
+
+com.codahale.metrics
+
${shadeBase}.com.codahale.metrics
+
[kylin] branch document created (now 758a8f7)
This is an automated email from the ASF dual-hosted git repository. nic pushed a change to branch document in repository https://gitbox.apache.org/repos/asf/kylin.git. at 758a8f7 Update website RESTful API page No new revisions were added by this update.
[kylin] branch master updated: KYLIN-4537 Give a friendly tips to the user when getting task list fails
This is an automated email from the ASF dual-hosted git repository.
nic pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kylin.git
The following commit(s) were added to refs/heads/master by this push:
new ed7658d KYLIN-4537 Give a friendly tips to the user when getting task
list fails
ed7658d is described below
commit ed7658d87104c9ab365062cbfed239341d33aa2b
Author: Guangxu Cheng
AuthorDate: Wed May 27 21:57:41 2020 +0800
KYLIN-4537 Give a friendly tips to the user when getting task list fails
---
webapp/app/js/model/jobListModel.js | 18 ++
1 file changed, 14 insertions(+), 4 deletions(-)
diff --git a/webapp/app/js/model/jobListModel.js
b/webapp/app/js/model/jobListModel.js
index c13ae66..0b67161 100755
--- a/webapp/app/js/model/jobListModel.js
+++ b/webapp/app/js/model/jobListModel.js
@@ -56,8 +56,13 @@ KylinApp.service('JobList',function(JobService, $q,
kylinConfig, jobConfig){
_this.jobs[id].dropped = false;
});
defer.resolve(jobs.length);
- },function(){
-defer.reject("failed to load jobs");
+},function(e){
+ var msg = 'failed to load jobs';
+ if (e.data && e.data.exception) {
+var message = e.data.exception;
+msg = !!(message) ? message : msg;
+ }
+ defer.reject(msg);
});
return defer.promise;
};
@@ -73,8 +78,13 @@ KylinApp.service('JobList',function(JobService, $q,
kylinConfig, jobConfig){
}
});
defer.resolve(jobsOverview);
- },function(){
-defer.reject("failed to load job overview");
+ },function(e){
+var msg = 'failed to load job overview';
+if (e.data && e.data.exception) {
+ var message = e.data.exception;
+ msg = !!(message) ? message : msg;
+}
+defer.reject(msg);
});
return defer.promise;
};
[kylin] 01/04: KYLIN-4505 Change guava cache to spring cache for user authentication
This is an automated email from the ASF dual-hosted git repository.
nic pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kylin.git
commit 84064d32a64c9f942e6c4cc0e1fac243a5ee2b89
Author: Zhong, Yanghong
AuthorDate: Wed May 20 16:41:45 2020 +0800
KYLIN-4505 Change guava cache to spring cache for user authentication
---
.../kylin/cache/cachemanager/CacheConstants.java | 1 +
.../cache/cachemanager/MemcachedCacheManager.java | 3 ++
.../rest/security/KylinAuthenticationProvider.java | 44 +++---
.../kylin/rest/service/KylinUserService.java | 14 ---
.../org/apache/kylin/rest/service/UserService.java | 10 ++---
server/src/main/resources/ehcache-test.xml | 7
server/src/main/resources/ehcache.xml | 7
7 files changed, 42 insertions(+), 44 deletions(-)
diff --git
a/cache/src/main/java/org/apache/kylin/cache/cachemanager/CacheConstants.java
b/cache/src/main/java/org/apache/kylin/cache/cachemanager/CacheConstants.java
index 07b15a5..12d0c7a 100644
---
a/cache/src/main/java/org/apache/kylin/cache/cachemanager/CacheConstants.java
+++
b/cache/src/main/java/org/apache/kylin/cache/cachemanager/CacheConstants.java
@@ -20,4 +20,5 @@ package org.apache.kylin.cache.cachemanager;
public class CacheConstants {
public static final String QUERY_CACHE = "StorageCache";
+public static final String USER_CACHE = "UserCache";
}
\ No newline at end of file
diff --git
a/cache/src/main/java/org/apache/kylin/cache/cachemanager/MemcachedCacheManager.java
b/cache/src/main/java/org/apache/kylin/cache/cachemanager/MemcachedCacheManager.java
index f5acc6f..2ae49c8 100644
---
a/cache/src/main/java/org/apache/kylin/cache/cachemanager/MemcachedCacheManager.java
+++
b/cache/src/main/java/org/apache/kylin/cache/cachemanager/MemcachedCacheManager.java
@@ -59,8 +59,11 @@ public class MemcachedCacheManager extends
AbstractCacheManager {
protected Collection loadCaches() {
Cache successCache = new MemCachedCacheAdaptor(
new
MemcachedChunkingCache(MemcachedCache.create(memcachedCacheConfig,
CacheConstants.QUERY_CACHE)));
+Cache userCache = new MemCachedCacheAdaptor(
+new MemcachedCache(MemcachedCache.create(memcachedCacheConfig,
CacheConstants.USER_CACHE, 86400)));
addCache(successCache);
+addCache(userCache);
Collection names = getCacheNames();
Collection caches = Lists.newArrayList();
diff --git
a/server-base/src/main/java/org/apache/kylin/rest/security/KylinAuthenticationProvider.java
b/server-base/src/main/java/org/apache/kylin/rest/security/KylinAuthenticationProvider.java
index 7ea3957..590c15a 100644
---
a/server-base/src/main/java/org/apache/kylin/rest/security/KylinAuthenticationProvider.java
+++
b/server-base/src/main/java/org/apache/kylin/rest/security/KylinAuthenticationProvider.java
@@ -18,16 +18,22 @@
package org.apache.kylin.rest.security;
+import static org.apache.kylin.cache.cachemanager.CacheConstants.USER_CACHE;
+
import java.nio.charset.Charset;
import java.util.Arrays;
-import java.util.concurrent.TimeUnit;
+
+import javax.annotation.PostConstruct;
import org.apache.kylin.common.KylinConfig;
import org.apache.kylin.rest.service.UserService;
+
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Qualifier;
+import org.springframework.cache.Cache;
+import org.springframework.cache.CacheManager;
import org.springframework.security.authentication.AuthenticationProvider;
import org.springframework.security.core.Authentication;
import org.springframework.security.core.AuthenticationException;
@@ -36,9 +42,7 @@ import
org.springframework.security.core.userdetails.UserDetails;
import org.springframework.security.core.userdetails.UsernameNotFoundException;
import org.springframework.util.Assert;
-import org.apache.kylin.shaded.com.google.common.cache.CacheBuilder;
-import org.apache.kylin.shaded.com.google.common.cache.RemovalListener;
-import org.apache.kylin.shaded.com.google.common.cache.RemovalNotification;
+import org.apache.kylin.shaded.com.google.common.base.Preconditions;
import org.apache.kylin.shaded.com.google.common.hash.HashFunction;
import org.apache.kylin.shaded.com.google.common.hash.Hashing;
@@ -49,21 +53,13 @@ public class KylinAuthenticationProvider implements
AuthenticationProvider {
private static final Logger logger =
LoggerFactory.getLogger(KylinAuthenticationProvider.class);
-private final static
org.apache.kylin.shaded.com.google.common.cache.Cache
userCache = CacheBuilder.newBuilder()
-
.maximumSize(KylinConfig.getInstanceFromEnv().getServerUserCacheMaxEntries())
-
.expireAfterWrite(KylinConfig.getInstanceFromEnv().getServerUserCacheExpireSeconds(),
TimeUnit.SECONDS)
-
[kylin] 04/04: KYLIN-4507 Override getSocketAddress() in TCPMemcachedNodeImpl to auto detect memcached instance ip change
This is an automated email from the ASF dual-hosted git repository.
nic pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kylin.git
commit a585119a59f4520114dd2d6fb357aeab41b32977
Author: Zhong, Yanghong
AuthorDate: Wed May 20 17:24:26 2020 +0800
KYLIN-4507 Override getSocketAddress() in TCPMemcachedNodeImpl to auto
detect memcached instance ip change
---
.../net/spy/memcached/protocol/TCPMemcachedNodeImpl.java | 11 ++-
.../cachemanager/RemoteLocalFailOverCacheManager.java | 5 +
.../cachemanager/RemoteLocalFailOverCacheManagerTest.java | 14 ++
3 files changed, 29 insertions(+), 1 deletion(-)
diff --git
a/cache/src/main/java/net/spy/memcached/protocol/TCPMemcachedNodeImpl.java
b/cache/src/main/java/net/spy/memcached/protocol/TCPMemcachedNodeImpl.java
index 22dd730..f7da57e 100644
--- a/cache/src/main/java/net/spy/memcached/protocol/TCPMemcachedNodeImpl.java
+++ b/cache/src/main/java/net/spy/memcached/protocol/TCPMemcachedNodeImpl.java
@@ -19,10 +19,12 @@
package net.spy.memcached.protocol;
import java.io.IOException;
+import java.net.InetSocketAddress;
import java.net.SocketAddress;
import java.nio.ByteBuffer;
import java.nio.channels.SelectionKey;
import java.nio.channels.SocketChannel;
+import java.nio.channels.UnsupportedAddressTypeException;
import java.util.ArrayList;
import java.util.Collection;
import java.util.concurrent.BlockingQueue;
@@ -42,6 +44,9 @@ import net.spy.memcached.protocol.binary.TapAckOperationImpl;
/**
* Represents a node with the memcached cluster, along with buffering and
* operation queues.
+ *
+ * This is a modified version of the
net.spy.memcached.protocol.TCPMemcachedNodeImpl
+ * Override the final method getSocketAddress() to refresh SocketAddress to
achieve same hostname with ip changing
*/
public abstract class TCPMemcachedNodeImpl extends SpyObject implements
MemcachedNode {
@@ -415,7 +420,11 @@ public abstract class TCPMemcachedNodeImpl extends
SpyObject implements Memcache
* @see net.spy.memcached.MemcachedNode#getSocketAddress()
*/
public final SocketAddress getSocketAddress() {
-return socketAddress;
+if (!(socketAddress instanceof InetSocketAddress)) {
+throw new UnsupportedAddressTypeException();
+}
+InetSocketAddress inetSocketAddress = (InetSocketAddress)
socketAddress;
+return new InetSocketAddress(inetSocketAddress.getHostName(),
inetSocketAddress.getPort());
}
/*
diff --git
a/cache/src/main/java/org/apache/kylin/cache/cachemanager/RemoteLocalFailOverCacheManager.java
b/cache/src/main/java/org/apache/kylin/cache/cachemanager/RemoteLocalFailOverCacheManager.java
index 22517f4..aae0d7c 100644
---
a/cache/src/main/java/org/apache/kylin/cache/cachemanager/RemoteLocalFailOverCacheManager.java
+++
b/cache/src/main/java/org/apache/kylin/cache/cachemanager/RemoteLocalFailOverCacheManager.java
@@ -68,4 +68,9 @@ public class RemoteLocalFailOverCacheManager extends
AbstractCacheManager {
void enableRemoteCacheManager() {
remoteCacheManager.setClusterHealth(true);
}
+
+@VisibleForTesting
+MemcachedCacheManager getRemoteCacheManager() {
+return remoteCacheManager;
+}
}
\ No newline at end of file
diff --git
a/cache/src/test/java/org/apache/kylin/cache/cachemanager/RemoteLocalFailOverCacheManagerTest.java
b/cache/src/test/java/org/apache/kylin/cache/cachemanager/RemoteLocalFailOverCacheManagerTest.java
index 243e386..c45dd6f 100644
---
a/cache/src/test/java/org/apache/kylin/cache/cachemanager/RemoteLocalFailOverCacheManagerTest.java
+++
b/cache/src/test/java/org/apache/kylin/cache/cachemanager/RemoteLocalFailOverCacheManagerTest.java
@@ -32,6 +32,8 @@ import org.springframework.cache.ehcache.EhCacheCache;
import org.springframework.test.context.ActiveProfiles;
import org.springframework.test.context.ContextConfiguration;
import org.springframework.test.context.junit4.SpringJUnit4ClassRunner;
+//
+//import net.spy.memcached.MemcachedClientIF;
@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration(locations = { "classpath:cacheContext.xml" })
@@ -58,5 +60,17 @@ public class RemoteLocalFailOverCacheManagerTest {
cacheManager.enableRemoteCacheManager();
Assert.assertTrue("Memcached enabled",
cacheManager.getCache(QUERY_CACHE) instanceof
MemcachedCacheManager.MemCachedCacheAdaptor);
+//
+//MemcachedCacheManager remoteCacheManager =
cacheManager.getRemoteCacheManager();
+//for (int i = 0; i < 1000; i++) {
+//MemcachedClientIF client = (MemcachedClientIF)
remoteCacheManager.getCache(QUERY_CACHE).getNativeCache();
+//System.out.println(i + " available servers: " +
client.getAvailableServers() + "; unavailable servers: "
+//+ client.getUnavailableServers());
+//
[kylin] 02/04: KYLIN-4506 Remove unresolved memcached servers before initialize MemcachedClient
This is an automated email from the ASF dual-hosted git repository.
nic pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kylin.git
commit f815c4ff1c4aa86692adbbbdfb93e4979f9cd49a
Author: Zhong, Yanghong
AuthorDate: Wed May 20 17:11:20 2020 +0800
KYLIN-4506 Remove unresolved memcached servers before initialize
MemcachedClient
---
.../org/apache/kylin/cache/memcached/MemcachedCache.java | 16 +++-
.../apache/kylin/cache/memcached/MemcachedCacheTest.java | 9 +
2 files changed, 24 insertions(+), 1 deletion(-)
diff --git
a/cache/src/main/java/org/apache/kylin/cache/memcached/MemcachedCache.java
b/cache/src/main/java/org/apache/kylin/cache/memcached/MemcachedCache.java
index e7ff75f..784665e 100644
--- a/cache/src/main/java/org/apache/kylin/cache/memcached/MemcachedCache.java
+++ b/cache/src/main/java/org/apache/kylin/cache/memcached/MemcachedCache.java
@@ -20,8 +20,11 @@ package org.apache.kylin.cache.memcached;
import java.io.IOException;
import java.io.Serializable;
+import java.net.InetSocketAddress;
import java.nio.ByteBuffer;
import java.util.Arrays;
+import java.util.Iterator;
+import java.util.List;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.Future;
import java.util.concurrent.TimeUnit;
@@ -136,13 +139,24 @@ public class MemcachedCache {
.setOpQueueMaxBlockTime(config.getTimeout()).setOpTimeout(config.getTimeout())
.setReadBufferSize(config.getReadBufferSize()).setOpQueueFactory(opQueueFactory).build();
return new MemcachedCache(new MemcachedClient(new
MemcachedConnectionFactory(connectionFactory),
-AddrUtil.getAddresses(hostsStr)), config, memcachedPrefix,
timeToLive);
+getResolvedAddrList(hostsStr)), config, memcachedPrefix,
timeToLive);
} catch (IOException e) {
logger.error("Unable to create MemcachedCache instance.", e);
throw Throwables.propagate(e);
}
}
+public static List getResolvedAddrList(String hostsStr)
{
+List addrs = AddrUtil.getAddresses(hostsStr);
+Iterator addrIterator = addrs.iterator();
+while (addrIterator.hasNext()) {
+if (addrIterator.next().isUnresolved()) {
+addrIterator.remove();
+}
+}
+return addrs;
+}
+
public String getName() {
return memcachedPrefix;
}
diff --git
a/cache/src/test/java/org/apache/kylin/cache/memcached/MemcachedCacheTest.java
b/cache/src/test/java/org/apache/kylin/cache/memcached/MemcachedCacheTest.java
index 4fcbc5e..a4f8fc0 100644
---
a/cache/src/test/java/org/apache/kylin/cache/memcached/MemcachedCacheTest.java
+++
b/cache/src/test/java/org/apache/kylin/cache/memcached/MemcachedCacheTest.java
@@ -21,6 +21,8 @@ package org.apache.kylin.cache.memcached;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.when;
+import java.net.InetSocketAddress;
+import java.util.List;
import java.util.Map;
import java.util.concurrent.TimeUnit;
@@ -81,4 +83,11 @@ public class MemcachedCacheTest extends
LocalFileMetadataTestCase {
Assert.assertEquals("The value should not change",
keyValueMap.get(key), memCachedAdaptor.get(key).get());
}
}
+
+@Test
+public void testGetResolvedAddrList() {
+String hostsStr = "localhost:11211,fafddafaf:11211,fadfafaerqr:11211";
+List addrList =
MemcachedCache.getResolvedAddrList(hostsStr);
+Assert.assertEquals(1, addrList.size());
+}
}
\ No newline at end of file
[kylin] 03/04: KYLIN-4507 Add hack file TCPMemcachedNodeImpl.java
This is an automated email from the ASF dual-hosted git repository.
nic pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kylin.git
commit 03e67bc1cf8fbb63ad59e868ae0287679bde4e5d
Author: Zhong, Yanghong
AuthorDate: Wed May 20 17:15:39 2020 +0800
KYLIN-4507 Add hack file TCPMemcachedNodeImpl.java
---
.../memcached/protocol/TCPMemcachedNodeImpl.java | 641 +
1 file changed, 641 insertions(+)
diff --git
a/cache/src/main/java/net/spy/memcached/protocol/TCPMemcachedNodeImpl.java
b/cache/src/main/java/net/spy/memcached/protocol/TCPMemcachedNodeImpl.java
new file mode 100644
index 000..22dd730
--- /dev/null
+++ b/cache/src/main/java/net/spy/memcached/protocol/TCPMemcachedNodeImpl.java
@@ -0,0 +1,641 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package net.spy.memcached.protocol;
+
+import java.io.IOException;
+import java.net.SocketAddress;
+import java.nio.ByteBuffer;
+import java.nio.channels.SelectionKey;
+import java.nio.channels.SocketChannel;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import net.spy.memcached.ConnectionFactory;
+import net.spy.memcached.FailureMode;
+import net.spy.memcached.MemcachedConnection;
+import net.spy.memcached.MemcachedNode;
+import net.spy.memcached.compat.SpyObject;
+import net.spy.memcached.ops.Operation;
+import net.spy.memcached.ops.OperationState;
+import net.spy.memcached.protocol.binary.TapAckOperationImpl;
+
+/**
+ * Represents a node with the memcached cluster, along with buffering and
+ * operation queues.
+ */
+public abstract class TCPMemcachedNodeImpl extends SpyObject implements
MemcachedNode {
+
+private final SocketAddress socketAddress;
+private final ByteBuffer rbuf;
+private final ByteBuffer wbuf;
+protected final BlockingQueue writeQ;
+private final BlockingQueue readQ;
+private final BlockingQueue inputQueue;
+private final long opQueueMaxBlockTime;
+private final long authWaitTime;
+private final ConnectionFactory connectionFactory;
+private AtomicInteger reconnectAttempt = new AtomicInteger(1);
+private SocketChannel channel;
+private int toWrite = 0;
+protected Operation optimizedOp = null;
+private volatile SelectionKey sk = null;
+private boolean shouldAuth = false;
+private CountDownLatch authLatch;
+private ArrayList reconnectBlocked;
+private long defaultOpTimeout;
+private volatile long lastReadTimestamp = System.nanoTime();
+private MemcachedConnection connection;
+
+// operation Future.get timeout counter
+private final AtomicInteger continuousTimeout = new AtomicInteger(0);
+
+public TCPMemcachedNodeImpl(SocketAddress sa, SocketChannel c, int
bufSize, BlockingQueue rq,
+BlockingQueue wq, BlockingQueue iq, long
opQueueMaxBlockTime, boolean waitForAuth,
+long dt, long authWaitTime, ConnectionFactory fact) {
+super();
+assert sa != null : "No SocketAddress";
+assert c != null : "No SocketChannel";
+assert bufSize > 0 : "Invalid buffer size: " + bufSize;
+assert rq != null : "No operation read queue";
+assert wq != null : "No operation write queue";
+assert iq != null : "No input queue";
+socketAddress = sa;
+connectionFactory = fact;
+this.authWaitTime = authWaitTime;
+setChannel(c);
+// Since these buffers are allocated rarely (only on client creation
+// or reconfigure), and are passed to Channel.read() and
Channel.write(),
+// use direct buffers to avoid
+// http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6214569
+rbuf = ByteBuffer.allocateDirect(bufSize);
+wbuf = ByteBuffer.allocateDirect(bufSize);
+getWbuf().clear();
+readQ = rq;
+writeQ = wq;
+inputQueue = iq;
+this.opQueueMaxBlockTime = opQueueMaxBlockTime;
+
[kylin] branch master updated: KYLIN-4523 Fix 'LinkageError' error after upgrading httpclient version to 4.3.6
This is an automated email from the ASF dual-hosted git repository. nic pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/kylin.git The following commit(s) were added to refs/heads/master by this push: new fc5bf0a KYLIN-4523 Fix 'LinkageError' error after upgrading httpclient version to 4.3.6 fc5bf0a is described below commit fc5bf0a1c3747e521491ecfa7542f898cbf99fc7 Author: Zhichao Zhang <441586...@qq.com> AuthorDate: Tue May 26 21:41:38 2020 +0800 KYLIN-4523 Fix 'LinkageError' error after upgrading httpclient version to 4.3.6 Problems: After upgrading httpclient version to 4.3.6, on CDH 5.7 env it will lead to LinkageError occured, because it loades 'HttpContext' class in two different ClassLoader (CustomizedWebappClassloader and AppClassLoader): Error message below: java.lang.LinkageError: loader constraint violation: when resolving method "org.apache.http.protocol.HttpCoreContext.(Lorg/apache/http/protocol/HttpContext;)V" the class loader (instance of org/apache/kylin/ext/CustomizedWebappClassloader) of the current class, org/apache/http/client/protocol/HttpClientContext, and the class loader (instance of sun/misc/Launcher$AppClassLoader) for the method's defining class, org/apache/http/protocol/HttpCoreContext, have different Class object [...] --- server/pom.xml | 13 + 1 file changed, 13 insertions(+) diff --git a/server/pom.xml b/server/pom.xml index f92518b..e0ec203 100644 --- a/server/pom.xml +++ b/server/pom.xml @@ -59,6 +59,11 @@ +org.apache.httpcomponents +httpclient + + + com.thetransactioncompany cors-filter @@ -111,6 +116,14 @@ javax.servlet.jsp jsp-api + +org.apache.httpcomponents +httpclient + + +org.apache.httpcomponents +httpcore +
[kylin] branch master updated: KYLIN-4496: Metric data missing
This is an automated email from the ASF dual-hosted git repository.
nic pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kylin.git
The following commit(s) were added to refs/heads/master by this push:
new 278aaba KYLIN-4496: Metric data missing
278aaba is described below
commit 278aababa11fff222327ec6b6f73e659b147b8d7
Author: tianhui5
AuthorDate: Thu May 14 18:34:49 2020 +0800
KYLIN-4496: Metric data missing
---
.../main/java/org/apache/kylin/metrics/lib/impl/hive/HiveProducer.java | 1 +
1 file changed, 1 insertion(+)
diff --git
a/metrics-reporter-hive/src/main/java/org/apache/kylin/metrics/lib/impl/hive/HiveProducer.java
b/metrics-reporter-hive/src/main/java/org/apache/kylin/metrics/lib/impl/hive/HiveProducer.java
index c2dcd7c..1ab3f89 100644
---
a/metrics-reporter-hive/src/main/java/org/apache/kylin/metrics/lib/impl/hive/HiveProducer.java
+++
b/metrics-reporter-hive/src/main/java/org/apache/kylin/metrics/lib/impl/hive/HiveProducer.java
@@ -123,6 +123,7 @@ public class HiveProducer {
public void close() {
tableFieldSchemaCache.cleanUp();
+closeFout();
}
public void send(final Record record) throws Exception {
[kylin] branch master updated (c3b1f0c -> 7437c1a)
This is an automated email from the ASF dual-hosted git repository. nic pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/kylin.git. from c3b1f0c KYLIN-4469 Fix clone model error add d896fce KYLIN-4281 Precisely set the data type of tuple expression add 5fd7249 KYLIN-4275 Fix result of count(case when) is not correct add 94b707f KYLIN-4282 support case when in count (distinct) add b0fadb0 KYLIN-4282 support case when in count (distinct) if all related columns are dims add 873d42c KYLIN-3361 Add a two layer udaf stddev_sum add 7437c1a KYLIN-3487 add a bitmap map-based new measure for precise count distinct No new revisions were added by this update. Summary of changes: .../apache/kylin/cube/CubeCapabilityChecker.java | 12 +- .../org/apache/kylin/cube/CubeDescManager.java | 3 +- .../java/org/apache/kylin/cube/CubeSegment.java| 5 + .../cube/gridtable/CuboidToGridTableMapping.java | 4 +- .../gridtable/CuboidToGridTableMappingExt.java | 11 - .../kylin/cube/gridtable/ScanRangePlannerBase.java | 14 +- .../java/org/apache/kylin/cube/model/CubeDesc.java | 37 +++ .../apache/kylin/cube/util/KeyValueBuilder.java| 37 +++ .../gridtable/GTAggregateTransformScanner.java | 300 + .../apache/kylin/gridtable/GTFunctionScanner.java | 3 +- .../java/org/apache/kylin/gridtable/GTRecord.java | 4 - .../org/apache/kylin/gridtable/GTScanRequest.java | 29 +- .../kylin/gridtable/GTScanRequestBuilder.java | 19 +- .../kylin/gridtable/GTTwoLayerAggregateParam.java | 96 +++ .../gridtable/GTTwoLayerAggregateScanner.java | 63 + .../java/org/apache/kylin/gridtable/GTUtil.java| 6 +- .../java/org/apache/kylin/cube/CubeDescTest.java | 6 + .../gridtable/GTTwoLayerAggregateParamTest.java| 59 .../kylin/measure/MeasureTransformation.java | 9 +- .../java/org/apache/kylin/measure/MeasureType.java | 5 + .../apache/kylin/measure/MeasureTypeFactory.java | 4 + .../kylin/measure/bitmap/BitmapSerializer.java | 3 + .../apache/kylin/measure/hllc/HLLCSerializer.java | 3 + .../apache/kylin/measure/map/MapKeySerializer.java | 22 +- .../kylin/measure/map/bitmap/BitmapCounterMap.java | 116 .../bitmap/BitmapCounterMapFactory.java} | 13 +- .../bitmap/BitmapMapAggregator.java} | 17 +- .../bitmap/BitmapMapDistinctCountAggFunc.java} | 22 +- .../bitmap/BitmapMapMeasureType.java} | 105 .../measure/map/bitmap/BitmapMapSerializer.java| 77 ++ .../map/bitmap/RoaringBitmapCounterMap.java| 23 +- .../bitmap/RoaringBitmapCounterMapFactory.java}| 29 +- .../map/bitmap/SegmentStartTimeKeySerializer.java} | 46 ++-- .../StandardDeviationAggFunc.java} | 36 +-- .../apache/kylin/measure/stddev/StdDevCounter.java | 89 ++ .../kylin/measure/stddev/StdDevSerializer.java | 46 ++-- .../StdDevSumMeasureType.java} | 61 +++-- .../StdDevTransformation.java} | 65 ++--- .../metadata/expression/BinaryTupleExpression.java | 110 ++-- .../metadata/expression/CaseTupleExpression.java | 25 +- .../metadata/expression/ColumnTupleExpression.java | 14 +- ...xpression.java => ConstantTupleExpression.java} | 60 +++-- .../expression/ExpressionColCollector.java | 11 +- .../expression/ExpressionCountDistributor.java | 23 +- .../metadata/expression/ExpressionVisitor.java | 16 +- .../metadata/expression/NoneTupleExpression.java | 2 +- .../expression/RexCallTupleExpression.java | 5 +- .../metadata/expression/StringTupleExpression.java | 89 -- .../kylin/metadata/expression/TupleExpression.java | 77 +- .../expression/TupleExpressionSerializer.java | 40 ++- ...CountDistinctExpressionDynamicFunctionDesc.java | 48 .../kylin/metadata/model/DynamicFunctionDesc.java | 52 +--- ...esc.java => ExpressionDynamicFunctionDesc.java} | 43 ++- .../apache/kylin/metadata/model/FunctionDesc.java | 18 +- .../metadata/model/StdDevSumFunctionDesc.java | 85 ++ java => SumExpressionDynamicFunctionDesc.java} | 35 ++- ...ntStatusEnum.java => TwoLayerFunctionDesc.java} | 19 +- .../measure/map/bitmap/BitmapCounterMapTest.java | 59 .../map/bitmap/BitmapMapAggregatorTest.java| 57 .../expression/ExpressionCountDistributorTest.java | 208 +- .../expression/TupleExpressionSerializerTest.java | 26 +- .../metadata/expression/TupleExpressionTest.java | 51 ++-- .../storage/gtrecord/CubeScanRangePlanner.java | 55 +++- .../kylin/storage/gtrecord/CubeTupleConverter.java | 2 +- .../mr/steps/UpdateCubeInfoAfterBuildStep.java | 15 ++ .../kylin/engine/spark/KylinKryoRegistrator.java | 9 + .../localmeta/cube_desc/ci_inner_join_cube.json| 2 +- .../resources/query/sql_expression
[kylin] branch master updated (178c737 -> c3b1f0c)
This is an automated email from the ASF dual-hosted git repository. nic pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/kylin.git. from 178c737 KYLIN-3698 Fix kylin.sh can't export spark/flink/kafka dependency add c3b1f0c KYLIN-4469 Fix clone model error No new revisions were added by this update. Summary of changes: .../src/main/java/org/apache/kylin/rest/controller/ModelController.java | 1 + 1 file changed, 1 insertion(+)
[kylin] branch document updated: Update website RESTful API page
This is an automated email from the ASF dual-hosted git repository.
nic pushed a commit to branch document
in repository https://gitbox.apache.org/repos/asf/kylin.git
The following commit(s) were added to refs/heads/document by this push:
new 758a8f7 Update website RESTful API page
758a8f7 is described below
commit 758a8f76997d2a0dab6b04d3d3b887fc33cff5b6
Author: yaqian.zhang <598593...@qq.com>
AuthorDate: Fri May 22 17:22:49 2020 +0800
Update website RESTful API page
---
website/_docs/howto/howto_use_restapi.cn.md | 465 +--
website/_docs/howto/howto_use_restapi.md| 478 ++--
2 files changed, 894 insertions(+), 49 deletions(-)
diff --git a/website/_docs/howto/howto_use_restapi.cn.md
b/website/_docs/howto/howto_use_restapi.cn.md
index b1970fd..217f80c 100644
--- a/website/_docs/howto/howto_use_restapi.cn.md
+++ b/website/_docs/howto/howto_use_restapi.cn.md
@@ -7,13 +7,20 @@ since: v0.7.1
---
This page lists the major RESTful APIs provided by Kylin.
-
-* Query
+* Authentication
* [Authentication](#authentication)
+* Query
* [Query](#query)
+ * [Prepare query](#prepare-query)
+ * [Save query](#save-query)
+ * [Remove saved query](#remove-saved-query)
+ * [Get saved queries](#get-saved-queries)
+ * [Get running queries](#get-running-queries)
+ * [Stop query](#stop-query)
* [List queryable tables](#list-queryable-tables)
* CUBE
* [Create cube](#create-cube)
+ * [Update cube](#update-cube)
* [List cubes](#list-cubes)
* [Get cube](#get-cube)
* [Get cube descriptor (dimension, measure info, etc)](#get-cube-descriptor)
@@ -24,10 +31,20 @@ This page lists the major RESTful APIs provided by Kylin.
* [Purge cube](#purge-cube)
* [Delete segment](#delete-segment)
* [Auto-Merge segment](#auto-merge-segment)
+ * [Get sql of a cube](#get-sql-of-a-cube)
+ * [Get sql of a cube segment](#get-sql-of-a-cube-segment)
+ * [Force rebuild lookup table
snapshot](#force-rebuild-lookup-table-snapshot)
+ * [Clone cube](#clone-cube)
+ * [Delete Cube](#delete-cube)
+ * [Get hbase info](#get-hbase-info)
+ * [Get current cuboid](#get-current-cuboid)
+ * [Migrate cube](#migrate-cube)
* MODEL
* [Create model](#create-model)
+ * [Update model](#update-model)
* [Get modelDescData](#get-modeldescdata)
* [Delete model](#delete-model)
+ * [Clone model](#clone-model)
* JOB
* [Resume job](#resume-job)
* [Pause job](#pause-job)
@@ -36,17 +53,29 @@ This page lists the major RESTful APIs provided by Kylin.
* [Get job status](#get-job-status)
* [Get job step output](#get-job-step-output)
* [Get job list](#get-job-list)
+ * [Get job status overview](#get-job-status-overview)
+ * [Resubmit realtime build job](#resubmit-realtime-build-job)
+ * [Rollback job](#rollback-job)
* Metadata
* [Get Hive Table](#get-hive-table)
* [Get Hive Tables](#get-hive-tables)
* [Load Hive Tables](#load-hive-tables)
+ * [Unload Hive Tables](#unload-hive-tables)
+ * [Show databases in hive](#show-databases-in-hive)
+ * [Show tables in a hive database](#show-tables-in-a-hive-database)
* Cache
* [Wipe cache](#wipe-cache)
+ * [Announce wipe cache](#announce-wipe-cache)
+ * [Hot load kylin config](#hot-load-kylin-config)
* Streaming
* [Initiate cube start position](#initiate-cube-start-position)
* [Build stream cube](#build-stream-cube)
* [Check segment holes](#check-segment-holes)
* [Fill segment holes](#fill-segment-holes)
+ * [Get streaming configs](#get-streaming-configs)
+ * [Get Kafka configs](#get-kafka-configs)
+ * [Create streaming schema](#create-streaming-schema)
+ * [Update streaming tables](#update-streaming-schema)
* ACL
* [Get users can query the table](#get-users-can-query-the-table)
* [Get users cannot query the table](#get-users-cannot-query-the-table)
@@ -70,23 +99,33 @@ python -c "import base64; print
base64.standard_b64encode('$UserName:$Password')
Response Sample
```sh
-{
- "userDetails":{
- "password":null,
- "username":"sample",
- "authorities":[
- {
-"authority":"ROLE_ANALYST"
- },
- {
-"authority":"ROLE_MODELER"
- }
- ],
- "accountNonExpired":true,
- "accountNonLocked":true,
- "credentialsNonExpired":true,
- "enabled":true
- }
+{
+"userDetails": {
+"username": "sample",
+"password": "null",
+"authorities": [
+{
+"authority": "ROLE_ADMIN"
+},
+{
+"authority": "ROLE_ANALYST"
+},
+{
+"authority": &q
[kylin] branch master updated: KYLIN-3698 Fix kylin.sh can't export spark/flink/kafka dependency
This is an automated email from the ASF dual-hosted git repository.
nic pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kylin.git
The following commit(s) were added to refs/heads/master by this push:
new 178c737 KYLIN-3698 Fix kylin.sh can't export spark/flink/kafka
dependency
178c737 is described below
commit 178c7377d39b004144e54b1c931c92be82fe76de
Author: yaqian.zhang <598593...@qq.com>
AuthorDate: Fri May 22 13:42:10 2020 +0800
KYLIN-3698 Fix kylin.sh can't export spark/flink/kafka dependency
---
build/bin/check-env.sh | 12 ++--
build/bin/find-flink-dependency.sh | 1 -
build/bin/find-kafka-dependency.sh | 1 -
build/bin/find-spark-dependency.sh | 1 -
4 files changed, 6 insertions(+), 9 deletions(-)
diff --git a/build/bin/check-env.sh b/build/bin/check-env.sh
index e3b61c7..573b600 100755
--- a/build/bin/check-env.sh
+++ b/build/bin/check-env.sh
@@ -84,27 +84,27 @@ else
fi
-bash ${dir}/find-spark-dependency.sh
+source ${dir}/find-spark-dependency.sh
rtn=$?
if [[ $rtn == 0 ]]; then
echo "...[`setColor 32
PASS`]"
-elif [[ $rtn == 3 ]]; then
+else
echo "...[`setColor 33
SKIP`]"
fi
-bash ${dir}/find-flink-dependency.sh
+source ${dir}/find-flink-dependency.sh
rtn=$?
if [[ $rtn == 0 ]]; then
echo "...[`setColor 32
PASS`]"
-elif [[ $rtn == 3 ]]; then
+else
echo "...[`setColor 33
SKIP`]"
fi
-bash ${dir}/find-kafka-dependency.sh
+source ${dir}/find-kafka-dependency.sh
rtn=$?
if [[ $rtn == 0 ]]; then
echo "...[`setColor 32
PASS`]"
-elif [[ $rtn == 3 ]]; then
+else
echo "...[`setColor 33
SKIP`]"
fi
diff --git a/build/bin/find-flink-dependency.sh
b/build/bin/find-flink-dependency.sh
index 10f3643..ea09a38 100755
--- a/build/bin/find-flink-dependency.sh
+++ b/build/bin/find-flink-dependency.sh
@@ -39,7 +39,6 @@ if [ ! -d "$flink_home/lib" ]
then
echo `setColor 33 "Optional dependency flink not found, if you need this;
set FLINK_HOME, or run bin/download-flink.sh"`
echo "echo 'skip flink_dependency'" > ${dir}/cached-flink-dependency.sh
-exit 3
else
flink_dependency=`find -L $flink_home/lib -name '*.jar' ! -name
'*shaded-hadoop*' ! -name 'kafka*' ! -name '*log4j*' ! -name '*slf4j*' ! -name
'*calcite*' ! -name '*doc*' ! -name '*test*' ! -name '*sources*' ''-printf
'%p:' | sed 's/:$//'`
if [ -z "$flink_dependency" ]
diff --git a/build/bin/find-kafka-dependency.sh
b/build/bin/find-kafka-dependency.sh
index 4782680..7a293b5 100755
--- a/build/bin/find-kafka-dependency.sh
+++ b/build/bin/find-kafka-dependency.sh
@@ -28,7 +28,6 @@ echo Retrieving kafka dependency...
if [ -z "$KAFKA_HOME" ]
then
echo "Couldn't find kafka home. If you want to enable streaming
processing, Please set KAFKA_HOME to the path which contains kafka
dependencies."
-exit 3
else
verbose "KAFKA_HOME is set to: $KAFKA_HOME, use it to locate kafka
dependencies."
kafka_home=$KAFKA_HOME
diff --git a/build/bin/find-spark-dependency.sh
b/build/bin/find-spark-dependency.sh
index 2321cea..0ea7ab2 100755
--- a/build/bin/find-spark-dependency.sh
+++ b/build/bin/find-spark-dependency.sh
@@ -59,7 +59,6 @@ if [ ! -d "$spark_home/jars" ]
then
echo `setColor 33 "Optional dependency spark not found, if you need this;
set SPARK_HOME, or run bin/download-spark.sh"`
echo "echo 'skip spark_dependency'" > ${dir}/cached-spark-dependency.sh
-exit 3
else
spark_dependency=`find -L $spark_home/jars -name '*.jar' ! -name '*slf4j*'
! -name '*calcite*' ! -name '*doc*' ! -name '*test*' ! -name '*sources*'
''-printf '%p:' | sed 's/:$//'`
if [ -z "$spark_dependency" ]
[kylin] branch master updated: KYLIN-3698 some enhancement for check-env.sh
This is an automated email from the ASF dual-hosted git repository.
nic pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kylin.git
The following commit(s) were added to refs/heads/master by this push:
new a471808 KYLIN-3698 some enhancement for check-env.sh
a471808 is described below
commit a471808626ee9349173255d329673c509b5f4037
Author: yaqian.zhang <598593...@qq.com>
AuthorDate: Wed May 20 18:37:46 2020 +0800
KYLIN-3698 some enhancement for check-env.sh
---
build/bin/check-env.sh | 38 ---
build/bin/find-flink-dependency.sh | 3 ++-
build/bin/find-kafka-dependency.sh | 5 -
build/bin/find-spark-dependency.sh | 41 --
build/bin/kylin.sh | 6 --
5 files changed, 54 insertions(+), 39 deletions(-)
diff --git a/build/bin/check-env.sh b/build/bin/check-env.sh
index a6250ed..e3b61c7 100755
--- a/build/bin/check-env.sh
+++ b/build/bin/check-env.sh
@@ -83,35 +83,31 @@ else
echo "...[`setColor 32
PASS`]"
fi
-source ${dir}/find-spark-dependency.sh
-if [[ $? == 0 ]]; then
+
+bash ${dir}/find-spark-dependency.sh
+rtn=$?
+if [[ $rtn == 0 ]]; then
echo "...[`setColor 32
PASS`]"
+elif [[ $rtn == 3 ]]; then
+echo "...[`setColor 33
SKIP`]"
fi
-source ${dir}/find-flink-dependency.sh
-if [[ $? == 0 ]]; then
+bash ${dir}/find-flink-dependency.sh
+rtn=$?
+if [[ $rtn == 0 ]]; then
echo "...[`setColor 32
PASS`]"
+elif [[ $rtn == 3 ]]; then
+echo "...[`setColor 33
SKIP`]"
fi
-SPARK_EVENTLOG_DIR=`bash $KYLIN_HOME/bin/get-properties.sh
kylin.engine.spark-conf.spark.eventLog.dir`
-if [ -n "$SPARK_EVENTLOG_DIR" ]
-then
-hadoop ${hadoop_conf_param} fs -mkdir -p $SPARK_EVENTLOG_DIR
-if [ $? != 0 ]
-then
-quit "Failed to create $SPARK_EVENTLOG_DIR. Please make sure the user
has right to access $SPARK_EVENTLOG_DIR"
-fi
+bash ${dir}/find-kafka-dependency.sh
+rtn=$?
+if [[ $rtn == 0 ]]; then
+echo "...[`setColor 32
PASS`]"
+elif [[ $rtn == 3 ]]; then
+echo "...[`setColor 33
SKIP`]"
fi
-SPARK_HISTORYLOG_DIR=`bash $KYLIN_HOME/bin/get-properties.sh
kylin.engine.spark-conf.spark.history.fs.logDirectory`
-if [ -n "$SPARK_HISTORYLOG_DIR" ]
-then
-hadoop ${hadoop_conf_param} fs -mkdir -p $SPARK_HISTORYLOG_DIR
-if [ $? != 0 ]
-then
-quit "Failed to create $SPARK_HISTORYLOG_DIR. Please make sure the
user has right to access $SPARK_HISTORYLOG_DIR"
-fi
-fi
${KYLIN_HOME}/bin/check-port-availability.sh || exit 1;
diff --git a/build/bin/find-flink-dependency.sh
b/build/bin/find-flink-dependency.sh
index 4485730..10f3643 100755
--- a/build/bin/find-flink-dependency.sh
+++ b/build/bin/find-flink-dependency.sh
@@ -37,8 +37,9 @@ fi
if [ ! -d "$flink_home/lib" ]
then
-echo "Optional dependency flink not found, if you need this; set
FLINK_HOME, or run bin/download-flink.sh"
+echo `setColor 33 "Optional dependency flink not found, if you need this;
set FLINK_HOME, or run bin/download-flink.sh"`
echo "echo 'skip flink_dependency'" > ${dir}/cached-flink-dependency.sh
+exit 3
else
flink_dependency=`find -L $flink_home/lib -name '*.jar' ! -name
'*shaded-hadoop*' ! -name 'kafka*' ! -name '*log4j*' ! -name '*slf4j*' ! -name
'*calcite*' ! -name '*doc*' ! -name '*test*' ! -name '*sources*' ''-printf
'%p:' | sed 's/:$//'`
if [ -z "$flink_dependency" ]
diff --git a/build/bin/find-kafka-dependency.sh
b/build/bin/find-kafka-dependency.sh
index b12e6a4..4782680 100755
--- a/build/bin/find-kafka-dependency.sh
+++ b/build/bin/find-kafka-dependency.sh
@@ -19,13 +19,16 @@
source ${KYLIN_HOME:-"$(cd -P -- "$(dirname -- "$0")" && pwd
-P)/../"}/bin/header.sh
+echo Retrieving Kafka dependency...
+
kafka_home=
echo Retrieving kafka dependency...
if [ -z "$KAFKA_HOME" ]
then
-verbose "Couldn't find kafka home. If you want to enable streaming
processing, Please set KAFKA_HOME to the path which contains kafka
dependencies."
+echo "Couldn't find kafka home. If you want to enable streaming
processing, Please set KAFKA_HOME to the path which contains kafka
dependencies."
+exit 3
else
verbose "KAFKA_HOME is set to: $KAFKA_HOME, use it to locate kafka
dependencies."
kafka_home=$KAFKA_HOME
diff --git a/build/bin/find-spark-dependency.sh
b/build/bin/find-spark-dependency
[kylin] branch document updated: Add security issue of CVE-2020-1956
This is an automated email from the ASF dual-hosted git repository. nic pushed a commit to branch document in repository https://gitbox.apache.org/repos/asf/kylin.git The following commit(s) were added to refs/heads/document by this push: new 5ed64e1 Add security issue of CVE-2020-1956 5ed64e1 is described below commit 5ed64e16129476e4502a1a2cf1b417ba359cfc40 Author: nichunen AuthorDate: Wed May 20 09:40:57 2020 +0800 Add security issue of CVE-2020-1956 --- website/_docs/security.md | 37 - 1 file changed, 36 insertions(+), 1 deletion(-) diff --git a/website/_docs/security.md b/website/_docs/security.md index 8dca57a..f905b3d 100644 --- a/website/_docs/security.md +++ b/website/_docs/security.md @@ -38,4 +38,39 @@ Users should upgrade to 3.0.1 or 2.6.5 __Credit__ -This issue was discovered by Jonathan Leitschuh \ No newline at end of file +This issue was discovered by Jonathan Leitschuh + +### [CVE-2020-1956](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-1956) Apache Kylin command injection vulnerability + +__Severity__ + + +Important + +__Vendor__ + +The Apache Software Foundation + +__Versions Affected__ + +Kylin 2.3.0 to 2.3.2 + +Kylin 2.4.0 to 2.4.1 + +Kylin 2.5.0 to 2.5.2 + +Kylin 2.6.0 to 2.6.5 + +Kylin 3.0.0-alpha, Kylin 3.0.0-alpha2, Kylin 3.0.0-beta, Kylin 3.0.0, Kylin 3.0.1 + +__Description__ + +Kylin has some restful api which will concat os command with the user input string, a user is likely to be able to execute any os command without any protection or validation. + +__Mitigation__ + +Users should upgrade to 3.0.2 or 2.6.6 or set kylin.tool.auto-migrate-cube.enabled to false to disable command execution. + +__Credit__ + +This issue was discovered by Johannes Dahse
[kylin] branch document updated: Doc update for Kylin 2.6.6&3.0.2
This is an automated email from the ASF dual-hosted git repository. nic pushed a commit to branch document in repository https://gitbox.apache.org/repos/asf/kylin.git The following commit(s) were added to refs/heads/document by this push: new 2c82e56 Doc update for Kylin 2.6.6&3.0.2 2c82e56 is described below commit 2c82e568fd5458414769f673e1459679a8f166be Author: nichunen AuthorDate: Tue May 19 10:37:32 2020 +0800 Doc update for Kylin 2.6.6&3.0.2 --- website/_docs/release_notes.md | 65 ++ website/download/index.md | 28 +- 2 files changed, 79 insertions(+), 14 deletions(-) diff --git a/website/_docs/release_notes.md b/website/_docs/release_notes.md index de5d1e7..1efdad8 100644 --- a/website/_docs/release_notes.md +++ b/website/_docs/release_notes.md @@ -15,6 +15,41 @@ or send to Apache Kylin mailing list: * User relative: [u...@kylin.apache.org](mailto:u...@kylin.apache.org) * Development relative: [d...@kylin.apache.org](mailto:d...@kylin.apache.org) +## v3.0.2 - 2020-05-19 +_Tag:_ [kylin-3.0.2](https://github.com/apache/kylin/tree/kylin-3.0.2) +This is a bugfix release after 3.0.1, with 6 enhancements and 19 bug fixes. + +__Improvement__ + +* [KYLIN-3628] - Query with lookup table always use latest snapshot +* [KYLIN-4132] - Kylin needn't use "org.apache.directory.api.util.Strings" to import api-util.jar +* [KYLIN-4388] - Refine the Dockerfile +* [KYLIN-4390] - Update tomcat to 7.0.100 +* [KYLIN-4400] - Use beeline as hive client in system-cube.sh +* [KYLIN-4437] - Should replace deprecated "mapred.job.name" + +__Bug Fix__ + +* [KYLIN-4119] - The admin of project can't operate the action of Hybrids +* [KYLIN-4206] - Build kylin on EMR 5.23. The kylin version is 2.6.4. When building the cube, the hive table cannot be found +* [KYLIN-4340] - Fix bug of get value of isSparkFactDistinctEnable for cube not correct +* [KYLIN-4353] - Realtime Segment is not closed in expected duration +* [KYLIN-4354] - Prune segment not using given filter when using jdbc preparestatement +* [KYLIN-4370] - Spark job failing with JDBC source on 8th step with error : org.apache.kylin.engine.spark.SparkCubingByLayer. Root cause: Table or view not found: `default`.`kylin_intermediate table' +* [KYLIN-4372] - Docker entrypoint delete file too later cause ZK started by HBase crash +* [KYLIN-4379] - Calculate column cardinality cannot use kylin config overwrite cause job failed +* [KYLIN-4383] - Kylin Integrated Issue with Amazon EMR and AWS Glue in HiveMetaStoreClientFactory.java +* [KYLIN-4385] - KYLIN system cube failing to update table when run on EMR with S3 as storage and EMRFS +* [KYLIN-4396] - File Descriptor Leakage in MR Build Engine +* [KYLIN-4397] - Use newLinkedHashMap in AssignmentUtil.java +* [KYLIN-4405] - Internal exception when trying to build cube whose modal has null PartitionDesc +* [KYLIN-4425] - Refactor Diagnosis Tool +* [KYLIN-4426] - Refine CliCommandExecutor +* [KYLIN-4433] - When uhc step is turned on, Build Dimension Dictionary job cannot get correct configuration +* [KYLIN-4438] - Null password may cause RuntimeException when starting up +* [KYLIN-4470] - The user cannot log in kylin normally after being assigned to a group +* [KYLIN-4481] - Project-level ACL lookups not working for non-admin SAML-federated users + ## v3.0.1 - 2020-02-20 _Tag:_ [kylin-3.0.1](https://github.com/apache/kylin/tree/kylin-3.0.1) This is a bugfix release after 3.0.0, with 10 enhancements and 14 bug fixes. @@ -303,6 +338,36 @@ __Bug Fix__ * [KYLIN-3922] - Fail to update coprocessor when run DeployCoprocessorCLI * [KYLIN-3923] - UT GeneralColumnDataTest fail +## v2.6.6 - 2020-05-19 +_Tag:_ [kylin-2.6.6](https://github.com/apache/kylin/tree/kylin-2.6.6) +This is a bugfix release after 2.6.5, with 6 enhancements and 14 bug fixes. + +__Improvement__ + +* [KYLIN-3628] - Query with lookup table always use latest snapshot +* [KYLIN-4132] - Kylin needn't use "org.apache.directory.api.util.Strings" to import api-util.jar +* [KYLIN-4160] - Auto redirect to host:port/kylin when user only enter host:port in broswer +* [KYLIN-4314] - Support union in intersect_count() function +* [KYLIN-4390] - Update tomcat to 7.0.100 +* [KYLIN-4400] - Use beeline as hive client in system-cube.sh + +__Bug Fix__ + +* [KYLIN-4072] - CDH 6.x find-hbase-dependency.sh return with "base-common lib not found" +* [KYLIN-4119] - The admin of project can't operate the action of Hybrids +* [KYLIN-4206] - Build kylin on EMR 5.23. The kylin version is 2.6.4. When building the cube, the hive table cannot be found +* [KYLIN-4245] - SqlConverter returns wrong syntax SQL when SqlParser fails for JDBC source +* [KYLIN-4354] - Prune segment not using given filter when using jdbc preparestatement +* [KYLIN-4370] - Spark job failing with JDBC source on 8th step with error : org.apache.kylin.engine.spark.SparkCub
[kylin] 03/07: KYLIN-4416 Disable htable compaction
This is an automated email from the ASF dual-hosted git repository.
nic pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kylin.git
commit 03274297b1dcdd26a93ee6903e764d92e8847a85
Author: Zhong, Yanghong
AuthorDate: Tue Mar 10 14:55:35 2020 +0800
KYLIN-4416 Disable htable compaction
---
.../src/main/java/org/apache/kylin/common/KylinConfigBase.java| 4
.../java/org/apache/kylin/storage/hbase/steps/CreateHTableJob.java| 4 ++--
.../java/org/apache/kylin/storage/hbase/steps/CubeHTableUtil.java | 1 +
3 files changed, 7 insertions(+), 2 deletions(-)
diff --git
a/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java
b/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java
index 7136a51..b0f7fec 100644
--- a/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java
+++ b/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java
@@ -1263,6 +1263,10 @@ public abstract class KylinConfigBase implements
Serializable {
return
Integer.parseInt(getOptional("kylin.storage.hbase.htable-available-retry",
"3"));
}
+public int getHBaseRegionCompactionThreshold() {
+return
Integer.parseInt(getOptional("kylin.storage.hbase.region-compaction-threshold",
"3"));
+}
+
public int getHBaseRegionCountMin() {
return
Integer.parseInt(getOptional("kylin.storage.hbase.min-region-count", "1"));
}
diff --git
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/CreateHTableJob.java
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/CreateHTableJob.java
index 7c970c2..4315957 100644
---
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/CreateHTableJob.java
+++
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/CreateHTableJob.java
@@ -280,8 +280,8 @@ public class CreateHTableJob extends AbstractHadoopJob {
hfileSizeMB = mbPerRegion / 2f;
}
-int compactionThreshold =
Integer.parseInt(hbaseConf.get("hbase.hstore.compactionThreshold", "3"));
-logger.info("hbase.hstore.compactionThreshold is {}",
compactionThreshold);
+int compactionThreshold =
kylinConfig.getHBaseRegionCompactionThreshold();
+logger.info("kylin.storage.hbase.region-compaction-threshold is " +
compactionThreshold);
if (hfileSizeMB > 0.0f && hfileSizeMB * compactionThreshold <
mbPerRegion) {
hfileSizeMB = ((float) mbPerRegion) / compactionThreshold;
}
diff --git
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/CubeHTableUtil.java
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/CubeHTableUtil.java
index 369c7bc..b99e46c 100644
---
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/CubeHTableUtil.java
+++
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/CubeHTableUtil.java
@@ -66,6 +66,7 @@ public class CubeHTableUtil {
KylinConfig kylinConfig = cubeDesc.getConfig();
HTableDescriptor tableDesc = new HTableDescriptor(tableName);
+tableDesc.setCompactionEnabled(false);
tableDesc.setValue(HTableDescriptor.SPLIT_POLICY,
DisabledRegionSplitPolicy.class.getName());
tableDesc.setValue(IRealizationConstants.HTableTag,
kylinConfig.getMetadataUrlPrefix());
tableDesc.setValue(IRealizationConstants.HTableCreationTime,
String.valueOf(System.currentTimeMillis()));
[kylin] branch master updated (e7ab5df -> 7886a24)
This is an automated email from the ASF dual-hosted git repository. nic pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/kylin.git. from e7ab5df KYLIN-4151 FileSplit ClassCastException in KafkaMRInput new 372f4dd KYLIN-4325 fix TypeFlag for global dictionary field in SelfDefineSortableKey new 20353a3 KYLIN-4415 HTable Creation with Retry new 0327429 KYLIN-4416 Disable htable compaction new 981cc27 KYLIN-4414 add a property jobTempDir for AbstractHadoopJob to indicate the directory for all of the temporary files new 8d2a53f KYLIN-4414 bulkload needs to follow locality new beb976a KYLIN-4417 Use hash rather than random to avoid potential issue in ConvergeCuboidDataPartitioner new 7886a24 KYLIN-4418 Bug fix for ShardingHash.getShard in HBaseLookupRowEncoder The 7 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../org/apache/kylin/common/KylinConfigBase.java | 8 ++ .../kylin/engine/mr/common/AbstractHadoopJob.java | 48 .../mr/steps/ConvergeCuboidDataPartitioner.java| 17 ++- .../engine/mr/steps/FactDistinctColumnsMapper.java | 6 +- .../hbase/lookup/HBaseLookupRowEncoder.java| 3 +- .../kylin/storage/hbase/steps/CreateHTableJob.java | 6 +- .../kylin/storage/hbase/steps/CubeHFileJob.java| 15 ++- .../kylin/storage/hbase/steps/CubeHTableUtil.java | 79 +++-- .../storage/hbase/steps/HFileOutputFormat3.java| 126 - 9 files changed, 251 insertions(+), 57 deletions(-)
[kylin] 04/07: KYLIN-4414 add a property jobTempDir for AbstractHadoopJob to indicate the directory for all of the temporary files
This is an automated email from the ASF dual-hosted git repository.
nic pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kylin.git
commit 981cc273be31bb3099c3de34c78f88f986b0db8d
Author: Zhong, Yanghong
AuthorDate: Tue Mar 10 16:11:34 2020 +0800
KYLIN-4414 add a property jobTempDir for AbstractHadoopJob to indicate the
directory for all of the temporary files
---
.../kylin/engine/mr/common/AbstractHadoopJob.java | 48 ++
1 file changed, 21 insertions(+), 27 deletions(-)
diff --git
a/engine-mr/src/main/java/org/apache/kylin/engine/mr/common/AbstractHadoopJob.java
b/engine-mr/src/main/java/org/apache/kylin/engine/mr/common/AbstractHadoopJob.java
index fd4d413..411caa7 100644
---
a/engine-mr/src/main/java/org/apache/kylin/engine/mr/common/AbstractHadoopJob.java
+++
b/engine-mr/src/main/java/org/apache/kylin/engine/mr/common/AbstractHadoopJob.java
@@ -157,6 +157,7 @@ public abstract class AbstractHadoopJob extends Configured
implements Tool {
protected OptionsHelper optionsHelper = new OptionsHelper();
protected Job job;
+private File jobTempDir;
public AbstractHadoopJob() {
super(HadoopUtil.getCurrentConfiguration());
@@ -595,10 +596,7 @@ public abstract class AbstractHadoopJob extends Configured
implements Tool {
protected void dumpKylinPropsAndMetadata(String prj, Set dumpList,
KylinConfig kylinConfig,
Configuration conf) throws IOException {
-File tmp = File.createTempFile("kylin_job_meta", "");
-FileUtils.forceDelete(tmp); // we need a directory, so delete the file
first
-
-File metaDir = new File(tmp, "meta");
+File metaDir = new File(getJobTempDir(), "meta");
metaDir.mkdirs();
// write kylin.properties
@@ -628,33 +626,29 @@ public abstract class AbstractHadoopJob extends
Configured implements Tool {
}
protected void cleanupTempConfFile(Configuration conf) {
-String[] tempfiles = StringUtils.split(conf.get("tmpfiles"), ",");
-if (tempfiles == null) {
-return;
-}
-for (String tempMetaFileString : tempfiles) {
-logger.trace("tempMetaFileString is : " + tempMetaFileString);
-if (tempMetaFileString != null) {
-if (tempMetaFileString.startsWith("file://")) {
-tempMetaFileString =
tempMetaFileString.substring("file://".length());
-File tempMetaFile = new File(tempMetaFileString);
-if (tempMetaFile.exists()) {
-try {
-
FileUtils.forceDelete(tempMetaFile.getParentFile());
-
-} catch (IOException e) {
-logger.warn("error when deleting " + tempMetaFile,
e);
-}
-} else {
-logger.info("" + tempMetaFileString + " does not
exist");
-}
-} else {
-logger.info("tempMetaFileString is not starting with
file:// :" + tempMetaFileString);
-}
+String tmpFilesString = conf.get("tmpfiles");
+logger.info("tmpFilesString is : " + tmpFilesString);
+if (jobTempDir != null) {
+try {
+FileUtils.forceDelete(jobTempDir);
+} catch (IOException e) {
+logger.warn("error when deleting " + jobTempDir, e);
}
}
}
+// It's not thread safe
+protected File getJobTempDir() throws IOException {
+if (jobTempDir != null && jobTempDir.isDirectory()) {
+return jobTempDir;
+}
+jobTempDir = File.createTempFile("kylin_job_meta", "");
+FileUtils.forceDelete(jobTempDir); // we need a directory, so delete
the file first
+
+jobTempDir.mkdirs();
+return jobTempDir;
+}
+
protected void deletePath(Configuration conf, Path path) throws
IOException {
HadoopUtil.deletePath(conf, path);
}
[kylin] 01/07: KYLIN-4325 fix TypeFlag for global dictionary field in SelfDefineSortableKey
This is an automated email from the ASF dual-hosted git repository.
nic pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kylin.git
commit 372f4ddf324dde8b1bc430710f7c3a09e7f30151
Author: Zhong, Yanghong
AuthorDate: Tue Mar 10 14:17:15 2020 +0800
KYLIN-4325 fix TypeFlag for global dictionary field in SelfDefineSortableKey
---
.../org/apache/kylin/engine/mr/steps/FactDistinctColumnsMapper.java | 6 +-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git
a/engine-mr/src/main/java/org/apache/kylin/engine/mr/steps/FactDistinctColumnsMapper.java
b/engine-mr/src/main/java/org/apache/kylin/engine/mr/steps/FactDistinctColumnsMapper.java
index 2c3bc8d..0fa12f9 100755
---
a/engine-mr/src/main/java/org/apache/kylin/engine/mr/steps/FactDistinctColumnsMapper.java
+++
b/engine-mr/src/main/java/org/apache/kylin/engine/mr/steps/FactDistinctColumnsMapper.java
@@ -285,7 +285,11 @@ public class FactDistinctColumnsMapper extends
FactDistinctColumnsMapperB
tmpbuf.put(Bytes.toBytes(reducerIndex)[3]);
tmpbuf.put(valueBytes);
outputKey.set(tmpbuf.array(), 0, tmpbuf.position());
-sortableKey.init(outputKey, type);
+if (cubeDesc.getDictionaryBuilderClass(allCols.get(colIndex)) == null)
{
+sortableKey.init(outputKey, type);
+} else {
+sortableKey.init(outputKey, (byte) 0);
+}
context.write(sortableKey, EMPTY_TEXT);
// log a few rows for troubleshooting
if (rowCount < 10) {
[kylin] 07/07: KYLIN-4418 Bug fix for ShardingHash.getShard in HBaseLookupRowEncoder
This is an automated email from the ASF dual-hosted git repository.
nic pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kylin.git
commit 7886a2498178ded085497d6204b4b367f56ce0c8
Author: Zhong, Yanghong
AuthorDate: Tue Mar 10 16:44:15 2020 +0800
KYLIN-4418 Bug fix for ShardingHash.getShard in HBaseLookupRowEncoder
---
.../org/apache/kylin/storage/hbase/lookup/HBaseLookupRowEncoder.java | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/lookup/HBaseLookupRowEncoder.java
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/lookup/HBaseLookupRowEncoder.java
index 7269465..51bd32b 100644
---
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/lookup/HBaseLookupRowEncoder.java
+++
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/lookup/HBaseLookupRowEncoder.java
@@ -88,7 +88,8 @@ public class HBaseLookupRowEncoder extends
AbstractLookupRowEncoder {
}
byte[] result = new byte[RowConstants.ROWKEY_SHARDID_LEN +
keyByteBuffer.position()];
System.arraycopy(keyByteBuffer.array(), 0, result,
RowConstants.ROWKEY_SHARDID_LEN, keyByteBuffer.position());
-short shard = ShardingHash.getShard(result,
RowConstants.ROWKEY_SHARDID_LEN, result.length, shardNum);
+short shard = ShardingHash.getShard(result,
RowConstants.ROWKEY_SHARDID_LEN,
+result.length - RowConstants.ROWKEY_SHARDID_LEN, shardNum);
BytesUtil.writeShort(shard, result, 0,
RowConstants.ROWKEY_SHARDID_LEN);
return result;
}
[kylin] 02/07: KYLIN-4415 HTable Creation with Retry
This is an automated email from the ASF dual-hosted git repository.
nic pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kylin.git
commit 20353a339054b71ae50386bf08397c780d2244e7
Author: Zhong, Yanghong
AuthorDate: Tue Mar 10 14:45:03 2020 +0800
KYLIN-4415 HTable Creation with Retry
---
.../org/apache/kylin/common/KylinConfigBase.java | 4 ++
.../kylin/storage/hbase/steps/CreateHTableJob.java | 2 +-
.../kylin/storage/hbase/steps/CubeHTableUtil.java | 78 +++---
3 files changed, 73 insertions(+), 11 deletions(-)
diff --git
a/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java
b/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java
index 16c07c1..7136a51 100644
--- a/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java
+++ b/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java
@@ -1259,6 +1259,10 @@ public abstract class KylinConfigBase implements
Serializable {
}
}
+public int getHBaseHTableAvailableRetry() {
+return
Integer.parseInt(getOptional("kylin.storage.hbase.htable-available-retry",
"3"));
+}
+
public int getHBaseRegionCountMin() {
return
Integer.parseInt(getOptional("kylin.storage.hbase.min-region-count", "1"));
}
diff --git
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/CreateHTableJob.java
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/CreateHTableJob.java
index 4b2218b..7c970c2 100644
---
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/CreateHTableJob.java
+++
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/CreateHTableJob.java
@@ -117,7 +117,7 @@ public class CreateHTableJob extends AbstractHadoopJob {
splitKeys = getRegionSplitsFromCuboidStatistics(cuboidSizeMap,
kylinConfig, cubeSegment,
partitionFilePath.getParent());
-CubeHTableUtil.createHTable(cubeSegment, splitKeys);
+CubeHTableUtil.createHTable(cubeSegment, splitKeys, true);
// export configuration in advance to avoid connecting to hbase from
spark
if (cubeDesc.getEngineType()== IEngineAware.ID_SPARK){
diff --git
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/CubeHTableUtil.java
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/CubeHTableUtil.java
index d06c993..369c7bc 100644
---
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/CubeHTableUtil.java
+++
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/CubeHTableUtil.java
@@ -19,16 +19,20 @@
package org.apache.kylin.storage.hbase.steps;
import java.io.IOException;
-
import java.util.Locale;
+
import org.apache.commons.io.IOUtils;
import org.apache.commons.lang.StringUtils;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HColumnDescriptor;
import org.apache.hadoop.hbase.HTableDescriptor;
+import org.apache.hadoop.hbase.TableExistsException;
import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.TableNotEnabledException;
+import org.apache.hadoop.hbase.TableNotFoundException;
import org.apache.hadoop.hbase.client.Admin;
import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.exceptions.TimeoutIOException;
import org.apache.hadoop.hbase.io.compress.Compression.Algorithm;
import org.apache.hadoop.hbase.io.encoding.DataBlockEncoding;
import org.apache.hadoop.hbase.regionserver.BloomType;
@@ -54,13 +58,14 @@ public class CubeHTableUtil {
private static final Logger logger =
LoggerFactory.getLogger(CubeHTableUtil.class);
-public static void createHTable(CubeSegment cubeSegment, byte[][]
splitKeys) throws IOException {
-String tableName = cubeSegment.getStorageLocationIdentifier();
+public static void createHTable(CubeSegment cubeSegment, byte[][]
splitKeys, boolean continueOnExists)
+throws IOException {
+TableName tableName =
TableName.valueOf(cubeSegment.getStorageLocationIdentifier());
CubeInstance cubeInstance = cubeSegment.getCubeInstance();
CubeDesc cubeDesc = cubeInstance.getDescriptor();
KylinConfig kylinConfig = cubeDesc.getConfig();
-HTableDescriptor tableDesc = new
HTableDescriptor(TableName.valueOf(cubeSegment.getStorageLocationIdentifier()));
+HTableDescriptor tableDesc = new HTableDescriptor(tableName);
tableDesc.setValue(HTableDescriptor.SPLIT_POLICY,
DisabledRegionSplitPolicy.class.getName());
tableDesc.setValue(IRealizationConstants.HTableTag,
kylinConfig.getMetadataUrlPrefix());
tableDesc.setValue(IRealizationConstants.HTableCreationTime,
String.valueOf(System.currentTimeMillis()));
@@ -95,16 +100,69 @@ public class CubeHTableUtil {
tableDesc.addFamily(cf);
}
-if (admin
[kylin] 05/07: KYLIN-4414 bulkload needs to follow locality
This is an automated email from the ASF dual-hosted git repository.
nic pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kylin.git
commit 8d2a53fcd7711a843b3f34eae5953a8bfff3d3be
Author: Zhong, Yanghong
AuthorDate: Tue Mar 10 15:29:11 2020 +0800
KYLIN-4414 bulkload needs to follow locality
---
.../kylin/storage/hbase/steps/CubeHFileJob.java| 15 ++-
.../storage/hbase/steps/HFileOutputFormat3.java| 126 -
2 files changed, 131 insertions(+), 10 deletions(-)
diff --git
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/CubeHFileJob.java
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/CubeHFileJob.java
index c0fae42..e403c20 100644
---
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/CubeHFileJob.java
+++
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/CubeHFileJob.java
@@ -18,8 +18,11 @@
package org.apache.kylin.storage.hbase.steps;
+import static org.apache.hadoop.hbase.HBaseConfiguration.merge;
+
import java.io.IOException;
import java.util.Collection;
+import java.util.Locale;
import org.apache.commons.cli.Options;
import org.apache.hadoop.conf.Configuration;
@@ -47,8 +50,6 @@ import org.apache.kylin.storage.hbase.HBaseConnection;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-import static org.apache.hadoop.hbase.HBaseConfiguration.merge;
-
/**
* @author George Song (ysong1)
*/
@@ -77,7 +78,7 @@ public class CubeHFileJob extends AbstractHadoopJob {
CubeInstance cube = cubeMgr.getCube(cubeName);
-// use current hbase configuration
+// construct configuration for the MR job cluster
Configuration configuration = new
Configuration(HBaseConnection.getCurrentHBaseConfiguration());
String[] allServices = getAllServices(configuration);
merge(configuration, getConf());
@@ -95,10 +96,14 @@ public class CubeHFileJob extends AbstractHadoopJob {
// add metadata to distributed cache
attachCubeMetadata(cube, job.getConfiguration());
-HTable htable = new HTable(configuration,
getOptionValue(OPTION_HTABLE_NAME));
+// construct configuration for the HBase cluster
+Configuration hbaseConf =
HBaseConnection.getCurrentHBaseConfiguration();
+HTable htable = new HTable(hbaseConf,
getOptionValue(OPTION_HTABLE_NAME).toUpperCase(Locale.ROOT));
// Automatic config !
HFileOutputFormat3.configureIncrementalLoad(job, htable);
+HFileOutputFormat3.configureHConnection(job, hbaseConf,
getJobTempDir());
+
reconfigurePartitions(configuration, partitionFilePath);
job.setInputFormatClass(SequenceFileInputFormat.class);
@@ -109,7 +114,7 @@ public class CubeHFileJob extends AbstractHadoopJob {
job.setSortComparatorClass(RowKeyWritable.RowKeyComparator.class);
// set block replication to 3 for hfiles
-configuration.set(DFSConfigKeys.DFS_REPLICATION_KEY, "3");
+job.getConfiguration().set(DFSConfigKeys.DFS_REPLICATION_KEY, "3");
this.deletePath(job.getConfiguration(), output);
diff --git
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HFileOutputFormat3.java
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HFileOutputFormat3.java
index 12c30ea..e14d012 100644
---
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HFileOutputFormat3.java
+++
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HFileOutputFormat3.java
@@ -17,8 +17,11 @@
*/
package org.apache.kylin.storage.hbase.steps;
+import java.io.File;
+import java.io.FileOutputStream;
import java.io.IOException;
import java.io.UnsupportedEncodingException;
+import java.net.InetSocketAddress;
import java.net.URLDecoder;
import java.net.URLEncoder;
import java.nio.charset.StandardCharsets;
@@ -39,15 +42,20 @@ import org.apache.hadoop.hbase.Cell;
import org.apache.hadoop.hbase.CellUtil;
import org.apache.hadoop.hbase.HColumnDescriptor;
import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.HRegionLocation;
import org.apache.hadoop.hbase.HTableDescriptor;
import org.apache.hadoop.hbase.KeyValue;
import org.apache.hadoop.hbase.KeyValueUtil;
+import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.classification.InterfaceAudience;
import org.apache.hadoop.hbase.classification.InterfaceStability;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
import org.apache.hadoop.hbase.client.HTable;
import org.apache.hadoop.hbase.client.Put;
import org.apache.hadoop.hbase.client.RegionLocator;
import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.fs.HFileSystem;
import org.apache.hadoop.hbase.io.ImmutableBy
[kylin] 06/07: KYLIN-4417 Use hash rather than random to avoid potential issue in ConvergeCuboidDataPartitioner
This is an automated email from the ASF dual-hosted git repository.
nic pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kylin.git
commit beb976a0ef3a3a6c1d488d4e3beeefaef1f545ce
Author: Zhong, Yanghong
AuthorDate: Tue Mar 10 16:42:49 2020 +0800
KYLIN-4417 Use hash rather than random to avoid potential issue in
ConvergeCuboidDataPartitioner
---
.../engine/mr/steps/ConvergeCuboidDataPartitioner.java | 17 -
1 file changed, 12 insertions(+), 5 deletions(-)
diff --git
a/engine-mr/src/main/java/org/apache/kylin/engine/mr/steps/ConvergeCuboidDataPartitioner.java
b/engine-mr/src/main/java/org/apache/kylin/engine/mr/steps/ConvergeCuboidDataPartitioner.java
index 605905a..3a31318 100644
---
a/engine-mr/src/main/java/org/apache/kylin/engine/mr/steps/ConvergeCuboidDataPartitioner.java
+++
b/engine-mr/src/main/java/org/apache/kylin/engine/mr/steps/ConvergeCuboidDataPartitioner.java
@@ -18,8 +18,6 @@
package org.apache.kylin.engine.mr.steps;
-import java.util.Random;
-
import org.apache.hadoop.conf.Configurable;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.io.Text;
@@ -28,10 +26,12 @@ import org.apache.kylin.cube.common.RowKeySplitter;
import org.apache.kylin.engine.mr.common.BatchConstants;
import com.google.common.base.Preconditions;
+import com.google.common.hash.HashFunction;
+import com.google.common.hash.Hashing;
public class ConvergeCuboidDataPartitioner extends Partitioner
implements Configurable {
-private Random rand = new Random();
+private static final HashFunction hashFunc = Hashing.murmur3_128();
private Configuration conf;
private boolean enableSharding;
@@ -40,12 +40,14 @@ public class ConvergeCuboidDataPartitioner extends
Partitioner imple
@Override
public int getPartition(Text key, Text value, int numReduceTasks) {
+long hash = hashFunc.hashBytes(key.getBytes()).asLong();
+
long cuboidID = RowKeySplitter.getCuboidId(key.getBytes(),
enableSharding);
// the first numReduceBaseCuboid are for base cuboid
if (cuboidID == baseCuboidID) {
-return rand.nextInt(numReduceBaseCuboid);
+return getRemainder(hash, numReduceBaseCuboid);
} else {
-return numReduceBaseCuboid + rand.nextInt(numReduceTasks -
numReduceBaseCuboid);
+return numReduceBaseCuboid + getRemainder(hash, numReduceTasks -
numReduceBaseCuboid);
}
}
@@ -64,4 +66,9 @@ public class ConvergeCuboidDataPartitioner extends
Partitioner imple
public Configuration getConf() {
return conf;
}
+
+private static int getRemainder(long val, int base) {
+int rem = (int) val % base;
+return rem >= 0 ? rem : rem + base;
+}
}
[kylin] branch master updated (a174194 -> 17da7e5)
This is an automated email from the ASF dual-hosted git repository. nic pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/kylin.git. from a174194 KYLIN-4407 fix IT add 17da7e5 KYLIN-4490 Fix minor bug which add measure on frontend No new revisions were added by this update. Summary of changes: webapp/app/partials/cubeDesigner/measures.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[kylin] 13/16: [maven-release-plugin] prepare release kylin-3.0.1
This is an automated email from the ASF dual-hosted git repository. nic pushed a commit to branch 3.0.x-hadoop3.1 in repository https://gitbox.apache.org/repos/asf/kylin.git commit c1cd85b329b2bc12e97b4510b94d56c3564b6c2b Author: nichunen AuthorDate: Mon Feb 10 13:23:03 2020 + [maven-release-plugin] prepare release kylin-3.0.1 --- assembly/pom.xml | 2 +- cache/pom.xml | 2 +- core-common/pom.xml| 2 +- core-cube/pom.xml | 2 +- core-dictionary/pom.xml| 2 +- core-job/pom.xml | 2 +- core-metadata/pom.xml | 2 +- core-metrics/pom.xml | 2 +- core-storage/pom.xml | 2 +- datasource-sdk/pom.xml | 2 +- engine-mr/pom.xml | 2 +- engine-spark/pom.xml | 2 +- jdbc/pom.xml | 2 +- kylin-it/pom.xml | 2 +- metrics-reporter-hive/pom.xml | 2 +- metrics-reporter-kafka/pom.xml | 2 +- pom.xml| 4 ++-- query/pom.xml | 2 +- server-base/pom.xml| 2 +- server/pom.xml | 2 +- source-hive/pom.xml| 2 +- source-jdbc/pom.xml| 2 +- source-kafka/pom.xml | 2 +- storage-hbase/pom.xml | 2 +- storage-stream/pom.xml | 2 +- stream-coordinator/pom.xml | 2 +- stream-core/pom.xml| 2 +- stream-receiver/pom.xml| 2 +- stream-source-kafka/pom.xml| 2 +- tomcat-ext/pom.xml | 2 +- tool-assembly/pom.xml | 2 +- tool/pom.xml | 2 +- 32 files changed, 33 insertions(+), 33 deletions(-) diff --git a/assembly/pom.xml b/assembly/pom.xml index 81ead33..4d9a957 100644 --- a/assembly/pom.xml +++ b/assembly/pom.xml @@ -26,7 +26,7 @@ kylin org.apache.kylin -3.0.1-SNAPSHOT +3.0.1 diff --git a/cache/pom.xml b/cache/pom.xml index a80c144..d59d7eb 100644 --- a/cache/pom.xml +++ b/cache/pom.xml @@ -26,7 +26,7 @@ org.apache.kylin kylin -3.0.1-SNAPSHOT +3.0.1 diff --git a/core-common/pom.xml b/core-common/pom.xml index aae0925..1eea967 100644 --- a/core-common/pom.xml +++ b/core-common/pom.xml @@ -28,7 +28,7 @@ org.apache.kylin kylin -3.0.1-SNAPSHOT +3.0.1 diff --git a/core-cube/pom.xml b/core-cube/pom.xml index f284e7a..804726c 100644 --- a/core-cube/pom.xml +++ b/core-cube/pom.xml @@ -28,7 +28,7 @@ org.apache.kylin kylin -3.0.1-SNAPSHOT +3.0.1 diff --git a/core-dictionary/pom.xml b/core-dictionary/pom.xml index f976e99..be3f94a 100644 --- a/core-dictionary/pom.xml +++ b/core-dictionary/pom.xml @@ -28,7 +28,7 @@ org.apache.kylin kylin -3.0.1-SNAPSHOT +3.0.1 diff --git a/core-job/pom.xml b/core-job/pom.xml index 5a63629..16c3369 100644 --- a/core-job/pom.xml +++ b/core-job/pom.xml @@ -28,7 +28,7 @@ org.apache.kylin kylin -3.0.1-SNAPSHOT +3.0.1 diff --git a/core-metadata/pom.xml b/core-metadata/pom.xml index 5c2b827..3b4fb5d 100644 --- a/core-metadata/pom.xml +++ b/core-metadata/pom.xml @@ -28,7 +28,7 @@ org.apache.kylin kylin -3.0.1-SNAPSHOT +3.0.1 diff --git a/core-metrics/pom.xml b/core-metrics/pom.xml index 746d37b..74abd9e 100644 --- a/core-metrics/pom.xml +++ b/core-metrics/pom.xml @@ -28,7 +28,7 @@ kylin org.apache.kylin -3.0.1-SNAPSHOT +3.0.1 diff --git a/core-storage/pom.xml b/core-storage/pom.xml index 0045235..1b83c72 100644 --- a/core-storage/pom.xml +++ b/core-storage/pom.xml @@ -28,7 +28,7 @@ org.apache.kylin kylin -3.0.1-SNAPSHOT +3.0.1 diff --git a/datasource-sdk/pom.xml b/datasource-sdk/pom.xml index 65e535d..47a4ca0 100644 --- a/datasource-sdk/pom.xml +++ b/datasource-sdk/pom.xml @@ -26,7 +26,7 @@ org.apache.kylin kylin -3.0.1-SNAPSHOT +3.0.1 diff --git a/engine-mr/pom.xml b/engine-mr/pom.xml index 7ce14f6..be7dd24 100644 --- a/engine-mr/pom.xml +++ b/engine-mr/pom.xml @@ -28,7 +28,7 @@ org.apache.kylin kylin -3.0.1-SNAPSHOT +3.0.1 diff --git a/engine-spark/pom.xml b/engine-spark/pom.xml index a0b0600..5cb3cac 100644 --- a/engine-spark/pom.xml +++ b/engine-spark/pom.xml @@ -28,7 +28,7 @@ org.apache.kylin kylin -3.0.1-SNAPSHOT +3.0.1 diff --git a/jdbc/pom.xml b/jdbc/pom.xml index 881d39b..2900a27 100644 --- a/jdbc/pom.xml +++ b/jdbc/pom.xml @@ -28,7 +28,7 @@ org.apache.kylin kylin -3.0.1-SNAPSHOT +3.0.1 diff --git a/kylin-it/pom.xml b/kylin-it/pom.xml index ee4e7b8..bfcd49d 100644
[kylin] 11/16: fix kylin jdbc driver guava class not found problem
This is an automated email from the ASF dual-hosted git repository.
nic pushed a commit to branch 3.0.x-hadoop3.1
in repository https://gitbox.apache.org/repos/asf/kylin.git
commit 6c36c78cee6f3409a01d56fd2c1a2e4af8cd80ed
Author: rupengwang
AuthorDate: Tue Dec 10 21:07:15 2019 +0800
fix kylin jdbc driver guava class not found problem
---
pom.xml | 1 -
1 file changed, 1 deletion(-)
diff --git a/pom.xml b/pom.xml
index 8257d4f..f79a731 100644
--- a/pom.xml
+++ b/pom.xml
@@ -572,7 +572,6 @@
com.google.guava
guava
${guava.version}
-provided
com.jcraft
[kylin] 04/16: Rebase master
This is an automated email from the ASF dual-hosted git repository.
nic pushed a commit to branch 3.0.x-hadoop3.1
in repository https://gitbox.apache.org/repos/asf/kylin.git
commit c8fc9f9ff600490cab1710ce89d91ae8a46f7cc9
Author: nichunen
AuthorDate: Tue Jun 18 21:56:07 2019 +0800
Rebase master
---
pom.xml| 53 ++
server-base/pom.xml| 5 --
.../apache/kylin/rest/job/KylinHealthCheckJob.java | 11 +++--
.../org/apache/kylin/rest/service/CubeService.java | 6 +--
.../kylin/storage/hbase/util/StorageCleanUtil.java | 13 +++---
5 files changed, 52 insertions(+), 36 deletions(-)
diff --git a/pom.xml b/pom.xml
index aadb233..bb82b02 100644
--- a/pom.xml
+++ b/pom.xml
@@ -16,14 +16,15 @@
See the License for the specific language governing permissions and
limitations under the License.
-->
-http://maven.apache.org/POM/4.0.0;
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/maven-v4_0_0.xsd;>
+http://www.w3.org/2001/XMLSchema-instance;
xmlns="http://maven.apache.org/POM/4.0.0;
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/maven-v4_0_0.xsd;>
4.0.0
org.apache
apache
19
-
+
@@ -123,6 +124,7 @@
9.3.22.v20171030
0.3.1
2.7.14
+1.9.5
1.7.0
@@ -130,6 +132,7 @@
1.5
1.4.0
1.18
+1.4
1.2.17
@@ -401,6 +404,33 @@
${project.version}
test-jar
+
+
+org.apache.kylin
+kylin-stream-core
+${project.version}
+
+
+org.apache.kylin
+kylin-stream-coordinator
+${project.version}
+
+
+org.apache.kylin
+kylin-stream-source-kafka
+${project.version}
+
+
+org.apache.kylin
+kylin-storage-stream
+${project.version}
+
+
+org.apache.kylin
+kylin-stream-receiver
+${project.version}
+
+
org.apache.kylin
kylin-datasource-sdk
@@ -904,12 +934,6 @@
opensaml
${opensaml.version}
-
-org.apache.curator
-curator-test
-2.12.0
-test
-
@@ -973,17 +997,15 @@
org.eclipse.jetty
jetty-server
${jetty.version}
-test
org.eclipse.jetty
-jetty-webapp
+jetty-servlet
${jetty.version}
-test
org.eclipse.jetty
-jetty-util
+jetty-webapp
${jetty.version}
test
@@ -1023,11 +1045,6 @@
scala-reflect
${scala.version}
-
-org.scala-lang
-scala-reflect
-${scala.version}
-
org.apache.curator
@@ -1269,7 +1286,7 @@
-
+
diff --git a/server-base/pom.xml b/server-base/pom.xml
index cc09aa8..49e33a5 100644
--- a/server-base/pom.xml
+++ b/server-base/pom.xml
@@ -248,11 +248,6 @@
jetty-webapp
test
-
-org.eclipse.jetty
-jetty-util
-test
-
junit
diff --git
a/server-base/src/main/java/org/apache/kylin/rest/job/KylinHealthCheckJob.java
b/server-base/src/main/java/org/apache/kylin/rest/job/KylinHealthCheckJob.java
index 0e25117..8b92941 100644
---
a/server-base/src/main/java/org/apache/kylin/rest/job/KylinHealthCheckJob.java
+++
b/server-base/src/main/java/org/apache/kylin/rest/job/KylinHealthCheckJob.java
@@ -24,8 +24,9 @@ import org.apache.commons.cli.OptionBuilder;
import org.apache.commons.cli.Options;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
-import org.apache.hadoop.hbase.HBaseConfiguration;
-import org.apache.hadoop.hbase.client.HBaseAdmin;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoo
[kylin] 16/16: [maven-release-plugin] prepare release kylin-3.0.2
This is an automated email from the ASF dual-hosted git repository. nic pushed a commit to branch 3.0.x-hadoop3.1 in repository https://gitbox.apache.org/repos/asf/kylin.git commit 4490cc2fa2fcb4ef51f7ed4b1d69ada76354171b Author: nichunen AuthorDate: Sat May 9 03:20:27 2020 + [maven-release-plugin] prepare release kylin-3.0.2 --- assembly/pom.xml | 2 +- cache/pom.xml | 2 +- core-common/pom.xml| 2 +- core-cube/pom.xml | 2 +- core-dictionary/pom.xml| 2 +- core-job/pom.xml | 2 +- core-metadata/pom.xml | 2 +- core-metrics/pom.xml | 2 +- core-storage/pom.xml | 2 +- datasource-sdk/pom.xml | 2 +- engine-mr/pom.xml | 2 +- engine-spark/pom.xml | 2 +- jdbc/pom.xml | 2 +- kylin-it/pom.xml | 2 +- metrics-reporter-hive/pom.xml | 2 +- metrics-reporter-kafka/pom.xml | 2 +- pom.xml| 4 ++-- query/pom.xml | 2 +- server-base/pom.xml| 2 +- server/pom.xml | 2 +- source-hive/pom.xml| 2 +- source-jdbc/pom.xml| 2 +- source-kafka/pom.xml | 2 +- storage-hbase/pom.xml | 2 +- storage-stream/pom.xml | 2 +- stream-coordinator/pom.xml | 2 +- stream-core/pom.xml| 2 +- stream-receiver/pom.xml| 2 +- stream-source-kafka/pom.xml| 2 +- tomcat-ext/pom.xml | 2 +- tool-assembly/pom.xml | 2 +- tool/pom.xml | 2 +- 32 files changed, 33 insertions(+), 33 deletions(-) diff --git a/assembly/pom.xml b/assembly/pom.xml index 4d9a957..abccf23 100644 --- a/assembly/pom.xml +++ b/assembly/pom.xml @@ -26,7 +26,7 @@ kylin org.apache.kylin -3.0.1 +3.0.2 diff --git a/cache/pom.xml b/cache/pom.xml index d59d7eb..af0f7e2 100644 --- a/cache/pom.xml +++ b/cache/pom.xml @@ -26,7 +26,7 @@ org.apache.kylin kylin -3.0.1 +3.0.2 diff --git a/core-common/pom.xml b/core-common/pom.xml index 1eea967..405c958 100644 --- a/core-common/pom.xml +++ b/core-common/pom.xml @@ -28,7 +28,7 @@ org.apache.kylin kylin -3.0.1 +3.0.2 diff --git a/core-cube/pom.xml b/core-cube/pom.xml index 804726c..2e3cf17 100644 --- a/core-cube/pom.xml +++ b/core-cube/pom.xml @@ -28,7 +28,7 @@ org.apache.kylin kylin -3.0.1 +3.0.2 diff --git a/core-dictionary/pom.xml b/core-dictionary/pom.xml index be3f94a..975ce41 100644 --- a/core-dictionary/pom.xml +++ b/core-dictionary/pom.xml @@ -28,7 +28,7 @@ org.apache.kylin kylin -3.0.1 +3.0.2 diff --git a/core-job/pom.xml b/core-job/pom.xml index 16c3369..bf2baff 100644 --- a/core-job/pom.xml +++ b/core-job/pom.xml @@ -28,7 +28,7 @@ org.apache.kylin kylin -3.0.1 +3.0.2 diff --git a/core-metadata/pom.xml b/core-metadata/pom.xml index 3b4fb5d..898a83c 100644 --- a/core-metadata/pom.xml +++ b/core-metadata/pom.xml @@ -28,7 +28,7 @@ org.apache.kylin kylin -3.0.1 +3.0.2 diff --git a/core-metrics/pom.xml b/core-metrics/pom.xml index 74abd9e..0c9747e 100644 --- a/core-metrics/pom.xml +++ b/core-metrics/pom.xml @@ -28,7 +28,7 @@ kylin org.apache.kylin -3.0.1 +3.0.2 diff --git a/core-storage/pom.xml b/core-storage/pom.xml index 1b83c72..7c00df3 100644 --- a/core-storage/pom.xml +++ b/core-storage/pom.xml @@ -28,7 +28,7 @@ org.apache.kylin kylin -3.0.1 +3.0.2 diff --git a/datasource-sdk/pom.xml b/datasource-sdk/pom.xml index 47a4ca0..7528660 100644 --- a/datasource-sdk/pom.xml +++ b/datasource-sdk/pom.xml @@ -26,7 +26,7 @@ org.apache.kylin kylin -3.0.1 +3.0.2 diff --git a/engine-mr/pom.xml b/engine-mr/pom.xml index be7dd24..0ae91c4 100644 --- a/engine-mr/pom.xml +++ b/engine-mr/pom.xml @@ -28,7 +28,7 @@ org.apache.kylin kylin -3.0.1 +3.0.2 diff --git a/engine-spark/pom.xml b/engine-spark/pom.xml index 5cb3cac..b737eba 100644 --- a/engine-spark/pom.xml +++ b/engine-spark/pom.xml @@ -28,7 +28,7 @@ org.apache.kylin kylin -3.0.1 +3.0.2 diff --git a/jdbc/pom.xml b/jdbc/pom.xml index 2900a27..e115fdb 100644 --- a/jdbc/pom.xml +++ b/jdbc/pom.xml @@ -28,7 +28,7 @@ org.apache.kylin kylin -3.0.1 +3.0.2 diff --git a/kylin-it/pom.xml b/kylin-it/pom.xml index bfcd49d..5aafce8 100644 --- a/kylin-it/pom.xml +++ b/kylin-it/pom.xml @@ -29,7 +29,7 @@ kylin org.apache.kylin
[kylin] 07/16: remove htable toUpperCase, for HBase tableName is case sensitive
This is an automated email from the ASF dual-hosted git repository.
nic pushed a commit to branch 3.0.x-hadoop3.1
in repository https://gitbox.apache.org/repos/asf/kylin.git
commit 038b9f6267ea1b366d84de863968c82e7c8c5584
Author: liuzx32
AuthorDate: Wed Jun 26 16:27:26 2019 +0800
remove htable toUpperCase, for HBase tableName is case sensitive
---
.../main/java/org/apache/kylin/storage/hbase/steps/CubeHFileJob.java | 4 +---
1 file changed, 1 insertion(+), 3 deletions(-)
diff --git
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/CubeHFileJob.java
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/CubeHFileJob.java
index e0ecc35..c6ec255 100644
---
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/CubeHFileJob.java
+++
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/CubeHFileJob.java
@@ -53,8 +53,6 @@ import org.apache.kylin.storage.hbase.HBaseConnection;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-import java.util.Locale;
-
import static org.apache.hadoop.hbase.HBaseConfiguration.merge;
/**
@@ -106,7 +104,7 @@ public class CubeHFileJob extends AbstractHadoopJob {
Configuration hbaseConf = HBaseConfiguration.create(getConf());
-String hTableName =
getOptionValue(OPTION_HTABLE_NAME).toUpperCase(Locale.ROOT);
+String hTableName = getOptionValue(OPTION_HTABLE_NAME);
connection = ConnectionFactory.createConnection(hbaseConf);
Table table = connection.getTable(TableName.valueOf(hTableName));
RegionLocator regionLocator =
connection.getRegionLocator(TableName.valueOf(hTableName));
[kylin] branch 3.0.x-hadoop3.1 updated (9287da5 -> 4490cc2)
This is an automated email from the ASF dual-hosted git repository.
nic pushed a change to branch 3.0.x-hadoop3.1
in repository https://gitbox.apache.org/repos/asf/kylin.git.
discard 9287da5 [maven-release-plugin] prepare release kylin-3.0.2
discard 1f2c818 Upgrade tomcat vertion to 8.5.51
discard 4b39914 Add keys
discard 37de34e [maven-release-plugin] prepare release kylin-3.0.1
discard fe53b76 Backport HBASE-22887 to Kylin HFileOutputFormat3
discard 1200a5b fix kylin jdbc driver guava class not found problem
discard 5c03c40 KYLIN-3973: InvalidProtocolBufferException: Protocol message
was too large.
discard 37736ba Package kafka and curator
discard 91a95f9 KYLIN-4086 Upgrade avatica to 1.12
discard a54f6c6 remove htable toUpperCase, for HBase tableName is case
sensitive
discard 4a1153a KYLIN-4072 find-hbase-dependency.sh don't work at CDH 6.x
discard 6e64b84 fix add jacoco.version in pom.xml (#699)
discard 583940f Rebase master
discard b7a353b fix beeline meta data parser on partition information
discard 5104404 KYLIN-3537
discard 5fefca5 KYLIN-2565 upgrade to hadoop 3.0 hbase 2.0, pass UT
add 90f0946 KYLIN-4298 KYLIN-4299
add 918b8a1 Fix for KYLIN-4206
add 04acd09 KYLIN-4340 Fix bug of get value of isSparkFactDistinctEnable
for cube not correct
add ee15808 KYLIN-4353 Add regular check for cube state convert.
add 8cf0a73 KYLIN-4372 Fix bug of delete file too later cause ZK started
by HBase crash
add 543f4de KYLIN-4383 Fix reflection error
add 31cca88 KYLIN-4251 Add livy to docker
add 9fcd39f Minor, change resource URL in Dockerfile
add e96075a KYLIN-4272: optimize docker build scripts
add c474815 KYLIN-4388 Refine the Dockerfile
new 7ede04e KYLIN-2565 upgrade to hadoop 3.0 hbase 2.0, pass UT
new 76daa6f KYLIN-3537
new 595ca19 fix beeline meta data parser on partition information
new c8fc9f9 Rebase master
new 84484b7 fix add jacoco.version in pom.xml (#699)
new 610e382 KYLIN-4072 find-hbase-dependency.sh don't work at CDH 6.x
new 038b9f6 remove htable toUpperCase, for HBase tableName is case
sensitive
new ab61383 KYLIN-4086 Upgrade avatica to 1.12
new 9c81ae9 Package kafka and curator
new 2036e15 KYLIN-3973: InvalidProtocolBufferException: Protocol message
was too large.
new 6c36c78 fix kylin jdbc driver guava class not found problem
new 745834d Backport HBASE-22887 to Kylin HFileOutputFormat3
new c1cd85b [maven-release-plugin] prepare release kylin-3.0.1
new 7ca7ce3 Add keys
new ce55f54 Upgrade tomcat vertion to 8.5.51
new 4490cc2 [maven-release-plugin] prepare release kylin-3.0.2
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (9287da5)
\
N -- N -- N refs/heads/3.0.x-hadoop3.1 (4490cc2)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 16 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../org/apache/kylin/common/KylinConfigBase.java | 33 ++---
.../org/apache/kylin/common/KylinConfigExt.java| 2 +
.../src/main/resources/kylin-defaults.properties | 4 +-
docker/Dockerfile | 103 +++
docker/Dockerfile_dev | 43 ++
docker/{Dockerfile => Dockerfile_hadoop} | 27 ++--
docker/README.md | 145 +
docker/build_image.sh | 22 +---
docker/entrypoint.sh | 19 ++-
docker/run_container.sh| 3 +-
docker/use-guide.md| 136 ---
.../engine/spark/SparkBatchCubingJobBuilder2.java | 3 +-
metrics-reporter-hive/pom.xml | 5 +-
.../kylin/metrics/lib/impl/hive/HiveProducer.java | 38 +++---
.../apache/kylin/source/hive/CLIHiveClient.java| 24 ++--
.../source/hive/HiveMetaStoreClientFactory.java| 62 +
.../coordinate/ReceiverClusterManager.java | 6 +-
.../core/storage/StreamingSegmentManager.java | 2 +-
.../kylin/stream/server/StreamingServer.java | 9 ++
19 files changed, 380 insertions(+),
[kylin] 06/16: KYLIN-4072 find-hbase-dependency.sh don't work at CDH 6.x
This is an automated email from the ASF dual-hosted git repository.
nic pushed a commit to branch 3.0.x-hadoop3.1
in repository https://gitbox.apache.org/repos/asf/kylin.git
commit 610e38211e69448ec770cf0b1d4de0c230c64d7c
Author: XiaoxiangYu
AuthorDate: Wed Jul 17 10:30:22 2019 +0800
KYLIN-4072 find-hbase-dependency.sh don't work at CDH 6.x
---
build/bin/find-hbase-dependency.sh | 14 --
build/bin/find-hive-dependency.sh | 2 +-
2 files changed, 13 insertions(+), 3 deletions(-)
diff --git a/build/bin/find-hbase-dependency.sh
b/build/bin/find-hbase-dependency.sh
index f8f0f1d..299b2d7 100755
--- a/build/bin/find-hbase-dependency.sh
+++ b/build/bin/find-hbase-dependency.sh
@@ -23,6 +23,9 @@ echo Retrieving hbase dependency...
hbase_classpath=`hbase classpath`
+hadoop version | head -1 | grep cdh6
+is_cdh6=$?
+
# special handling for Amazon EMR, to prevent re-init of hbase-setenv
is_aws=`uname -r | grep amzn`
if [ -n "$is_aws" ] && [ -d "/usr/lib/oozie/lib" ]; then
@@ -42,10 +45,17 @@ done
if [ -z "$hbase_common_path" ]
then
-quit "hbase-common lib not found"
+if [[ $is_cdh6 -ne 0 ]]; then
+ quit "hbase-common lib not found"
+fi
+fi
+
+if [[ $is_cdh6 -eq 0 ]]; then
+hbase_dependency=${hbase_classpath}
+else
+hbase_dependency=${hbase_common_path}
fi
-hbase_dependency=${hbase_common_path}
verbose "hbase dependency: $hbase_dependency"
export hbase_dependency
echo "export HBASE_ENV_INIT=$HBASE_ENV_INIT
diff --git a/build/bin/find-hive-dependency.sh
b/build/bin/find-hive-dependency.sh
index 06e2a11..80c68d7 100755
--- a/build/bin/find-hive-dependency.sh
+++ b/build/bin/find-hive-dependency.sh
@@ -37,7 +37,7 @@ else
hive_env=`hive ${hive_conf_properties} -e set 2>&1 | grep 'env:CLASSPATH'`
fi
-if [ -z $hive_env ]
+if [ -z "${hive_env}" ]
then
hive_permission=`hive ${hive_conf_properties} -e set 2>&1 | grep 'No valid
credentials provided'`
if [ -n "$hive_permission" ]
[kylin] 05/16: fix add jacoco.version in pom.xml (#699)
This is an automated email from the ASF dual-hosted git repository.
nic pushed a commit to branch 3.0.x-hadoop3.1
in repository https://gitbox.apache.org/repos/asf/kylin.git
commit 84484b7c96ed3af3756a56f41220862ddf0ae1ec
Author: liuzx32
AuthorDate: Mon Jun 24 14:56:39 2019 +0800
fix add jacoco.version in pom.xml (#699)
* fix add jacoco.version in pom.xml
---
pom.xml | 1 +
1 file changed, 1 insertion(+)
diff --git a/pom.xml b/pom.xml
index bb82b02..6debb06 100644
--- a/pom.xml
+++ b/pom.xml
@@ -165,6 +165,7 @@
2.3
+0.8.0
jacoco
reuseReports
${project.basedir}/../target/jacoco.exec
[kylin] 15/16: Upgrade tomcat vertion to 8.5.51
This is an automated email from the ASF dual-hosted git repository.
nic pushed a commit to branch 3.0.x-hadoop3.1
in repository https://gitbox.apache.org/repos/asf/kylin.git
commit ce55f542b0545266071a0e756eda8b3bdccfb8b5
Author: rupengwang
AuthorDate: Sat Feb 29 13:12:21 2020 +0800
Upgrade tomcat vertion to 8.5.51
---
build/script/download-tomcat.sh | 8
pom.xml | 2 +-
server/src/main/java/org/apache/kylin/rest/DebugTomcat.java | 2 +-
.../java/org/apache/kylin/tool/extractor/HBaseUsageExtractor.java | 2 +-
4 files changed, 7 insertions(+), 7 deletions(-)
diff --git a/build/script/download-tomcat.sh b/build/script/download-tomcat.sh
index 0f580e9..ae562a3 100755
--- a/build/script/download-tomcat.sh
+++ b/build/script/download-tomcat.sh
@@ -27,19 +27,19 @@ if [[ `uname -a` =~ "Darwin" ]]; then
alias md5cmd="md5 -q"
fi
-tomcat_pkg_version="7.0.100"
-tomcat_pkg_md5="79be4ba5a6e770730a4be3d5cb3c7862"
+tomcat_pkg_version="8.5.51"
+tomcat_pkg_md5="be0e56e47b2b3d56a2fec527d3e14ee1"
if [ ! -f "build/apache-tomcat-${tomcat_pkg_version}.tar.gz" ]
then
echo "no binary file found"
-wget --directory-prefix=build/
http://archive.apache.org/dist/tomcat/tomcat-7/v${tomcat_pkg_version}/bin/apache-tomcat-${tomcat_pkg_version}.tar.gz
|| echo "Download tomcat failed"
+wget --directory-prefix=build/
http://archive.apache.org/dist/tomcat/tomcat-8/v${tomcat_pkg_version}/bin/apache-tomcat-${tomcat_pkg_version}.tar.gz
|| echo "Download tomcat failed"
else
if [ `md5cmd build/apache-tomcat-${tomcat_pkg_version}.tar.gz | awk
'{print $1}'` != "${tomcat_pkg_md5}" ]
then
echo "md5 check failed"
rm build/apache-tomcat-${tomcat_pkg_version}.tar.gz
-wget --directory-prefix=build/
http://archive.apache.org/dist/tomcat/tomcat-7/v${tomcat_pkg_version}/bin/apache-tomcat-${tomcat_pkg_version}.tar.gz
|| echo "download tomcat failed"
+wget --directory-prefix=build/
http://archive.apache.org/dist/tomcat/tomcat-8/v${tomcat_pkg_version}/bin/apache-tomcat-${tomcat_pkg_version}.tar.gz
|| echo "download tomcat failed"
fi
fi
unalias md5cmd
diff --git a/pom.xml b/pom.xml
index 902cbd4..859a056 100644
--- a/pom.xml
+++ b/pom.xml
@@ -146,7 +146,7 @@
3.2.4
2.4.0
2.5
-7.0.100
+8.5.51
3.1
2.3.23
5.9.2
diff --git a/server/src/main/java/org/apache/kylin/rest/DebugTomcat.java
b/server/src/main/java/org/apache/kylin/rest/DebugTomcat.java
index a238a8f..96698f9 100644
--- a/server/src/main/java/org/apache/kylin/rest/DebugTomcat.java
+++ b/server/src/main/java/org/apache/kylin/rest/DebugTomcat.java
@@ -21,7 +21,7 @@ package org.apache.kylin.rest;
import org.apache.catalina.Context;
import org.apache.catalina.core.AprLifecycleListener;
import org.apache.catalina.core.StandardServer;
-import org.apache.catalina.deploy.ErrorPage;
+import org.apache.tomcat.util.descriptor.web.ErrorPage;
import org.apache.catalina.startup.Tomcat;
import org.apache.commons.io.FileUtils;
import org.apache.commons.lang3.StringUtils;
diff --git
a/tool/src/main/java/org/apache/kylin/tool/extractor/HBaseUsageExtractor.java
b/tool/src/main/java/org/apache/kylin/tool/extractor/HBaseUsageExtractor.java
index 72129c3..f576339 100644
---
a/tool/src/main/java/org/apache/kylin/tool/extractor/HBaseUsageExtractor.java
+++
b/tool/src/main/java/org/apache/kylin/tool/extractor/HBaseUsageExtractor.java
@@ -90,7 +90,7 @@ public class HBaseUsageExtractor extends
AbstractInfoExtractor {
extractor.execute(args);
}
-private String getHBaseMasterUrl() throws IOException, KeeperException {
+private String getHBaseMasterUrl() {
String host = conf.get("hbase.master.info.bindAddress");
if (host.equals("0.0.0.0")) {
try {
[kylin] 12/16: Backport HBASE-22887 to Kylin HFileOutputFormat3
This is an automated email from the ASF dual-hosted git repository.
nic pushed a commit to branch 3.0.x-hadoop3.1
in repository https://gitbox.apache.org/repos/asf/kylin.git
commit 745834d1a4f5b746cb5ef9df332ebc4192d7e80a
Author: langdamao
AuthorDate: Fri Dec 13 14:13:02 2019 +0800
Backport HBASE-22887 to Kylin HFileOutputFormat3
Signed-off-by: langdamao
---
.../kylin/storage/hbase/steps/CreateHTableJob.java | 3 +-
.../kylin/storage/hbase/steps/CubeHFileJob.java| 3 +-
.../storage/hbase/steps/HFileOutputFormat3.java| 794 ++---
3 files changed, 537 insertions(+), 263 deletions(-)
diff --git
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/CreateHTableJob.java
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/CreateHTableJob.java
index 3bc6cfb..85933d0 100644
---
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/CreateHTableJob.java
+++
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/CreateHTableJob.java
@@ -38,7 +38,6 @@ import org.apache.hadoop.hbase.KeyValueUtil;
import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.client.Connection;
import org.apache.hadoop.hbase.client.HTable;
-import org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2;
import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.io.SequenceFile;
import org.apache.hadoop.mapreduce.Job;
@@ -136,7 +135,7 @@ public class CreateHTableJob extends AbstractHadoopJob {
Connection conn = HBaseConnection.get(kylinConfig.getStorageUrl());
HTable htable = (HTable)
conn.getTable(TableName.valueOf(hbaseTableName));
-HFileOutputFormat2.configureIncrementalLoadMap(job,
htable.getDescriptor());
+HFileOutputFormat3.configureIncrementalLoadMap(job,
htable.getDescriptor());
logger.info("Saving HBase configuration to {}", hbaseConfPath);
FileSystem fs = HadoopUtil.getWorkingFileSystem();
diff --git
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/CubeHFileJob.java
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/CubeHFileJob.java
index c6ec255..6490ff7 100644
---
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/CubeHFileJob.java
+++
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/CubeHFileJob.java
@@ -33,7 +33,6 @@ import org.apache.hadoop.hbase.client.ConnectionFactory;
import org.apache.hadoop.hbase.client.RegionLocator;
import org.apache.hadoop.hbase.client.Table;
import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
-import org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2;
import org.apache.hadoop.hdfs.DFSConfigKeys;
import org.apache.hadoop.io.SequenceFile;
import org.apache.hadoop.io.Writable;
@@ -109,7 +108,7 @@ public class CubeHFileJob extends AbstractHadoopJob {
Table table = connection.getTable(TableName.valueOf(hTableName));
RegionLocator regionLocator =
connection.getRegionLocator(TableName.valueOf(hTableName));
// Automatic config !
-HFileOutputFormat2.configureIncrementalLoad(job, table,
regionLocator);
+HFileOutputFormat3.configureIncrementalLoad(job, table,
regionLocator);
reconfigurePartitions(hbaseConf, partitionFilePath);
job.setInputFormatClass(SequenceFileInputFormat.class);
diff --git
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HFileOutputFormat3.java
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HFileOutputFormat3.java
index 12c30ea..1ec5887 100644
---
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HFileOutputFormat3.java
+++
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HFileOutputFormat3.java
@@ -14,60 +14,76 @@
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
-*/
+ */
+
package org.apache.kylin.storage.hbase.steps;
+import static
org.apache.hadoop.hbase.regionserver.HStoreFile.BULKLOAD_TASK_KEY;
+import static
org.apache.hadoop.hbase.regionserver.HStoreFile.BULKLOAD_TIME_KEY;
+import static
org.apache.hadoop.hbase.regionserver.HStoreFile.EXCLUDE_FROM_MINOR_COMPACTION_KEY;
+import static
org.apache.hadoop.hbase.regionserver.HStoreFile.MAJOR_COMPACTION_KEY;
+
import java.io.IOException;
import java.io.UnsupportedEncodingException;
+import java.net.InetSocketAddress;
import java.net.URLDecoder;
import java.net.URLEncoder;
-import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.Arrays;
-import java.util.Collection;
import java.util.List;
import java.util.Map;
+import java.util.Set;
import java.util.TreeMap;
import java.util.TreeSet;
+import java.util.function.Function;
+import java.util.stream.Collectors;
-import org.apache.commons.logging.Lo
[kylin] 02/16: KYLIN-3537
This is an automated email from the ASF dual-hosted git repository.
nic pushed a commit to branch 3.0.x-hadoop3.1
in repository https://gitbox.apache.org/repos/asf/kylin.git
commit 76daa6f0f442f03b1acc2aeb55df349544c76bcd
Author: alexandr.sidorchuk
AuthorDate: Thu Feb 21 15:26:55 2019 +0300
KYLIN-3537
Use Spark to build Cube on Yarn faild at Setp8 on HDP3.0
add HBase dependicies to avoid java.lang.NoClassDefFoundError
Signed-off-by: shaofengshi
---
.../apache/kylin/storage/hbase/steps/HBaseSparkSteps.java | 14 ++
1 file changed, 14 insertions(+)
diff --git
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HBaseSparkSteps.java
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HBaseSparkSteps.java
index d636e7d..91a1206 100644
---
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HBaseSparkSteps.java
+++
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HBaseSparkSteps.java
@@ -73,6 +73,20 @@ public class HBaseSparkSteps extends HBaseJobSteps {
StringUtil.appendWithSeparator(jars,
ClassUtil.findContainingJar("org.apache.hadoop.hbase.regionserver.MetricsRegionServerSourceFactoryImpl",
null));//hbase-hadoop2-compat-1.1.1.jar
+//KYLIN-3537
+StringUtil.appendWithSeparator(jars,
+
ClassUtil.findContainingJar("org.apache.hadoop.hbase.io.hfile.HFileWriterImpl",
null));//hbase-server.jar
+StringUtil.appendWithSeparator(jars,
+
ClassUtil.findContainingJar("org.apache.hbase.thirdparty.com.google.common.cache.CacheLoader",
null));//hbase-shaded-miscellaneous.jar
+StringUtil.appendWithSeparator(jars,
+
ClassUtil.findContainingJar("org.apache.hadoop.hbase.metrics.MetricRegistry",
null));//hbase-metrics-api.jar
+StringUtil.appendWithSeparator(jars,
+
ClassUtil.findContainingJar("org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl",
null));//hbase-metrics.jar
+StringUtil.appendWithSeparator(jars,
+
ClassUtil.findContainingJar("org.apache.hbase.thirdparty.com.google.protobuf.Message",
null));//hbase-shaded-protobuf.jar
+StringUtil.appendWithSeparator(jars,
+
ClassUtil.findContainingJar("org.apache.hadoop.hbase.shaded.protobuf.generated.HFileProtos",
null));//hbase-protocol-shaded.jar
+
StringUtil.appendWithSeparator(jars,
seg.getConfig().getSparkAdditionalJars());
sparkExecutable.setJars(jars.toString());
[kylin] 03/16: fix beeline meta data parser on partition information
This is an automated email from the ASF dual-hosted git repository.
nic pushed a commit to branch 3.0.x-hadoop3.1
in repository https://gitbox.apache.org/repos/asf/kylin.git
commit 595ca1983a7389cdcf6525c6df2bf4c12e1f85dc
Author: sickcate
AuthorDate: Tue Apr 23 17:08:14 2019 +0800
fix beeline meta data parser on partition information
---
.../apache/kylin/source/hive/BeelineHiveClient.java | 19 ++-
1 file changed, 10 insertions(+), 9 deletions(-)
diff --git
a/source-hive/src/main/java/org/apache/kylin/source/hive/BeelineHiveClient.java
b/source-hive/src/main/java/org/apache/kylin/source/hive/BeelineHiveClient.java
index 29538aa..333e281 100644
---
a/source-hive/src/main/java/org/apache/kylin/source/hive/BeelineHiveClient.java
+++
b/source-hive/src/main/java/org/apache/kylin/source/hive/BeelineHiveClient.java
@@ -213,20 +213,21 @@ public class BeelineHiveClient implements IHiveClient {
}
}
-private void parseResultEntry(ResultSet resultSet, HiveTableMetaBuilder
builder) throws SQLException {
+private void parseResultEntry(ResultSet resultSet, HiveTableMetaBuilder
builder) throws SQLException{
List partitionColumns =
Lists.newArrayList();
if ("# Partition Information".equals(resultSet.getString(1).trim())) {
resultSet.next();
Preconditions.checkArgument("#
col_name".equals(resultSet.getString(1).trim()));
resultSet.next();
-
Preconditions.checkArgument("".equals(resultSet.getString(1).trim()));
-while (resultSet.next()) {
-if ("".equals(resultSet.getString(1).trim())) {
-break;
-}
-partitionColumns.add(new
HiveTableMeta.HiveTableColumnMeta(resultSet.getString(1).trim(),
-resultSet.getString(2).trim(),
resultSet.getString(3).trim()));
-}
+if ("".equals(resultSet.getString(1).trim()))
+resultSet.next();
+do {
+ if ("".equals(resultSet.getString(1).trim())) {
+ break;
+ }
+ partitionColumns.add(new
HiveTableMeta.HiveTableColumnMeta(resultSet.getString(1).trim(),
+ resultSet.getString(2).trim(),
resultSet.getString(3).trim()));
+} while (resultSet.next());
builder.setPartitionColumns(partitionColumns);
}
[kylin] 09/16: Package kafka and curator
This is an automated email from the ASF dual-hosted git repository.
nic pushed a commit to branch 3.0.x-hadoop3.1
in repository https://gitbox.apache.org/repos/asf/kylin.git
commit 9c81ae9b5256c548603e6b0e553f1fefc50ceed3
Author: nichunen
AuthorDate: Tue Jul 23 18:09:44 2019 +0800
Package kafka and curator
---
pom.xml | 22 ++
1 file changed, 18 insertions(+), 4 deletions(-)
diff --git a/pom.xml b/pom.xml
index 6debb06..8257d4f 100644
--- a/pom.xml
+++ b/pom.xml
@@ -533,19 +533,34 @@
org.apache.curator
curator-framework
${curator.version}
-provided
+
+
+org.apache.zookeeper
+zookeeper
+
+
org.apache.curator
curator-recipes
${curator.version}
-provided
+
+
+org.apache.zookeeper
+zookeeper
+
+
org.apache.curator
curator-client
${curator.version}
-provided
+
+
+org.apache.zookeeper
+zookeeper
+
+
com.google.code.findbugs
@@ -770,7 +785,6 @@
org.apache.kafka
kafka_2.11
${kafka.version}
-provided
