[GitHub] [incubator-kyuubi] yaooqinn opened a new issue #779: Remove obsolete CODE_OF_CONDUCT.md
yaooqinn opened a new issue #779: URL: https://github.com/apache/incubator-kyuubi/issues/779 ### 1. Describe the feature We shall switch to use https://www.apache.org/foundation/policies/conduct.html now ### 2. Motivation ### 3. Describe the solution ### 4. Additional context -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn opened a new pull request #778: [BUILD] Enable .asf.yaml features
yaooqinn opened a new pull request #778: URL: https://github.com/apache/incubator-kyuubi/pull/778 ### _Why are the changes needed?_ Follow https://cwiki.apache.org/confluence/display/INFRA/git+-+.asf.yaml+features to enable `.asf.yaml` with some Github features - desc - homepage - tags/labels - merge buttons - squash only - set master as protected, which need at least 1 approval before getting merged - enabling Github issuses - notifications w/ mailing lists ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on pull request #778: [BUILD] Enable .asf.yaml features
yaooqinn commented on pull request #778: URL: https://github.com/apache/incubator-kyuubi/pull/778#issuecomment-878418791 cc @WillemJiang @Apache9 @aajisaka @zjffdu @pan3793 @turboFei @ulysses-you @jhx1008 thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] zhaomin1423 commented on pull request #758: [KYUUBI 661] Add UDF system_user
zhaomin1423 commented on pull request #758: URL: https://github.com/apache/incubator-kyuubi/pull/758#issuecomment-878371360 @ulysses-you Should the system_user return the UserGroupInformation.getCurrentUser? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] Yikun commented on a change in pull request #776: Run kyuubi on kubernetes - make basic dockerfile and example
Yikun commented on a change in pull request #776:
URL: https://github.com/apache/incubator-kyuubi/pull/776#discussion_r667992957
##
File path: docker/Dockerfile
##
@@ -0,0 +1,61 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# Usage:
+# docker build -f docker/Dockerfile -t yaooqinn/kyuubi:tagname .
+# Options:
+# -f this docker file
+# -t the target repo and tag name
+# more options can be found with -h
+# TODO: REPALCE it with offical spark image iff Apache Spark community deploy
Review comment:
learned, lol
##
File path: docker/Dockerfile
##
@@ -0,0 +1,61 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# Usage:
+# docker build -f docker/Dockerfile -t yaooqinn/kyuubi:tagname .
+# Options:
+# -f this docker file
+# -t the target repo and tag name
+# more options can be found with -h
+# TODO: REPALCE it with offical spark image iff Apache Spark community deploy
+# Or make one after kyuubi to be setup under ASF
+
+ARG SPARK_IMAGE=yaooqinn/spark:3.0.3
+FROM ${SPARK_IMAGE}
+
+ARG kyuubi_uid=10009
+
+USER root
+
+ENV KYUUBI_HOME /opt/kyuubi
+ENV KYUUBI_LOG_DIR ${KYUUBI_HOME}/logs
+ENV KYUUBI_PID_DIR ${KYUUBI_HOME}/pid
+ENV KYUUBI_WORK_DIR_ROOT ${KYUUBI_HOME}/work
+
+RUN set -ex && \
+sed -i 's/http:\/\/deb.\(.*\)/https:\/\/deb.\1/g' /etc/apt/sources.list &&
\
+apt-get update && \
+apt install -y bash tini libc6 libpam-modules krb5-user libnss3 procps && \
+useradd -u ${kyuubi_uid} -g root kyuubi && \
+mkdir ${KYUUBI_HOME} && \
+mkdir ${KYUUBI_LOG_DIR} && \
+mkdir ${KYUUBI_PID_DIR} && \
+mkdir ${KYUUBI_WORK_DIR_ROOT} && \
+chmod ug+rw -R ${KYUUBI_HOME} && \
+chmod a+rwx -R ${KYUUBI_WORK_DIR_ROOT} && \
+rm -rf /var/cache/apt/*
+
+COPY bin ${KYUUBI_HOME}/bin
+COPY conf ${KYUUBI_HOME}/conf
+COPY jars ${KYUUBI_HOME}/jars
+COPY externals/engines/spark ${KYUUBI_HOME}/externals/engines/spark
+
+WORKDIR ${KYUUBI_HOME}
+
+CMD [ "/opt/kyuubi/bin/kyuubi", "run" ]
+
+USER ${kyuubi_uid}
Review comment:
OK, my bad understand on
https://docs.docker.com/engine/reference/builder/#user. So, looks like CMD is
regared as "for any RUN, CMD and ENTRYPOINT instructions that **follow** it in
the Dockerfile"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on a change in pull request #778: [BUILD] Enable .asf.yaml features
pan3793 commented on a change in pull request #778: URL: https://github.com/apache/incubator-kyuubi/pull/778#discussion_r668088689 ## File path: .asf.yaml ## @@ -0,0 +1,51 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +# See: https://cwiki.apache.org/confluence/display/INFRA/git+-+.asf.yaml+features + +github: + description: "Apache Kyuubi is a distributed multi-tenant JDBC server for large-scale data processing and analytics, built on top of Apache Spark" + homepage: https://kyuubi.apache.org/ + labels: +- spark +- jdbc +- thrift +- data-lake +- spark-sql +- hadoop +- hive +- sql +- kubernetes + enabled_merge_buttons: +squash: true +merge: false +rebase: false + protected_branches: +master: Review comment: This strategy may break our merge script -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] ulysses-you closed pull request #777: Refactor Kyuubi kubernetes block cleaner
ulysses-you closed pull request #777: URL: https://github.com/apache/incubator-kyuubi/pull/777 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] ulysses-you opened a new pull request #777: Refactor Kyuubi kubernetes block cleaner
ulysses-you opened a new pull request #777: URL: https://github.com/apache/incubator-kyuubi/pull/777 ### _Why are the changes needed?_ * fix bug as a blockMgr dir is created with no file at the begining. * simplify code * add test ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on a change in pull request #777: Refactor Kyuubi kubernetes block cleaner
yaooqinn commented on a change in pull request #777:
URL: https://github.com/apache/incubator-kyuubi/pull/777#discussion_r668371348
##
File path:
tools/kubernetes/spark-block-cleaner/src/test/scala/org.apache.kyuubi.tools/KubernetesSparkBlockCleanerTest.scala
##
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.tools
+
+import java.io.File
+import java.nio.file.{Files, Paths}
+import java.util.UUID
+
+import org.apache.kyuubi.KyuubiFunSuite
+
+class KubernetesSparkBlockCleanerTest extends KyuubiFunSuite {
Review comment:
KubernetesSparkBlockCleanerSuite for better regex matching
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on a change in pull request #777: Refactor Kyuubi kubernetes block cleaner
yaooqinn commented on a change in pull request #777:
URL: https://github.com/apache/incubator-kyuubi/pull/777#discussion_r668371688
##
File path:
tools/kubernetes/spark-block-cleaner/src/test/scala/org.apache.kyuubi.tools/KubernetesSparkBlockCleanerTest.scala
##
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.tools
+
+import java.io.File
+import java.nio.file.{Files, Paths}
+import java.util.UUID
+
+import org.apache.kyuubi.KyuubiFunSuite
+
+class KubernetesSparkBlockCleanerTest extends KyuubiFunSuite {
+ import KubernetesSparkBlockCleanerConstants._
+
+ private val rootDir = System.getProperty("java.io.tmpdir")
Review comment:
can we use `Utils.createXXX`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on a change in pull request #777: Refactor Kyuubi kubernetes block cleaner
yaooqinn commented on a change in pull request #777:
URL: https://github.com/apache/incubator-kyuubi/pull/777#discussion_r668374585
##
File path:
tools/kubernetes/spark-block-cleaner/src/main/scala/org/apache/kyuubi/tools/KubernetesSparkBlockCleaner.scala
##
@@ -54,153 +56,142 @@ object KubernetesSparkBlockCleaner extends Logging {
private val cacheDirs = if (envMap.containsKey(CACHE_DIRS_KEY)) {
envMap.get(CACHE_DIRS_KEY).split(",").filter(!_.equals(""))
} else {
-throw new IllegalArgumentException(s"the env ${CACHE_DIRS_KEY} can not be
null")
+throw new IllegalArgumentException(s"the env $CACHE_DIRS_KEY must be set")
}
+ private val isTesting = envMap.getOrDefault("kyuubi.testing",
"false").toBoolean
+ checkConfiguration()
+
+ /**
+ * one thread clean one dir
+ */
+ private val threadPool = Executors.newFixedThreadPool(cacheDirs.length)
+
+ private def checkConfiguration(): Unit = {
+require(fileExpiredTime > 0,
+ s"the env $FILE_EXPIRED_TIME_KEY should be greater than 0")
+require(deepCleanFileExpiredTime > 0,
+ s"the env $DEEP_CLEAN_FILE_EXPIRED_TIME_KEY should be greater than 0")
+require(sleepTime > 0,
+ s"the env $SLEEP_TIME_KEY should be greater than 0")
+require(freeSpaceThreshold > 0 && freeSpaceThreshold < 100,
+ s"the env $FREE_SPACE_THRESHOLD_KEY should between 0 and 100")
Review comment:
can we make these `kvs` more human-readable?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [incubator-kyuubi] timothy65535 opened a new pull request #780: [KYUUBI 770] Fix flaky test: Add config to control if cancel invoke interrupt task on engine
timothy65535 opened a new pull request #780:
URL: https://github.com/apache/incubator-kyuubi/pull/780
### _Why are the changes needed?_
Fix flaky test: Add config to control if cancel invoke interrupt task on
engine.
```
The code passed to eventually never returned normally. Attempted 1783 times
over 30.01599386 seconds. Last failure message: 0 did not equal 1.
ScalaTestFailureLocation: org.apache.kyuubi.engine.spark.SparkEngineSuites
at (IndividualSparkSuite.scala:62)
```
**ROOT CAUSE**
When the operating system is under pressure at a certain point in time and
the `forceCancel` is true.
The statement `assert(System.currentTimeMillis() -
taskEnd.taskInfo.launchTime < 3000)` will throw Exception, it
cause the following statement can not be executed.
```
override def onTaskEnd(taskEnd: SparkListenerTaskEnd): Unit = {
assert(taskEnd.reason.isInstanceOf[TaskKilled])
if (forceCancel.get()) {
assert(System.currentTimeMillis() - taskEnd.taskInfo.launchTime < 3000)
index.incrementAndGet()
} else {
assert(System.currentTimeMillis() - taskEnd.taskInfo.launchTime >= 4000)
index.incrementAndGet()
}
}
```
### _How was this patch tested?_
- [X] Add some test cases that check the changes thoroughly including
negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run
test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests)
locally before make a pull request
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on pull request #780: [KYUUBI 770] Fix flaky test: Add config to control if cancel invoke interrupt task on engine
yaooqinn commented on pull request #780: URL: https://github.com/apache/incubator-kyuubi/pull/780#issuecomment-878723786 cc @ulysses-you -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn opened a new issue #781: Fix status badges in README to point the current repo
yaooqinn opened a new issue #781:
URL: https://github.com/apache/incubator-kyuubi/issues/781
### 1. Describe the bug
[](https://github.com/NetEase/kyuubi/releases)
[](https://codecov.io/gh/NetEase/kyuubi)
[](https://travis-ci.com/NetEase/kyuubi)
NetEase -> apache
kyuubi -> incubator-kyuubi
### 2. Environments
2.1 Versions
- Kyuubi
- [ ] 1.1.0
- [ ] 1.0.3
- [ ] 0.8.x and earlier
- Spark
- [ ] 3.1.x
- [ ] 3.0.x
- [ ] 2.4.x and earlier
2.2 Kyuubi Configurations
2.3 Spark Configurations
### 3. Steps to reproduce
3.1 Operations
3.2 Expected behavior
3.3 Actual behavior
### 4. Additional context
4.1 Log of the server
4.2 Log of the corresponding engine
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on a change in pull request #778: [BUILD] Enable .asf.yaml features
yaooqinn commented on a change in pull request #778: URL: https://github.com/apache/incubator-kyuubi/pull/778#discussion_r668387169 ## File path: .asf.yaml ## @@ -0,0 +1,51 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +# See: https://cwiki.apache.org/confluence/display/INFRA/git+-+.asf.yaml+features + +github: + description: "Apache Kyuubi is a distributed multi-tenant JDBC server for large-scale data processing and analytics, built on top of Apache Spark" + homepage: https://kyuubi.apache.org/ + labels: +- spark +- jdbc +- thrift +- data-lake +- spark-sql +- hadoop +- hive +- sql +- kubernetes + enabled_merge_buttons: +squash: true +merge: false +rebase: false + protected_branches: +master: Review comment: we can try first -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn opened a new issue #782: SparkProcessBuilder fails to generate SPARK_HOME after codebase migration
yaooqinn opened a new issue #782:
URL: https://github.com/apache/incubator-kyuubi/issues/782
### 1. Describe the bug
It causes the Github action and Travis CI fail
### 2. Environments
2.1 Versions
- Kyuubi
- [ ] 1.1.0
- [ ] 1.0.3
- [ ] 0.8.x and earlier
- Spark
- [ ] 3.1.x
- [ ] 3.0.x
- [ ] 2.4.x and earlier
2.2 Kyuubi Configurations
2.3 Spark Configurations
### 3. Steps to reproduce
3.1 Operations
3.2 Expected behavior
3.3 Actual behavior
### 4. Additional context
4.1 Log of the server
4.2 Log of the corresponding engine
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [incubator-kyuubi] ulysses-you opened a new pull request #783: [KYUUBI #782] SparkProcessBuilder fails to generate SPARK_HOME after codebase migration
ulysses-you opened a new pull request #783: URL: https://github.com/apache/incubator-kyuubi/pull/783 ### _Why are the changes needed?_ Fix failed test ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on pull request #780: [KYUUBI 770] Fix flaky test: Add config to control if cancel invoke interrupt task on engine
yaooqinn commented on pull request #780: URL: https://github.com/apache/incubator-kyuubi/pull/780#issuecomment-878749331 please wait for https://github.com/apache/incubator-kyuubi/pull/783 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn closed issue #782: SparkProcessBuilder fails to generate SPARK_HOME after codebase migration
yaooqinn closed issue #782: URL: https://github.com/apache/incubator-kyuubi/issues/782 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn closed pull request #783: [KYUUBI #782] SparkProcessBuilder fails to generate SPARK_HOME after codebase migration
yaooqinn closed pull request #783: URL: https://github.com/apache/incubator-kyuubi/pull/783 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on pull request #783: [KYUUBI #782] SparkProcessBuilder fails to generate SPARK_HOME after codebase migration
yaooqinn commented on pull request #783: URL: https://github.com/apache/incubator-kyuubi/pull/783#issuecomment-878757566 thanks, merged to master -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on pull request #780: [KYUUBI 770] Fix flaky test: Add config to control if cancel invoke interrupt task on engine
yaooqinn commented on pull request #780: URL: https://github.com/apache/incubator-kyuubi/pull/780#issuecomment-878761605 #783 merged, please trigger ci again thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on pull request #767: Event Tracking: For statement
yaooqinn commented on pull request #767: URL: https://github.com/apache/incubator-kyuubi/pull/767#issuecomment-878762753 How about add some unit tests? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn edited a comment on pull request #767: Event Tracking: For statement
yaooqinn edited a comment on pull request #767: URL: https://github.com/apache/incubator-kyuubi/pull/767#issuecomment-878762753 How about adding some unit tests? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on a change in pull request #767: Event Tracking: For statement
yaooqinn commented on a change in pull request #767: URL: https://github.com/apache/incubator-kyuubi/pull/767#discussion_r668413526 ## File path: externals/kyuubi-spark-monitor/src/main/scala/org/apache/kyuubi/engine/spark/monitor/entity/KyuubiStatementInfo.scala ## @@ -0,0 +1,48 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.kyuubi.engine.spark.monitor.entity + +import scala.collection.mutable.Map + +import org.apache.spark.sql.execution.QueryExecution + +import org.apache.kyuubi.KyuubiSQLException +import org.apache.kyuubi.operation.OperationState.OperationState + +/** + * This object store the summary infomation about statement. + * You can use statementId to get all jobs' or stages' metric that this statement has. + * + * @param statementId + * @param statement + * @param appId + * @param sessionId + * @param physicalPlan: contains physicalPlan, logicPlan and so on + * @param sparkUser + * @param exception + * @param stateTimeMap: store this statement's every state and the time of occurrence + */ +case class KyuubiStatementInfo( +statementId: String, +statement: String, +appId: String, +sessionId: String, +var physicalPlan: QueryExecution, Review comment: var queryExecution: QueryExecutio -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on a change in pull request #767: Event Tracking: For statement
yaooqinn commented on a change in pull request #767: URL: https://github.com/apache/incubator-kyuubi/pull/767#discussion_r668413699 ## File path: externals/kyuubi-spark-monitor/src/main/scala/org/apache/kyuubi/engine/spark/monitor/entity/KyuubiStatementInfo.scala ## @@ -0,0 +1,48 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.kyuubi.engine.spark.monitor.entity + +import scala.collection.mutable.Map + +import org.apache.spark.sql.execution.QueryExecution + +import org.apache.kyuubi.KyuubiSQLException +import org.apache.kyuubi.operation.OperationState.OperationState + +/** + * This object store the summary infomation about statement. + * You can use statementId to get all jobs' or stages' metric that this statement has. + * + * @param statementId + * @param statement + * @param appId + * @param sessionId + * @param physicalPlan: contains physicalPlan, logicPlan and so on + * @param sparkUser + * @param exception + * @param stateTimeMap: store this statement's every state and the time of occurrence + */ +case class KyuubiStatementInfo( +statementId: String, +statement: String, +appId: String, +sessionId: String, +var physicalPlan: QueryExecution, +sparkUser: String, Review comment: remove this, we can manage this in session info -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on a change in pull request #767: Event Tracking: For statement
yaooqinn commented on a change in pull request #767: URL: https://github.com/apache/incubator-kyuubi/pull/767#discussion_r668413957 ## File path: externals/kyuubi-spark-monitor/src/main/scala/org/apache/kyuubi/engine/spark/monitor/entity/KyuubiStatementInfo.scala ## @@ -0,0 +1,48 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.kyuubi.engine.spark.monitor.entity + +import scala.collection.mutable.Map + +import org.apache.spark.sql.execution.QueryExecution + +import org.apache.kyuubi.KyuubiSQLException +import org.apache.kyuubi.operation.OperationState.OperationState + +/** + * This object store the summary infomation about statement. + * You can use statementId to get all jobs' or stages' metric that this statement has. + * + * @param statementId + * @param statement + * @param appId + * @param sessionId + * @param physicalPlan: contains physicalPlan, logicPlan and so on + * @param sparkUser + * @param exception + * @param stateTimeMap: store this statement's every state and the time of occurrence + */ +case class KyuubiStatementInfo( +statementId: String, +statement: String, +appId: String, +sessionId: String, +var physicalPlan: QueryExecution, +sparkUser: String, +var exception: KyuubiSQLException, +stateTimeMap: Map[OperationState, Long]) Review comment: stateToTime -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on a change in pull request #767: Event Tracking: For statement
yaooqinn commented on a change in pull request #767:
URL: https://github.com/apache/incubator-kyuubi/pull/767#discussion_r668414546
##
File path:
externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/operation/ExecuteStatement.scala
##
@@ -156,6 +167,16 @@ class ExecuteStatement(
spark.sparkContext.removeSparkListener(operationListener)
super.cleanup(targetState)
}
+
+ override def setState(newState: OperationState): Unit = {
+super.setState(newState)
+ kyuubiStatementInfo.stateTimeMap.put(newState, lastAccessTime)
Review comment:
remove ` `
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on a change in pull request #767: Event Tracking: For statement
yaooqinn commented on a change in pull request #767: URL: https://github.com/apache/incubator-kyuubi/pull/767#discussion_r668415174 ## File path: externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/operation/ExecuteStatement.scala ## @@ -58,6 +62,12 @@ class ExecuteStatement( private val operationListener: SQLOperationListener = new SQLOperationListener(this, spark) + private val kyuubiStatementInfo = KyuubiStatementInfo( +statementId, statement, spark.sparkContext.applicationId, +session.getTypeInfo.identifier.toString, null, +spark.sparkContext.sparkUser, null, Map(state -> lastAccessTime)) + KyuubiStatementMonitor.addStatementDetailForOperationId(statementId, kyuubiStatementInfo) Review comment: could this lead to memory leak? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on a change in pull request #778: [BUILD] Enable .asf.yaml features
pan3793 commented on a change in pull request #778: URL: https://github.com/apache/incubator-kyuubi/pull/778#discussion_r668415183 ## File path: .asf.yaml ## @@ -0,0 +1,54 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +# See: https://cwiki.apache.org/confluence/display/INFRA/git+-+.asf.yaml+features + +github: + description: "Apache Kyuubi is a distributed multi-tenant JDBC server for large-scale data processing and analytics, built on top of Apache Spark" Review comment: I'm not sure if the **Apache Kyuubi** without **(Incubating)** meets the Apache trademark requirements. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on a change in pull request #778: [BUILD] Enable .asf.yaml features
yaooqinn commented on a change in pull request #778: URL: https://github.com/apache/incubator-kyuubi/pull/778#discussion_r668416649 ## File path: .asf.yaml ## @@ -0,0 +1,54 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +# See: https://cwiki.apache.org/confluence/display/INFRA/git+-+.asf.yaml+features + +github: + description: "Apache Kyuubi is a distributed multi-tenant JDBC server for large-scale data processing and analytics, built on top of Apache Spark" Review comment: These rules are generalized to describe ASF software associated with the trademark "Apache ProjectName", or more generally "ProjectName" when it is understood to refer to our specific Apache ProjectName software product. Like most ASF software, this ProjectName software is maintained by the Apache ProjectName project, or by the ProjectName sub-project of another project, such as the "Apache Incubator" (itself an ASF trademark). https://incubator.apache.org/projects/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on a change in pull request #778: [BUILD] Enable .asf.yaml features
yaooqinn commented on a change in pull request #778: URL: https://github.com/apache/incubator-kyuubi/pull/778#discussion_r668416649 ## File path: .asf.yaml ## @@ -0,0 +1,54 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +# See: https://cwiki.apache.org/confluence/display/INFRA/git+-+.asf.yaml+features + +github: + description: "Apache Kyuubi is a distributed multi-tenant JDBC server for large-scale data processing and analytics, built on top of Apache Spark" Review comment: These rules are generalized to describe ASF software associated with the trademark "Apache ProjectName", or more generally "ProjectName" when it is understood to refer to our specific Apache ProjectName software product. Like most ASF software, this ProjectName software is maintained by the Apache ProjectName project, or by the ProjectName sub-project of another project, such as the "Apache Incubator" (itself an ASF trademark). https://incubator.apache.org/projects/ https://www.apache.org/foundation/marks/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi-website] turboFei opened a new pull request #1: Enable kyuubi.staged.apache.org for staging preview
turboFei opened a new pull request #1: URL: https://github.com/apache/incubator-kyuubi-website/pull/1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi-website] turboFei merged pull request #1: Enable kyuubi.staged.apache.org for staging preview
turboFei merged pull request #1: URL: https://github.com/apache/incubator-kyuubi-website/pull/1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn closed pull request #778: [BUILD] Enable .asf.yaml features
yaooqinn closed pull request #778: URL: https://github.com/apache/incubator-kyuubi/pull/778 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on pull request #778: [BUILD] Enable .asf.yaml features
yaooqinn commented on pull request #778: URL: https://github.com/apache/incubator-kyuubi/pull/778#issuecomment-878785756 thanks, merged to master -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] ulysses-you commented on pull request #780: [KYUUBI 770] Fix flaky test: Add config to control if cancel invoke interrupt task on engine
ulysses-you commented on pull request #780: URL: https://github.com/apache/incubator-kyuubi/pull/780#issuecomment-878789072 hi @timothy65535 the change LGTM, please rebase on latest master to trigger test -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] cutiechi commented on issue #774: How to disable ha zk?
cutiechi commented on issue #774: URL: https://github.com/apache/incubator-kyuubi/issues/774#issuecomment-878789315 @pan3793 Ok, thx -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] cutiechi closed issue #774: How to disable ha zk?
cutiechi closed issue #774: URL: https://github.com/apache/incubator-kyuubi/issues/774 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] cutiechi commented on issue #773: How to configure spark executor pod name prefix?
cutiechi commented on issue #773: URL: https://github.com/apache/incubator-kyuubi/issues/773#issuecomment-878789490 @ulysses-you Ok, thx -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] cutiechi closed issue #773: How to configure spark executor pod name prefix?
cutiechi closed issue #773: URL: https://github.com/apache/incubator-kyuubi/issues/773 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi-website] turboFei opened a new pull request #2: Refactor Kyuubi HomePage
turboFei opened a new pull request #2: URL: https://github.com/apache/incubator-kyuubi-website/pull/2 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] timothy65535 commented on pull request #780: [KYUUBI 770] Fix flaky test: Add config to control if cancel invoke interrupt task on engine
timothy65535 commented on pull request #780: URL: https://github.com/apache/incubator-kyuubi/pull/780#issuecomment-878794668 Thanks @yaooqinn @ulysses-you -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi-website] turboFei merged pull request #2: Refactor kyuubi website home page.
turboFei merged pull request #2: URL: https://github.com/apache/incubator-kyuubi-website/pull/2 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on issue #781: Fix status badges in README to point the current repo
yaooqinn commented on issue #781: URL: https://github.com/apache/incubator-kyuubi/issues/781#issuecomment-878799359 cc @wuchunfu, are you interested in this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on pull request #777: Refactor Kyuubi kubernetes block cleaner
yaooqinn commented on pull request #777: URL: https://github.com/apache/incubator-kyuubi/pull/777#issuecomment-878800268 can you fix the ci? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on a change in pull request #767: Event Tracking: For statement
yaooqinn commented on a change in pull request #767: URL: https://github.com/apache/incubator-kyuubi/pull/767#discussion_r668449269 ## File path: externals/kyuubi-spark-monitor/src/main/scala/org/apache/kyuubi/engine/spark/monitor/entity/KyuubiStatementInfo.scala ## @@ -0,0 +1,48 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.kyuubi.engine.spark.monitor.entity + +import scala.collection.mutable.Map + +import org.apache.spark.sql.execution.QueryExecution + +import org.apache.kyuubi.KyuubiSQLException +import org.apache.kyuubi.operation.OperationState.OperationState + +/** + * This object store the summary infomation about statement. + * You can use statementId to get all jobs' or stages' metric that this statement has. + * + * @param statementId + * @param statement + * @param appId + * @param sessionId + * @param physicalPlan: contains physicalPlan, logicPlan and so on + * @param sparkUser + * @param exception + * @param stateTimeMap: store this statement's every state and the time of occurrence + */ +case class KyuubiStatementInfo( +statementId: String, +statement: String, +appId: String, +sessionId: String, Review comment: use session handle? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] wuchunfu commented on issue #781: Fix status badges in README to point the current repo
wuchunfu commented on issue #781: URL: https://github.com/apache/incubator-kyuubi/issues/781#issuecomment-878806223 @yaooqinn OK, I'll do the task. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi-website] ulysses-you opened a new pull request #3: Add *.iml to .gitignore
ulysses-you opened a new pull request #3: URL: https://github.com/apache/incubator-kyuubi-website/pull/3 Make this project friendly for idea. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on issue #779: Remove obsolete CODE_OF_CONDUCT.md
yaooqinn commented on issue #779: URL: https://github.com/apache/incubator-kyuubi/issues/779#issuecomment-878815713 cc @zhang1002 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on pull request #777: Refactor Kyuubi kubernetes block cleaner
yaooqinn commented on pull request #777: URL: https://github.com/apache/incubator-kyuubi/pull/777#issuecomment-878826788 please also update module name from `Kyuubi Tool Kubernetes Spark Block Cleaner` to `Kyuubi Project Spark Block Cleaner` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] ulysses-you opened a new pull request #784: Remove striction in .asf.yaml
ulysses-you opened a new pull request #784: URL: https://github.com/apache/incubator-kyuubi/pull/784 ### _Why are the changes needed?_ Make `dev/merge_kyuubi_pr.py` available. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] wuchunfu opened a new pull request #785: [KYUUBI #781] Fix status badges in README to point the current repo
wuchunfu opened a new pull request #785: URL: https://github.com/apache/incubator-kyuubi/pull/785 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] ulysses-you merged pull request #784: Remove striction in .asf.yaml
ulysses-you merged pull request #784: URL: https://github.com/apache/incubator-kyuubi/pull/784 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] ulysses-you commented on pull request #784: Remove striction in .asf.yaml
ulysses-you commented on pull request #784: URL: https://github.com/apache/incubator-kyuubi/pull/784#issuecomment-878837943 thanks, merging to master -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] wForget opened a new issue #786: Engine load balancing strategy
wForget opened a new issue #786: URL: https://github.com/apache/incubator-kyuubi/issues/786 ### 1. Describe the feature The last engine is acquired by default. Is there any plan to support other engine load balancing strategies? Such as load balancing according to the current number of sessions of the engine. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] ulysses-you commented on issue #786: Engine load balancing strategy
ulysses-you commented on issue #786: URL: https://github.com/apache/incubator-kyuubi/issues/786#issuecomment-878858349 Good idea, actually we have some discussion about this internal. Maybe we can have something like engine pool to provide the ability of load balancing. And about the strategies, we can support `round robin`, `session/statment-number-weight-based` or other user defined strategy which is used the interface provided by this framework. As you konw, Kyuubi is stateless in a cluster so it should be considered first how to share the engine pool status between several Kyuubi services. So we need more details and some design of this. It's also very helpful if you can share some use case of this feature. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on issue #786: Engine load balancing strategy
yaooqinn commented on issue #786: URL: https://github.com/apache/incubator-kyuubi/issues/786#issuecomment-878858464 For engine, we'd call it service discovery, not load balancing. The engine discovery logic is better to bind with client requests, a.k.a, JDBC connections, as an engine is stateful with cache tables, udf, etc. We already have implemented 3 share levels, CONNECTION/USER/SERVER for different kinds of usage scenarios. - For CONNECTION mode, the engine and all states are within a JDBC connection. - For USER mode, the engine and all states might/could share across all JDBC connections of a user with a `subDomain` if any. For `The last engine is acquired by default`, it's a potential bug since `1.2.0-rc`s as a distributed lock for engine creation. When an engine is created but killed forcefully and externally, the service uri of this engine will not get cleared. And another thread might get a malformed engine after it gains the lock. For this part, I think we need to introduce a retry and recreation of engine -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] loovia opened a new issue #787: [question] How to connect kyuubi(ha mode) by beeline
loovia opened a new issue #787: URL: https://github.com/apache/incubator-kyuubi/issues/787 I add kyuubi.ha configures in conf/kyuubi-defaults.conf ``` kyuubi.ha.enabled true kyuubi.ha.zookeeper.namespace kyuubi kyuubi.ha.zookeeper.quorum 1.zookeeper.xyz,2.zookeeper.xyz,3.zookeeper.xyz kyuubi.ha.zookeeper.client.port 2181 ``` and kyuubi has been registered in zooKeeper. when I execute `/opt/spark/spark-3.1.1-bin-hadoop2.7/bin/beeline -u jdbc:hive2://1.zookeeper.xyz:2181,2.zookeeper.xyz:2181,3.zookeeper.xyz:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNameSpace=kyuubi `, it's wrong: > Connecting to jdbc:hive2://1.zookeeper.xyz:2181,2.zookeeper.xyz:2181,3.zookeeper.xyz:2181/ 21/07/13 15:53:47 INFO Utils: Supplied authorities: 1.zookeeper.xyz:2181,2.zookeeper.xyz:2181,3.zookeeper.xyz:2181 21/07/13 15:53:47 INFO Utils: Resolved authority: 1.zookeeper.xyz:2181 21/07/13 15:53:47 WARN HiveConnection: Failed to connect to 1.zookeeper.xyz:2181 Unexpected end of file when reading from HS2 server. The root cause might be too many concurrent connections. Please ask the administrator to check the number of active connections, and adjust hive.server2.thrift.max.worker.threads if applicable. Error: Could not open client transport with JDBC Uri: jdbc:hive2://1.zookeeper.xyz:2181/: null (state=08S01,code=0) Beeline version 2.3.7 by Apache Hive beeline> I don't know How to connect kyuubi(ha mode) by beeline. Please take a look~ thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] wForget commented on issue #787: [question] How to connect the HA mode Kyuubi with Beeline
wForget commented on issue #787: URL: https://github.com/apache/incubator-kyuubi/issues/787#issuecomment-878868040 maybe you need to use quoted string, like: ``` opt/spark/spark-3.1.1-bin-hadoop2.7/bin/beeline -u "jdbc:hive2://1.zookeeper.xyz:2181,2.zookeeper.xyz:2181,3.zookeeper.xyz:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNameSpace=kyuubi" ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] wuchunfu commented on issue #779: Remove obsolete CODE_OF_CONDUCT.md
wuchunfu commented on issue #779: URL: https://github.com/apache/incubator-kyuubi/issues/779#issuecomment-878869619 @yaooqinn If @zhang1002 doesn't have time, I can do it -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn opened a new pull request #788: [BUILD] Minimize the build of travis
yaooqinn opened a new pull request #788: URL: https://github.com/apache/incubator-kyuubi/pull/788 ### _Why are the changes needed?_ the Travis ci is not very powerful, we low its workload to a test with default profiles and use it as a backup when GA is out of service sometimes. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on issue #779: Remove obsolete CODE_OF_CONDUCT.md
yaooqinn commented on issue #779: URL: https://github.com/apache/incubator-kyuubi/issues/779#issuecomment-878875983 @wuchunfu thanks~ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on pull request #785: [KYUUBI #781] Fix status badges in README to point the current repo
yaooqinn commented on pull request #785: URL: https://github.com/apache/incubator-kyuubi/pull/785#issuecomment-878882938 [](https://github.com/apache/incubator-kyuubi)  [](https://github.com/apache/incubator-kyuubi/releases) [](https://codecov.io/gh/apache/incubator-kyuubi) [](https://travis-ci.com/apache/incubator-kyuubi)  [](https://kyuubi.apache.org/?badge=latest) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] wuchunfu opened a new pull request #789: [KYUUBI #779] Remove obsolete CODE_OF_CONDUCT.md
wuchunfu opened a new pull request #789: URL: https://github.com/apache/incubator-kyuubi/pull/789 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn opened a new pull request #790: BUILD] Update PR Merge tool
yaooqinn opened a new pull request #790: URL: https://github.com/apache/incubator-kyuubi/pull/790 ### _Why are the changes needed?_ change GITHUB_API_BASE = "https://api.github.com/repos/NetEase/kyuubi"; to GITHUB_API_BASE = "https://api.github.com/repos/apache/incubator-kyuubi"; ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn merged pull request #785: [KYUUBI #781] Fix status badges in README to point the current repo
yaooqinn merged pull request #785: URL: https://github.com/apache/incubator-kyuubi/pull/785 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on pull request #789: [KYUUBI #779] Remove obsolete CODE_OF_CONDUCT.md
yaooqinn commented on pull request #789: URL: https://github.com/apache/incubator-kyuubi/pull/789#issuecomment-878920118 please remove the change in README? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] wuchunfu commented on pull request #789: [KYUUBI #779] Remove obsolete CODE_OF_CONDUCT.md
wuchunfu commented on pull request #789: URL: https://github.com/apache/incubator-kyuubi/pull/789#issuecomment-878926401 @yaooqinn I have restored the changes in README. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn merged pull request #790: [BUILD] Update PR Merge tool
yaooqinn merged pull request #790: URL: https://github.com/apache/incubator-kyuubi/pull/790 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] timothy65535 opened a new issue #791: Fix point to right repo in PULL_REQUEST_TEMPLATE
timothy65535 opened a new issue #791:
URL: https://github.com/apache/incubator-kyuubi/issues/791
### 1. Describe the bug
Fix point to right repo in PULL_REQUEST_TEMPLATE.
### 2. Environments
2.1 Versions
- Kyuubi
- [ ] 1.1.0
- [ ] 1.0.3
- [ ] 0.8.x and earlier
- Spark
- [ ] 3.1.x
- [ ] 3.0.x
- [ ] 2.4.x and earlier
2.2 Kyuubi Configurations
2.3 Spark Configurations
### 3. Steps to reproduce
3.1 Operations
3.2 Expected behavior
3.3 Actual behavior
### 4. Additional context
4.1 Log of the server
4.2 Log of the corresponding engine
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [incubator-kyuubi] timothy65535 opened a new pull request #792: [KYUUBI 791] Fix point to right repo in PULL_REQUEST_TEMPLATE
timothy65535 opened a new pull request #792: URL: https://github.com/apache/incubator-kyuubi/pull/792 ### _Why are the changes needed?_ Fix point to right repo in PULL_REQUEST_TEMPLATE. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on pull request #792: [KYUUBI 791] Fix point to right repo in PULL_REQUEST_TEMPLATE
yaooqinn commented on pull request #792: URL: https://github.com/apache/incubator-kyuubi/pull/792#issuecomment-878953773 docs/community/contributions.md can we update this too? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn merged pull request #789: [KYUUBI #779] Remove obsolete CODE_OF_CONDUCT.md
yaooqinn merged pull request #789: URL: https://github.com/apache/incubator-kyuubi/pull/789 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn opened a new pull request #793: Set protected_branches explicitly in a nonstrict way
yaooqinn opened a new pull request #793: URL: https://github.com/apache/incubator-kyuubi/pull/793 ### _Why are the changes needed?_ ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] ulysses-you merged pull request #793: Set protected_branches explicitly in a nonstrict way
ulysses-you merged pull request #793: URL: https://github.com/apache/incubator-kyuubi/pull/793 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] ulysses-you merged pull request #780: [KYUUBI 770] Fix flaky test: Add config to control if cancel invoke interrupt task on engine
ulysses-you merged pull request #780: URL: https://github.com/apache/incubator-kyuubi/pull/780 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] ulysses-you commented on pull request #780: [KYUUBI 770] Fix flaky test: Add config to control if cancel invoke interrupt task on engine
ulysses-you commented on pull request #780: URL: https://github.com/apache/incubator-kyuubi/pull/780#issuecomment-879011684 thanks, merging to master for v1.3.0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] ulysses-you closed issue #770: [TEST] Flaky: Add config to control if cancel invoke interrupt task on engine
ulysses-you closed issue #770: URL: https://github.com/apache/incubator-kyuubi/issues/770 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] ulysses-you closed pull request #777: Refactor Kyuubi kubernetes block cleaner
ulysses-you closed pull request #777: URL: https://github.com/apache/incubator-kyuubi/pull/777 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] ulysses-you opened a new pull request #794: Block cleaner 2
ulysses-you opened a new pull request #794: URL: https://github.com/apache/incubator-kyuubi/pull/794 ### _Why are the changes needed?_ * fix bug as a blockMgr dir is created with no file at the begining. * simplify code * simplify log * add test ### _How was this patch tested?_ the release directory structure  -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] ulysses-you commented on pull request #777: Refactor Kyuubi kubernetes block cleaner
ulysses-you commented on pull request #777: URL: https://github.com/apache/incubator-kyuubi/pull/777#issuecomment-879039815 see https://github.com/apache/incubator-kyuubi/pull/794 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] timothy65535 commented on pull request #792: [KYUUBI 791] Fix point to right repo in PULL_REQUEST_TEMPLATE
timothy65535 commented on pull request #792: URL: https://github.com/apache/incubator-kyuubi/pull/792#issuecomment-879079678 > docs/community/contributions.md can we update this too? Yeah. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn opened a new pull request #795: Docker2
yaooqinn opened a new pull request #795: URL: https://github.com/apache/incubator-kyuubi/pull/795 ### _Why are the changes needed?_ ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] timothy65535 commented on pull request #788: [BUILD] Minimize the build of travis
timothy65535 commented on pull request #788: URL: https://github.com/apache/incubator-kyuubi/pull/788#issuecomment-879142462 It seems that we can remove `.travis.yml` directly, all the jobs of travis have been replaced by github action. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn merged pull request #788: [BUILD] Minimize the build of travis
yaooqinn merged pull request #788: URL: https://github.com/apache/incubator-kyuubi/pull/788 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on pull request #788: [BUILD] Minimize the build of travis
yaooqinn commented on pull request #788: URL: https://github.com/apache/incubator-kyuubi/pull/788#issuecomment-879149444 > It seems that we can remove `.travis.yml` directly, all the jobs of travis have been replaced by Github action. 1. We have experienced several Github action crashes so far. With one CI work can help us unblock PRs 2. We also detect some hidden bugs because Travis's slowness. LOL. Anyway, we can make Travis faster via separating and parallelizing tests from module to module -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn merged pull request #792: [KYUUBI #791] Fix point to right repo in PULL_REQUEST_TEMPLATE
yaooqinn merged pull request #792: URL: https://github.com/apache/incubator-kyuubi/pull/792 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on pull request #792: [KYUUBI #791] Fix point to right repo in PULL_REQUEST_TEMPLATE
yaooqinn commented on pull request #792: URL: https://github.com/apache/incubator-kyuubi/pull/792#issuecomment-879153520 thanks, merged to master -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn closed pull request #776: Run kyuubi on kubernetes - make basic dockerfile and example
yaooqinn closed pull request #776: URL: https://github.com/apache/incubator-kyuubi/pull/776 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn closed issue #791: Fix point to right repo in PULL_REQUEST_TEMPLATE
yaooqinn closed issue #791: URL: https://github.com/apache/incubator-kyuubi/issues/791 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn closed issue #781: Fix status badges in README to point the current repo
yaooqinn closed issue #781: URL: https://github.com/apache/incubator-kyuubi/issues/781 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] pan3793 opened a new pull request #796: [INFRA] Format asf yaml
pan3793 opened a new pull request #796: URL: https://github.com/apache/incubator-kyuubi/pull/796 ### _Why are the changes needed?_ ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] pan3793 opened a new pull request #797: [INFRA] Disable branches protection
pan3793 opened a new pull request #797: URL: https://github.com/apache/incubator-kyuubi/pull/797 ### _Why are the changes needed?_ ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn merged pull request #797: [INFRA] Disable branches protection
yaooqinn merged pull request #797: URL: https://github.com/apache/incubator-kyuubi/pull/797 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on pull request #797: [INFRA] Disable branches protection
yaooqinn commented on pull request #797: URL: https://github.com/apache/incubator-kyuubi/pull/797#issuecomment-879534529 ``` GitHub Protected Branches has set requires signature setting on branch 'master' to 'False' With regards, ASF Infra. ``` Ideally, we shall receive a message in our private mailing list like the above from the INFRA team. But it seems that this is not triggered but will be triggered if we send another PR to disable/enable the Protected Branches -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] ulysses-you commented on pull request #794: Refactor Kyuubi kubernetes block cleaner
ulysses-you commented on pull request #794: URL: https://github.com/apache/incubator-kyuubi/pull/794#issuecomment-879534631 cc @zwangsheng @yaooqinn -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn opened a new issue #798: Make distribution with --spark-provided as default an only in release.yml of GA
yaooqinn opened a new issue #798: URL: https://github.com/apache/incubator-kyuubi/issues/798 ### 1. Describe the feature ### 2. Motivation ### 3. Describe the solution ### 4. Additional context -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-kyuubi] zwangsheng commented on a change in pull request #794: Refactor Kyuubi kubernetes block cleaner
zwangsheng commented on a change in pull request #794:
URL: https://github.com/apache/incubator-kyuubi/pull/794#discussion_r669235075
##
File path:
tools/spark-block-cleaner/src/main/scala/org/apache/kyuubi/tools/KubernetesSparkBlockCleaner.scala
##
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.tools
+
+import java.io.File
+import java.nio.file.{Files, Paths}
+import java.util.concurrent.{CountDownLatch, Executors}
+
+import org.apache.kyuubi.Logging
+
+/*
+* Spark storage shuffle data as the following structure.
+*
+* local-dir1/
+* blockmgr-uuid/
+* hash-sub-dir/
+* shuffle-data
+* shuffle-index
+*
+* local-dir2/
+* blockmgr-uuid/
+* hash-sub-dir/
+* shuffle-data
+* shuffle-index
+*
+* ...
+*/
+object KubernetesSparkBlockCleaner extends Logging {
+ import KubernetesSparkBlockCleanerConstants._
+
+ private val envMap = System.getenv()
+
+ private val freeSpaceThreshold =
envMap.getOrDefault(FREE_SPACE_THRESHOLD_KEY,
+"60").toInt
+ private val fileExpiredTime = envMap.getOrDefault(FILE_EXPIRED_TIME_KEY,
+"60480").toLong
+ private val scheduleInterval = envMap.getOrDefault(SCHEDULE_INTERNAL,
+"360").toLong
+ private val deepCleanFileExpiredTime =
envMap.getOrDefault(DEEP_CLEAN_FILE_EXPIRED_TIME_KEY,
+"43200").toLong
+ private val cacheDirs = if (envMap.containsKey(CACHE_DIRS_KEY)) {
+envMap.get(CACHE_DIRS_KEY).split(",").filter(!_.equals(""))
+ } else {
+throw new IllegalArgumentException(s"the env $CACHE_DIRS_KEY must be set")
+ }
+ private val isTesting = envMap.getOrDefault("kyuubi.testing",
"false").toBoolean
+ checkConfiguration()
+
+ /**
+ * one thread clean one dir
+ */
+ private val threadPool = Executors.newFixedThreadPool(cacheDirs.length)
+
+ private def checkConfiguration(): Unit = {
+require(fileExpiredTime > 0,
+ s"the env $FILE_EXPIRED_TIME_KEY should be greater than 0")
+require(deepCleanFileExpiredTime > 0,
+ s"the env $DEEP_CLEAN_FILE_EXPIRED_TIME_KEY should be greater than 0")
+require(scheduleInterval > 0,
+ s"the env $SCHEDULE_INTERNAL should be greater than 0")
+require(freeSpaceThreshold > 0 && freeSpaceThreshold < 100,
+ s"the env $FREE_SPACE_THRESHOLD_KEY should between 0 and 100")
+require(cacheDirs.nonEmpty, s"the env $CACHE_DIRS_KEY must be set")
+cacheDirs.foreach { dir =>
+ val path = Paths.get(dir)
+ require(Files.exists(path),
+s"the input cache dir: $dir does not exists")
+ require(Files.isDirectory(path),
+s"the input cache dir: $dir should be a directory")
+}
+
+info(s"finish initializing configuration, " +
+ s"use $CACHE_DIRS_KEY: ${cacheDirs.mkString(",")}, " +
+ s"$FILE_EXPIRED_TIME_KEY: $fileExpiredTime, " +
+ s"$FREE_SPACE_THRESHOLD_KEY: $freeSpaceThreshold, " +
+ s"$SCHEDULE_INTERNAL: $scheduleInterval, " +
+ s"$DEEP_CLEAN_FILE_EXPIRED_TIME_KEY: $deepCleanFileExpiredTime")
+ }
+
+ private def doClean(dir: File, time: Long) {
+// clean blockManager shuffle file
+
dir.listFiles.filter(_.isDirectory).filter(_.getName.startsWith("blockmgr"))
+ .foreach { blockManagerDir =>
+info(s"start check blockManager dir
${blockManagerDir.getCanonicalPath}")
+// check blockManager directory
+val released = blockManagerDir.listFiles.filter(_.isDirectory).map {
subDir =>
+ debug(s"start check sub dir ${subDir.getCanonicalPath}")
+ // check sub directory
+ val subDirReleased = subDir.listFiles.map(file =>
checkAndDeleteFile(file, time))
+ // delete empty sub directory
+ checkAndDeleteFile(subDir, time, true)
+ subDirReleased.sum
+}
+// delete empty blockManager directory
+checkAndDeleteFile(blockManagerDir, time, true)
+info(s"finished clean blockManager dir
${blockManagerDir.getCanonicalPath}, " +
+ s"released space: ${released.sum / 1024 / 1024} MB.")
+ }
+
+// clean spark cache file
+dir.listFiles.filter(_.isDirectory).filter(_.getName.startsWith("spark"))
+ .foreach { cacheDir =>
+info(s"start chec
[GitHub] [incubator-kyuubi] pan3793 closed pull request #796: [DO-NOT-MERGE] [INFRA] Format asf yaml
pan3793 closed pull request #796: URL: https://github.com/apache/incubator-kyuubi/pull/796 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
