Hey Frank, I've deployed the apache/druid:32.0.0-rc2 image on minikube with druid-operator and it has fully qualified hostnames in zk [1] So it seems to me that it works correctly - even in case there is another cluster in another namespace using the same ZK I guess that should work too.
While I was digginig into history: there was * Use canonical hostname instead of ip by default https://github.com/apache/druid/pull/16386 * which was merged by you - and it essentially made the same changes as you see right now on k8s * reverted by https://github.com/apache/druid/pull/17347 * reapplied with restriction to only affect k8s in https://github.com/apache/druid/pull/17697 To be sure to get on the same page; could you answer these questions: * you could get back the earlier behaviour by setting the environment variable `DRUID_SET_HOST_IP=1` - isn't that an option? * why is the modification of that `properties` out-of-scope - can't you remove druid.host=localhost from it ; and where its coming from? * is this deployment follows some playbook which should be supported out-of-the-box? if yes; could you point me to it? cheers, Zoltan [1] $ echo 'ls -R /' | kubectl exec -it pod/tiny-cluster-zk-0 zkCli.sh [...] /druid/internal-discovery/BROKER/druid-tiny-cluster-brokers-0.druid-tiny-cluster-brokers.druid.svc.cluster.local:8088 /druid/internal-discovery/COORDINATOR/druid-tiny-cluster-coordinators-0.druid-tiny-cluster-coordinators.druid.svc.cluster.local:8088 /druid/internal-discovery/HISTORICAL/druid-tiny-cluster-historicals-0.druid-tiny-cluster-historicals.druid.svc.cluster.local:8088 /druid/internal-discovery/OVERLORD/druid-tiny-cluster-coordinators-0.druid-tiny-cluster-coordinators.druid.svc.cluster.local:8088 /druid/internal-discovery/ROUTER/druid-tiny-cluster-routers-0.druid-tiny-cluster-routers.druid.svc.cluster.local:8088 [...] On 4/17/25 12:52, Frank Chen wrote:
In the previous release, druid.host is SET by the starting script to ip address, now, since KUBERNETES_SERVICE_HOST environment variable is detected in the K8S env, the druid.host will NOT be set, and the localhost defined in the property file will be used. I think druid_host should be allowed to be customized instead of fully controlled by our script. One case is that if a druid cluster is deployed in two K8S clusters, services in two K8S clusters can't communicate with each other by the default host name. we need to add some suffix to the host name to use CoreDNS to resolve the druid_host. On Thu, Apr 17, 2025 at 4:46 PM Zoltan Haindrich <k...@rxd.hu> wrote:Hey Frank, Thank you for checking it out - I was digging into it a bit ; but I'm not sure if I understand all aspects of your setup/expectations If you were you following a guide for k8s - could you point to it; it might help a lot to get on the same page! Let me try to condense how I think this works right now: * druid.host might be set in a properties file * depending on the values of `KUBERNETES_SERVICE_HOST` and `DRUID_SET_HOST_IP` it might be set in a script * default behaviour for k8s seems to be to not set `druid.host` to anything * if it remains unset it will be null and populated in DruidNode [1] to `InetAddress.getLocalHost().getCanonicalHostName()` I believe for a k8s deployment you should not set druid.host to anything and let `DruidNode` set its hostname. I suspect earlier it worked because it was set to the ip address on k8s as well - regardless what the config contained; I believe hostnames are much friendlier to use - so it would be great to use them. Regarding the alternatives: > 1. if the druid_host environment is not provided, use the hostname command > to set the druid.host property if it's under the K8S environment to > override the default one in the property file I see that there is a `druid_host` env [2] ; but instead of adding more custom scripts I would rather leave that to `InetAddress.getLocalHost().getCanonicalHostName()` if possible. > 2. or remove the druid.host default property from the file and always set > it via starting scripts I think you mean to remove it from conf/druid/cluster/_common/common.runtime.properties [3] ? But I feel like that might be done automatically people following: https://druid.apache.org/docs/latest/tutorials/cluster/ opened PR to unset it: https://github.com/apache/druid/pull/17930 > BTW, I think we should add this change(use the hostname under the K8S > environment) to the release note too. added a suggestion to the release notes PR relating to this. cheers, Zoltan [1] https://github.com/apache/druid/blob/3766e3a14fb49ca98ab3b02495bf1336c7eee75c/server/src/main/java/org/apache/druid/server/DruidNode.java#L168 [2] https://github.com/apache/druid/blob/3766e3a14fb49ca98ab3b02495bf1336c7eee75c/extensions-core/kubernetes-overlord-extensions/src/main/java/org/apache/druid/k8s/overlord/common/DruidK8sConstants.java#L43 [3] https://github.com/apache/druid/blob/3766e3a14fb49ca98ab3b02495bf1336c7eee75c/examples/conf/druid/cluster/_common/common.runtime.properties#L37 On 4/17/25 05:37, Frank Chen wrote:-1(binding) because https://github.com/apache/druid/pull/17697 preventsusfrom upgrading our old clusters to this release. Our clusters are running in K8S, above PR skips the setting of druid.host properties, however, this property is defined as localhost by default in the common.properties, so after upgrading, all druid service names registered in the zookeeperarelocalhost, which can't be accessed by other services. Possible solutions might be: 1. if the druid_host environment is not provided, use the hostnamecommandto set the druid.host property if it's under the K8S environment to override the default one in the property file 2. or remove the druid.host default property from the file and always set it via starting scripts BTW, I think we should add this change(use the hostname under the K8S environment) to the release note too. On Thu, Apr 17, 2025 at 8:25 AM Clint Wylie <cwy...@apache.org> wrote:+1 (binding) src package: * verified checksum and signature * checked for LICENSE and NOTICE * rat check successful * built binary distribution, ran MSQ ingestion, kafka ingestion, compaction, then some queries binary package: * verified checksum and signature * checked for LICENSE and NOTICE * ran json batch ingestion and some queries docker: * verified checksum * started cluster with docker-compose, ran native json ingestion, some queries I did see an out of memory failure on coordinator-overlord when using the old 'micro quickstart' configuration when I was testing kafka ingestion... I would guess it is related to the sampler? However, I did not look much further into it since I think the recommended way to run tutorials these days is with the auto sizing script, so not sure if it is a big deal or not. Did not see the problem when using that script instead. On Wed, Apr 16, 2025 at 12:39 AM Zoltan Haindrich <k...@rxd.hu> wrote:Hi all, I have created a build for Apache Druid 33.0.0, release candidate 2.(RC1 was botched).Thanks for everyone who has helped contribute to the release! You canreadthe proposed release notes here: https://github.com/apache/druid/pull/17872 The release candidate has been tagged in GitHub as druid-33.0.0-rc2(63477ea1b7866dc2f61c9d1d893dea150735cbff),available here: https://github.com/apache/druid/tree/druid-33.0.0-rc2 The artifacts to be voted on are located here: https://dist.apache.org/repos/dist/dev/druid/33.0.0-rc2/ A staged Maven repository is available for review at:https://repository.apache.org/content/repositories/orgapachedruid-1086/Staged druid.apache.org website documentation is available here: https://druid.staged.apache.org/docs/33.0.0/ A Docker image containing the binary of the release candidate can be retrieved via: docker pull apache/druid:33.0.0-rc2 artifact checksums: 33.0.0-rc2/apache-druid-33.0.0-bin.tar.gzcd6adc2840c87521312534cd6a786b0d92559a652f275a6d773e717cef1c5bd0f6aacac67626e2a26fdad96f2353c5314e53b8a2533b37cbdc953d96025a1e04apache-druid-33.0.0-src.tar.gzd4f78b627e0c46db0d328ce53460ae3f5639831f20efc50ed6c25c5999aa310dd089d4d7813c0994f7c6714a2df24594ab7b14ab087e330bb2fbc5992e6e7293docker:sha256:528a78a68b1dde47a5984e60975203d8cdf0bd4f477f8b97b84a8b2b44e3c683Release artifacts are signed with the following key: https://people.apache.org/keys/committer/kgyrtkirk.asc This key and the key of other committers can also be found in theproject'sKEYS file here: https://dist.apache.org/repos/dist/release/druid/KEYS (If you are a committer, please feel free to add your own key to thatfileby following the instructions in the file's header.) Verify checksums: diff <(shasum -a512 apache-druid-33.0.0-src.tar.gz | cut -d ' ' -f1)<(cat apache-druid-33.0.0-src.tar.gz.sha512 ; echo)diff <(shasum -a512 apache-druid-33.0.0-bin.tar.gz | cut -d ' ' -f1)<(cat apache-druid-33.0.0-bin.tar.gz.sha512 ; echo)Verify signatures: gpg --verify apache-druid-33.0.0-src.tar.gz.ascapache-druid-33.0.0-src.tar.gzgpg --verify apache-druid-33.0.0-bin.tar.gz.ascapache-druid-33.0.0-bin.tar.gzPlease review the proposed artifacts and vote. Note that Apache has specific requirements that must be met before +1 binding votes can becastby PMC members. Please refer to the policy at http://www.apache.org/legal/release-policy.html#policy for moredetails.As part of the validation process, the release artifacts can begeneratedfrom source by running: mvn clean install -Papache-release,dist -Dgpg.skip The RAT license check can be run from source by: mvn apache-rat:check -Prat This vote will be open for at least 72 hours. The vote will pass if a majority of at least three +1 PMC votes are cast. [ ] +1 Release this package as Apache Druid 0.17.0 [ ] 0 I don't feel strongly about it, but I'm okay with the release [ ] -1 Do not release this package because... cheers, Zoltan--------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org For additional commands, e-mail: dev-h...@druid.apache.org
OpenPGP_signature.asc
Description: OpenPGP digital signature
