AW: [DISCUSS] Refactor the way we're building/using Thrift
Well it seems it’s not quite that simple. Because for example the CMakeList.txt references directories outside of the module itself. https://github.com/apache/iotdb/blob/master/iotdb-client/client-cpp/src/main/CMakeLists.txt If you wanted to build the client without having built thrift, this will not work. I’m currently working on making the thrift module sort of independent from the rest. The idea is to build the artifact once on every target platform and to make the produced artifacts available via Maven. Instead of just having one executable downloaded, I’d propose to have an archive containing the: * Thrift executable * Thrift runtime shared libraries * Thrift header files With this every module, would just unpack that archive and use that locally. This removes any circular dependencies. However, this module sort of doesn’t fit nicely into the build and it will cause problems. It will always just build thrift for the one platform the RM is currently using. So, I would propose to move this tiny project into the other git repository, that we currently use to serve the pre-compiled binaries. There whenever we need a new OS, CPU Architecture or Thrift version, we update that module and release it (and stage new thrift assembies in Nexus) This would eliminate all complexities in the build. Chris Von: Xiangdong Huang Datum: Samstag, 29. Juli 2023 um 08:37 An: dev@iotdb.apache.org Betreff: Re: [DISCUSS] Refactor the way we're building/using Thrift Hi Chris, > I would like to move the compile-tools directory into the root of the project > and detach it from the rest of the project (So it’s not called from the build > if we build the project). I quickly check current compile-tool, now it is only used by client-cpp. So is it really necessary to move to the root folder? > This only needs to be run every now and then if something fundamental changes > (Like new OSes or new CPU Architectures). Yes, I also agree to compile tools ourselves, rather than download from somewhere.. (and just run once for each operation system.) > I would like to update that module, so it will not only build an executable, > but also package the includes and the libraries for that OS and Architecture +1. Best, --- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 Christofer Dutz 于2023年7月28日周五 20:42写道： > > Hi, > > So, I've been fighting to simplify our use of Thrift, as there were several > problems with how we do it today on my M2 Mac (Mostly however to an updated > Homebrew, which uses different paths). > > While going through the project, I noticed that everything in "protocol" > downloads a pre-built executable, which is simply downloaded and run this > doesn’t seem to run on Arm64 (new M1 and M2 Macs). > > I would like to move the compile-tools directory into the root of the project > and detach it from the rest of the project (So it’s not called from the build > if we build the project). > > This only needs to be run every now and then if something fundamental changes > (Like new OSes or new CPU Architectures). > I would like to update that module, so it will not only build an executable, > but also package the includes and the libraries for that OS and Architecture, > so we can use that in the client-cpp and the client-cpp-example. > I'd like to give it a try if we could also use this approach for the libs and > completely get rid of the thrift compilation as part of the build. > > What do you think? > > Chris >
[DISCUSS] How about building a Test utility class to start IoTDB on random free ports?
Hi, as you might now, I’m currently working hard on untangling a bit of IoTDB’s build. Unfortunately, I now know, why all READMEs refer to using “-DskipTests”. It turns out that our build in its current form has a number of cycles in them. The probably most problematic one is that client-cpp needs the distribution to run its integration-tests and distribution bundles client-cpp … this is a classic cycle in maven. So, if I try to build IoTDB after a clean, at least this module will fail, as distribution is empty and the failure is a bit hard to diagnose at first. It would be good, if that integration-test wasn’t requiring distribution to be built first. As I’ve done so in my Apache Historian experiment, I know we could just run IoTDB from the JVM of the maven build and run it locally in the current modules target directory. This has the benefit of the processes being killed after the build automatically. I usually have the process viewer open while running the builds, as I have to kill zombie iotdb processes, if the build fails. This brings me to the next problem … we always run the tests on the default ports … this means if we have zombies running around, it is impossible to build as many tests will fail. We could also use this IoTDB Test Started to start IoTDB on random available ports. This would eliminate the problem of being blocked by zombie processes, and also enable me to run the build as parallel build. My Mac has way too many cores to just let one do the work, so I usually run builds in parallel. However, with us using static ports, this is not possible, as if two modules run tests in parallel, that need access to these ports, only one will succeed. What do you think? Chris
Re: 如何设置10720和10750端口的监听地址
Hi, This should be caused by confingode's cn_internal_address(in confignode-properties) set to 0.0.0.0, which needs to be set to the actual Intranet ip, and it does not support modification at present. If you need to change it, you can configure cn_internal_address to hostname and then modify the host file externally yourself to support changing the ip. Thanks — Xinyu Tan > 2023年7月28日 11:23，Ivan Soeng 写道： > > 您好！ > > > 目前10720和10750两个端口的监听地址是全IP范围，如何设置监听指定的IP地址？ > > > [root@localhost iotdb-cluster]# netstat -nltp|grep -E > '10667|10710|10720|10730|10740|10750|10760' > tcp6 0 0 :::10750:::*LISTEN > 10015/java > tcp6 0 0 :::10720:::*LISTEN > 7607/java > tcp6 0 0 192.168.1.1:10760 :::*LISTEN > 10015/java > tcp6 0 0 192.168.1.1:10730 :::*LISTEN > 10015/java > tcp6 0 0 192.168.1.1:10667 :::*LISTEN > 10015/java > tcp6 0 0 192.168.1.1:10740 :::*LISTEN > 10015/java > tcp6 0 0 192.168.1.1:10710 :::*LISTEN > 7607/java
Re: [DISCUSS] Adding the generation of sboms to our build?
Cool, CycloneDX is famous. Look forward! --- Xiangdong Huang Christofer Dutz 于2023年7月15日周六 22:59写道： > > Well in PLC4X the plugin generates an XML version of the SBOM. > We’re using this plugin: > https://github.com/CycloneDX/cyclonedx-maven-plugin > > Chris > > Von: Xiangdong Huang > Datum: Samstag, 15. Juli 2023 um 07:58 > An: dev@iotdb.apache.org > Betreff: Re: [DISCUSS] Adding the generation of sboms to our build? > Hi Chris, > > Look forward! SBOM has also received a lot of attention in China. > Which kind of format/standard it will obey? > > Best, > --- > Xiangdong Huang > > Christofer Dutz 于2023年7月14日周五 21:28写道： > > > > Hi all, > > > > here in Europe we’re currently preparing for quite a bit of an earthquake > > caused by the Cyber-Resiliency-Act. In some projects I’m involved in > > (Mainly PLC4X) I’ve started initiating small changes which could make us > > come out without too many problems. > > > > One thing that seems to be coming up in both the EU as well as the US acts, > > are the requirement to publish SBOM information (Software Bill Of > > Material). As we are also using Maven as a build tool, I’ve got a > > configuration in our poms that ensures an Apache release also produces an > > SBOM, that we will be able to deploy. > > > > Are we interested in adding that to the IoTDB build? > > > > Chris
Re: Great job on the restructuring!
welcome to continuously refactor and optimize the project structure :D --- Xiangdong Huang Christofer Dutz 于2023年7月27日周四 22:48写道： > > Hi Folks, > > I sort of missed most of it as git tends to keep the moved directories, if > there’s something in there … as they all contained “target” directories, > I just noticed the cleaned-up structure of the project … awesome work. > > I took the liberty of applying the same structural changes on the reactor (No > worries, in a branch) > There I introduced intermediate sub-parent poms. > Also did I clean up the pom structure, where it needed some cleaning up. > > Now in IntelliJ (and probably every other Maven-Enabled IDE) the project > should also be structured nicely, and the modules have equally formatted > names. > > The changes should not have any real impact on the project and the build. > > In general, I would prefer adjusting the artifact-ids to replicate the module > structure, but this is already a lot better, and I wasn’t sure of the > implications of artifact renaming in upstream builds. > > Chris
Re: [DISCUSS] Refactor the way we're building/using Thrift
Hi Chris, > I would like to move the compile-tools directory into the root of the project > and detach it from the rest of the project (So it’s not called from the build > if we build the project). I quickly check current compile-tool, now it is only used by client-cpp. So is it really necessary to move to the root folder? > This only needs to be run every now and then if something fundamental changes > (Like new OSes or new CPU Architectures). Yes, I also agree to compile tools ourselves, rather than download from somewhere.. (and just run once for each operation system.) > I would like to update that module, so it will not only build an executable, > but also package the includes and the libraries for that OS and Architecture +1. Best, --- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 Christofer Dutz 于2023年7月28日周五 20:42写道： > > Hi, > > So, I've been fighting to simplify our use of Thrift, as there were several > problems with how we do it today on my M2 Mac (Mostly however to an updated > Homebrew, which uses different paths). > > While going through the project, I noticed that everything in "protocol" > downloads a pre-built executable, which is simply downloaded and run this > doesn’t seem to run on Arm64 (new M1 and M2 Macs). > > I would like to move the compile-tools directory into the root of the project > and detach it from the rest of the project (So it’s not called from the build > if we build the project). > > This only needs to be run every now and then if something fundamental changes > (Like new OSes or new CPU Architectures). > I would like to update that module, so it will not only build an executable, > but also package the includes and the libraries for that OS and Architecture, > so we can use that in the client-cpp and the client-cpp-example. > I'd like to give it a try if we could also use this approach for the libs and > completely get rid of the thrift compilation as part of the build. > > What do you think? > > Chris >
[BUILD-STABLE]: Job 'IoTDB/IoTDB-pip-new/master [master] [304]'
BUILD-STABLE: Job 'IoTDB/IoTDB-pip-new/master [master] [304]': Is back to normal.
