And again sorry, turns out I selected the branches the wrong way around … this is the right PR: https://github.com/apache/iotdb/pull/10742
Chris Von: Christofer Dutz <christofer.d...@c-ware.de> Datum: Montag, 31. Juli 2023 um 09:33 An: dev@iotdb.apache.org <dev@iotdb.apache.org> Betreff: AW: [DISCUSS] Refactor the way we're building/using Thrift Hi Folks, So, I’ve literally spent all weekend working on my refactoring and must admit I’m quite happy with the results. However, did I decide to take the PR out of this repo and work on a fork in my GitHub account as I didn’t want to swamp you folks with build failure emails. So, I closed the old PR and opened a new one: https://github.com/apache/iotdb/pull/10741 Right now, I still have two TODOs that I would like to work on: * Make the cpp-client build again on Windows (AARCH64 and x86_64) (Currently struggling with it not detecting that it needs a different socket implementation) * Introduce an “iotdb-test-server” module, that integration tests use for running tests against a server: * The current usage of calling scripts in distribution introduces a logical cycle in Maven (Example: cpp-client needs distribution to run tests, distribution bundles cpp-client) * I had many build failures, as the IoTDB server instances are not shut down cleanly. I have noticed that we generally seem to have a number of flaky tests, that randomly fail … as I’ve been running dozens of full builds with integration tests on all supported platforms, I’ve had to re-run many builds to continue from the last failure. I’ve started compiling a list of these tests and I think we should fix them, as the CI should generally pass and not require an arbitrary number of re-runs. Chris Von: Christofer Dutz <christofer.d...@c-ware.de> Datum: Samstag, 29. Juli 2023 um 16:22 An: dev@iotdb.apache.org <dev@iotdb.apache.org> Betreff: AW: [DISCUSS] Refactor the way we're building/using Thrift Well it seems it’s not quite that simple. Because for example the CMakeList.txt references directories outside of the module itself. https://github.com/apache/iotdb/blob/master/iotdb-client/client-cpp/src/main/CMakeLists.txt If you wanted to build the client without having built thrift, this will not work. I’m currently working on making the thrift module sort of independent from the rest. The idea is to build the artifact once on every target platform and to make the produced artifacts available via Maven. Instead of just having one executable downloaded, I’d propose to have an archive containing the: * Thrift executable * Thrift runtime shared libraries * Thrift header files With this every module, would just unpack that archive and use that locally. This removes any circular dependencies. However, this module sort of doesn’t fit nicely into the build and it will cause problems. It will always just build thrift for the one platform the RM is currently using. So, I would propose to move this tiny project into the other git repository, that we currently use to serve the pre-compiled binaries. There whenever we need a new OS, CPU Architecture or Thrift version, we update that module and release it (and stage new thrift assembies in Nexus) This would eliminate all complexities in the build. Chris Von: Xiangdong Huang <saint...@gmail.com> Datum: Samstag, 29. Juli 2023 um 08:37 An: dev@iotdb.apache.org <dev@iotdb.apache.org> Betreff: Re: [DISCUSS] Refactor the way we're building/using Thrift Hi Chris, > I would like to move the compile-tools directory into the root of the project > and detach it from the rest of the project (So it’s not called from the build > if we build the project). I quickly check current compile-tool, now it is only used by client-cpp. So is it really necessary to move to the root folder? > This only needs to be run every now and then if something fundamental changes > (Like new OSes or new CPU Architectures). Yes, I also agree to compile tools ourselves, rather than download from somewhere.. (and just run once for each operation system.) > I would like to update that module, so it will not only build an executable, > but also package the includes and the libraries for that OS and Architecture +1. Best, ----------------------------------- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 Christofer Dutz <christofer.d...@c-ware.de> 于2023年7月28日周五 20:42写道: > > Hi, > > So, I've been fighting to simplify our use of Thrift, as there were several > problems with how we do it today on my M2 Mac (Mostly however to an updated > Homebrew, which uses different paths). > > While going through the project, I noticed that everything in "protocol" > downloads a pre-built executable, which is simply downloaded and run this > doesn’t seem to run on Arm64 (new M1 and M2 Macs). > > I would like to move the compile-tools directory into the root of the project > and detach it from the rest of the project (So it’s not called from the build > if we build the project). > > This only needs to be run every now and then if something fundamental changes > (Like new OSes or new CPU Architectures). > I would like to update that module, so it will not only build an executable, > but also package the includes and the libraries for that OS and Architecture, > so we can use that in the client-cpp and the client-cpp-example. > I'd like to give it a try if we could also use this approach for the libs and > completely get rid of the thrift compilation as part of the build. > > What do you think? > > Chris >