Hey all , happy new year 🥳 I am very happy to announce that my Guile Scheme CI/CD project, byggsteg (https://codeberg.org/jjba23/byggsteg) on SQLite + base64 + Artanis is a reality, and it's got an amazing performance boost, all also thanks to your contributions, help and hinting, and my hours 😄 Byggsteg works real nicely on all my computers.
I was having a bit of problems deploying the new version to my Guix server (https://codeberg.org/jjba23/wolk-jjba), specially due to the addition of guile-dbd and guile-dbi-sqlite but I have resolved this. I also managed to now configure everything properly about hostnames and redirections of HTTP to HTTPS. Byggsteg almost works 100% on the server, but somehow the Shepherd service crashes and restarts when i try to start a new job. I was suspicious of threading capabilities of my server, but all works as expected there, also checked from the Guix REPL, and hardware is powerful and up to date enough. So now I suspect that my database operations somehow are making things crash. Could you please help me figure this out, I am at a loss cause there is no useful error being displayed 😢 I attached the logs from my server, maybe they tell you more than they tell me. The code where things go BOOM 💣 I think? is at lib/byggsteg/job/pipeline.scm, also attached to this thread. Find below also the Shepherd service definition I use (in byggsteg readme as well) Thank you in advance! ``` (define (wolk-jjba-byggsteg-service config) (list (shepherd-service (documentation "Run byggsteg as a daemon") (provision '(byggsteg)) (requirement '()) (start #~(make-forkexec-constructor '("make" "production-server") #:directory "/var/log/byggsteg/job-clone/byggsteg/trunk" #:environment-variables (list "GUILE_LOAD_PATH=/run/current-system/profile/share/guile/site/3.0" "GUILE_DBD_PATH=/run/current-system/profile/lib" "C_INCLUDE_PATH=/run/current-system/profile/include" "GUILE_LOAD_COMPILED_PATH=/run/current-system/profile/lib/guile/3.0/site-ccache:/run/current-system/profile/share/guile/site/3.0" "LIBRARY_PATH=/run/current-system/profile/lib" "GIT_SSL_NO_VERIFY=1" "LANG=nl_NL.UTF-8" "GUILE_AUTO_COMPILE=0" "PWD=/var/log/byggsteg/job-clone/byggsteg/trunk" "PATH=/run/privileged/bin:/run/current-system/profile/bin:/run/current-system/profile/sbin"))) (stop #~(make-kill-destructor)) (auto-start? #t) (respawn? #t)))) (define wolk-jjba-byggsteg-service-type (service-type (name 'byggsteg) (description "Run byggsteg as a daemon") (extensions (list (service-extension shepherd-root-service-type wolk-jjba-byggsteg-service))) (default-value '()))) ```
Jan 1 22:19:34 localhost shepherd[1]: Service byggsteg has been started. Jan 1 22:19:34 localhost shepherd[1]: make[901] art work -c conf/production.conf Jan 1 22:19:35 localhost shepherd[1]: make[901] Backtrace: Jan 1 22:19:35 localhost shepherd[1]: make[901] In ice-9/boot-9.scm: Jan 1 22:19:35 localhost shepherd[1]: make[901] 1752:10 1 (with-exception-handler _ _ #:unwind? _ # _) Jan 1 22:19:35 localhost shepherd[1]: make[901] In unknown file: Jan 1 22:19:35 localhost shepherd[1]: make[901] 0 (apply-smob/0 #<thunk 7f2a2ba41a60>) Jan 1 22:19:35 localhost shepherd[1]: make[901] Jan 1 22:19:35 localhost shepherd[1]: make[901] ERROR: In procedure apply-smob/0: Jan 1 22:19:35 localhost shepherd[1]: make[901] In procedure dbi-close: Wrong type argument in position 1: #<finalized smob 7f2a28f02ed0> Jan 1 22:20:17 localhost shepherd[1]: make[901] Loading conf/production.conf...done. Jan 1 22:20:17 localhost shepherd[1]: make[901] Loading server engine 'ragnarok' ... Jan 1 22:20:17 localhost shepherd[1]: make[901] Loading server engine 'ragnarok' ... Jan 1 22:20:17 localhost shepherd[1]: make[901] User wants to use Database, initializing... Jan 1 22:20:17 localhost shepherd[1]: make[901] connection pools are initilizing...<sqlite3> byggsteg.db Jan 1 22:20:17 localhost shepherd[1]: make[901] <sqlite3> byggsteg.db Jan 1 22:20:17 localhost last message repeated 62 times Jan 1 22:20:17 localhost shepherd[1]: make[901] DB pool init ok! Jan 1 22:20:17 localhost shepherd[1]: make[901] Now the size of connection pool is 64. Jan 1 22:20:17 localhost shepherd[1]: make[901] DB init done! Jan 1 22:20:17 localhost shepherd[1]: make[901] Initilizing session backend `SIMPLE'... Jan 1 22:20:17 localhost shepherd[1]: make[901] Session with SIMPLE backend init done! Jan 1 22:20:17 localhost shepherd[1]: make[901] Loading controllers... Jan 1 22:20:17 localhost shepherd[1]: make[901] Creating table `job' defined in model Jan 1 22:20:17 localhost shepherd[1]: make[901] ...... Jan 1 22:20:17 localhost shepherd[1]: make[901] select * from sqlite_master where type='table' and name='job' Jan 1 22:20:17 localhost shepherd[1]: make[901] Table `job' already exists! Jan 1 22:20:17 localhost shepherd[1]: make[901] Done. Jan 1 22:20:17 localhost shepherd[1]: make[901] Creating table `job' defined in model Jan 1 22:20:17 localhost shepherd[1]: make[901] ...... Jan 1 22:20:17 localhost shepherd[1]: make[901] select * from sqlite_master where type='table' and name='job' Jan 1 22:20:17 localhost shepherd[1]: make[901] Table `job' already exists! Jan 1 22:20:17 localhost shepherd[1]: make[901] Done. Jan 1 22:20:17 localhost shepherd[1]: make[901] Creating table `log' defined in model Jan 1 22:20:17 localhost shepherd[1]: make[901] ...... Jan 1 22:20:17 localhost shepherd[1]: make[901] select * from sqlite_master where type='table' and name='log' Jan 1 22:20:17 localhost shepherd[1]: make[901] Table `log' already exists! Jan 1 22:20:17 localhost shepherd[1]: make[901] Done. Jan 1 22:20:17 localhost shepherd[1]: make[901] Creating table `log' defined in model Jan 1 22:20:17 localhost shepherd[1]: make[901] ...... Jan 1 22:20:17 localhost shepherd[1]: make[901] select * from sqlite_master where type='table' and name='log' Jan 1 22:20:17 localhost shepherd[1]: make[901] Table `log' already exists! Jan 1 22:20:17 localhost shepherd[1]: make[901] Done. Jan 1 22:20:17 localhost shepherd[1]: make[901] Creating table `profile' defined in model Jan 1 22:20:17 localhost shepherd[1]: make[901] ...... Jan 1 22:20:17 localhost shepherd[1]: make[901] select * from sqlite_master where type='table' and name='profile' Jan 1 22:20:17 localhost shepherd[1]: make[901] Table `profile' already exists! Jan 1 22:20:17 localhost shepherd[1]: make[901] Done. Jan 1 22:20:17 localhost shepherd[1]: make[901] Creating table `profile' defined in model Jan 1 22:20:17 localhost shepherd[1]: make[901] ...... Jan 1 22:20:17 localhost shepherd[1]: make[901] select * from sqlite_master where type='table' and name='profile' Jan 1 22:20:17 localhost shepherd[1]: make[901] Table `profile' already exists! Jan 1 22:20:17 localhost shepherd[1]: make[901] Done. Jan 1 22:20:17 localhost shepherd[1]: make[901] Loading restful API... Jan 1 22:20:17 localhost shepherd[1]: make[901] Regenerating route cache ... Jan 1 22:20:17 localhost shepherd[1]: make[901] Server core: ragnarok Jan 1 22:20:17 localhost shepherd[1]: make[901] http://byggsteg.jointhefreeworld.org Jan 1 22:20:17 localhost shepherd[1]: make[901] Anytime you want to quit just try Ctrl+C, thanks! Jan 1 22:20:17 localhost shepherd[1]: make[901] Loading server engine 'ragnarok' ... Jan 1 22:20:17 localhost shepherd[1]: make[901] Using Non-Blocking I/O Jan 1 22:20:17 localhost shepherd[1]: make[901] Installed suspendable ports Jan 1 22:20:17 localhost shepherd[1]: make[901] Starting `ragnarok' engine loader ... Jan 1 22:20:17 localhost shepherd[1]: make[901] Enter ragnarok-http-gateway-run Jan 1 22:20:17 localhost shepherd[1]: make[901] Listen socket is #<input-output: socket 8> Jan 1 22:20:17 localhost shepherd[1]: make[901] generating work-table Jan 1 22:20:17 localhost shepherd[1]: make[901] Prepare for regnarok-open Jan 1 22:20:17 localhost shepherd[1]: make[901] Added listenning port to epoll Jan 1 22:20:17 localhost shepherd[1]: make[901] Prepare for main-loop Jan 1 22:20:17 localhost shepherd[1]: make[901] Checking event (8 . 1) Jan 1 22:20:17 localhost shepherd[1]: make[901] listenning-port? (8 . 1) Jan 1 22:20:17 localhost shepherd[1]: make[901] Is listen-socket? 8 ?= #<input-output: socket 8> Jan 1 22:20:17 localhost shepherd[1]: make[901] New connection from listening socket (8 . 1) Jan 1 22:20:17 localhost shepherd[1]: make[901] make ragnarok client (#<input-output: socket 80> . #(2 2130706433 34876)) Jan 1 22:20:17 localhost shepherd[1]: make[901] Async read! Jan 1 22:20:17 localhost shepherd[1]: make[901] I would break #<input-output: socket 8> Jan 1 22:20:17 localhost shepherd[1]: make[901] Get 1 new connections. Jan 1 22:20:17 localhost shepherd[1]: make[901] New coming connections: 1 Jan 1 22:20:17 localhost shepherd[1]: make[901] Enter main-loop, protocol is http Jan 1 22:20:17 localhost shepherd[1]: make[901] Prepare to serve one request #<input-output: socket 80> Jan 1 22:20:17 localhost shepherd[1]: make[901] START: serve #<input-output: socket 80> Jan 1 22:20:17 localhost shepherd[1]: make[901] Is continuable work #<input-output: socket 80> Jan 1 22:20:17 localhost shepherd[1]: make[901] Ragnarok: new request #<input-output: socket 80> Jan 1 22:20:17 localhost shepherd[1]: make[901] Register #<input-output: socket 80> as RW event Jan 1 22:20:17 localhost shepherd[1]: make[901] register End Jan 1 22:20:17 localhost shepherd[1]: make[901] Ragnarok: new task from #<input-output: socket 80> Jan 1 22:20:17 localhost shepherd[1]: make[901] Ragnarok: start to read client #<ragnarok-client treasure: (#<input-output: socket 80> . #(2 2130706433 34876))> Jan 1 22:20:17 localhost shepherd[1]: make[901] ragnarok-read 127.0.0.1 Jan 1 22:20:17 localhost shepherd[1]: make[901] Enter http-read #<input-output: socket 80> Jan 1 22:20:17 localhost shepherd[1]: make[901] url-need-websocket? /api/v1/jobs/manage/submit Jan 1 22:20:17 localhost shepherd[1]: make[901] url-need-inexclusive-websocket? /api/v1/jobs/manage/submit Jan 1 22:20:17 localhost shepherd[1]: make[901] try to read request body guile: uncaught exception: Jan 1 22:20:17 localhost shepherd[1]: make[901] In procedure dbi-close: Wrong type argument in position 1: #<finalized smob 7f2a28f02d10> Jan 1 22:20:17 localhost shepherd[1]: make[901] #<<request> method: POST uri: #<<uri> scheme: #f userinfo: #f host: #f port: #f path: "/api/v1/jobs/manage/submit" query: #f fragment: #f> version: (1 . 0) headers: ((host "byggsteg-proxy" . #f) (connection close) (content-length . 184) (user-agent . "curl/7.88.1") (accept (*/*)) (content-type application/x-www-form-urlencoded) (authorization basic . "$BYGGSTEG_KEY")) meta: () port: #<input-output: socket 80>> Jan 1 22:20:17 localhost shepherd[1]: make[901] handle request Jan 1 22:20:17 localhost shepherd[1]: make[901] prepare the handler Jan 1 22:20:17 localhost shepherd[1]: make[901] [Client] POST /api/v1/jobs/manage/submit #<input-output: socket 80> Jan 1 22:20:17 localhost shepherd[1]: make[901] insert into job (created_at,job_failed,job_in_progress,job_success,job_data,human_id,id) values ('1735766417','0','1','0','YCgocHJvamVjdCAuICJieWdnc3RlZyIpKGJyYW5jaC1uYW1lIC4gInRydW5rIiko Jan 1 22:20:17 localhost shepherd[1]: make[901] Jan 1 22:20:17 localhost shepherd[1]: make[901] dGFzayAuICJieWdnc3RlZy12ZXJzaW9uIikoY2xvbmUtdXJsIC4gImh0dHBzOi8v Jan 1 22:20:17 localhost shepherd[1]: make[901] Jan 1 22:20:17 localhost shepherd[1]: make[901] Y29kZWJlcmcub3JnL2pqYmEyMy9ieWdnc3RlZyIpKQ==','YnlnZ3N0ZWdfXzIyOjIwOjE3X18wMS0wMS0yMDI1LmJ5Z2dzdGVnLmxvZw==','55542c99-0470-459c-9bcd-f07dfcdfb9db'); Jan 1 22:20:17 localhost shepherd[1]: make[901] insert into log (created_at,log_data,job_id,human_id,id) values ('1735766417','','55542c99-0470-459c-9bcd-f07dfcdfb9db','YnlnZ3N0ZWdfXzIyOjIwOjE3X18wMS0wMS0yMDI1LmJ5Z2dzdGVnLmxvZw==','c5c81421-7380-4e20-a852-9f9e5e5b42ad'); Jan 1 22:20:17 localhost shepherd[1]: make[901] Jan 1 22:20:17 localhost shepherd[1]: make[901] Jan 1 22:20:17 localhost shepherd[1]: make[901] ====================== Jan 1 22:20:17 localhost shepherd[1]: make[901] #<route-context handler: #<procedure 7f2a26456160 at ice-9/eval.scm:333:13 (a)> keys: () regexp: #(*irregex-tag* #f #f #<procedure 7f2a26515480 at artanis/irregex.scm:3411:9 (cnk init src str i end matches fail)> 3 0 #((26 . 26)) ()) request: #<<request> method: POST uri: #<<uri> scheme: #f userinfo: #f host: #f port: #f path: "/api/v1/jobs/manage/submit" query: #f fragment: #f> version: (1 . 0) headers: ((host "byggsteg-proxy" . #f) (connection close) (content-length . 184) (user-agent . "curl/7. Jan 1 22:20:17 localhost shepherd[1]: 88.1") (ac Jan 1 22:20:17 localhost shepherd[1]: make[901] cept (*/*)) (content-type application/x-www-form-urlencoded) (authorization basic . "$BYGGSTEG_KEY")) meta: () port: #<input-output: socket 80>> path: "/api/v1/jobs/manage/submit" qt: #f method: POST rhk: (POST . "^/api/v1/jobs/manage/submit$") bt: () body: #vu8(106 111 98 45 99 111 100 101 61 37 54 48 37 50 56 37 50 56 112 114 111 106 101 99 116 43 46 43 37 50 50 98 121 103 103 115 116 101 103 37 50 50 37 50 57 37 50 56 98 114 97 110 99 104 45 110 97 109 101 43 46 43 37 50 50 116 114 117 110 107 Jan 1 22:20:17 localhost shepherd[1]: 37 50 50 3 Jan 1 22:20:17 localhost shepherd[1]: make[901] 7 50 57 37 50 56 116 97 115 107 43 46 43 37 50 50 98 121 103 103 115 116 101 103 45 118 101 114 115 105 111 110 37 50 50 37 50 57 37 50 56 99 108 111 110 101 45 117 114 108 43 46 43 37 50 50 104 116 116 112 115 37 51 65 37 50 70 37 50 70 99 111 100 101 98 101 114 103 46 111 114 103 37 50 70 106 106 98 97 50 51 37 50 70 98 121 103 103 115 116 101 103 37 50 50 37 50 57 37 50 57) date: #f cookie: () set-cookie: () lpc: #f oht: #<hash-table 7f2a2645f860 1/31> conn: #f> Jan 1 22:20:17 localhost shepherd[1]: make[901] ====================== Jan 1 22:20:17 localhost shepherd[1]: make[901] Jan 1 22:20:17 localhost last message repeated 2 times Jan 1 22:20:17 localhost shepherd[1]: Service byggsteg (PID 901) exited with 2. Jan 1 22:20:17 localhost shepherd[1]: make[901] ====================== Jan 1 22:20:17 localhost shepherd[1]: make[901] ((job-uuid . "55542c99-0470-459c-9bcd-f07dfcdfb9db") (log-uuid . "c5c81421-7380-4e20-a852-9f9e5e5b42ad")) Jan 1 22:20:17 localhost shepherd[1]: make[901] ====================== Jan 1 22:20:17 localhost shepherd[1]: make[901] Jan 1 22:20:17 localhost last message repeated 2 times Jan 1 22:20:17 localhost shepherd[1]: make[901] ====================== Jan 1 22:20:17 localhost shepherd[1]: make[901] "Before calling new thread" Jan 1 22:20:17 localhost shepherd[1]: make[901] ====================== Jan 1 22:20:17 localhost shepherd[1]: make[901] Jan 1 22:20:17 localhost shepherd[1]: make[901] make: *** [Makefile:16: production-server] Fout 1 Jan 1 22:20:17 localhost shepherd[1]: Respawning byggsteg. Jan 1 22:20:17 localhost shepherd[1]: Starting service byggsteg... Jan 1 22:20:17 localhost shepherd[1]: Service byggsteg started. Jan 1 22:20:17 localhost shepherd[1]: Service byggsteg running with value #<<process> id: 911 command: ("make" "production-server")>. Jan 1 22:20:17 localhost shepherd[1]: Service byggsteg has been started. Jan 1 22:20:17 localhost shepherd[1]: make[911] art work -c conf/production.conf Jan 1 22:20:17 localhost shepherd[1]: make[911] Backtrace: Jan 1 22:20:17 localhost shepherd[1]: make[911] In ice-9/boot-9.scm: Jan 1 22:20:17 localhost shepherd[1]: make[911] 1752:10 1 (with-exception-handler _ _ #:unwind? _ # _) Jan 1 22:20:17 localhost shepherd[1]: make[911] In unknown file: Jan 1 22:20:17 localhost shepherd[1]: make[911] 0 (apply-smob/0 #<thunk 7f8026e4e920>) Jan 1 22:20:17 localhost shepherd[1]: make[911] Jan 1 22:20:17 localhost shepherd[1]: make[911] ERROR: In procedure apply-smob/0: Jan 1 22:20:17 localhost shepherd[1]: make[911] In procedure dbi-close: Wrong type argument in position 1: #<finalized smob 7f80244f41b0>
;;; byggsteg - the CI/CD orchestrator written in Guile Scheme ;; Copyright (C) 2024 "Josep Bigorra" known as "jjba23" <[email protected]> ;; Copyright (C) 2024 "Mu Lei" known as "NalaGinrut" <[email protected]> ;; byggsteg is free software: you can redistribute it and/or modify ;; it under the terms of the GNU General Public License as published by ;; the Free Software Foundation, either version 3 of the License, or ;; (at your option) any later version. ;; byggsteg is distributed in the hope that it will be useful, ;; but WITHOUT ANY WARRANTY; without even the implied warranty of ;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the ;; GNU General Public License for more details. ;; You should have received a copy of the GNU General Public License ;; along with byggsteg. If not, see <https://www.gnu.org/licenses/>. (define-module (byggsteg job pipeline) #:use-module (byggsteg preferences) #:use-module (byggsteg process) #:use-module (byggsteg job steps) #:use-module (byggsteg job utils) #:use-module (byggsteg uuid) #:use-module (artanis artanis) #:use-module (app models log) #:use-module (app models job) #:use-module (byggsteg base64) #:use-module (byggsteg debug) #:use-module (app models job) #:use-module (app models log) #:use-module (ice-9 string-fun) #:use-module (ice-9 threads)) (begin (define* (async-job-pipeline #:key rc job-uuid log-uuid) (debug-print rc) (debug-print `((job-uuid unquote job-uuid) (log-uuid unquote log-uuid))) (define* (run-pipeline #:key project branch-name clone-url task) (display "\nstarting new job...\n") (clone-repo-step #:project project #:branch-name branch-name #:clone-url clone-url) (cond ;; Stack test + sdist ((eq? task 'stack-test) (stack-step #:project project #:branch-name branch-name #:clone-url clone-url #:stack-task "test") (stack-step #:project project #:branch-name branch-name #:clone-url clone-url #:stack-task "sdist --tar-dir .")) ;; Stack build + sdist ((eq? task 'stack-build) (stack-step #:project project #:branch-name branch-name #:clone-url clone-url #:stack-task "build") (stack-step #:project project #:branch-name branch-name #:clone-url clone-url #:stack-task "sdist --tar-dir .")) ;; Byggsteg version update ((eq? task 'byggsteg-version) (byggsteg-version-step #:project project #:branch-name branch-name #:clone-url clone-url)) ;; Nix generic build ((eq? task 'nix-build) (nix-build-step #:project project #:branch-name branch-name #:clone-url clone-url)) ;; Herd pull and restart ((eq? task 'pull-and-restart) (pull-and-restart-step #:project project #:branch-name branch-name #:clone-url clone-url)) ;; Scala SBT test ((eq? task 'sbt-test) (sbt-test-step #:project project #:branch-name branch-name #:clone-url clone-url)) ;; Pull and make deploy ((eq? task 'pull-and-deploy) (pull-and-deploy-step #:project project #:branch-name branch-name #:clone-url clone-url)) ;; Otherwise attempt generic `make build' (else (make-build-step #:project project #:branch-name branch-name #:clone-url clone-url)))) (debug-print "Before calling new thread") (call-with-new-thread (lambda () (let* ((j (car ($job 'get #:ret 'top #:condition (where #:id job-uuid)))) (l (car ($log 'get #:ret 'top #:condition (where #:id log-uuid)))) (old-log-data (safe-base64-decode (assoc-ref l "log_data"))) (kv (eval-string (safe-base64-decode (assoc-ref j "job_data")))) (project (assoc-ref kv 'project)) (branch-name (assoc-ref kv 'branch-name)) (clone-url (assoc-ref kv 'clone-url)) (task (string->symbol (assoc-ref kv 'task))) (o (with-output-to-string (lambda () (run-pipeline #:project project #:branch-name branch-name #:clone-url clone-url #:task task)))) (new-log-data (string-append old-log-data "\n" o))) (display o) (DB-query (DB-open rc) (->sql update 'log set `((log_data ,(safe-base64-encode new-log-data)) (updated_at ,(current-time))) (where #:id log-uuid))) (DB-query (DB-open rc) (->sql update 'job set `((job_success 1) (job_in_progress 0) (job_failed 0) (updated_at ,(current-time))) (where #:id job-uuid))) ) )) ) (export async-job-pipeline))
