GAlexIHU opened a new issue, #5997: URL: https://github.com/apache/incubator-devlake/issues/5997
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-devlake/issues?q=is%3Aissue) and found no similar issues. ### What happened When installing via the helm chart on an AKS cluster the migration script for tapd fails. Values used for installing the chart: ```yaml grafana: enabled: true ingress: annotations: cert-manager.io/cluster-issuer: letsencrypt kubernetes.io/ingress.class: nginx kubernetes.io/tls-acme: "true" enableHttps: true enabled: true hostname: devlake.qwlth.alex-goth.com tlsSecretName: devlake-tls useDefaultNginx: true lake: encryptionSecret: secret: PPXRGSOIDEQVACDYPOQINZUVFEWAOMSMSGUJULKFEFEIDINSOVZDJLXYZVQCAWUNGWLWPRLXZKLNCWWRVXVLLRIOPUAISCPAATYQIQFDDYGIRSARTLLMXWXGNUBYNAUB envs: - name: API_TIMEOUT value: 120s - name: API_RETRY value: "3" - name: API_REQUESTS_PER_HOUR value: "10000" - name: PIPELINE_MAX_PARALLEL value: "1" - name: IN_SECURE_SKIP_VERIFY value: "false" - name: LOGGING_DIR value: /app/logs - name: LOGGING_LEVEL value: debug - name: IN_SECURE_SKIP_VERIFY value: "true" - name: GIN_MODE value: debug options: database: mysql ``` Log output of the devlake-lake container on initial startup: ``` Version: v0.18.0-beta8@ time="2023-08-31 09:29:18" level=info msg="migration initialized" time="2023-08-31 09:29:18" level=info msg="plugin loaded ae" time="2023-08-31 09:29:19" level=info msg="plugin loaded bamboo" time="2023-08-31 09:29:19" level=info msg="plugin loaded bitbucket" time="2023-08-31 09:29:19" level=info msg="plugin loaded customize" time="2023-08-31 09:29:19" level=info msg="plugin loaded dbt" time="2023-08-31 09:29:19" level=info msg="plugin loaded dora" time="2023-08-31 09:29:19" level=info msg="plugin loaded feishu" time="2023-08-31 09:29:19" level=info msg="plugin loaded gitee" time="2023-08-31 09:29:19" level=info msg="plugin loaded gitextractor" time="2023-08-31 09:29:20" level=info msg="plugin loaded github" time="2023-08-31 09:29:20" level=info msg="plugin loaded github_graphql" time="2023-08-31 09:29:20" level=info msg="plugin loaded gitlab" time="2023-08-31 09:29:20" level=info msg="plugin loaded icla" time="2023-08-31 09:29:20" level=info msg="plugin loaded jenkins" time="2023-08-31 09:29:20" level=info msg="plugin loaded jira" time="2023-08-31 09:29:21" level=info msg="plugin loaded org" time="2023-08-31 09:29:21" level=info msg="plugin loaded pagerduty" time="2023-08-31 09:29:21" level=info msg="plugin loaded refdiff" time="2023-08-31 09:29:21" level=info msg="plugin loaded slack" time="2023-08-31 09:29:21" level=info msg="plugin loaded sonarqube" time="2023-08-31 09:29:22" level=info msg="plugin loaded starrocks" time="2023-08-31 09:29:22" level=info msg="plugin loaded tapd" time="2023-08-31 09:29:22" level=info msg="plugin loaded teambition" time="2023-08-31 09:29:22" level=info msg="plugin loaded trello" time="2023-08-31 09:29:22" level=info msg="plugin loaded webhook" time="2023-08-31 09:29:23" level=info msg="plugin loaded zentao" time="2023-08-31 09:29:23" level=info msg="Loading remote plugins" time="2023-08-31 09:29:36" level=info msg="remote plugin loaded azuredevops" time="2023-08-31 09:29:36" level=info msg=" [migrator] applying migration script add commit line change table:202209221033" time="2023-08-31 09:29:36" level=info msg=" [migrator] applying migration script add column `repo_id` and `commit_sha` at _tool_bitbucket_pipelines:2023041111938" time="2023-08-31 09:29:36" level=info msg=" [migrator] applying migration script create init schemas:20220406212344" time="2023-08-31 09:29:37" level=info msg=" [migrator] applying migration script Jira init schemas:20220407201138" time="2023-08-31 09:29:37" level=info msg=" [migrator] applying migration script Rename step to stage :20220505212344" time="2023-08-31 09:29:37" level=info msg=" [migrator] applying migration script Rename source to connection :20220505212344" time="2023-08-31 09:29:37" level=info msg=" [migrator] applying migration script add column `subtasks` at _devlake_tasks:20220601000005" time="2023-08-31 09:29:37" level=info msg=" [migrator] applying migration script add mode field to blueprint:20220616110537" time="2023-08-31 09:29:37" level=info msg=" [migrator] applying migration script blueprint normal mode support:20220622110537" time="2023-08-31 09:29:37" level=info msg=" [migrator] applying migration script create domain layer init schemas:20220707232344" time="2023-08-31 09:29:41" level=info msg=" [migrator] applying migration script create subtask schema:20220711000001" time="2023-08-31 09:29:41" level=info msg=" [migrator] applying migration script AE init schemas:20220714201133" time="2023-08-31 09:29:42" level=info msg=" [migrator] applying migration script Jenkins init schemas:20220714201237" time="2023-08-31 09:29:42" level=info msg=" [migrator] applying migration script Gitlab init schemas:20220714231236" time="2023-08-31 09:29:45" level=info msg=" [migrator] applying migration script Gitee init schemas:20220714231268" time="2023-08-31 09:29:47" level=info msg=" [migrator] applying migration script Github init schemas 20220707:20220715000001" time="2023-08-31 09:29:49" level=info msg=" [migrator] applying migration script Jira init schemas:20220716201138" time="2023-08-31 09:29:52" level=info msg=" [migrator] applying migration script add commit_file_components components table,update commit_files table:20220722165805" time="2023-08-31 09:29:52" level=info msg=" [migrator] applying migration script add project_mapping table to domain layer:20220725152355" time="2023-08-31 09:29:53" level=info msg=" [migrator] applying migration script remove notes from domain layer:20220727165805" time="2023-08-31 09:29:53" level=info msg=" [migrator] applying migration script Github add github_runs table:20220728000001" time="2023-08-31 09:29:53" level=info msg=" [migrator] applying migration script Github add github_jobs table:20220729000001" time="2023-08-31 09:29:53" level=info msg=" [migrator] applying migration script rename pull_request_number to pull_request_key, issue_number to issue_key:20220729165805" time="2023-08-31 09:29:53" level=info msg=" [migrator] applying migration script pipeline and job:20220729231236" time="2023-08-31 09:29:53" level=info msg=" [migrator] applying migration script Jenkins modify build and job:20220729231237" time="2023-08-31 09:29:54" level=info msg=" [migrator] applying migration script add NoPKModel to commit_parents:20220801162735" time="2023-08-31 09:29:55" level=info msg=" [migrator] applying migration script Github add github_pipelines table:20220803000001" time="2023-08-31 09:29:55" level=info msg=" [migrator] applying migration script Bitbucket init schema 20220803:20220803220824" time="2023-08-31 09:29:56" level=info msg=" [migrator] applying migration script add pipeline id to job:20220804230912" time="2023-08-31 09:29:56" level=info msg=" [migrator] applying migration script add cicd models:20220818232735" time="2023-08-31 09:29:57" level=info msg=" [migrator] applying migration script rename user_id to account_id in pull_request_comments and issue_comments:20220826120824" time="2023-08-31 09:29:57" level=info msg=" [migrator] applying migration script modify tables for dora:20220829232735" time="2023-08-31 09:29:57" level=info msg=" [migrator] applying migration script add column `type` at boards:20220830142321" time="2023-08-31 09:29:58" level=info msg=" [migrator] applying migration script encrypt Blueprint:20220904142321" time="2023-08-31 09:29:58" level=info msg=" [migrator] applying migration script encrypt Pipeline:20220904162121" time="2023-08-31 09:29:58" level=info msg=" [migrator] applying migration script modify cicd pipeline:20220905232735" time="2023-08-31 09:29:59" level=info msg=" [migrator] applying migration script UpdateSchemas for fixDurationToFloat8:20220906000005" time="2023-08-31 09:29:59" level=info msg=" [migrator] applying migration script gitlab add _tool_gitlab_pipeline_projects table:20220907230912" time="2023-08-31 09:29:59" level=info msg=" [migrator] applying migration script webhook init schemas:20220908000001" time="2023-08-31 09:29:59" level=info msg=" [migrator] applying migration script Github delete github_pipelines table:20220908000001" time="2023-08-31 09:29:59" level=info msg=" [migrator] applying migration script modify cicd tasks:20220909232735" time="2023-08-31 09:29:59" level=info msg=" [migrator] applying migration script add length of commit_file file_path:20220913165805" time="2023-08-31 09:30:00" level=info msg=" [migrator] applying migration script modify board repos:20220913232735" time="2023-08-31 09:30:01" level=info msg=" [migrator] applying migration script add origin change lead time for pr:20220913235535" time="2023-08-31 09:30:02" level=info msg=" [migrator] applying migration script bitbucket add _tool_bitbucket_pipelines table:20220914111223" time="2023-08-31 09:30:02" level=info msg=" [migrator] applying migration script UpdateSchemas for renamePipelineCommits:20220915000025" time="2023-08-31 09:30:03" level=info msg=" [migrator] applying migration script Jenkins modify build primary key:20220916231237" time="2023-08-31 09:30:04" level=info msg=" [migrator] applying migration script add column `head_repo_id` at tool_github_pull_requests:20220919223048" time="2023-08-31 09:30:04" level=info msg=" [migrator] applying migration script add dora benchmark:20220928000001" time="2023-08-31 09:30:04" level=info msg=" [migrator] applying migration script modify lead_time_minutes:20220929145125" time="2023-08-31 09:30:05" level=info msg=" [migrator] applying migration script bitbucket add _tool_bitbucket_pull_requests_commits table:20221008182354" time="2023-08-31 09:30:05" level=info msg=" [migrator] applying migration script add snapshot table:20221009111241" time="2023-08-31 09:30:05" level=info msg=" [migrator] applying migration script bitbucket add _tool_bitbucket_deployments table:20221013152349" time="2023-08-31 09:30:05" level=info msg=" [migrator] applying migration script add column `repo_id` and `commit_sha` at _tool_bitbucket_pipelines:20221014114623" time="2023-08-31 09:30:05" level=info msg=" [migrator] applying migration script create _tool_icla_committers:20221021183022" time="2023-08-31 09:30:06" level=info msg=" [migrator] applying migration script Create collector state table:20221101000001" time="2023-08-31 09:30:06" level=info msg=" [migrator] applying migration script Remove cicd_pipeline_relation:20221107000001" time="2023-08-31 09:30:06" level=info msg=" [migrator] applying migration script add skip_on_fail to _devlake_pipelines and _devlake_blueprint:20221107095744" time="2023-08-31 09:30:06" level=info msg=" [migrator] applying migration script add job path for builds:20221108231237" time="2023-08-31 09:30:06" level=info msg=" [migrator] applying migration script add fields for jenkins jobs:20221108231237" time="2023-08-31 09:30:07" level=info msg=" [migrator] applying migration script add project tables:20221109194734" time="2023-08-31 09:30:07" level=info msg=" [migrator] applying migration script Tapd init schemas:20221109201138" time="2023-08-31 09:30:11" level=info msg=" [migrator] applying migration script modify commits diffs:20221109232735" time="2023-08-31 09:30:12" level=info msg=" [migrator] applying migration script add url to jenkinsJob:20221110231237" time="2023-08-31 09:30:12" level=info msg=" [migrator] applying migration script add project pr metric tables:20221111000001" time="2023-08-31 09:30:13" level=info msg=" [migrator] applying migration script UpdateSchemas for addEnableGraphqlForConnection:20221111000008" time="2023-08-31 09:30:13" level=info msg=" [migrator] applying migration script PagerDuty init schemas:20221115000001" time="2023-08-31 09:30:14" level=info msg=" [migrator] applying migration script add labels' schema for blueprint and pipeline:20221115000034" time="2023-08-31 09:30:14" level=info msg=" [migrator] applying migration script add table _tool_jira_transformation_rules, add transformation_rule_id to _tool_jira_boards:20221117122534" time="2023-08-31 09:30:14" level=info msg=" [migrator] applying migration script add project to blueprint:20221117184342" time="2023-08-31 09:30:14" level=info msg=" [migrator] applying migration script add project issue metric tables:20221120000001" time="2023-08-31 09:30:15" level=info msg=" [migrator] applying migration script save state for collector:20221125000038" time="2023-08-31 09:30:15" level=info msg=" [migrator] applying migration script add table _tool_gitlab_transformation_rules, add transformation_rule_id and html_url to _tool_gitlab_projects:20221125102502" time="2023-08-31 09:30:15" level=info msg=" [migrator] applying migration script add full name for builds:20221131000008" time="2023-08-31 09:30:15" level=info msg=" [migrator] applying migration script rename fields in project pr metric tables:20221201000001" time="2023-08-31 09:30:15" level=info msg=" [migrator] applying migration script add enable to project metric:20221201190341" time="2023-08-31 09:30:16" level=info msg=" [migrator] applying migration script add table _tool_jenkins_transformation_rules, add transformation_rule_id to _tool_jenkins_jobs:20221205113502" time="2023-08-31 09:30:16" level=info msg=" [migrator] applying migration script add cicd scope and add cicd_scope_id and drop builds&jobs:20221207000001" time="2023-08-31 09:30:16" level=info msg=" [migrator] applying migration script add table _tool_github_transformation_rules, add transformation_rule_id&clone_url to _tool_github_repos:20221214095902" time="2023-08-31 09:30:16" level=info msg=" [migrator] applying migration script concat owner and name for old github_repos and drop owner for github_repos:20221214115900" time="2023-08-31 09:30:17" level=info msg=" [migrator] applying migration script add project_name to _tool_jira_issues:20221215142316" time="2023-08-31 09:30:17" level=info msg=" [migrator] applying migration script add original_project to issues:20221215142543" time="2023-08-31 09:30:17" level=info msg=" [migrator] applying migration script add error_name to _devlake_tasks and _devlake_pipelines:20221221150548" time="2023-08-31 09:30:17" level=info msg=" [migrator] applying migration script encrypt task.options:20221221162121" time="2023-08-31 09:30:17" level=info msg=" [migrator] applying migration script add std type to gitlab issue:20221230095900" time="2023-08-31 09:30:18" level=info msg=" [migrator] applying migration script add std type to github issue:20221230095900" time="2023-08-31 09:30:18" level=info msg=" [migrator] applying migration script add original type to issue:20221230162121" time="2023-08-31 09:30:18" level=info msg=" [migrator] applying migration script rename project metrics:20222123191424" time="2023-08-31 09:30:18" level=info msg=" [migrator] applying migration script Tapd encode connection token:20230115201138" time="2023-08-31 09:30:18" level=info msg=" [migrator] applying migration script add connection config fields:20230123000000" time="2023-08-31 09:30:18" level=info msg=" [migrator] applying migration script add multiauth to _tool_jira_connections:20230129115901" time="2023-08-31 09:30:18" level=info msg=" [migrator] applying migration script add _tool_customized_fields:20230201093311" time="2023-08-31 09:30:19" level=info msg=" [migrator] applying migration script add endpoint/proxy to pagerduty connection:20230203201814" time="2023-08-31 09:30:19" level=info msg=" [migrator] applying migration script add scope and table _tool_bitbucket_transformation_rules:20230206000008" time="2023-08-31 09:30:20" level=info msg=" [migrator] applying migration script add time_after to _devlake_collector_latest_state:20230213200040" time="2023-08-31 09:30:20" level=info msg=" [migrator] applying migration script modify issues story_point from int64 to float64:20230214145125" time="2023-08-31 09:30:20" level=info msg=" [migrator] applying migration script remove _tool_jira_issues column std_story_point:20230214232735" time="2023-08-31 09:30:20" level=info msg=" [migrator] applying migration script add pipeline step:20230215000009" time="2023-08-31 09:30:21" level=info msg=" [migrator] applying migration script add commit_sha index for commit_files:20230222145127" time="2023-08-31 09:30:22" level=info msg=" [migrator] applying migration script add host, namespace, repo_name to issue_repo_commits:20230222153154" time="2023-08-31 09:30:22" level=info msg=" [migrator] applying migration script add remotelink_repo_pattern to _tool_jira_transformation_rules:20230223112532" time="2023-08-31 09:30:22" level=info msg=" [migrator] applying migration script remove created_date_after from _devlake_collector_latest_state:20230223200040" time="2023-08-31 09:30:22" level=info msg=" [migrator] applying migration script sonarqube init schemas:20230227220071" time="2023-08-31 09:30:23" level=info msg=" [migrator] applying migration script add IsDetailRequired to _tool_gitlab_merge_requests and _tool_gitlab_pipelines:20230228161031" time="2023-08-31 09:30:24" level=info msg=" [migrator] applying migration script add code quality domain:20230301000082" time="2023-08-31 09:30:24" level=info msg=" [migrator] applying migration script add transformation rules to pagerduty:20230303001814" time="2023-08-31 09:30:24" level=info msg=" [migrator] applying migration script trello init schemas:20230305000014" time="2023-08-31 09:30:25" level=info msg=" [migrator] applying migration script teambition init schemas:20230314000001" time="2023-08-31 09:30:27" level=info msg=" [migrator] applying migration script bamboo init schemas:20230315205035" time="2023-08-31 09:30:29" level=info msg=" [migrator] applying migration script sonarqube modify character set:20230316220071" time="2023-08-31 09:30:29" level=info msg=" [migrator] applying migration script expend project_key:20230321000003" time="2023-08-31 09:30:29" level=info msg=" [migrator] applying migration script add connection_id to _tool_jenkins_transformation_rules:20230322150357" time="2023-08-31 09:30:29" level=info msg=" [migrator] applying migration script add connection_id to _tool_gitlab_transformation_rules:20230322150357" time="2023-08-31 09:30:30" level=info msg=" [migrator] applying migration script add connection_id to _tool_bamboo_transformation_rules:20230322150357" time="2023-08-31 09:30:30" level=info msg=" [migrator] applying migration script add connection_id to _tool_trello_transformation_rules:20230322150357" time="2023-08-31 09:30:30" level=info msg=" [migrator] applying migration script add connection_id to _tool_github_transformation_rules:20230322150357" time="2023-08-31 09:30:30" level=info msg=" [migrator] applying migration script add type/env to github runs and run_jobs:20230322150357" time="2023-08-31 09:30:31" level=info msg=" [migrator] applying migration script add connection_id to _tool_bitbucket_transformation_rules:20230322150357" time="2023-08-31 09:30:31" level=info msg=" [migrator] applying migration script add connection_id to _tool_jira_transformation_rules:20230322150357" time="2023-08-31 09:30:31" level=info msg=" [migrator] applying migration script add type/env to gitlab pipelines:20230322150357" time="2023-08-31 09:30:31" level=info msg=" [migrator] applying migration script expand _tool_jira_remotelinks.url and _tool_jira_issue_commits.commit_url to LONGTEXT:20230322153324" time="2023-08-31 09:30:31" level=info msg=" [migrator] applying migration script Tapd add transformation, update _raw_data_params:20230323000003" time="2023-08-31 09:30:32" level=info msg=" [migrator] applying migration script Tapd delete issue and remote link:20230323000004" time="2023-08-31 09:30:32" level=info msg=" [migrator] applying migration script add issue_assignees table:20230402000041" time="2023-08-31 09:30:32" level=info msg=" [migrator] applying migration script Rename Collector state table for Singer Taps:20230405000001" time="2023-08-31 09:30:32" level=info msg=" [migrator] applying migration script Rename cicd_piopeline_commits repo to repo_url:20230406170701" time="2023-08-31 09:30:32" level=info msg=" [migrator] applying migration script add type/env to bitbucket pipelines and pipeline_steps:20230410150357" time="2023-08-31 09:30:33" level=info msg=" [migrator] applying migration script modify tapd custom field name:20230411000004" ``` After the pod exists in this line after the restart startup seems normal but in fact devlake gets into a broken state. The UI shows <img width="819" alt="image" src="https://github.com/apache/incubator-devlake/assets/64845621/246a6527-ad35-4a07-b6ac-d10e743fa994";> but trying to trigger the migrations always fails with the following log output from the container: ``` time="2023-08-31 10:02:43" level=info msg=" [migrator] applying migration script modify tapd custom field name:20230411000004" time="2023-08-31 10:02:43" level=info msg="/app/impls/dalgorm/dalgorm.go:139 Error 1054 (42S22): Unknown column 'custom_field6' in '_tool_tapd_stories' [15.263ms] [rows:0] ALTER TABLE `_tool_tapd_stories` RENAME COLUMN `custom_field6` TO `custom_field_six`" time="2023-08-31 10:02:43" level=error msg="HTTP 500 error caused by: error executing migration (500) Wraps: (2) Error 1054 (42S22): Unknown column 'custom_field6' in '_tool_tapd_stories' (500) Wraps: (3) Error 1054 (42S22): Unknown column 'custom_field6' in '_tool_tapd_stories' Error types: (1) *hintdetail.withDetail (2) *hintdetail.withDetail (3) *mysql.MySQLError" [GIN] 2023/08/31 - 10:02:43 | 500 | 21.879533ms | 10.224.0.5 | GET "/proceed-db-migration" ``` Funny as is, this issue only happens to me on my AKS cluster, in my local docker-desktop everything works correctly. I've tested both with the mysql included in the chart as well as an external Azure MariaDB but the issue persists. ### What do you expect to happen For the initial installation to succeed. ### How to reproduce Install the chart with the provided values on an AKS cluster. ### Anything else _No response_ ### Version 0.18.0-beta8 (also tested on 17.0.0, same issue) ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
