SundayHK

KubeSphere DevOps 流水线JAVA项目配置

2024-11-03 285 0

环境说明

硬件最低要求：4核16G

修改kubesphere/ks-installer:v3.4.1 为 kubesphere/ks-installer:v3.4.1-patch.0
该补丁镜像更新证书有效期到2051年，解决了devops s2i证书2024-02-14过期问题

kubectl set image deployment/ks-installer -n kubesphere-system ks-installer=kubesphere/ks-installer:v3.4.1-patch.0

kubectl get deployment ks-installer -n kubesphere-system -o jsonpath="{.spec.template.spec.containers[0].image}"

root@ks-master01:~# kubectl rollout status deployment/ks-installer -n kubesphere-system
deployment "ks-installer" successfully rolled out # 即成功更新

https://ask.kubesphere.io/forum/d/22841-340shi-yong-kk3012an-zhuang-hao-zhi-hou-kai-qi-devopsshi-bai-delete-job-migrate-because-helm-upgrade

开启DevOps

https://kubesphere.io/zh/docs/v3.4/pluggable-components/devops/

注：jenkins内存不要小于官方默认值，不然会导致jenkins安装失败或jenkins流水线图形无法显示编辑

kubectl edit cc -n kubesphere-system ks-installer

devops: 
  enabled: true # 将“false”更改为“true”。

SonarQube 配置

部署sonarqube

节点内存建议 16G，不然内存压力会反复重建。
https://kubesphere.io/zh/docs/v3.4/devops-user-guide/how-to-integrate/sonarqube/#create-sonarqube-token-for-new-project

root@ks-master01:~# helm version
version.BuildInfo{Version:"v3.16.1", GitCommit:"5a5449dc42be07001fd5771d56429132984ab3ab", GitTreeState:"clean", GoVersion:"go1.22.7"}
root@ks-master01:~# helm upgrade --install sonarqube sonarqube --repo https://charts.kubesphere.io/main -n kubesphere-devops-system  --create-namespace --set service.type=NodePort
Release "sonarqube" does not exist. Installing it now.
NAME: sonarqube
LAST DEPLOYED: Wed Oct  9 13:28:15 2024
NAMESPACE: kubesphere-devops-system
STATUS: deployed
REVISION: 1
NOTES:
1. Get the application URL by running these commands:
  export NODE_PORT=$(kubectl get --namespace kubesphere-devops-system -o jsonpath="{.spec.ports[0].nodePort}" services sonarqube-sonarqube)
  export NODE_IP=$(kubectl get nodes --namespace kubesphere-devops-system -o jsonpath="{.items[0].status.addresses[0].address}")
  echo http://$NODE_IP:$NODE_PORT
root@ks-master01:~#   export NODE_PORT=$(kubectl get --namespace kubesphere-devops-system -o jsonpath="{.spec.ports[0].nodePort}" services sonarqube-sonarqube)
  export NODE_IP=$(kubectl get nodes --namespace kubesphere-devops-system -o jsonpath="{.items[0].status.addresses[0].address}")
  echo http://$NODE_IP:$NODE_PORT
root@ks-master01:~# kubectl get pod -n kubesphere-devops-system
NAME                                  READY   STATUS      RESTARTS   AGE
devops-28836210-nbd5j                 0/1     Completed   0          85m
devops-28836240-sd2hl                 0/1     Completed   0          55m
devops-28836270-v64mp                 0/1     Completed   0          25m
devops-apiserver-57b84ddf48-mq2sc     1/1     Running     0          5d22h
devops-controller-7779fd7c4f-z2s6v    1/1     Running     0          5d22h
devops-jenkins-8d6f46c68-z9l7f        1/1     Running     0          5d22h
s2ioperator-0                         1/1     Running     0          5d22h
sonarqube-postgresql-0                1/1     Running     0          16m
sonarqube-sonarqube-564574684-h7qxv   1/1     Running     0          6m51s

浏览器中访问 SonarQube 控制台 http://<Node IP>:<NodePort>
http://192.168.77.131:31429
使用默认帐户 admin/admin 登录。

创建 SonarQube 管理员令牌 (Token)

点击右上角字母 A，然后从菜单中选择 My Account 以转到 Profile 页面。
点击 Security 并输入令牌名称，如 kubesphere

image.png

image.png

记录Tokens: 01b1497e7aed7d1edd907ceb44f88a058a42dce7

创建 Webhook 服务器

命令获取 SonarQube Webhook 的地址

export NODE_PORT=$(kubectl get --namespace kubesphere-devops-system -o jsonpath="{.spec.ports[0].nodePort}" services devops-jenkins)
export NODE_IP=$(kubectl get nodes --namespace kubesphere-devops-system -o jsonpath="{.items[0].status.addresses[0].address}")
echo http://$NODE_IP:$NODE_PORT/sonarqube-webhook/

http://192.168.77.131:30180/sonarqube-webhook/

依次点击 Administration、Configuration 和 Webhooks 创建一个 Webhook。

image.png
image.png

将 SonarQube 配置添加到 ks-installer

kubectl edit cc -n kubesphere-system ks-installer

搜索devops，添加字段 sonarqube 并在其下方指定 externalSonarUrl 和 externalSonarToken。

  devops:
    enabled: true
    jenkinsCpuLim: 2
    jenkinsCpuReq: 0.5
    jenkinsMemoryLim: 2Gi
    jenkinsMemoryReq: 2Gi
    jenkinsVolumeSize: 8Gi
    sonarqube:
      externalSonarToken: 01b1497e7aed7d1edd907ceb44f88a058a42dce7
      externalSonarUrl: http://192.168.77.131:31429

Jenkins配置

输出Jenkins访问地址

export NODE_PORT=$(kubectl get --namespace kubesphere-devops-system -o jsonpath="{.spec.ports[0].nodePort}" services devops-jenkins) export NODE_IP=$(kubectl get nodes --namespace kubesphere-devops-system -o jsonpath="{.items[0].status.addresses[0].address}") echo http://$NODE_IP:$NODE_PORT

http://192.168.77.131:30180

请使用地址 http://:30180 访问 Jenkins。安装 KubeSphere 时，默认情况下也会安装 Jenkins 仪表板。此外，Jenkins 还配置有 KubeSphere LDAP，这意味着您可以直接使用 KubeSphere 帐户（例如 admin/P@88w0rd）登录 Jenkins。

注：
有时会出现 修改密码，jenkins 与 kubesphere 的 ldap 同步失败的情况，这时可以 重新修改下 密码，会同步成功。 或者可以先用 默认密码：P@88w0rd 来登录 jenkins。

系统管理-Manage Credentials-Jenkins-全局凭据-添加凭据
secret处填写sonarqube token

image.png

系统管理-系统配置

image.png

将 sonarqubeURL 添加到 KubeSphere 控制台

kubectl edit  cm -n kubesphere-system  ks-console-config
    client:
      version:
        kubesphere: v3.3.2
        kubernetes: v1.28.8
        openpitrix: v3.3.2
      enableKubeConfig: true
      devops: # 手动添加该字段。
        sonarqubeURL: http://192.168.77.131:31429
    defaultClusterName: default

重启服务

kubectl -n kubesphere-devops-system rollout restart deploy devops-apiserver
kubectl -n kubesphere-system rollout restart deploy ks-console

Jenkins邮件修改

https://kubesphere.io/zh/docs/v3.3/devops-user-guide/how-to-use/pipelines/jenkins-email/

kubectl edit deployment -n kubesphere-devops-system devops-jenkins
        - name: kubernetes.connection.timeout
          value: "60000"
        - name: kubernetes.request.timeout
          value: "60000"
        - name: EMAIL_SMTP_HOST
          value: smtp.qq.com
        - name: EMAIL_SMTP_PORT
          value: "465"
        - name: EMAIL_USE_SSL
          value: "false"
        - name: EMAIL_FROM_NAME
          value: KubeSphere
        - name: EMAIL_FROM_ADDR
          value: sundayle@qq.com
        - name: EMAIL_FROM_PASS
          value: 授权码
        image: kubesphere/ks-jenkins:v3.4.0-2.319.3-1
        imagePullPolicy: Always

应用后需要等几分钟才可使用

https://ask.kubesphere.io/forum/d/22841-340shi-yong-kk3012an-zhuang-hao-zhi-hou-kai-qi-devopsshi-bai-delete-job-migrate-because-helm-upgrade

kubectl -n kubesphere-system rollout restart deployment ks-installer

DevOps流水线

https://kubesphere.io/zh/docs/v3.4/devops-user-guide/how-to-use/pipelines/create-a-pipeline-using-graphical-editing-panel/

克隆java项目 https://github.com/kubesphere/devops-maven-sample 推送到私有gitlab

Git Token推送使用

gitlab token 推送
偏好设置 是全局token
image.png

#git clone -b https://{user}:{token}@gitlab.com/xxx/xxx.git
#git set-url origin https://{user}:{token}@gitlab.com/xxx/xxx.git
git push https://{user}:{token}@gitlab.com/xxx/xxx.git

github token 推送

image.png
image.png

git push https://{token}@github.com/xxx/xxx.git

image.png
image.png
构建参数：

  • REGISTRY # Harbor仓库域名
  • DOCKER_NAMESPACE # 名称空间
  • APP_NAME
  • TAG_NAME
  • GITLAB_ACCOUNT
  • GITLAB_DOMAIN

流水线代码

pipeline {
  agent {
    node {
      label 'maven'
    }

  }
  stages {
    stage('Checkout SCM') {
      agent none
      steps {
        git(url: 'http://git.sundayhk.com/zsp/kubesphere-devops-maven-sample.git', credentialsId: 'gitlab-id', branch: 'master', changelog: true, poll: false)
      }
    }

    stage('Unit Test') {
      agent none
      steps {
        container('maven') {
          sh 'mvn clean test'
        }

      }
    }

    stage('Code Analysis') {
      agent none
      steps {
        container('maven') {
          withCredentials([string(credentialsId: 'sonar-token', variable: 'SONAR_TOKEN')]) {
            withSonarQubeEnv('sonar') {
              sh 'mvn sonar:sonar -Dsonar.login=$SONAR_TOKEN'
            }

          }

          timeout(unit: 'HOURS', activity: true, time: 1) {
            waitForQualityGate 'true'
          }

        }

      }
    }

    stage('Build And Push') {
      agent none
      steps {
        container('maven') {
          sh 'mvn -Dmaven.test.skip=true clean package'
          withCredentials([usernamePassword(credentialsId: 'harbor-id', passwordVariable: 'HARBOR_PASSWORD', usernameVariable: 'HARBOR_USERNAME')]) {
            sh 'podman build -f Dockerfile-online -t $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:SNAPSHOT-$BUILD_NUMBER .'
            sh 'echo "$HARBOR_PASSWORD" | podman login $REGISTRY -u "$HARBOR_USERNAME" --password-stdin'
            sh 'podman push $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:SNAPSHOT-$BUILD_NUMBER'
          }

        }

      }
    }

    stage('Artifacts') {
      agent none
      steps {
        archiveArtifacts(artifacts: 'target/*.jar', followSymlinks: false)
      }
    }

    stage('Deploy to Dev') {
      agent none
      steps {
        input(message: 'deploy to dev? @project-admin', submitter: '')
        container('maven') {
          withCredentials([kubeconfigContent(credentialsId: 'demo-kubeconfig', variable: 'KUBECONFIG_CONTENT')]) {
            sh '''mkdir ~/.kube
echo "$KUBECONFIG_CONTENT" > ~/.kube/config
envsubst < deploy/no-branch-dev/devops-sample-svc.yaml | kubectl apply -f -
envsubst < deploy/no-branch-dev/devops-sample.yaml | kubectl apply -f -'''
          }

        }

      }
    }

    stage('Push with tag') {
      agent none
      when {
        expression {
          return params.TAG_NAME =~ /v.*/
        }

      }
      steps {
        container('maven') {
          input(message: 'release image prod?@project-admin', submitter: '')
          withCredentials([string(credentialsId: 'gitlab-token', variable: 'GITLAB_TOKEN')]) {
            sh '''git config --global user.email "sundayhk@qq.com"
git config --global user.name "sundayhk"
git tag -a $TAG_NAME -m "$TAG_NAME"
git remote set-url origin http://$GITLAB_ACCOUNT:$GITLAB_TOKEN@$GITLAB_DOMAIN/$GITLAB_ACCOUNT/$APP_NAME.git
git push --tags --ipv4'''
            sh 'podman tag $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:SNAPSHOT-$BUILD_NUMBER $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:$TAG_NAME'
            sh 'podman push $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:$TAG_NAME'
          }

        }

      }
    }

    stage('Deploy to Prod') {
      agent none
      when {
        expression {
          return params.TAG_NAME =~ /v.*/
        }

      }
      steps {
        input(message: 'deploy to prod?@project-admin', submitter: '')
        container('maven') {
          withCredentials([kubeconfigContent(credentialsId: 'demo-kubeconfig', variable: 'KUBECONFIG_CONTENT')]) {
            sh 'envsubst < deploy/prod-all-in-one/devops-sample.yaml | kubectl apply -f -'
          }

        }

      }
    }

  }
}

image.png
image.png
image.png
image.png
image.png

报错解决

sonarqube 报错

[INFO] Analysis total time: 7.830 s
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 10.370 s
[INFO] Finished at: 2024-10-30T06:48:35Z
[INFO] ------------------------------------------------------------------------

Wait for SonarQube analysis to be completed and return quality gate status58 ms失败

Checking status of SonarQube task 'AZLcLuu1VQF99l2mwYaO' on server 'sonar'
java.lang.IllegalStateException: Unable to parse response from http://192.168.77.131:31429//api/ce/task?id=AZLcLuu1VQF99l2mwYaO:
<!doctype html><html lang="en"><head><meta http-equiv="content-type" content="text/html; charset=UTF-8" charset="UTF-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge">

image.png

原因是 jenkins 配置SonarQube Server URL 最后多出/
image.png
将http://192.168.77.131:31429/ 修改为 http://192.168.77.131:31429 即可

maven 提示没有docker

image.png
解决： 使用podman
![Uploading file...m50uj]()

参考

https://blog.csdn.net/qq_37892401/article/details/128851129

sunday
Kubernetes
0 0

相关文章

虚拟机热添加内存 Kubernetes未生效
Containerd镜像加速及私有仓库配置(用户密码和忽略HTTPS)
在Kubernetes集群部署kubesphere
使用KubeKey快速部署Kubernetes集群1.28.8
Rancher 快速创建RKE K8S集群
Kubernetes日志收集方案 EFK Pod部署

发布评论

sunday

文章129 评论0