环境说明
硬件最低要求:4核16G
修改kubesphere/ks-installer:v3.4.1 为 kubesphere/ks-installer:v3.4.1-patch.0
该补丁镜像更新证书有效期到2051年,解决了devops s2i证书2024-02-14过期问题
kubectl set image deployment/ks-installer -n kubesphere-system ks-installer=kubesphere/ks-installer:v3.4.1-patch.0
kubectl get deployment ks-installer -n kubesphere-system -o jsonpath="{.spec.template.spec.containers[0].image}"
root@ks-master01:~# kubectl rollout status deployment/ks-installer -n kubesphere-system
deployment "ks-installer" successfully rolled out # 即成功更新
开启DevOps
https://kubesphere.io/zh/docs/v3.4/pluggable-components/devops/
注:jenkins内存不要小于官方默认值,不然会导致jenkins安装失败或jenkins流水线图形无法显示编辑
kubectl edit cc -n kubesphere-system ks-installer
devops:
enabled: true # 将“false”更改为“true”。
SonarQube 配置
部署sonarqube
节点内存建议 16G,不然内存压力会反复重建。
https://kubesphere.io/zh/docs/v3.4/devops-user-guide/how-to-integrate/sonarqube/#create-sonarqube-token-for-new-project
root@ks-master01:~# helm version
version.BuildInfo{Version:"v3.16.1", GitCommit:"5a5449dc42be07001fd5771d56429132984ab3ab", GitTreeState:"clean", GoVersion:"go1.22.7"}
root@ks-master01:~# helm upgrade --install sonarqube sonarqube --repo https://charts.kubesphere.io/main -n kubesphere-devops-system --create-namespace --set service.type=NodePort
Release "sonarqube" does not exist. Installing it now.
NAME: sonarqube
LAST DEPLOYED: Wed Oct 9 13:28:15 2024
NAMESPACE: kubesphere-devops-system
STATUS: deployed
REVISION: 1
NOTES:
1. Get the application URL by running these commands:
export NODE_PORT=$(kubectl get --namespace kubesphere-devops-system -o jsonpath="{.spec.ports[0].nodePort}" services sonarqube-sonarqube)
export NODE_IP=$(kubectl get nodes --namespace kubesphere-devops-system -o jsonpath="{.items[0].status.addresses[0].address}")
echo http://$NODE_IP:$NODE_PORT
root@ks-master01:~# export NODE_PORT=$(kubectl get --namespace kubesphere-devops-system -o jsonpath="{.spec.ports[0].nodePort}" services sonarqube-sonarqube)
export NODE_IP=$(kubectl get nodes --namespace kubesphere-devops-system -o jsonpath="{.items[0].status.addresses[0].address}")
echo http://$NODE_IP:$NODE_PORT
root@ks-master01:~# kubectl get pod -n kubesphere-devops-system
NAME READY STATUS RESTARTS AGE
devops-28836210-nbd5j 0/1 Completed 0 85m
devops-28836240-sd2hl 0/1 Completed 0 55m
devops-28836270-v64mp 0/1 Completed 0 25m
devops-apiserver-57b84ddf48-mq2sc 1/1 Running 0 5d22h
devops-controller-7779fd7c4f-z2s6v 1/1 Running 0 5d22h
devops-jenkins-8d6f46c68-z9l7f 1/1 Running 0 5d22h
s2ioperator-0 1/1 Running 0 5d22h
sonarqube-postgresql-0 1/1 Running 0 16m
sonarqube-sonarqube-564574684-h7qxv 1/1 Running 0 6m51s
浏览器中访问 SonarQube 控制台 http://<Node IP>:<NodePort>
http://192.168.77.131:31429
使用默认帐户 admin/admin
登录。
创建 SonarQube 管理员令牌 (Token)
点击右上角字母 A,然后从菜单中选择 My Account 以转到 Profile 页面。
点击 Security 并输入令牌名称,如 kubesphere
记录Tokens: 01b1497e7aed7d1edd907ceb44f88a058a42dce7
创建 Webhook 服务器
命令获取 SonarQube Webhook 的地址
export NODE_PORT=$(kubectl get --namespace kubesphere-devops-system -o jsonpath="{.spec.ports[0].nodePort}" services devops-jenkins)
export NODE_IP=$(kubectl get nodes --namespace kubesphere-devops-system -o jsonpath="{.items[0].status.addresses[0].address}")
echo http://$NODE_IP:$NODE_PORT/sonarqube-webhook/
http://192.168.77.131:30180/sonarqube-webhook/
依次点击 Administration、Configuration 和 Webhooks 创建一个 Webhook。
将 SonarQube 配置添加到 ks-installer
kubectl edit cc -n kubesphere-system ks-installer
搜索devops,添加字段 sonarqube 并在其下方指定 externalSonarUrl 和 externalSonarToken。
devops:
enabled: true
jenkinsCpuLim: 2
jenkinsCpuReq: 0.5
jenkinsMemoryLim: 2Gi
jenkinsMemoryReq: 2Gi
jenkinsVolumeSize: 8Gi
sonarqube:
externalSonarToken: 01b1497e7aed7d1edd907ceb44f88a058a42dce7
externalSonarUrl: http://192.168.77.131:31429
Jenkins配置
输出Jenkins访问地址
export NODE_PORT=$(kubectl get --namespace kubesphere-devops-system -o jsonpath="{.spec.ports[0].nodePort}" services devops-jenkins) export NODE_IP=$(kubectl get nodes --namespace kubesphere-devops-system -o jsonpath="{.items[0].status.addresses[0].address}") echo http://$NODE_IP:$NODE_PORT
http://192.168.77.131:30180
请使用地址 http://admin/P@88w0rd
)登录 Jenkins。
注:
有时会出现 修改密码,jenkins 与 kubesphere 的 ldap 同步失败的情况,这时可以 重新修改下 密码,会同步成功。 或者可以先用 默认密码:P@88w0rd 来登录 jenkins。
系统管理-Manage Credentials-Jenkins-全局凭据-添加凭据
secret处填写sonarqube token
系统管理-系统配置
将 sonarqubeURL 添加到 KubeSphere 控制台
kubectl edit cm -n kubesphere-system ks-console-config
client:
version:
kubesphere: v3.3.2
kubernetes: v1.28.8
openpitrix: v3.3.2
enableKubeConfig: true
devops: # 手动添加该字段。
sonarqubeURL: http://192.168.77.131:31429
defaultClusterName: default
重启服务
kubectl -n kubesphere-devops-system rollout restart deploy devops-apiserver
kubectl -n kubesphere-system rollout restart deploy ks-console
Jenkins邮件修改
https://kubesphere.io/zh/docs/v3.3/devops-user-guide/how-to-use/pipelines/jenkins-email/
kubectl edit deployment -n kubesphere-devops-system devops-jenkins
- name: kubernetes.connection.timeout
value: "60000"
- name: kubernetes.request.timeout
value: "60000"
- name: EMAIL_SMTP_HOST
value: smtp.qq.com
- name: EMAIL_SMTP_PORT
value: "465"
- name: EMAIL_USE_SSL
value: "false"
- name: EMAIL_FROM_NAME
value: KubeSphere
- name: EMAIL_FROM_ADDR
value: sundayle@qq.com
- name: EMAIL_FROM_PASS
value: 授权码
image: kubesphere/ks-jenkins:v3.4.0-2.319.3-1
imagePullPolicy: Always
应用后需要等几分钟才可使用
kubectl -n kubesphere-system rollout restart deployment ks-installer
DevOps流水线
克隆java项目 https://github.com/kubesphere/devops-maven-sample 推送到私有gitlab
Git Token推送使用
gitlab token 推送
偏好设置 是全局token
#git clone -b https://{user}:{token}@gitlab.com/xxx/xxx.git
#git set-url origin https://{user}:{token}@gitlab.com/xxx/xxx.git
git push https://{user}:{token}@gitlab.com/xxx/xxx.git
github token 推送
git push https://{token}@github.com/xxx/xxx.git
构建参数:
- REGISTRY # Harbor仓库域名
- DOCKER_NAMESPACE # 名称空间
- APP_NAME
- TAG_NAME
- GITLAB_ACCOUNT
- GITLAB_DOMAIN
流水线代码
pipeline {
agent {
node {
label 'maven'
}
}
stages {
stage('Checkout SCM') {
agent none
steps {
git(url: 'http://git.sundayhk.com/zsp/kubesphere-devops-maven-sample.git', credentialsId: 'gitlab-id', branch: 'master', changelog: true, poll: false)
}
}
stage('Unit Test') {
agent none
steps {
container('maven') {
sh 'mvn clean test'
}
}
}
stage('Code Analysis') {
agent none
steps {
container('maven') {
withCredentials([string(credentialsId: 'sonar-token', variable: 'SONAR_TOKEN')]) {
withSonarQubeEnv('sonar') {
sh 'mvn sonar:sonar -Dsonar.login=$SONAR_TOKEN'
}
}
timeout(unit: 'HOURS', activity: true, time: 1) {
waitForQualityGate 'true'
}
}
}
}
stage('Build And Push') {
agent none
steps {
container('maven') {
sh 'mvn -Dmaven.test.skip=true clean package'
withCredentials([usernamePassword(credentialsId: 'harbor-id', passwordVariable: 'HARBOR_PASSWORD', usernameVariable: 'HARBOR_USERNAME')]) {
sh 'podman build -f Dockerfile-online -t $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:SNAPSHOT-$BUILD_NUMBER .'
sh 'echo "$HARBOR_PASSWORD" | podman login $REGISTRY -u "$HARBOR_USERNAME" --password-stdin'
sh 'podman push $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:SNAPSHOT-$BUILD_NUMBER'
}
}
}
}
stage('Artifacts') {
agent none
steps {
archiveArtifacts(artifacts: 'target/*.jar', followSymlinks: false)
}
}
stage('Deploy to Dev') {
agent none
steps {
input(message: 'deploy to dev? @project-admin', submitter: '')
container('maven') {
withCredentials([kubeconfigContent(credentialsId: 'demo-kubeconfig', variable: 'KUBECONFIG_CONTENT')]) {
sh '''mkdir ~/.kube
echo "$KUBECONFIG_CONTENT" > ~/.kube/config
envsubst < deploy/no-branch-dev/devops-sample-svc.yaml | kubectl apply -f -
envsubst < deploy/no-branch-dev/devops-sample.yaml | kubectl apply -f -'''
}
}
}
}
stage('Push with tag') {
agent none
when {
expression {
return params.TAG_NAME =~ /v.*/
}
}
steps {
container('maven') {
input(message: 'release image prod?@project-admin', submitter: '')
withCredentials([string(credentialsId: 'gitlab-token', variable: 'GITLAB_TOKEN')]) {
sh '''git config --global user.email "sundayhk@qq.com"
git config --global user.name "sundayhk"
git tag -a $TAG_NAME -m "$TAG_NAME"
git remote set-url origin http://$GITLAB_ACCOUNT:$GITLAB_TOKEN@$GITLAB_DOMAIN/$GITLAB_ACCOUNT/$APP_NAME.git
git push --tags --ipv4'''
sh 'podman tag $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:SNAPSHOT-$BUILD_NUMBER $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:$TAG_NAME'
sh 'podman push $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:$TAG_NAME'
}
}
}
}
stage('Deploy to Prod') {
agent none
when {
expression {
return params.TAG_NAME =~ /v.*/
}
}
steps {
input(message: 'deploy to prod?@project-admin', submitter: '')
container('maven') {
withCredentials([kubeconfigContent(credentialsId: 'demo-kubeconfig', variable: 'KUBECONFIG_CONTENT')]) {
sh 'envsubst < deploy/prod-all-in-one/devops-sample.yaml | kubectl apply -f -'
}
}
}
}
}
}
报错解决
sonarqube 报错
[INFO] Analysis total time: 7.830 s
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 10.370 s
[INFO] Finished at: 2024-10-30T06:48:35Z
[INFO] ------------------------------------------------------------------------
Wait for SonarQube analysis to be completed and return quality gate status58 ms失败
Checking status of SonarQube task 'AZLcLuu1VQF99l2mwYaO' on server 'sonar'
java.lang.IllegalStateException: Unable to parse response from http://192.168.77.131:31429//api/ce/task?id=AZLcLuu1VQF99l2mwYaO:
<!doctype html><html lang="en"><head><meta http-equiv="content-type" content="text/html; charset=UTF-8" charset="UTF-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge">
原因是 jenkins 配置SonarQube Server URL 最后多出/
将http://192.168.77.131:31429/
修改为 http://192.168.77.131:31429
即可
maven 提示没有docker
解决: 使用podman
![Uploading file...m50uj]()