以前常做的事是自行弄個 Jenkins 在搭配 Ansible 連續技完成 CI/CD 的任務，今年要來推廣 Docker 生態，把簡單的任務包裝成 Docker 降低人員維護的成本，在此挑選 Gitlab.com 作為出發點，接著會使用 Gitlab CI/CD 發布到 Amazon Elastic Container Service。這篇筆記不限於 Laravel framework ，但以他當作範例來進行。
- Deploy to Amazon Elastic Container Service
- Deploy to AWS from GitLab CI/CD
首先就是弄個 gitlab project，可以是 Create blank project 也可以是 Create from template，接著就到 Settings -> CI/CD 欣賞一下有哪些設定選項
General pipelines
Auto DevOps
Runners
Artifacts
Variables
Pipeline trigger tokens
Deploy freezes
Job token permissions
Secure files
在 General pipelines 介面上，可以看到滿多設定的，但整體上起源是在 "Deploy to AWS from GitLab CI/CD" 這篇文章中，整個精髓是在 .gitlab-ci.yml 檔案上，因此可以參考 "Create from template" 的素材，當下可以按 preview 先欣賞一下，多參考幾個就有感覺了：
簡言之 .gitlab-ci.yml 就是定義想做的指令，並且可以指定要用哪個環境(docker image)來運行這些指令。例如 Laravel 範例使用的 image 是 edbizarro/gitlab-ci-pipeline-php:latest 位置，對應的就是在 https://hub.docker.com/r/edbizarro/gitlab-ci-pipeline-php 和 https://github.com/edbizarro/gitlab-ci-pipeline-php ，搞懂後就來建立自己的 .gitlab-ci.yml 檔案，目標是直接依照專案內的 Dockerfile 來編譯 Docker Images：
variables:
DOCKER_IMAGE_NAME_FOR_LARAVEL_PROJECT: "myapp-php-fpm"
DOCKER_IMAGE_NAME_FOR_NGINX: "myapp-nginx"
DOCKER_TLS_CERTDIR: "/certs"
stages:
- build
build:
stage: build
image: docker:latest
services:
- docker:dind
script:
- ls -la $DOCKER_TLS_CERTDIR
- docker info
- docker build -t $DOCKER_IMAGE_NAME_FOR_LARAVEL_PROJECT:latest -f docker/php-fpm/Dockerfile .
- docker build -t $DOCKER_IMAGE_NAME_FOR_NGINX:latest -f docker/nginx/Dockerfile .
- docker images
- pwd
- ls -la
- docker save $DOCKER_IMAGE_NAME_FOR_LARAVEL_PROJECT:latest > $DOCKER_IMAGE_NAME_FOR_LARAVEL_PROJECT.image.tar
- docker save $DOCKER_IMAGE_NAME_FOR_NGINX:latest > $DOCKER_IMAGE_NAME_FOR_NGINX.image.tar
- ls -la
artifacts:
paths:
- $DOCKER_IMAGE_NAME_FOR_LARAVEL_PROJECT.image.tar
- $DOCKER_IMAGE_NAME_FOR_NGINX.image.tar
expire_in: 1 week
rules:
- changes:
- Dockerfile
- docker/**/*
- if: '$CI_PIPELINE_SOURCE == "web"'
- if: '$CI_PIPELINE_SOURCE == "api"'
#tags:
# - docker
如此，回到 Gitlab Project 左邊的 Build 項目，就可以在 Pipelines 去觸發 New pipeline 來觸發 Build docker images ，並且會把產出的 *.image.tar 壓縮起來供下載。此外，在 Build 內的 Pipeline editor 也是滿好用的架構，可以線上編輯 .gitlab-ci.yml 檔案內容，還可以驗證格式內容是否有錯誤。
接下來又做更多事，像是 Laravel framework 相關專案設定和初始化（包括 npm install && npm run build && php artisan optimize）:
variables:
DOCKER_IMAGE_NAME_FOR_LARAVEL_PROJECT: "myapp-php-fpm"
DOCKER_IMAGE_NAME_FOR_NGINX: "myapp-nginx"
DOCKER_TLS_CERTDIR: "/certs"
stages:
- prepare
- build
composer:
stage: prepare
image: php:8.4-cli
before_script:
- apt-get update && apt-get install -y unzip libzip-dev
- docker-php-ext-install zip
- curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer
script:
# https://gitlab.com/gitlab-org/project-templates/laravel/-/blob/main/.gitlab-ci.yml?ref_type=heads
- cd www
- composer install --no-dev --optimize-autoloader
#- php artisan optimize:clear
- php artisan optimize
artifacts:
paths:
- www/
expire_in: 1 day
rules:
- changes:
- www/**/*
- if: '$CI_PIPELINE_SOURCE == "web"'
- if: '$CI_PIPELINE_SOURCE == "api"'
npm:
stage: prepare
image: node:22
dependencies:
- composer
script:
- cd www
- npm install
- npm run build
artifacts:
paths:
- www/node_modules/
- www/public/build/
expire_in: 1 day
rules:
- changes:
- www/**/*
- if: '$CI_PIPELINE_SOURCE == "web"'
- if: '$CI_PIPELINE_SOURCE == "api"'
build:
stage: build
image: docker:latest
services:
- docker:dind
dependencies:
- composer
- npm
script:
- docker info
- docker build -t $DOCKER_IMAGE_NAME_FOR_LARAVEL_PROJECT:latest -f docker/php-fpm/Dockerfile .
- docker build -t $DOCKER_IMAGE_NAME_FOR_NGINX:latest -f docker/nginx/Dockerfile .
- docker save $DOCKER_IMAGE_NAME_FOR_LARAVEL_PROJECT:latest > $DOCKER_IMAGE_NAME_FOR_LARAVEL_PROJECT.image.tar
- docker save $DOCKER_IMAGE_NAME_FOR_NGINX:latest > $DOCKER_IMAGE_NAME_FOR_NGINX.image.tar
artifacts:
paths:
- $DOCKER_IMAGE_NAME_FOR_LARAVEL_PROJECT.image.tar
- $DOCKER_IMAGE_NAME_FOR_NGINX.image.tar
expire_in: 1 week
rules:
- changes:
- Dockerfile
- docker/**/*
- www/**/* # 加入 www 目錄的變更觸發
- if: '$CI_PIPELINE_SOURCE == "web"'
- if: '$CI_PIPELINE_SOURCE == "api"'
最後，再來增加把 Docker images 發布到 AWS ECS ，目前應當有兩招可以發布：
- 使用 Gitlab Container Registry
- 使用 AWS ECS Private Registry
先試試看 Gitlab Container Registry ，過程就是在 Build stage 中，增加登入到 Gitlab Container Registry 機制:
before_script:
- echo "$CI_REGISTRY_PASSWORD" | docker login -u "$CI_REGISTRY_USER" "$CI_REGISTRY" --password-stdin
實際 Job 運行會看到這段資訊：
$ echo "$CI_REGISTRY_PASSWORD" | docker login -u "$CI_REGISTRY_USER" "$CI_REGISTRY" --password-stdin
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credential-stores
Login Succeeded
接著再把這些串起來，在 build stage 時，就把產生的 docker image 送進 Gitlab Container Registry，並且在 Gitlab Project -> Deploy -> Container Registry 看到成果：
variables:
DOCKER_IMAGE_NAME_FOR_LARAVEL_PROJECT: "myapp-php-fpm"
DOCKER_IMAGE_NAME_FOR_NGINX: "myapp-nginx"
DOCKER_TLS_CERTDIR: "/certs"
PHP_IMAGE_PATH: "$CI_REGISTRY_IMAGE/$DOCKER_IMAGE_NAME_FOR_LARAVEL_PROJECT"
NGINX_IMAGE_PATH: "$CI_REGISTRY_IMAGE/$DOCKER_IMAGE_NAME_FOR_NGINX"
stages:
- prepare
- build
composer:
stage: prepare
image: php:8.4-cli
before_script:
- apt-get update && apt-get install -y unzip libzip-dev
- docker-php-ext-install zip
- curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer
script:
# https://gitlab.com/gitlab-org/project-templates/laravel/-/blob/main/.gitlab-ci.yml?ref_type=heads
- cd www
- composer install --no-dev --optimize-autoloader
#- php artisan optimize:clear
- php artisan optimize
artifacts:
paths:
- www/
expire_in: 1 day
rules:
- changes:
- www/**/*
- if: '$CI_PIPELINE_SOURCE == "web"'
- if: '$CI_PIPELINE_SOURCE == "api"'
npm:
stage: prepare
image: node:22
dependencies:
- composer
script:
- cd www
- npm install
- npm run build
artifacts:
paths:
- www/node_modules/
- www/public/build/
expire_in: 1 day
rules:
- changes:
- www/**/*
- if: '$CI_PIPELINE_SOURCE == "web"'
- if: '$CI_PIPELINE_SOURCE == "api"'
build:
stage: build
image: docker:latest
services:
- docker:dind
dependencies:
- composer
- npm
before_script:
- echo "$CI_REGISTRY_PASSWORD" | docker login -u "$CI_REGISTRY_USER" "$CI_REGISTRY" --password-stdin
script:
- docker info
- docker build -t $DOCKER_IMAGE_NAME_FOR_LARAVEL_PROJECT:latest -f docker/php-fpm/Dockerfile .
- docker build -t $DOCKER_IMAGE_NAME_FOR_NGINX:latest -f docker/nginx/Dockerfile .
- docker save $DOCKER_IMAGE_NAME_FOR_LARAVEL_PROJECT:latest > $DOCKER_IMAGE_NAME_FOR_LARAVEL_PROJECT.image.tar
- docker save $DOCKER_IMAGE_NAME_FOR_NGINX:latest > $DOCKER_IMAGE_NAME_FOR_NGINX.image.tar
- docker tag $DOCKER_IMAGE_NAME_FOR_LARAVEL_PROJECT:latest $PHP_IMAGE_PATH:$CI_COMMIT_SHA
- docker tag $DOCKER_IMAGE_NAME_FOR_LARAVEL_PROJECT:latest $PHP_IMAGE_PATH:latest
- docker tag $DOCKER_IMAGE_NAME_FOR_NGINX:latest $NGINX_IMAGE_PATH:$CI_COMMIT_SHA
- docker tag $DOCKER_IMAGE_NAME_FOR_NGINX:latest $NGINX_IMAGE_PATH:latest
- docker push $PHP_IMAGE_PATH:$CI_COMMIT_SHA
- docker push $PHP_IMAGE_PATH:latest
- docker push $NGINX_IMAGE_PATH:$CI_COMMIT_SHA
- docker push $NGINX_IMAGE_PATH:latest
artifacts:
paths:
- $DOCKER_IMAGE_NAME_FOR_LARAVEL_PROJECT.image.tar
- $DOCKER_IMAGE_NAME_FOR_NGINX.image.tar
expire_in: 1 week
rules:
- changes:
- Dockerfile
- docker/**/*
- www/**/* # 加入 www 目錄的變更觸發
- if: '$CI_PIPELINE_SOURCE == "web"'
- if: '$CI_PIPELINE_SOURCE == "api"'
如此就只需要到 AWS ECS 後台去建立 Task 改從 Gitlab Container Registry 取資料即可完成發佈，這個架構的優勢是避免在 Gitlab.com 關聯太多 AWS 存取方式，也可以避免 AWS 相關資訊外洩，若要更加保護的話，可以連 Gitlab CI/CD 的 Runner 都挑自己的機器會更穩。
最後，還有一點要留意的，對 Gitlab.com 免費服務有 5GB 空間的限制，假設 Docker Image 隨便就佔了幾百GB，那發佈幾次後就會沒空間了，因此，再補上一段只保留最後三版 Docker Image 的機制：
cleanup_images:
stage: cleanup
image: docker:latest
variables:
GIT_STRATEGY: none
before_script:
- apk add --no-cache curl jq
- echo "CI_API_V4_URL = ${CI_API_V4_URL}"
- echo "CI_PROJECT_ID = ${CI_PROJECT_ID}"
# https://docs.gitlab.com/ee/api/container_registry.html
- echo "CI_REGISTRY_PASSWORD = ${CI_REGISTRY_PASSWORD}"
- echo "CI_JOB_TOKEN = ${CI_JOB_TOKEN}"
- echo "API REQUEST URL = ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/registry/repositories"
- |
curl --header "JOB-TOKEN: ${CI_JOB_TOKEN}" "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/registry/repositories" | jq '.'
script:
- docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
- |
REGISTRY_TARGET=$(curl --header "JOB-TOKEN: ${CI_JOB_TOKEN}" "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/registry/repositories" | jq -r '. | if length > 0 then .[].id else empty end')
for REGISTRY_ID in $REGISTRY_TARGET; do
echo "REGISTRY_ID = $REGISTRY_ID"
curl --header "JOB-TOKEN: ${CI_JOB_TOKEN}" "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/registry/repositories/${REGISTRY_ID}/tags"
REGISTRY_LAST_TAG=$(curl --header "JOB-TOKEN: ${CI_JOB_TOKEN}" "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/registry/repositories/${REGISTRY_ID}/tags" | jq -r '[ .[] | select(.name != "latest") ] | sort_by(.name) | if length > 3 then .[:-3] | .[].name else empty end')
for REGISTRY_TAG in $REGISTRY_LAST_TAG; do
echo "REGISTRY_ID = $REGISTRY_ID, REGISTRY_TAG = $REGISTRY_TAG"
curl --request DELETE --header "JOB-TOKEN: ${CI_JOB_TOKEN}" "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/registry/repositories/${REGISTRY_ID}/tags/${REGISTRY_TAG}"
done
done
rules:
- when: on_success
主要是透過 CI_JOB_TOKEN 跟 Gitlab API 溝通，像是取出該專案下有幾個 Container Registry ，並且善用 docker tag 時，把 tag prefix 增加 timestamp ，方便此時 sorting 並挑選舊的 tag 且移除他們，如此就可以節省 Gitlab Project 使用空間。
至於在 AWS ECS 的設定，就主要先看 Gitlab 跟 AWS ECS 官方文件了，由於 Docker Image 我們擺在 Gitlab.com，這時 AWS ECS -> Cluster -> Task ，在定義 Container 要從哪邊 Container Registry 取出時，要寫入 registry.gitlab.com/YourGitlabGroup/YourGitlabProject/YourContainerRegistryProject:latest 等類似的位置，下一刻則是 AWS ECS 運行時，此時 ecsTaskExecutionRole 怎麼存取的議題
流程：
- 在 Gitlab Project 上，先建立一個 Deploy Token，會拿到一個帳號跟密碼
- Settings -> Repository -> Deploy tokens -> Add token
- 在 AWS Secrets Manager 上，添加一組，選擇其他類型項目即可
- 假設產出是 arn:aws:secretsmanager:ap-northeast-1:#####:secret:gitlab.com-container-registry
- 在 AWS Identity and Access Management (IAM) 上，找到 ecsTaskExecutionRole ，幫它添加可以存取 Secrets Manager 的權限
- 開放權限可以只提供 READOnly ，甚至細微到只能對 arn:aws:secretsmanager:ap-northeast-1:#####:secret:gitlab.com-container-registry 操作
- 沒設定好的錯誤訊息：
- ResourceInitializationError: unable to pull secrets or registry auth: execution resource retrieval failed: unable to get registry auth from asm: service call has been retried 1 time(s): failed to fetch secret arn:aws:secretsmanager:ap-northeast-1:#####:secret:gitlab.com-container-registry from secrets manager: AccessDeniedException: User: arn:aws:sts::#####:assumed-role/ecsTaskExecutionRole/##### is not authorized to perform: secretsmanager:GetSecretValue on resource: arn:aws:secretsmanager:ap-northeast-1:#####:secret:gitlab.com-container-registry because no identity-based policy allows the secretsmanager:GetSecretValue action status code: 400
- 設定範例 JSON
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"secretsmanager:GetRandomPassword",
"secretsmanager:GetResourcePolicy",
"secretsmanager:GetSecretValue",
"secretsmanager:DescribeSecret",
"secretsmanager:ListSecretVersionIds",
"secretsmanager:ListSecrets",
"secretsmanager:BatchGetSecretValue"
],
"Resource": "*"
}
]
}
- 在 AWS ECS 的 Task 就可以好好設定 Container 的存取機制
- 私有登錄檔 -> Secrets Manager ARN 或名稱 -> arn:aws:secretsmanager:ap-northeast-1:#####:secret:gitlab.com-container-registry
如此在 AWS ECS -> Cluster -> Create Service 時，就會觸發 AWS ECS Task 運行，就可以看到 Docker Container 被創立出來
目前研究先告一個段落，在 AWS ECS 的設置規劃其實也要不小的篇幅的。
再碎碎念一下，直接實驗 AWS ECS 直接去跟 Gitlab.com Container Registry 要資料時，有時會出現彷彿無法正常取出的問題，這個我還不是很清楚排除的方式，目前還有偷懶隔天再弄一次，一樣的方式又正常可通過 Orz 或許最佳還是改用 AWS ECS Private Container Registry 維護，這樣服務發布架構可以省去一些不穩定的額外煩惱。
類似錯誤訊息：
- esourceInitializationError: unable to pull secrets or registry auth: unable to get registry auth from asm: There is a connection issue between the task and AWS Secrets Manager.
數個資安提問：
- 假設你的 Docker Images 是在 Gitlab.com 產出，要 docker push 到 AWS ECS Container Registry 時，就會要面對在 Gitlab.com 上暴露 AWS 存取的資訊
- 假設 Build docker images 是用 Gitlab.com 機器(Runner)，對應的也是有敏感資訊會在這些機器上，你是否信任他們？
- 假設 Build docker images 時，用了其他人製作的 Docker images ，你的敏感資訊是否有妥善保護？惡意的 docker images 可以在啟動或結束時，把用戶的敏感資訊傳遞出去
ref:
