2025年2月12日 星期三

Docker 開發筆記 - 從 Gitlab 觸發 CI/CD，製作 Docker Image 並發布到 Amazon Elastic Container Service (AWS ECS)


以前常做的事是自行弄個 Jenkins 在搭配 Ansible 連續技完成 CI/CD 的任務，今年要來推廣 Docker 生態，把簡單的任務包裝成 Docker 降低人員維護的成本，在此挑選 Gitlab.com 作為出發點，接著會使用 Gitlab CI/CD 發布到 Amazon Elastic Container Service。這篇筆記不限於 Laravel framework ，但以他當作範例來進行。
首先就是弄個 gitlab project，可以是 Create blank project 也可以是 Create from template，接著就到 Settings -> CI/CD 欣賞一下有哪些設定選項

General pipelines
Auto DevOps
Runners
Artifacts
Variables
Pipeline trigger tokens
Deploy freezes
Job token permissions
Secure files

在 General pipelines 介面上，可以看到滿多設定的，但整體上起源是在 "Deploy to AWS from GitLab CI/CD" 這篇文章中，整個精髓是在 .gitlab-ci.yml 檔案上，因此可以參考 "Create from template" 的素材，當下可以按 preview 先欣賞一下，多參考幾個就有感覺了：

簡言之 .gitlab-ci.yml 就是定義想做的指令，並且可以指定要用哪個環境(docker image)來運行這些指令。例如 Laravel 範例使用的 image 是 edbizarro/gitlab-ci-pipeline-php:latest 位置，對應的就是在 https://hub.docker.com/r/edbizarro/gitlab-ci-pipeline-php 和 https://github.com/edbizarro/gitlab-ci-pipeline-php ，搞懂後就來建立自己的 .gitlab-ci.yml 檔案，目標是直接依照專案內的 Dockerfile 來編譯 Docker Images：

variables:
  DOCKER_IMAGE_NAME_FOR_LARAVEL_PROJECT: "myapp-php-fpm"
  DOCKER_IMAGE_NAME_FOR_NGINX: "myapp-nginx"
  DOCKER_TLS_CERTDIR: "/certs"

stages:
  - build

build:
  stage: build
  image: docker:latest
  services:
    - docker:dind
  script:
    - ls -la $DOCKER_TLS_CERTDIR 
    - docker info
    - docker build -t $DOCKER_IMAGE_NAME_FOR_LARAVEL_PROJECT:latest -f docker/php-fpm/Dockerfile .
    - docker build -t $DOCKER_IMAGE_NAME_FOR_NGINX:latest -f docker/nginx/Dockerfile .
    - docker images
    - pwd
    - ls -la
    - docker save $DOCKER_IMAGE_NAME_FOR_LARAVEL_PROJECT:latest > $DOCKER_IMAGE_NAME_FOR_LARAVEL_PROJECT.image.tar
    - docker save $DOCKER_IMAGE_NAME_FOR_NGINX:latest > $DOCKER_IMAGE_NAME_FOR_NGINX.image.tar
    - ls -la
  artifacts:
    paths:
      - $DOCKER_IMAGE_NAME_FOR_LARAVEL_PROJECT.image.tar
      - $DOCKER_IMAGE_NAME_FOR_NGINX.image.tar
    expire_in: 1 week
  rules:
    - changes:
        - Dockerfile
        - docker/**/*
    - if: '$CI_PIPELINE_SOURCE == "web"'
    - if: '$CI_PIPELINE_SOURCE == "api"'
  #tags:
  #  - docker


如此，回到 Gitlab Project 左邊的 Build 項目，就可以在 Pipelines 去觸發 New pipeline 來觸發 Build docker images ，並且會把產出的 *.image.tar 壓縮起來供下載。此外，在 Build 內的 Pipeline editor 也是滿好用的架構，可以線上編輯 .gitlab-ci.yml 檔案內容，還可以驗證格式內容是否有錯誤。

接下來又做更多事，像是 Laravel framework 相關專案設定和初始化（包括 npm install && npm run build && php artisan optimize）:

variables:
  DOCKER_IMAGE_NAME_FOR_LARAVEL_PROJECT: "myapp-php-fpm"
  DOCKER_IMAGE_NAME_FOR_NGINX: "myapp-nginx"
  DOCKER_TLS_CERTDIR: "/certs"

stages:
  - prepare
  - build

composer:
  stage: prepare
  image: php:8.4-cli
  before_script:
    - apt-get update && apt-get install -y unzip libzip-dev
    - docker-php-ext-install zip
    - curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer
  script:
    # https://gitlab.com/gitlab-org/project-templates/laravel/-/blob/main/.gitlab-ci.yml?ref_type=heads
    - cd www
    - composer install --no-dev --optimize-autoloader
    #- php artisan optimize:clear
    - php artisan optimize
  artifacts:
    paths:
      - www/
    expire_in: 1 day
  rules:
    - changes:
        - www/**/*
    - if: '$CI_PIPELINE_SOURCE == "web"'
    - if: '$CI_PIPELINE_SOURCE == "api"'

npm:
  stage: prepare
  image: node:22
  dependencies:
    - composer
  script:
    - cd www
    - npm install
    - npm run build
  artifacts:
    paths:
      - www/node_modules/
      - www/public/build/
    expire_in: 1 day
  rules:
    - changes:
        - www/**/*
    - if: '$CI_PIPELINE_SOURCE == "web"'
    - if: '$CI_PIPELINE_SOURCE == "api"'

build:
  stage: build
  image: docker:latest
  services:
    - docker:dind
  dependencies:
    - composer
    - npm
  script:
    - docker info
    - docker build -t $DOCKER_IMAGE_NAME_FOR_LARAVEL_PROJECT:latest -f docker/php-fpm/Dockerfile .
    - docker build -t $DOCKER_IMAGE_NAME_FOR_NGINX:latest -f docker/nginx/Dockerfile .
    - docker save $DOCKER_IMAGE_NAME_FOR_LARAVEL_PROJECT:latest > $DOCKER_IMAGE_NAME_FOR_LARAVEL_PROJECT.image.tar
    - docker save $DOCKER_IMAGE_NAME_FOR_NGINX:latest > $DOCKER_IMAGE_NAME_FOR_NGINX.image.tar
  artifacts:
    paths:
      - $DOCKER_IMAGE_NAME_FOR_LARAVEL_PROJECT.image.tar
      - $DOCKER_IMAGE_NAME_FOR_NGINX.image.tar
    expire_in: 1 week
  rules:
    - changes:
        - Dockerfile
        - docker/**/*
        - www/**/*  # 加入 www 目錄的變更觸發
    - if: '$CI_PIPELINE_SOURCE == "web"'
    - if: '$CI_PIPELINE_SOURCE == "api"'

最後，再來增加把 Docker images 發布到 AWS ECS ，目前應當有兩招可以發布：

先試試看 Gitlab Container Registry ，過程就是在 Build stage 中，增加登入到 Gitlab Container Registry 機制:

before_script:
  - echo "$CI_REGISTRY_PASSWORD" | docker login -u "$CI_REGISTRY_USER" "$CI_REGISTRY" --password-stdin

實際 Job 運行會看到這段資訊：

$ echo "$CI_REGISTRY_PASSWORD" | docker login -u "$CI_REGISTRY_USER" "$CI_REGISTRY" --password-stdin
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credential-stores
Login Succeeded

接著再把這些串起來，在 build stage 時，就把產生的 docker image 送進 Gitlab Container Registry，並且在 Gitlab Project -> Deploy -> Container Registry 看到成果：

variables:
  DOCKER_IMAGE_NAME_FOR_LARAVEL_PROJECT: "myapp-php-fpm"
  DOCKER_IMAGE_NAME_FOR_NGINX: "myapp-nginx"
  DOCKER_TLS_CERTDIR: "/certs"
  PHP_IMAGE_PATH: "$CI_REGISTRY_IMAGE/$DOCKER_IMAGE_NAME_FOR_LARAVEL_PROJECT"
  NGINX_IMAGE_PATH: "$CI_REGISTRY_IMAGE/$DOCKER_IMAGE_NAME_FOR_NGINX"

stages:
  - prepare
  - build

composer:
  stage: prepare
  image: php:8.4-cli
  before_script:
    - apt-get update && apt-get install -y unzip libzip-dev
    - docker-php-ext-install zip
    - curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer
  script:
    # https://gitlab.com/gitlab-org/project-templates/laravel/-/blob/main/.gitlab-ci.yml?ref_type=heads
    - cd www
    - composer install --no-dev --optimize-autoloader
    #- php artisan optimize:clear
    - php artisan optimize
  artifacts:
    paths:
      - www/
    expire_in: 1 day
  rules:
    - changes:
        - www/**/*
    - if: '$CI_PIPELINE_SOURCE == "web"'
    - if: '$CI_PIPELINE_SOURCE == "api"'

npm:
  stage: prepare
  image: node:22
  dependencies:
    - composer
  script:
    - cd www
    - npm install
    - npm run build
  artifacts:
    paths:
      - www/node_modules/
      - www/public/build/
    expire_in: 1 day
  rules:
    - changes:
        - www/**/*
    - if: '$CI_PIPELINE_SOURCE == "web"'
    - if: '$CI_PIPELINE_SOURCE == "api"'

build:
  stage: build
  image: docker:latest
  services:
    - docker:dind
  dependencies:
    - composer
    - npm
  before_script:
    - echo "$CI_REGISTRY_PASSWORD" | docker login -u "$CI_REGISTRY_USER" "$CI_REGISTRY" --password-stdin
  script:
    - docker info
    - docker build -t $DOCKER_IMAGE_NAME_FOR_LARAVEL_PROJECT:latest -f docker/php-fpm/Dockerfile .
    - docker build -t $DOCKER_IMAGE_NAME_FOR_NGINX:latest -f docker/nginx/Dockerfile .
    - docker save $DOCKER_IMAGE_NAME_FOR_LARAVEL_PROJECT:latest > $DOCKER_IMAGE_NAME_FOR_LARAVEL_PROJECT.image.tar
    - docker save $DOCKER_IMAGE_NAME_FOR_NGINX:latest > $DOCKER_IMAGE_NAME_FOR_NGINX.image.tar
    - docker tag $DOCKER_IMAGE_NAME_FOR_LARAVEL_PROJECT:latest $PHP_IMAGE_PATH:$CI_COMMIT_SHA
    - docker tag $DOCKER_IMAGE_NAME_FOR_LARAVEL_PROJECT:latest $PHP_IMAGE_PATH:latest
    - docker tag $DOCKER_IMAGE_NAME_FOR_NGINX:latest $NGINX_IMAGE_PATH:$CI_COMMIT_SHA
    - docker tag $DOCKER_IMAGE_NAME_FOR_NGINX:latest $NGINX_IMAGE_PATH:latest
    - docker push $PHP_IMAGE_PATH:$CI_COMMIT_SHA
    - docker push $PHP_IMAGE_PATH:latest
    - docker push $NGINX_IMAGE_PATH:$CI_COMMIT_SHA
    - docker push $NGINX_IMAGE_PATH:latest
  artifacts:
    paths:
      - $DOCKER_IMAGE_NAME_FOR_LARAVEL_PROJECT.image.tar
      - $DOCKER_IMAGE_NAME_FOR_NGINX.image.tar
    expire_in: 1 week
  rules:
    - changes:
        - Dockerfile
        - docker/**/*
        - www/**/*  # 加入 www 目錄的變更觸發
    - if: '$CI_PIPELINE_SOURCE == "web"'
    - if: '$CI_PIPELINE_SOURCE == "api"'

如此就只需要到 AWS ECS 後台去建立 Task 改從 Gitlab Container Registry 取資料即可完成發佈，這個架構的優勢是避免在 Gitlab.com 關聯太多 AWS 存取方式，也可以避免 AWS 相關資訊外洩，若要更加保護的話，可以連 Gitlab CI/CD 的 Runner 都挑自己的機器會更穩。



最後，還有一點要留意的，對 Gitlab.com 免費服務有 5GB 空間的限制，假設 Docker Image 隨便就佔了幾百GB，那發佈幾次後就會沒空間了，因此，再補上一段只保留最後三版 Docker Image 的機制：

cleanup_images:
  stage: cleanup
  image: docker:latest
  variables:
    GIT_STRATEGY: none
  before_script:
    - apk add --no-cache curl jq
    - echo "CI_API_V4_URL = ${CI_API_V4_URL}"
    - echo "CI_PROJECT_ID = ${CI_PROJECT_ID}"
    # https://docs.gitlab.com/ee/api/container_registry.html
    - echo "CI_REGISTRY_PASSWORD = ${CI_REGISTRY_PASSWORD}"
    - echo "CI_JOB_TOKEN = ${CI_JOB_TOKEN}" 
    - echo "API REQUEST URL = ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/registry/repositories"
    - |
      curl --header "JOB-TOKEN: ${CI_JOB_TOKEN}" "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/registry/repositories" | jq '.' 
  script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    - |
      REGISTRY_TARGET=$(curl --header "JOB-TOKEN: ${CI_JOB_TOKEN}" "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/registry/repositories" | jq -r '. | if length > 0 then .[].id else empty end')
      for REGISTRY_ID in $REGISTRY_TARGET; do
        echo "REGISTRY_ID = $REGISTRY_ID"
        curl --header "JOB-TOKEN: ${CI_JOB_TOKEN}" "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/registry/repositories/${REGISTRY_ID}/tags"
        REGISTRY_LAST_TAG=$(curl --header "JOB-TOKEN: ${CI_JOB_TOKEN}" "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/registry/repositories/${REGISTRY_ID}/tags" | jq -r '[ .[] | select(.name != "latest") ] | sort_by(.name) | if length > 3 then .[:-3] | .[].name else empty end')
        for REGISTRY_TAG in $REGISTRY_LAST_TAG; do
          echo "REGISTRY_ID = $REGISTRY_ID, REGISTRY_TAG = $REGISTRY_TAG"
          curl --request DELETE  --header "JOB-TOKEN: ${CI_JOB_TOKEN}" "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/registry/repositories/${REGISTRY_ID}/tags/${REGISTRY_TAG}"
        done
      done
  rules:
    - when: on_success

主要是透過 CI_JOB_TOKEN 跟 Gitlab API 溝通，像是取出該專案下有幾個 Container Registry ，並且善用 docker tag 時，把 tag prefix 增加 timestamp ，方便此時 sorting 並挑選舊的 tag 且移除他們，如此就可以節省 Gitlab Project 使用空間。

至於在 AWS ECS 的設定，就主要先看 Gitlab 跟 AWS ECS 官方文件了，由於 Docker Image 我們擺在 Gitlab.com，這時 AWS ECS -> Cluster -> Task ，在定義 Container 要從哪邊 Container Registry 取出時，要寫入 registry.gitlab.com/YourGitlabGroup/YourGitlabProject/YourContainerRegistryProject:latest 等類似的位置，下一刻則是 AWS ECS 運行時，此時 ecsTaskExecutionRole 怎麼存取的議題

流程：
  1. 在 Gitlab Project 上，先建立一個 Deploy Token，會拿到一個帳號跟密碼
    • Settings -> Repository -> Deploy tokens -> Add token
  2. 在 AWS Secrets Manager 上，添加一組，選擇其他類型項目即可


    • 假設產出是 arn:aws:secretsmanager:ap-northeast-1:#####:secret:gitlab.com-container-registry
  3. 在 AWS Identity and Access Management (IAM) 上，找到 ecsTaskExecutionRole ，幫它添加可以存取 Secrets Manager 的權限
    • 開放權限可以只提供 READOnly ，甚至細微到只能對 arn:aws:secretsmanager:ap-northeast-1:#####:secret:gitlab.com-container-registry 操作
    • 沒設定好的錯誤訊息：
      • ResourceInitializationError: unable to pull secrets or registry auth: execution resource retrieval failed: unable to get registry auth from asm: service call has been retried 1 time(s): failed to fetch secret arn:aws:secretsmanager:ap-northeast-1:#####:secret:gitlab.com-container-registry from secrets manager: AccessDeniedException: User: arn:aws:sts::#####:assumed-role/ecsTaskExecutionRole/##### is not authorized to perform: secretsmanager:GetSecretValue on resource: arn:aws:secretsmanager:ap-northeast-1:#####:secret:gitlab.com-container-registry because no identity-based policy allows the secretsmanager:GetSecretValue action status code: 400
    • 設定範例 JSON
      • {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Sid": "VisualEditor0",
                    "Effect": "Allow",
                    "Action": [
                        "secretsmanager:GetRandomPassword",
                        "secretsmanager:GetResourcePolicy",
                        "secretsmanager:GetSecretValue",
                        "secretsmanager:DescribeSecret",
                        "secretsmanager:ListSecretVersionIds",
                        "secretsmanager:ListSecrets",
                        "secretsmanager:BatchGetSecretValue"
                    ],
                    "Resource": "*"
                }
            ]
        }
  4. 在 AWS ECS 的 Task 就可以好好設定 Container 的存取機制
    • 私有登錄檔 -> Secrets Manager ARN 或名稱 -> arn:aws:secretsmanager:ap-northeast-1:#####:secret:gitlab.com-container-registry
如此在 AWS ECS -> Cluster -> Create Service 時，就會觸發 AWS ECS Task 運行，就可以看到 Docker Container 被創立出來

目前研究先告一個段落，在 AWS ECS 的設置規劃其實也要不小的篇幅的。

再碎碎念一下，直接實驗 AWS ECS 直接去跟 Gitlab.com Container Registry 要資料時，有時會出現彷彿無法正常取出的問題，這個我還不是很清楚排除的方式，目前還有偷懶隔天再弄一次，一樣的方式又正常可通過 Orz 或許最佳還是改用 AWS ECS Private Container Registry 維護，這樣服務發布架構可以省去一些不穩定的額外煩惱。

類似錯誤訊息：
  • esourceInitializationError: unable to pull secrets or registry auth: unable to get registry auth from asm: There is a connection issue between the task and AWS Secrets Manager. 
數個資安提問：
  • 假設你的 Docker Images 是在 Gitlab.com 產出，要 docker push 到 AWS ECS Container Registry 時，就會要面對在 Gitlab.com 上暴露 AWS 存取的資訊
  • 假設 Build docker images 是用 Gitlab.com 機器(Runner)，對應的也是有敏感資訊會在這些機器上，你是否信任他們？
  • 假設 Build docker images 時，用了其他人製作的 Docker images ，你的敏感資訊是否有妥善保護？惡意的 docker images 可以在啟動或結束時，把用戶的敏感資訊傳遞出去
ref: 
張貼者： 於
標籤： , , , , , , , , ,

沒有留言:

張貼留言

訂閱： 張貼留言 (Atom)