Docker 開發筆記 - 使用 Docker Compose 架設 Gitlab 服務 / 處理自訂 Ports / HTTPS SSL 憑證 @ macOS 14.2.1

延續上一篇 Docker 開發筆記 - 使用 Docker Compose 架設 Jenkins 服務 @ macOS 14.2.1 活動，該寫一下 gitlab 架設筆記。其實過年期間有播空試試，但是處理很不順，再加上跑去玩樂就荒廢了。昨晚終於可以收尾一下，把一些使用過程列一下。當時踩坑的原因是自己沒有把環境清乾淨，花了大把時間除錯。

先來個環境簡介：

% docker version 

Client:

 Cloud integration: v1.0.35+desktop.10

 Version:           25.0.3

 API version:       1.44

 Go version:        go1.21.6

 Git commit:        4debf41

 Built:             Tue Feb  6 21:13:26 2024

 OS/Arch:           darwin/arm64

 Context:           desktop-linux

Server: Docker Desktop 4.27.2 (137060)

 Engine:

  Version:          25.0.3

  API version:      1.44 (minimum version 1.24)

  Go version:       go1.21.6

  Git commit:       f417435

  Built:            Tue Feb  6 21:14:22 2024

  OS/Arch:          linux/arm64

  Experimental:     false

 containerd:

  Version:          1.6.28

  GitCommit:        ae07eda36dd25f8a1b98dfbf587313b99c0190bb

 runc:

  Version:          1.1.12

  GitCommit:        v1.1.12-0-g51d5e94

 docker-init:

  Version:          0.19.0

  GitCommit:        de40ad0

清乾淨後再重啟：

% docker-compose down -v

% rm -rf ~/docker-gitlab

% docker-compose up

總之先來為回顧官網的 docker 教學吧！依照 gitlab 官網的安裝簡介 可以很快速地裝起來 ：

% cat /etc/hosts | grep gitlab

127.0.0.1 gitlab.example.com

% cat docker-compose.yml 

# https://docs.docker.com/compose/compose-file/compose-versioning/

version: '3.8' 

services:

  gitlab:

    image: gitlab/gitlab-ee:latest

    container_name: gitlab

    restart: always

    hostname: 'gitlab.example.com'

    environment:

      GITLAB_OMNIBUS_CONFIG: |

        external_url 'http://gitlab.example.com:8929'

        gitlab_rails['gitlab_shell_ssh_port'] = 2424

    ports:

      - '8929:8929'

      - '2424:2424'

    volumes:

      - '~/docker-gitlab/config:/etc/gitlab'

      - '~/docker-gitlab/logs:/var/log/gitlab'

      - '~/docker-gitlab/data:/var/opt/gitlab'

    shm_size: '256m'

% docker-compose up

...

% docker container ls                   

CONTAINER ID   IMAGE                     COMMAND             CREATED         STATUS                   PORTS                                                             NAMES

XXXXXXXXXXXX   gitlab/gitlab-ee:latest   "/assets/wrapper"   3 minutes ago   Up 3 minutes (healthy)   22/tcp, 443/tcp, 0.0.0.0:20080->80/tcp, 0.0.0.0:20022->2424/tcp   gitlab

主要是看到 docker container 狀態要顯示 healthy ，接著就可以去瀏覽 http://gitlab.example.com:8929 位置了（註：gitlab.example.com被我設定成 127.0.0.1）。

接著我還在惡搞切換 nginx port，以及碰到 chrome browser 的 ERR_UNSAFE_PORT，最後延宕了好一陣子 :P 就把剩下的流水帳心得都記錄一下：

• 關於 gitlab/gitlab-ee:latest 和 gitlab/gitlab-ce:latest ，據說 gitlab/gitlab-ee:latest 沒有序號啟動時，就等同於 gitlab/gitlab-ce:latest ，就統一用 gitlab/gitlab-ee:latest 即可
• 記得初次使用時，登入帳號是 root ，密碼躲在 /etc/gitlab/initial_root_password

% docker container ls

CONTAINER ID   IMAGE                     COMMAND             CREATED          STATUS                    PORTS                                                              NAMES

XXXXXXXX   gitlab/gitlab-ee:latest   "/assets/wrapper"   20 minutes ago   Up 18 minutes (healthy)   80/tcp, 443/tcp, 0.0.0.0:20443->20443/tcp, 0.0.0.0:20022->22/tcp   gitlab

% docker exec -it XXXXXXXX cat /etc/gitlab/initial_root_password

# WARNING: This value is valid only in the following conditions

#          1. If provided manually (either via `GITLAB_ROOT_PASSWORD` environment variable or via `gitlab_rails['initial_root_password']` setting in `gitlab.rb`, it was provided before database was seeded for the first time (usually, the first reconfigure run).

#          2. Password hasn't been changed manually, either via UI or via command line.

#

#          If the password shown here doesn't work, you must reset the admin password following https://docs.gitlab.com/ee/security/reset_user_password.html#reset-your-root-password.

Password: yNRnhTRu9IZ/eBvlC3BCDeuK6zn6BUBmGB+a89SMpn0=

# NOTE: This file will be automatically deleted in the first reconfigure run after 24 hours.

• 使用 GITLAB_OMNIBUS_CONFIG 可以便利的完成絕大部分的設定
• 自訂的 port 請避開 chrome browser 定義的 ERR_UNSAFE_PORT 清單，這個雷不小心會耗掉非常多時間的，例如我偷懶把 80 增加個 10000 變成 10080 ...就中招，讓我以為有什麼服務沒啟動成功
• https://chromium.googlesource.com/chromium/src.git/+/refs/heads/main/net/base/port_util.cc#27
• 善用 external_url 設定外部連進去的資訊，並且把 HOST:CONTAINER Ports 都填寫一樣是最輕鬆的方式：

    environment:

      GITLAB_OMNIBUS_CONFIG: |

        external_url 'http://gitlab.example.com:20080'

        gitlab_rails['gitlab_shell_ssh_port'] = 20022

    ports:

      - '20080:20080'

      - '20022:20022'

• 想要來惡搞讓 nginx 聽在不同 port ，那就要設定更多東西

    environment:

      GITLAB_OMNIBUS_CONFIG: |

        external_url 'http://gitlab.example.com:20080'

        nginx['listen_port'] = 80

        gitlab_rails['gitlab_shell_ssh_port'] = 22

    ports:

      - '20080:80'

      - '20022:22'

• 想要啟用加密連線，單靠 external_url 更新成 `https://` 的描述也會默認啟動 SSL 加密連線服務，但下一刻還得處理憑證問題，連續動作：

% mkdir -p ssl

% test -e ./ssl/localhost.key || openssl genpkey -algorithm RSA -out ./ssl/localhost.key

% test -e ./ssl/localhost.crt || openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout ./ssl/localhost.key -out ./ssl/localhost.crt -subj '/C=US/ST=State/L=City/O=Organization/OU=Unit/CN=localhost'

% tree ssl 

ssl

├── localhost.crt

└── localhost.key

1 directory, 2 files 

 

% cat docker-compose.yml

 ...

     environment:

       GITLAB_OMNIBUS_CONFIG: |

         external_url 'https://gitlab.example.com:20443'

         #nginx['listen_port'] = 443

         nginx['ssl_certificate'] = "/etc/gitlab-ssl-usage/localhost.crt"

         nginx['ssl_certificate_key'] = "/etc/gitlab-ssl-usage/localhost.key"

         gitlab_rails['gitlab_shell_ssh_port'] = 22

     ports:

       - '20443:20443'

       - '20022:22'

     volumes:

       - './ssl:/etc/gitlab-ssl-usage'

• 若不想靠 volumes 掛進來，也可以改用 command 來發動

     command: ["sh", "-c", "mkdir -p /etc/gitlab-ssl-usage && (test -e /etc/gitlab-ssl-usage/localhost.key || openssl genpkey -algorithm RSA -out /etc/gitlab-ssl-usage/localhost.key ) && ( test -e /etc/gitlab-ssl-usage/localhost.crt || openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/gitlab-ssl-usage/localhost.key -out /etc/gitlab-ssl-usage/localhost.crt -subj '/C=US/ST=State/L=City/O=Organization/OU=Unit/CN=localhost' ) && /assets/wrapper "]

     #command: ["sh", "-c", "/tmp/config/setup.sh"]

     environment:

       GITLAB_OMNIBUS_CONFIG: |

         external_url 'https://gitlab.example.com:20443'

         nginx['ssl_certificate'] = "/etc/gitlab-ssl-usage/localhost.crt"

         nginx['ssl_certificate_key'] = "/etc/gitlab-ssl-usage/localhost.key"

         gitlab_rails['gitlab_shell_ssh_port'] = 22

     ports:

       - '20443:20443'

       - '20022:22'

• 最初實驗時還曾碰過 redis 跟 postgres 無法跑起來的問題 ( /var/opt/gitlab/postgresql/ , /var/opt/gitlab/redis/ )，以至於變成非常臭長的架構，我想沒事都可以不用這樣惡搞了，在此順便留戀一下

# https://docs.docker.com/compose/compose-file/compose-versioning/

version: '3.8' 

services:

  redis:

    restart: unless-stopped 

    image: redis:latest

    container_name: gitlab-redis

    volumes:

      - ~/docker_gitlab_home/redis:/data

      - ~/docker_gitlab_home/socket-redis:/var/run/redis

  postgres:

    image: postgres:latest

    container_name: gitlab-postgres

    restart: unless-stopped

    environment:

      POSTGRES_USER: gitlab

      POSTGRES_PASSWORD: gitlabAdmin

    volumes:

      - ~/docker_gitlab_home/postgres:/var/lib/postgresql/data

      - ~/docker_gitlab_home/socket-postgresql:/var/run/postgresql

  gitlab:

    # https://docs.gitlab.com/ee/install/docker.html#install-gitlab-using-docker-compose

    # https://hub.docker.com/r/gitlab/gitlab-ee/

    # https://hub.docker.com/r/gitlab/gitlab-ce

    image: gitlab/gitlab-ee:latest

    container_name: gitlab-main

    depends_on:

      - postgres

      - redis

    # https://docs.docker.com/config/containers/start-containers-automatically/#use-a-restart-policy

    restart: unless-stopped 

    hostname: 'localhost'

    command: ["sh", "-c", "mkdir -p /etc/gitlab-ssl-usage && (test -e /etc/gitlab-ssl-usage/localhost.key || openssl genpkey -algorithm RSA -out /etc/gitlab-ssl-usage/localhost.key ) && ( test -e /etc/gitlab-ssl-usage/localhost.crt || openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/gitlab-ssl-usage/localhost.key -out /etc/gitlab-ssl-usage/localhost.crt -subj '/C=US/ST=State/L=City/O=Organization/OU=Unit/CN=localhost' ) && /assets/wrapper "]

    environment:

      GITLAB_OMNIBUS_CONFIG: |

        # Add any other gitlab.rb configuration here, each on its own line

        #external_url 'http://localhost:20080'

        #nginx['listen_port'] = 80

        external_url 'https://localhost:20443'

        gitlab_rails['gitlab_shell_ssh_port'] = 22

        nginx['listen_port'] = 443

        nginx['listen_https'] = true

        nginx['ssl_certificate'] = "/etc/gitlab-ssl-usage/localhost.crt"

        nginx['ssl_certificate_key'] = "/etc/gitlab-ssl-usage/localhost.key"

        #letsencrypt['enable'] = false

        gitlab_rails['db_username'] = "gitlab"

        gitlab_rails['db_password'] = "gitlabAdmin"

    ports:

      # note: ERR_UNSAFE_PORT - https://chromium.googlesource.com/chromium/src.git/+/refs/heads/main/net/base/port_util.cc#27

      # HOST:CONTAINER

      - 20443:443

      #- 20080:80

      - 20022:22

    volumes:

      - ~/docker_gitlab_home/config:/etc/gitlab

      - ~/docker_gitlab_home/logs:/var/log/gitlab

      - ~/docker_gitlab_home/data:/var/opt/gitlab

      - ~/docker_gitlab_home/redis:/var/opt/gitlab/data/redis

      - ~/docker_gitlab_home/postgresql:/var/opt/gitlab/postgresql

      - ~/docker_gitlab_home/socket-postgresql:/var/opt/gitlab/postgresql/

      - ~/docker_gitlab_home/socket-redis:/var/opt/gitlab/redis/ 

Docker 開發筆記 - 使用 Docker Compose 架設 Jenkins 服務 @ macOS 14.2.1

過年找點樂子，用 Docker 把一些工作上常見的服務都架設一次好了 XD 整體工作上仍主要都還是 ansible 管理數百台機器，近期有同事對 docker 很感興趣，我就努力推坑，推坑前也得親自走一下是吧 :P

連續動作：

% cat docker-compose.yml 

# https://docs.docker.com/compose/compose-file/compose-versioning/

version: '3.8' 

services:

  jenkins:

    # https://hub.docker.com/_/jenkins

    image: jenkins/jenkins:lts

    # https://docs.docker.com/config/containers/start-containers-automatically/#use-a-restart-policy

    restart: unless-stopped 

    privileged: true

    user: root

    ports:

      # HOST:CONTAINER

      - 8080:8080 

    container_name: jenkins

    volumes:

      - ~/docker_jenkins_home:/var/jenkins_home

% docker-compose up 

[+] Running 1/0

 ✔ Container jenkins  Created                                                                                                                                          0.0s 

Attaching to jenkins

jenkins  | Running from: /usr/share/jenkins/jenkins.war

jenkins  | webroot: /var/jenkins_home/war

...

jenkins  | *************************************************************

jenkins  | *************************************************************

jenkins  | *************************************************************

jenkins  | 

jenkins  | Jenkins initial setup is required. An admin user has been created and a password generated.

jenkins  | Please use the following password to proceed to installation:

jenkins  | 

jenkins  | 54b6458ba37b4178bdc77f7d9eccbd0f

jenkins  | 

jenkins  | This may also be found at: /var/jenkins_home/secrets/initialAdminPassword

jenkins  | 

jenkins  | *************************************************************

jenkins  | *************************************************************

jenkins  | *************************************************************

這時就可以到 http://localhost:8080 繼續走完 jenkins 的架設，包括也看到的啟動碼，當然，也可以搞剛用 docker exec 指令來列出：

% docker exec jenkins cat /var/jenkins_home/secrets/initialAdminPassword

54b6458ba37b4178bdc77f7d9eccbd0f

除此之外，在本機 ~/docker_jenkins_home 也可以看到相關結構：

% tree -L 1 ~/docker_jenkins_home

/Users/user/docker_jenkins_home

├── config.xml

├── copy_reference_file.log

├── hudson.model.UpdateCenter.xml

├── jenkins.telemetry.Correlator.xml

├── jobs

├── nodeMonitors.xml

├── nodes

├── plugins

├── secret.key

├── secret.key.not-so-secret

├── secrets

├── updates

├── userContent

├── users

└── war

9 directories, 7 files

只是剩下的就靠 http://localhost:8080 走安裝流程，收工

而原先跑 docker-compose up 的那個環境可以用 ctrl+c 來停掉，未來可以靠 docker-compose up -d 來重新啟動

^CGracefully stopping... (press Ctrl+C again to force)

[+] Stopping 1/1

 ✔ Container jenkins  Stopped                                                                                                                                          0.2s 

canceled

Kubernetes/k8s 開發筆記 - 在 Ubuntu 16.04 安裝 Kubeadm 以及處理 docker、containerd 版本過舊問題

之前已經用 docker 來封裝一些非常重的工作任務，像是 build fw 等。現在來試試看 kubeadm 這個工具，將維護整套系統的維度從 docker 轉進到 Kubernetes cluster，往後可以靠 k8s 來維護算力資源，像是動態調配算力單元等等。這些感覺滿像十多年前在 AWS 靠 autoscaling 做的事，真是熟悉的陌生人。

這篇僅處理在 Ubuntu 16.04 安裝 Kubeadm 後的啟動問題，並沒有處理其他使用細節，包括建立 node server 、 連上即加入 master server 等。

環境簡介：

$ lsb_release -a

No LSB modules are available.

Distributor ID: Ubuntu

Description:    Ubuntu 16.04.6 LTS

Release:        16.04

Codename:       xenial

$ curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -

$ echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee -a /etc/apt/sources.list.d/kubernetes.list

$ sudo apt update

$ sudo apt install kubeadm

$ sudo apt-mark hold kubelet kubeadm kubectl

$ dpkg -l  | grep kube

ii  kubeadm                 1.28.2-00          amd64        Kubernetes Cluster Bootstrapping Tool

ii  kubectl                 1.28.2-00          amd64        Kubernetes Command Line Tool

ii  kubelet                 1.28.2-00          amd64        Kubernetes Node Agent

ii  kubernetes-cni          1.2.0-00           amd64        Kubernetes CNI

接著：

$ sudo kubeadm init --v=5

...

validating the existence and emptiness of directory /var/lib/etcd

[preflight] Some fatal errors occurred:

[ERROR CRI]: container runtime is not running: output: level=fatal msg="validate service connection: CRI v1 runtime API is not implemented for endpoint \"unix:///var/run/containerd/containerd.sock\": rpc error: code = Unimplemented desc = unknown service runtime.v1.RuntimeService"

發現有些問題，進行排除研究，部分資訊推導應當跟 docker , containerd 版本有高度相關，就先把 docker 跟 containerd 盡可能升級上去:

$ dpkg -l | grep containerd

ii  containerd              1.2.6-0ubuntu1~16.04.6+esm1  amd64        daemon to control runC

$ dpkg -l | grep docker

rc  docker                                     1.5-1                                           amd64        System tray for KDE3/GNOME2 docklet applications

ii  docker.io                                  18.09.7-0ubuntu1~16.04.7                        amd64        Linux container runtime

$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

$ echo "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

$ sudo apt update

$ sudo apt install docker-ce docker-ce-cli containerd.io

$ sudo docker version

Client: Docker Engine - Community

 Version:           20.10.7

 API version:       1.41

 Go version:        go1.13.15

 Git commit:        f0df350

 Built:             Wed Jun  2 11:56:47 2021

 OS/Arch:           linux/amd64

 Context:           default

 Experimental:      true

Server: Docker Engine - Community

 Engine:

  Version:          20.10.7

  API version:      1.41 (minimum version 1.12)

  Go version:       go1.13.15

  Git commit:       b0f5bc3

  Built:            Wed Jun  2 11:54:58 2021

  OS/Arch:          linux/amd64

  Experimental:     false

 containerd:

  Version:          1.4.6

  GitCommit:        d71fcd7d8303cbf684402823e425e9dd2e99285d

 runc:

  Version:          1.0.0-rc95

  GitCommit:        b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7

 docker-init:

  Version:          0.19.0

  GitCommit:        de40ad0

接著追蹤可能是 cri 插件的部分，試著排除：

$ cat /etc/containerd/config.toml | grep cri

enabled_plugins = ["cri"]

無效，繼續努力：

$ sudo mv /etc/containerd/config.toml /etc/containerd/config.toml.bak

$ containerd config default | sudo tee /etc/containerd/config.toml

$ sudo systemctl restart containerd

$ containerd config default | grep containerd.sock

  address = "/run/containerd/containerd.sock"

接著在試著  kubeadm init 還是有一樣的問題，查詢了細節滿有可能是 containerd 版本還是太舊了，有個關鍵資訊是說 1.6 版本以前會缺乏溝通介面

$ dpkg -L containerd.io | grep bin

/usr/bin

/usr/bin/containerd-shim-runc-v2

/usr/bin/containerd-shim

/usr/bin/containerd

/usr/bin/runc

/usr/bin/ctr

/usr/bin/containerd-shim-runc-v1

直接到 containerd.io 官網下載最新版 1.7.11 版的 binary 方案：

$ wget https://github.com/containerd/containerd/releases/download/v1.7.11/containerd-1.7.11-linux-amd64.tar.gz

$ tar xvf containerd-1.7.11-linux-amd64.tar.gzl

$ tar -tzvf containerd-1.7.11-linux-amd64.tar.gz

drwxr-xr-x root/root         0 2023-12-09 07:41 bin/

-rwxr-xr-x root/root  12185600 2023-12-09 07:41 bin/containerd-shim-runc-v2

-rwxr-xr-x root/root  28330360 2023-12-09 07:41 bin/ctr

-rwxr-xr-x root/root   7061504 2023-12-09 07:41 bin/containerd-shim

-rwxr-xr-x root/root   8761344 2023-12-09 07:41 bin/containerd-shim-runc-v1

-rwxr-xr-x root/root  26184312 2023-12-09 07:41 bin/containerd-stress

-rwxr-xr-x root/root  55551616 2023-12-09 07:41 bin/containerd

處理一下系統內部的：

$ sudo systemctl stop containerd

$ sudo mkdir -p /usr/bin/containerd-1.4.6

$ sudo mv /usr/bin/containerd* /usr/bin/containerd-1.4.6/

$ sudo mv /usr/bin/ctr /usr/bin/containerd-1.4.6/

$ tree /usr/bin/containerd-1.4.6/

/usr/bin/containerd-1.4.6/

├── containerd

├── containerd-shim

├── containerd-shim-runc-v1

├── containerd-shim-runc-v2

└── ctr

0 directories, 5 files

$ sudo cp ~/bin/c* /usr/bin/

準備重新啟動：

$ containerd --version

containerd github.com/containerd/containerd v1.7.11 64b8a811b07ba6288238eefc14d898ee0b5b99ba

$ containerd config default | sudo tee /etc/containerd/config.toml

$ sudo systemctl stop containerd

$ sudo systemctl start containerd

$ sudo systemctl status containerd

● containerd.service - containerd container runtime

   Loaded: loaded (/lib/systemd/system/containerd.service; enabled; vendor preset: enabled)

   Active: active (running); 14min ago

     Docs: https://containerd.io

  Process: 19396 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)

 Main PID: 19406 (containerd)

    Tasks: 32

   Memory: 24.5M

      CPU: 187ms

   CGroup: /system.slice/containerd.service

           └─19406 /usr/bin/containerd

$ sudo systemctl stop docker

$ sudo systemctl start docker

$ sudo docker version

Client: Docker Engine - Community

 Version:           20.10.7

 API version:       1.41

 Go version:        go1.13.15

 Git commit:        f0df350

 Built:             Wed Jun  2 11:56:47 2021

 OS/Arch:           linux/amd64

 Context:           default

 Experimental:      true

Server: Docker Engine - Community

 Engine:

  Version:          20.10.7

  API version:      1.41 (minimum version 1.12)

  Go version:       go1.13.15

  Git commit:       b0f5bc3

  Built:            Wed Jun  2 11:54:58 2021

  OS/Arch:          linux/amd64

  Experimental:     false

 containerd:

  Version:          v1.7.11

  GitCommit:        64b8a811b07ba6288238eefc14d898ee0b5b99ba

 runc:

  Version:          1.0.0-rc95

  GitCommit:        b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7

 docker-init:

  Version:          0.19.0

  GitCommit:        de40ad0

終於讓 docker version 也認到 containerd v1.7.11 了，接著就可以回到 kubeadm 啦 

$ sudo kubeadm init  --v=5

....

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube

  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.

Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:

  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join ip:6443 --token ###### --discovery-token-ca-cert-hash sha256:###### 

此外，kubernetes 本身是建議關閉 swap 的使用來確保整體性能，由於我是在一台本身就有 swap 的機器上運行，由於不能關閉 swap ，只好設法去略過 swap 的檢查 (增加 --fail-swap-on=false ):

$ cat /etc/systemd/system/kubelet.service.d/10-kubeadm.conf | grep ExecStart

ExecStart=

ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS --fail-swap-on=false

相關資訊:

• github.com/containerd/containerd/issues/8139
• kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/

Javascript 開發筆記 - 盡可能使用 npm 指令處理套件安全更新，此例升級 electron 和 electron-builder

之前寫的練習擺在 github 一直收到套件安全的升級通知信（這只是其中一條），以前在處理這個時，都是人工改掉 package.json ，這次來用指令試試：

% git clone https://github.com/changyy/node-electron-based

% cd node-electron-based

% source env_nvm.sh 

Now using node v16.20.2 (npm v8.19.4)

% npm install

npm WARN deprecated asar@3.2.0: Please use @electron/asar moving forward.  There is no API change, just a package name change

npm WARN deprecated electron-osx-sign@0.6.0: Please use @electron/osx-sign moving forward. Be aware the API is slightly different

added 557 packages, and audited 558 packages in 10s

69 packages are looking for funding

  run `npm fund` for details

11 vulnerabilities (4 moderate, 7 high)

To address issues that do not require attention, run:

  npm audit fix

To address all issues (including breaking changes), run:

  npm audit fix --force

Run `npm audit` for details.

接著就如上述所言，用 npm audit fix 查看一下：

% npm audit fix

up to date, audited 558 packages in 860ms

69 packages are looking for funding

  run `npm fund` for details

# npm audit report

electron  <=22.3.24

Severity: high

Depends on vulnerable versions of @electron/get

Electron vulnerable to out-of-package code execution when launched with arbitrary cwd - https://github.com/advisories/GHSA-7x97-j373-85x5

Electron context isolation bypass via nested unserializable return value - https://github.com/advisories/GHSA-p7v2-p9m8-qqg7

Electron affected by libvpx's heap buffer overflow in vp8 encoding - https://github.com/advisories/GHSA-qqvq-6xgj-jw8g

ASAR Integrity bypass via filetype confusion in electron - https://github.com/advisories/GHSA-7m48-wc93-9g85

fix available via `npm audit fix --force`

Will install electron@28.1.4, which is a breaking change

node_modules/electron

got  <11.8.5

Severity: moderate

Got allows a redirect to a UNIX socket - https://github.com/advisories/GHSA-pfrx-2q88-qq97

fix available via `npm audit fix --force`

Will install electron@28.1.4, which is a breaking change

node_modules/got

  @electron/get  <=1.14.1

  Depends on vulnerable versions of got

  node_modules/@electron/get

minimatch  <3.0.5

Severity: high

minimatch ReDoS vulnerability - https://github.com/advisories/GHSA-f8q6-p94x-37v3

fix available via `npm audit fix`

node_modules/dir-compare/node_modules/minimatch

  dir-compare  <=2.4.0

  Depends on vulnerable versions of minimatch

  node_modules/dir-compare

    @electron/universal  1.0.1 - 1.3.3

    Depends on vulnerable versions of dir-compare

    node_modules/@electron/universal

      app-builder-lib  22.10.4 - 24.0.0-alpha.13

      Depends on vulnerable versions of @electron/universal

      node_modules/app-builder-lib

        dmg-builder  22.10.4 - 24.0.0-alpha.13

        Depends on vulnerable versions of app-builder-lib

        node_modules/dmg-builder

          electron-builder  19.25.0 || 22.10.4 - 24.6.0

          Depends on vulnerable versions of app-builder-lib

          Depends on vulnerable versions of dmg-builder

          Depends on vulnerable versions of simple-update-notifier

          node_modules/electron-builder

semver  7.0.0 - 7.5.1

Severity: moderate

semver vulnerable to Regular Expression Denial of Service - https://github.com/advisories/GHSA-c2qf-rxjj-qqgw

fix available via `npm audit fix`

node_modules/simple-update-notifier/node_modules/semver

  simple-update-notifier  1.0.7 - 1.1.0

  Depends on vulnerable versions of semver

  node_modules/simple-update-notifier

11 vulnerabilities (4 moderate, 7 high)

To address issues that do not require attention, run:

  npm audit fix

To address all issues (including breaking changes), run:

  npm audit fix --force

使用 npm audit fix --force 處理：

% npm audit fix --force

npm WARN using --force Recommended protections disabled.

npm WARN audit Updating electron to 28.1.4, which is a SemVer major change.

added 10 packages, removed 13 packages, changed 15 packages, and audited 555 packages in 4s

76 packages are looking for funding

  run `npm fund` for details

# npm audit report

minimatch  <3.0.5

Severity: high

minimatch ReDoS vulnerability - https://github.com/advisories/GHSA-f8q6-p94x-37v3

fix available via `npm audit fix`

node_modules/dir-compare/node_modules/minimatch

  dir-compare  <=2.4.0

  Depends on vulnerable versions of minimatch

  node_modules/dir-compare

    @electron/universal  1.0.1 - 1.3.3

    Depends on vulnerable versions of dir-compare

    node_modules/@electron/universal

      app-builder-lib  22.10.4 - 24.0.0-alpha.13

      Depends on vulnerable versions of @electron/universal

      node_modules/app-builder-lib

        dmg-builder  22.10.4 - 24.0.0-alpha.13

        Depends on vulnerable versions of app-builder-lib

        node_modules/dmg-builder

          electron-builder  19.25.0 || 22.10.4 - 24.6.0

          Depends on vulnerable versions of app-builder-lib

          Depends on vulnerable versions of dmg-builder

          Depends on vulnerable versions of simple-update-notifier

          node_modules/electron-builder

semver  7.0.0 - 7.5.1

Severity: moderate

semver vulnerable to Regular Expression Denial of Service - https://github.com/advisories/GHSA-c2qf-rxjj-qqgw

fix available via `npm audit fix`

node_modules/simple-update-notifier/node_modules/semver

  simple-update-notifier  1.0.7 - 1.1.0

  Depends on vulnerable versions of semver

  node_modules/simple-update-notifier

8 vulnerabilities (2 moderate, 6 high)

To address all issues, run:

  npm audit fix

% git diff

diff --git a/package.json b/package.json

index f470e2b..4b7ac3d 100644

--- a/package.json

+++ b/package.json

@@ -19,7 +19,7 @@

     "concurrently": "^7.4.0",

     "copy-webpack-plugin": "^11.0.0",

     "cross-env": "^7.0.3",

-    "electron": "^20.1.1",

+    "electron": "^28.1.4",

     "electron-builder": "^23.3.3",

     "webpack-cli": "^4.10.0",

     "webpack-dev-server": "^4.11.0"

這樣應當算有處理到一個項目的更新，最後再看看之前的 build code 是否正常

% npm install

% npm run build

> simple-electron-app@1.0.0 build

> webpack-cli && electron-builder -mwl

assets by path ../../ 2.47 KiB

  assets by path ../../renderer/*.js 958 bytes

    asset ../../renderer/mainRenederer.js 910 bytes [emitted] [from: src/renderer/mainRenederer.js] [copied] [minimized]

    asset ../../renderer/index.js 48 bytes [emitted] [from: src/renderer/index.js] [copied] [minimized]

  asset ../../main/index.js 1.26 KiB [emitted] [from: src/main/index.js] [copied] [minimized]

  asset ../../preload/mainRenderer.js 289 bytes [emitted] [from: src/preload/mainRenderer.js] [copied] [minimized]

asset index.html 495 bytes [emitted] [from: src/html/mainRenderer/index.html] [copied]

asset index.js 253 bytes [emitted] [minimized] (name: mainRendererHTML)

./src/html/mainRenderer/index.js 380 bytes [built] [code generated]

WARNING in configuration

The 'mode' option has not been set, webpack will fallback to 'production' for this value.

Set 'mode' option to 'development' or 'production' to enable defaults for each environment.

You can also set it to 'none' to disable any default behavior. Learn more: https://webpack.js.org/configuration/mode/

webpack 5.89.0 compiled with 1 warning in 146 ms

  • electron-builder  version=23.6.0 os=23.2.0

  • description is missed in the package.json  appPackageFile=/Volumes/Data/UserData/tmp/node-electron-based/package.json

  • writing effective config  file=dist/builder-effective-config.yaml

  • packaging       platform=darwin arch=arm64 electron=28.1.4 appOutDir=dist/mac-arm64

  • downloading     url=https://github.com/electron/electron/releases/download/v28.1.4/electron-v28.1.4-darwin-arm64.zip size=95 MB parts=8

  • downloaded      url=https://github.com/electron/electron/releases/download/v28.1.4/electron-v28.1.4-darwin-arm64.zip duration=36.971s

  • default Electron icon is used  reason=application icon is not set

  • skipped macOS application code signing  reason=cannot find valid "Developer ID Application" identity or custom non-Apple code signing certificate, it could cause some undefined behaviour, e.g. macOS localized description not visible, see https://electron.build/code-signing allIdentities=     0 identities found

                                                Valid identities only

     0 valid identities found

  • building        target=macOS zip arch=arm64 file=dist/simple-electron-app-1.0.0-arm64-mac.zip

  • building        target=DMG arch=arm64 file=dist/simple-electron-app-1.0.0-arm64.dmg

  • Detected arm64 process, HFS+ is unavailable. Creating dmg with APFS - supports Mac OSX 10.12+

  • packaging       platform=linux arch=x64 electron=28.1.4 appOutDir=dist/linux-unpacked

  • downloading     url=https://github.com/electron/electron/releases/download/v28.1.4/electron-v28.1.4-linux-x64.zip size=102 MB parts=8

  • building block map  blockMapFile=dist/simple-electron-app-1.0.0-arm64.dmg.blockmap

  • building block map  blockMapFile=dist/simple-electron-app-1.0.0-arm64-mac.zip.blockmap

  • downloaded      url=https://github.com/electron/electron/releases/download/v28.1.4/electron-v28.1.4-linux-x64.zip duration=30.749s

  • building        target=snap arch=x64 file=dist/simple-electron-app_1.0.0_amd64.snap

  • building        target=AppImage arch=x64 file=dist/simple-electron-app-1.0.0.AppImage

  • application Linux category is set to default "Utility"  reason=linux.category is not set and cannot map from macOS docs=https://www.electron.build/configuration/linux

  • default Electron icon is used  reason=application icon is not set

  • application Linux category is set to default "Utility"  reason=linux.category is not set and cannot map from macOS docs=https://www.electron.build/configuration/linux

  • packaging       platform=win32 arch=arm64 electron=28.1.4 appOutDir=dist/win-arm64-unpacked

  • downloading     url=https://github.com/electron/electron/releases/download/v28.1.4/electron-v28.1.4-win32-arm64.zip size=108 MB parts=8

  • downloading     url=https://github.com/electron-userland/electron-builder-binaries/releases/download/snap-template-4.0-2/snap-template-electron-4.0-2-amd64.tar.7z size=1.5 MB parts=1

  • downloading     url=https://github.com/electron-userland/electron-builder-binaries/releases/download/appimage-12.0.1/appimage-12.0.1.7z size=1.6 MB parts=1

  • downloaded      url=https://github.com/electron-userland/electron-builder-binaries/releases/download/appimage-12.0.1/appimage-12.0.1.7z duration=5.225s

  • downloaded      url=https://github.com/electron-userland/electron-builder-binaries/releases/download/snap-template-4.0-2/snap-template-electron-4.0-2-amd64.tar.7z duration=5.425s

  • downloaded      url=https://github.com/electron/electron/releases/download/v28.1.4/electron-v28.1.4-win32-arm64.zip duration=42.26s

  • default Electron icon is used  reason=application icon is not set

  • downloading     url=https://github.com/electron-userland/electron-builder-binaries/releases/download/winCodeSign-2.6.0/winCodeSign-2.6.0.7z size=5.6 MB parts=1

  • downloaded      url=https://github.com/electron-userland/electron-builder-binaries/releases/download/winCodeSign-2.6.0/winCodeSign-2.6.0.7z duration=4.001s

  • downloading     url=https://github.com/electron-userland/electron-builder-binaries/releases/download/wine-4.0.1-mac/wine-4.0.1-mac.7z size=19 MB parts=3

  • downloaded      url=https://github.com/electron-userland/electron-builder-binaries/releases/download/wine-4.0.1-mac/wine-4.0.1-mac.7z duration=13.249s

  • building        target=nsis file=dist/simple-electron-app Setup 1.0.0.exe archs=arm64 oneClick=true perMachine=false

  • downloading     url=https://github.com/electron-userland/electron-builder-binaries/releases/download/nsis-3.0.4.1/nsis-3.0.4.1.7z size=1.3 MB parts=1

  • downloaded      url=https://github.com/electron-userland/electron-builder-binaries/releases/download/nsis-3.0.4.1/nsis-3.0.4.1.7z duration=5.637s

  • downloading     url=https://github.com/electron-userland/electron-builder-binaries/releases/download/nsis-resources-3.4.1/nsis-resources-3.4.1.7z size=731 kB parts=1

  • downloaded      url=https://github.com/electron-userland/electron-builder-binaries/releases/download/nsis-resources-3.4.1/nsis-resources-3.4.1.7z duration=4.013s

  • building block map  blockMapFile=dist/simple-electron-app Setup 1.0.0.exe.blockmap

編譯正常，接下來再繼續處理 

% npm audit fix

up to date, audited 555 packages in 730ms

76 packages are looking for funding

  run `npm fund` for details

# npm audit report

minimatch  <3.0.5

Severity: high

minimatch ReDoS vulnerability - https://github.com/advisories/GHSA-f8q6-p94x-37v3

fix available via `npm audit fix`

node_modules/dir-compare/node_modules/minimatch

  dir-compare  <=2.4.0

  Depends on vulnerable versions of minimatch

  node_modules/dir-compare

    @electron/universal  1.0.1 - 1.3.3

    Depends on vulnerable versions of dir-compare

    node_modules/@electron/universal

      app-builder-lib  22.10.4 - 24.0.0-alpha.13

      Depends on vulnerable versions of @electron/universal

      node_modules/app-builder-lib

        dmg-builder  22.10.4 - 24.0.0-alpha.13

        Depends on vulnerable versions of app-builder-lib

        node_modules/dmg-builder

          electron-builder  19.25.0 || 22.10.4 - 24.6.0

          Depends on vulnerable versions of app-builder-lib

          Depends on vulnerable versions of dmg-builder

          Depends on vulnerable versions of simple-update-notifier

          node_modules/electron-builder

semver  7.0.0 - 7.5.1

Severity: moderate

semver vulnerable to Regular Expression Denial of Service - https://github.com/advisories/GHSA-c2qf-rxjj-qqgw

fix available via `npm audit fix`

node_modules/simple-update-notifier/node_modules/semver

  simple-update-notifier  1.0.7 - 1.1.0

  Depends on vulnerable versions of semver

  node_modules/simple-update-notifier

8 vulnerabilities (2 moderate, 6 high)

To address all issues, run:

  npm audit fix

故事大概起源於 electron-builder ，只好先把他升級看看

% cat package.json | jq '.devDependencies["electron-builder"]'

"^23.3.3"

% npm update --dev electron-builder

% cat package.json | jq '.devDependencies["electron-builder"]'

"^23.3.3"

看來只好靠這招了：

% npm install electron-builder@latest --save-dev

% cat package.json | jq '.devDependencies["electron-builder"]'

"^24.9.1"

如此也確認都沒事了：

% npm audit

found 0 vulnerabilities

最終，其實也只是改 package.json 兩行 XD

% git diff package.json 

diff --git a/package.json b/package.json

index f470e2b..4c830f1 100644

--- a/package.json

+++ b/package.json

@@ -19,8 +19,8 @@

     "concurrently": "^7.4.0",

     "copy-webpack-plugin": "^11.0.0",

     "cross-env": "^7.0.3",

-    "electron": "^20.1.1",

-    "electron-builder": "^23.3.3",

+    "electron": "^28.1.4",

+    "electron-builder": "^24.9.1",

     "webpack-cli": "^4.10.0",

     "webpack-dev-server": "^4.11.0"

   }

Synology NAS 使用 Private IP, 自訂網域 與 Wildcard SSL 來提供 HTTPS 服務並完成憑證定期更新與設定 @ DSM 7.2.1, DS723+

之前已經處理過 DS216play 的情況，撰寫過兩篇筆記：

• 2018年 - [NAS] Private IP Server 與 Let's Encrypt Wildcard SSL 憑證 / HTTPS 服務
• 2022年 - [NAS] Private IP / Custom Domain / Wildcard SSL / HTTPS 憑證定期更新與設定 @ DS216play / DSM 7.1

這次在處理 DS723+ / DSM 7.2.1 時，有在追細一點，但整體上的流程 2022年的那篇只做了一半，他並沒有做到自動更新，而當時追的時候也沒追清楚。

簡略的流程：

1. 參考 2022年的方式，先透過 acme.sh 產出憑證（例如用 Cloudflare 維護 DNS record 時，可以輕鬆呼叫 Cloudflare API 完成 txt record 更新），接著到 Synology 網頁管理 -> 控制台 -> 連線能力 -> 憑證 -> 新增 -> 新增憑證 -> 匯入憑證
2. 這時匯入完畢後，其實在系統底層可以看到資訊，我原先在 2022年那篇誤認為 /usr/syno/etc/www/certificate/system_default/cert.conf 的描述位置（因為從 Nginx 設定檔有關），細追後，發現真正的位置在 /usr/syno/etc/certificate/_archive/ 某個目錄內
• 用 cat /usr/syno/etc/certificate/_archive/INFO 可以看到你自己上傳的憑證，假設是在 /usr/syno/etc/certificate/MySelf 這目錄，內有 cert.pem, chain.pem, fullchain.pem, privkey.pem
3. 這時若把 acme.sh 安置好，可以在 Synology NAS 內某個使用者的家目錄運行，那在他家目錄中就會有 ~/.acme.sh/*.YourDomain_ecc/ 這類位置，內有 ca.cer, '*.YourDomain.cer', '*.YourDomain.key', 'fullchain.cer' 和 '*.YourDomain.pkcs8'，這時，就要把 /usr/syno/etc/certificate/MySelf 內的完成同步更新，例如使用 symbolic link 
4. 後續，就回到 Synology 網頁介面 -> 控制台 -> 任務排程表 -> 新增一則任務並使用 root 權限 -> 設定每月執行一次 -> 在使用者定義指令碼那邊，可以直接呼叫 script (例如包裝成 renew-ssl.sh）在指定使用者家目錄完成憑證更新、接著在把 nginx reload
• sudo -u YourNASUser bash /var/services/homes/YourNASUser/renew-ssl.sh
• sudo nginx -t && sudo synopkg restart --service nginx && sudo synosystemctl reload nginx

目前看來，在 (2) 那邊找到的實際儲存位置時，透過 symbolic link 改到取 (3) 的資料，如此在控制台憑證列表可以立即觀看到 acme.sh renew 後的延長期限數字，這邊有提到 '*.YourDomain.pkcs8' 檔案格式，是跟 Synology NAS 使用的格式有關，他在呼叫 acme.sh renew 時，可以多添加 --to-pkcs8 參數來產生。

現況應當還有個隱憂，那就是 /usr/syno/etc/certificate/MySelf/*.pem 可完成自動更新，那 nginx 設定取得的憑證資訊是否也會更新？目前推猜好像不是同步的，且更新時機也還不明，只能等下個期週期後再來看看了。暫時推論是在設定憑證的系統預設時，才從 /usr/syno/etc/certificate/MySelf/*.pem 複製到 /usr/syno/etc/www/certificate/system_default/ 使用