centos7的系统服务管理（Harbor企业级镜像管理系统实战）

0.安装harbor0.1 下载安装包

github下载巨慢

链接：HTTPS:///s/1_l_FaqkKleqoiR3FAi2p5A 提取码：7b1r --来自百度网盘超级会员V4的分享

0.2安装

解压下载的压缩包修改配置并执行install

tar -zxvf harbor-offline-installer-v1.10.10.tgz harbor]# ll -rw-r--r-- 1 root root 612306524 Jan 12 12:09 harbor.v1.10.10.tar.gz -rw-r--r-- 1 root root 5895 Apr 22 10:02 harbor.yml -rwxr-xr-x 1 root root 2284 Jan 12 12:08 install.sh -rw-r--r-- 1 root root 11347 Jan 12 12:08 LICENSE -rwxr-xr-x 1 root root 1750 Jan 12 12:08 prepare #修改配置 修改hostname 和port 以及数据存储目录 hostname: 10.50.10.185 http: # port for http, default is 80. If https enabled, this port will redirect to https port port: 8199 # The default data volume data_volume: /opt/harbor/data [root@p1edaspk02 harbor]# sh ./install.sh [Step 0]: checking if Docker is installed ... Note: docker version: 18.06.3 [Step 1]: checking docker-compose is installed ... Note: docker-compose version: 1.29.1 [Step 2]: loading Harbor images ... Loaded image: goharbor/harbor-portal:v1.10.10 ... [Step 3]: preparing environment ... [Step 4]: preparing harbor configs ... prepare base dir is set to /opt/harbor /usr/src/app/utils/configs.py:100: YAMLLoadWARNING: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details. configs = yaml.load(f) .... [Step 5]: starting Harbor ... Creating nginx ... done ✔ ----Harbor has been installed and started successfully.----

0.3启动和关闭

# 启动harbor docker-compose up -d # 关闭harbor docker-compose down

要配置HTTPS，必须创建SSL证书。您可以使用由受信任的第三方CA签名的证书，也可以使用自签名证书

默认情况下，Harbor不附带证书。可以在没有安全性的情况下部署Harbor，以便您可以通过HTTP连接到它。但是，只有在没有外部网络连接的空白测试或开发环境中，才可以使用HTTP。在没有空隙的环境中使用HTTP会使您遭受中间人攻击。在生产环境中，请始终使用HTTPS。如果启用Content Trust with Notary来正确签名所有图像，则必须使用HTTPS。

1. 生成证书颁发机构证书

在生产环境中，您应该从CA获得证书。在测试或开发环境中，您可以生成自己的CA。要生成CA证书，请运行以下命令。

1.1 生成CA证书私钥。

openssl genrsa -out ca.key 4096

1.2 生成CA证书

调整-subj选项中的值以反映您的组织。如果使用FQDN连接Harbor主机，则必须将其指定为通用名称（CN）属性。

openssl req -x509 -new -nodes -sha512 -days 3650 \ -subj "/C=CN/ST=XianYang/L=XianYang/O=example/OU=Personal/CN=10.50.10.185" \ -key ca.key \ -out ca.crt

如果是ip访问， 将 10.50.10.185 改成 ip地址

2. 生成服务器证书

证书通常包含一个.crt文件和一个.key文件

2.1 生成私钥

Copyopenssl genrsa -out 10.50.10.185.key 4096

2.2 生成证书签名请求（CSR）

openssl req -sha512 -new \ -subj "/C=CN/ST=XianYang/L=XianYang/O=example/OU=Personal/CN=10.50.10.185" \ -key 10.50.10.185.key \ -out 10.50.10.185.csr

如果是ip访问， 将 10.50.10.185 改成 ip地址

2.3 生成一个x509 v3扩展文件

无论您使用FQDN还是IP地址连接到Harbor主机，都必须创建此文件，以便可以为您的Harbor主机生成符合主题备用名称（SAN）和x509 v3的证书扩展要求。替换DNS条目以反映您的域

Copycat > v3.ext <<-EOF authorityKeyIdentifier=keyid,issuer basicConstraints=CA:FALSE keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment extendedKeyUsage = serverAuth subjectAltName = @alt_names [alt_names] DNS.1=10.50.10.185 DNS.2=10.50.10.185 DNS.3=10.50.10.185 EOF

• 如果是ip访问

cat > v3.ext <<-EOF authorityKeyIdentifier=keyid,issuer basicConstraints=CA:FALSE keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment extendedKeyUsage = serverAuth subjectAltName = IP:10.50.10.185 EOF

2.4 使用该v3.ext文件为您的Harbor主机生成证书

openssl x509 -req -sha512 -days 3650 \ -extfile v3.ext \ -CA ca.crt -CAkey ca.key -CAcreateserial \ -in 10.50.10.185.csr \ -out 10.50.10.185.crt

如果是ip访问， 将 10.50.10.185 改成 ip地址

3. 提供证书给Harbor和Docker

生成后ca.crt，10.50.10.185.crt和10.50.10.185.key文件，必须将它们提供给Harbor和docker，重新配置它们

3.1 将服务器证书和密钥复制到Harbor主机上的/data/cert/文件夹中

mkdir -p /data/cert/ cp 10.50.10.185.crt /data/cert/ cp 10.50.10.185.key /data/cert/

3.2 转换10.50.10.185.crt为10.50.10.185.cert，供Docker使用

Docker守护程序将.crt文件解释为CA证书，并将.cert文件解释为客户端证书

openssl x509 -inform PEM -in 10.50.10.185.crt -out 10.50.10.185.cert

3.3 将服务器证书，密钥和CA文件复制到Harbor主机上的Docker证书文件夹中。您必须首先创建适当的文件夹

mkdir -p /etc/docker/certs.d/10.50.10.185/ cp 10.50.10.185.cert /etc/docker/certs.d/10.50.10.185/ cp 10.50.10.185.key /etc/docker/certs.d/10.50.10.185/ cp ca.crt /etc/docker/certs.d/10.50.10.185/ harbor]# tree /etc/docker/certs.d/10.50.10.185/ /etc/docker/certs.d/10.50.10.185/ ├── 10.50.10.185.cert ├── 10.50.10.185.key └── ca.crt

如果将默认nginx端口443 映射到其他端口，请创建文件夹/etc/docker/certs.d/yourdomain.com:port或/etc/docker/certs.d/harbor_IP:port

例如https的port为8443

mkdir -p /etc/docker/certs.d/10.50.10.185:8843 cp 10.50.10.185.cert 10.50.10.185.key ca.crt /etc/docker/certs.d/10.50.10.185:8843

3.4 重新启动Docker Engine

systemctl restart docker

3.5 证书的目录结构

ca]# tree /etc/docker/certs.d/ /etc/docker/certs.d/ └── 10.50.10.185 ├── 10.50.10.185.cert ├── 10.50.10.185.key └── ca.crt

4. 部署或重新配置Harbor

harbor.yml hostname: 10.50.10.185 http: port: 8199 https: port: 443 certificate: /opt/harbor/ca/10.50.10.185.crt private_key: /opt/harbor/ca/10.50.10.185.key external_url: https://10.50.10.185 harbor_admin_password: Harbor12345 database: password: root123 max_idle_conns: 50 max_open_conns: 100 data_volume: /data/harbor clair: updaters_interval: 12 jobservice: max_job_workers: 10 notification: webhook_job_max_retry: 10 chart: absolute_url: disabled log: level: info local: rotate_count: 50 rotate_size: 200M location: /data/harbor/logs _version: 1.10.0 proxy: http_proxy: https_proxy: no_proxy: components: - core - jobservice - clair

4.1 运行prepare脚本以启用HTTPS

Harbor将nginx实例用作所有服务的反向代理。您可以使用prepare脚本来配置nginx为使用HTTPS

./prepare

4.2 如果Harbor正在运行，请停止并删除现有实例

您的images数据保留在文件系统中，因此不会丢失任何数据

harbor]# docker-compose down -v Stopping harbor-jobservice ... done Stopping nginx ... done Stopping harbor-core ... done Stopping registryctl ... done Stopping harbor-db ... done Stopping redis ... done Stopping registry ... done Stopping harbor-portal ... done Stopping harbor-log ... done Removing harbor-jobservice ... done Removing nginx ... done Removing harbor-core ... done Removing registryctl ... done Removing harbor-db ... done Removing redis ... done Removing registry ... done Removing harbor-portal ... done Removing harbor-log ... done Removing network harbor_harbor

4.3 重启Harbor

harbor]# docker-compose up -d Creating network "harbor_harbor" with the default driver Creating harbor-log ... done Creating harbor-db ... done Creating registryctl ... done Creating redis ... done Creating harbor-portal ... done Creating registry ... done Creating harbor-core ... done Creating harbor-jobservice ... done Creating nginx ... done

找到Harbor服务的服务目录

~]# find / -name harbor.yml -type f /opt/harbor/harbor.yml

查看Harbor服务状态

2 ~]# cd /opt/harbor/ You have mail in /var/spool/mail/root [root@p1edaspk02 harbor]# docker-compose ps Name Command State Ports ----------------------------------------------------------------------------------------------------------------- harbor-core /harbor/harbor_core Up (healthy) harbor-db /docker-entrypoint.sh Up (healthy) 5432/tcp harbor-jobservice /harbor/harbor_jobservice ... Up (healthy) harbor-log /bin/sh -c /usr/local/bin/ ... Up (healthy) 127.0.0.1:1514->10514/tcp harbor-portal nginx -g daemon off; Up (healthy) 8080/tcp nginx nginx -g daemon off; Up (healthy) 0.0.0.0:8199->8080/tcp, 0.0.0.0:443->8443/tcp redis redis-server /etc/redis.conf Up (healthy) 6379/tcp registry /home/harbor/entrypoint.sh Up (healthy) 5000/tcp registryctl /home/harbor/start.sh Up (healthy)

5. 验证HTTPS连接

打开浏览器，然后输入https://10.50.10.185。它应该显示Harbor界面

6. 推送以及拉去镜像6.1 登录harbor仓库

# docker login harbor域名地址:端口号 harbor]# docker login https://10.50.10.185 -u admin Password: WARNING! Your password will be stored unencrypted in /root/.docker/config.json. Configure a credential helper to remove this warning. See https://docs.docker.com/engine/reference/commandline/login/#credentials-store Login Succeeded

如上所示为登录成功

如果登录报错:

harbor]# docker login 10.50.10.185:8199 -u admin Password: Error response from daemon: Get https://10.50.10.185:8199/v2/: http: server gave HTTP response to HTTPS client

6.2 在harbor dashboar创建项目

项目 -> 新建项目 ->

在项目中标记镜像： docker tag SOURCE_IMAGE[:TAG] 10.50.10.185/harbortest/IMAGE[:TAG] docker tag registry.aliyuncs.com/openspug/spug:latest 10.50.10.185/harbortest/registry.aliyuncs.com/openspug/spug:latest ``` 在项目中标记镜像： docker tag SOURCE_IMAGE[:TAG] 10.50.10.185/harbortest/IMAGE[:TAG] 推送镜像到当前项目： docker push 10.50.10.185/harbortest/IMAGE[:TAG] ​```

例子：

harbor]# docker tag registry.aliyuncs.com/openspug/spug:latest 10.50.10.185/harbortest/registry.aliyuncs.com/openspug/spug:latest [root@p1edaspk02 harbor]# [root@p1edaspk02 harbor]# [root@p1edaspk02 harbor]# docker login https://10.50.10.185 -u admin Password: WARNING! Your password will be stored unencrypted in /root/.docker/config.json. Configure a credential helper to remove this warning. See https://docs.docker.com/engine/reference/commandline/login/#credentials-store Login Succeeded [root@p1edaspk02 harbor]# [root@p1edaspk02 harbor]# [root@p1edaspk02 harbor]# docker push 10.50.10.185/harbortest/registry.aliyuncs.com/openspug/spug:latest The push refers to repository [10.50.10.185/harbortest/registry.aliyuncs.com/openspug/spug] 7f7d97906ce8: Pushed 260a2403f5c7: Pushed 26b6e6155c9a: Pushed 8e1aef93890d: Pushed b220652480d3: Pushed d30f3e7469cb: Pushed 8d395243207e: Pushed 4b4158158262: Pushed 5fcede3d79f6: Pushed 72bd99349a58: Pushed 27e935fbee66: Pushed 737c272b1ba6: Pushed fb82b029bea0: Pushed latest: digest: sha256:8137ad64f0e6ae455171fd4c45a4c0ca42d069262d66f15f66a487f357312350 size: 3032

注意

然后登陆推送镜像测试， 如果服务器要推送代码到harbor， 必须在docker的配置文件的目录 /etc/docker/certs.d/10.50.10.185/ 配置 服务器证书（10.50.10.185.cert），密钥（10.50.10.185.key）和CA文件（ca.crt）

7. harbor 配合kubernetes使用7.1 k8s拉取镜像的方式

Always：当容器失效时，由Kubelet自动重启该容器。RestartPolicy的默认值。 OnFailure：当容器终止运行且退出码不为0时由Kubelet重启。 Never：无论何种情况下，Kubelet都不会重启该容器。 注意，这里的重启是指在 Pod 所在 Node 上面本地重启，并不会调度到其他 Node 上去。

7.2使用私有镜像仓库拉去镜像7.2.1 k8s-node节点添加验证

将harbor服务器上如下三个文件分发到kubernetes集群的node节点/etc/docker/certs.d/10.50.10.185/的这目录

0. 服务器证书（10.50.10.185.cert）
1. 密钥（10.50.10.185.key）
2. CA文件（ca.crt）

7.2.2 拉去镜像

在node节点执行

docker pull 10.50.10.185/harbortest/nginx:latest

如果可以成功拉取代表node节点的证书已生效

7.2.3 创建一个 docker registry secret

使用私有仓库

kubectl create secret docker-registry regsecret --docker-server=https://10.50.10.185 --docker-username=admin --docker-password=Harb2323 --docker-email=ninesun@126.com

使用 Azure Container Registry（ACR）： https://kubernetes.feisky.xyz/concepts/objects/pod

ACR_NAME=dregistry SERVICE_PRINCIPAL_NAME=acr-service-principal # Populate the ACR login server and resource id. ACR_LOGIN_SERVER=$(az acr show --name $ACR_NAME --query loginServer --output tsv) ACR_REGISTRY_ID=$(az acr show --name $ACR_NAME --query id --output tsv) # Create a contributor role assignment with a scope of the ACR resource. SP_PASSWD=$(az ad sp create-for-rbac --name $SERVICE_PRINCIPAL_NAME --role Reader --scopes $ACR_REGISTRY_ID --query password --output tsv) # Get the service principle client id. CLIENT_ID=$(az ad sp show --id http://$SERVICE_PRINCIPAL_NAME --query appId --output tsv) # Create secret kubectl create secret docker-registry acr-auth --docker-server $ACR_LOGIN_SERVER --docker-username $CLIENT_ID --docker-password $SP_PASSWD --docker-email local@local.domain

7.2.4 引用docker registry secret的两种方式直接在pod中引用

apiVersion: v1 kind: Pod metadata: name: harbor-test spec: containers: - name: harbor-test image: 10.50.10.185/harbortest/nginx:latest imagePullSecrets: - name: regsecret

secret 添加到 service account 中 s并通过ervice account 引用

显然如果将secret添加到sa中，就又多了一层抽象，不用在每个pod或者deployment中的每个container都写一遍imagePullSecrets。而且向用户屏蔽了细节。用户不需要关心

kubectl patch serviceaccount default -p '{"imagePullSecrets": [{"name": "myregistrykey"}]}'

kubectl patch serviceaccount default -p '{"imagePullSecrets": [{"name": "regsecret"}]}' /opt/k8s]#kubectl get sa -oyaml apiVersion: v1 items: - apiVersion: v1 imagePullSecrets: - name: regsecret kind: ServiceAccount metadata: creationTimestamp: "2022-03-18T12:31:44Z" name: default namespace: default resourceVersion: "4202955" uid: a9b88295-630e-4121-94e1-ab53a17f4f49 secrets: - name: default-token-qvnrc kind: List metadata: resourceVersion: "" selfLink: ""

实战

使用deployment部署nginx

apiVersion: apps/v1 kind: Deployment metadata: name: harbor-test labels: app: nginx spec: replicas: 10 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx #image: nginx:latest image: 10.50.10.185/harbortest/nginx:latest ports: - containerPort: 80

<!--== 注意 ==-->

image字段一定要写harbor全路径，否则将使用默认的源拉去镜像

如何查找harbor中的镜像

# vim Harbor-image-list-100.sh #!/bin/bash #镜像清单文件，将获取到的镜像信息存到该文件中 File=harbor-images-`date ' %Y-%m-%d'`.txt ## 定义Harbor连接地址,这里需要改为你们自己的Harbor地址 Address=https://10.50.10.185 ## 定义连接Harbor的用户名和密码（因为是获取全部的镜像，只有admin用户才有该权限） Hamin=admin:Harbor12345 ## 获取Harbor中有哪些项目（Project） Project_List=$(curl -u "$Hamin" -X GET $Address/api/projects -H "Content-Type: application/json" | grep name | awk '/"name": /' | awk -F '"' '{print $4}') for Project in $Project_List;do # 循环获取每个项目下所有的镜像 Image_Names=$(curl -u "$Hamin" -X GET $Address/api/search?q=$Project -H "Content-Type: application/json" | grep "repository_name" | awk -F "\"" '{print $4}') for Image in $Image_Names;do # 循环获取每个镜像所有的标签（版本） Image_Tags=$(curl -u "$Hamin" -X GET $Address/api/repositories/$Image/tags -H "Content-Type: application/json" | awk '/"name": /' | awk -F '"' '{print $4}') for Tag in $Image_Tags;do # 将获取到的镜像完整路径存档到镜像清单文件 echo "$Address/$Image:$Tag" | grep -v Base | grep -v Image | grep -v CentOS >> $File done done done

,