Docker swarm mode cluster under the private warehouse harbor setting example

Docker swarm mode cluster under the private warehouse harbor setup process

Description

  Test domain name: reg.goluk.cn # Intranet effective target: through the visit https://reg.goluk.cn can create a management of all projects in the private warehouse; harbor docker swarm mode as a network of warehouse, each docker node can docker Login reg.goluk.cn login, can push and pull images. 

Docker swarm mode

Docker swarm mode cluster installation and configuration

Ceph please refer to (this article is used in three mon, 4 osd, mds master and backup, to provide cephfs as a back-end storage)

Ceph on centos7.2 on the installation and basic configuration

Download the online installation package

  Wget -c https://github.com/vmware/harbor/releases 

Decompress the installation

  Tar xvf harbor-online-installer- <version> .tgz 

Modify harbor.cfg

  Hostname = reg.goluk.cn 

Ui_url_protocol = http


Email_identity = lihui@goluk.com

Email_server = smtp.mydomain.com
Email_server_port = 25
Email_username = sample_admin@mydomain.com
Email_password = abc
Email_from = admin <sample_admin@mydomain.com>
Email_ssl = false

Harbor_admin_password = Harbor12345

Auth_mode = db_auth

Ldap_url = ldaps: //ldap.mydomain.com



Ldap_basedn = ou = people, dc = mydomain, dc = com


Ldap_uid = uid

Ldap_scope = 3

Db_password = mobnote @ 123

Self_registration = on

Use_compressed_js = on

Max_job_workers = 3

Token_expiration = 30

Verify_remote_cert = on

Customize_crt = on

Crt_country = CN
Crt_state = State
Crt_location = CN
Crt_organization = goluk
Crt_organizationalunit = goluk
Crt_commonname = goluk.cn
Crt_email = lihui@goluk.com

Project_creation_restriction = everyone

Ssl_cert = /data/cert/server.crt
Ssl_cert_key = /data/cert/server.key

Run the installation script

  ./install.sh 

Modify docker-compose.yml to form docker stack compatible files

 Cat harbor2.yml 
Version: '3'
Services:
Log:
Image: vmware / harbor-log: 0.5.0
Volumes:
- / var / log / harbor /: / var / log / docker /
Ports:
- 1514: 514
Registry:
Image: library / registry: 2.5.0
Volumes:
- / mnt / cephfs / app / harbor / data / registry: / storage
- / mnt / cephfs / app / harbor / harbor / common / config / registry /: / etc / registry /
Environment:
- GODEBUG = netdns = cgo
Command:
[Serve "," /etc/registry/config.yml "]
Depends_on
- log
Logging:
Driver: "syslog"
Options:
Syslog-address: "tcp: //127.0.0.1: 1514"
Tag: "registry"
Mysql:
Image: vmware / harbor-db: 0.5.0
Volumes:
- / mnt / cephfs / app / harbor / data / database: / var / lib / mysql
Environment:
- MYSQL_ROOT_PASSWORD = mobnote @ 123
Depends_on
- log
Logging:
Driver: "syslog"
Options:
Syslog-address: "tcp: //127.0.0.1: 1514"
Tag: "mysql"
Ui
Image: vmware / harbor-ui: 0.5.0
Environment:
- MYSQL_HOST = mysql
- MYSQL_PORT = 3306
- MYSQL_USR = root
- MYSQL_PWD = mobnote @ 123
- REGISTRY_URL = http: // registry: 5000
- JOB_SERVICE_URL = http: // jobservice
- UI_URL = http: // ui
- CONFIG_PATH = / etc / ui / app.conf
- EXT_REG_URL = reg.goluk.cn
- HARBOR_ADMIN_PASSWORD = Harbor12345
- AUTH_MODE = db_auth
- LDAP_URL = ldaps: //ldap.mydomain.com
- LDAP_SEARCH_DN =
- LDAP_SEARCH_PWD =
- LDAP_BASE_DN = ou = people, dc = mydomain, dc = com
- LDAP_FILTER =
- LDAP_UID = uid
- LDAP_SCOPE = 3
- UI_SECRET = YEiVW92oM0szGsWa
- SECRET_KEY = 4tDRVqYEj4YjCdNI
- SELF_REGISTRATION = on
- USE_COMPRESSED_JS = on
- LOG_LEVEL = debug
- GODEBUG = netdns = cgo
- EXT_ENDPOINT = http: //reg.goluk.cn
- TOKEN_ENDPOINT = http: // ui
- VERIFY_REMOTE_CERT = on
- TOKEN_EXPIRATION = 30
- PROJECT_CREATION_RESTRICTION = everyone
Volumes:
- /mnt/cephfs/app/harbor/harbor/common/config/ui/app.conf:/etc/ui/app.conf
- /mnt/cephfs/app/harbor/harbor/common/config/ui/private_key.pem:/etc/ui/private_key.pem
- / mnt / cephfs / app / harbor / data: / harbor_storage
Depends_on
- log
Logging:
Driver: "syslog"
Options:
Syslog-address: "tcp: //127.0.0.1: 1514"
Tag: "ui"
Jobservice:
Image: vmware / harbor-jobservice: 0.5.0
Environment:
- MYSQL_HOST = mysql
- MYSQL_PORT = 3306
- MYSQL_USR = root
- MYSQL_PWD = mobnote @ 123
- UI_SECRET = YEiVW92oM0szGsWa
- SECRET_KEY = 4tDRVqYEj4YjCdNI
- CONFIG_PATH = / etc / jobservice / app.conf
- REGISTRY_URL = http: // registry: 5000
- VERIFY_REMOTE_CERT = on
- MAX_JOB_WORKERS = 3
- LOG_LEVEL = debug
- LOG_DIR = / var / log / jobs
- GODEBUG = netdns = cgo
- EXT_ENDPOINT = http: //reg.goluk.cn
- TOKEN_ENDPOINT = http: // ui
Volumes:
- / mnt / cephfs / app / harbor / data / job_logs: / var / log / jobs
- /mnt/cephfs/app/harbor/harbor/common/config/jobservice/app.conf:/etc/jobservice/app.conf
Depends_on
- ui
Logging:
Driver: "syslog"
Options:
Syslog-address: "tcp: //127.0.0.1: 1514"
Tag: "jobservice"
Proxy:
Image: nginx: 1.11.5
Volumes:
- / mnt / cephfs / app / harbor / harbor / common / config / nginx: / etc / nginx
Ports:
- 80:80
- 443: 443
Depends_on
- mysql
- registry
- ui
- log
Logging:
Driver: "syslog"
Options:
Syslog-address: "tcp: //127.0.0.1: 1514"
Tag: "proxy"

Run harbor

  Docker stack deploy -c harbor2.yml harbor 
[Root @ swarm2 ~] # docker stack ls
NAME SERVICES
Harbor 6

View running services

  [Root @ swarm2 ~] # docker stack services harbor 
ID NAME MODE REPLICAS IMAGE
Eyipo6gng5su harbor_jobservice replicated 1/1 vmware / harbor-jobservice: 0.5.0
Mzbeq1oqguqd harbor_registry replicated 1/1 library / registry: 2.5.0
N3sbohiie3x4 harbor_mysql replicated 1/1 vmware / harbor-db: 0.5.0
Prf0jhe0j31x harbor_ui replicated 1/1 vmware / harbor-ui: 0.5.0
Qcmxi1g8x16f harbor_proxy replicated 1/1 nginx: 1.11.5
Ss29zpgdmlut harbor_log replicated 1/1 vmware / harbor-log: 0.5.0

Configure the self-signed certificate for harbor

Create a self-signed ca certificate

  Openssl req \ 
-newkey rsa: 4096 -nodes -sha256 -keyout ca.key \
-x509 -days 365 -out ca.crt
[Root @ swarm2 ca] # openssl req \
> -newkey rsa: 4096 -nodes -sha256 -keyout ca.key \
> -x509 -days 365 -out ca.crt
Generating a 4096 bit RSA private key
.................................................. .................................................. .................................................. .................................................. .................................................. .................................................. ......................................... ++
.................................................. ...................... ++
Writing new private key to 'ca.key'
-----
You are about to be asked to enter information that will be incorporated
Into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN?
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', The field will be left blank.
-----
Country Name (2 letter code) [XX]: cn
State or Province Name (full name) []: Beijing
Locality Name (eg, city) [Default City]: Chaoyang
Organization Name (eg, company) [Default Company Ltd]: Goluk
Organizational Unit Name (eg, section) []:
Common Name (eg, your name or your server's hostname) []: reg.goluk.cn
Email Address []: lihui@goluk.com
[Root @ swarm2 ca] # ls
Ca.crt ca.key

Generate a certificate signing request

  Openssl req \ 
-newkey rsa: 4096 -nodes -keyout reg.goluk.cn.key \
-out reg.goluk.cn.csr

[Root @ swarm2 ca] # openssl req -newkey rsa: 4096 -nodes -sha256 -keyout reg.goluk.cn.key -out reg.goluk.cn.csr
Generating a 4096 bit RSA private key
.................................................. .................................................. ............ ++
.................................................. .................................................. .................................................. ............ ++
Writing new private key to 'reg.goluk.cn.key'
-----
You are about to be asked to enter information that will be incorporated
Into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN?
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', The field will be left blank.
-----
Country Name (2 letter code) [XX]: cn
State or Province Name (full name) []: Beijing
Locality Name (eg, city) [Default City]: Chaoyang
Organization Name (eg, company) [Default Company Ltd]: Goluk
Organizational Unit Name (eg, section) []:
Common Name (eg, your name or your server's hostname) []: reg.goluk.cn
Email Address []: lihui@goluk.com

Please enter the following 'extra' attributes
To be sent with your certificate request
A challenge password []:
An optional company name []:
[Root @ swarm2 ca] # ls -al
Total amount of 11
Drwxr-xr-x 1 root root 4 月 13 月 16 日 16:00.
Drwxr-xr-x 1 root root 15 February 13 15:54 ..
-rw-r-r-- 1 root root 2037 February 13 15:55 ca.crt
-rw-r-r-- 1 root root 3272 February 13 15:55 ca.key
-rw-r - r-- 1 root root 1708 February 13 16:00 reg.goluk.cn.csr
-rw-r - r-- 1 root root 3272 February 13 16:00 reg.goluk.cn.key

Generate a certificate for the private warehouse host

  Openssl x509 -req -days 365 -in reg.goluk.cn.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out reg.goluk.cn.crt 
[Root @ swarm2 ca] # openssl x509 -req -days 365 -in reg.goluk.cn.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out reg.goluk.cn.crt
Signature ok
Subject = / C = cn / ST = Beijing / L = Chaoyang / O = Goluk / CN = reg / emailAddress=lihui@goluk.com
Getting CA Private Key

Configure the certificate to the harbor directory

  Cp reg.goluk.cn.crt / mnt / cephfs / app / harbor / data / cert 
Cp reg.goluk.cn.key / mnt / cephfs / app / harbor / data / cert

Modify harbor.cfg

  Ui_url_protocol = https 
Crt_country = cn ## This line to crt_email
The line needs to be consistent with the value of the above self-signed certificate
Crt_state = Beijing
Crt_location = cn
Crt_organization = Goluk
Crt_organizationalunit =
Crt_commonname = reg.goluk.cn
Crt_email = lihui@goluk.com
Ssl_cert = /mnt/cephfs/app/harbor/data/cert/reg.goluk.cn.crt ## Fill in the file and path of the certificate
Ssl_cert_key = /mnt/cephfs/app/harbor/data/cert/reg.goluk.cn.key ## ibid

Run prepare.sh

  ./prepare.sh 

OK to stop harbor running

  Docker stack rm harbor 

Re-run harbor

  Docker stack deploy -c harbor2.yml 

Configure each docker host that needs to access harbor

Place the reg certificate for reg.goluk.cn in the directory of the docker's configuration certificate

  [Root @ swarm3 ~] # mkdir -p /etc/docker/certs.d/reg.goluk.cn 
[Root @ swarm3 ~] # cp /mnt/cephfs/app/harbor/harbor/ca/ca.crt/etc/docker/certs.d/reg.goluk.cn/

Configure the system level of the docker host to trust the self-signed certificate

  [Root @ swarm3 ~] #cp /mnt/cephfs/app/harbor/data/cert/reg.goluk.cn.crt / etc / pki / ca-trust / source / anchors / 
Update-ca-trust ## Update the list of trusted certificates

Docker login test

  Docker login reg.goluk.cn 

Log in harbor ui build a base repository and test push image

Docker images list

  [Root @ swarm3 ~] # docker images 
REPOSITORY TAG IMAGE ID CREATED SIZE
Reg.goluk.cn/base/harbor-log latest eebc987a891b 2 months ago 190 MB
Vmware / harbor-log 0.5.0 eebc987a891b 2 months ago 190 MB
Vmware / harbor-jobservice 0.5.0 995368e96860 2 months ago 169 MB
Vmware / harbor-ui 0.5.0 232a8664541a 2 months ago 233 MB
Vmware / harbor-db 0.5.0 84c4ce8e9b6c 2 months ago 327 MB
Nginx 1.11.5 05a60462f8ba 3 months ago 181 MB
Registry 2.5.0 c6c14b3960bd 6 months ago 33.3 MB

Docker tag

  [Root @ swarm3 ~] # docker tag eebc987a891b reg.goluk.cn/base/harbor-log 
[Root @ swarm3 ~] # docker tag 232a8664541a reg.goluk.cn/base/harbor-ui
[Root @ swarm3 ~] # docker tag 995368e96860 reg.goluk.cn/base/harbor-jobservice
[Root @ swarm3 ~] # docker tag 84c4ce8e9b6c reg.goluk.cn/base/harbor/db
[Root @ swarm3 ~] # docker tag 05a60462f8ba reg.goluk.cn/base/nginx
[Root @ swarm3 ~] # docker tag c6c14b3960bd reg.goluk.cn/base/registry

Docker push

  [Root @ swarm3 ~] # docker push reg.goluk.cn/base/registry 
The push reference to a repository [reg.goluk.cn/base/registry]
3bb5bc5ad373: Pushed
35039a507f7a: Pushed
D00444e19d65: Pushed
Aa3a31ee27f3: Pushed
4fe15f8d0ae6: Pushed
Latest: digest: sha256: 04cc36f8f72c4272f07325075586b3a0a73db23d3822a7ed1ce34f86f3f410c3 size: 1363
...

Login harbor ui can see the mirror has been pushed

Migrate original images to harbor warehouse

Set the docker host to trust the self-signed certificate

Ibid

Docker tag

Ibid

Docker push

Ibid

Heads up! This alert needs your attention, but it's not super important.