proxy 部署多个swift

2013-02-21 17:27:33 sraing 阅读数 2846

5台服务器 A B C D E

A为负载均衡

BCDE 既做proxy server 又做storage server


负载均衡工具的选择:

nginx:0.6.33版本后不允许POST方法访问静态资源,swift 只有head get方法能用,post put等方法被阻挡在nginx,无法下发到proxyserver。错误表现为:405 Not Allowed

修改配置文件和改代码的方法都没有解决。


pound  CPU负载过高(据说)


HAProxy  :配置简单,均衡策略多样,负载小。还有一个简单的web监控页面,很方便。

使用roundrobin 策略时,引发了多proxy server环境下 swift tempauth验证方式的问题,根本原因是多proxy之间无法共享各自产生的token。

又在A上部署了keystone,将B C D E上swift的验证方式改为 keystone,成功!



未完待续...


性能待测...


2015-06-04 21:17:45 chinagissoft 阅读数 2042

1.Swift介绍

1.1.       Swift架构概述

官方给出的标准架构进行分析,如图,分别为存储节点(Storage node)、代理节点(Proxy node)和认证节点(Auth node)三部分。


1.1.1.   代理节点

代理节点可以说是Swift的核心,运行着swift-proxy-server进程。它提供Swift API的服务,负责Swift其余组件间的相互通信。对于每个客户端的请求,它在Ring中查询相应Account、Container以及Object的位置,并且转发这些请求。从安全的角度来说,很少有直接暴露底层服务接口的实现方式,这个代理可以做集群。

它提供了Rest-full API,开发者可以通过这个接口快捷构建定制的客户端与Swift交互。

大量的存储错误也由Proxy Server处理:当一个服务器无法对一个对象的PUT操作进行响应,它将从Ring中查询一个可以接手的服务器并将请求传递给它。

² Ring

Ring是数据到物理设备映射的集合,通过Zone(区域)、Device(设备)、Partition(分区)和Replica(副本)来维护映射信息。

1.1.2.   存储节点

存储节点需要运行account和容器、对象存储等服务。

² Object Server

对象服务是一个简单的二进制大对象存储服务,可以用来存储、检索和删除本地设备上的对象。在文件系统上,对象以二进制文件的形式存储,它的元数据存储在文件系统的扩展属性(xattrs)中。这要求用于对象服务的文件系统需要支持文件有扩展属性。

² Container Server

容器服务的首要工作是处理对象的列表。容器服务并不知道对象存在哪,只知道指定容器里存的哪些对象。 这些对象信息以sqlite数据库文件的形式存储,和对象一样在集群上做类似的备份。容器服务也做一些跟踪统计,比如对象的总数,容器的使用情况。 

² Account Server

帐号服务与容器服务非常相似,除了它是负责处理容器的列表而不是对象。1.3.   认证节点

在存储节点和代理节点接入外网之前,首先要通过一个附加的认证节点对用户进行比对认证。在早期,Swift使用单独的认证服务对用户进行核对,随着OpenStack项目的逐渐发展,目前Swift以及结合了OpenStack的认证Keystone,形成统一的云认证体系。


在官方文档中建议至少使用5个zone,一个zone就是一组独立节点,官方给出的标配例图如图所示:


1.3.       组件间的通信

Swift代理节点与存储节点间的通信方式如下:


1.4.       存储原理

Swift利用一致性哈希算法构建了一个冗余的可扩展的分布式对象存储集群,但一致性哈希算法不能保证绝对的平衡,当节点较少的时候,需要存储的对象不能被均匀地映射到节点上,这样每个节点的存储压力不平衡。为了解决这个问题,swift在一致性哈希算法的基础上,引入虚拟节点(partition)的概念。

虚拟节点是实际节点在环形空间的复制品,一个实际节点对应了若干个虚拟节点,通过中间一层虚拟节点的存储再分配,数据可以比较均匀得分布在各个节点上了。如图所示:

由此可见,Storage node与partition,partition与object间的映射关系如下:


在Storage node上运行着Linux系统并使用了XFS文件系统,逻辑上使用一致性哈希算法将固定总数的partition映射到每个Storage node上,每个object也使用同样的哈希算法映射到Partition上。


安装规划

通过上面对Swift的原理解释,我们在原有的控制节点、计算节点的基础上,新增加了三个存储节点,关于Swift的代理服务器和授权服务器,都安装在控制节点上。

三个存储服务器:

  • Storage1:192.168.3.190
  • Storage2:192.168.3.191
  • Storage3:192.168.3.192
保证所有的节点,防火墙关闭和时间同步。

实际部署

在控制节点操作如下:
1、创建swift用户和角色
keystone user-create --name=swift --pass=swift --email=swift@example.com
sm@controller:~$ keystone user-create --name=swift --pass=swift --email=swift@example.com
+----------+----------------------------------+
| Property |              Value               |
+----------+----------------------------------+
|  email   |        swift@example.com         |
| enabled  |               True               |
|    id    | 895cdefbbffc40579cdde991ce0e4eca |
|   name   |              swift               |
| username |              swift               |
+----------+----------------------------------+

创建角色
keystone user-role-add --user=swift --tenant=service --role=admin

2、创建swift服务
keystone service-create --name=swift --type=object-store --description="OpenStack Object Storage"
sm@controller:~$ keystone service-create --name=swift --type=object-store --description="OpenStack Object Storage"
+-------------+----------------------------------+
|   Property  |              Value               |
+-------------+----------------------------------+
| description |     OpenStack Object Storage     |
|   enabled   |               True               |
|      id     | 2414df50d9d8481f8bfc559e08705aab |
|     name    |              swift               |
|     type    |           object-store           |
+-------------+----------------------------------+

3、创建Swift接入点
keystone endpoint-create --service-id=$(keystone service-list | awk '/ object-store / {print$2}') --
publicurl='http://192.168.3.180:8080/v1/AUTH_%(tenant_id)s' --internalurl='http://192.168.3.180:8080/v1/AUTH_
%(tenant_id)s' --adminurl=http://192.168.3.180:8080
sm@controller:~$ keystone endpoint-create --service-id=$(keystone service-list | awk '/ object-store / {print$2}') --

publicurl='http://192.168.3.180:8080/v1/AUTH_%(tenant_id)s' --internalurl='http://192.168.3.180:8080/v1/AUTH_

%(tenant_id)s' --adminurl=http://192.168.3.180:8080
+-------------+-------------------------------------------------+
|   Property  |                      Value                      |
+-------------+-------------------------------------------------+
|   adminurl  |            http://192.168.3.180:8080            |
|      id     |         bfb3916d18674ca49aaee613abeab55b        |
| internalurl | http://192.168.3.180:8080/v1/AUTH_%(tenant_id)s |
|  publicurl  | http://192.168.3.180:8080/v1/AUTH_%(tenant_id)s |
|    region   |                    regionOne                    |
|  service_id |         2414df50d9d8481f8bfc559e08705aab        |
+-------------+-------------------------------------------------+

4、创建swift文件夹
(注:这里既包括控制节点(验证服务器和代理服务器),也包括所有的存储节点)
sudo mkdir -p /etc/swift


5、编辑Swift配置文件
(注:这里既包括控制节点(验证服务器和代理服务器),也包括所有的存储节点)
sudo vi /etc/swift/swift.conf
[swift-hash]
# random unique string that can never change (DO NOT LOSE)
swift_hash_path_prefix = xrfuniounenqjnw
swift_hash_path_suffix = fLIbertYgibbitZ

在所有的存储节点操作如下

1、安装软件
sudo apt-get install swift swift-account swift-container swift-object xfsprogs

2、对所有存储节点添加硬盘进行分区。
我使用VMWare虚拟机,为所有的存储节点添加一块20GB的硬盘,然后对新的存储区间进行分区操作。
root@storage1:~# fdisk -l

Disk /dev/sda: 128.8 GB, 128849018880 bytes
255 heads, 63 sectors/track, 15665 cylinders, total 251658240 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0004a791

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048   247463935   123730944   83  Linux
/dev/sda2       247465982   251656191     2095105    5  Extended
/dev/sda5       247465984   251656191     2095104   82  Linux swap / Solaris

Disk /dev/sdb: 21.5 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders, total 41943040 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/sdb doesn't contain a valid partition table
root@storage1:~# fdisk /dev/sdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0x763836c7.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): n
Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p): p
Partition number (1-4, default 1): 1
First sector (2048-41943039, default 2048): +10G
Last sector, +sectors or +size{K,M,G} (20971520-41943039, default 41943039):
Using default value 41943039

Command (m for help): p

Disk /dev/sdb: 21.5 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders, total 41943040 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x763836c7

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1        20971520    41943039    10485760   83  Linux

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
root@storage1:~# fdisk -l

Disk /dev/sda: 128.8 GB, 128849018880 bytes
255 heads, 63 sectors/track, 15665 cylinders, total 251658240 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0004a791

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048   247463935   123730944   83  Linux
/dev/sda2       247465982   251656191     2095105    5  Extended
/dev/sda5       247465984   251656191     2095104   82  Linux swap / Solaris

Disk /dev/sdb: 21.5 GB, 21474836480 bytes
213 heads, 34 sectors/track, 5791 cylinders, total 41943040 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x763836c7

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1        20971520    41943039    10485760   83  Linux

3、格式化磁盘为xfs格式 mkfs.xfs /dev/sdb1
root@storage1:~# mkfs.xfs /dev/sdb1
meta-data=/dev/sdb1              isize=256    agcount=4, agsize=655360 blks
         =                       sectsz=512   attr=2, projid32bit=0
data     =                       bsize=4096   blocks=2621440, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal log           bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0



4、 添加开机自启动
echo "/dev/sdb1 /srv/node/sdb1 xfs noatime,nodiratime,nobarrier,logbufs=8 0 0" >> /etc/fstab
5、 创建挂载目录mkdir -p /srv/node/sdb1
6、 挂载磁盘mount /srv/node/sdb1
7、 设置目录权限chown -R swift:swift /srv/node
8、配置rsync,创建配置文件sudo vi /etc/rsyncd.conf
uid = swift
gid = swift
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
address = 127.0.0.1(该IP为存储节点的物理IP)

[account]
max connections = 2
path = /swift/node
read only = false
lock file = /var/lock/account.lock

[container]
max connections = 2
path = /swift/node
read only = false
lock file = /var/lock/container.lock

[object]
max connections = 2
path = /swift/node
read only = false
lock file = /var/lock/object.lock

9、修改配置文件sudo vi /etc/default/rsync
将RSYNC_ENABLE=true设置为true

10、启动rsync服务;sudo service rsync start

11、创建swift的recon文件夹并设置权限
sudo  mkdir -p /var/swift/recon
sudo chown -R swift:swift /var/swift/recon


控制节点操作(Proxy服务器)

1、安装软件
sudo apt-get install swift swift-proxy memcached python-keystoneclient python-swiftclient python-webob

2、sudo vi /etc/memcached.conf
-l PROXY_LOCAL_NET_IP(192.168.3.180)

3、重启memcached服务 sudo service memcached restart

4、修改代理服务器配置文件sudo vi /etc/swift/proxy-server.conf
[DEFAULT]
bind_port = 8080
user = swift
[pipeline:main]
pipeline = healthcheck cache authtoken keystoneauth proxy-server
[app:proxy-server]
use = egg:swift#proxy
allow_account_management = true
account_autocreate = true
[filter:keystoneauth]
use = egg:swift#keystoneauth
operator_roles = Member,admin,swiftoperator
[filter:authtoken]
paste.filter_factory = keystoneclient.middleware.auth_token:filter_factory
# Delaying the auth decision is required to support token-less
# usage for anonymous referrers ('.r:*').
delay_auth_decision = true
# auth_* settings refer to the Keystone server
auth_protocol = http
auth_host = 192.168.3.180
auth_port = 35357

# the service tenant and swift username and password created in Keystone
admin_tenant_name = service
admin_user = swift
admin_password = swift
[filter:cache]
use = egg:swift#memcache
[filter:catch_errors]
use = egg:swift#catch_errors
[filter:healthcheck]
use = egg:swift#healthcheck

5、创建ring
 cd /etc/swift
 sudo swift-ring-builder account.builder create 18 3 1
 sudo swift-ring-builder container.builder create 18 3 1
 sudo swift-ring-builder object.builder create 18 3 1

注:后面的数字第一个为存储分区数,第二个为备份次数,备份次数不能大于ZONE数目,第三个为延迟删除的时间,以小时为单位


6、向ring添加设备
swift-ring-builder account.builder add z1-192.168.3.190:6002R192.168.3.190:6005/sdb1 100
swift-ring-builder container.builder add z1-192.168.3.190:6001R192.168.3.190:6004/sdb1 100
swift-ring-builder object.builder add z1-192.168.3.190:6000R192.168.3.190:6003/sdb1 100


swift-ring-builder account.builder add z2-192.168.3.191:6002R192.168.3.191:6005/sdb1 100
swift-ring-builder container.builder add z2-192.168.3.191:6001R192.168.3.191:6004/sdb1 100
swift-ring-builder object.builder add z2-192.168.3.191:6000R192.168.3.191:6003/sdb1 100


swift-ring-builder account.builder add z3-192.168.3.192:6002R192.168.3.192:6005/sdb1 100
swift-ring-builder container.builder add z3-192.168.3.192:6001R192.168.3.192:6004/sdb1 100
swift-ring-builder object.builder add z3-192.168.3.192:6000R192.168.3.192:6003/sdb1 100

7、检验每个环里面的条目
sudo swift-ring-builder account.builder
sudo  swift-ring-builder container.builder
sudo  swift-ring-builder object.builder
sm@controller:/etc/swift$ sudo swift-ring-builder account.builder
account.builder, build version 1
262144 partitions, 3.000000 replicas, 1 regions, 1 zones, 1 devices, 100.00 balance
The minimum number of hours before a partition can be reassigned is 1
Devices: id region zone ip address port replication ip  replication port name weight partitions balance meta
0       1     1   192.168.3.190  6002   192.168.3.190              6005      sdb1 100.00          0 -

100.00
sm@controller:/etc/swift$ sudo  swift-ring-builder container.builder
container.builder, build version 1
262144 partitions, 3.000000 replicas, 1 regions, 1 zones, 1 devices, 100.00 balance
The minimum number of hours before a partition can be reassigned is 1
Devices:    id  region  zone      ip address  port  replication ip  replication port      name weight partitions balance meta
0       1     1   192.168.3.190  6001   192.168.3.190       6004      sdb1 100.00          0 -100.00
sm@controller:/etc/swift$ sudo  swift-ring-builder object.builder
object.builder, build version 1
262144 partitions, 3.000000 replicas, 1 regions, 1 zones, 1 devices, 100.00 balance
The minimum number of hours before a partition can be reassigned is 1
Devices:    id  region  zone      ip address  port  replication ip  replication port      name weight partitions balance meta
 0       1     1   192.168.3.190  6000   192.168.3.190      6003      sdb1 100.00          0 -100.00

8、平衡环
sudo   swift-ring-builder account.builder rebalance
sudo   swift-ring-builder container.builder rebalance
sudo   swift-ring-builder object.builder rebalance
sm@controller:/etc/swift$ sudo swift-ring-builder account.builder rebalance
Reassigned 262144 (100.00%) partitions. Balance is now 100.00.
-------------------------------------------------------------------------------
NOTE: Balance of 100.00 indicates you should push this
      ring, wait at least 1 hours, and rebalance/repush.
-------------------------------------------------------------------------------
sm@controller:/etc/swift$ sudo swift-ring-builder container.builder rebalance
Reassigned 262144 (100.00%) partitions. Balance is now 100.00.
-------------------------------------------------------------------------------
NOTE: Balance of 100.00 indicates you should push this
      ring, wait at least 1 hours, and rebalance/repush.
-------------------------------------------------------------------------------
sm@controller:/etc/swift$ sudo swift-ring-builder object.builder rebalance
Reassigned 262144 (100.00%) partitions. Balance is now 100.00.
-------------------------------------------------------------------------------
NOTE: Balance of 100.00 indicates you should push this
      ring, wait at least 1 hours, and rebalance/repush.
-------------------------------------------------------------------------------

该操作可能会稍等一段时间,执行完毕之后,会在执行目录下面生成三个*.gz文件,我们需要将这三个文件同步到所有存储节点的/etc/swift文件夹中
sm@controller:/etc/swift$ ll
total 6320
drwxr-xr-x   3 swift swift    4096 Jun  1 14:21 ./
drwxr-xr-x 112 root  root     4096 Jun  1 14:12 ../
-rw-r--r--   1 swift swift 2100180 Jun  1 14:21 account.builder
-rw-r--r--   1 root  root    48018 Jun  1 14:21 account.ring.gz
drwxr-xr-x   2 swift swift    4096 Jun  1 14:21 backups/
-rw-r--r--   1 swift swift 2100182 Jun  1 14:21 container.builder
-rw-r--r--   1 root  root    48065 Jun  1 14:21 container.ring.gz
-rw-r--r--   1 swift swift 2100182 Jun  1 14:21 object.builder
-rw-r--r--   1 root  root    48101 Jun  1 14:21 object.ring.gz
-rw-r--r--   1 swift swift     938 May 17 04:43 proxy-server.conf
-rw-r--r--   1 swift swift     155 May 17 03:58 swift.conf

9、修改所有节点的/etc/swift文件夹权限
sudo chown -R swift:swift /etc/swift
10、重启代理服务器服务 sudo service swift-proxy restart

所有存储节点操作


1、重启存储节点的服务 sudo swift-init all start
root@storage1:/etc/swift# sudo swift-init all start
container-updater running (1803 - /etc/swift/container-server.conf)
container-updater already started...
account-auditor running (1868 - /etc/swift/account-server.conf)
account-auditor already started...
object-replicator running (1721 - /etc/swift/object-server.conf)
object-replicator already started...
Unable to locate config for proxy-server
container-replicator running (1789 - /etc/swift/container-server.conf)
container-replicator already started...
object-auditor running (1747 - /etc/swift/object-server.conf)
object-auditor already started...
Unable to locate config for object-expirer
Starting container-auditor...(/etc/swift/container-server.conf)
container-server running (1773 - /etc/swift/container-server.conf)
container-server already started...
account-server running (1821 - /etc/swift/account-server.conf)
account-server already started...
account-reaper running (1851 - /etc/swift/account-server.conf)
account-reaper already started...
Starting container-sync...(/etc/swift/container-server.conf)
account-replicator running (1837 - /etc/swift/account-server.conf)
account-replicator already started...
object-updater running (1734 - /etc/swift/object-server.conf)
object-updater already started...
object-server running (1699 - /etc/swift/object-server.conf)
object-server already started...


Swift验证

1、查看统计信息
sm@controller:~$ swift stat
       Account: AUTH_b500268a41e34752aba847c7ea464870
    Containers: 0
       Objects: 0
         Bytes: 0
  Content-Type: text/plain; charset=utf-8
   X-Timestamp: 1433141462.32076
    X-Trans-Id: tx4c9d683713834709bac22-00556c00d6
X-Put-Timestamp: 1433141462.32076

2、编辑两个文本文件,然后上传,再次查看统计信息
sm@controller:~$ vi test.txt
sm@controller:~$ vi test2.txt
sm@controller:~$ swift upload myfiles test.txt
test.txt
sm@controller:~$ swift upload myfiles test2.txt
test2.txt
sm@controller:~$ swift stat
       Account: AUTH_b500268a41e34752aba847c7ea464870
    Containers: 1
       Objects: 1
         Bytes: 18
 Accept-Ranges: bytes
   X-Timestamp: 1433141536.45037
    X-Trans-Id: tx14074b3b7bdf458bb7486-00556c012d
  Content-Type: text/plain; charset=utf-8


通过horizon查看

打开horizon,我们可以看到左边菜单包含一个容器界面,刚刚上传的两个文本文件也可以查看到,同样,我们也可以通过horizon管理界面进行上传文件。


通过简单的使用Swift,还没有发现更多Swift的好处,目前只感觉可以上传和下载,或者说就像帮助所说,Swift可以将廉价的硬盘在没有RAID的情况下,实现类似于RAID的功能,如果只是上传和下载功能,这个也不是我所需要的,假设用户有存储设备,个人认为没有必要安装Swift组件,也许Swift更多的功能还在持续研究和发现中。
2018-02-12 11:31:44 ch648966459 阅读数 2725

环境

  • 硬件

    这里只使用了一台服务器,既作为Controller Node,也作为Storage Node

主机名 IP OS 磁盘 文件系统
sf-dev 10.202.127.4 Centos-7.4 /dev/sdb
/dev/sdc
/dev/sdd
XFS
  • 软件

    我们使用Openstack Pike版本

  • 配置可用的Openstack源

    这里使用了清华开源镜像。配置服务器镜像:

    cd /etc/yum.repos.d/
    vim CentOS-Base.repo
    

    增加如下配置

    ...
    [openstack]
    name=Openstack
    baseurl=https://mirrors.tuna.tsinghua.edu.cn/centos/7/cloud/x86_64/openstack-pike/
    gpgcheck=0
    ...
    

    使用YUM跟新库

    yum update -y
    

Swift组件安装于配置

1. 安装必要的组件包

# yum install openstack-swift-proxy python-swiftclient \
  python-keystoneclient python-keystonemiddleware \
  memcached
  • 从Swift源镜像获取代理服务配置文件,并进行配置

    # curl -o /etc/swift/proxy-server.conf https://git.openstack.org/cgit/openstack/swift/plain/etc/proxy-server.conf-sample?h=stable/pike
    

    编辑代理服务器配置文件/etc/swift/proxy-server.conf

    1. 编辑 [DEFAULT] 段内容,配置如下内容

      [DEFAULT]
      ...
      bind_port = 8080
      user = swift
      swift_dir = /etc/swift
      
    2. 编辑 [pipeline:main]

      [pipeline:main]
      pipeline = catch_errors gatekeeper healthcheck proxy-logging cache container_sync bulk tempurl ratelimit tempauth copy container-quotas account-quotas slo dlo versioned_writes proxy-logging proxy-server  
      
    3. 编辑 [app:proxy-server] 段内容,允许自动创建账户

      [app:proxy-server]
      use = egg:swift#proxy
      ...
      account_autocreate = True
      
    4. [filter:tempauth] 段中,设置允许的的账户/用户

      [filter:tempauth]
      ...
      user_admin_admin = admin .admin .reseller_admin
      user_test_tester = testing .admin
      
    5. [filter:cache] 段中,设置memcache位置

      [filter:cache]
      use = egg:swift#memcache
      ...
      memcache_servers = 127.0.0.1:11211
      

注意:如果控制节点与存储节点分离,以上配置只需在控制节点进行配置,如果使用keystone请参考配置说明,更详细的说明请阅读官方文档

2. 存储节点组件包安装于配置说明

  • 安装实用软件包

    # yum install xfsprogs rsync
    
  • 格式化磁盘

    # mkfs.xfs /dev/sdb -f
    # mkfs.xfs /dev/sdc -f
    # mkfs.xfs /dev/sdd -f
    
  • 创建mount点

    # mkdir -p /srv/node/sdb
    # mkdir -p /srv/node/sdc
    # mkdir -p /srv/node/sdd
    
  • 编辑 /etc/fstab 文件,添加如下内容:

    /dev/sdb /srv/node/sdb xfs noatime,nodiratime,nobarrier,logbufs=8 0 2
    /dev/sdc /srv/node/sdc xfs noatime,nodiratime,nobarrier,logbufs=8 0 2
    /dev/sdd /srv/node/sdd xfs noatime,nodiratime,nobarrier,logbufs=8 0 2
    
  • Mount 设备

    # mount /srv/node/sdb
    # mount /srv/node/sdc
    # mount /srv/node/sdd
    
  • 创建并编辑 /etc/rsyncd.conf 文件,内容如下:

    uid = swift
    gid = swift
    log file = /var/log/rsyncd.log
    pid file = /var/run/rsyncd.pid
    address = 10.202.127.4 #控制节点IP
    
    [account]
    max connections = 2
    path = /srv/node/
    read only = False
    lock file = /var/lock/account.lock
    
    [container]
    max connections = 2
    path = /srv/node/
    read only = False
    lock file = /var/lock/container.lock
    
    [object]
    max connections = 2
    path = /srv/node/
    read only = False
    lock file = /var/lock/object.lock
    
  • 存储节点软件包安装

    # yum install openstack-swift-account openstack-swift-container \
      openstack-swift-object
    
  • 从Swift源镜像获取账户(accounting)、容器(container)以及对象(object)服务配置文件,并进行配置

    # curl -o /etc/swift/account-server.conf https://git.openstack.org/cgit/openstack/swift/plain/etc/account-server.conf-sample?h=stable/pike
    # curl -o /etc/swift/container-server.conf https://git.openstack.org/cgit/openstack/swift/plain/etc/container-server.conf-sample?h=stable/pike
    # curl -o /etc/swift/object-server.conf https://git.openstack.org/cgit/openstack/swift/plain/etc/object-server.conf-sample?h=stable/pike
    
  • 编辑账户服务配置文件 /etc/swift/account-server.conf

    1. [DEFAULT] 中配置如下信息:

      [DEFAULT]
      ...
      bind_ip = 10.202.127.4
      bind_port = 6202
      user = swift
      swift_dir = /etc/swift
      devices = /srv/node
      mount_check = True
      
    2. 编辑 [pipeline:main]

      [pipeline:main]
      pipeline = healthcheck recon account-server
      
    3. [filter:recon] 中,设置recon cache 目录:

      [filter:recon]
      use = egg:swift#recon
      ...
      recon_cache_path = /var/cache/swift
      
  • 编辑账户服务配置文件 /etc/swift/container-server.conf

    1. [DEFAULT] 中配置如下信息:

      [DEFAULT]
      ...
      bind_ip = 10.202.127.4
      bind_port = 6201
      user = swift
      swift_dir = /etc/swift
      devices = /srv/node
      mount_check = True
      
    2. 编辑 [pipeline:main]

      [pipeline:main]
      pipeline = healthcheck recon container-server
      
    3. [filter:recon] 中,设置recon cache 目录:

      [filter:recon]
      use = egg:swift#recon
      ...
      recon_cache_path = /var/cache/swift
      
  • 编辑账户服务配置文件 /etc/swift/object-server.conf

    1. [DEFAULT] 中配置如下信息:

      [DEFAULT]
      ...
      bind_ip = 10.202.127.4
      bind_port = 6200
      user = swift
      swift_dir = /etc/swift
      devices = /srv/node
      mount_check = True
      
    2. 编辑 [pipeline:main]

      [pipeline:main]
      pipeline = healthcheck recon object-server
      
    3. [filter:recon] 中,设置recon cache 目录:

      [filter:recon]
      use = egg:swift#recon
      ...
      recon_cache_path = /var/cache/swift
      recon_lock_path = /var/lock
      
  • Mount 点目录所属权限设置

    # chown -R swift:swift /srv/node
    
  • 创建 recon目录,并设置目录所属权限

    # mkdir -p /var/cache/swift
    # chown -R root:swift /var/cache/swift
    # chmod -R 775 /var/cache/swift
    

3. 创建Rings

在启动对象存储服务前,需要创建并初始化account,container,object的rings。这里使用一个region,1个zone。总共设置2^10(1024)个最大分区(partitions),使用3副本策略(replicas),设置1小时来限制需要多次移动分区时为最小间隔

  • 创建account ring

    cd /etc/swift
    # swift-ring-builder account.builder create 10 3 1
    

    添加所有存储到ring

    # swift-ring-builder account.builder add \
      --region 1 --zone 1 --ip 10.202.127.4 --port 6202 --device sdb --weight 100
    # swift-ring-builder account.builder add \
      --region 1 --zone 1 --ip 10.202.127.4 --port 6202 --device sdc --weight 100
    # swift-ring-builder account.builder add \
      --region 1 --zone 1 --ip 10.202.127.4 --port 6202 --device sdd --weight 100
    

    Rebalance the ring

    # swift-ring-builder account.builder rebalance
    

    确认ring

    # swift-ring-builder account.builder
    
  • 创建container ring

    cd /etc/swift
    # swift-ring-builder container.builder create 10 3 1
    

    添加所有存储到ring

    # swift-ring-builder container.builder add \
      --region 1 --zone 1 --ip 10.202.127.4 --port 6201 --device sdb --weight 100
    # swift-ring-builder container.builder add \
      --region 1 --zone 1 --ip 10.202.127.4 --port 6201 --device sdc --weight 100
    # swift-ring-builder container.builder add \
      --region 1 --zone 1 --ip 10.202.127.4 --port 6201 --device sdd --weight 100
    

    Rebalance the ring

    # swift-ring-builder container.builder rebalance
    

    确认ring

    # swift-ring-builder container.builder
    
  • 创建object ring

    cd /etc/swift
    # swift-ring-builder object.builder create 10 3 1
    

    添加所有存储到ring

    # swift-ring-builder object.builder add \
      --region 1 --zone 1 --ip 10.202.127.4 --port 6200 --device sdb --weight 100
    # swift-ring-builder object.builder add \
      --region 1 --zone 1 --ip 10.202.127.4 --port 6200 --device sdc --weight 100
    # swift-ring-builder object.builder add \
      --region 1 --zone 1 --ip 10.202.127.4 --port 6200 --device sdd --weight 100
    

    Rebalance the ring

    # swift-ring-builder object.builder rebalance
    

    确认ring

    # swift-ring-builder object.builder
    

注意:如果有多个存储节点,需要将ring配置文件 account.ring.gz, container.ring.gzobject.ring.gz 分发到各个存储节点的 /etc/swift 目录下

3. 配置swift.conf 并启动各组件服务

  • 从swift git仓库中获取配置文件到 /etc/swift/ 目录

    # curl -o /etc/swift/swift.conf \
      https://git.openstack.org/cgit/openstack/swift/plain/etc/swift.conf-sample?h=stable/pike
    

    编辑配置文件/etc/swift/swift.conf

    1. [swift-hash] 中,设置如下内容:

      [swift-hash]
      ...
      swift_hash_path_suffix = HASH_PATH_SUFFIX #替换为自己的内容,如swift
      swift_hash_path_prefix = HASH_PATH_PREFIX #替换为自己的内容,如swift
      
    2. 创建并配置默认策略

      [storage-policy:0]
      ...
      name = Policy-0
      default = yes
      
    3. 如果是多存储节点,将配置文件 swift.conf 拷贝到各节点的 /etc/swift 目录下

    4. 修改所有节点 /etc/swift/ 目录所属权限如下:

      # chown -R root:swift /etc/swift
      
  • 控制节点启动代理服务和memcache服务

    # systemctl enable openstack-swift-proxy.service memcached.service
    # systemctl start openstack-swift-proxy.service memcached.service
    
  • 存储节点启动如下服务

    # systemctl enable openstack-swift-account.service openstack-swift-account-auditor.service \
      openstack-swift-account-reaper.service openstack-swift-account-replicator.service
    # systemctl start openstack-swift-account.service openstack-swift-account-auditor.service \
      openstack-swift-account-reaper.service openstack-swift-account-replicator.service
    # systemctl enable openstack-swift-container.service \
      openstack-swift-container-auditor.service openstack-swift-container-replicator.service \
      openstack-swift-container-updater.service
    # systemctl start openstack-swift-container.service \
      openstack-swift-container-auditor.service openstack-swift-container-replicator.service \
      openstack-swift-container-updater.service
    # systemctl enable openstack-swift-object.service openstack-swift-object-auditor.service \
      openstack-swift-object-replicator.service openstack-swift-object-updater.service
    # systemctl start openstack-swift-object.service openstack-swift-object-auditor.service \
      openstack-swift-object-replicator.service openstack-swift-object-updater.service
    

服务验证

新建并编辑 testrc 文件

cd $HOME
vim testrc

写入 /etc/swift/proxy-server.conf 配置文件中 配置的用户权限信息,如下:

export ST_AUTH=http://10.202.127.4:8080/auth/v1.0
export ST_USER=test:tester
export ST_KEY=testing

执行上诉配置脚本,使配置信息生效

. testrc

查看swift服务状态

swift stat

获取 X-Storage-UrlX-Auth-Token

curl -v -H 'X-Storage-User: test:tester' -H 'X-Storage-Pass: testing' http://127.0.0.1:8080/auth/v1.0

查看账户

curl -v -H 'X-Auth-Token: <token-from-x-auth-token-above>' <url-from-x-storage-url-above>

我们可以使用如下命令获取账户下容器列表

swift list

获取容器下的对象列表

swift list <container>
2012-09-07 14:46:58 lifeifei2010 阅读数 1201

原文地址:http://swift.openstack.org/howto_installmultinode.html

多结点 Swift 安装 (Ubuntu)

先决条件

  • Ubuntu Server 12.04 LTS 安装媒介
  • 网络配置
  • 编辑  /etc/network/interfaces:

  • auto lo
    iface lo inet loopback

    auto eth0
    iface eth0 inet static
          address 10.1.0.35  
          netmask 255.255.255.0
          broadcast 10.1.0.255
          gateway 10.1.0.1
          dns-nameservers 8.8.8.8

    auto eth1
    iface eth1 inet static
         address 192.168.3.1
         netmask 255.255.255.0
         network 192.168.3.0
         broadcast 192.168.3.255

     Restart the network now
     sudo /etc/init.d/networking restart

     

基本结构和一些术语

  • node - 提供一种或多种 Swift 服务的主机
  • Proxy node - 提供 Proxy 服务的node; 同时也提供 TempAuth 的服务
  • Storage node - 提供 Account, Container, 和 Object services服务
  • ring - Swift数据和物理设备之间的一系列映射

本文介绍一个由以下node组成的机群:

  • 一个 Proxy node

    • 运行 swift-proxy-server 进程, 会转送来自客户端的 request 到合适的

      Storage nodes。 proxy server 也可以提供 WSGI 中间件形式的 TempAuth 服务

  • 五个 Storage node

    • 运行 swift-account-server, swift-container-server, 和swift-object-server 进程, 管理account、container数据和实际存储的object。

本文描述的每个 Storage node 都安排在 ring 中不同的 zone 里。推荐至少配置5个 zone. 一个 zone 是一组隔离的 node (各自隔离的服务器,网络,电源配置,甚至是地理位置)。ring 维护着每个副本( replica )存储在不同的 zone 上。关于 ring 和 zone 的更多信息请查看:The Rings.

为了增强可靠性,你可能会增加其他的 Proxy server,这里你会得到帮助 增加一个 Proxy Server.

网络配置的一些注意事项

本文提到了两个网络。“外部网络”用来连接 Proxy server, “内部存储网络”在机群外则是不可见的。内部网络用来node之间的互联。所有的Swift 服务,包括每个Storage node中的 rsync 守护进程,都时时监听STORAGE_LOCAL_NET,即内网中分配到的IP地址。

通用的 OS 配置

  1. 所有 node 安装 Ubuntu Server 12.04 LTS

  2. 安装 Swift 的先决条件:

    apt-get install python-software-properties
    add-apt-repository ppa:swift-core/ppa
    apt-get update
    apt-get install swift openssh-server
  3. 创建 Swift 的工作目录:

    mkdir -p /etc/swift
    chown -R swift:swift /etc/swift/
  4. 在第一个 node , 创建 /etc/swift/swift.conf:

    cat >/etc/swift/swift.conf <<EOF
    [swift-hash]
    # random unique string that can never change (DO NOT LOSE)
    swift_hash_path_suffix = `od -t x8 -N 8 -A n </dev/random`
    EOF
  5. 后续的 node : 拷贝上一步的swift.conf文件到相应目录,这个文件在所有的 node 上都必须相同:

    scp firstnode.example.com:/etc/swift/swift.conf /etc/swift/
  6. 定义内部网络 IP 地址,之后的配置会遇到:

    export STORAGE_LOCAL_NET_IP=10.1.2.3
    export PROXY_LOCAL_NET_IP=10.1.2.4

配置 Proxy node

  1. 安装 swift-proxy 服务:

    apt-get install swift-proxy memcached
  2. 创建SSL自签名证书 cert

    cd /etc/swift
    openssl req -new -x509 -nodes -out cert.crt -keyout cert.key
  1. 配置 memcached ,监听默认的端口。推荐配置为内部的、非公用的IP网络地址。编辑 /etc/memcached.conf,并做配置,例如:

    perl -pi -e "s/-l 127.0.0.1/-l $PROXY_LOCAL_NET_IP/" /etc/memcached.conf
  2. 重启 memcached server:

    service memcached restart
  3. 创建 /etc/swift/proxy-server.conf:

    cat >/etc/swift/proxy-server.conf <<EOF
    [DEFAULT]
    cert_file = /etc/swift/cert.crt
    key_file = /etc/swift/cert.key
    bind_port = 8080
    workers = 8
    user = swift
    
    [pipeline:main]
    pipeline = healthcheck cache tempauth proxy-server
    
    [app:proxy-server]
    use = egg:swift#proxy
    allow_account_management = true
    
    [filter:tempauth]
    use = egg:swift#tempauth
    default_swift_cluster = local#https://$PROXY_LOCAL_NET_IP:8080/v1
    # Highly recommended to change this key to something else!
    super_admin_key = tempauth
    
    [filter:healthcheck]
    use = egg:swift#healthcheck
    
    [filter:cache]
    use = egg:swift#memcache
    memcache_servers = <PROXY_LOCAL_NET_IP>:11211
    EOF
  4. 创建 account, container 和 object ring。swift-ring-builder 命令创建相应的.builder文件会有几个参数。18代表partition的数目会到2^18幂,这个数字取决于你希望一个ring中会有多少个partition。3代表每个object的副本数。最后一个参数1,代表一个partition至少在1小时之后才能被移动。:

    cd /etc/swift
    swift-ring-builder account.builder create 18 3 1
    swift-ring-builder container.builder create 18 3 1
    swift-ring-builder object.builder create 18 3 1
  5. ring 需要记录每个 Store Node /srv/node 中记录的存储设备(下例为sdb1):

    export ZONE=                    # 为存储设备设定ZONE的编号
    export STORAGE_LOCAL_NET_IP=    #  IP 地址
    export WEIGHT=100               # 相对权重( weight) (更大或者更快的磁盘应有更大的权重)
    export DEVICE=sdb1
    swift-ring-builder account.builder add z$ZONE-$STORAGE_LOCAL_NET_IP:6002/$DEVICE $WEIGHT
    swift-ring-builder container.builder add z$ZONE-$STORAGE_LOCAL_NET_IP:6001/$DEVICE $WEIGHT
    swift-ring-builder object.builder add z$ZONE-$STORAGE_LOCAL_NET_IP:6000/$DEVICE $WEIGHT
  6. 确认 ring 的内容是否正确:

    swift-ring-builder account.builder
    swift-ring-builder container.builder
    swift-ring-builder object.builder
  7. 平衡(Rebalance) ring:

    swift-ring-builder account.builder rebalance
    swift-ring-builder container.builder rebalance
    swift-ring-builder object.builder rebalance
  8. 拷贝 account.ring.gz, container.ring.gz, object.ring.gz 到其它 Proxy node 和 Storage node 的 /etc/swift 目录下。

  9. 确保所有的配置文件都属于 swift 用户:

    chown -R swift:swift /etc/swift
  10. 开启 Proxy 服务:

    swift-init proxy start

配置 Storage node

  1. 安装配置 Storage node 相关:

    apt-get install swift-account swift-container swift-object xfsprogs
  2. 对 Storage node 的每一个存储设备, 安装 XFS 逻辑卷 (下例中为 /dev/sdb ):

    fdisk /dev/sdb  (挂载了单独的一个物理卷,或磁盘)
    mkfs.xfs -i size=1024 /dev/sdb1
    echo "/dev/sdb1 /srv/node/sdb1 xfs noatime,nodiratime,nobarrier,logbufs=8 0 0" >> /etc/fstab
    mkdir -p /srv/node/sdb1
    mount /srv/node/sdb1
    chown -R swift:swift /srv/node
  3. 创建 /etc/rsyncd.conf:

    cat >/etc/rsyncd.conf <<EOF
    uid = swift
    gid = swift
    log file = /var/log/rsyncd.log
    pid file = /var/run/rsyncd.pid
    address = $STORAGE_LOCAL_NET_IP
    
    [account]
    max connections = 2
    path = /srv/node/
    read only = false
    lock file = /var/lock/account.lock
    
    [container]
    max connections = 2
    path = /srv/node/
    read only = false
    lock file = /var/lock/container.lock
    
    [object]
    max connections = 2
    path = /srv/node/
    read only = false
    lock file = /var/lock/object.lock
    EOF
  4. 修改 /etc/default/rsync 中的 RSYNC_ENABLE= 属性

    perl -pi -e 's/RSYNC_ENABLE=false/RSYNC_ENABLE=true/' /etc/default/rsync
  5. 开启 rsync 守护进程:

    service rsync start
  6. 创建 /etc/swift/account-server.conf:

    cat >/etc/swift/account-server.conf <<EOF
    [DEFAULT]
    bind_ip = $STORAGE_LOCAL_NET_IP
    workers = 2
    
    [pipeline:main]
    pipeline = account-server
    
    [app:account-server]
    use = egg:swift#account
    
    [account-replicator]
    
    [account-auditor]
    
    [account-reaper]
    EOF
  7. 创建 /etc/swift/container-server.conf:

    cat >/etc/swift/container-server.conf <<EOF
    [DEFAULT]
    bind_ip = $STORAGE_LOCAL_NET_IP
    workers = 2
    
    [pipeline:main]
    pipeline = container-server
    
    [app:container-server]
    use = egg:swift#container
    
    [container-replicator]
    
    [container-updater]
    
    [container-auditor]
    EOF
  8. 创建 /etc/swift/object-server.conf:

    cat >/etc/swift/object-server.conf <<EOF
    [DEFAULT]
    bind_ip = $STORAGE_LOCAL_NET_IP
    workers = 2
    
    [pipeline:main]
    pipeline = object-server
    
    [app:object-server]
    use = egg:swift#object
    
    [object-replicator]
    
    [object-updater]
    
    [object-auditor]
    EOF
  9. 开启 storage 服务. 下面的命令会开启所有具有相关配置文件的服务(比如proxy-server.conf,auth-server.conf存在,会开启proxy 和auth服务), 并可能因为部分配置文件不存在而抛出警告:

    swift-init all start

如果你希望一次只启动一个服务,按照如下格式手动启动服务即可。有一点需要注意,您遇到问题的server 在用swift-init启动时,可能把标准输出、错误重定向到了 /dev/null。 如果您遇到什么问题,停止相应 server 的服务,并手动启动它可能是更好的选择。所有的 server 都可以通过 “swift-$SERVER-$SERVICE /etc/swift/$SERVER-config” 的格式启动。其中 $SERVER 可以是 object, continer, 或者 account,  $SERVICE 可以是 server,replicator, updater, 或者 auditor。:

swift-init object-server start
swift-init object-replicator start
swift-init object-updater start
swift-init object-auditor start
swift-init container-server start
swift-init container-replicator start
swift-init container-updater start
swift-init container-auditor start
swift-init account-server start
swift-init account-replicator start
swift-init account-auditor start

创建 Swift 管理员 account 并作测试

在 Proxy node 运行如下命令

  1. 获得 X-Storage-Url 和 X-Auth-Token:

    curl -k -v -H 'X-Storage-User: system:root' -H 'X-Storage-Pass: testpass' https://$PROXY_LOCAL_NET_IP:8080/auth/v1.0
  2. 检测是否可以使用 HEAD 命令查询 account 状态:

    curl -k -v -H 'X-Auth-Token: <token-from-x-auth-token-above>' <url-from-x-storage-url-above>
  3. 检测 swift 命令是否工作正常  (输出应为 zero containers, zero objects, 和 zero bytes):

    swift -A https://$PROXY_LOCAL_NET_IP:8080/auth/v1.0 -U system:root -K testpass stat
  4. 使用 swift 命令上载 ‘bigfile[1-2].tgz’两个文件到’myfiles’的 container

    swift -A https://$PROXY_LOCAL_NET_IP:8080/auth/v1.0 -U system:root -K testpass upload myfiles bigfile1.tgz
    swift -A https://$PROXY_LOCAL_NET_IP:8080/auth/v1.0 -U system:root -K testpass upload myfiles bigfile2.tgz
  5. 使用 swift 命令下载 container ‘myfiles’中的所有文件

    swift -A https://$PROXY_LOCAL_NET_IP:8080/auth/v1.0 -U system:root -K testpass download myfiles
  6. 使用 swift 保存ring的.builder 构建文件到container ‘builders’. 这些构建文件很重要,不要丢失!:

    swift -A https://$PROXY_LOCAL_NET_IP:8080/auth/v1.0 -U system:root -K testpass upload builders /etc/swift/*.builder
  7. 使用 swift 命令列表特定用户的 container

    swift -A https://$PROXY_LOCAL_NET_IP:8080/auth/v1.0 -U system:root -K testpass list
  8. 使用 swift 命令列表container ‘builders’中的 object

    swift -A https://$PROXY_LOCAL_NET_IP:8080/auth/v1.0 -U system:root -K testpass list builders
  9. 使用 swift 命令下载container ‘builders’中的object

    swift -A https://$PROXY_LOCAL_NET_IP:8080/auth/v1.0 -U system:root -K testpass download builders

增加一个 Proxy Server

从系统可靠性的角度,我们更希望有多个proxy server。这一小节介绍如何在已有一个proxy server 的基础上增加另一个 Proxy Server。

一旦有超过两个proxy server,我们可能期望增加负载均衡的机制,这往往意为着存储结构的变化(增加了一层负载均衡设备)。我们可以选择不同的负载均衡策略。 比如循环dns(round robin dns)算法,或者在多个proxy server之前使用负载均衡的中间件 (比如 pound) , 把指向 storage url 的请求转嫁给负载均衡器。

参考 配置 Proxy node 中的初始安装, 之后按如下步骤操作。

  1. 所有的Proxy server中,更新/etc/swift/proxy-server.conf的memcache server配置。如果有多个memcache server,用以下的格式进行分隔: IP:port ,例如在每个Proxy server 的配置文件中:10.1.2.3:11211,10.1.2.4:11211 ,说明是以10.1.2.3、10.1.2.4作为共享的缓存:

    [filter:cache]
    use = egg:swift#memcache
    memcache_servers = $PROXY_LOCAL_NET_IP:11211
  2. 改变/etc/swift/proxy-server.conf中storage url的指向,使所有请求指向负载均衡设备的url,而不是初始设置时的那台Proxy server。

    [filter:tempauth]
    use = egg:swift#tempauth
    user_system_root = testpass .admin http[s]://<LOAD_BALANCER_HOSTNAME>:<PORT>/v1/AUTH_system
  3. 之后,拷贝所有ring 的信息到所有node中,包括新加入的proxy node, 确保ring 的内容包括了所有的storage node。

  4. 同步(?)所有的结点之后, 确保管理员( admin)可以访问 /etc/swift 目录,并且确保所有ring的.gz文件所有者都是正确的。

故障检测的便签

如果遇到什么问题,查看 var/log/syslog 中日志信息是非常必要的。

在 Rackspace,我们会查看/var/log/kern.log找到磁盘故障的一些线索。

Administrator’s Guide中会有更多的调试技巧

2018-12-10 11:40:00 dengyuelin 阅读数 719

目的:通过k8s可以快速创建启动swift集群

思路:刚开始想全自动的,后来实现有点困难,主要对k8s不是很熟,所以先半自动了。。。先创建swift proxy的镜像,再在worker节点中随便找一台制作swift  存储的镜像。swift对象存储的多节点集群模式暂时采用temauth临时认证,具体的安装文档可以参考官网了解下:https://docs.openstack.org/project-install-guide/object-storage/newton/,但是官网是采用的keystone认证方式。在这里我们k8s的master就不介绍了,因为已经建好了,但需要用到添加k8s的node节点。

k8s集群机器:

        master节点:

               k8smaster:192.168.18.73

        worker节点:

                 192.168.18.173   主机名 zabbix  作为swift的proxy节点

                 192.168.18.100   主机名:compute1   作swift 的node节点

                 192.168.18.84      主机名  object2         swift的node节点

                  192.168.18.172    主机名 tian-7            swift的node节点

 

在任意一个有docker的机器上 制作proxy镜像:这里在proxy0

  先安装docker制作proxy镜像:
   curl -fsSL https://mirrors.ustc.edu.cn/docker-ce/linux/ubuntu/gpg | sudo apt-key add -
   echo 'deb https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial main' >> /etc/apt/sources.list
   apt update
  apt upgrade
  apt install -y docker-ce

  apt install docker-ce=18.06.1~ce~3-0~ubuntu -y 

在当前目录创建proxy目录,然后再proxy目录下创建files目录和Dockerfile文件:

在当前目录创建storage目录,然后再storage目录下创建files目录和Dockerfile文件:

进入proxy目录

 root@proxy0:~/dockerfile/proxy# ls files/
proxy-server.conf  swift.conf

root@proxy0:~/dockerfile/proxy# cat files/proxy-server.conf |grep -v ^#
[DEFAULT]
bind_port = 8080
swift_dir = /etc/swift
user = swift
log_facility = LOG_LOCAL1

[pipeline:main]
pipeline= healthcheck proxy-logging cache tempauth proxy-logging proxy-server


[app:proxy-server]
use = egg:swift#proxy
account_autocreate = True
allow_account_management = true 

[filter:tempauth]
use = egg:swift#tempauth
reseller_prefix = '' 


token_life = 86400000
user_admin_admin = admin .admin .reseller_admin
user_test_tester = testing .admin
user_test2_tester2 = testing2 .admin
user_test_tester3 = testing3
user_test5_tester5 = testing5 service


[filter:healthcheck]
use = egg:swift#healthcheck

[filter:cache]
use = egg:swift#memcache
memcache_servers = 192.168.18.178:11211

[filter:ratelimit]
use = egg:swift#ratelimit


[filter:domain_remap]
use = egg:swift#domain_remap


[filter:catch_errors]
use = egg:swift#catch_errors

[filter:cname_lookup]
use = egg:swift#cname_lookup

[filter:staticweb]
use = egg:swift#staticweb

[filter:tempurl]
use = egg:swift#tempurl

[filter:formpost]
use = egg:swift#formpost

[filter:name_check]
use = egg:swift#name_check

[filter:list-endpoints]
use = egg:swift#list_endpoints

[filter:proxy-logging]
use = egg:swift#proxy_logging

[filter:bulk]
use = egg:swift#bulk


[filter:slo]
use = egg:swift#slo

[filter:dlo]
use = egg:swift#dlo

[filter:container-quotas]
use = egg:swift#container_quotas

[filter:account-quotas]
use = egg:swift#account_quotas

[filter:gatekeeper]
use = egg:swift#gatekeeper

[filter:container_sync]
use = egg:swift#container_sync

[filter:xprofile]
use = egg:swift#xprofile

[filter:versioned_writes]
use = egg:swift#versioned_writes

root@proxy0:~/dockerfile/proxy# cat Dockerfile 
FROM ubuntu:16.04
CMD /bin/bash
MAINTAINER dyl <dylisbean@gmail.com>

# Install repository  for latest version of swift installation

RUN  apt-get update -y
RUN  apt-get upgrade -y

# Install swift packages  
RUN mkdir -p /etc/swift
RUN apt-get install swift swift-proxy python-swiftclient  memcached -y
ADD files/proxy-server.conf /etc/swift/proxy-server.conf
ADD files/swift.conf /etc/swift/swift.conf


# Permission for swift
RUN chown -R swift:swift /etc/swift
#RUN service supervisor restart
 

这里先不创建ring了,因为创建ring要使用到swift storage的地址,这里无确定

 

接着在Dockerfile所在目录使用Dockerfile创建proxy的镜像:

docker build -t swift-proxy .

接着制作swift-storage镜像

进入storage目录:

root@proxy0:~/dockerfile# cd storage/
root@proxy0:~/dockerfile/storage# ls 
Dockerfile  files
root@proxy0:~/dockerfile/storage# ls files/
account-server.conf  container-server.conf  object-server.conf  proxy-server.conf  rsync  rsyncd.conf  rsyslog.conf  swift.conf

root@proxy0:~/dockerfile/storage# cat files/account-server.conf |grep -v ^#
[DEFAULT]
bind_ip = 0.0.0.0
bind_port = 6002
workers = 2
user = swift
swift_dir = /etc/swift
devices = /srv/node
mount_check = True
log_facility = LOG_LOCAL4

[pipeline:main]
pipeline = healthcheck recon account-server

[filter:healthcheck]
use = egg:swift#healthcheck

[app:account-server]
use = egg:swift#account

[filter:recon]
use = egg:swift#recon
recon_cache_path = /var/cache/swift

[account-replicator]

[account-auditor]

[account-reaper]

root@proxy0:~/dockerfile/storage# cat files/object-server.conf |grep -v ^#
[DEFAULT]
bind_ip = 0.0.0.0
bind_port = 6000
workers = 2
user = swift
swift_dir = /etc/swift
devices = /srv/node
mount_check = True
log_facility = LOG_LOCAL2


[pipeline:main]
pipeline = healthcheck recon object-server


[filter:healthcheck]
use = egg:swift#healthcheck


[filter:recon]
use = egg:swift#recon
recon_cache_path = /var/cache/swift
recon_lock_path = /var/lock

[app:object-server]
use = egg:swift#object

[object-replicator]

[object-updater]

[object-auditor]

root@proxy0:~/dockerfile/storage# cat files/rsyncd.conf |grep -v ^#
uid = swift
gid = swift
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
address = 0.0.0.0 

[account]
max connections = 2
path = /srv/node
read only = false
lock file = /var/lock/account.lock

[container]
max connections = 4
path = /srv/node
read only = false
lock file = /var/lock/container.lock

[object]
max connections = 8
path = /srv/node
read only = false
lock file = /var/lock/object.lock

root@proxy0:~/dockerfile/storage# cat files/rsyslog.conf |grep -v ^#

module(load="imuxsock") # provides support for local system logging
module(load="imklog")   # provides kernel logging support

$KLogPermitNonKernelFacility on


$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

$RepeatedMsgReduction on

$FileOwner syslog
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022
$PrivDropToUser syslog
$PrivDropToGroup syslog

$WorkDirectory /var/spool/rsyslog

$IncludeConfig /etc/rsyslog.d/*.conf

root@proxy0:~/dockerfile/storage# cat files/rsy |grep -v ^#
rsync         rsyncd.conf   rsyslog.conf  
root@proxy0:~/dockerfile/storage# cat files/rsync |grep -v ^#

RSYNC_ENABLE=true


RSYNC_OPTS=''

RSYNC_NICE=''

root@proxy0:~/dockerfile/storage# cat Dockerfile 
FROM ubuntu:16.04
CMD /bin/bash
MAINTAINER dyl <dylisbean@gmail.com>

# Install repository  for latest version of swift installation

RUN  apt-get update -y
RUN  apt-get upgrade -y

# Install supervisor for handling sub-process of swift

#RUN sudo apt-get install supervisor -y
#RUN mkdir -p /var/log/supervisor

#Installing swift supporting file system. 

RUN apt-get install xfsprogs rsync -y
#RUN mkdir -p /etc/swift

# Installing storage packages
RUN  apt-get install swift swift-account swift-container swift-object -y

# Add swift storage file
#ADD files/supervisord.conf /etc/supervisor/conf.d/supervisord.conf
ADD files/rsyncd.conf /etc/rsyncd.conf 
ADD files/account-server.conf /etc/swift/account-server.conf
ADD files/object-server.conf /etc/swift/object-server.conf
ADD files/container-server.conf /etc/swift/container-server.conf
ADD files/swift.conf /etc/swift/swift.conf

ADD files/rsync /etc/default/rsync

#RUN mkdir -p /srv/node/sdb
#RUN mkdir -p /var/cache/swift
#RUN chown -R swift:swift /srv/node
#RUN chown -R swift:swift /var/cache/swift
 

在storage目录下Dockerfile所在目录利用Dockerfile创建swift-storage镜像:

docker build -t swift-storage .

 

打包镜像:

docker save -o  swift-storage.tar swift-storage 
docker save -o swift-proxy-server.tar swift-proxy-server

从tar包导入镜像:

   docker load -i swift-proxy-server.tar
   docker load -i swift-storage.tar

创建k8s节点,将节点加入k8s master:

先在master上面生成有效的token:

kubeadm token create --print-join-command

生成这段命令,将其放在worker.sh创建的时候时候:

kubeadm join 192.168.18.73:6443 --token whsk3i.qrp01768uirp7ibs --discovery-token-ca-cert-hash sha256:2fafddd03602bcc0d61c1c67fba09d5dfa5672e7cdd848a00211b88c0724d711

再在k8s的worker节点也就是下面这些机器上分别执行创建worker脚本

        worker节点:

                 192.168.18.173   主机名 zabbix  作为swift的proxy节点

                 192.168.18.100   主机名:compute1   作swift 的node节点

                 192.168.18.84      主机名  object2         swift的node节点

                  192.168.18.172    主机名 tian-7            swift的node节点

root@compute1:/usr/local/src# cat worker.sh 
apt remove -y docker-ce kubelet kubeadm kubectl 
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4
curl -fsSL https://mirrors.ustc.edu.cn/docker-ce/linux/ubuntu/gpg | sudo apt-key add -
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
add-apt-repository \
    "deb [arch=amd64] https://mirrors.ustc.edu.cn/docker-ce/linux/ubuntu \
    $(lsb_release -cs) \
    stable"
echo 'deb https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial main' >> /etc/apt/sources.list
apt-get update
apt install docker-ce=18.06.1~ce~3-0~ubuntu
systemctl enable docker && systemctl start docker
apt-get install kubeadm=1.12.2-00 kubectl=1.12.2-00 kubelet=1.12.2-00 --allow-unauthenticated
systemctl daemon-reload
systemctl restart kubelet
swapoff -a
kubeadm join 192.168.18.73:6443 --token whsk3i.qrp01768uirp7ibs --discovery-token-ca-cert-hash sha256:2fafddd03602bcc0d61c1c67fba09d5dfa5672e7cdd848a00211b88c0724d711
cd /usr/local/src/kubeadm1.12.2/ &&  docker load -i flannel.tar &&  docker load -i kube-proxy.tar &&  docker load -i pause.tar 

systemctl daemon-reload
systemctl restart kubelet
 

root@compute1:/usr/local/src# sh worker.sh 

来到master上面查看节点是否添加成功,可以看到都添加成功了

 


 

 

 


 

 

 

 

 

 

                  

【Swift】部署学习

阅读数 307