Multipath made of iscsi on VMware

Linux Comments

[TOC]

How to create Multipath Link by iscsi on VMware workstation

提示
为了能够理解此篇文章，请了解iscsi和multipath理论知识和基础操作

为了在VMware上进行Multipath的相关理论实践，
没有机器，就只打算仅在一台虚拟机(同时兼做服务端和客户端)上采用iscsi建立共享磁盘，通过Multiple Sessions per Initiator建立两条链路。
需要2条链路，于是需要2个IP，就随便临时配一个浮动IP。

注意：target是0.0.0.0:3260 ,discover的时候需要discover 2 个IP：
iscsiadm -m discoverydb -t st -p 192.168.1.10 –discover
iscsiadm -m discoverydb -t st -p 192.168.1.11 –discover
然后执行iscsiadm -m node -l 来一次性登入。
或者分两次登录：
iscsiadm –mode node –targetname iqn.2020-02.com.baidu:shareddisk –portal 192.168.1.10:3260 –login
iscsiadm –mode node –targetname iqn.2020-02.com.baidu:shareddisk –portal 192.168.1.11:3260 –login

1个iSCSI Initiator透过多个Session连入同1个iSCSI Target设备，以便利用多网卡或iSCSI HBA启用负载平衡与故障失效切换机制，也可称作Multiple Sessions per Initiator。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
[root@centos77 tmp]# uname -a
Linux centos77 3.10.0-1062.el7.x86_64 #1 SMP Wed Aug 7 18:08:02 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
[root@centos77 ~]# ifconfig ens33:0 192.168.1.11 netmask 255.255.255.0
[root@centos77 ~]# ifconfig
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.1.10  netmask 255.255.255.0  broadcast 192.168.1.255
        ether 00:0c:29:19:60:fc  txqueuelen 1000  (Ethernet)
        RX packets 30001  bytes 17144152 (16.3 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 8388  bytes 1237295 (1.1 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens33:0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.1.11  netmask 255.255.255.0  broadcast 192.168.1.255
        ether 00:0c:29:19:60:fc  txqueuelen 1000  (Ethernet)

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 9741  bytes 11586392 (11.0 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 9741  bytes 11586392 (11.0 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
1
2
3
4
5
6
[root@centos77 tmp]# yum --disablerepo=epel install device-mapper-multipath
[root@centos77 ~]# yum --disablerepo=epel install targetcli iscsi-initiator-utils
[root@centos77 ~]# dd if=/dev/zero of=/tmp/image.iso bs=1M count=2048
[root@centos77 tmp]# losetup -f
[root@centos77 tmp]# losetup /dev/loop0 /tmp/image.iso
[root@centos77 tmp]# lsblk
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
[root@centos77 tmp]# targetcli
/> backstores/block create dev=/dev/loop0 name=shareddisk
/> iscsi/ create iqn.2020-02.com.baidu:shareddisk
/> iscsi/iqn.2020-02.com.baidu:shareddisk/tpg1/acls create iqn.2020-02.com.baidu:client
/> /iscsi/iqn.2020-02.com.baidu:shareddisk/tpg1/luns create /backstores/block/shareddisk
/> saveconfig
/> exit
Global pref auto_save_on_exit=true
Configuration saved to /etc/target/saveconfig.json

[root@centos77 ~]# targetcli
targetcli shell version 2.1.fb49
Copyright 2011-2013 by Datera, Inc and others.
For help on commands, type 'help'.

/> ls
o- / ................................................................................................... [...]
  o- backstores ........................................................................................ [...]
  | o- block ............................................................................ [Storage Objects: 1]
  | | o- shared_disk .............................................. [/dev/loop0 (2.0GiB) write-thru activated]
  | |   o- alua ............................................................................. [ALUA Groups: 1]
  | |     o- default_tg_pt_gp ................................................. [ALUA state: Active/optimized]
  | o- fileio ........................................................................... [Storage Objects: 0]
  | o- pscsi ............................................................................ [Storage Objects: 0]
  | o- ramdisk .......................................................................... [Storage Objects: 0]
  o- iscsi ...................................................................................... [Targets: 1]
  | o- iqn.2020-02.com.baidu:shareddisk ............................................................ [TPGs: 1]
  |   o- tpg1 ......................................................................... [no-gen-acls, no-auth]
  |     o- acls .................................................................................... [ACLs: 1]
  |     | o- iqn.2020-02.com.baidu:client ................................................... [Mapped LUNs: 1]
  |     |   o- mapped_lun0 ..................................................... [lun0 block/shared_disk (rw)]
  |     o- luns .................................................................................... [LUNs: 1]
  |     | o- lun0 ........................................ [block/shared_disk (/dev/loop0) (default_tg_pt_gp)]
  |     o- portals .............................................................................. [Portals: 1]
  |       o- 0.0.0.0:3260 ............................................................................... [OK]
  o- loopback ................................................................................... [Targets: 0]
/> 
[root@centos77 tmp]# systemctl restart target
[root@centos77 tmp]# systemctl status target
[root@centos77 tmp]# systemctl status iscsid
[root@centos77 ~]# iscsiadm -m discoverydb -t st -p 192.168.1.10 --discover
iscsiadm: Could not bind connection 0 to ens33:0
iscsiadm: cannot make connection to 192.168.1.10: No such device
iscsiadm: Could not bind connection 0 to ens33:0
iscsiadm: cannot make connection to 192.168.1.10: No such device
iscsiadm: Could not bind connection 0 to ens33:0
iscsiadm: cannot make connection to 192.168.1.10: No such device
iscsiadm: Could not bind connection 0 to ens33:0
iscsiadm: cannot make connection to 192.168.1.10: No such device
iscsiadm: Could not bind connection 0 to ens33:0
iscsiadm: cannot make connection to 192.168.1.10: No such device
iscsiadm: Could not bind connection 0 to ens33:0
iscsiadm: cannot make connection to 192.168.1.10: No such device
iscsiadm: connection login retries (reopen_max) 5 exceeded
192.168.1.10:3260,1 iqn.2020-02.com.baidu:shareddisk
[root@centos77 ~]# iscsiadm -m discoverydb -t st -p 192.168.1.11 --discover
iscsiadm: Could not bind connection 0 to ens33:0
iscsiadm: cannot make connection to 192.168.1.11: No such device
iscsiadm: Could not bind connection 0 to ens33:0
iscsiadm: cannot make connection to 192.168.1.11: No such device
iscsiadm: Could not bind connection 0 to ens33:0
iscsiadm: cannot make connection to 192.168.1.11: No such device
iscsiadm: Could not bind connection 0 to ens33:0
iscsiadm: cannot make connection to 192.168.1.11: No such device
iscsiadm: Could not bind connection 0 to ens33:0
iscsiadm: cannot make connection to 192.168.1.11: No such device
iscsiadm: Could not bind connection 0 to ens33:0
iscsiadm: cannot make connection to 192.168.1.11: No such device
iscsiadm: connection login retries (reopen_max) 5 exceeded
192.168.1.11:3260,1 iqn.2020-02.com.baidu:shareddisk
[root@centos77 ~]# iscsiadm -m node
192.168.1.10:3260,1 iqn.2020-02.com.baidu:shareddisk
192.168.1.11:3260,1 iqn.2020-02.com.baidu:shareddisk
[root@centos77 tmp]# iscsiadm -m iface -I iface0 --op=new //删除iface为：iscsiadm -m iface -I iface0 --op=delete
New interface iface0 added
[root@centos77 tmp]# iscsiadm -m iface -I iface1 --op=new //删除iface为：iscsiadm -m iface -I iface1 --op=delete
New interface iface1 added
[root@centos77 tmp]# ls -l /var/lib/iscsi/ifaces/
total 8
-rw-------. 1 root root 419 Feb 15 10:01 iface0
-rw-------. 1 root root 419 Feb 15 10:01 iface1
[root@centos77 tmp]# iscsiadm -m iface -I iface0 --op=update -n iface.net_ifacename -v ens33
iface0 updated.
[root@centos77 tmp]# iscsiadm -m iface -I iface1 --op=update -n iface.net_ifacename -v ens33:0
iface0 updated.
[root@centos77 ~]# iscsiadm -m iface
default tcp,<empty>,<empty>,<empty>,<empty>
iser iser,<empty>,<empty>,<empty>,<empty>
iface0 tcp,<empty>,<empty>,ens33,<empty>
iface1 tcp,<empty>,<empty>,ens33:0,<empty>
[root@centos77 tmp]#  iscsiadm -m discoverydb -t st -p 192.168.1.10 -I iface0 -I iface1
# BEGIN RECORD 6.2.0.874-10
discovery.startup = manual
discovery.type = sendtargets
discovery.sendtargets.address = 192.168.1.10
discovery.sendtargets.port = 3260
discovery.sendtargets.auth.authmethod = None
discovery.sendtargets.auth.username = <empty>
discovery.sendtargets.auth.password = <empty>
discovery.sendtargets.auth.username_in = <empty>
discovery.sendtargets.auth.password_in = <empty>
discovery.sendtargets.timeo.login_timeout = 15
discovery.sendtargets.use_discoveryd = No
discovery.sendtargets.discoveryd_poll_inval = 30
discovery.sendtargets.reopen_max = 5
discovery.sendtargets.timeo.auth_timeout = 45
discovery.sendtargets.timeo.active_timeout = 30
discovery.sendtargets.iscsi.MaxRecvDataSegmentLength = 32768
# END RECORD
[root@centos77 ~]# iscsiadm --mode node --targetname iqn.2020-02.com.baidu:shareddisk --portal 192.168.1.10:3260 --login
[root@centos77 ~]# iscsiadm --mode node --targetname iqn.2020-02.com.baidu:shareddisk --portal 192.168.1.11:3260 --login
或者[root@centos77 ~]# iscsiadm -m node -l //登出为：iscsiadm -m node -u
[root@centos77 ~]# iscsiadm -m node
192.168.1.10:3260,1 iqn.2020-02.com.baidu:shareddisk
192.168.1.11:3260,1 iqn.2020-02.com.baidu:shareddisk
[root@centos77 ~]# iscsiadm -m session //查看登录会话
tcp: [5] 192.168.1.10:3260,1 iqn.2020-02.com.baidu:shareddisk (non-flash)
tcp: [6] 192.168.1.11:3260,1 iqn.2020-02.com.baidu:shareddisk (non-flash)
//也可以查看登录会话
[root@centos77 ~]# netstat -anp | grep 3260
tcp        0      0 0.0.0.0:3260            0.0.0.0:*               LISTEN      -                   
tcp        0      0 192.168.1.10:57166      192.168.1.11:3260       ESTABLISHED 1727/iscsid         
tcp        0      0 192.168.1.11:3260       192.168.1.10:57166      ESTABLISHED -                   
tcp        0      0 192.168.1.10:41850      192.168.1.10:3260       ESTABLISHED 1727/iscsid         
tcp        0      0 192.168.1.10:3260       192.168.1.10:41850      ESTABLISHED - 

[root@centos77 tmp]# lsblk 
NAME            MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda               8:0    0    7G  0 disk 
├─sda1            8:1    0    1G  0 part /boot
└─sda2            8:2    0    6G  0 part 
  ├─centos-root 253:0    0  5.3G  0 lvm  /
  └─centos-swap 253:1    0  716M  0 lvm  [SWAP]
sdb               8:16   0    2G  0 disk 
sdc               8:32   0    2G  0 disk 
sr0              11:0    1 1024M  0 rom  
loop0             7:0    0    2G  0 loop
//其中loop0是dd出来的作为共享的挂载在/dev/loop0上的/tmp/images.iso
//sdb,sdc是通过initiator登录target识别出来的2块硬盘(其实就是一个loop0共享出来的target)
//所以说，target，initiator，multipath同在一台机器上
[root@centos77 tmp]# netstat -anp | grep 3260
tcp        0      0 0.0.0.0:3260            0.0.0.0:*               LISTEN      -                   
tcp        0      0 192.168.1.10:3260       192.168.1.10:46320      ESTABLISHED -                   
tcp        0      0 192.168.1.10:45494      192.168.1.11:3260       ESTABLISHED 2290/iscsid         
tcp        0      0 192.168.1.10:46320      192.168.1.10:3260       ESTABLISHED 2290/iscsid         
tcp        0      0 192.168.1.11:3260       192.168.1.10:45494      ESTABLISHED - 
//如上输出也能证明，所以说，target，initiator，multipath同在一台机器上

[root@centos77 ~]# cat /proc/scsi/scsi 
Attached devices:
Host: scsi0 Channel: 00 Id: 00 Lun: 00
  Vendor: VMware,  Model: VMware Virtual S Rev: 1.0 
  Type:   Direct-Access                    ANSI  SCSI revision: 02
Host: scsi2 Channel: 00 Id: 00 Lun: 00
  Vendor: NECVMWar Model: VMware IDE CDR10 Rev: 1.00
  Type:   CD-ROM                           ANSI  SCSI revision: 05
Host: scsi15 Channel: 00 Id: 00 Lun: 00
  Vendor: LIO-ORG  Model: shared_disk      Rev: 4.0 
  Type:   Direct-Access                    ANSI  SCSI revision: 05
Host: scsi16 Channel: 00 Id: 00 Lun: 00
  Vendor: LIO-ORG  Model: shared_disk      Rev: 4.0 
  Type:   Direct-Access                    ANSI  SCSI revision: 05
[root@centos77 ~]# grep -v "zZzZ" -H /sys/class/scsi_host/host{0,1,2,15,16}/state
/sys/class/scsi_host/host0/state:running
/sys/class/scsi_host/host1/state:running
/sys/class/scsi_host/host2/state:running
/sys/class/scsi_host/host15/state:running
/sys/class/scsi_host/host16/state:running
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
[root@centos77 tmp]# multipath -ll
Feb 15 10:31:52 | DM multipath kernel driver not loaded
Feb 15 10:31:52 | /etc/multipath.conf does not exist, blacklisting all devices.
Feb 15 10:31:52 | A default multipath.conf file is located at
Feb 15 10:31:52 | /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf
Feb 15 10:31:52 | You can run /sbin/mpathconf --enable to create
Feb 15 10:31:52 | /etc/multipath.conf. See man mpathconf(8) for more details
Feb 15 10:31:52 | DM multipath kernel driver not loaded
[root@centos77 tmp]# modprobe dm_multipath
[root@centos77 tmp]# modprobe dm-multipath
[root@centos77 tmp]# modprobe dm-round-robin
[root@centos77 tmp]# service multipathd start
[root@centos77 tmp]# lsmod | grep dm_
dm_multipath           27792  0 
dm_mirror              22289  0 
dm_region_hash         20813  1 dm_mirror
dm_log                 18411  2 dm_region_hash,dm_mirror
dm_mod                124501  9 dm_multipath,dm_log,dm_mirror



--- 补充
在 Linux 7下，可以使用如下命令：
# /usr/lib/udev/scsi_id -g -u /dev/sdb

在 Linux 6下，可以使用如下命令：
# /sbin/scsi_id -g -u /dev/sdb

在 Linux 5下，可以使用如下命令：
# /sbin/scsi_id -g -u -s /block/sdb/sdb


[root@centos77 block]# /usr/lib/udev/scsi_id -g -u /dev/sdb
3600140565524db35468485692d22ebc2
[root@centos77 block]# /usr/lib/udev/scsi_id -g -u /dev/sdc
3600140565524db35468485692d22ebc2
[root@centos77 block]# ll /dev/disk/by-id/
total 0
lrwxrwxrwx. 1 root root  9 Feb 15 09:27 ata-VMware_Virtual_IDE_CDROM_Drive_10000000000000000001 -> ../../sr0
lrwxrwxrwx. 1 root root 10 Feb 15 09:27 dm-name-centos-root -> ../../dm-0
lrwxrwxrwx. 1 root root 10 Feb 15 09:27 dm-name-centos-swap -> ../../dm-1
lrwxrwxrwx. 1 root root 10 Feb 15 09:27 dm-uuid-LVM-knguIEmnftLwQDqkmD2j8t1X3PhKdvBs6uzAGweaam1OoGS7uo0sKH0o5YeY78P9 -> ../../dm-0
lrwxrwxrwx. 1 root root 10 Feb 15 09:27 dm-uuid-LVM-knguIEmnftLwQDqkmD2j8t1X3PhKdvBse8RGeQgafMPcyDCCQP3RN4c0tMfvwM9c -> ../../dm-1
lrwxrwxrwx. 1 root root 10 Feb 15 09:27 lvm-pv-uuid-5di517-Kstv-hVjb-Ags4-gJzc-a7e7-WPnfgY -> ../../sda2
lrwxrwxrwx. 1 root root  9 Feb 15 10:27 scsi-3600140565524db35468485692d22ebc2 -> ../../sdc
lrwxrwxrwx. 1 root root  9 Feb 15 10:27 wwn-0x600140565524db35468485692d22ebc2 -> ../../sdc
[root@centos77 block]# 

[root@centos77 tmp]# mpathconf --enable
[root@centos77 tmp]# vim /etc/multipath.conf
[root@centos77 ~]# grep ^[^#] /etc/multipath.conf
defaults {
    user_friendly_names yes
    find_multipaths yes
}
blacklist {
    devnode "sda"
    devnode "loop0"
    devnode "sr0"
}
[root@centos77 ~]# systemctl restart multipathd
[root@centos77 ~]# multipath -ll
mpatha (3600140565524db35468485692d22ebc2) dm-2 LIO-ORG ,shared_disk     
size=2.0G features='0' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=1 status=active
| `- 15:0:0:0 sdb 8:16 active ready running
`-+- policy='service-time 0' prio=1 status=enabled
  `- 16:0:0:0 sdc 8:32 active ready running
[root@centos77 ~]# lsblk 
NAME            MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
sda               8:0    0    7G  0 disk  
├─sda1            8:1    0    1G  0 part  /boot
└─sda2            8:2    0    6G  0 part  
  ├─centos-root 253:0    0  5.3G  0 lvm   /
  └─centos-swap 253:1    0  716M  0 lvm   [SWAP]
sdb               8:16   0    2G  0 disk  
└─mpatha        253:2    0    2G  0 mpath 
sdc               8:32   0    2G  0 disk  
└─mpatha        253:2    0    2G  0 mpath 
sr0              11:0    1 1024M  0 rom   
loop0             7:0    0    2G  0 loop


[root@centos77 tmp]# vim /etc/multipath.conf
[root@centos77 ~]# grep ^[^#] /etc/multipath.conf 
defaults {
    user_friendly_names yes
    find_multipaths yes
}
blacklist {
    devnode "sda"
    devnode "loop0"
    devnode "sr0"
}
multipaths{
    multipath{
        wwid 3600140565524db35468485692d22ebc2
        alias shareddisk
    }
}
[root@centos77 ~]# systemctl restart multipathd
[root@centos77 ~]# multipath -ll
shareddisk (3600140565524db35468485692d22ebc2) dm-2 LIO-ORG ,shared_disk     
size=2.0G features='0' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=1 status=active
| `- 15:0:0:0 sdb 8:16 active ready running
`-+- policy='service-time 0' prio=1 status=enabled
  `- 16:0:0:0 sdc 8:32 active ready running
[root@centos77 ~]# lsblk 
NAME            MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
sda               8:0    0    7G  0 disk  
├─sda1            8:1    0    1G  0 part  /boot
└─sda2            8:2    0    6G  0 part  
  ├─centos-root 253:0    0  5.3G  0 lvm   /
  └─centos-swap 253:1    0  716M  0 lvm   [SWAP]
sdb               8:16   0    2G  0 disk  
└─shareddisk    253:2    0    2G  0 mpath 
sdc               8:32   0    2G  0 disk  
└─shareddisk    253:2    0    2G  0 mpath 
sr0              11:0    1 1024M  0 rom   
loop0             7:0    0    2G  0 loop 


[root@centos77 tmp]# vim /etc/multipath.conf
[root@centos77 ~]# grep ^[^#] /etc/multipath.conf 
defaults {
    user_friendly_names yes
    find_multipaths yes
    polling_interval     10
    path_selector        "round-robin 0"
    path_grouping_policy    multibus
    uid_attribute        ID_SERIAL
    prio            alua
    path_checker        readsector0
    rr_min_io        100
    max_fds            8192
    rr_weight        priorities
    failback        immediate
    no_path_retry        fail
}
blacklist {
    devnode "sda"
    devnode "loop0"
    devnode "sr0"
}

multipaths{
    multipath{
        wwid 3600140565524db35468485692d22ebc2
        alias shareddisk
    }
}
[root@centos77 ~]# systemctl restart multipathd
[root@centos77 ~]# multipath -ll
shareddisk (3600140565524db35468485692d22ebc2) dm-2 LIO-ORG ,shared_disk     
size=2.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=50 status=active
  |- 15:0:0:0 sdb 8:16 active ready running
  `- 16:0:0:0 sdc 8:32 active ready running


[root@centos77 ~]# lsblk
NAME            MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
sda               8:0    0    7G  0 disk  
├─sda1            8:1    0    1G  0 part  /boot
└─sda2            8:2    0    6G  0 part  
  ├─centos-root 253:0    0  5.3G  0 lvm   /
  └─centos-swap 253:1    0  716M  0 lvm   [SWAP]
sdb               8:16   0    2G  0 disk  
└─shareddisk    253:2    0    2G  0 mpath 
sdc               8:32   0    2G  0 disk  
└─shareddisk    253:2    0    2G  0 mpath 
sr0              11:0    1 1024M  0 rom   
loop0             7:0    0    2G  0 loop


[root@centos77 ~]# sg_map -x
/dev/sg0  0 0 0 0  0  /dev/sda
/dev/sg1  2 0 0 0  5  /dev/sr0
/dev/sg2  15 0 0 0  0  /dev/sdb
/dev/sg3  16 0 0 0  0  /dev/sdc
[root@centos77 ~]# ls -lrht /dev/mapper/*
crw-------. 1 root root 10, 236 Feb 15 09:27 /dev/mapper/control
lrwxrwxrwx. 1 root root       7 Feb 15 09:27 /dev/mapper/centos-swap -> ../dm-1
lrwxrwxrwx. 1 root root       7 Feb 15 09:27 /dev/mapper/centos-root -> ../dm-0
lrwxrwxrwx. 1 root root       7 Feb 15 11:15 /dev/mapper/shareddisk -> ../dm-2

[root@centos77 ~]# ll /sys/class/scsi_host/
total 0
lrwxrwxrwx. 1 root root 0 Feb 15 09:27 host0 -> ../../devices/pci0000:00/0000:00:10.0/host0/scsi_host/host0
lrwxrwxrwx. 1 root root 0 Feb 15 09:27 host1 -> ../../devices/pci0000:00/0000:00:07.1/ata1/host1/scsi_host/host1
lrwxrwxrwx. 1 root root 0 Feb 15 10:24 host15 -> ../../devices/platform/host15/scsi_host/host15
lrwxrwxrwx. 1 root root 0 Feb 15 10:27 host16 -> ../../devices/platform/host16/scsi_host/host16
lrwxrwxrwx. 1 root root 0 Feb 15 09:27 host2 -> ../../devices/pci0000:00/0000:00:07.1/ata2/host2/scsi_host/host2
[root@centos77 ~]# ll /sys/class/block/
total 0
lrwxrwxrwx. 1 root root 0 Feb 15 09:27 dm-0 -> ../../devices/virtual/block/dm-0
lrwxrwxrwx. 1 root root 0 Feb 15 09:27 dm-1 -> ../../devices/virtual/block/dm-1
lrwxrwxrwx. 1 root root 0 Feb 15 10:49 dm-2 -> ../../devices/virtual/block/dm-2
lrwxrwxrwx. 1 root root 0 Feb 15 09:51 loop0 -> ../../devices/virtual/block/loop0
lrwxrwxrwx. 1 root root 0 Feb 15 09:27 sda -> ../../devices/pci0000:00/0000:00:10.0/host0/target0:0:0/0:0:0:0/block/sda
lrwxrwxrwx. 1 root root 0 Feb 15 09:27 sda1 -> ../../devices/pci0000:00/0000:00:10.0/host0/target0:0:0/0:0:0:0/block/sda/sda1
lrwxrwxrwx. 1 root root 0 Feb 15 09:27 sda2 -> ../../devices/pci0000:00/0000:00:10.0/host0/target0:0:0/0:0:0:0/block/sda/sda2
lrwxrwxrwx. 1 root root 0 Feb 15 10:35 sdb -> ../../devices/platform/host15/session13/target15:0:0/15:0:0:0/block/sdb
lrwxrwxrwx. 1 root root 0 Feb 15 10:35 sdc -> ../../devices/platform/host16/session14/target16:0:0/16:0:0:0/block/sdc
lrwxrwxrwx. 1 root root 0 Feb 15 09:27 sr0 -> ../../devices/pci0000:00/0000:00:07.1/ata2/host2/target2:0:0/2:0:0:0/block/sr0
[root@centos77 ~]# ll /dev/disk/by-id
total 0
lrwxrwxrwx. 1 root root  9 Feb 15 09:27 ata-VMware_Virtual_IDE_CDROM_Drive_10000000000000000001 -> ../../sr0
lrwxrwxrwx. 1 root root 10 Feb 15 09:27 dm-name-centos-root -> ../../dm-0
lrwxrwxrwx. 1 root root 10 Feb 15 09:27 dm-name-centos-swap -> ../../dm-1
lrwxrwxrwx. 1 root root 10 Feb 15 11:07 dm-name-shareddisk -> ../../dm-2
lrwxrwxrwx. 1 root root 10 Feb 15 09:27 dm-uuid-LVM-knguIEmnftLwQDqkmD2j8t1X3PhKdvBs6uzAGweaam1OoGS7uo0sKH0o5YeY78P9 -> ../../dm-0
lrwxrwxrwx. 1 root root 10 Feb 15 09:27 dm-uuid-LVM-knguIEmnftLwQDqkmD2j8t1X3PhKdvBse8RGeQgafMPcyDCCQP3RN4c0tMfvwM9c -> ../../dm-1
lrwxrwxrwx. 1 root root 10 Feb 15 11:07 dm-uuid-mpath-3600140565524db35468485692d22ebc2 -> ../../dm-2
lrwxrwxrwx. 1 root root 10 Feb 15 09:27 lvm-pv-uuid-5di517-Kstv-hVjb-Ags4-gJzc-a7e7-WPnfgY -> ../../sda2
lrwxrwxrwx. 1 root root  9 Feb 15 10:49 scsi-3600140565524db35468485692d22ebc2 -> ../../sdb
lrwxrwxrwx. 1 root root  9 Feb 15 10:49 wwn-0x600140565524db35468485692d22ebc2 -> ../../sdb
[root@centos77 ~]# ll /dev/disk/by-path/
total 0
lrwxrwxrwx. 1 root root  9 Feb 15 10:49 fc---lun-0 -> ../../sdb
lrwxrwxrwx. 1 root root 10 Feb 15 09:27 fc---lun-0-part1 -> ../../sda1
lrwxrwxrwx. 1 root root 10 Feb 15 09:27 fc---lun-0-part2 -> ../../sda2
lrwxrwxrwx. 1 root root  9 Feb 15 10:49 ip-192.168.1.10:3260-iscsi-iqn.2020-02.com.baidu:shareddisk-lun-0 -> ../../sdb
lrwxrwxrwx. 1 root root  9 Feb 15 10:49 ip-192.168.1.11:3260-iscsi-iqn.2020-02.com.baidu:shareddisk-lun-0 -> ../../sdc
lrwxrwxrwx. 1 root root  9 Feb 15 09:27 pci-0000:00:07.1-ata-2.0 -> ../../sr0
lrwxrwxrwx. 1 root root  9 Feb 15 09:27 pci-0000:00:10.0-scsi-0:0:0:0 -> ../../sda
lrwxrwxrwx. 1 root root 10 Feb 15 09:27 pci-0000:00:10.0-scsi-0:0:0:0-part1 -> ../../sda1
lrwxrwxrwx. 1 root root 10 Feb 15 09:27 pci-0000:00:10.0-scsi-0:0:0:0-part2 -> ../../sda2

unimportant

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
[root@centos77 ~]# multipath -ll
shareddisk (3600140565524db35468485692d22ebc2) dm-2 LIO-ORG ,shared_disk     
size=2.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=50 status=active
  |- 15:0:0:0 sdb 8:16 active ready running
  `- 16:0:0:0 sdc 8:32 active ready running
[root@centos77 ~]# multipath -F
[root@centos77 ~]# multipath -ll
[root@centos77 ~]# multipath -v2
create: shareddisk (3600140565524db35468485692d22ebc2) undef LIO-ORG ,shared_disk     
size=2.0G features='0' hwhandler='0' wp=undef
`-+- policy='round-robin 0' prio=50 status=undef
  |- 15:0:0:0 sdb 8:16 undef ready running
  `- 16:0:0:0 sdc 8:32 undef ready running
[root@centos77 ~]# multipath -ll
shareddisk (3600140565524db35468485692d22ebc2) dm-2 LIO-ORG ,shared_disk     
size=2.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=50 status=active
  |- 15:0:0:0 sdb 8:16 active ready running
  `- 16:0:0:0 sdc 8:32 active ready running
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147

[root@centos77 ~]# multipath -ll
shareddisk (3600140565524db35468485692d22ebc2) dm-2 LIO-ORG ,shared_disk     
size=2.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=50 status=active
  |- 15:0:0:0 sdb 8:16 active ready running
  `- 16:0:0:0 sdc 8:32 active ready running
[root@centos77 ~]# multipath -F
[root@centos77 ~]# multipath -ll
[root@centos77 ~]# multipath -v3
Feb 15 11:21:39 | set open fds limit to 8192/8192
Feb 15 11:21:39 | loading /lib64/multipath/libcheckdirectio.so checker
Feb 15 11:21:39 | loading /lib64/multipath/libprioconst.so prioritizer
Feb 15 11:21:39 | sr0: not found in pathvec
Feb 15 11:21:39 | sr0: device node name blacklisted
Feb 15 11:21:39 | sda: not found in pathvec
Feb 15 11:21:39 | sda: device node name blacklisted
Feb 15 11:21:39 | sdb: not found in pathvec
Feb 15 11:21:39 | sdb: mask = 0x3f
Feb 15 11:21:39 | sdb: dev_t = 8:16
Feb 15 11:21:39 | sdb: size = 4194304
Feb 15 11:21:39 | sdb: vendor = LIO-ORG 
Feb 15 11:21:39 | sdb: product = shared_disk     
Feb 15 11:21:39 | sdb: rev = 4.0 
Feb 15 11:21:39 | sdb: h:b:t:l = 15:0:0:0
Feb 15 11:21:39 | sdb: tgt_node_name = iqn.2020-02.com.baidu:shareddisk
Feb 15 11:21:39 | sdb: path state = running

Feb 15 11:21:39 | sdb: 1009 cyl, 67 heads, 62 sectors/track, start at 0
Feb 15 11:21:39 | sdb: serial = 65524db3-5468-4856-92d2-2ebc2471e3da
Feb 15 11:21:39 | sdb: get_state
Feb 15 11:21:39 | sdb: detect_checker = 1 (config file default)
Feb 15 11:21:39 | loading /lib64/multipath/libcheckreadsector0.so checker
Feb 15 11:21:39 | sdb: path checker = readsector0 (config file default)
Feb 15 11:21:39 | sdb: checker timeout = 30000 ms (sysfs setting)
Feb 15 11:21:39 | sdb: readsector0 state = up
Feb 15 11:21:39 | sdb: uid_attribute = ID_SERIAL (config file default)
Feb 15 11:21:39 | sdb: uid = 3600140565524db35468485692d22ebc2 (udev)
Feb 15 11:21:39 | sdb: detect_prio = 1 (config file default)
Feb 15 11:21:39 | loading /lib64/multipath/libprioalua.so prioritizer
Feb 15 11:21:39 | sdb: prio = alua (config file default)
Feb 15 11:21:39 | sdb: prio args = (null) (config file default)
Feb 15 11:21:39 | reported target port group is 0
Feb 15 11:21:39 | aas = 00 [active/optimized]
Feb 15 11:21:39 | sdb: alua prio = 50
Feb 15 11:21:39 | sdc: not found in pathvec
Feb 15 11:21:39 | sdc: mask = 0x3f
Feb 15 11:21:39 | sdc: dev_t = 8:32
Feb 15 11:21:39 | sdc: size = 4194304
Feb 15 11:21:39 | sdc: vendor = LIO-ORG 
Feb 15 11:21:39 | sdc: product = shared_disk     
Feb 15 11:21:39 | sdc: rev = 4.0 
Feb 15 11:21:39 | sdc: h:b:t:l = 16:0:0:0
Feb 15 11:21:39 | sdc: tgt_node_name = iqn.2020-02.com.baidu:shareddisk
Feb 15 11:21:39 | sdc: path state = running

Feb 15 11:21:39 | sdc: 1009 cyl, 67 heads, 62 sectors/track, start at 0
Feb 15 11:21:39 | sdc: serial = 65524db3-5468-4856-92d2-2ebc2471e3da
Feb 15 11:21:39 | sdc: get_state
Feb 15 11:21:39 | sdc: detect_checker = 1 (config file default)
Feb 15 11:21:39 | sdc: path checker = readsector0 (config file default)
Feb 15 11:21:39 | sdc: checker timeout = 30000 ms (sysfs setting)
Feb 15 11:21:39 | sdc: readsector0 state = up
Feb 15 11:21:39 | sdc: uid_attribute = ID_SERIAL (config file default)
Feb 15 11:21:39 | sdc: uid = 3600140565524db35468485692d22ebc2 (udev)
Feb 15 11:21:39 | sdc: detect_prio = 1 (config file default)
Feb 15 11:21:39 | sdc: prio = alua (config file default)
Feb 15 11:21:39 | sdc: prio args = (null) (config file default)
Feb 15 11:21:39 | reported target port group is 0
Feb 15 11:21:39 | aas = 00 [active/optimized]
Feb 15 11:21:39 | sdc: alua prio = 50
Feb 15 11:21:39 | loop0: not found in pathvec
Feb 15 11:21:39 | loop0: device node name blacklisted
Feb 15 11:21:39 | dm-0: not found in pathvec
Feb 15 11:21:39 | dm-0: device node name blacklisted
Feb 15 11:21:39 | dm-1: not found in pathvec
Feb 15 11:21:39 | dm-1: device node name blacklisted
===== paths list =====
uuid                              hcil     dev dev_t pri dm_st chk_st vend/pro
3600140565524db35468485692d22ebc2 15:0:0:0 sdb 8:16  50  undef undef  LIO-ORG 
3600140565524db35468485692d22ebc2 16:0:0:0 sdc 8:32  50  undef undef  LIO-ORG 
Feb 15 11:21:39 | found multiple paths with wwid 3600140565524db35468485692d22ebc2, multipathing sdb
Feb 15 11:21:39 | sdb: ownership set to shareddisk
Feb 15 11:21:39 | sdb: not found in pathvec
Feb 15 11:21:39 | sdb: mask = 0xc
Feb 15 11:21:39 | sdb: path state = running

Feb 15 11:21:39 | sdb: get_state
Feb 15 11:21:39 | sdb: readsector0 state = up
Feb 15 11:21:39 | reported target port group is 0
Feb 15 11:21:39 | aas = 00 [active/optimized]
Feb 15 11:21:39 | sdb: alua prio = 50
Feb 15 11:21:39 | sdc: ownership set to shareddisk
Feb 15 11:21:39 | sdc: not found in pathvec
Feb 15 11:21:39 | sdc: mask = 0xc
Feb 15 11:21:39 | sdc: path state = running

Feb 15 11:21:39 | sdc: get_state
Feb 15 11:21:39 | sdc: readsector0 state = up
Feb 15 11:21:39 | reported target port group is 0
Feb 15 11:21:39 | aas = 00 [active/optimized]
Feb 15 11:21:39 | sdc: alua prio = 50
Feb 15 11:21:39 | shareddisk: pgfailback = -2 (config file default)
Feb 15 11:21:39 | shareddisk: pgpolicy = multibus (config file default)
Feb 15 11:21:39 | shareddisk: selector = round-robin 0 (config file default)
Feb 15 11:21:39 | shareddisk: features = 0 (config file default)
Feb 15 11:21:39 | shareddisk: retain_attached_hw_handler = 1 (config file default)
Feb 15 11:21:39 | shareddisk: hwhandler = 0 (internal default)
Feb 15 11:21:39 | shareddisk: rr_weight = 2 (config file default)
Feb 15 11:21:39 | shareddisk: minio = 1 rq (config file default)
Feb 15 11:21:39 | shareddisk: no_path_retry = -1 (config file default)
Feb 15 11:21:39 | shareddisk: pg_timeout = NONE (internal default)
Feb 15 11:21:39 | shareddisk: fast_io_fail_tmo = 5 (config file default)
Feb 15 11:21:39 | shareddisk: deferred_remove = 1 (config file default)
Feb 15 11:21:39 | delay_watch_checks = DISABLED (internal default)
Feb 15 11:21:39 | delay_wait_checks = DISABLED (internal default)
Feb 15 11:21:39 | marginal_path_err_sample_time = DISABLED (internal default)
Feb 15 11:21:39 | marginal_path_err_rate_threshold = DISABLED (internal default)
Feb 15 11:21:39 | marginal_path_err_recheck_gap_time = DISABLED (internal default)
Feb 15 11:21:39 | marginal_path_double_failed_time = DISABLED (internal default)
Feb 15 11:21:39 | skip_kpartx = 1 (config file default)
Feb 15 11:21:39 | unpriv_sgio = 1 (config file default)
Feb 15 11:21:39 | shareddisk: remove queue_if_no_path from '0'
Feb 15 11:21:39 | shareddisk: assembled map [0 0 1 1 round-robin 0 2 1 8:16 50 8:32 50]
Feb 15 11:21:39 | shareddisk: set ACT_CREATE (map does not exist)
Feb 15 11:21:39 | ghost_delay = -1 (config file default)
Feb 15 11:21:39 | shareddisk: unset queue_if_no_path feature
create: shareddisk (3600140565524db35468485692d22ebc2) undef LIO-ORG ,shared_disk     
size=2.0G features='0' hwhandler='0' wp=undef
`-+- policy='round-robin 0' prio=50 status=undef
  |- 15:0:0:0 sdb 8:16 undef ready running
  `- 16:0:0:0 sdc 8:32 undef ready running
Feb 15 11:21:39 | readsector0 checker refcount 2
Feb 15 11:21:39 | alua prioritizer refcount 2
Feb 15 11:21:39 | readsector0 checker refcount 1
Feb 15 11:21:39 | alua prioritizer refcount 1
Feb 15 11:21:39 | unloading alua prioritizer
Feb 15 11:21:39 | unloading const prioritizer
Feb 15 11:21:39 | unloading readsector0 checker
Feb 15 11:21:39 | unloading directio checker
[root@centos77 ~]# multipath -ll
shareddisk (3600140565524db35468485692d22ebc2) dm-2 LIO-ORG ,shared_disk     
size=2.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=50 status=active
  |- 15:0:0:0 sdb 8:16 active ready running
  `- 16:0:0:0 sdc 8:32 active ready running

n条链路

配置N个浮动IP就可以有N+1(真实物理IP)条链路

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
[root@centos77 ~]# ifconfig ens33:1 192.168.1.12 netmask 255.255.255.0
[root@centos77 ~]# iscsiadm -m iface -I iface2 --op=new
New interface iface2 added
[root@centos77 ~]# iscsiadm -m iface -I iface2 --op=update -n iface.net_ifacename -v ens33:1
iface2 updated.
[root@centos77 ~]# iscsiadm -m iface
default tcp,<empty>,<empty>,<empty>,<empty>
iser iser,<empty>,<empty>,<empty>,<empty>
iface1 tcp,<empty>,<empty>,ens33:0,<empty>
iface0 tcp,<empty>,<empty>,ens33,<empty>
iface2 tcp,<empty>,<empty>,ens33:1,<empty>
[root@centos77 ~]# iscsiadm -m discoverydb -t st -p 192.168.1.10 -I iface0 -I iface1 -I iface2
# BEGIN RECORD 6.2.0.874-10
discovery.startup = manual
discovery.type = sendtargets
discovery.sendtargets.address = 192.168.1.10
discovery.sendtargets.port = 3260
discovery.sendtargets.auth.authmethod = None
discovery.sendtargets.auth.username = <empty>
discovery.sendtargets.auth.password = <empty>
discovery.sendtargets.auth.username_in = <empty>
discovery.sendtargets.auth.password_in = <empty>
discovery.sendtargets.timeo.login_timeout = 15
discovery.sendtargets.use_discoveryd = No
discovery.sendtargets.discoveryd_poll_inval = 30
discovery.sendtargets.reopen_max = 5
discovery.sendtargets.timeo.auth_timeout = 45
discovery.sendtargets.timeo.active_timeout = 30
discovery.sendtargets.iscsi.MaxRecvDataSegmentLength = 32768
# END RECORD

[root@centos77 ~]# iscsiadm -m discoverydb -t st -p 192.168.1.12 --discover
iscsiadm: Could not bind connection 0 to ens33:0
iscsiadm: cannot make connection to 192.168.1.12: No such device
iscsiadm: Could not bind connection 0 to ens33:0
iscsiadm: cannot make connection to 192.168.1.12: No such device
iscsiadm: Could not bind connection 0 to ens33:0
iscsiadm: cannot make connection to 192.168.1.12: No such device
iscsiadm: Could not bind connection 0 to ens33:0
iscsiadm: cannot make connection to 192.168.1.12: No such device
iscsiadm: Could not bind connection 0 to ens33:0
iscsiadm: cannot make connection to 192.168.1.12: No such device
iscsiadm: Could not bind connection 0 to ens33:0
iscsiadm: cannot make connection to 192.168.1.12: No such device
iscsiadm: connection login retries (reopen_max) 5 exceeded
iscsiadm: Could not bind connection 0 to ens33:1
iscsiadm: cannot make connection to 192.168.1.12: No such device
iscsiadm: Could not bind connection 0 to ens33:1
iscsiadm: cannot make connection to 192.168.1.12: No such device
iscsiadm: Could not bind connection 0 to ens33:1
iscsiadm: cannot make connection to 192.168.1.12: No such device
iscsiadm: Could not bind connection 0 to ens33:1
iscsiadm: cannot make connection to 192.168.1.12: No such device
iscsiadm: Could not bind connection 0 to ens33:1
iscsiadm: cannot make connection to 192.168.1.12: No such device
iscsiadm: Could not bind connection 0 to ens33:1
iscsiadm: cannot make connection to 192.168.1.12: No such device
iscsiadm: connection login retries (reopen_max) 5 exceeded
192.168.1.12:3260,1 iqn.2020-02.com.baidu:shareddisk
[root@centos77 ~]# iscsiadm -m node
192.168.1.10:3260,1 iqn.2020-02.com.baidu:shareddisk
192.168.1.11:3260,1 iqn.2020-02.com.baidu:shareddisk
192.168.1.12:3260,1 iqn.2020-02.com.baidu:shareddisk
[root@centos77 ~]# iscsiadm --mode node --targetname iqn.2020-02.com.baidu:shareddisk --portal 192.168.1.12:3260 --login
Logging in to [iface: iface0, target: iqn.2020-02.com.baidu:shareddisk, portal: 192.168.1.12,3260] (multiple)
Login to [iface: iface0, target: iqn.2020-02.com.baidu:shareddisk, portal: 192.168.1.12,3260] successful.
[root@centos77 ~]# iscsiadm -m session
tcp: [5] 192.168.1.10:3260,1 iqn.2020-02.com.baidu:shareddisk (non-flash)
tcp: [6] 192.168.1.11:3260,1 iqn.2020-02.com.baidu:shareddisk (non-flash)
tcp: [7] 192.168.1.12:3260,1 iqn.2020-02.com.baidu:shareddisk (non-flash)
[root@centos77 ~]# lsblk 
NAME            MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
sda               8:0    0    7G  0 disk  
├─sda1            8:1    0    1G  0 part  /boot
└─sda2            8:2    0    6G  0 part  
  ├─centos-root 253:0    0  5.3G  0 lvm   /
  └─centos-swap 253:1    0  716M  0 lvm   [SWAP]
sdb               8:16   0    2G  0 disk  
└─shareddisk    253:2    0    2G  0 mpath 
sdc               8:32   0    2G  0 disk  
└─shareddisk    253:2    0    2G  0 mpath 
sdd               8:48   0    2G  0 disk  
└─shareddisk    253:2    0    2G  0 mpath 
sr0              11:0    1 1024M  0 rom   
loop0             7:0    0    2G  0 loop  
[root@centos77 ~]# multipath -ll
shareddisk (3600140565524db35468485692d22ebc2) dm-2 LIO-ORG ,shared_disk     
size=2.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=50 status=active
  |- 7:0:0:0 sdb 8:16 active ready running
  |- 8:0:0:0 sdc 8:32 active ready running
  `- 9:0:0:0 sdd 8:48 active ready running
[root@centos77 ~]#

测试断路

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
[root@centos77 ~]# ifconfig
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.1.10  netmask 255.255.255.0  broadcast 192.168.1.255
        ether 00:0c:29:19:60:fc  txqueuelen 1000  (Ethernet)
        RX packets 13470  bytes 1225681 (1.1 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 4841  bytes 798163 (779.4 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens33:0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.1.11  netmask 255.255.255.0  broadcast 192.168.1.255
        ether 00:0c:29:19:60:fc  txqueuelen 1000  (Ethernet)

ens33:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.1.12  netmask 255.255.255.0  broadcast 192.168.1.255
        ether 00:0c:29:19:60:fc  txqueuelen 1000  (Ethernet)

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 15034  bytes 16187808 (15.4 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 15034  bytes 16187808 (15.4 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[root@centos77 ~]# multipath -ll
shareddisk (3600140565524db35468485692d22ebc2) dm-2 LIO-ORG ,shared_disk     
size=2.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=50 status=active
  |- 7:0:0:0 sdb 8:16 active ready running
  |- 8:0:0:0 sdc 8:32 active ready running
  `- 9:0:0:0 sdd 8:48 active ready running
[root@centos77 ~]# ip addr del 192.168.1.12 dev ens33:1
Warning: Executing wildcard deletion to stay compatible with old scripts.
         Explicitly specify the prefix length (192.168.1.12/32) to avoid this warning.
         This special behaviour is likely to disappear in further releases,
         fix your scripts!
[root@centos77 ~]# ifconfig
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.1.10  netmask 255.255.255.0  broadcast 192.168.1.255
        ether 00:0c:29:19:60:fc  txqueuelen 1000  (Ethernet)
        RX packets 13502  bytes 1228927 (1.1 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 4873  bytes 803439 (784.6 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens33:0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.1.11  netmask 255.255.255.0  broadcast 192.168.1.255
        ether 00:0c:29:19:60:fc  txqueuelen 1000  (Ethernet)

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 15160  bytes 16200828 (15.4 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 15160  bytes 16200828 (15.4 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[root@centos77 ~]# multipath -ll
shareddisk (3600140565524db35468485692d22ebc2) dm-2 LIO-ORG ,shared_disk     
size=2.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=50 status=active
  |- 7:0:0:0 sdb 8:16 active ready running
  |- 8:0:0:0 sdc 8:32 active ready running
  `- 9:0:0:0 sdd 8:48 active faulty running
[root@centos77 ~]# ifconfig ens33:1 192.168.1.12 netmask 255.255.255.0
[root@centos77 ~]# multipath -ll
shareddisk (3600140565524db35468485692d22ebc2) dm-2 LIO-ORG ,shared_disk     
size=2.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=50 status=active
  |- 7:0:0:0 sdb 8:16 active ready running
  |- 8:0:0:0 sdc 8:32 active ready running
  `- 9:0:0:0 sdd 8:48 failed ready running
[root@centos77 ~]# multipath -ll
shareddisk (3600140565524db35468485692d22ebc2) dm-2 LIO-ORG ,shared_disk     
size=2.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=50 status=active
  |- 7:0:0:0 sdb 8:16 active ready running
  |- 8:0:0:0 sdc 8:32 active ready running
  `- 9:0:0:0 sdd 8:48 failed ready running
[root@centos77 ~]# multipath -ll
shareddisk (3600140565524db35468485692d22ebc2) dm-2 LIO-ORG ,shared_disk     
size=2.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=50 status=active
  |- 7:0:0:0 sdb 8:16 active ready running
  |- 8:0:0:0 sdc 8:32 active ready running
  `- 9:0:0:0 sdd 8:48 failed ready running
[root@centos77 ~]# multipath -ll
shareddisk (3600140565524db35468485692d22ebc2) dm-2 LIO-ORG ,shared_disk     
size=2.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=50 status=active
  |- 7:0:0:0 sdb 8:16 active ready running
  |- 8:0:0:0 sdc 8:32 active ready running
  `- 9:0:0:0 sdd 8:48 failed ready running
[root@centos77 ~]# multipath -ll
shareddisk (3600140565524db35468485692d22ebc2) dm-2 LIO-ORG ,shared_disk     
size=2.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=50 status=active
  |- 7:0:0:0 sdb 8:16 active ready running
  |- 8:0:0:0 sdc 8:32 active ready running
  `- 9:0:0:0 sdd 8:48 active ready running
[root@centos77 ~]#

参考

素丶素丶丶wodetian

忍一时越想越气，退一步越想越亏。