목표
- Kubernetes Node에서 사용할 iSCSI 구성
서버
| IP | hostname | 비고 |
| 192.168.110.99 | sung-haproxy | haproxy, iSCSI Target 서버 |
| 192.168.110.100 | sung-ubuntu01 | master01 |
| 192.168.110.101 | sung-ubuntu02 | master02 |
| 192.168.110.102 | sung-ubuntu03 | master03 |
| 192.168.110.103 | sung-ubuntu04 | worker01 |
| 192.168.110.104 | sung-ubuntu05 | worker02 |
1. iSCSI Target 구성
(1) Package 설치
# install iSCSI Target Package
root@sung-haproxy:~# apt install tgt
root@sung-haproxy:~# systemctl status tgt
● tgt.service - (i)SCSI target daemon
Loaded: loaded (/lib/systemd/system/tgt.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2022-08-03 01:52:54 UTC; 25s ago
Docs: man:tgtd(8)
Main PID: 27367 (tgtd)
Status: "Starting event loop..."
Tasks: 1
Memory: 1.2M
CGroup: /system.slice/tgt.service
└─27367 /usr/sbin/tgtd -f
Aug 03 01:52:54 sung-haproxy systemd[1]: Starting (i)SCSI target daemon...
Aug 03 01:52:54 sung-haproxy tgtd[27367]: tgtd: iser_ib_init(3431) Failed to initialize RDMA; load kernel modules?
Aug 03 01:52:54 sung-haproxy tgtd[27367]: tgtd: work_timer_start(146) use timer_fd based scheduler
Aug 03 01:52:54 sung-haproxy tgtd[27367]: tgtd: bs_init(387) use signalfd notification
Aug 03 01:52:54 sung-haproxy systemd[1]: Started (i)SCSI target daemon.
(2) sung-haproxy 서버에 디스크 추가 /vdb, /vdc
- 오픈스택을 사용하였으나, kvm으로 구성해도 동일 함.
root@sung-haproxy:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
loop0 7:0 0 47M 1 loop /snap/snapd/16010
loop1 7:1 0 61.9M 1 loop /snap/core20/1518
loop2 7:2 0 67.8M 1 loop /snap/lxd/22753
loop3 7:3 0 47M 1 loop /snap/snapd/16292
loop4 7:4 0 62M 1 loop /snap/core20/1587
vda 252:0 0 50G 0 disk
├─vda1 252:1 0 49.9G 0 part /
├─vda14 252:14 0 4M 0 part
└─vda15 252:15 0 106M 0 part /boot/efi
vdb 252:16 0 50G 0 disk
vdc 252:32 0 50G 0 disk
(3) lv 생성 및 마운트
root@sung-haproxy:~# fdisk /dev/vdb
Welcome to fdisk (util-linux 2.34).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Device does not contain a recognized partition table.
Created a new DOS disklabel with disk identifier 0x6a0c98a4.
Command (m for help): n
Partition type
p primary (0 primary, 0 extended, 4 free)
e extended (container for logical partitions)
Select (default p): p
Partition number (1-4, default 1):
First sector (2048-104857599, default 2048):
Last sector, +/-sectors or +/-size{K,M,G,T,P} (2048-104857599, default 104857599):
Created a new partition 1 of type 'Linux' and of size 50 GiB.
Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.
# partprobe : partition recognize without reboot
root@sung-haproxy:/# partprobe /dev/vdb
root@sung-haproxy:/iscsi/iscsi1# mkfs.btrfs /dev/vdb1
btrfs-progs v5.4.1
See http://btrfs.wiki.kernel.org for more information.
Label: (null)
UUID: e92bab2c-ffba-4b6e-8ee5-ca26120decee
Node size: 16384
Sector size: 4096
Filesystem size: 50.00GiB
Block group profiles:
Data: single 8.00MiB
Metadata: DUP 256.00MiB
System: DUP 8.00MiB
SSD detected: no
Incompat features: extref, skinny-metadata
Checksum: crc32c
Number of devices: 1
Devices:
ID SIZE PATH
1 50.00GiB /dev/vdb1
#vdc 생략
# vg 생성
root@sung-haproxy:/iscsi/iscsi1# vgcreate iscsi_vg /dev/vdb1 /dev/vdc1
Volume group "iscsi_vg" successfully created
root@sung-haproxy:/iscsi/iscsi1# vgdisplay iscsi_vg
--- Volume group ---
VG Name iscsi_vg
System ID
Format lvm2
Metadata Areas 2
Metadata Sequence No 1
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 0
Open LV 0
Max PV 0
Cur PV 2
Act PV 2
VG Size 99.99 GiB
PE Size 4.00 MiB
Total PE 25598
Alloc PE / Size 0 / 0
Free PE / Size 25598 / 99.99 GiB
VG UUID 4P8XCb-cBnA-WDDt-d4vv-EZPl-nVN5-pL8Yqi
# lv 생성
root@sung-haproxy:/dev/mapper# lvcreate -n iscsi_lv -l 100%FREE iscsi_vg
Logical volume "iscsi_lv" created.
root@sung-haproxy:/dev/mapper# lvdisplay
--- Logical volume ---
LV Path /dev/iscsi_vg/iscsi_lv
LV Name iscsi_lv
VG Name iscsi_vg
LV UUID RoYxzG-yZHa-uS08-l3oF-ed4G-e6sq-YjA9zz
LV Write Access read/write
LV Creation host, time sung-haproxy, 2022-08-03 03:52:25 +0000
LV Status available
# open 0
LV Size 99.99 GiB
Current LE 25598
Segments 2
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:0
#
root@sung-haproxy:/dev/mapper# mkfs.btrfs /dev/iscsi_vg/iscsi_lv
btrfs-progs v5.4.1
See http://btrfs.wiki.kernel.org for more information.
Label: (null)
UUID: db07ddef-5a3a-46de-a542-ffd60492f2dc
Node size: 16384
Sector size: 4096
Filesystem size: 99.99GiB
Block group profiles:
Data: single 8.00MiB
Metadata: DUP 1.00GiB
System: DUP 8.00MiB
SSD detected: no
Incompat features: extref, skinny-metadata
Checksum: crc32c
Number of devices: 1
Devices:
ID SIZE PATH
1 99.99GiB /dev/iscsi_vg/iscsi_lv
mkdir -p /iscsi
root@sung-haproxy:/# mount /dev/iscsi_vg/iscsi_lv /iscsi/
root@sung-haproxy:/# df -h | grep iscsi
/dev/mapper/iscsi_vg-iscsi_lv 100G 3.8M 98G 1% /iscsi
--------------------------------------------------------
그냥 btrfs subvolume 생성만 해도 되나?
root@sung-haproxy:/iscsi# chmod 755 /iscsi/disk
root@sung-haproxy:/iscsi/disk# btrfs subvolume create /iscsi/disk1
Create subvolume '/iscsi/disk1'
root@sung-haproxy:/iscsi# dd if=/dev/zero of=/iscsi/disk1/sung-ubuntu01-disk.img bs=1G count=5
50+0 records in
50+0 records out
53687091200 bytes (54 GB, 50 GiB) copied, 573.436 s, 93.6 MB/s
#dd if=/dev/zero of=/iscsi/disk1/sung-ubuntu02-disk.img bs=1G count=5
#dd if=/dev/zero of=/iscsi/disk1/sung-ubuntu03-disk.img bs=1G count=5
#dd if=/dev/zero of=/iscsi/disk1/sung-ubuntu04-disk.img bs=1G count=5
#dd if=/dev/zero of=/iscsi/disk1/sung-ubuntu05-disk.img bs=1G count=5
##한 번에..
#for i in {1..5}; do echo $i; dd if=/dev/zero of=/iscsi/disk1/sung-ubuntu0$i-disk.img bs=1GB count=5; sleep 5; done
(4) 새로운 iSCSI Target 생성
# 1. 새로운 Target의 이름 및 id (tid) 생성
tgtadm --lld iscsi --op new --mode target --tid 3466 -T iqn.server.ubuntu:sllee3466
# New Target Name : iscsi
# tid : 3466
# ISCSI Qualified Name(iqn) : iqn.server.ubuntu:sllee
root@sung-haproxy:/iscsi# tgtadm --lld iscsi --mode target --op show
Target 3466: iqn.server.ubuntu:sllee3466
System information:
Driver: iscsi
State: ready
I_T nexus information:
LUN information:
LUN: 0
Type: controller
SCSI ID: IET 0d8a0000
SCSI SN: beaf34660
Size: 0 MB, Block size: 1
Online: Yes
Removable media: No
Prevent removal: No
Readonly: No
SWP: No
Thin-provisioning: No
Backing store type: null
Backing store path: None
Backing store flags:
Account information:
ACL information:
# 2. tid에 대응되는 logicalunit에 디스크 이미지 연결
tgtadm --lld iscsi --op new --mode logicalunit --tid 3466 --lun 1 -b /iscsi/disk1/sung-ubuntu01-disk.img
tgtadm --lld iscsi --op new --mode logicalunit --tid 3466 --lun 2 -b /iscsi/disk1/sung-ubuntu02-disk.img
tgtadm --lld iscsi --op new --mode logicalunit --tid 3466 --lun 3 -b /iscsi/disk1/sung-ubuntu03-disk.img
tgtadm --lld iscsi --op new --mode logicalunit --tid 3466 --lun 4 -b /iscsi/disk1/sung-ubuntu04-disk.img
tgtadm --lld iscsi --op new --mode logicalunit --tid 3466 --lun 5 -b /iscsi/disk1/sung-ubuntu05-disk.img
# 3. tid 접속 허가 설정
tgtadm --lld iscsi --op bind --mode target --tid 3466 -I 192.168.110.100
tgtadm --lld iscsi --op bind --mode target --tid 3466 -I 192.168.110.101
tgtadm --lld iscsi --op bind --mode target --tid 3466 -I 192.168.110.102
tgtadm --lld iscsi --op bind --mode target --tid 3466 -I 192.168.110.103
tgtadm --lld iscsi --op bind --mode target --tid 3466 -I 192.168.110.104
#LUN에 특정 노드 할당 할 수 있을까?
# - SAN Storage에서는 가능한 것 같은데, iscsiadm으로는 안되는 것 같음
# 설정 확인
root@sung-haproxy:/iscsi/disk1# tgtadm --lld iscsi --mode target --op show
Target 3466: iqn.server.ubuntu:sllee3466
System information:
Driver: iscsi
State: ready
I_T nexus information:
LUN information:
LUN: 0
Type: controller
SCSI ID: IET 0d8a0000
SCSI SN: beaf34660
Size: 0 MB, Block size: 1
Online: Yes
Removable media: No
Prevent removal: No
Readonly: No
SWP: No
Thin-provisioning: No
Backing store type: null
Backing store path: None
Backing store flags:
LUN: 1
Type: disk
SCSI ID: IET 0d8a0001
SCSI SN: beaf34661
Size: 5000 MB, Block size: 512
Online: Yes
Removable media: No
Prevent removal: No
Readonly: No
SWP: No
Thin-provisioning: No
Backing store type: rdwr
Backing store path: /iscsi/disk1/sung-ubuntu01-disk.img
Backing store flags:
LUN: 2
Type: disk
SCSI ID: IET 0d8a0002
SCSI SN: beaf34662
Size: 5000 MB, Block size: 512
Online: Yes
Removable media: No
Prevent removal: No
Readonly: No
SWP: No
Thin-provisioning: No
Backing store type: rdwr
Backing store path: /iscsi/disk1/sung-ubuntu02-disk.img
Backing store flags:
LUN: 3
Type: disk
SCSI ID: IET 0d8a0003
SCSI SN: beaf34663
Size: 5000 MB, Block size: 512
Online: Yes
Removable media: No
Prevent removal: No
Readonly: No
SWP: No
Thin-provisioning: No
Backing store type: rdwr
Backing store path: /iscsi/disk1/sung-ubuntu03-disk.img
Backing store flags:
LUN: 4
Type: disk
SCSI ID: IET 0d8a0004
SCSI SN: beaf34664
Size: 5000 MB, Block size: 512
Online: Yes
Removable media: No
Prevent removal: No
Readonly: No
SWP: No
Thin-provisioning: No
Backing store type: rdwr
Backing store path: /iscsi/disk1/sung-ubuntu04-disk.img
Backing store flags:
LUN: 5
Type: disk
SCSI ID: IET 0d8a0005
SCSI SN: beaf34665
Size: 5000 MB, Block size: 512
Online: Yes
Removable media: No
Prevent removal: No
Readonly: No
SWP: No
Thin-provisioning: No
Backing store type: rdwr
Backing store path: /iscsi/disk1/sung-ubuntu05-disk.img
Backing store flags:
Account information:
ACL information:
192.168.110.100
192.168.110.101
192.168.110.102
192.168.110.103
192.168.110.104
## 생성된 target 삭제
# tgtadm --lld iscsi --op delete --mode target --tid 3466
## 사용중 target 강제삭제
#tgtadm --lld iscsi --op delete --force --mode target --tid 3466
(5) iSCSI Initiarot 구성
root@sung-ubuntu05:~# apt -y install open-iscsi
root@sung-ubuntu05:/etc/iscsi# vi /etc/iscsi/initiatorname.iscsi
InitiatorName=iqn.server.ubuntu:sllee3466
root@sung-ubuntu05:/etc/iscsi# iscsiadm -m discovery -t sendtargets -p 192.168.110.99
192.168.110.99:3260,1 iqn.server.ubuntu:sllee3466
root@sung-ubuntu05:/etc/iscsi# iscsiadm -m node -o show
root@sung-ubuntu05:/etc/iscsi# iscsiadm -m node --login
Logging in to [iface: default, target: iqn.server.ubuntu:sllee3466, portal: 192.168.110.99,3260] (multiple)
Login to [iface: default, target: iqn.server.ubuntu:sllee3466, portal: 192.168.110.99,3260] successful.
root@sung-ubuntu05:/etc/iscsi# iscsiadm -m session -o show
tcp: [1] 192.168.110.99:3260,1 iqn.server.ubuntu:sllee3466 (non-flash)
ls
# fdisk -l 명령어로 확인해도 됨.
root@sung-ubuntu05:/etc/iscsi# cat /proc/partitions
major minor #blocks name
7 1 63444 loop1
7 2 48080 loop2
7 3 69440 loop3
7 4 48088 loop4
7 5 63448 loop5
252 0 52428800 vda
252 1 52315119 vda1
252 14 4096 vda14
252 15 108544 vda15
252 16 1048576 vdb
252 17 1047552 vdb1
8 0 52428800 sda # 추가 됨
#연결된 iscsi 확인
root@sung-ubuntu05:/dev/disk/by-path# ls -al /dev/disk/by-path
root@sung-ubuntu05:/etc/iscsi# parted --script /dev/sda "mklabel gpt"
root@sung-ubuntu05:/etc/iscsi# parted --script /dev/sda "mkpart primary 0% 100%"
root@sung-ubuntu05:/etc/iscsi# mkfs.ext4 /dev/sda
root@sung-ubuntu05:/etc/iscsi# mkdir /iscsi
root@sung-ubuntu05:/etc/iscsi# mount /dev/sda1 /iscsi
### Remove initiator node
#1. umount iSCSI device/filesystems
root@sung-ubuntu05:/etc/iscsi# umount /iscsi
#2. logout session
root@sung-ubuntu05:/etc/iscsi# iscsiadm -m node -T iqn.server.ubuntu:sllee3466 --portal 192.168.110.99:3260 -u
Logging out of session [sid: 1, target: iqn.server.ubuntu:sllee3466, portal: 192.168.110.99,3260]
Logout of [sid: 1, target: iqn.server.ubuntu:sllee3466, portal: 192.168.110.99,3260] successful.
root@sung-ubuntu05:/etc/iscsi# iscsiadm -m session -o show
iscsiadm: No active sessions.
#3. delete node
iscsiadm -m node -T iqn.server.ubuntu:sllee3466 --portal 192.168.110.99:3260 -o delete
#4. check
root@sung-ubuntu05:~# iscsiadm -m node
후기
- Target에 5개의 LUN을 생성하여, sung-ubuntu01~05서버에서 login을 할 경우
- 각 sung-ubuntuXX 서버에서 모든 LUN이 보이고, 모든 디스크가 검색이 됨
- 서버-LUN을 1:1로 연결해야하는데, 1:N으로 마운트가 되어서 문제가 발생 됨.
- SAN Storage에서는 되는 것 같은데…확인필요..
- 운영환경 스토리지 구조도 한 번 확인해봐야 겠음.
'Linux' 카테고리의 다른 글
| [Linux] CentOS repository 구성 (0) | 2022.10.21 |
|---|---|
| [Ubuntu] 특정 IP 차단/허용 (0) | 2022.10.13 |
| [ubuntu] lvm 구성 (0) | 2022.08.03 |
| [Ubuntu] swap 파티션 생성 (0) | 2022.07.21 |
| [base64] 인코딩/디코딩 (0) | 2022.06.07 |
