A simple shell script to mount qcow2 file through qemu-nbd and pop shell inside
the chrooted partition over the network through socat
Why ?
I needed to inspect an lvm parition (/root) inside a qcow2 disk file.
For this there is many tools, many solutions like:
- nbd-server | nbd-client
- qemu-nbd
- nbdkit (didn't used, just look arround the code and it use qemu-nbd so...)
nbd-server and nbd-client are nice for standards partitions type but lvm paritions are not displayed (I don't know why).
qemu-nbd works very well but over the network, the lvm partition is not displayed.
This can be solved by using socat in a chrooted partition(socat have to be present in the mounted system).
For this script qemu-nbd is used.
But there is one other problem:
After working on the mounted partition, time is to unmount the partition and disconnect the /dev/nbdX device. This is what happen after umount /mount/path and qemu-nbd --disconnect /dev/nbdX:
# lsblk
nbd0 43:0 0 20G 0 disk
|-nbd0p1 43:1 0 487M 0 part
|-nbd0p2 43:2 0 1K 0 part
|-nbd0p5 43:5 0 9.5G 0 part
|-system--tocheck-root 253:7 0 4.7G 0 lvm
|-system--tocheck-var 253:8 0 952M 0 lvm
|-system--tocheck-tmp 253:9 0 952M 0 lvm
|-system--tocheck-swap 253:10 0 1.1G 0 lvm
# lsmod |grep nbd | awk '{print $3}' | xargs -I % echo "Number of modules using nbd: %"
Number of modules using nbd: 3
# lsof | grep nbd
knbd0-rec 1172 root cwd DIR 8,5 4096 2 /
knbd0-rec 1172 root rtd DIR 8,5 4096 2 /
knbd0-rec 1172 root txt unknown /proc/1172/exe
Easy to see that the nbd kernel module is still used by somthing. Just disconnect the /dev/nbdX device looks not enough.
So, what happen ?
What is using nbd module ?
Why lvm partitions are still visible through lsblk command ?
How ?
lvm is the right way, and deeply, the device mapper.
I saw the lvm volume groups from the lsblk command so i decide to check with the file command:
# file /dev/mapper/*
/dev/mapper/system--tocheck-root: symbolic link to ../dm-7
/dev/mapper/system--tocheck-swap: symbolic link to ../dm-10
/dev/mapper/system--tocheck-tmp: symbolic link to ../dm-9
/dev/mapper/system--tocheck-var: symbolic link to ../dm-8
Interesting right ?
What I see here is how lvm works internally, the device mapper kernel driver create a /dev/dm-* device linked with the logical partition (lvm)
Ok, it means that maybe it is possible to disconnect also the /dev/dm-*.
The correct way is to umount mounted partitions, disconnect dm-* devices and diconnect nbd device.
But I wanted network access to the block or the partition, like I said, lvm partition not appears like this.
It's ok there is always at least many solutions to one problem, the goal is to find the better one for the immediat need !
The partition network acces can easily be done with running socat to provide remote shell from the chrooted partition.
The finale idea is to be able to mount remote qcow2 file on a remote server and get shell acces inside a mounted partition
without to do any other action.
- Run the script on the remote server from ssh
- List and choose the partition to mount
- Mount the partition
- Provide chrooted shell
Code !
First, the usage of the script:function usage(){
echo "Usage $(basename $0) {-m|-u} -d {VMDISK} -p {MOUNTPATH} -P {PORT} -D"
echo -e "\t-m : Ask to mount"
echo -e "\t-u : Ask to umount"
echo -e "\t-d VMDISK : Target disk file .img|.qcow2 to mount"
echo -e "\t-p MOUNTPATH : Target path eg: /mnt/root_part"
echo -e "\t-P PORT : Port to open (ssl)"
echo -e "\t-D : debug=1 (default debug=0) dont send socat in background"
}
To mount a qcow2 file the script will looks like:
mount_nbd.sh -m -p /path/to/mount_dir -d file.qcow2 -P 8080
The first called function is the mount_vmdisk() function:
function mount_vmdisk(){
qemu-nbd --connect=/dev/nbd0 $vmdisk
sleep 0.3
choose_partition
}
who will execute the first connection to the nbd device /dev/nbd0 then a little sleep to let the sync time to the kernel
and finaly the choose_partition() function.
function choose_partition(){
partitions_list=`file /dev/mapper/* |grep tocheck |cut -d: -f1`
declare -a PARTITIONS
while read line; do
PARTITIONS+=("$line")
done < <(echo "$partitions_list")
id=0
for part in "${PARTITIONS[@]}"
do
echo "$id - $part"
((id++))
done
while read -p "Choose partition to mount: " id; do
PART_TO_MOUNT="${PARTITIONS[$id]}"
[ ! -z "$PART_TO_MOUNT" ] && \
break || \
echo "Partition doesn't exists"
done
mount_part "$PART_TO_MOUNT"
}
0 - /dev/mapper/system--tocheck-root
1 - /dev/mapper/system--tocheck-swap
2 - /dev/mapper/system--tocheck-tmp
3 - /dev/mapper/system--tocheck-var
Choose partition to mount: 0
First it is the set of the partitions_list variable who will contains the logicals partitions paths, it is a multi-line variable.
Then it is the declaration of the PARTITIONS array and it is populate from the partitions_list.
A list type 'id - partition' is displayed to choose the id linked to the wanted partition, started at 0 like arrays.
This id is used to select the retrieve the partition path in the PARTITIONS array.
Finaly the retrieved partition is passed as parameter to the mount_part function.
The mount_part() function:
function mount_part(){
partition="$1"
mount -w $partition $mountpath
echo "$partition -> $mountpath"
}
It is a simple mount of the selected partiton in the mountpath parameter -p.
Now the wanted partition is mounted, it is the time to serve shell inside this partition to be like in ssh in the partition.
function open_shell_in_mounted_part(){
generate_ssl_cert
mount -t proc none /proc
mount -t sysfs /sys $mountpath/sys
mount -o bind /run mountpath/run
mount -o bind /dev $mountpath/dev
mount -o bind /dev/pts $mountpath/dev/pts
if [[ $debug -eq 1 ]]; then
chroot $mountpath socat -d -d openssl-listen:$port,cert=/tmp/nbdserver.run.pem,verify=0,fork EXEC:"/bin/bash -li",pty,stderr,setsid,sigint,sane
else
nohup bash -c "chroot $mountpath socat -d -d openssl-listen:$port,cert=/tmp/nbdserver.run.pem,verify=0,fork EXEC:\"/bin/bash -li\",pty,stderr,setsid,sigint,sane" &
fi
}
The shell is simply encrypted by a self-signed ssl certificate with the generate_ssl_cert function.
For the ability to get a proper prompt shell with basic function /dev/pts from the host have to be binded in the $mountpath/dev and $mountpath/dev/pts.
And the last part, the chroot and the encrypted socat inside the mounted partition who will server the shell EXEC:"/bin/bash -li".
Here the generate_ssl_cert function:
function generate_ssl_cert(){
openssl req -new -newkey rsa:4096 -days 99999 -nodes -x509 \
-subj "/C=FR/ST=PA/L=PA/O=NBD-Server/CN=nbdserver.run" \
-keyout $mountpath/tmp/nbdserver.run.key -out $mountpath/tmp/nbdserver.run.crt
cat $mountpath/tmp/nbdserver.run.key $mountpath/tmp/nbdserver.run.crt > $mountpath/tmp/nbdserver.run.pem
}
When the work is done, umount_vmdisk function.
function umount_vmdisk(){
umount -l $mountpath/run
umount -l $mountpath/proc
umount -l $mountpath/sys
umount -l $mountpath/dev/pts
umount -l $mountpath/dev
umount -l $mountpath > /dev/null 2>&1
# remove device-mapper
file /dev/mapper/* |grep tocheck | grep -Eo 'dm.*' |xargs -l -I % dmsetup remove --force /dev/% > /dev/null 2>&1
# disable vg *-tocheck
vgchange -a n > /dev/null 2>&1
qemu-nbd --disconnect /dev/nbd0 2>/dev/null
rmmod nbd > /dev/null 2>&1
}
This part was long to solve, it was the problem described before, qemu-nbd doesn't remove the dm-* devices and was solved by this command.
dmsetup remove --force /dev/dm-X
The previous command executed by xargs with a list of dm-* devices piped to it:
file /dev/mapper/* |grep tocheck | grep -Eo 'dm.*' |xargs -l -I % dmsetup remove --force /dev/%
Final touch, deactivate the lvm group, disconnect the nbd device and unload the nbd module.
vgchange -a n > /dev/null 2>&1
qemu-nbd --disconnect /dev/nbd0
rmmod nbd
The full code:
#!/bin/bash
# This script thinks he's on a <server> and you alread moved the vmdisk to mount(from hypv server) on the <server>
function usage(){
echo "Usage $(basename $0) {-m|-u} -d {VMDISK} -p {MOUNTPATH} -P {PORT} -D"
echo -e "\t-m : Ask to mount"
echo -e "\t-u : Ask to umount"
echo -e "\t-d VMDISK : Target disk file .img|.qcow2 to mount"
echo -e "\t-p MOUNTPATH : Target path eg: /mnt/root_part"
echo -e "\t-P PORT : Port to open (ssl)"
echo -e "\t-D : debug=1 (default debug=0) dont send socat in background"
}
function check_nbd_mod(){
if ! lsmod | grep nbd > /dev/null; then
modprobe nbd max_part=8
fi
}
function mount_part(){
partition="$1"
mount -w $partition $mountpath
echo "$partition -> $mountpath"
}
function choose_partition(){
partitions_list=`file /dev/mapper/* |grep tocheck |cut -d: -f1`
declare -a PARTITIONS
while read line; do
PARTITIONS+=("$line")
done < <(echo "$partitions_list")
id=0
for part in "${PARTITIONS[@]}"
do
echo "$id - $part"
((id++))
done
while read -p "Choose partition to mount: " id; do
PART_TO_MOUNT="${PARTITIONS[$id]}"
[ ! -z "$PART_TO_MOUNT" ] && \
break || \
echo "Partition doesn't exists"
done
mount_part "$PART_TO_MOUNT"
}
function mount_vmdisk(){
qemu-nbd --connect=/dev/nbd0 $vmdisk
sleep 0.3
choose_partition
}
function umount_vmdisk(){
umount -l $mountpath/run
umount -l $mountpath/proc
umount -l $mountpath/sys
umount -l $mountpath/dev/pts
umount -l $mountpath/dev
umount -l $mountpath > /dev/null 2>&1
# remove device-mapper
file /dev/mapper/* |grep tocheck | grep -Eo 'dm.*' |xargs -l -I % dmsetup remove --force /dev/% > /dev/null 2>&1
# disable vg *-tocheck
vgchange -a n > /dev/null 2>&1
qemu-nbd --disconnect /dev/nbd0 2>/dev/null
rmmod nbd > /dev/null 2>&1
}
function generate_ssl_cert(){
openssl req -new -newkey rsa:4096 -days 99999 -nodes -x509 \
-subj "/C=FR/ST=PA/L=PA/O=NBD-Server/CN=nbdserver.run" \
-keyout $mountpath/tmp/nbdserver.run.key -out $mountpath/tmp/nbdserver.run.crt
cat $mountpath/tmp/nbdserver.run.key $mountpath/tmp/nbdserver.run.crt > $mountpath/tmp/nbdserver.run.pem
}
function open_shell_in_mounted_part(){
generate_ssl_cert
mount -t proc none $mountpath/proc
mount -t sysfs /sys $mountpath/sys
mount -o bind /run $mountpath/run #grub
mount -o bind /dev $mountpath/dev
mount -o bind /dev/pts $mountpath/dev/pts
if [[ $debug -eq 1 ]]; then
chroot $mountpath socat -d -d openssl-listen:$port,cert=/tmp/nbdserver.run.pem,verify=0,fork EXEC:"/bin/bash -li",pty,stderr,setsid,sigint,sane
else
nohup bash -c "chroot $mountpath socat -d -d openssl-listen:$port,cert=/tmp/nbdserver.run.pem,verify=0,fork EXEC:\"/bin/bash -li\",pty,stderr,setsid,sigint,sane" &
fi
}
mount=0
umount=0
debug=0
param_d=0
param_p=0
param_P=0
while getopts "mud:p:P:" opt; do
case $opt in
m) mount=1 ;;
d) vmdisk=$OPTARG; param_d=1 ;;
p) mountpath=$OPTARG; param_p=1 ;;
P) port=$OPTARG; param_P=1 ;;
D) debug=1;;
u) umount=1 ;;
*) usage; exit 1 ;;
esac
done
shift $(( OPTIND - 1 ))
if [[ $mount -eq 1 ]]; then
(($param_d)) && (($param_p)) && (($param_P)) \
&& check_nbd_mod; mount_vmdisk; open_shell_in_mounted_part \
|| usage
fi
if [[ $umount -eq 1 ]]; then
(($param_p)) \
&& umount_vmdisk \
|| usage
fi
In the "main" the is the case to parse the script's options and the check of the first option to decide mount or unmount.
In these check, (($param_d)) && (($param_p)) && (($param_P)), these variables are evaluated as 0 for false and 1 for true.
If true, run the functions:
check_nbd_mod; mount_vmdisk; open_shell_in_mounted_part
The script has to be on the server who host the qcow2 disk file.
The idea to run this script it is to execute is over ssh:
ssh -t <server> "mount_nbd.sh -m -d /path/to/file.qcow2 -p /path/to/mount -P 8080"
Let's pop the shell !
socat - openssl:<server>:8080,verify=0
root@host:/#
To unmount:
ssh <server> "mount_nbd.sh -u -p /path/to/mount"
Time to debug the system freshly mounted :)