东东东 陈煜东的博客

最近的项目是虚拟机在PCI pass-through下做动态迁移,目前还没有一个好的解决方案,需要我们小组去讨论并提出一个解决方案来.既然是要做PCI pass through下做动态迁移的解决方案,那就得要先在这样的情况下做个迁移,看看到底会出现什么样的问题.

方案一:使用virsh进行动态迁移

在前面的文章中,我们进行的虚拟机的pci pass through,如果对pci pass through有不会的地方,参考之前的文章来进行设置.

接下来,我们要进行虚拟机的动态迁移.使用virsh启动虚拟机,运行以下命令进行迁移

virsh migrate --live vdisk qemu+ssh://211.87.***.45/system --unsafe

运行完后会出现问题.报了一个这样的错误.

error: Requested operation is not valid: domain has assigned non-USB host devices

上面的错误说明,现在的请求操作的非法的,因为虚拟机使用了非USB设备.

可以看到libvirt的源码 src/qemu/qemu_migration.c

/* Validate whether the domain is safe to migrate.  If vm is NULL,
 * then this is being run in the v2 Prepare stage on the destination
 * (where we only have the target xml); if vm is provided, then this
 * is being run in either v2 Perform or v3 Begin (where we also have
 * access to all of the domain's metadata, such as whether it is
 * marked autodestroy or has snapshots).  While it would be nice to
 * assume that checking on source is sufficient to prevent ever
 * talking to the destination in the first place, we are stuck with
 * the fact that older servers did not do checks on the source. */
bool
qemuMigrationIsAllowed(virQEMUDriverPtr driver, virDomainObjPtr vm,
                       virDomainDefPtr def, bool remote, bool abort_on_error)
{
    int nsnapshots;
    int pauseReason;
    bool forbid;
    size_t i;

    if (vm) {
        if (qemuProcessAutoDestroyActive(driver, vm)) {
            virReportError(VIR_ERR_OPERATION_INVALID,
                           "%s", _("domain is marked for auto destroy"));
            return false;
        }

        /* perform these checks only when migrating to remote hosts */
        if (remote) {
            nsnapshots = virDomainSnapshotObjListNum(vm->snapshots, NULL, 0);
            if (nsnapshots < 0)
                return false;

            if (nsnapshots > 0) {
                virReportError(VIR_ERR_OPERATION_INVALID,
                               _("cannot migrate domain with %d snapshots"),
                               nsnapshots);
                return false;
            }

            /* cancel migration if disk I/O error is emitted while migrating */
            if (abort_on_error &&
                virDomainObjGetState(vm, &pauseReason) == VIR_DOMAIN_PAUSED &&
                pauseReason == VIR_DOMAIN_PAUSED_IOERROR) {
                virReportError(VIR_ERR_OPERATION_INVALID, "%s",
                               _("cannot migrate domain with I/O error"));
                return false;
            }

        }

        if (virDomainHasDiskMirror(vm)) {
            virReportError(VIR_ERR_OPERATION_INVALID, "%s",
                           _("domain has an active block job"));
            return false;
        }

        def = vm->def;
    }

    /* Migration with USB host devices is allowed, all other devices are
     * forbidden.
     */
    forbid = false;
    for (i = 0; i < def->nhostdevs; i++) {
        virDomainHostdevDefPtr hostdev = def->hostdevs[i];
        if (hostdev>mode != VIR_DOMAIN_HOSTDEV_MODE_SUBSYS ||
            hostdev->source.subsys.type != VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_USB) {
            forbid = true;
            break;
        }
    }
    if (forbid) {
        virReportError(VIR_ERR_OPERATION_INVALID,"%s",
                       _("domain has assigned non-USB host devices"));
        return false;
    }

    return true;
}

可以看到源码上显示,只要检测到虚拟机上使用了非USB的PCI设备,直接将borbid设置为true,禁止虚拟机的迁移.

如果要添加可以迁移的PCI设备,那么需要在这里进行相应的修改.

方案二:使用qemu进行动态迁移

在source host运行以下命令以启动虚拟机

/usr/bin/qemu-kvm -name vdisk -enable-kvm -m 512 -smp 2 
-hda /mnt/nfs/vdisk.img 
-monitor stdio 
-vnc 0.0.0.0:0 
-device pci-assign,host=00:1b.0

在dest host上运行以下命令,等待虚拟机的迁移:

/usr/bin/qemu-kvm -name vdisk -enable-kvm -m 512 -smp 2 
-hda /mnt/nfs/vdisk.img 
-monitor stdio 
-vnc 0.0.0.0:0 
-incoming tcp:0.0.0.0:8888

• 参数-incoming tcp:0.0.0.0:8888说明的是,虚拟机在监听tcp的8888端口,等待源虚拟机被迁移过来.

此时虚拟机的状态是inmigration状态,等待被迁移过来.

在source host上输入以下命令,进行虚拟机的迁移.

(qemu) migrate -d -b tcp:dest_ip:8888

 -d 可以在迁移的过程中查询迁移状态，否则只能在迁移结束后查询。
 -b 迁移虚拟机存储文件
 tcp:dest_ip:8888 数据迁移的协议、目的主机和端口。协议和端口必须和目的主机上虚拟机的 -incoming 参数一致。

运行这个命令后,出报这样的错误

(qemu) migrate -d tcp:dest_ip:8888
migrate: State blocked by non-migratable device '0000:00:04.0/pci-assign'

应该也是进行了检查,使用了一个不可迁移设备即pci pass through的设备.

通过libvirt和qemu的动态迁移出先的问题,基本可以定位到相应的代码,那么只要提供一种相应的解决方案,那么就可以迁移其他的设备了.

打赏