Set up your Open Stack

There are several ways to deploy openstack, Devstack is easily for developer to deploy Open Stack.

Please follow this URL to set up your Open Stack.

nested KVM support:

• Kernel
  • First check if your system has already configured nested kvm:

root@minicloud-allinone-controller0-i7bnc6baumzl:~# dpkg -S /etc/modprobe.d/qemu-system-x86.confqemu-system-x86: /etc/modprobe.d/qemu-system-x86.confroot@minicloud-allinone-controller0-i7bnc6baumzl:~# cat /etc/modprobe.d/qemu-system-x86.confoptions kvm_intel nested=1

• If not, run:

echo “options kvm-intel nested=1″ | sudo tee /etc/modprobe.d/kvm-intel.conf

• then reload kvm_intel.ko

rmmod kvm_intelmodprobe kvm_intel

• Nova
  • Optional: Modify nova.conf and add host-passthrough. Please note it may impact migration.

libvirt_cpu_mode = host-passthrough

• Restart nova services.

PCI PassThrough Support

There are two ways to support PCI PassThrough.

1. legacy UIO framework(PCI Assign);
2. VFIO, it is recommended

For PCI SR-IOV device we have a specially way to setup.

 Preparation for PCI PassThrough

(1) Enable the Intel VT-d extensions

Setup in your bios. Check you have enable it.

$sudo dmesg | grep -e DMAR -e IOMMU

(2) Activate Intel VT-d in the kernel

$ sudo echo 1 > /sys/module/kvm/parameters/allow_unsafe_assigned_interrupts

update the the kernel cmdline.

$ sudo su$ sudo sed -i -e 's/\(^GRUB_CMDLINE_LINUX_DEFAULT.*\)"$/\1 intel_iommu=on"/' /etc/default/grub$ sudo update-grub$ sudo grep intel_iommu=on /boot/grub/grub.cfg

Use legacy UIO framework instead of VFIO

Why we still need the legacy UIO framework, for there are some thing wrong with vfio and igb in some PC.

Set parameter KVM allow_unsafe_assigned_interrupts

Transient configure (take effect Immediately)

$ sudo echo 1 > /sys/module/kvm/parameters/allow_unsafe_assigned_interrupts

Or persistent

$ sudo echo "options kvm allow_unsafe_assigned_interrupts=1" >>  /etc/modprobe.d/kvm_iommu.conf

Search the PCI device

$ lspci -n01:00.0 0200: 8086:10fb (rev 01)01:00.1 0200: 8086:10fb (rev 01)$ lspci01:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)01:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)

Bind pci_stub driver to PCI device

$ sudo modprobe pci_stub$ sudo su(take effect Immediately)$ echo "8086 10fb" > /sys/bus/pci/drivers/pci-stub/new_id$ echo 0000:01:00.0   > /sys/bus/pci/devices/0000\:01\:00.0/driver/unbind$ echo 0000:01:00.0  > /sys/bus/pci/drivers/pci-stub/bind$ echo 0000:01:00.1   > /sys/bus/pci/devices/0000\:01\:00.1/driver/unbind$ echo 0000:01:00.1  > /sys/bus/pci/drivers/pci-stub/bind

Start attach PCI device to guest and start

1. in this way, if you want to use virsh to start your guest, you must assign the driver name, or you will failed for libvirt will use vfio driver as default.
2. name='kvm' means: the legacy pci device assignment handled directly by the KVM kernel module

$ echo "
      
       
         
     
  
    " > new-pci.xml$ virsh attach-device guest-name --config new-pci.xml$ virsh start guest-name

 Use VFIO

There are tons of tutorials on how to get VT-d working with KVM/libvirt/Qemu/etc. It’s still a fairly new and unused feature, which means that it is getting updated pretty frequently. Unfortunately, the Internet doesn’t update as quickly, so most of the documentation out there is not accurate or the best approach. readlink You can refer this when any trouble.

We need this command to load the kernel modules at boot time

$ sudo echo "vfiovfio_iommu_type1vfio_pci" >> /etc/modules

Set vfio_iommu_type1 allow_unsafe_interrupts parameters If you're overriding the ivrs table to enable intremap you dont need this line Persistent (need reboot.)

$ echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" > /etc/modprobe.d/vfio_iommu_type1.conf

Or transient (take effect Immediately)

$ sudo echo 1 >  /sys/module/vfio_iommu_type1/parameters/allow_unsafe_interrupts

Disable AppArmor for libvirtd

Actually, we do not need to disable Apparmor, this is a bug on ubuntu, we need to work around it. (It may can not works)

$ apt-get install apparmor-utils$ sudo aa-complain /usr/sbin/libvirtd$ sudo su$ for x in /etc/apparmor.d/libvirt/libvirt-*[0-9a-b];do aa-complain $x; done

Or (It may can not works)

$ sudo ln -s  /etc/apparmor.d/usr.sbin.libvirtd /etc/apparmor.d/disable/$ sudo ln -s /etc/apparmor.d/usr.lib.libvirt.virt-aa-helper /etc/apparmor.d/disable/$ sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.libvirtd$ sudo apparmor_parser -R /etc/apparmor.d/usr.lib.libvirt.virt-aa-helper$ sudo service libvirt-bin restart

Or remove it (the best way)

$ sudo /etc/init.d/apparmor stop$ sudo update-rc.d -f apparmor remove$ sudo apt-get --purge remove apparmor apparmor-utils libapparmor-perl libapparmor1

It may cause the ubuntu-desktop crash, please reinstall it.

$ sudo apt-get install ubuntu-desktop

Find the all the pci devices in sam iommu group

We must binding all devices in same group to the vfio-pci driver creates.

$ readlink /sys/bus/pci/devices/0000\:01\:00.0/iommu_group../../../../kernel/iommu_groups/1$ ls -l /sys/bus/pci/devices/0000\:01\:00.0/iommu_group/devices

Bind vfio driver to PCI device

$ sudo su$ echo "8086 10fb" > /sys/bus/pci/drivers/vfio-pci/new_id$ echo "0000:01:00.1" > /sys/bus/pci/devices/0000\:01\:00.1/driver/unbind$ echo "0000:01:00.1" > /sys/bus/pci/drivers/vfio-pci/bind$ echo "0000:01:00.0" > /sys/bus/pci/devices/0000\:01\:00.0/driver/unbind$ echo "0000:01:00.0" > /sys/bus/pci/drivers/vfio-pci/bind

Start attach PCI device to guest and start

$ echo "
      
       
         
     
  
    " > new-pci.xml$ virsh attach-device guest-name --config new-pci.xml$ virsh start guest-name

Configure for igb module for SR-IOV support.

Persistent configure for igb module.

$ sudo echo 'options igb max_vfs=7' > /etc/modprobe.d/igb.conf$ sudo echo "igb" >> /etc/modules$ sudo update-initramfs -k all -t -u

Transient configure for igb module.

$ sudo su$ sudo rmmod igb && sudo modprobe igb max_vfs=7

It maill failed as follow on PC:

Jan 10 04:42:18 localhost kernel: [ 8441.914658] ixgbe 0000:01:00.1: SR-IOV: bus number out of rangeJan 10 04:42:18 localhost kernel: [ 8441.914662] ixgbe 0000:01:00.1 0000:01:00.1 (uninitialized): Failed to enable PCI sriov: -12

This is intel official document, it is some different from the above.

Work around if pci-passthrough can not work.

 Upgrade your kernel.

Download the latest stable kernel.

$ wget $ tar -xvf linux-3.18.1.tar.xz$ cd linux-3.18.1$ cp /boot/config-3.13.0-43-generic ./.config$ echo "" | make oldconfig

Please enable CONFIG_IP_NF_NAT the config, kernel 3.17 change the macro define for nat.

$ sed -i -e "s/# CONFIG_IP_NF_NAT.*/CONFIG_IP_NF_NAT=m/" ./.config$ make -j8$ make modules_install$ make install$ sudo reboot

 compile your owen qemu

$ git clone git://git.qemu.org/qemu.git$ apt-get install libglib2.0-dev zlib1g-dev libspice-server-dev libspice-protocol-dev$ ./configure '--target-list=x86_64-softmmu' '--enable-debug' '--enable-kvm' '--enable-spice' '--prefix=/home/shhfeng/qemu/'$ make

 

 re make you initrd

$ gunzip # uncompress your initrd-*.img file $ mkdir tmp$ cd tmp/$ cpio -i --make-directories < "your uncompress  initrd direction."$ tree -C .

# tree: option -C color show，-n shutdown color show.

or

$ cd tmp/$ gzip -dc ../sth.gz |cpio -id

# edit your initrd，

$ cd tmp/

# add or remove *.ko

$ find ./ | cpio -H newc -o > /boot/initrd-2.6.22.6.img-ahci$ gzip /boot/initrd-2.6.22.6.img-ahci$ mv initrd-2.6.22.6.img-ahci.gz /boot/initrd-2.6.22.6.img-ahci

 Build up your skills to develop Open stack

 Doc In Repository

Please read the docs in /opt/stack/nova/doc/source/devref first. Then you will get some ideas about Open Stack. Read up these files, you can ignore them if you already know them.

• gerrit.rst

It introduce the gerrit work flow.

• development.environment.rst

It introduce how to set up development environment.

• architecture.rst

Introduce the profile of the Open Stack.

• rpc.rst

Open Stack rpc is different with the traditional SUN IPC, it base on a AMQP broker.

Learn more about implementation of rpc. Please seen the Open Stack Common Library.

$ git clone git://git.openstack.org/openstack/oslo.git

Also learn more about kombu:

 

• threading.rst

All OpenStack services use *green thread* model of threading, If your know coroutine, you will learn greenthread quickly.

All the above files also can be access by this url link.

import component

You had better do some exercise about some technologies that Open Stack uses. This is helpful for you to know Open Stack.

• Rabbitmq python tutorials

Tutorial:

(a Chinese Document for Rabbitmq) (a Chinese Simple Document for Rabbitmq)

Tutorial Source Code:

• sqlalchemy ()

Tutorial Source Code:

• WSGI And eventlet

Read the Red hat REST API Guide. It is useful to understand REST API.

Join The Open Stack Community to do Contributions

• Join the IRC

IRC:

• Subscribe the mail-list

Dev:

General:

• To submit the patch

Workflow:

Play with Open Stack

DashBoard

Try to access

 CURL

I have write a small script to get token ID, as follow, you can name it as “get-token”:

#!/bin/bashreturn=`curl -i ' -X POST -H "Content-Type: application/json" -H "Accept: application/json"  -d '{"auth": {"tenantName": "admin", "passwordCredentials": {"username": "admin", "password": "123"}}}'`echo $returnecho "-----------------------------------------------------------------------------"ID=""for x in $return; do    if [ "x$ID" = "xtokenID" ] ;then         id=`echo $x | tr -d '[",]'`        export TOKEN=$id        echo "TOKEN=$id"        ID=""        break    fi    if [ "x$x" = 'x"id":' ] ;then         ID="tokenID"    fidoneecho ""

This script just is a sample, you can improve it.

Run it by

$ . get-token

Then you can run other commands use the TOKEN environment value, such as get all the tenents, as follow:

$ curl -i -X GET  -H "User-Agent: python-keystoneclient" -H "X-Auth-Token: $TOKEN"

 Open Stack Client

You should export the OS_USERNAME and OS_TENANT_NAME environment by the follow command firstly.

$ cd devstack/$ . openrc OS_USERNAME OS_TENANT_NAME

update your /etc/nova/nova.conf

Add the follow option to “[DEFAULT]” section.

$ grep -e DEFAULT -e pci_passthrough -e pci_alias -e scheduler_available_filters /etc/nova/nova.conf [DEFAULT]pci_passthrough_whitelist={"product_id":"10fb", "vendor_id":"8086", "deviceids":"0000:01:00.0, 0000:01:00.1"}pci_passthrough_whitelist={"product_id":"8c20", "vendor_id":"8086", "deviceids":"0000:00:1b.0"}pci_alias = {"vendor_id":"8086", "product_id":"10fb", "name":"intelnic"}scheduler_available_filters = nova.scheduler.filters.pci_passthrough_filter.PciPassthroughFilter

Create a instance with pci-passthrough device.

$ nova flavor-list$ nova flavor-key m1.tiny set pci_passthrough:alias=intelnic:1$ nova boot  pci  --flavor m1.tiny  --image=cirros-0.3.2-x86_64-uec

Debug with Open Stack

I usually use “print” for debugging. The “print” is on the stack screen terminal.

$ cd devstack/$ ./rejoin-stack.sh

Or

$ screen –lsThere is a screen on:	13846.stack$ screen -r stack

Some useful operation:

run a command by “Ctrl + A + :”,  you can input help command to show more info.Swith to any window by “Ctrl + A + '  ”, input the windows number you want to switch.Swith to any window by “Ctrl + A + "  ”, select the windows you want to switch.

write testcase