Project 10 - Hypervisor data collection

Primary mentor: Brian Hay (US)
Student: Brandon Marken

Project Overview:
In this Project I will port existing VIX tools from the Xen Hypervisor to run on the KVM Hypervisor. To do this I need to develop a kernel module which will allow us to access functions for KVM. This module will allow us to use the functions to pause/resume a guest, connect/disconnect from a guest, and access memory directly. While libvirt allows us the ability to connect and disconnect as well as pause and resume it does not allow us direct access to guest memory. I can then use this module to convert the existing toolset to work on KVM.

Next I will develop a tool to dynamically determine the kernel offsets for the guest OS. To do this I will look directly at the portion kernel and look for patterns that can be recognized as kernel structs. These patterns can then be used to determine the locations of the kernel structs in arbitrary kernels.

The last step will be to develop binary distributions of the VIX tool set in a .deb and .rpm format. This will increase the usability of the toolset.
Project Plan:
Version of Vix for KVM - June 20
Tool to dynamically determine locations of kernel structs - July 20.
Testing completed on both tools - August 1.
Documentation improvements - August 8
Packaged software available - August 13

Updates:
Attempting to use the libvirt virDomainMemoryPeek function to get information off the guest systems. (5-30-11)
Altering libvirt to debug the memory return issues.
VirDomainMemoryPeek successfully reads memory. (june-5-11)
Moving libvirt functions into actual vix tools (june-5-11)
Vix tools compiling with libvirt (june-13-11)

Currently debugging tools for June 20 deadline (june-13-11)
Still debugging vix tools. Currently tracking (june-27-11)
down a stack overflow error.

Fixed segfault. Currently addressing issues with correct results (July -4-11)
Still working on fixing address issues. I'm no longer 100% convinced (July-11-11)
that libvirt's virDomainMemoryPeek works as it advertised

Has virDomainMemoryPeek getting memory correctly now (July-13-11)
Attempting to reproduce the issue make appropriate documentation.
Issue and solution explained below *

Has vt-date working now. Needs to fix handling for programs which require multiple pages such as vt-ps (July-18-2011).

Fixed a memory allocation issue involving multiple multiple pages working on some other memory issues now. (July-25-11)

vt-date, vt-ps, vt-lsof, vt-lsmod, vt-ifconfig now working. Next goal is to finish up vt-dump-pmem (july-31-11)

Still working on vt-dump-pmem. It currently works in kernel space but not in user space (8-9-11)
Issues:
virDomainMemoryPeek function returns all zeros in the return buffer rather than the memory expected.
Running into a huge design decision on how to do the release for the final deliverable. We have to include the newest version of libvirt which at present doesn't come with many Linux distributions. This means we may have to include a version of libvirt with our final product.

Tools were significantly less hypervisor agnostic than they appeared at first. This has caused the transition to take longer than I had anticipated.

Currently trying to eliminate a stack overflow error which occurs in our page mapping. This is causing the vix-tools to segfualt when they attempt to map guest memory.

Currently all VIX-tools are running without segfaulting and are giving results though they are not necessarily correct or consistantly the same results.

With my new update issues I'm not 100% sure that virDomainMemoryPeek works as claimed. In the default version used in Ubuntu Linux returns all 0s when a memory grab is made (indicating its not actually reading memory). In fedora teh default version returns an error. So I compiled my own version from the snapshot and ran a test and it pulled information out of memory so I began to incorporate it into the VIX toolset. I have been unable to get the tools to return correct or consistent results. I set up a simple program to connect to a domain and make a memory grab to get the xtime variable.
xtime has a known address Located in the Systemmap inside the virtual machine and contains the time stamp for the machine. The program has not returned the correct time (indicating it is sometime in the 1970s though the result varies quite a bit. ) My task over the next week will be to run some more tests to determine whether this function is working properly or if its just a problem with this test.

* During a routine update of my system did the command apt-get autoremove. This command removes any packages in the system which are not in use as a tool or not referred to by another program. This removed libxenstore.so.0.3.0. As a result the libvirt daemon would not as by default libvirt builds with support for ESX, Xen, KVM and all the other hypervisors it supports. Upon rebuilding the library with the option --with-xen=no to disable support for the xen hypervisor I was able to run daemon again with no issue. On a whim I decided run my test program again to determine the system time from a manual memory grab of that page. The result gave the correct UNIX timestamp indicating the problem occurring before has been resolved.


Having a working memory function I was able to get vt-date to work. However the current problem I'm running into (which I anticipate I will finish in the next day or so ) is that the Xen memory mapping functions and the Libvirt functions work significantly differently. As a result there are some differences in the way you need to handle memory. So I handle them pretty well with 1 page but not with multiple pages.

Still having issues getting memory out of user space I'm not entirely sure why.