Process Virtualization … Can it Help?

Virtualization has been growing at an ever increasing rate over the past several years, now that nearly every processor in the market supports virtualization at the processor level. Of course there are the full blown hypervisor implementations such as Windows Hyper-V and VMWare which support running a complete OS within a virtual environment. But more recently there has been a push for micro-hypervisors which scale down the requirements and can run individual processes or process groups within a virtual environment. Here at Kernel Drivers we decided to take a deeper look at what is required and what advantages there could be by running processes within a virtual environment.

In some recent blog posts, I talked about the details of Process Isolation but from a non-virtualized environment perspective. What if we could run a process within a virtual environment to better isolate possible changes the process could effect on the operating system as a whole? As you may have heard, Microsoft recently released some forms of this in their Container technology. They currently offer 2 types of containers. First, there are Application level containers which sound a lot like what was previously discussed in the “Security Through Process Isolation” blog series. Second, they also offer a Virtualization Container where an application is run within a hypervisor with support of a scaled down kernel also running within this virtual environment. But what if you could separate out the user mode and kernel mode portions of the running process and run the user mode code within a virtual container and have full support of the non-virtualized kernel? What benefits could this design offer in terms of security and isolation?

One feature that this model would offer is a very clean interface to trap all the user to kernel transitions. In doing so one could implement an isolation layer that would be completely infallible. Imagine, every call made by the user mode process would be required to go through this security layer allowing for absolute control over how and what the process does. There would be no more isolation through file system, registry, object, etc. processing, it would be a single interface trapping all calls made by the process into the kernel. As an added benefit, the process running in the virtual environment wouldn’t be able to communicate directly with any other user mode process without going through this security layer.

This all sounds great but how can this be accomplished? How can a single process be executed within a virtual environment and somehow notify the micro-hypervisor that it is performing a kernel mode call? Well, before diving into the details of how to get this implemented, which we have achieved, let’s cover some of the basics about how hypervisors work and how they are able to control the processes running within the environment they create.

Over the next series of blog entries, I’ll be covering these details along with the problems encountered and how they were solved, in general, when trying to get a process to run in a virtual environment. Of course, some of what we’ll be talking about is OS independent and more focused on the support for Virtualization Technology, or VT, by the underlying processor. But a large part of the discussion will be how the OS runs a process from a fundamental perspective.