Nathan Gauer

Vulkan-ize Virglrenderer - experiment

Tue, 24 Jul 2018 00:00:00 +0000

Virglrenderer provides OpenGL acceleration to a guest running on QEMU.

My current GSoC project is to add support for the Vulkan API.

Vulkan is drastically different to OpenGL. Thus, this addition is not straight-forward. My current idea is to add an alternative path for Vulkan. Currently, two different states are kept, one for OpenGL, and one for Vulkan. Commands will either go to the OpenGL or Vulkan front-end.

For now, only compute shaders are supported. The work is divided in two parts: a Vulkan ICD in MESA, and a new front-end for Virgl and vtest.

If you have any feedback, do not hesitate !

This experiment can be tested using this repository. If you have an Intel driver in use, you might be able to use the Dockerfile provided.

Each part is also available independently:

GSoC 2017 - 3D acceleration using VirtIOGPU

Sun, 27 Aug 2017 00:00:00 +0000

Several months ago started the GSoC 2017. Among all the projects available one got my attention:

Add OpenGL support on a Windows guest using VirGL

In a VM, to access real hardware, we have two methods: passthrough, and virtualization extensions (Intel VT-x, AMD-V..). When it comes to GPUs possibilities drop down to one: passtrough. Intel has a virtualization extension (GVT), but we want to support every devices. Thus, we need to fall-back to a software based method.

Emulation ? Since we want 3D acceleration, better forget it
API-forwarding ? This means we need to have the same OpenGL API between guest host, also no.
Paravirtualization ? Yes !

Since a couple years, VirtIO devices became a good standard on QEMU. Then, Dave Airlie started to work on VirGL and a VirtIO-gpu. Both help provide a descent virtual-GPU which rely on the host graphic stack.

This article will present VirtIO devices, and what kind of operations a guest can do using VirGL.

I also invite you to read a previous article I wrote about Linux’s graphic stack

VirtIO devices

Since we will use a VirtIO based device, let’s see how it works. First, these devices behave as regular PCI devices. We have a config space, some dedicated memory, and interruptions. Second very important point, VirtIO devices communicate with ring-buffers used as FIFO queues. This device is entirely emulated in QEMU, and can realize DMA transfers by sharing common pages between the guest and the host.

Communication queues

On our v-gpu, we have 2 queues. One dedicated to the hardware cursor, and another for everything else. To send a command in the queue, it goes like this:

guest: allocate pages on the host
guest: send a header and pointers to our physical pages (guest POV) in the ring buffer.
guest: send an interruption
VMExit
host: QEMU read our header and pointers. Translate addresses to match local virtual address range.
host: read the command, execute it
host: write back to ring buffer
host: send interruption
guest: handle interruption, read ring buffer and handle answer

VirGL

VirGL can be summed up as a simple state-machine, keeping track of resources, and translating command buffers to a sequence of OpenGL calls. It exposes two kinds of commands: let’s say 2D and 3D.

2D commands are mainly focused on resources management. We can allocate memory on the host by creating a 2D resource. Then initialize a DMA transfer by linking this resource’s memory areas to guest’s physical pages. To ease resource management between applications on the guest, VirGL also adds a simple context feature. Resource creation is global, but to use them, you must attach them to the context.

Then, 3D commands. These are close to what we can find in a API like Vulkan. We can setup a viewport, scissor state, create a VBO, and draw it. Shaders are also supported, but we first need to translate them to TGSI; an assembly-like representation. Once on the host, they will be re-translated to GLSL and sent to OpenGL.

You can find a part of the spec on this repository

OpenGL on Windows

Windows graphic stack can be decomposed as follows:

Interresting parts are:

OpenGL ICD (Installable client driver):

This is our OpenGL implementation -> the state machine, which can speak to our kernel driver.

GDI.dll:

A simple syscall wrapper for us.

D3D Subsystem:

First part of the kernel graphic stack. It exposes a 3D and 2D API. Since we are not a licensed developer, let’s try to avoid this. From the documentation, we have a some functions to bypass it: DxgkDdiEscape is one. This functions takes a buffer, a size, and lets it pass trough this subsystem, directly to the underlying driver.

DOD (Display Only Driver)

Our kernel driver. This part will have to communicate to both kernel/ICD and VirtIO-gpu.

OpenGL State-Tracker

OpenGL rely on a state machine we have to implement. Let’s start by drawing on the frame-buffer.

We start a new application, want to split it from the rest. So we start by creating a VirGL context. Then create a 2D resource (800x600 RGBA seams great), and attach it to our VGL-context.

We might want to draw something now. We have two options, either use the 3D command INLINE_WRITE, or DMA. Using INLINE_WRITE means sending all our pixels through a VirtIO queue. So let’s use DMA !

We start by allocating our memory pages on the guest.
Then, send physical addresses to VirGL (guest POV)
VirGL will translate PA addresses to local virtual addresses, and link these pages to our resource.
Back to the guest, we can write our pixels to the frame-buffer.
To notify the V-gpu, we use the TRANSFER_TO_HOST_2D command, which tells QEMU to sync resources.

Now, let’s draw some pixels on this frame-buffer. We will need :

create an OpenGL context
setup our viewport and scissor settings (ie: screen limits)
create a VBO
link the VBO to a vertex/normals/color buffer
create vertex and frag shaders
setup a rasterizer
setup the frame-buffer to use the one we created earlier
create a constant buffer
send the draw call

A 3D Command is a set of UINT32. The first one is used as a header, followed by N arguments. A command buffer can contains several commands stacked together in one big UINT32 array.

Earlier, we created resources in VGL-Contexts. Now we will need 3D objects. These are created sending 3D commands, and are not shared between VGL contexts. Once created, we have to bind them to the current opengl-context.

Now, if everything goes well, we should be able to display something like that:

Once more, explaining all the commands would be uninteresting, but there is a spec for that !

If you are still interested, here are couple of links:

GIST to present the project
DOD Driver: The kernel driver needed on the Windows guest to communicate with the VirtIO-gpu
ICD Driver: opengl32.dll, the userland driver including a basic state-tracker:
VirGL Reference : partial reference of VirGL 2D and 3D commands

GSoC 2017 - API Forwarding

Sat, 13 May 2017 00:00:00 +0000

Writing an ICD is a problem in itself. Add to this Windows kernel interfaces, virtIO queues management, resources transfer between host and guest, and BOOM, you are lost. This brings us to our first step: something not efficient but simpler, API Forwarding.

Tasks

Hook OpenGL calls
Serialize function calls
Send them to the miniport-driver, then the host
De-serialize calls and execute them on the host.
Send some data back to the guest

Realization

ICD

The ICD part (Userland) is pretty straightforward. Make your own opengl32.dll, serialize the calls. Now find a sweet function in gdi32.dll to throw your mumbo-jumbo on the kernel side. Fortunately, we have this:

NTSTATUS APIENTRY DxgkDdiEscape(
  _In_ const HANDLE         hAdapter,
  _In_ const DXGKARG_ESCAPE *pEscape
)
{ ... }

A beautiful function available on both DOD and full display driver. It takes a pointer on an userland buffer, and send it to our display driver. Wait… userland buffer, no check, kernel part ? Mmmmm…. What could go wrong ?

Kernel part

To initialize a display driver, you must call a function: DxgkInitialize This function will take a big structure, containing function pointers to your driver. For a display only driver, you will have a reduced set of function to implement. And for a full featured driver, well…

Anyway, now the game is to run the driver, and see where we crash. Sadly we cannot just hope to add some functions, and run only using the working DOD code base. Windows wants something more, and the game is to find what, Yay ! Since we have a working DOD driver, let’s find how we could trick.

ICD <=> Kernel communication

We can register two type of driver: a DOD driver using DxgkInitializeDisplayOnlyDriver and DxgkInitialize. Windows will then know which kind of features each driver can support (fine tune will be done using query callbacks). Both drivers can implement DxgkDdiEscape. Great, we will fool Windows and use this DOD as a fully featured 3D driver ! WRONG !

Setup of the ICD part, sending everything through our escape functions ? check. But return values seams off. After investigation, and any function taking a userland buffer, I came to a conclusion: OpenGL ICD part cannot communicate with a DOD driver. Windows knows we are display only, and fall-back our ICD calls on it’s own driver.

So now, what’s the plan ? Let’s put this problem aside, and try to focus on the real part: create proper commands for the host.

GSoC 2017 - Project presentation

Thu, 11 May 2017 00:00:00 +0000

On my arrival at the lab, I started a little project: working on a display only driver for Windows. A good way to start learning what was hidden under the hood of an OpenGL application. Google Summer of Code 2017 arrived, and subject were published. Among these, QEMU’s ‘Windows Virgl driver’. Great ! Let’s apply !

Applications closed early April. I took a look at the already existing DOD driver (non official repo)

and also decided to learn a bit more about Vulkan. Results came, and I was selected, excellent !

Mission

The idea is to bring 3d acceleration on Windows guests running with QEMU. Using VirtIO devices and Virgl3d.

Context

On this stack, we can work on three parts: opengl32.dll, ICD and Miniport driver.

OpenGL32.dll is just a dynamic library used to communicate with out runtime driver.
ICD: this is the OpenGL implementation. This part is the equivalent of Mesa on Linux.
Miniport-driver: this is the kernel driver. Hardware specific, we are going to do our hypercalls here.

Problems

Windows is not open source. We have some basic ideas about D3DKrnl subsystem behaviour, but nothing is certain.
To develop a complete OpenGL state tracker is a lot of work.
Virgl3D takes some calls, and bytecode for shaders, re-translate it to GLSL, and call OpenGL again. Which means we will do the same work twice. Once on the host, once on the guest.