Several months ago started the GSoC 2018. Once again, I found a project which
got my attention.
Vulkan-ize VirglRenderer
Virglrenderer is a library designed to provide
QEMU guests with OpenGL acceleration. It is composed of several components:
- a MESA driver, on the guest, which generates Virgl commands
- a lib, on the host, which takes virgl commands and generated OpenGL calls
from it.
If you want to read more: 3D Acceleration using VirtIO.
This library was built with OpenGL in mind. Today, Vulkan is correctly
supported, and is becoming a new standard. It might be time to bring Vulkan to
QEMU’s guests !
To do so, we will need to work on two components:
- A Vulkan ICD. Writting one for MESA sounds like a good idea.
- A Vulkan back-end for Virglrenderer.
Now, we can face the first issue: Vulkan is not designed with abstraction in
mind. The time of the old GlBegin/glVertex is kinda dead.
If we want to avoid any unnecessary abstraction, we cannot easily reduce the
amount of calls made to the API.
Thus, the vast majority of the VK calls will be forwarded to the host.
However, there is some area in which we can bend the rules a bit.
The rest of this post contains the same content as the announce email (virgl ML).
Project status
- Several Vulkan objects can be created
- Memory can be mapped and altered on the client.
- Changes are written/read to/from the server on flush/invalidation
- Basic features for command buffers are supported.
As a result, a sample compute shader can be ran, and the results can
be read.
I only use vtest for now. The client part lies in mesa/srv/virgl.
Current behavior
To compile virglrenderer with vulkan, the option –with-vulkan is needed.
Running the server as-is does not enable Vulkan. And for now, Vulkan
cannot be used in parallel with OpenGL (Issue #1).
To enable Vulkan, the environment variable VTEST_USE_VULKAN must be set.
Initialization:
The client driver is registered as a classic Vulkan ICD.
When the loader call icdNegociateLoaderICDInterfaceVersion, the driver
connects to the server.
On failure, the driver reports as an invalid driver.
Once connected, the ICD will fetch and cache all physical devices.
It will also fetch information about queue, memory and so.
Physical devices are then exposed as virtual-gpus.
Memory areas are showed as-is, except for the
VK_MEMORY_PROPERTY_HOST_COHERENT bit, which is disabled.
This forces the application to notify every modification made to a
mapped memory.
The object creation part relies heavily on API-Forwarding. For now, I
don’t see how I could avoid that.
Memory transfers
Once basic objects are created, the client will ask to map some
memory. For now, no clever thing is done.
The ICD will provide a buffer. On flush, a transfer command is issued.
Virglrenderer will then map the corresponding memory region,
write/read, and unmap it.
A memory manager could be used on the server in the future to avoid
mapping/unmapping regions each time a transfer occurs.
Commands and execution
Command pool creation is forwarded to the server. For now, a command
buffer is attached to its pool.
To retrieve a command buffer from a handle, I need to know from which
pool it came from. (Issue #2)
Command buffer creation is also forwarded to the server.
Command buffers state is managed on the client. Each vkCmd* call will
modify an internal state.
Once vkEndCommandBuffer is called, the state is sent to the server.
The server will then call corresponding vkCmd* functions to match
retrieved the state.
Code generation
Vulkan entry points are generated at compile time. Heavily inspired
from Intel’s entry-point generation.
However, since object creation relies on API-Forwarding, I started to
work on a code generator for these functions.
Using a json, the interesting informations are outlined. Then a Python
script will generate functions used to forward object creation to the
vtest pipe.
Even-though the Vulkan API seams pretty consistent, some specific
cases and time constraints forced me to abandon it.
This script is still available in the mesa/src/virgl/tools and
virglrenderer/tools folder, but is lacking features.
Also, since I had different needs on both sides of vtest, scripts
diverge a lot.
The most recent version is the Virglrenderer one. It’s a second
iteration, and it might be easier to work with.
In the current state, I use it to generate a skeleton for vtest
functions, and then fixes the implementation.
In the future, it could save us some time, especially if we use the
same protocol for VirtIO commands.
Issues
1: (Virglrenderer) Vulkan cannot be used next to OpenGL.
There is no reason for it except a badly though integration of the vulkan
initialization part into virglrenderer.
2: (Virglrenderer) Command buffers are scattered into several pools
Command buffers are scattered into several pools the client created.
To fetch a command buffer vk-handle, I need to
first fetch the corresponding pool from a logical device, then fetch
the command buffer.
Since VirtIO ant Vtest provides a FIFO, maybe we could drop the
command pool creation forwarding. Use only one pool per instance, and
thus simplify command buffers lookups.
3: (MESA) Vtest and VirtIO switch is not straightforward right now.
An
idea could be to add a level between vgl_vk* functions an vtest.
vgl_vk* function would still manage the state of the ICD.
the mid-layer would convert handles and payload to a common protocol
for both VirtIO and Vtest. (Both could use vgl handles and some metadata).
Then, a backend function, which would choose between vtest and virtio.
The handles could be either forwarded as-is (vtest case)
Or translated to real virgl handles in the case of a kernel driver
which could do a translation, or check them. But the metadata should
not change.
4: (Virglrenderer/MESA) vtest error handling is bad.
Each command sends a result payload, and optionally, data. This result payload
contains two informations. An error code, and a numerical value. Use as a
handle, or a size. On server failure, error-codes should be used.
5: bugs, bugs and bugs.
This project is absolutely NOT usable right now.
Next steps
My first step should be to rebase this project onto the current
virglrenderer version, and rewrite the history. In the mean time,
rewrite the initialization part to allow both OpenGL and Vulkan to be ran.
Then, fix the vtest/virtio architecture. Add this new mid-layer.
Once refactorized, I should work on the error handling for
client-server interactions.
Once in a sane state, other issues will have to be addressed.
How to test it
There is a main repo used to build and test it rapidly.
In it, a bash script and a dockerfile (+ readme, todo)
The bash script in itself should be enough. But if the compilation
fails for a reason, the dockerfile could be used.
The README provided should be enough to make the sample app run.
Repositories
