Creating A DLL With Rust¶

In this four-part, project-based series I'll show you how to:

Create a Windows dynamic-link library (DLL) using Rust
Inject DLLs into processes using Process Hacker
Inject DLLs into processes using Rust
Create application windows using Rust
Override a window's behavior using subclassing

By the end of the series, you will build an injectable DLL that let's you "full-screen" a browser window without it taking up your entire screen:

Pretty cool, eh?

Background¶

My interest in writing DLLs started, as with many things, as an attempt to scratch an itch.

See, I have an ultrawide, and the problem with all ultrawides, is window management; more specifically, the lack of sane window management.

Instead of emulating a dual- or triple- monitor setup, operating systems treat your monitor as ONE BIGASS MONITOR, which I guess, well, yeah, who would have thought...

Anyways, after evaluating many, many, many potential solutions, I landed on Microsoft's very own Fancy Zones, perfect in all ways, apart from one, when you "Full Screen" a YouTube video, the zone boundary is broken and the video consumes the entire screen.

After a few weeks of hacking, I was able to cobble together a C++-based solution which utilized DLL-injection: peddamat/PowerToys

This article series will walk through the development of an identical solution utilizing Rust.

What Are DLLs?¶

Microsoft describes them as:

... a kind of executable file that acts as a shared library of functions and resources....

More importantly:

... (they) run in the context of the applications that call them. The operating system loads the DLL into an application's memory space. It's done either when the application is loaded (implicit linking), or on demand at runtime (explicit linking)...

And an added bonus:

Multiple applications can access the contents of a single copy of a DLL in memory at the same time.

Key points being, DLLs are:

similar to .exe's
loaded into an application's memory space
shared between multiple applications

Compiling Your First DLL¶

hello.dll¶

Let's begin by creating a new library crate called hello:

$ cargo new hello --lib

This should feel familiar, the --lib bit lets cargo know we want a library crate, which results in:

$ tree
.
├── Cargo.toml
└── src
    └── lib.rs

Cargo.toml feels familiar…

Cargo.toml

[package]
name = "hello"
version = "0.1.0"
edition = "2021"

[dependencies]

… and lib.rs contains the library equivalent of a "Hello World":

lib.rs

pub fn add(left: usize, right: usize) -> usize {
    left + right
}

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn it_works() {
        let result = add(2, 2);
        assert_eq!(result, 4);
    }
}

To build the crate, use:

$ cd hello
$ cargo build

Which produces the following:

$ tree
...
└── target
    └── debug
        ...
        └── libhello.rlib

Unfortunately, libhello.rlib isn’t a .dll. It’s actually something called a 'Rust library', intended for internal consumption within and between Rust programs.

To produce a .dll, we must add a crate-type specifier to Cargo.toml:

Cargo.toml

[package]
name = "hello"
version = "0.1.0"
edition = "2021"

[lib]
crate-type = ["cdylib"]

[dependencies]

Building the project now results in:

$ cargo build
   Compiling hello v0.1.0 (C:\Users\me\source\blog_qa\hello)
    Finished dev [unoptimized + debuginfo] target(s) in 0.66s

$ tree
.
└── target
    └── debug
        ...
        └── hello.dll

Hooray!

Wait, what?¶

The crate-type attribute allows us to specify what sort of artifacts we want cargo to generate for us.

The full list can be found here, but a good summary is:

bin: ordinary executable.
lib: a library to be linked statically into Rust programs.
dylib: a library to be linked dynamically into Rust programs.
staticlib: a library to be linked statically into non-Rust programs.
cdylib: a library to be linked dynamically into non-Rust programs.
source

We use the cdylib crate type because our ultimate goal is to inject hello.dll into an existing Windows process, which, let's be honest, is likely non-Rust. However, feel free to try this project with using the dylib.

In addition to hello.dll, Rust produces a number of other artifacts:

*.d — Makefile-compatible dependency lists
*.rlib — Rust library files. Contain the compiled code of a dependency
build — Directories for build scripts to use as scratch space
deps — Your compiled dependencies
examples — Binaries from the examples directory
incremental — A directory for the incremental compilation cache
*-{hash} — Binaries from cargo test
executables — Your target binaries

You can read more about them: here

Using Your First DLL¶

hello_runner.exe¶

Let's put our fresh hello.dll to use by creating a new binary crate called hello_runner:

$ cd ..
$ cargo new hello_runner

Next, let's indicate a dependency on the hello crate by adding a line to the [dependencies] block in Cargo.toml:

Cargo.toml

[package]
name = "hello_runner"
version = "0.1.0"
edition = "2021"

[dependencies]
hello = { path = "../hello" }

Let's replace hello_runner's main.rs with something simple, just to make sure hello.dll is linked:

main.rs

fn main() {
    println!("2+2={}", add(2,2));
}

Oh look, rust-analyzer is already screaming at us:

It seems merely specifying the dependency on hello in Cargo.toml isn't enough for Rust to find add() in hello.dll.

So to tell Rust that there indeed exists an add() out there, somewhere, we can add an extern block to the top of main.rs:

main.rs

extern {
    fn add(left: usize, right: usize) -> usize;
}

rust-analyzer is still dissatisfied:

Following along carefully, we add an unsafe block, and...

extern {
    fn add(left: usize, right: usize) -> usize;
}

fn main() {
    unsafe {
        println!("2+2={}", add(2,2));
    }
}

... silence, perfect!

Doing a quick cargo build:

$ cargo build

warning: The package `hello` provides no linkable target. The compiler might raise an error while compiling `hello_runner`. Consider adding 'dylib' or 'rlib' to key `crate-type` in `hello`'s Cargo.toml. This warning might turn into a hard error in the future.
   Compiling hello v0.1.0 (C:\Users\me\source\blog_qa\hello)
   Compiling hello_runner v0.1.0 (C:\Users\me\source\blog_qa\hello_runner)
error: linking with `link.exe` failed: exit code: 1120
  |
...
  = note: hello_runner.3x2hoz24wh045sx8.rcgu.o : error LNK2019: unresolved external symbol add referenced in function _ZN12hello_runner4main17h5d530bb84e958f44E
          C:\Users\me\source\blog_qa\hello_runner\target\debug\deps\hello_runner.exe : fatal error LNK1120: 1 unresolved externals
error: could not compile `hello_runner` due to previous error

What fresh hell...

Ok, re-reading The Book's External Block section a bit more thoroughly, we see that we can specify a link attribute:

main.rs

#[link(name = "hello.dll", kind="dylib")]
extern {
    fn add(left: usize, right: usize) -> usize;
}

fn main() {
    unsafe {
        println!("2+2={}", add(2,2));
    }
}

cargo build:

$ cargo build
warning: The package `hello` provides no linkable target. The compiler might raise an error while compiling `hello_runner`. Consider adding 'dylib' or 'rlib' to key `crate-type` in `hello`'s Cargo.toml. This warning might turn into a hard error in the future.
   Compiling hello_runner v0.1.0 (C:\Users\me\source\blog_qa\hello_runner)
error: linking with `link.exe` failed: exit code: 1120
  |
...
  = note: hello_runner.3x2hoz24wh045sx8.rcgu.o : error LNK2019: unresolved external symbol __imp_add referenced in function _ZN12hello_runner4main17h5d530bb84e958f44E
          C:\Users\me\source\blog_qa\hello_runner\target\debug\deps\hello_runner.exe : fatal error LNK1120: 1 unresolved externals
error: could not compile `hello_runner` due to previous error

Weeps.

Ok, pulling out the goddamn cdylib RFC, which helpfully explains:

Symbol visibility - rdylibs will expose all symbols as rlibs do, cdylibs will expose symbols as executables do. This means that pub fn foo() {} will not be an exported symbol, but #[no_mangle] pub extern fn foo() {} will be an exported symbol.

Adding the #[no_mangle] pub extern to add():

main.rs

#[no_mangle]
pub extern fn add(left: usize, right: usize) -> usize {
    left + right
}

cargo build:

$ cargo build
warning: The package `hello` provides no linkable target. The compiler might raise an error while compiling `hello_runner`. Consider adding 'dylib' or 'rlib' to key `crate-type` in `hello`'s Cargo.toml. This warning might turn into a hard error in the future.
   Compiling hello v0.1.0 (C:\Users\me\source\blog_qa\hello)
   Compiling hello_runner v0.1.0 (C:\Users\me\source\blog_qa\hello_runner)
    Finished dev [unoptimized + debuginfo] target(s) in 0.80s

Thank you god. That warning is odd, though...

$ tree
.
└── target
    └── debug
        ...
        ├── hello_runner.exe
        ...

Let's see if hello_runner.exe runs using cargo run:

$ cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.01s
     Running `target\debug\hello_runner.exe`
2+2=4

Da dum!

Upon Further Examination...¶

Ok, thus far we've been using cargo run to execute hello_runner, but for shits and giggles, let's try running hello_runner.exe directly:

$ target\debug\hello_runner.exe

$

wat.

$ ls -l
build
deps
examples
hello_runner.d
hello_runner.exe
hello_runner.pdb
incremental

Ah... hello.dll isn't in the same directory as hello_runner.exe...

$ cp deps\hello.dll .
$ hello_runner.exe
2+2=4

Bingo. Ok, so that little exercise gives us a fairly good idea that hello_runner.exe has indeed “dynamically”-linked hello.dll, but how do we know for sure?

If you have Visual Studio installed, chances are you have a handy tool called dumpbin.exe, which is available using a "Developer Command Prompt for VS 2022".

View hello.dll's exported functions using dumpbin /exports:

$ dumpbin /exports hello.dll
Microsoft (R) COFF/PE Dumper Version 14.34.31942.0
Copyright (C) Microsoft Corporation.  All rights reserved.

Dump of file hello.dll

File Type: DLL

  Section contains the following exports for hello.dll

    00000000 characteristics
    FFFFFFFF time date stamp
        0.00 version
           1 ordinal base
           1 number of functions
           1 number of names

    ordinal hint RVA      name

          1    0 00001000 add = add

  Summary

        1000 .data
        1000 .pdata
        7000 .rdata
        1000 .reloc
       18000 .text

View hello_runner.exe's imported functions using dumpbin /imports:

$ dumpbin /imports hello_runner.exe |more
Microsoft (R) COFF/PE Dumper Version 14.34.31942.0
Copyright (C) Microsoft Corporation.  All rights reserved.


Dump of file hello_runner.exe

File Type: EXECUTABLE IMAGE

  Section contains the following imports:

    hello.dll
             14001E2A0 Import Address Table
             140026CC8 Import Name Table
                     0 time date stamp
                     0 Index of first forwarder reference

                           0 add

     KERNEL32.dll
  ...

For the more visually included, the venerable x64dbg or Dependencies can also be used to inspect hello_runner.exe:

Wait, what?¶

Eagle-eyed readers may have noticed the the kind="dylib" in the main.rs #[link... block:

main.rs

#[link(name = "hello.dll", kind="dylib")]
extern {
    fn add(left: usize, right: usize) -> usize;
}

The external blocks#link-attribute section explains :

dylib — Indicates a dynamic library. This is the default if kind is not specified.
static — Indicates a static library.
framework — Indicates a macOS framework. This is only valid for macOS targets.
raw-dylib — Indicates a dynamic library where the compiler will generate an import library to link against (see dylib versus raw-dylib below for details). This is only valid for Windows targets.

Pulling Bootstraps¶

As it currently stands, our humble hello.dll doesn't do much; essentially a glorified wastebasket at the beck and call of any old binary that links up with it.

Let's give our hello.dll a bit of agency by defining an entry-point.

DllMain¶

What is an entry-point? In short, whenever Windows loads a DLL, it checks to see if it exports a function named DllMain. If so, the operating system calls the function with a DLL_PROCESS_ATTACH or DLL_PROCESS_DETACH when attaching or detaching the DLL to processes.

Let's see how this works be adding a DllMain to hello.dll.

First, we'll need to add a few windows-rs crates to the hello crate's cargo.toml:

Cargo.toml

[dependencies.windows]
version = "0.*"
features = [
    "Win32_Foundation",
    "Win32_System_SystemServices",
    "Win32_UI_WindowsAndMessaging",
]

Next, we can add a barebones implementation of DllMain to the top of lib.rs:

lib.rs

use windows::{ Win32::Foundation::*, Win32::System::SystemServices::*, };

#[no_mangle]
#[allow(non_snake_case, unused_variables)]
extern "system" fn DllMain(
    dll_module: HINSTANCE,
    call_reason: u32,
    _: *mut ())
    -> bool
{
    match call_reason {
        DLL_PROCESS_ATTACH => (),
        DLL_PROCESS_DETACH => (),
        _ => ()
    }

    true
}

A quick cargo check in either /hello or /hello_runner shows that we're on the right path:

$ cargo
    Finished dev [unoptimized + debuginfo] target(s) in 0.04s

Let's have our DLL do something when it receives a DLL_PROCESS_ATTACH and DLL_PROCESS_DETACH:

lib.rs

use windows::{ Win32::Foundation::*, Win32::System::SystemServices::*, };
use windows::{ core::*, Win32::UI::WindowsAndMessaging::MessageBoxA, };

#[no_mangle]
#[allow(non_snake_case, unused_variables)]
extern "system" fn DllMain(
    dll_module: HINSTANCE,
    call_reason: u32,
    _: *mut ())
    -> bool
{
    match call_reason {
        DLL_PROCESS_ATTACH => attach(),
        DLL_PROCESS_DETACH => detach(),
        _ => ()
    }

    true
}

fn attach() {
    unsafe {
        // Create a message box
        MessageBoxA(HWND(0),
            s!("ZOMG!"),
            s!("hello.dll"),
            Default::default()
        );
    };
}

fn detach() {
    unsafe {
        // Create a message box
        MessageBoxA(HWND(0),
            s!("GOODBYE!"),
            s!("hello.dll"),
            Default::default()
        );
    };
}

Run the hello_runner crate using cargo run:

$ cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.06s
     Running `target\debug\hello_runner.exe`
2+2=4

and...

and then...

Boom!

Well... I'll admit that I may have been talking things a bit up.

That did feel a bit anti-climatic; theoretical, even; far too ivory tower to be satisfying.

Let's remedy this by slamming hello.dll into a random process using DLL injection.

Injecting The GD Thing Into Notepad.exe¶

So what is "DLL Injection"? Basically, it's leetspeek for using publically available Windows API calls to load a DLL into an unsuspecting application's memory space, with bonus points if you can actually execute code from the DLL in the application's memory space.

We will do both, but since the day grows old, we'll take the script kiddie route and use the handy "Process Hacker" tool.

But First...¶

Let's make a small tweak to the attach() function in lib.rs, because we really want to feel the next part in our veins.

Add this to the top of lib.rs:

lib.rs

use windows::Win32::System::Threading::GetCurrentProcessId;

Since we added that, we need to also update our [dependencies.windows] in Cargo.toml`:

Cargo.toml

[dependencies.windows]
version = "0.*"
features = [
    "Win32_Foundation",
    "Win32_System_SystemServices",
    "Win32_UI_WindowsAndMessaging",
    "Win32_System_Threading"
]

And make attach() look like this

lib.rs

fn attach() {
    unsafe {
        let pid = GetCurrentProcessId();

        MessageBoxA(HWND(0),
            PCSTR(std::format!("Hello from process: {}!\0", pid).as_ptr()),
            s!("hello.dll"),
            Default::default()
        );
    };
}

And finally:

$ cargo run
warning: The package `hello` provides no linkable target. The compiler might raise an error while compiling `hello_runner`. Consider adding 'dylib' or 'rlib' to key `crate-type` in `hello`'s Cargo.toml. This warning might turn into a hard error in the future.
   Compiling windows v0.44.0
   Compiling hello v0.1.0 (C:\Users\me\source\blog_qa\hello)
   Compiling hello_runner v0.1.0 (C:\Users\me\source\blog_qa\hello_runner)
    Finished dev [unoptimized + debuginfo] target(s) in 6.74s
     Running `target\debug\hello_runner.exe`

And we get our updated pop-up, good.

Process Hacker¶

Now, grab Process Hacker: here and launch the x64 version.

If you closed the previous pop-up, do another cargo run and search for "hello_runner.exe" in the little search bar near the top right of the window.

Compare the contents of the PID column with the number in the pop-up. For me, I get the same number, 33640.

Now hit Enter (or right-click -> Properties) to open the "Properties" panel:

The 'Threads' tab shows us that hello_runner.exe is using a single thread with a "Thread ID" (TID) of 18676.

Clicking on the 'Modules' tab shows us that hello.dll is indeed loaded into hello_runner.exe's address space, in fact, it's at 0x7ffb5e7d0000 for me:

All of this makes sense, since hello_runner.exe is literally loading hello.dll, right?

Well, I don't remember using LoadLibraryW anywhere... but we did slap #[link(name = "hello.dll", kind="dylib")] on that extern in hello_runner's main.rs.

Implicit vs Explicit linking¶

If we stroll down Memory Lane, recall:

... (they) run in the context of the applications that call them. The operating system loads the DLL into an application's memory space. It's done either when the application is loaded (implicit linking), or on demand at runtime (explicit linking)...

Specifically,

It's done either when the application is loaded (implicit linking), or on demand at runtime (explicit linking)...

So, though we didn't directly call LoadLibraryW to load hello.dll we did define a runtime dependency on it; we explicitly linked it.

Later on, when we actually do use LoadLibraryW, we'll be implicitly linking it. Something to keep in mind in case you never need that information.

Actually Injecting teh DLL¶

Ok, finally , open up a Notepad instance, and find it in Process Hacker:

Next, right click on the notepad.exe process and find Inject DLL under the Miscellaneous context menu.

Find hello.dll in the hello_runner\target\debug\deps folder and click "Open":

If you didn't have Notepad in the foreground, you may have to hunt for it...

But you will be rewarded with a...

You can confirm that hello.dll has indeed been loaded into notepad.exe's address space by opening up the Modules tab in Process Hacker's Properties panel.

Be sure to Unload hello.dll before you go:

Giving you a:

In general, it is always good practice to make sure your DLL is well behaved both loading and unloading.

Just A Little Further¶

Rust Workspaces¶

When injecting hello.dll above, you may noticed that there are actually two hello.dll binaries on our filesystem: one in \hello_runner\target\debug\deps and one in \hello\target\debug.

If you had the misfortune of trying to inject hello.dll from the latter, you'd have noticed that nothing happened. Why? Because it was a stale binary, left over from an earlier part of the tutorial when we were working in the hello crate context.

To reduce this confusion, Rust provides us with the workspace concept, which allows us to group dependent crates into a single... workspace.

First, let's do a bit of tidying, do a cargo clean from both the hello and hello_runner directories.

Now, in the parent folder, create a Cargo.toml containing:

Cargo.toml

[workspace]

members = [
    "hello",
    "hello_runner",
]

The directory should look like this:

$ tree -L 1 -a
.
├── Cargo.toml
├── hello
└── hello_runner

Do a cargo build:

$ cargo build
warning: The package `hello` provides no linkable target. The compiler might raise an error while compiling `hello_runner`. Consider adding 'dylib' or 'rlib' to key `crate-type` in `hello`'s Cargo.toml. This warning might turn into a hard error in the future.
   Compiling windows_x86_64_msvc v0.42.1
   Compiling windows-targets v0.42.1
   Compiling windows v0.44.0
   Compiling hello v0.1.0 (C:\Users\me\source\blog_qa\hello)
   Compiling hello_runner v0.1.0 (C:\Users\me\source\blog_qa\hello_runner)
    Finished dev [unoptimized + debuginfo] target(s) in 6.92s

Which results in:

$ tree -L 3 -a
.
├── Cargo.lock
├── Cargo.toml
├── hello
│   ├── .gitignore
│   ├── Cargo.lock
│   ├── Cargo.toml
│   └── src
│       └── lib.rs
├── hello_runner
│   ├── .gitignore
│   ├── Cargo.lock
│   ├── Cargo.toml
│   └── src
│       └── main.rs
└── target
    ...
    └── debug
        ...
        ├── hello.dll
        ├── hello_runner.exe
        ...

Notice that we now only have one target directory.

If you do a cargo run, you'll notice cargo is smart enough to execute hello_runner.exe:

$ cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.05s
     Running `target\debug\hello_runner.exe`
2+2=4

Lastly, if you're so inclined, do a git init and call it a night!

Part 2 is here: Creating A Window With Rust

Having trouble? The code this series can be found here: peddamat/how-to-create-a-dll-using-rust

Last update: 2023-03-16

Creating A DLL With Rust¶

Background¶

What Are DLLs?¶

Compiling Your First DLL¶

hello.dll¶

Wait, what?¶

Further Reading¶

Using Your First DLL¶

hello_runner.exe¶

Upon Further Examination...¶

Wait, what?¶

Further Reading¶

Pulling Bootstraps¶

DllMain¶

Further Reading¶

Injecting The GD Thing Into Notepad.exe¶

But First...¶

Process Hacker¶

Implicit vs Explicit linking¶

Actually Injecting teh DLL¶

Just A Little Further¶

Rust Workspaces¶