asm: introduce a new x64 assembler (#10110)

* asm: add initial infrastructure for an external assembler

This change adds some initial logic implementing an external assembler
for Cranelift's x64 backend, as proposed in RFC [#41].

This adds two crates:
- the `cranelift/assembler/meta` crate defines the instructions; to
  print out the defined instructions use `cargo run -p
  cranelift-assembler-meta`
- the `cranelift/assembler` crate exposes the generated Rust code for
  those instructions; to see the path to the generated code use `cargo
  run -p cranelift-assembler`

The assembler itself is straight-forward enough (modulo the code
generation, of course); its integration into `cranelift-codegen` is what
is most tricky about this change. Instructions that we will emit in the
new assembler are contained in the `Inst::External` variant. This
unfortunately increases the memory size of `Inst`, but only temporarily
if we end up removing the extra `enum` indirection by adopting the new
assembler wholesale. Another integration point is ISLE: we generate ISLE
definitions and a Rust helper macro to make the external assembler
instructions accessible to ISLE lowering.

This change introduces some duplication: the encoding logic (e.g. for
REX instructions) currently lives both in `cranelift-codegen` and the
new assembler crate. The `Formatter` logic for the assembler `meta`
crate is quite similar to the other `meta` crate. This minimal
duplication felt worth the additional safety provided by the new
assembler.

The `cranelift-assembler` crate is fuzzable (see the `README.md`). It
will generate instructions with randomized operands and compare their
encoding and pretty-printed string to a known-good disassembler,
currently `capstone`. This gives us confidence we previously didn't have
regarding emission. In the future, we may want to think through how to
fuzz (or otherwise check) the integration between `cranelift-codegen`
and this new assembler level.

[#41]: bytecodealliance/rfcs#41

* asm: bless Cranelift file tests

Using the new assembler's pretty-printing results in slightly different
disassembly of compiled CLIF. This is because the assembler matches a
certain configuration of `capstone`, causing the following obvious
differences:

- instructions with only two operands only print two operands; the
  original `MInst` instructions separate out the read-write operand into
  two separate operands (SSA-like)
- the original instructions have some space padding after the
  instruction mnemonic, those from the new assembler do not

This change uses the slightly new style as-is, but this is open for
debate; we can change the configuration of `capstone` that we fuzz
against. My only preferences would be to (1) retain some way to visually
distinguish the new assembler instructions in the disassembly
(temporarily, for debugging) and (2) eventually transition to
pretty-printing instructions in Intel-style (`rw, r`) instead of the
current (`r, rw`).

* ci: skip formatting when `rustfmt` not present

Though it is likely that `rustfmt` is present in a Rust environment,
some CI tasks do not have this tool installed. To handle this case
(plus the chance that other Wasmtime builds are similar), this change
skips formatting with a `stderr` warning when `rustfmt` fails.

* vet: audit `arbtest` for use as a dev-dependency

* ci: make assembler crates publishable

In order to satisfy `ci/publish.rs`, it would appear that we need to use
a version that matches the rest of the Cranelift crates.

* review: use Cargo workspace values

* review: document `Inst`, move `Inst::name`

* review: clarify 'earlier' doc comment

* review: document multi-byte opcodes

* review: document `Rex` builder methods

* review: document encoding rules

* review: clarify 'bits' -> 'width'

* review: clarify confusing legacy prefixes

* review: tweak IA-32e language

* review: expand documentation for format

* review: move feature list closer to enum

* review: add a TODO to remove AT&T operand ordering

* review: move prefix emission to separate lines

* review: add testing note

* review: fix incomplete sentence

* review: rename `MinusRsp` to `NonRspGpr`

* review: add TODO for commented out instructions

* review: add conservative down-conversion to `is_imm*`

* Fuzzing updates for cranelift-assembler-x64 (#10)

* Fuzzing updates for cranelift-assembler-x64

* Ensure fuzzers build on CI
* Move fuzz crate into the main workspace
* Move `fuzz.rs` support code directly into fuzzer
* Move `capstone` dependency into the fuzzer

* Make `arbitrary` an optional dependency

Shuffle around a few things in a few locations for this.

* vet: skip audit for `cranelift-assembler-x64-fuzz`

Co-authored-by: Alex Crichton <alex@alexcrichton.com>

* review: use 32-bit form for 8-bit and 16-bit reg-reg

Cranelift's existing lowering for 8-bit and 16-bit reg-reg `AND` used
the wider version of the instruction--the 32-bit reg-reg `AND`. As
pointed out by @cfallin [here], this was likely due to avoid partial
register stalls. This change keeps that lowering by distinguishing more
precisely between `GprMemImm` that are in register or memory.

[here]: #10110 (comment)

* fix: skip `rustfmt` on generated code in more cases

Apparently `rustfmt` is not found on the `x86_64-unknown-illumos` build.
This change skips the action in this new case.

prtest:full

* fix: feed Cargo the meta crate version

This fixes errors with the `publish.rs` script.

prtest:full

---------

Co-authored-by: Alex Crichton <alex@alexcrichton.com>

Author: abrown

.github/workflows/main.yml

@@ -634,6 +634,7 @@ jobs:
     - run: cargo fuzz build --dev
     - run: cargo fuzz build --dev --fuzz-dir ./cranelift/isle/fuzz
     - run: cargo fuzz build --dev --fuzz-dir ./crates/environ/fuzz --features component-model
+    - run: cargo fuzz build --dev --fuzz-dir ./cranelift/assembler-x64/fuzz
 
     # common logic to cancel the entire run if this job fails
     - uses: ./.github/actions/cancel-on-failure

Cargo.lock

@@ -139,6 +139,7 @@ opt-level = 0
 resolver = '2'
 members = [
   "cranelift",
+  "cranelift/assembler-x64/fuzz",
   "cranelift/isle/fuzz",
   "cranelift/isle/islec",
   "cranelift/isle/veri/veri_engine",

Cargo.toml

@@ -0,0 +1,26 @@
+[package]
+name = "cranelift-assembler-x64"
+description = "A Cranelift-specific x64 assembler"
+version = "0.117.0"
+license = "Apache-2.0 WITH LLVM-exception"
+edition.workspace = true
+rust-version.workspace = true
+
+[dependencies]
+arbitrary = { workspace = true, features = ["derive"], optional = true }
+
+[dev-dependencies]
+arbtest = "0.3.1"
+
+[build-dependencies]
+cranelift-assembler-x64-meta = { path = "meta", version = "0.117.0" }
+
+[lints.clippy]
+all = "deny"
+pedantic = "warn"
+module_name_repetitions = { level = "allow", priority = 1 }
+similar_names = { level = "allow", priority = 1 }
+wildcard_imports = { level = "allow", priority = 1 }
+
+[features]
+arbitrary = ['dep:arbitrary']

cranelift/assembler-x64/Cargo.toml

@@ -0,0 +1,36 @@
+# `cranelift-assembler-x64`
+
+A Cranelift-specific x64 assembler. Unlike the existing `cranelift-codegen`
+assembler, this assembler uses instructions, not instruction classes, as the
+core abstraction.
+
+### Use
+
+Like `cranelift-codegen`, using this assembler starts with `enum Inst`. For
+convenience, a `main.rs` script prints the path to this generated code:
+
+```console
+$ cat $(cargo run)
+#[derive(arbitrary::Arbitrary, Debug)]
+pub enum Inst {
+    andb_i(andb_i),
+    andw_i(andw_i),
+    andl_i(andl_i),
+    ...
+```
+
+### Test
+
+In order to check that this assembler emits correct machine code, we fuzz it
+against a known-good disassembler. We can run a quick, one-second check:
+
+```console
+$ cargo test -- --nocapture
+```
+
+Or we can run the fuzzer indefinitely:
+
+```console
+$ cargo +nightly fuzz run -s none roundtrip -j16
+```
+

cranelift/assembler-x64/README.md

@@ -0,0 +1,24 @@
+use cranelift_assembler_x64_meta as meta;
+use std::env;
+use std::path::Path;
+
+fn main() {
+    println!("cargo:rerun-if-changed=build.rs");
+
+    let out_dir = env::var("OUT_DIR").expect("The OUT_DIR environment variable must be set");
+    let out_dir = Path::new(&out_dir);
+    let built_files = [
+        meta::generate_rust_assembler(out_dir.join("assembler.rs")),
+        meta::generate_isle_macro(out_dir.join("assembler-isle-macro.rs")),
+        meta::generate_isle_definitions(out_dir.join("assembler-definitions.isle")),
+    ];
+
+    println!(
+        "cargo:rustc-env=ASSEMBLER_BUILT_FILES={}",
+        built_files
+            .iter()
+            .map(|p| p.to_string_lossy().to_string())
+            .collect::<Vec<_>>()
+            .join(":")
+    );
+}

cranelift/assembler-x64/build.rs

@@ -0,0 +1,4 @@
+target
+corpus
+artifacts
+coverage

cranelift/assembler-x64/fuzz/.gitignore

@@ -0,0 +1,22 @@
+[package]
+name = "cranelift-assembler-x64-fuzz"
+version = "0.0.0"
+publish = false
+edition.workspace = true
+rust-version.workspace = true
+
+[package.metadata]
+cargo-fuzz = true
+
+[dependencies]
+libfuzzer-sys = { workspace = true }
+cranelift-assembler-x64 = { path = "..", features = ['arbitrary'] }
+capstone = { workspace = true }
+arbitrary = { workspace = true, features = ['derive'] }
+
+[[bin]]
+name = "roundtrip"
+path = "fuzz_targets/roundtrip.rs"
+test = false
+doc = false
+bench = false