Add procedure HTTP request API for WASM modules and the Rust module bindings library (#3684)

# Description of Changes

Closes #3517 .

With this PR, procedures (at least, those defined in Rust modules) can
perform HTTP requests! This is performed through a new field on the
`ProcedureContext`, `http: HttpClient`, which has a method `send` for
sending an `http::Request`, as well as a convenience wrapper `get`.

Internally, these methods hit the `procedure_http_request` ABI call /
host function, which uses reqwest to perform an HTTP request. The
request is run with a user-configurable timeout which defaults and is
clamped to 500 ms.
Rather than exposing the HTTP stream to modules, we download the entire
response body immediately, within the same timeout.

I've added an example usage of `get` to `module-test` which performs a
request against `localhost:3000` to read its own schema/moduledef.

This PR also makes all procedure-related definitions in the Rust module
bindings library `#[cfg(feature = "unstable")]`, as per #3644 . The
rename of the `/v1/database/:name/procedure/:name` route is not included
in this PR, so this does not close #3644 .

Left as TODOs are:
- Metrics for recording request and response size.
- Improving performance by stashing a long-lived `reqwest::Client`
someplace.
  Currently we build a new `Client` for each request.
- Improving performance (possibly) by passing the request-future to the
global tokio executor
  rather than running it on the single-threaded database executor.

# API and ABI breaking changes

Adds new APIs, which are marked as unstable. Adds a new ABI, which is
not unstable in any meaningful way (we can't really do that). Marks
unreleased APIs as unstable. Does not affect any pre-existing
already-released APIs or ABIs.

# Expected complexity level and risk

3 or so: networking is scary, and even though we impose a timeout which
prevents these connections from being truly long-lived, they're still
potentially long-lived on the scale of Tokio futures. It's possible that
running them on the database core is problematic in some way, and so
what I've left as a performance TODO could actually be a
concurrency-correctness issue.

# Testing

- [x] Manually wrote and executed some procedures which make HTTP
requests.
- [x] Added two automated tests to the `sdk-test` suite,
`procedure::http_ok` and `procedure::http_err`, which make successful
and failing requests respectively, then return its result. A client then
makes some assertions about the result.

---------

Co-authored-by: Noa <coolreader18@gmail.com>

Author: gefjon

Cargo.lock

@@ -196,6 +196,7 @@ hex = "0.4.3"
 home = "0.5"
 hostname = "^0.3"
 http = "1.0"
+http-body-util= "0.1.3"
 humantime = "2.1.0"
 hyper = "1.0"
 hyper-util = { version = "0.1", features = ["tokio"] }

Cargo.toml

@@ -732,6 +732,47 @@ pub mod raw {
         ///   This currently does not happen as anonymous read transactions
         ///   are not exposed to modules.
         pub fn procedure_abort_mut_tx() -> u16;
+
+        /// Perform an HTTP request as specified by the buffer `request_ptr[..request_len]`,
+        /// suspending execution until the request is complete,
+        /// then return its response via a [`BytesSource`] written to `out`.
+        ///
+        /// `request_ptr[..request_len]` should store a BSATN-serialized `spacetimedb_lib::http::Request` object
+        /// containing the details of the request to be performed.
+        ///
+        /// If the request is successful, a [`BytesSource`] is written to `out`
+        /// containing a BSATN-encoded `spacetimedb_lib::http::Response` object.
+        /// "Successful" in this context includes any connection which results in any HTTP status code,
+        /// regardless of the specified meaning of that code.
+        ///
+        /// # Errors
+        ///
+        /// Returns an error:
+        ///
+        /// - `WOULD_BLOCK_TRANSACTION` if there is currently a transaction open.
+        ///   In this case, `out` is not written.
+        /// - `BSATN_DECODE_ERROR` if `request_ptr[..request_len]` does not contain
+        ///   a valid BSATN-serialized `spacetimedb_lib::http::Request` object.
+        ///   In this case, `out` is not written.
+        /// - `HTTP_ERROR` if an error occurs while executing the HTTP request.
+        ///   In this case, a [`BytesSource`] is written to `out`
+        ///   containing a BSATN-encoded `spacetimedb_lib::http::Error` object.
+        ///
+        /// # Traps
+        ///
+        /// Traps if:
+        ///
+        /// - `request_ptr` is NULL or `request_ptr[..request_len]` is not in bounds of WASM memory.
+        /// - `out` is NULL or `out[..size_of::<RowIter>()]` is not in bounds of WASM memory.
+        /// - `request_ptr[..request_len]` does not contain a valid BSATN-serialized `spacetimedb_lib::http::Request` object.
+        #[cfg(feature = "unstable")]
+        pub fn procedure_http_request(
+            request_ptr: *const u8,
+            request_len: u32,
+            body_ptr: *const u8,
+            body_len: u32,
+            out: *mut [BytesSource; 2],
+        ) -> u16;
     }
 
     /// What strategy does the database index use?
@@ -1397,4 +1438,42 @@ pub mod procedure {
     pub fn procedure_abort_mut_tx() -> Result<()> {
         call_no_ret(|| unsafe { raw::procedure_abort_mut_tx() })
     }
+
+    #[inline]
+    #[cfg(feature = "unstable")]
+    /// Perform an HTTP request as specified by `http_request_bsatn`,
+    /// suspending execution until the request is complete,
+    /// then return its response or error.
+    ///
+    /// `http_request_bsatn` should be a BSATN-serialized `spacetimedb_lib::http::Request`.
+    ///
+    /// If the request completes successfully,
+    /// this function returns `Ok(bytes)`, where `bytes` contains a BSATN-serialized `spacetimedb_lib::http::Response`.
+    /// All HTTP response codes are treated as successful for these purposes;
+    /// this method only returns an error if it is unable to produce any HTTP response whatsoever.
+    /// In that case, this function returns `Err(bytes)`, where `bytes` contains a BSATN-serialized `spacetimedb_lib::http::Error`.
+    pub fn http_request(
+        http_request_bsatn: &[u8],
+        body: &[u8],
+    ) -> Result<(raw::BytesSource, raw::BytesSource), raw::BytesSource> {
+        let mut out = [raw::BytesSource::INVALID; 2];
+
+        let res = unsafe {
+            super::raw::procedure_http_request(
+                http_request_bsatn.as_ptr(),
+                http_request_bsatn.len() as u32,
+                body.as_ptr(),
+                body.len() as u32,
+                &mut out as *mut [raw::BytesSource; 2],
+            )
+        };
+
+        match super::Errno::from_code(res) {
+            // Success: `out` is a `spacetimedb_lib::http::Response`.
+            None => Ok((out[0], out[1])),
+            // HTTP_ERROR: `out` is a `spacetimedb_lib::http::Error`.
+            Some(errno) if errno == super::Errno::HTTP_ERROR => Err(out[0]),
+            Some(errno) => panic!("{errno}"),
+        }
+    }
 }

crates/bindings-sys/src/lib.rs

@@ -26,7 +26,9 @@ spacetimedb-bindings-macro.workspace = true
 spacetimedb-primitives.workspace = true
 
 bytemuck.workspace = true
+bytes.workspace = true
 derive_more.workspace = true
+http.workspace = true
 log.workspace = true
 scoped-tls.workspace = true
 

crates/bindings/Cargo.toml

@@ -0,0 +1,210 @@
+//! Types and utilities for performing HTTP requests in [procedures](crate::procedure).
+//!
+//! Perform an HTTP request using methods on [`crate::ProcedureContext::http`],
+//! which is of type [`HttpClient`].
+//! The [`get`](HttpClient::get) helper can be used for simple `GET` requests,
+//! while [`send`](HttpClient::send) allows more complex requests with headers, bodies and other methods.
+
+use bytes::Bytes;
+pub use http::{Request, Response};
+pub use spacetimedb_lib::http::{Error, Timeout};
+
+use crate::{
+    rt::{read_bytes_source_as, read_bytes_source_into},
+    IterBuf,
+};
+use spacetimedb_lib::{bsatn, http as st_http};
+
+/// Allows performing HTTP requests via [`HttpClient::send`] and [`HttpClient::get`].
+///
+/// Access an `HttpClient` from within [procedures](crate::procedure)
+/// via [the `http` field of the `ProcedureContext`](crate::ProcedureContext::http).
+#[non_exhaustive]
+pub struct HttpClient {}
+
+impl HttpClient {
+    /// Send the HTTP request `request` and wait for its response.
+    ///
+    /// For simple `GET` requests with no headers, use [`HttpClient::get`] instead.
+    ///
+    /// Include a [`Timeout`] in the [`Request::extensions`] via [`http::request::RequestBuilder::extension`]
+    /// to impose a timeout on the request.
+    /// All HTTP requests in SpacetimeDB are subject to a maximum timeout of 500 milliseconds.
+    /// All other extensions in `request` are ignored.
+    ///
+    /// The returned [`Response`] may have a status code other than 200 OK.
+    /// Callers should inspect [`Response::status`] to handle errors returned from the remote server.
+    /// This method returns `Err(err)` only when a connection could not be initiated or was dropped,
+    /// e.g. due to DNS resolution failure or an unresponsive server.
+    ///
+    /// # Example
+    ///
+    /// Send a `POST` request with the header `Content-Type: text/plain`, a string body,
+    /// and a timeout of 100 milliseconds, then treat the response as a string and log it:
+    ///
+    /// ```norun
+    /// # use spacetimedb::{procedure, ProcedureContext};
+    /// # use spacetimedb::http::{Request, Timeout};
+    /// # use std::time::Duration;
+    /// # #[procedure]
+    /// # fn post_somewhere(ctx: &mut ProcedureContext) {
+    /// let request = Request::builder()
+    ///     .uri("https://some-remote-host.invalid/upload")
+    ///     .method("POST")
+    ///     .header("Content-Type", "text/plain")
+    ///     // Set a timeout of 100 ms, further restricting the default timeout.
+    ///     .extension(Timeout::from(Duration::from_millis(100)))
+    ///     .body("This is the body of the HTTP request")
+    ///     .expect("Building `Request` object failed");
+    ///
+    /// match ctx.http.send(request) {
+    ///     Err(err) => {
+    ///         log::error!("HTTP request failed: {err}");
+    ///     },
+    ///     Ok(response) => {
+    ///         let (parts, body) = response.into_parts();
+    ///         log::info!(
+    ///             "Got response with status {}, body {}",
+    ///             parts.status,
+    ///             body.into_string_lossy(),
+    ///         );
+    ///     }
+    /// }
+    /// # }
+    ///
+    /// ```
+    pub fn send<B: Into<Body>>(&self, request: Request<B>) -> Result<Response<Body>, Error> {
+        let (request, body) = request.map(Into::into).into_parts();
+        let request = st_http::Request::from(request);
+        let request = bsatn::to_vec(&request).expect("Failed to BSATN-serialize `spacetimedb_lib::http::Request`");
+
+        match spacetimedb_bindings_sys::procedure::http_request(&request, &body.into_bytes()) {
+            Ok((response_source, body_source)) => {
+                let response = read_bytes_source_as::<st_http::Response>(response_source);
+                let response =
+                    http::response::Parts::try_from(response).expect("Invalid http response returned from host");
+                let mut buf = IterBuf::take();
+                read_bytes_source_into(body_source, &mut buf);
+                let body = Body::from_bytes(buf.clone());
+
+                Ok(http::Response::from_parts(response, body))
+            }
+            Err(err_source) => {
+                let error = read_bytes_source_as::<st_http::Error>(err_source);
+                Err(error)
+            }
+        }
+    }
+
+    /// Send a `GET` request to `uri` with no headers and wait for the response.
+    ///
+    /// # Example
+    ///
+    /// Send a `GET` request, then treat the response as a string and log it:
+    ///
+    /// ```no_run
+    /// # use spacetimedb::{procedure, ProcedureContext};
+    /// # #[procedure]
+    /// # fn get_from_somewhere(ctx: &mut ProcedureContext) {
+    /// match ctx.http.get("https://some-remote-host.invalid/download") {
+    ///     Err(err) => {
+    ///         log::error!("HTTP request failed: {err}");
+    ///     }
+    ///     Ok(response) => {
+    ///         let (parts, body) = response.into_parts();
+    ///         log::info!(
+    ///             "Got response with status {}, body {}",
+    ///             parts.status,
+    ///             body.into_string_lossy(),
+    ///         );
+    ///     }
+    /// }
+    /// # }
+    /// ```
+    pub fn get(&self, uri: impl TryInto<http::Uri, Error: Into<http::Error>>) -> Result<Response<Body>, Error> {
+        self.send(
+            http::Request::builder()
+                .method("GET")
+                .uri(uri)
+                .body(Body::empty())
+                .map_err(|err| Error::from_display(&err))?,
+        )
+    }
+}
+
+/// Represents the body of an HTTP request or response.
+pub struct Body {
+    inner: BodyInner,
+}
+
+impl Body {
+    /// Treat the body as a sequence of bytes.
+    pub fn into_bytes(self) -> Bytes {
+        match self.inner {
+            BodyInner::Bytes(bytes) => bytes,
+        }
+    }
+
+    /// Convert the body into a [`String`], erroring if it is not valid UTF-8.
+    pub fn into_string(self) -> Result<String, std::string::FromUtf8Error> {
+        String::from_utf8(self.into_bytes().into())
+    }
+
+    /// Convert the body into a [`String`], replacing invalid UTF-8 with
+    /// `U+FFFD REPLACEMENT CHARACTER`, which looks like this: �.
+    ///
+    /// See [`String::from_utf8_lossy`] for more details on the conversion.
+    pub fn into_string_lossy(self) -> String {
+        self.into_string()
+            .unwrap_or_else(|e| String::from_utf8_lossy(e.as_bytes()).into_owned())
+    }
+
+    /// Construct a `Body` consisting of `bytes`.
+    pub fn from_bytes(bytes: impl Into<Bytes>) -> Body {
+        Body {
+            inner: BodyInner::Bytes(bytes.into()),
+        }
+    }
+
+    /// An empty body, suitable for a `GET` request.
+    pub fn empty() -> Body {
+        ().into()
+    }
+
+    /// Is `self` exactly zero bytes?
+    pub fn is_empty(&self) -> bool {
+        match &self.inner {
+            BodyInner::Bytes(bytes) => bytes.is_empty(),
+        }
+    }
+}
+
+impl Default for Body {
+    fn default() -> Self {
+        Self::empty()
+    }
+}
+
+macro_rules! impl_body_from_bytes {
+    ($bytes:ident : $t:ty => $conv:expr) => {
+        impl From<$t> for Body {
+            fn from($bytes: $t) -> Body {
+                Body::from_bytes($conv)
+            }
+        }
+    };
+    ($t:ty) => {
+        impl_body_from_bytes!(bytes : $t => bytes);
+    };
+}
+
+impl_body_from_bytes!(String);
+impl_body_from_bytes!(Vec<u8>);
+impl_body_from_bytes!(Box<[u8]>);
+impl_body_from_bytes!(&'static [u8]);
+impl_body_from_bytes!(&'static str);
+impl_body_from_bytes!(_unit: () => Bytes::new());
+
+enum BodyInner {
+    Bytes(Bytes),
+}