Use RSC payload to render server components on server (#515)

# Using RSC Payload to Render Server Components on Server

## Overview

This PR implements a significant architectural improvement to the React
Server Components (RSC) rendering flow in both `react_on_rails` and
`react_on_rails_pro`. Previously, server components were rendered twice
- once for HTML generation and again for RSC payload. This PR unifies
the process by generating the RSC payload first and using it for
server-side rendering, eliminating double rendering and reducing HTTP
requests.

## Previous Rendering Flow

Before this improvement, the React Server Components rendering process
worked as follows:

1. Initial HTML Generation:
   - The `stream_react_component` helper is called in the view
   - Makes a request to the node renderer
- Renderer uses the **Server Bundle** to generate HTML for all
components
   - HTML is streamed to the client

2. RSC Payload Generation:
   - Browser shows the initial html
   - Browser makes a separate fetch request to the RSC payload URL
   - Calls `rsc_payload_react_component` on the server
   - Node renderer uses the **RSC Bundle** to generate the RSC payload
   - Server components are rendered and serialized
   - Client components are included as references

3. Client Hydration:
   - RSC payload is processed by React runtime
   - Client component chunks are fetched based on references
   - Components are hydrated progressively

This approach had two main inefficiencies:

1. **Double Rendering**: Server components are rendered twice:
   - Once for HTML generation using the server bundle
   - Again for RSC payload generation using the RSC bundle

2. **Multiple Requests**: Requires two separate HTTP requests:
   - Initial request for HTML
   - Secondary request for RSC payload

```mermaid
sequenceDiagram
    participant Browser
    participant View
    participant NodeRenderer
    participant ServerBundle
    participant RSCBundle
    
    Note over Browser,RSCBundle: 1. Initial HTML Generation
    Browser->>View: Request page
    View->>NodeRenderer: stream_react_component
    NodeRenderer->>ServerBundle: Generate HTML
    ServerBundle-->>NodeRenderer: HTML for all components
    NodeRenderer-->>Browser: Stream HTML
    
    Note over Browser,RSCBundle: 2. RSC Payload Generation
    Browser->>NodeRenderer: Fetch RSC payload
    NodeRenderer->>RSCBundle: rsc_payload_react_component
    RSCBundle-->>NodeRenderer: RSC payload with:<br/>- Server components<br/>- Client component refs
    NodeRenderer-->>Browser: Stream RSC payload
    
    Note over Browser: 3. Client Hydration
    Browser->>Browser: Process RSC payload
    loop For each client component
        Browser->>Browser: Fetch component chunk
        Browser->>Browser: Hydrate component
    end
```

> [!NOTE]
> For simplicity, this diagram shows the RSC payload being fetched after
the HTML is fully streamed to the client. In reality, the browser begins
fetching the RSC payload and starts hydration immediately as soon as it
receives the necessary HTML, without waiting for the complete page to be
streamed. This parallel processing enables faster page interactivity and
better performance.

## Key Improvements

1. **Elimination of Double Rendering**: Server components are now
rendered only once - in the RSC bundle - and the resulting payload is
used by both server and client.
2. **Single HTTP Request**: The client no longer needs to make a
separate request for the RSC payload, as it's embedded in the initial
HTML response.
3. **Improved Performance**: Reduced server load and faster page
interactivity by eliminating redundant rendering and requests.

## Technical Implementation

### Inter-Bundle Communication

The core of this implementation is a mechanism for communication between
bundles in the node renderer:

- Introduced `runOnOtherBundle` function in the VM context that allows
executing code in a different bundle
- Added `generateRSCPayload` function that uses `runOnOtherBundle` to
run rendering requests in the RSC bundle

```javascript
// Simplified representation of the flow
globalThis.generateRSCPayload = function generateRSCPayload(componentName, props, railsContext) {
  const { renderingRequest, rscBundleHash } = railsContext.serverSideRSCPayloadParameters;
  const propsString = JSON.stringify(props);
  const newRenderingRequest = renderingRequest.replace(/\(\s*\)\s*$/, `('${componentName}', ${propsString})`);
  return runOnOtherBundle(rscBundleHash, newRenderingRequest);
}
```

### Key Components for RSC Rendering

Two new components are introduced in React on Rails to facilitate
server-side RSC rendering:

#### RSCServerRoot

`RSCServerRoot` is the core component responsible for:

- Requesting the RSC payload from the RSC bundle via the
`generateRSCPayload` function
- Creating server component elements using the RSC payload
- Rendering both the server component HTML and embedding the RSC payload
for client hydration

This component operates as follows:
1. Receives the component name and props to render
2. Creates an SSR manifest by merging client and server manifests
3. Fetches the RSC payload stream for the component
4. Splits the stream to use for both rendering and embedding
5. Renders the server component on the server using the RSC payload
6. Returns a React fragment containing both the rendered component and
the RSCPayloadContainer

#### RSCPayloadContainer

`RSCPayloadContainer` handles embedding the RSC payload within the HTML
response by:

- Accepting an RSC payload stream as input
- Processing the stream chunks progressively
- Creating script elements that inject the payload into
`self.REACT_ON_RAILS_RSC_PAYLOAD`
- Suspending rendering until new chunks arrive, enabling streaming

This creates a seamless experience where:
- Server components render immediately on the server
- RSC payload is embedded directly in the HTML response
- Client can hydrate components progressively without a separate request

### New Rendering Flow

1. When `stream_react_component` is called:
- The server bundle rendering function calls `generateRSCPayload` with
the component name and props
   - This executes the component rendering in the RSC bundle
- RSC bundle generates the payload containing server component data and
client component references
   - The payload is returned to the server bundle

2. The server bundle then:
   - Uses the RSC payload to generate HTML for server components
   - Renders client components normally
   - Embeds the RSC payload within the HTML response
   - Streams the result to the client

3. On the client:
   - HTML is displayed immediately
   - React runtime uses the embedded RSC payload for hydration
   - Client components are hydrated progressively

### Configuration and Detection

- Added conditional rendering function selection based on bundle type:
```javascript
ReactOnRails[ReactOnRails.isRSCBundle ? 'serverRenderRSCReactComponent' : 'streamServerRenderedReactComponent']
```

- RSC bundles now expose `ReactOnRails.isRSCBundle` flag to identify
their special purpose
- Server bundles automatically detect and use RSC payload when available

## Visual Representation

```mermaid
sequenceDiagram
    participant Browser
    participant View
    participant NodeRenderer
    participant RSCBundle
    participant ServerBundle
    
    Note over Browser,ServerBundle: 1. Initial Request
    Browser->>View: Request page
    View->>NodeRenderer: stream_react_component
    NodeRenderer->>ServerBundle: Execute rendering request
    ServerBundle->>RSCBundle: generateRSCPayload(component, props)
    RSCBundle-->>ServerBundle: RSC payload with:<br/>- Server components<br/>- Client component refs
    ServerBundle-->>NodeRenderer: Generate HTML using RSC payload
    
    Note over Browser,ServerBundle: 2. Single Response
    NodeRenderer-->>Browser: Stream HTML with embedded RSC payload
    
    Note over Browser: 3. Client Hydration
    Browser->>Browser: Process embedded RSC payload
    loop For each client component
        Browser->>Browser: Fetch component chunk
        Browser->>Browser: Hydrate component
    end
```

## Testing

- Added tests for the `serverRenderRSCReactComponent` function
- Verified correct payload generation and embedding
- Ensured proper hydration of client components using the embedded
payload
- Confirmed reduced network requests and rendering time

## Performance Improvements

- **Server Load**: Reduced overhead of rendering server components twice
- **Network Traffic**: Reduced by eliminating separate RSC payload
requests.
- **Time to Interactive**: Improved by streamlining the rendering and
hydration process



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Enhanced React Server Components (RSC) rendering flow to eliminate
double rendering and reduce HTTP requests, resulting in faster
interactivity.
- Added support for simultaneous upload and management of multiple
bundles and assets.
- Introduced a global function for inter-bundle communication during
rendering requests.
- Added Redis-based streaming support for server-rendered React
components, enabling advanced data-fetching patterns.
- Provided new guides and documentation for rendering RSC inside client
components and optimizing hydration flows.
- Added new sample pages, components, and routes demonstrating advanced
RSC, streaming, and router integration patterns.

- **Improvements**
- Upgraded the protocol version between Node Renderer and Rails to
2.0.0.
- Improved error handling and validation for asset and bundle
operations.
- Optimized bundle and asset management for better concurrency and
resource usage.
  - Enhanced test coverage for multi-bundle and streaming scenarios.
  - Updated dependencies for improved compatibility and performance.

- **Bug Fixes**
- Fixed edge cases in bundle upload, asset existence checks, and error
propagation during streaming.

- **Documentation**
- Updated and expanded documentation to reflect the new RSC rendering
flow, protocol changes, and advanced usage patterns.

- **Chores**
- Updated development dependencies and configuration for improved
tooling and testing support.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Author: AbanoubGhadban

CHANGELOG.md

@@ -16,6 +16,18 @@ You can find the **package** version numbers from this repo's tags and below in
 ## [Unreleased]
 *Add changes in master not yet tagged.*
 
+## [4.0.0-rc.14] - 2025-06-22
+
+### Improved
+- Improved RSC rendering flow by eliminating double rendering of server components and reducing the number of HTTP requests.
+- Updated communication protocol between Node Renderer and Rails to version 2.0.0 which supports the ability to upload multiple bundles at once.
+- Added the ability to communicate between different bundles on the renderer by using the `runOnOtherBundle` function which is globally available for the rendering request.
+
+[PR 515](https://github.com/shakacode/react_on_rails_pro/pull/515) by [AbanoubGhadban](https://github.com/AbanoubGhadban).
+
+
+## [4.0.0-rc.13] - 2025-03-07
+
 ### Added
 - 🚀 **Introducing React Server Components Support!** 🎉
   - Experience the future of React with full RSC integration

Gemfile.development_dependencies

@@ -5,7 +5,7 @@ git_source(:github) { |repo| "https://github.com/#{repo}.git" }
 
 ruby '3.3.7'
 
-gem "react_on_rails", "15.0.0.alpha.2" # keep in sync with package.json files
+gem "react_on_rails", "15.0.0.rc.1" # keep in sync with package.json files
 
 # For local development
 # Add the following line to the Gemfile.local file

Gemfile.lock

@@ -284,7 +284,7 @@ GEM
       ffi (~> 1.0)
     rdoc (6.12.0)
       psych (>= 4.0.0)
-    react_on_rails (15.0.0.alpha.2)
+    react_on_rails (15.0.0.rc.1)
       addressable
       connection_pool
       execjs (~> 2.5)
@@ -462,7 +462,7 @@ DEPENDENCIES
   pry-theme
   puma (~> 6)
   rails (~> 7.1)
-  react_on_rails (= 15.0.0.alpha.2)
+  react_on_rails (= 15.0.0.rc.1)
   react_on_rails_pro!
   rspec-rails
   rspec-retry

babel.config.js

@@ -1,3 +1,4 @@
 module.exports = {
   presets: [['@babel/preset-env', { targets: { node: 'current' } }], '@babel/preset-typescript'],
+  plugins: ['@babel/plugin-syntax-import-attributes'],
 };

docs/node-renderer/js-configuration.md

@@ -17,7 +17,7 @@ Here are the options available for the JavaScript renderer configuration object,
 1. **logHttpLevel** (default: `process.env.RENDERER_LOG_HTTP_LEVEL || 'error'`) - The HTTP server log level (same allowed values as `logLevel`).
 1. **fastifyServerOptions** (default: `{}`) - Additional options to pass to the Fastify server factory. See [Fastify documentation](https://fastify.dev/docs/latest/Reference/Server/#factory).
 1. **bundlePath** (default: `process.env.RENDERER_BUNDLE_PATH || '/tmp/react-on-rails-pro-node-renderer-bundles'` ) - path to a temp directory where uploaded bundle files will be stored. For example you can set it to `path.resolve(__dirname, './.node-renderer-bundles')` if you configured renderer from the `/` directory of your app.
-1. **workersCount** (default: `env.RENDERER_WORKERS_COUNT || defaultWorkersCount()` where default is your CPUs count - 1) - Number of workers that will be forked to serve rendering requests. If you set this manually make sure that value is a **Number** and is `>= 1`.
+1. **workersCount** (default: `process.env.RENDERER_WORKERS_COUNT || defaultWorkersCount()` where default is your CPUs count - 1) - Number of workers that will be forked to serve rendering requests. If you set this manually make sure that value is a **Number** and is `>= 0`. Setting this to `0` will run the renderer in a single process mode without forking any workers, which is useful for debugging purposes. For production use, the value should be `>= 1`.
 1. **password** (default: `env.RENDERER_PASSWORD`) - The password expected to receive from the **Rails client** to authenticate rendering requests.
 If no password is set, no authentication will be required.
 1. **allWorkersRestartInterval** (default: `env.RENDERER_ALL_WORKERS_RESTART_INTERVAL`) - Interval in minutes between scheduled restarts of all workers. By default restarts are not enabled. If restarts are enabled, `delayBetweenIndividualWorkerRestarts` should also be set.
@@ -63,10 +63,14 @@ const config = {
   // All other values are the defaults, as described above
 };
 
+// For debugging, run in single process mode
+if (process.env.NODE_ENV === 'development') {
+  config.workersCount = 0;
+}
 // Renderer detects a total number of CPUs on virtual hostings like Heroku or CircleCI instead
 // of CPUs number allocated for current container. This results in spawning many workers while
 // only 1-2 of them really needed.
-if (process.env.CI) {
+else if (process.env.CI) {
   config.workersCount = 2;
 }