use RSC payload to SSR server components

Author: AbanoubGhadban

CHANGELOG.md

@@ -16,6 +16,16 @@ You can find the **package** version numbers from this repo's tags and below in
 ## [Unreleased]
 *Add changes in master not yet tagged.*
 
+### Improved
+- Improved RSC rendering flow by eliminating double rendering of server components and reducing the number of HTTP requests.
+- Updated communication protocol between Node Renderer and Rails to version 2.0.0 which supports the ability to upload multiple bundles at once.
+- Added the ability to communicate between different bundles on the renderer by using the `runOnOtherBundle` function which is globally available for the rendering request.
+
+[PR 515](https://github.com/shakacode/react_on_rails_pro/pull/515) by [AbanoubGhadban](https://github.com/AbanoubGhadban).
+
+
+## [4.0.0-rc.13] - 2025-03-07
+
 ### Added
 - 🚀 **Introducing React Server Components Support!** 🎉
   - Experience the future of React with full RSC integration

Gemfile.ci

@@ -0,0 +1 @@
+gem "react_on_rails", github: "shakacode/react_on_rails", branch: "abanoubghadban/pro465/use-rsc-payload-to-render-server-components-on-server"

Gemfile.lock

@@ -6,6 +6,18 @@ GIT
       byebug (~> 11.0)
       pry (>= 0.13, < 0.15)
 
+GIT
+  remote: https://github.com/shakacode/react_on_rails.git
+  revision: c7d4d42682f4620a8ff02cf519e0a61a96534441
+  branch: abanoubghadban/pro465/use-rsc-payload-to-render-server-components-on-server
+  specs:
+    react_on_rails (15.0.0.alpha.2)
+      addressable
+      connection_pool
+      execjs (~> 2.5)
+      rails (>= 5.2)
+      rainbow (~> 3.0)
+
 PATH
   remote: .
   specs:
@@ -284,12 +296,6 @@ GEM
       ffi (~> 1.0)
     rdoc (6.12.0)
       psych (>= 4.0.0)
-    react_on_rails (15.0.0.alpha.2)
-      addressable
-      connection_pool
-      execjs (~> 2.5)
-      rails (>= 5.2)
-      rainbow (~> 3.0)
     regexp_parser (2.9.2)
     reline (0.6.0)
       io-console (~> 0.5)
@@ -462,7 +468,7 @@ DEPENDENCIES
   pry-theme
   puma (~> 6)
   rails (~> 7.1)
-  react_on_rails (= 15.0.0.alpha.2)
+  react_on_rails!
   react_on_rails_pro!
   rspec-rails
   rspec-retry

docs/node-renderer/js-configuration.md

@@ -17,7 +17,7 @@ Here are the options available for the JavaScript renderer configuration object,
 1. **logHttpLevel** (default: `process.env.RENDERER_LOG_HTTP_LEVEL || 'error'`) - The HTTP server log level (same allowed values as `logLevel`).
 1. **fastifyServerOptions** (default: `{}`) - Additional options to pass to the Fastify server factory. See [Fastify documentation](https://fastify.dev/docs/latest/Reference/Server/#factory).
 1. **bundlePath** (default: `process.env.RENDERER_BUNDLE_PATH || '/tmp/react-on-rails-pro-node-renderer-bundles'` ) - path to a temp directory where uploaded bundle files will be stored. For example you can set it to `path.resolve(__dirname, './.node-renderer-bundles')` if you configured renderer from the `/` directory of your app.
-1. **workersCount** (default: `env.RENDERER_WORKERS_COUNT || defaultWorkersCount()` where default is your CPUs count - 1) - Number of workers that will be forked to serve rendering requests. If you set this manually make sure that value is a **Number** and is `>= 1`.
+1. **workersCount** (default: `env.RENDERER_WORKERS_COUNT || defaultWorkersCount()` where default is your CPUs count - 1) - Number of workers that will be forked to serve rendering requests. If you set this manually make sure that value is a **Number** and is `>= 0`. Setting this to `0` will run the renderer in a single process mode without forking any workers, which is useful for debugging purposes. For production use, the value should be `>= 1`.
 1. **password** (default: `env.RENDERER_PASSWORD`) - The password expected to receive from the **Rails client** to authenticate rendering requests.
 If no password is set, no authentication will be required.
 1. **allWorkersRestartInterval** (default: `env.RENDERER_ALL_WORKERS_RESTART_INTERVAL`) - Interval in minutes between scheduled restarts of all workers. By default restarts are not enabled. If restarts are enabled, `delayBetweenIndividualWorkerRestarts` should also be set.
@@ -63,10 +63,14 @@ const config = {
   // All other values are the defaults, as described above
 };
 
+// For debugging, run in single process mode
+if (process.env.NODE_ENV === 'development') {
+  config.workersCount = 0;
+}
 // Renderer detects a total number of CPUs on virtual hostings like Heroku or CircleCI instead
 // of CPUs number allocated for current container. This results in spawning many workers while
 // only 1-2 of them really needed.
-if (process.env.CI) {
+else if (process.env.CI) {
   config.workersCount = 2;
 }
 

docs/react-server-components-rendering-flow.md

@@ -24,91 +24,35 @@ In a React Server Components project, there are three distinct types of bundles:
 - Code splitting occurs automatically for client components
 - Chunks are loaded on-demand during client component hydration
 
-## Current Rendering Flow
+## Rendering Flow
 
-When a request is made to a page using React Server Components, the following sequence occurs:
+When a request is made to a page using React Server Components, the following optimized sequence occurs:
 
-1. Initial HTML Generation:
+1. Initial Request Processing:
    - The `stream_react_component` helper is called in the view
    - Makes a request to the node renderer
-   - Renderer uses the **Server Bundle** to generate HTML for all components
-   - HTML is streamed to the client
-
-2. RSC Payload Generation:
-   - Browser shows the initial html
-   - Browser makes a separate fetch request to the RSC payload URL
-   - Calls `rsc_payload_react_component` on the server
-   - Node renderer uses the **RSC Bundle** to generate the RSC payload
-   - Server components are rendered and serialized
-   - Client components are included as references
+   - Server bundle's rendering function calls `generateRSCPayload` with the component name and props
+   - This executes the component rendering in the RSC bundle
+   - RSC bundle generates the payload containing server component data and client component references
+   - The payload is returned to the server bundle
+
+2. Server-Side Rendering with RSC Payload:
+   - The server bundle uses the RSC payload to generate HTML for server components using `RSCServerRoot`
+   - `RSCServerRoot` splits the RSC payload stream into two parts:
+     - One stream for rendering server components as HTML
+     - Another stream for embedding the RSC payload in the response
+   - `RSCPayloadContainer` component embeds the RSC payload within the HTML response
+   - HTML and embedded RSC payload are streamed together to the client
 
 3. Client Hydration:
-   - RSC payload is processed by React runtime
-   - Client component chunks are fetched based on references
-   - Components are hydrated progressively
-
-## Current Limitations
-
-This approach has two main inefficiencies:
-
-1. **Double Rendering**: Server components are rendered twice:
-   - Once for HTML generation using the server bundle
-   - Again for RSC payload generation using the RSC bundle
-
-2. **Multiple Requests**: Requires two separate HTTP requests:
-   - Initial request for HTML
-   - Secondary request for RSC payload
-
-```mermaid
-sequenceDiagram
-    participant Browser
-    participant View
-    participant NodeRenderer
-    participant ServerBundle
-    participant RSCBundle
-    
-    Note over Browser,RSCBundle: 1. Initial HTML Generation
-    Browser->>View: Request page
-    View->>NodeRenderer: stream_react_component
-    NodeRenderer->>ServerBundle: Generate HTML
-    ServerBundle-->>NodeRenderer: HTML for all components
-    NodeRenderer-->>Browser: Stream HTML
-    
-    Note over Browser,RSCBundle: 2. RSC Payload Generation
-    Browser->>NodeRenderer: Fetch RSC payload
-    NodeRenderer->>RSCBundle: rsc_payload_react_component
-    RSCBundle-->>NodeRenderer: RSC payload with:<br/>- Server components<br/>- Client component refs
-    NodeRenderer-->>Browser: Stream RSC payload
-    
-    Note over Browser: 3. Client Hydration
-    Browser->>Browser: Process RSC payload
-    loop For each client component
-        Browser->>Browser: Fetch component chunk
-        Browser->>Browser: Hydrate component
-    end
-```
-
-> [!NOTE]
-> For simplicity, this diagram shows the RSC payload being fetched after the HTML is fully streamed to the client. In reality, the browser begins fetching the RSC payload and starts hydration immediately as soon as it receives the necessary HTML, without waiting for the complete page to be streamed. This parallel processing enables faster page interactivity and better performance.
-
-## Future Improvements
-
-These inefficiencies will be addressed in the upcoming ["Use RSC payload to render server components on server"](https://github.com/shakacode/react_on_rails_pro/pull/515) PR. The new flow will be:
-
-1. Initial Request:
-   - `stream_react_component` triggers node renderer
-   - Renderer uses **RSC Bundle** to generate RSC payload
-   - Payload is passed to rendering function in **Server Bundle**
-   - HTML of server components is generated using RSC payload
-   - Client component references are filled with HTML of the client components
-
-2. Single Response:
-   - HTML and RSC payload are streamed together, with the RSC payload embedded inside the HTML page
    - Browser displays HTML immediately
-   - React runtime uses embedded RSC payload for hydration
-   - Client components are hydrated progressively
+   - React runtime uses the embedded RSC payload for hydration
+   - Client components are hydrated progressively without requiring a separate HTTP request
 
-This improved approach eliminates double rendering and reduces HTTP requests, resulting in better performance and resource utilization.
+This approach offers significant advantages:
+- Eliminates double rendering of server components
+- Reduces HTTP requests by embedding the RSC payload within the initial HTML response
+- Provides faster interactivity through streamlined rendering and hydration
 
 ```mermaid
 sequenceDiagram
@@ -121,9 +65,9 @@ sequenceDiagram
     Note over Browser,ServerBundle: 1. Initial Request
     Browser->>View: Request page
     View->>NodeRenderer: stream_react_component
-    NodeRenderer->>RSCBundle: Generate RSC payload
-    RSCBundle-->>NodeRenderer: RSC payload
-    NodeRenderer->>ServerBundle: Pass RSC payload
+    NodeRenderer->>ServerBundle: Execute rendering request
+    ServerBundle->>RSCBundle: generateRSCPayload(component, props)
+    RSCBundle-->>ServerBundle: RSC payload with:<br/>- Server components<br/>- Client component refs
     ServerBundle-->>NodeRenderer: Generate HTML using RSC payload
     
     Note over Browser,ServerBundle: 2. Single Response

lib/react_on_rails_pro/request.rb

@@ -14,7 +14,7 @@ def reset_connection
 
       def render_code(path, js_code, send_bundle)
         Rails.logger.info { "[ReactOnRailsPro] Perform rendering request #{path}" }
-        form = form_with_code(js_code, send_bundle, is_rsc_payload: false)
+        form = form_with_code(js_code, send_bundle)
         perform_request(path, form: form)
       end
 
@@ -28,27 +28,55 @@ def render_code_as_stream(path, js_code, is_rsc_payload:)
         end
 
         ReactOnRailsPro::StreamRequest.create do |send_bundle|
-          form = form_with_code(js_code, send_bundle, is_rsc_payload: is_rsc_payload)
+          form = form_with_code(js_code, send_bundle)
           perform_request(path, form: form, stream: true)
         end
       end
 
       def upload_assets
         Rails.logger.info { "[ReactOnRailsPro] Uploading assets" }
-        perform_request("/upload-assets", form: form_with_assets_and_bundle)
 
-        return unless ReactOnRailsPro.configuration.enable_rsc_support
+        # Check if server bundle exists before trying to upload assets
+        server_bundle_path = ReactOnRails::Utils.server_bundle_js_file_path
+        unless File.exist?(server_bundle_path)
+          raise ReactOnRailsPro::Error, "Server bundle not found at #{server_bundle_path}. " \
+                                        "Please build your bundles before uploading assets."
+        end
+
+        # Create a list of bundle timestamps to send to the node renderer
+        pool = ReactOnRailsPro::ServerRenderingPool::NodeRenderingPool
+        target_bundles = [pool.server_bundle_hash]
+
+        # Add RSC bundle if enabled
+        if ReactOnRailsPro.configuration.enable_rsc_support
+          rsc_bundle_path = ReactOnRails::Utils.rsc_bundle_js_file_path
+          unless File.exist?(rsc_bundle_path)
+            raise ReactOnRailsPro::Error, "RSC bundle not found at #{rsc_bundle_path}. " \
+                                          "Please build your bundles before uploading assets."
+          end
+          target_bundles << pool.rsc_bundle_hash
+        end
+
+        form = form_with_assets_and_bundle
+        form["targetBundles"] = target_bundles
 
-        perform_request("/upload-assets", form: form_with_assets_and_bundle(is_rsc_payload: true))
-        # Explicitly return nil to ensure consistent return value regardless of whether
-        # enable_rsc_support is true or false. Without this, the method would return nil
-        # when RSC is disabled but return the response object when RSC is enabled.
-        nil
+        perform_request("/upload-assets", form: form)
       end
 
       def asset_exists_on_vm_renderer?(filename)
         Rails.logger.info { "[ReactOnRailsPro] Sending request to check if file exist on node-renderer: #{filename}" }
-        response = perform_request("/asset-exists?filename=#{filename}", json: common_form_data)
+
+        form_data = common_form_data
+
+        # Add targetBundles from the current bundle hash and RSC bundle hash
+        pool = ReactOnRailsPro::ServerRenderingPool::NodeRenderingPool
+        target_bundles = [pool.server_bundle_hash]
+
+        target_bundles << pool.rsc_bundle_hash if ReactOnRailsPro.configuration.enable_rsc_support
+
+        form_data["targetBundles"] = target_bundles
+
+        response = perform_request("/asset-exists?filename=#{filename}", json: form_data)
         JSON.parse(response.body)["exists"] == true
       end
 
@@ -118,42 +146,54 @@ def perform_request(path, **post_options) # rubocop:disable Metrics/AbcSize,Metr
         response
       end
 
-      def form_with_code(js_code, send_bundle, is_rsc_payload:)
+      def form_with_code(js_code, send_bundle)
         form = common_form_data
         form["renderingRequest"] = js_code
-        populate_form_with_bundle_and_assets(form, is_rsc_payload: is_rsc_payload, check_bundle: false) if send_bundle
+        populate_form_with_bundle_and_assets(form, check_bundle: false) if send_bundle
         form
       end
 
-      def populate_form_with_bundle_and_assets(form, is_rsc_payload:, check_bundle:)
-        server_bundle_path = if is_rsc_payload
-                               ReactOnRails::Utils.rsc_bundle_js_file_path
-                             else
-                               ReactOnRails::Utils.server_bundle_js_file_path
-                             end
-        if check_bundle && !File.exist?(server_bundle_path)
-          raise ReactOnRailsPro::Error, "Bundle not found #{server_bundle_path}"
+      def populate_form_with_bundle_and_assets(form, check_bundle:)
+        pool = ReactOnRailsPro::ServerRenderingPool::NodeRenderingPool
+
+        add_bundle_to_form(
+          form,
+          bundle_path: ReactOnRails::Utils.server_bundle_js_file_path,
+          bundle_file_name: pool.renderer_bundle_file_name,
+          bundle_hash: pool.server_bundle_hash,
+          check_bundle: check_bundle
+        )
+
+        if ReactOnRailsPro.configuration.enable_rsc_support
+          add_bundle_to_form(
+            form,
+            bundle_path: ReactOnRails::Utils.rsc_bundle_js_file_path,
+            bundle_file_name: pool.rsc_renderer_bundle_file_name,
+            bundle_hash: pool.rsc_bundle_hash,
+            check_bundle: check_bundle
+          )
         end
 
-        pool = ReactOnRailsPro::ServerRenderingPool::NodeRenderingPool
-        renderer_bundle_file_name = if is_rsc_payload
-                                      pool.rsc_renderer_bundle_file_name
-                                    else
-                                      pool.renderer_bundle_file_name
-                                    end
-        form["bundle"] = {
-          body: get_form_body_for_file(server_bundle_path),
+        add_assets_to_form(form)
+      end
+
+      def add_bundle_to_form(form, bundle_path:, bundle_file_name:, bundle_hash:, check_bundle:)
+        raise ReactOnRailsPro::Error, "Bundle not found #{bundle_path}" if check_bundle && !File.exist?(bundle_path)
+
+        form["bundle_#{bundle_hash}"] = {
+          body: get_form_body_for_file(bundle_path),
           content_type: "text/javascript",
-          filename: renderer_bundle_file_name
+          filename: bundle_file_name
         }
-
-        add_assets_to_form(form, is_rsc_payload: is_rsc_payload)
       end
 
-      def add_assets_to_form(form, is_rsc_payload:)
+      def add_assets_to_form(form)
         assets_to_copy = ReactOnRailsPro.configuration.assets_to_copy || []
-        # react_client_manifest file is needed to generate react server components payload
-        assets_to_copy << ReactOnRails::Utils.react_client_manifest_file_path if is_rsc_payload
+        # react_client_manifest and react_server_manifest files are needed to generate react server components payload
+        if ReactOnRailsPro.configuration.enable_rsc_support
+          assets_to_copy << ReactOnRails::Utils.react_client_manifest_file_path
+          assets_to_copy << ReactOnRails::Utils.react_server_client_manifest_file_path
+        end
 
         return form unless assets_to_copy.present?
 
@@ -180,18 +220,14 @@ def add_assets_to_form(form, is_rsc_payload:)
         form
       end
 
-      def form_with_assets_and_bundle(is_rsc_payload: false)
+      def form_with_assets_and_bundle
         form = common_form_data
-        populate_form_with_bundle_and_assets(form, is_rsc_payload: is_rsc_payload, check_bundle: true)
+        populate_form_with_bundle_and_assets(form, check_bundle: true)
         form
       end
 
       def common_form_data
-        {
-          "gemVersion" => ReactOnRailsPro::VERSION,
-          "protocolVersion" => "1.0.0",
-          "password" => ReactOnRailsPro.configuration.renderer_password
-        }
+        ReactOnRailsPro::Utils.common_form_data
       end
 
       def create_connection