Merge pull request #2 from ros-realtime/add-minimal-scheduling

Add minimal_scheduling package

Author: carlossvg

README.md

@@ -5,9 +5,10 @@ create real-time applications.
 
 * [minimal_memory_lock](minimal_memory_lock/README.md): shows how to lock the
   process memory to avoid memory page faults
-  
+* [minimal_scheduling](minimal_scheduling/README.md): shows how to configure the thread(s)
+scheduling policy and priority
+
 TODO: Add other packages after review (currently in rolling-experimental branch): 
-* minimal_scheduling: shows how to configure the thread(s) scheduling policy and priority (TODO: add SCHED_DEADLINE example)
 * minimal_cpu_affinity: shows how to set the process and threads CPU affinity
 * minimal_realtime_loop: shows different approaches to created typical real-time time based loops
 * minimal_deadline_qos: shows how to use the DDS deadline QoS policy

minimal_scheduling/CMakeLists.txt

@@ -0,0 +1,42 @@
+cmake_minimum_required(VERSION 3.5)
+project(minimal_scheduling)
+
+# Default to C++14
+if(NOT CMAKE_CXX_STANDARD)
+  set(CMAKE_CXX_STANDARD 14)
+endif()
+
+if(CMAKE_COMPILER_IS_GNUCXX OR CMAKE_CXX_COMPILER_ID MATCHES "Clang")
+  add_compile_options(-Wall -Wextra -Wpedantic)
+endif()
+
+find_package(ament_cmake REQUIRED)
+find_package(rclcpp REQUIRED)
+find_package(std_msgs REQUIRED)
+
+add_executable(minimal_scheduling_main_thread minimal_scheduling_main_thread.cpp)
+ament_target_dependencies(minimal_scheduling_main_thread rclcpp std_msgs)
+
+add_executable(minimal_scheduling_middleware_threads minimal_scheduling_middleware_threads.cpp)
+ament_target_dependencies(minimal_scheduling_middleware_threads rclcpp std_msgs)
+
+add_executable(minimal_scheduling_spin_thread minimal_scheduling_spin_thread.cpp)
+ament_target_dependencies(minimal_scheduling_spin_thread rclcpp std_msgs)
+
+add_executable(minimal_scheduling_callback_group minimal_scheduling_callback_group.cpp)
+ament_target_dependencies(minimal_scheduling_callback_group rclcpp std_msgs)
+
+install(TARGETS
+  minimal_scheduling_main_thread
+  minimal_scheduling_middleware_threads
+  minimal_scheduling_spin_thread
+  minimal_scheduling_callback_group
+  DESTINATION lib/${PROJECT_NAME}
+)
+
+if(BUILD_TESTING)
+  find_package(ament_lint_auto REQUIRED)
+  ament_lint_auto_find_test_dependencies()
+endif()
+
+ament_package()

minimal_scheduling/README.md

@@ -0,0 +1,212 @@
+# minimal_scheduling
+
+These examples show how to configure the thread scheduling policy and priority using
+[pthread APIs](https://linux.die.net/man/7/pthreads).
+
+In order to show the effects of using a real-time scheduling policy, the examples show
+the number of involuntary context switches in each callback execution. An involuntary context 
+switches happens when the kernel scheduler preempts a task to schedule a higher priority thread
+or by a I/O request.
+
+Using a real-time scheduling policy and using a real-time priority removes or reduces the
+number of involuntary context switches.
+
+**Note: The code shown in the examples is not necessarily real-time safe. The aim of these
+examples is to show a minimal usage example for a particular concept.**
+
+## Requirements 
+
+Adjust permissions for the scheduler. Add to `/etc/security/limits.conf` (as sudo):
+
+```bash
+<your username>    -   rtprio   98
+```
+
+## How to run
+
+### minimal_scheduling_main_thread
+
+For this example, we will be configuring the main thread scheduling policy of the main thread,
+where the node is spinning. In this case, the middleware threads will not inherit these settings
+because they are created after the scheduling configuration.
+
+## Using SCHED_OTHER scheduling policy
+
+We start with the default settings, which means we don't configure the main thread scheduling
+policy and priority. In Linux, by default, the `SCHED_OTHER` scheduling policy is used. Since,
+this is not a real-time scheduling policy the main thread might be preempted frequently. For this 
+reason, we expect to see a high number of involuntary context switches.
+
+```bash
+$ ros2 run minimal_scheduling minimal_scheduling_main_thread
+```
+
+<script id="asciicast-gxZvQuyvNZQR1Y75nFUCq7GQ8" src="https://asciinema.org/a/gxZvQuyvNZQR1Y75nFUCq7GQ8.js" async></script>
+
+## Using SCHED_FIFO scheduling policy
+
+Now we run the same example but using the `SCHED_FIFO` scheduling policy and setting a priority
+of 80.
+
+There are just a few kernel threads running with a higher priority, so we expect to see a few
+number of involuntary context switches or none.
+
+```bash
+$ ros2 run minimal_scheduling minimal_scheduling_main_thread --sched SCHED_FIFO --priority 80
+```
+
+Using `htop` or `ps` we can observe the priorities and the scheduling policies of the threads.
+
+```bash
+$ ps -C minimal_schedul -L -o tid,comm,rtprio,cls,psr
+    TID COMMAND         RTPRIO CLS PSR
+   6775 minimal_sched_f     80  FF   7
+   6776 minimal_sched_f      -  TS   1
+   6777 gc                   -  TS   1
+   6778 dq.builtins          -  TS   0
+   6779 dq.user              -  TS   6
+   6780 tev                  -  TS   2
+   6781 recv                 -  TS   4
+   6782 recvMC               -  TS   0
+   6783 recvUC               -  TS   1
+   6784 minimal_sched_f      -  TS   0
+```
+
+Note we are not configuring the threads created by the RMW middleware. These threads will run
+with normal priority. This option might be preferred when middleware communications shall not
+participate in the real-time execution. Also, some DDS implementations allow to configure
+each thread scheduling individually, allowing to set a priority different creator thread.
+
+<script id="asciicast-XzlFpNwZKw0hea9vhfKFCMgj4" src="https://asciinema.org/a/XzlFpNwZKw0hea9vhfKFCMgj4.js" async></script>
+
+## Using SCHED_RR scheduling policy
+
+Now we run the same example but using the `SCHED_RR` scheduling policy and setting a priority
+of 80. We expect to see the same results as in the previous case.
+
+```bash
+$ ros2 run minimal_scheduling minimal_scheduling_main_thread --sched SCHED_RR --priority 80
+```
+
+Using `htop` or `ps` we can observe the priorities and the scheduling policies of the threads.
+
+```bash
+$ ps -C minimal_schedul -L -o tid,comm,rtprio,cls,psr
+    TID COMMAND         RTPRIO CLS PSR
+  16359 minimal_schedul     80  RR   3
+  16360 minimal_schedul      -  TS   6
+  16361 gc                   -  TS   1
+  16362 dq.builtins          -  TS   6
+  16363 dq.user              -  TS   1
+  16364 tev                  -  TS   5
+  16365 recv                 -  TS   7
+  16366 recvMC               -  TS   0
+  16367 recvUC               -  TS   7
+  16368 minimal_schedul      -  TS   0
+```
+
+<script id="asciicast-OOGN7OVgLkOE4b3ifTPdx6DWr" src="https://asciinema.org/a/OOGN7OVgLkOE4b3ifTPdx6DWr.js" async></script>
+
+### minimal_scheduling_middleware_threads
+
+For this example, we will be configuring the main thread scheduling policy before the node
+creation. The middleware threads will inherit these settings because they are created after
+the scheduling configuration.
+
+```bash
+$ ros2 run minimal_scheduling minimal_scheduling_middleware_threads --sched SCHED_FIFO --priority 80
+```
+
+Using `htop` or `ps` we can observe the priorities and the scheduling policies of the threads.
+
+```bash
+$ ps -C minimal_schedul -L -o tid,comm,rtprio,cls,psr
+    TID COMMAND         RTPRIO CLS PSR
+   6794 minimal_sched_f     80  FF   2
+   6795 minimal_sched_f     80  FF   6
+   6796 gc                  80  FF   6
+   6797 dq.builtins         80  FF   3
+   6798 dq.user             80  FF   6
+   6799 tev                 80  FF   6
+   6800 recv                80  FF   7
+   6801 recvMC              80  FF   7
+   6802 recvUC              80  FF   6
+   6803 minimal_sched_f     80  FF   7
+```
+
+<script id="asciicast-WtoeazuOds4CCpBDA0Mfkg8Ly" src="https://asciinema.org/a/WtoeazuOds4CCpBDA0Mfkg8Ly.js" async></script>
+
+### minimal_scheduling_spin_thread
+
+For this example, we will be configuring the scheduling policy of a thread used to spin the
+executor. In this case, the middleware threads will not inherit these settings because
+the settings apply only to the executor thread.
+
+```bash
+$ ros2 run minimal_scheduling minimal_scheduling_spin_thread --sched SCHED_FIFO --priority 80
+```
+
+Using `htop` or `ps` we can observe the priorities and the scheduling policies of the threads.
+
+```bash
+$ ps -C minimal_schedul -L -o tid,comm,rtprio,cls,psr
+    TID COMMAND         RTPRIO CLS PSR
+   6822 minimal_sched_f      -  TS   1
+   6823 minimal_sched_f      -  TS   7
+   6824 gc                   -  TS   5
+   6825 dq.builtins          -  TS   7
+   6826 dq.user              -  TS   2
+   6827 tev                  -  TS   0
+   6828 recv                 -  TS   2
+   6829 recvMC               -  TS   0
+   6830 recvUC               -  TS   7
+   6831 minimal_sched_f      -  TS   0
+   6832 minimal_sched_f     80  FF   7
+```
+
+<script id="asciicast-nLpeLDCmtwT5JqH9QYvVPGVXT" src="https://asciinema.org/a/nLpeLDCmtwT5JqH9QYvVPGVXT.js" async></script>
+
+### minimal_scheduling_callback_group
+
+For this example, we will be using multiple callback groups to separate callbacks, so it is
+possible to execute them by threads with different scheduling priorities.
+
+We run the example using `SCHED_FIFO` and priority 80 for the thread running the real-time
+callbacks. The other callbacks run with normal priority. We expect to see a high number of
+involuntary context switches logged only from the non-real-time subscription callbacks.
+
+```bash
+$ ros2 run minimal_scheduling minimal_scheduling_callback_group --sched SCHED_FIFO --priority 80
+```
+
+Using `htop` or `ps` we can observe the priorities and the scheduling policies of the threads.
+
+```bash
+$ ps -C minimal_schedul -L -o tid,comm,rtprio,cls,psr
+    TID COMMAND         RTPRIO CLS PSR
+   6847 minimal_sched_f      -  TS   1
+   6848 minimal_sched_f      -  TS   0
+   6849 gc                   -  TS   1
+   6850 dq.builtins          -  TS   1
+   6851 dq.user              -  TS   7
+   6852 tev                  -  TS   1
+   6853 recv                 -  TS   3
+   6854 recvMC               -  TS   0
+   6855 recvUC               -  TS   7
+   6856 minimal_sched_f      -  TS   0
+   6857 minimal_sched_f     80  FF   0
+```
+
+<script id="asciicast-FncZQ5gSgp6sNIxHj4weDra3n" src="https://asciinema.org/a/FncZQ5gSgp6sNIxHj4weDra3n.js" async></script>
+
+## Resources
+
+- [https://www.man7.org/linux/man-pages/man7/sched.7.html](https://www.man7.org/linux/man-pages/man7/sched.7.html)
+- [https://wiki.linuxfoundation.org/realtime/documentation/technical_basics/sched_policy_prio/start](https://wiki.linuxfoundation.org/realtime/documentation/technical_basics/sched_policy_prio/start)
+- [https://wiki.linuxfoundation.org/realtime/documentation/howto/applications/application_base](https://wiki.linuxfoundation.org/realtime/documentation/howto/applications/application_base)
+- [https://linux.die.net/man/3/pthread_setschedparam](https://linux.die.net/man/3/pthread_setschedparam)
+- [https://linux.die.net/man/3/pthread_create](https://linux.die.net/man/3/pthread_create)
+- [https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/performance_tuning_guide/s-cpu-scheduler](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/performance_tuning_guide/s-cpu-scheduler)
+- Callback groups executor - Ralph Lange [video](https://www.youtube.com/watch?v=5Sd5bvvZeb0), [slides](https://www.apex.ai/_files/ugd/984e93_f3791ae0711042a883bfc40f827d6268.pdf)
+- [https://github.com/ros2/examples/tree/master/rclcpp/executors/cbg_executor](https://github.com/ros2/examples/tree/master/rclcpp/executors/cbg_executor)
+- [Basic investigation of priority of DDS-generated threads.](https://discourse.ros.org/uploads/short-url/p2fAAbrKHkrSqZ9oJkZNwOOf2Hi.pdf)

minimal_scheduling/burn_cpu_cycles.hpp

@@ -0,0 +1,61 @@
+// Copyright (c) 2020 Robert Bosch GmbH
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#ifndef MINIMAL_SCHEDULING__BURN_CPU_CYCLES_HPP_
+#define MINIMAL_SCHEDULING__BURN_CPU_CYCLES_HPP_
+
+#include <pthread.h>
+
+#include <chrono>
+#include <thread>
+
+// Original code taken from:
+// https://github.com/ros2/examples/blob/master/rclcpp/executors/cbg_executor/src/examples_rclcpp_cbg_executor/utilities.hpp
+// https://github.com/ros2/examples/blob/14c743af8bbca6a40e83da7a470c5332dce66d9d/rclcpp/executors/cbg_executor/src/examples_rclcpp_cbg_executor/pong_node.cpp#L75-L88
+
+/// Returns the time of the given native thread handle as std::chrono
+/// timestamp. This allows measuring the execution time of this thread.
+template<typename T>
+std::chrono::nanoseconds get_native_thread_time(T native_handle)
+{
+  clockid_t id;
+  pthread_getcpuclockid(native_handle, &id);
+  timespec spec;
+  clock_gettime(id, &spec);
+  return std::chrono::seconds{spec.tv_sec} + std::chrono::nanoseconds{spec.tv_nsec};
+}
+
+/// Returns the time of the current thread as std::chrono timestamp.
+/// This allows measuring the execution time of this thread.
+inline std::chrono::nanoseconds get_current_thread_time()
+{
+  return get_native_thread_time(pthread_self());
+}
+
+void burn_cpu_cycles(std::chrono::nanoseconds duration)
+{
+  if (duration > std::chrono::nanoseconds::zero()) {
+    auto end_time = get_current_thread_time() + duration;
+    int x = 0;
+    bool do_again = true;
+    while (do_again) {
+      while (x != std::rand() && x % 1000 != 0) {
+        x++;
+      }
+      do_again = (get_current_thread_time() < end_time);
+    }
+  }
+}
+
+#endif  // MINIMAL_SCHEDULING__BURN_CPU_CYCLES_HPP_