Merge branch 'stackhpc/2025.1' into pulp-tls-update

Author: jackhodgkiss

doc/source/configuration/wazuh.rst

@@ -205,8 +205,27 @@ Reinstall the role if required:
 
 ``kayobe control host bootstrap``
 
+Secrets
+-------
+
+Wazuh requires that secrets or passwords are set for itself and the services with which it communicates.
+Wazuh secrets playbook is located in ``$KAYOBE_CONFIG_PATH/ansible/deployment/wazuh-secrets.yml``.
+Running this playbook will generate and put pertinent security items into secrets
+vault file which will be placed in ``$KAYOBE_CONFIG_PATH/deployment/wazuh-secrets.yml``.
+If using environments it ends up in ``$KAYOBE_CONFIG_PATH/environments/<env_name>/deployment/wazuh-secrets.yml``
+Remember to encrypt!
+
+Wazuh secrets template is located in ``$KAYOBE_CONFIG_PATH/ansible/templates/wazuh-secrets.yml.j2``.
+It will be used by wazuh secrets playbook to generate wazuh secrets vault file.
 
-Edit the playbook and variables to your needs:
+
+.. code-block:: console
+
+  kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/deployment/wazuh-secrets.yml
+
+.. note:: Use ``ansible-vault`` to view the secrets:
+
+  ``ansible-vault view --vault-password-file ~/vault.password $KAYOBE_CONFIG_PATH/inventory/group_vars/wazuh-manager/deployment/wazuh-secrets.yml``
 
 Wazuh manager configuration
 ---------------------------
@@ -242,28 +261,6 @@ You may need to modify some of the variables, including:
 
 You'll need to run ``wazuh-manager.yml`` playbook again to apply customisation.
 
-Secrets
--------
-
-Wazuh requires that secrets or passwords are set for itself and the services with which it communiticates.
-Wazuh secrets playbook is located in ``$KAYOBE_CONFIG_PATH/ansible/deployment/wazuh-secrets.yml``.
-Running this playbook will generate and put pertinent security items into secrets
-vault file which will be placed in ``$KAYOBE_CONFIG_PATH/deployment/wazuh-secrets.yml``.
-If using environments it ends up in ``$KAYOBE_CONFIG_PATH/environments/<env_name>/deployment/wazuh-secrets.yml``
-Remember to encrypt!
-
-Wazuh secrets template is located in ``$KAYOBE_CONFIG_PATH/ansible/templates/wazuh-secrets.yml.j2``.
-It will be used by wazuh secrets playbook to generate wazuh secrets vault file.
-
-
-.. code-block:: console
-
-  kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/deployment/wazuh-secrets.yml
-
-.. note:: Use ``ansible-vault`` to view the secrets:
-
-  ``ansible-vault view --vault-password-file ~/vault.password $KAYOBE_CONFIG_PATH/inventory/group_vars/wazuh-manager/deployment/wazuh-secrets.yml``
-
 Configure Wazuh Dashboard's Server Host
 ---------------------------------------
 
@@ -424,6 +421,13 @@ Verification
 The Wazuh agents should register with the Wazuh manager. This can be verified via the agents page in Wazuh Portal.
 Check CIS benchmark output in agent section.
 
+Wazuh manager removal
+---------------------
+
+The following playbook can be used to purge all Wazuh manager components from a host. This is particularly useful for Wazuh manager servers that are not hosted on an infra-vm.
+
+``kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/tools/wazuh-manager-purge.yml``
+
 Additional resources
 --------------------
 

doc/source/operations/bifrost-hardware-inventory-management.rst

@@ -83,7 +83,7 @@ Replacing a Failing Hypervisor
 
 To replace a failing hypervisor, proceed as follows:
 
-* :ref:`Disable the hypervisor to avoid scheduling any new instance on it <taking-a-hypervisor-out-of-service>`
+* :ref:`Disable the hypervisor to avoid scheduling any new instances on it <taking-a-hypervisor-out-of-service>`
 * :ref:`Evacuate all instances <evacuating-all-instances>`
 * :ref:`Set the node to maintenance mode in Bifrost <set-bifrost-maintenance-mode>`
 * Physically fix or replace the node
@@ -102,6 +102,54 @@ To deprovision an existing hypervisor, run:
    system. Running this command without a limit will deprovision all overcloud
    hosts.
 
+Removing a Hypervisor
+---------------------
+
+To remove a hypervisor without replacing it, proceed as follows:
+
+#. :ref:`Disable the hypervisor to avoid scheduling any new instances on it <taking-a-hypervisor-out-of-service>`
+#. :ref:`Evacuate all instances <evacuating-all-instances>`
+#. (optionally) Deprovision the hypervisor
+
+   .. code-block:: console
+
+      kayobe overcloud deprovision --limit <Hypervisor hostname>
+
+   .. warning::
+
+      Always use ``--limit`` with ``kayobe overcloud deprovision`` on a production
+      system. Running this command without a limit will deprovision all overcloud
+      hosts.
+
+#. Physically remove the node from the deployment
+
+#. Delete the node in Bifrost:
+
+   .. code-block:: console
+
+      docker exec -it bifrost_deploy bash
+      (bifrost-deploy)[root@seed bifrost-base]# export OS_CLOUD=bifrost
+      (bifrost-deploy)[root@seed bifrost-base]# openstack baremetal node delete <Hostname>
+
+#. Delete the compute service in OpenStack:
+
+   .. code-block:: console
+
+      openstack compute service list | grep <Hypervisor hostname>
+      openstack compute service delete <Service ID>
+
+#. Delete the network agents in OpenStack:
+
+   .. code-block:: console
+
+      openstack network agent list | grep <Hypervisor hostname>
+      openstack network agent delete <Agent IDs>
+
+#. Remove the node from the Kayobe configuration. Ensure the node is removed
+   from the inventory and ``network-allocation.yml``. Other configuration files
+   may also be removed, but this is dependent on the deployment. Recursive
+   ``grep`` can help here.
+
 .. _evacuating-all-instances:
 
 Evacuating all instances

doc/source/operations/ubuntu-noble.rst

@@ -47,8 +47,10 @@ The following types of hosts will be covered in the future:
 
 .. warning::
 
-   Ceph node upgrades have not yet been performed outside of a virtualised test
-   environment. Proceed with caution.
+   Due to `Bug 66389 <https://tracker.ceph.com/issues/66389>`__, do not upgrade
+   Ceph hosts to Noble until the Ceph cluster has been upgraded to at least
+   Reef v18.2.5. Upgrading a host prematurely will prevent its Ceph daemons
+   from starting, and it will not be able to rejoin the cluster.
 
 Prerequisites
 =============
@@ -353,6 +355,8 @@ Storage
 Potential issues
 ----------------
 
+-  Ensure the Ceph cluster is running at least Reef v18.2.5.
+   Upgrading hosts with an older Ceph version will cause daemons to fail.
 -  It is recommended that you upgrade the bootstrap host last.
 -  Before upgrading the bootstrap host, it can be beneficial to backup
    ``/etc/ceph`` and ``/var/lib/ceph``, as sometimes the keys, config, etc.

doc/source/operations/upgrading-ceph.rst

@@ -9,6 +9,37 @@ The Ceph release series is not strictly dependent upon the StackHPC OpenStack
 release, however this configuration does define a default Ceph release series
 and container image tag. The default release series is currently |ceph_series|.
 
+Known issues
+============
+
+Slow ceph-volume activate
+-------------------------
+
+A large slowdown of ``ceph-volume activate`` has been reported on version
+19.2.3 (`bug 73107 <https://tracker.ceph.com/issues/73107>`__).
+
+On Reef, a host with 15 OSDs was measured taking around 10 seconds to activate
+all OSDs while exiting maintenance mode. On Squid 19.2.3, a host with 22 OSDs
+was measured taking 2 minutes to activate all OSDs.
+
+This bug may be a blocker for deployments with large number of OSDs per host.
+
+Elastic Shared Blob crash
+-------------------------
+
+There is a `known bug causing OSDs created on Squid to crash
+<https://tracker.ceph.com/issues/70390>`__. To avoid it, `disable the
+Elastic Shared Blob feature
+<https://docs.clyso.com/blog/#squid-deployed-osds-are-crashing>`__ before
+any OSDs are created or replaced:
+
+.. code-block:: bash
+
+   ceph config set osd bluestore_elastic_shared_blobs 0
+
+This needs to be done after the upgrade is complete as the option is not
+available on Reef.
+
 Prerequisites
 =============
 
@@ -171,6 +202,21 @@ versions``. Once confirmed, run the following command:
 
       ceph config set osd bluestore_elastic_shared_blobs 0
 
+Finally, verify the value of ``ceph osd get-require-min-compat-client``. On
+older Ceph deployments, it may still be set to ``jewel``, which would prevent
+using the `upmap balancer mode
+<https://docs.ceph.com/en/latest/rados/operations/balancer/#modes>`__ which
+requires ``luminous`` or later. Similarly, the more recent `read balancer
+<https://docs.ceph.com/en/latest/rados/operations/read-balancer/>`__ requires
+``reef``.
+
+Run ``ceph features`` to identify client versions and consider setting the
+minimum to an appropriate value:
+
+.. code-block:: console
+
+   ceph osd set-require-min-compat-client reef
+
 Upgrade Cephadm
 ===============
 

doc/source/operations/upgrading-openstack.rst

@@ -132,7 +132,7 @@ For example:
       enabled: "{{ seed_pulp_container_enabled | bool }}"
 
 Ansible playbook subdirectories
---------------------------------------
+-------------------------------
 
 The playbooks under ``etc/kayobe/ansible`` have been subdivided into different
 categories to make them easier to navigate. This change may result in merge
@@ -147,6 +147,10 @@ To mitigate the impact of these changes, two scripts have been added:
 * ``tools/magic-symlink-fix.sh`` - Uses the previous script to attempt to fix
   any broken symlinks in the kayobe configuration.
 
+If playbooks are referenced in different methods other than symlinks, they'll
+need to be manually resolved by operators. (e.g. Shell scripts running
+playbooks with file paths, ``import_playbook`` command in custom playbooks)
+
 Known issues
 ============
 
@@ -370,6 +374,14 @@ You can find more information from the :ref:`beokay` documentation.
    For Rocky Linux 9, ``beokay create`` must be used with the ``--python python3.12``
    option to specify Beokay to use Python 3.12 as it is not the default.
 
+Kayobe Automation
+~~~~~~~~~~~~~~~~~
+
+For deployments using Kayobe Automation CI, the Kayobe Docker image also needs
+to be rebuilt with Python 3.12. In GitHub, run the ``Build Kayobe Docker
+Image`` workflow. In GitLab, run the ``build_kayobe_image`` pipeline. In either
+case, the image will automatically be rebuilt with Python 3.12.
+
 Preparation
 ===========
 

etc/kayobe/ansible.cfg

@@ -1,6 +1,6 @@
 [defaults]
 forks = 20
-# Use the YAML stdout callback plugin.
+# Use YAML stdout callback output.
 callback_result_format = yaml
 # Use the stdout_callback when running ad-hoc commands.
 bin_ansible_callbacks = True

etc/kayobe/ansible/requirements.yml

@@ -1,7 +1,7 @@
 ---
 collections:
   - name: stackhpc.cephadm
-    version: 1.21.0
+    version: 1.22.0
   # NOTE: Pinning pulp.squeezer to 0.0.13 because 0.0.14+ depends on the
   # pulp_glue Python library being installed.
   - name: pulp.squeezer

etc/kayobe/ansible/tools/install-doca.yml

@@ -8,6 +8,7 @@
       ansible.builtin.command:
         cmd: "uname -r"
       register: kernel
+      check_mode: false
 
     - name: Install kernel repo
       ansible.builtin.dnf:

etc/kayobe/ansible/tools/wazuh-manager-purge.yml

@@ -0,0 +1,105 @@
+---
+# This is the playbook version of the wazuh purge tool from:
+# https://github.com/stackhpc/wazuh-server-purge
+
+- name: Purge Wazuh Server Components
+  hosts: wazuh-manager
+  become: true
+  become_user: root
+  tasks:
+# Dashboard
+    - name: Disable and stop wazuh-dashboard service
+      ansible.builtin.systemd_service:
+        name: wazuh-dashboard
+        state: stopped
+        enabled: no
+        daemon_reload: true
+      register: svc_result
+      failed_when:
+        - svc_result.failed
+        - "'Could not find the requested service' not in svc_result.msg"
+
+    - name: Remove wazuh-dashboard and files
+      ansible.builtin.package:
+        name: wazuh-dashboard
+        state: absent
+
+    - name: Remove wazuh-dashboard directories
+      ansible.builtin.file:
+        path: "{{ item }}"
+        state: absent
+      loop:
+        - /var/lib/wazuh-dashboard
+        - /usr/share/wazuh-dashboard
+        - /etc/wazuh-dashboard
+# Manager
+    - name: Remove wazuh-manager service
+      ansible.builtin.systemd_service:
+        name: wazuh-manager
+        state: stopped
+        enabled: no
+        daemon_reload: true
+      register: svc_result
+      failed_when:
+        - svc_result.failed
+        - "'Could not find the requested service' not in svc_result.msg"
+
+    - name: Remove wazuh-manager and files
+      ansible.builtin.package:
+        name: wazuh-manager
+        state: absent
+
+    - name: Remove wazuh-manager directories
+      ansible.builtin.file:
+        path: /var/ossec
+        state: absent
+# Filebeat
+    - name: Disable and stop filebeat service
+      ansible.builtin.systemd_service:
+        name: filebeat
+        state: stopped
+        enabled: no
+        daemon_reload: true
+      register: svc_result
+      failed_when:
+        - svc_result.failed
+        - "'Could not find the requested service' not in svc_result.msg"
+
+    - name: Remove filebeat and files
+      ansible.builtin.package:
+        name: filebeat
+        state: absent
+
+    - name: Remove filebeat directories
+      ansible.builtin.file:
+        path: "{{ item }}"
+        state: absent
+      loop:
+        - /var/lib/filebeat
+        - /usr/share/filebeat
+        - /etc/filebeat
+# Indexer
+    - name: Disable and stop wazuh-indexer service
+      ansible.builtin.systemd_service:
+        name: wazuh-indexer
+        state: stopped
+        enabled: no
+        daemon_reload: true
+      register: svc_result
+      failed_when:
+        - svc_result.failed
+        - "'Could not find the requested service' not in svc_result.msg"
+
+    - name: Remove wazuh-indexer and files
+      ansible.builtin.package:
+        name: wazuh-indexer
+        state: absent
+
+    - name: Remove wazuh-indexer directories
+      ansible.builtin.file:
+        path: "{{ item }}"
+        state: absent
+      loop:
+        - /var/lib/wazuh-indexer
+        - /usr/share/wazuh-indexer
+        - /etc/wazuh-indexer

etc/kayobe/environments/ci-aio/automated-setup.sh

@@ -255,7 +255,7 @@ generate_openstack_env_script() {
 
 python3 -m venv $BASE_PATH/venvs/openstack
 $BASE_PATH/venvs/openstack/bin/pip install -U pip
-$BASE_PATH/venvs/openstack/bin/pip install -U python-openstackclient -c https://raw.githubusercontent.com/stackhpc/requirements/refs/heads/stackhpc/stackhpc/$OPENSTACK_RELEASE/upper-constraints.txt
+$BASE_PATH/venvs/openstack/bin/pip install -U python-openstackclient -c https://raw.githubusercontent.com/stackhpc/requirements/refs/heads/stackhpc/$OPENSTACK_RELEASE/upper-constraints.txt
 cat > $BASE_PATH/openstack-env.sh <<EOF
 
 source $BASE_PATH/venvs/openstack/bin/activate