macapptree

macapptree is a Python package that extracts the accessibility tree of a macOS applications screen in JSON format. It also provides an option to capture screenshots of the application, including labeled bounding boxes with different colors representing various element types. This tool is useful for accessibility testing, UI automation, and visual debugging. You can also capture the accessibility of all currently running, visible apps and include the Dock and top Menu Bar.

---

Features

• Accessibility Tree Extraction of Apps: Retrieve the accessibility hierarchy of a single or multiple macOS applications in JSON format.
• Screenshot Capture: Capture a cropped screenshot of the application window and full screen.
• Labeled Visual Output: Generate a segmented screenshot with bounding boxes highlighting UI elements, colored by their types.

---

Installation

To install macapptree, use pip:

pip install macapptree

Note: This package requires macOS and Python 3.8+.

---

Usage

The library provides two primary functions:

• get_tree: Extracts the accessibility tree of a macOS app.
• get_tree_screenshot: Extracts the accessibility tree and generates screenshots.

Example

from macapptree import get_tree, get_tree_screenshot, get_app_bundle

# Get the app bundle identifier (for example: "com.apple.TextEdit")
bundle = get_app_bundle("TextEdit")

# Retrieve the accessibility tree, a cropped screenshot, and a segmented screenshot
tree, im, im_seg = get_tree_screenshot(bundle)

# `tree`: JSON-like structure of the accessibility elements
# `im`: Cropped screenshot of the application window
# `im_seg`: Labeled screenshot with bounding boxes indicating UI elements

CLI Example (multi-app)

Capture the accessibility of all currently running and visible apps (with upper menu and dock included):

python -m macapptree.main \
  --oa shots/dump.json \
  --os shots \
  --all-apps \
  --include-menubar \
  --include-dock

Or specify apps explicitly:

python -m macapptree.main \
  -a com.apple.Safari com.google.Chrome \
  --oa shots/dump.json \
  --os shots \
  --include-menubar \
  --include-dock

Output

• tree: A Python dictionary representing the accessibility hierarchy.
• im: A cropped PIL.Image object of the app window.
• im_seg: A PIL.Image object with bounding boxes drawn on top, colored based on the element type.

---

Example Tree Output

{
    "id": "9d72c04ce9df11c8ab938ead88723de1",
    "name": "Untitled",
    "role": "AXWindow",
    "description": null,
    "role_description": "standard window",
    "value": null,
    "absolute_position": "214.00;119.00",
    "position": "0.00;0.00",
    "size": "586;476",
    "enabled": false,
    "bbox": [
        0,
        0,
        586,
        476
    ],
    "visible_bbox": [
        0,
        0,
        586,
        476
    ],
    "children": [
        {
            "id": "422f5e0df37aa872341d3b6a47faf320",
            "name": null,
            "role": "AXScrollArea",
            "description": null,
            "role_description": "scroll area",
            "value": null,
            "absolute_position": "214.00;175.00",
            "position": "0.00;56.00",
            "size": "586;420",
            "enabled": false,
            "bbox": [
                0,
                56,
                586,
                476
            ],
            "visible_bbox": [
                0,
                56,
                586,
                476
            ],
            "children": [
                ...

---

License

This project is licensed under the MIT License. See the LICENSE file for details.

MacPaw Research

Visit our site to learn more 😉

https://research.macpaw.com