Merge branch 'main' into release-test

Author: justincorrigible

.eslintignore

@@ -7,3 +7,5 @@ lerna-debug.log
 **/.DS_Store
 **/*.env*
 !**/.env.schema
+
+*.tsbuildinfo

.eslintrc.js

@@ -0,0 +1,103 @@
+# Contributing to Overture
+
+We appreciate your interest in contributing to this project. We are the Genome Informatics Software Engineering team from Ontario Institute for Cancer Research. At OICR we develop new software, databases and other necessary components to store, organize and process the large and complex datasets being generated by our cancer research programs. Embodying OICR's values of collaboration and community, we are firm believers in open-source and open-science. As such we strongly believe in the collective power of expertise and shared resources.
+
+## Code of Conduct
+
+By participating in this project, you are expected to abide by our [Code of Conduct](https://docs.overture.bio/community/code-of-conduct). Please take the time to read it carefully before contributing.
+
+## Get Involved
+
+**Getting Started:** Our primary platform for community support, feature requests, and general discussions is GitHub Discussions. This allows us to keep all conversations in one place and make them easily searchable for future reference.
+
+
+**Community Support:** Our primary platform for community support, feature requests, and general discussions is [GitHub Discussions](https://github.com/overture-stack/docs/discussions). This allows us to keep all conversations in one place and make them easily searchable for future reference.
+
+- **Getting Help:** If you need assistance with Overture, please create a [new discussion in our support category](https://github.com/overture-stack/docs/discussions/categories/support).
+  - Before creating a new discussion, please search existing discussions to see if your question has already been answered.
+- **Feature Requests & Proposals:** We love hearing your ideas for improving Overture! Before making a feature request, please check our [**feature roadmap**](https://github.com/orgs/overture-stack/projects/11/views/1) to see if your idea is already planned.
+  - If your idea isn't on the roadmap, feel free to [**submit a feature request**](https://github.com/overture-stack/docs/discussions/categories/ideas) by creating a new discussion in our Ideas category 
+- **Report a Potential Bug:** We use GitHub Issues primarily for tracking confirmed bugs and ticketing development tasks. If you come across a potential bug or issue, please first post it to our [**GitHub support discussion forum**](https://github.com/overture-stack/docs/discussions/categories/support).
+  - This allows us to confirm the issue and gather more information if needed. If we determine that further development is required, we will create and tag you into a GitHub Issue from your discussion post.
+
+## Our Development Process
+
+We use GitHub issues and pull requests for communication related to code changes. 
+
+### Branch Organization
+
+We use the following standard branches:
+
+- `main` is for stable production code
+- `develop` is the integration branch for new features
+- `feature/<name>` for feature branches
+- `release/v<version>` for release branches
+- `hotfix/<name>` for hotfix branches
+
+## Pull Requests
+
+### Submitting a Pull Request
+
+We welcome and encourage pull requests from the community. To submit a pull request, please follow these steps:
+
+1. **Fork the Repository**: Fork the Overture repository on GitHub.
+2. **Clone Your Fork**: Clone your forked repository to your local machine.
+3. **Create a New Branch**: Create a new branch for your changes. Use lowercase and hyphens (e.g., `feature/user-authentication`). Include ticket/issue numbers when applicable (e.g., `feature/PROJ-123-user-authentication`).
+4. **Make Your Changes**: Implement your changes and commit them to your branch. Write clear, concise commit messages in present tense (e.g., "Add feature" not "Added feature"). Reference issue numbers in commits when applicable.
+5. **Push Your Changes**: Push your changes to your forked repository.
+6. **Submit a Pull Request**: Open a pull request against the main repository.
+
+### Best Practices
+
+1. **Keep PRs as small as possible:** Focus on one feature or bug fix per pull request. Break large changes into smaller, more manageable pieces making it easier for reviewers to understand and approve your changes.
+
+2. **Use descriptive titles:** Start with a verb (e.g., "Add", "Fix", "Update", "Refactor"), briefly summarize the main purpose of the PR and include the issue number if applicable (e.g., "Fix user authentication bug (#123)").
+
+3. **Describe how you tested it:** Explain the testing process you followed and mention any new automated tests you've added.
+
+4. **Provide a clear description:** Explain the purpose of your changes and list the main modifications you've made. Mention any potential side effects or areas that might need extra attention.
+
+5. **Link related issues:** Reference any related issues or pull requests. Use GitHub keywords to automatically link issues (e.g., "Closes #123", "Fixes #456").
+6. **Keep the PR's branch up-to-date:** Regularly rebase your branch on the latest main branch and resolve any merge conflicts promptly.
+
+7. **Respond to feedback:** Be open to suggestions and willing to make changes. Address all comments from reviewers. If you disagree with a suggestion, explain your reasoning politely.
+
+8. **Include documentation updates:** If your changes affect user-facing features, update or create and issue detailing the relevant changes need to the documentation. Where appropriate include inline comments for complex code sections.
+
+10. **Be patient:** Reviewers will likely be unable to respond immediately. However, feel free to follow up politely if you haven't received feedback after a reasonable time.
+
+### Using Draft Pull Requests
+
+Draft Pull Requests are an excellent way to document work in progress and facilitate early feedback. Use them to:
+
+- Organize your thoughts and process
+- Share early work and ideas with the team
+- Get feedback on implementation approaches before finalizing code
+- Track progress on long-running features
+
+Guidelines for Draft Pull Requests:
+
+1. **Creation**:
+   - Open a pull request and select "Create draft pull request"
+   - Clearly mark the title with [WIP] or [DRAFT] prefix
+2. **Description**:
+   - Outline the current state of the work
+   - List planned tasks or improvements
+   - Highlight areas where feedback is specifically needed
+3. **Updates**:
+   - Regularly update the description or provide comments following commits with progress notes
+- Use task lists (using `- [ ]` in Markdown) to track completion of sub-tasks
+4. **Collaboration**:
+   - Encourage early feedback and discussion
+   - Use the pull request comments for design discussions
+5. **Finalization**:
+   - Complete all planned work and address feedback
+   - Update tests and documentation
+   - Click "Ready for review" to move out of draft state
+
+### Merging a Pull Request
+
+- Ensure all CI checks pass
+- Obtain the required number of approvals
+- Use the project's specified merge strategy (Typically squash and merge)
+- Delete the source branch after merging if no longer needed

.gitignore

@@ -1,2 +1,2 @@
 @Library(value='jenkins-pipeline-library@master', changelog=false) _
-pipelineOVERTUREArranger()
+pipelineOvertureArranger()

CONTRIBUTING.md

@@ -129,7 +129,7 @@ clear-es-documents:
 		-d '{"query":{"match_all":{}}}'
 
 seed-es:
-	@echo $(YELLOW)$(INFO_HEADER) "Initializing file_centric index" $(END)
+	@echo $(YELLOW)$(INFO_HEADER) "Initializing index" $(END)
 	@$(ES_LOAD_SCRIPT) $(ES_USER) $(ES_PASS) $(ES_HOST) $(ES_INDEX) $(ES_DATA_DIR) $(ES_DOCS_DIR)
 
 get-es-indices:

Jenkinsfile

@@ -1,99 +1,49 @@
-# Arranger - Data Portal API and UI component Generation
+# Arranger
 
-[<img hspace="5" src="https://img.shields.io/badge/chat-on--slack-blue?style=for-the-badge">](http://slack.overture.bio)
-[<img hspace="5" src="https://img.shields.io/badge/License-gpl--v3.0-blue?style=for-the-badge">](https://github.com/overture-stack/arranger/blob/develop/LICENSE)
-[<img hspace="5" src="https://img.shields.io/badge/Code%20of%20Conduct-2.1-blue?style=for-the-badge">](code_of_conduct.md)
-
-<div>
-<img align="right" width="120vw" src="icon-arranger.png" alt="arranger-logo"/>
-</div>
-
-Arranger integrates with your underlying Elasticsearch cluster to automatically generate a powerful search API based on your configured index mapping. It consists of two main modules, **Arranger Server** and **Arranger Components**.
-
-**Arranger Server** is a GraphQL API that communicates with an Elasticsearch index. Arranger Server uses a consistent and custom filter notation called SQON. SQON is designed to be user-friendly, allowing humans to easily understand and create custom filters while being straightforward for software systems to interpret and process.
-
-**Arranger Components** are interactive and configurable UI components specifically designed to display and query complex datasets from a web browser. 
-
-<!--Blockqoute-->
+Arranger is a versatile, data-agnostic GraphQL search API that leverages Elasticsearch, designed to simplify the process of creating powerful search interfaces for complex datasets. It's accompanied by its own React component library to generate interactive and highly configurable search UIs.
 
 </br>
 
-> 
-> <div>
-> <img align="left" src="ov-logo.png" height="90"/>
-> </div>
-> 
-> *Arranger is a core service within the [Overture](https://www.overture.bio/) research software ecosystem. See our [related products](#related-products) for more information on how Overture helps organize data and enable discovery.*
-> 
-> 
-
-<!--Blockqoute-->
+> <img align="left" src="ov-logo.png" height="50"/>
+>
+> _Arranger is part of [Overture](https://www.overture.bio/), a collection of open-source software microservices used to create platforms for researchers to organize and share genomics data._
 
 ## Documentation
 
-- For development resources, including set up and technical details, see our [Developer Documentation.](https://github.com/overture-stack/arranger/wiki)
-
-- For more information on installation, configuration, and usage see our [User Documentation.](https://www.overture.bio/documentation/arranger/installation/installation/)
-
-## Quickstart Docker Install
-
-To install Arranger using Docker, follow these steps:
+Technical resources for those working with or contributing to the project are available from our official documentation site, the following content can also be read and updated within the `/docs` folder of this repository.
 
-1. Clone the Arranger repository
-2. Navigate to the project directory
-3. With Docker running, execute the quickstart `make` target:
+- **[Arranger Overview](https://docs.overture.bio/docs/core-software/Arranger/overview)**
+- [**Setting up the Development Enviornment**](https://docs.overture.bio/docs/core-software/Arranger/setup)
+- [**Common Usage Docs**](https://docs.overture.bio/docs/core-software/Arranger/setup)
 
-```shell
-make start
-```
+## Development Environment
 
-The deployed services will be accessible through the following ports:
-
-| Service | Port |
-|--|--|
-| Arranger Server | localhost:5050/graphql |
-| Elasticsearch | localhost:9200 |
-| Kibana |  localhost:5601 |
-
-**Note(s):**
-
--  To generate the API Arranger requires an Elasticsearch Instance and an index mapping. For information on setting Arranger up with a default index mapping, see our [official documentation page on supplying an index mapping.](https://www.overture.bio/documentation/arranger/installation/configuration/es/)
-
-- Arranger is undergoing refactoring work for what will become version 3+. An _upgrade guide_ is on its way. Current users of v2 may look at the `legacy` branch, meanwhile, where we will continue fixing newly reported bugs until further notice.
-
-- You may need to add the `--legacy-peer-deps` flag when integrating our modules into your apps. Along with the rewrite, we're steadily updating the internal dependencies, so this isn't necessary.
+- [Node.js](https://nodejs.org/) (v22 or higher)
+- [Docker](https://www.docker.com/) (v4.39.0 or higher)
 
 ## Support & Contributions
 
-- Filing an [issue](https://github.com/overture-stack/arranger/issues)
-- Making a [contribution](CONTRIBUTING.md)
-- Connect with us on [Slack](https://overture-bio.slack.com/)
-- Add or Upvote a [feature request](https://github.com/overture-stack/arranger/issues?q=is%3Aopen+is%3Aissue+label%3Anew-feature+sort%3Areactions-%2B1-desc)
-
-## Related Products 
+- For support, feature requests, and bug reports, please see our [Support Guide](https://docs.overture.bio/community/support).
+- For detailed information on how to contribute to this project, please see our [Contributing Guide](https://docs.overture.bio/docs/contribution).
 
-<div>
-  <img align="right" alt="Overture overview" src="https://www.overture.bio/static/124ca0fede460933c64fe4e50465b235/a6d66/system-diagram.png" width="45%" hspace="5">
-</div>
+## Related Software
 
-Overture is an ecosystem of research software tools, each with narrow responsibilities, designed to reduce redundant programming efforts and enable developers and data scientists to build reliable systems that organize and share big data.
+The Overture platform includes the following components:
 
-All our core microservices are included in the Overture **Data Management System** (DMS). The DMS offers turnkey installation, configuration, and deployment of Overture software. For more information on the DMS, read our [DMS documentation](https://www.overture.bio/documentation/dms/).
+</br>
 
-See the links below for additional information on our other research software tools:
+| Software                                                | Description                                                                               |
+| ------------------------------------------------------- | ----------------------------------------------------------------------------------------- |
+| [Score](https://github.com/overture-stack/score/)       | Transfer data to and from any cloud-based storage system                                  |
+| [Song](https://github.com/overture-stack/song/)         | Catalogue and manage metadata associated to file data spread across cloud storage systems |
+| [Maestro](https://github.com/overture-stack/maestro/)   | Organizing your distributed data into a centralized Elasticsearch index                   |
+| [Arranger](https://github.com/overture-stack/arranger/) | A search API with reusable UI components                                                  |
+| [Stage](https://github.com/overture-stack/stage)        | A React-based web portal scaffolding                                                      |
+| [Lyric](https://github.com/overture-stack/lyric)        | A model-agnostic, tabular data submission system                                          |
+| [Lectern](https://github.com/overture-stack/lectern)    | Schema Manager, designed to validate, store, and manage collections of data dictionaries. |
 
-</br>
+If you'd like to get started using our platform [check out our quickstart guides](https://docs.overture.bio/guides/getting-started)
 
-|Software|Description|
-|---|---|
-|[Ego](https://www.overture.bio/products/ego/)|An authorization and user management service|
-|[Ego UI](https://www.overture.bio/products/ego-ui/)|A UI for managing Ego authentication and authorization services|
-|[Score](https://www.overture.bio/products/score/)| Transfer data to and from any cloud-based storage system|
-|[Song](https://www.overture.bio/products/song/)|Catalog and manage metadata associated to file data spread across cloud storage systems|
-|[Maestro](https://www.overture.bio/products/maestro/)|Organizing your distributed data into a centralized Elasticsearch index|
-|[Arranger](https://www.overture.bio/products/arranger/)|A search API with reusable UI components that build into configurable and functional data portals|
-|[DMS-UI](https://github.com/overture-stack/dms-ui)|A simple web browser UI that integrates Ego and Arranger|
-
-## Acknowledgements
+## Funding Acknowledgement
 
 Overture is supported by grant #U24CA253529 from the National Cancer Institute at the US National Institutes of Health, and additional funding from Genome Canada, the Canada Foundation for Innovation, the Canadian Institutes of Health Research, Canarie, and the Ontario Institute for Cancer Research.

Makefile

@@ -3,28 +3,29 @@ version: '3.7'
 services:
   elasticsearch:
     image: docker.elastic.co/elasticsearch/elasticsearch:7.17.1
-
+    container_name: elasticsearch.local
     environment:
       discovery.type: single-node
       cluster.name: workflow.elasticsearch
       ES_JAVA_OPTS: -Xms512m -Xmx2048m
       ELASTIC_PASSWORD: '$ES_PASS'
       xpack.security.enabled: 'true'
+    healthcheck:
+      test: 'curl --silent --fail localhost:9200/_cluster/health?wait_for_status=yellow&timeout=50s || exit 1'
+      interval: 1m30s
+      timeout: 50s
+      retries: 5
     logging:
       driver: 'json-file'
       options:
         max-size: '50m'
         max-file: '10'
+    networks:
+      - arranger
     ports:
       - 9200:9200
       - 9300:9300
 
-    healthcheck:
-      test: 'curl --silent --fail localhost:9200/_cluster/health?wait_for_status=yellow&timeout=50s || exit 1'
-      interval: 1m30s
-      timeout: 50s
-      retries: 5
-
   kibana:
     image: docker.elastic.co/kibana/kibana:7.17.1
 
@@ -42,7 +43,7 @@ services:
     ports:
       - 5601:5601
 
-  arranger-server:
+  server:
     build:
       context: ./
       dockerfile: ./docker/Dockerfile.local
@@ -51,9 +52,21 @@ services:
     environment:
       ENABLE_LOGS: '${ENABLE_LOGS:-false}'
       ES_HOST: '${ES_HOST:-http://host.docker.internal:9200}'
-      ES_USER: '$ES_USER'
-      ES_PASS: '$ES_PASS'
+      ES_USER: '${ES_USER:-elastic}'
+      ES_PASS: '${ES_PASS:-badpassword}'
+    networks:
+      - arranger
     ports:
       - 5050:5050
     volumes:
       - '${CONFIG_PATH:-./docker/server}:/app/modules/server/configs'
+
+  ui:
+    image: ghcr.io/overture-stack/stage:edge
+    container_name: stage.local
+    depends_on:
+      - server
+    networks:
+      - arranger
+    ports:
+      - '3000:3000'

README.md

@@ -8,7 +8,7 @@ ARG ES_HOST=http://localhost:9200
 #######################################################
 # Builder
 #######################################################
-FROM node:18-alpine as builder
+FROM node:22-alpine as builder
 
 ARG APP_FOLDER
 ARG APP_GID
@@ -43,4 +43,4 @@ USER $APP_USER
 
 EXPOSE 5050
 
-CMD make _ping_elasticsearch_server -e && npm run server:prod
+CMD make _ping_elasticsearch_server -e && npm run server