WIP: Add pipeline setup scripts and platform-aware tool install #51
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Added .gitignore, Makefile, install.sh, odk.sh, and config/prefixes.csv to support pipeline setup and execution. Enhanced run_pipeline.py to detect OS/architecture and download the correct DuckDB and yq binaries, improving cross-platform compatibility.
Refactor Makefile and run_pipeline.py to use a two-level directory structure: a shared working directory for tools and a run-specific subdirectory for data and results. Add --run-name option to customize the run directory, defaulting to the current date. Update all file operations to use the run directory, improve tool installation logic, and clarify documentation and help messages.
Updated the --comparison argument in run_pipeline.py to accept multiple values, enabling users to run several comparisons in a single command. Output filenames are now stable and no longer include version numbers. Refactored main logic to handle multiple or all comparisons as specified by the user.
Add pyproject.toml with dependencies and configuration, create the monarch_semsim package with CLI and exomiserdb modules, and implement a CLI command to convert semantic similarity files to Exomiser SQL format.
Renamed 'custom-phenio-%' Makefile target to 'semsim-phenio-%' and added new targets for downloading OLS embedding files. Updated SQL generation rules to use new working directory and simplified output filenames by removing version suffixes. Adjusted help text and output file naming in run_pipeline.py to match these changes.
Updated the _format_row function to provide default values for optional fields such as jaccard_similarity, ancestor_information_content, phenodigm_score, and ancestor_id. This improves robustness when input data is missing these fields.
Introduces a --threshold option to the CLI and underlying functions for filtering results by minimum score when generating Exomiser SQL files. Updates the Makefile to use a default threshold of 0.5 and refactors code to centralize score extraction and filtering logic.
Contributor
|
Would you like to change the project name to |
Author
|
@caufieldjh I dont know if that is necessary - let us come back to the naming once I am done with implementing everything |
Updated the Makefile to use tar.gz archives for intermediate results, standardized target names, and improved SQL import/compaction steps. Refactored exomiserdb.py to use CURIE formatting, enforce label length limits, and add stricter ID validation for Exomiser SQL output. These changes improve robustness, compatibility, and maintainability of the phenotype comparison pipeline.
Introduces a --random CLI option to overwrite score values with random numbers in a specified range during conversion to Exomiser SQL format. Updates internal functions to support random score generation and improves label handling and CURIE formatting for robustness.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Added .gitignore, Makefile, install.sh, odk.sh, and config/prefixes.csv to support pipeline setup and execution. Enhanced run_pipeline.py to detect OS/architecture and download the correct DuckDB and yq binaries, improving cross-platform compatibility.