Skip to content

Commit d149bbf

Browse files
committed
docs: fix more urls in vignette
1 parent 067df4f commit d149bbf

File tree

1 file changed

+23
-23
lines changed

1 file changed

+23
-23
lines changed

vignettes/batchtools.Rmd

Lines changed: 23 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -34,16 +34,16 @@ Furthermore, you may provide functions to list queued/running jobs and to kill j
3434
Usually you do not have to start from scratch but can just use one of the cluster functions which ship with the package:
3535

3636

37-
* Interactive Cluster Functions (default): [docs](https://mlr-org.github.io/batchtools/reference/makeClusterFunctionsInteractive), [implementation](https://github.com/mllg/batchtools/blob/master/R/clusterFunctionsInteractive.R)
38-
* Multicore Cluster Functions: [docs](https://mlr-org.github.io/batchtools/reference/makeClusterFunctionsMulticore), [implementation](https://github.com/mllg/batchtools/blob/master/R/clusterFunctionsMulticore.R)
39-
* Socket Cluster Functions: [docs](https://mlr-org.github.io/batchtools/reference/makeClusterFunctionsSocket), [implementation](https://github.com/mllg/batchtools/blob/master/R/clusterFunctionsSocket.R)
40-
* Makeshift SSH cluster: [docs](https://mlr-org.github.io/batchtools/reference/makeClusterFunctionsSSH), [implementation](https://github.com/mllg/batchtools/blob/master/R/clusterFunctionsSSH.R)
41-
* Docker Swarm: [docs](https://mlr-org.github.io/batchtools/reference/makeClusterFunctionsDocker), [implementation](https://github.com/mllg/batchtools/blob/master/R/clusterFunctionsDocker.R)
42-
* IBM Spectrum Load Sharing Facility (LSF): [docs](https://mlr-org.github.io/batchtools/reference/makeClusterFunctionsLSF), [implementation](https://github.com/mllg/batchtools/blob/master/R/clusterFunctionsLSF.R)
43-
* OpenLava: [docs](https://mlr-org.github.io/batchtools/reference/makeClusterFunctionsOpenLava), [implementation](https://github.com/mllg/batchtools/blob/master/R/clusterFunctionsOpenLava.R)
44-
* Univa Grid Engine / Oracle Grid Engine (OGE) / Sun Grid Engine (SGE): [docs](https://mlr-org.github.io/batchtools/reference/makeClusterFunctionsSGE), [implementation](https://github.com/mllg/batchtools/blob/master/R/clusterFunctionsSGE.R)
45-
* Slurm: [docs](https://mlr-org.github.io/batchtools/reference/makeClusterFunctionsSlurm), [implementation](https://github.com/mllg/batchtools/blob/master/R/clusterFunctionsSlurm.R)
46-
* TORQUE/OpenPBS: [docs](https://mlr-org.github.io/batchtools/reference/makeClusterFunctionsTORQUE), [implementation](https://github.com/mllg/batchtools/blob/master/R/clusterFunctionsTORQUE.R)
37+
* Interactive Cluster Functions (default): [docs](https://mlr-org.github.io/batchtools/reference/makeClusterFunctionsInteractive), [implementation](https://github.com/mlr-org/batchtools/blob/master/R/clusterFunctionsInteractive.R)
38+
* Multicore Cluster Functions: [docs](https://mlr-org.github.io/batchtools/reference/makeClusterFunctionsMulticore), [implementation](https://github.com/mlr-org/batchtools/blob/master/R/clusterFunctionsMulticore.R)
39+
* Socket Cluster Functions: [docs](https://mlr-org.github.io/batchtools/reference/makeClusterFunctionsSocket), [implementation](https://github.com/mlr-org/batchtools/blob/master/R/clusterFunctionsSocket.R)
40+
* Makeshift SSH cluster: [docs](https://mlr-org.github.io/batchtools/reference/makeClusterFunctionsSSH), [implementation](https://github.com/mlr-org/batchtools/blob/master/R/clusterFunctionsSSH.R)
41+
* Docker Swarm: [docs](https://mlr-org.github.io/batchtools/reference/makeClusterFunctionsDocker), [implementation](https://github.com/mlr-org/batchtools/blob/master/R/clusterFunctionsDocker.R)
42+
* IBM Spectrum Load Sharing Facility (LSF): [docs](https://mlr-org.github.io/batchtools/reference/makeClusterFunctionsLSF), [implementation](https://github.com/mlr-org/batchtools/blob/master/R/clusterFunctionsLSF.R)
43+
* OpenLava: [docs](https://mlr-org.github.io/batchtools/reference/makeClusterFunctionsOpenLava), [implementation](https://github.com/mlr-org/batchtools/blob/master/R/clusterFunctionsOpenLava.R)
44+
* Univa Grid Engine / Oracle Grid Engine (OGE) / Sun Grid Engine (SGE): [docs](https://mlr-org.github.io/batchtools/reference/makeClusterFunctionsSGE), [implementation](https://github.com/mlr-org/batchtools/blob/master/R/clusterFunctionsSGE.R)
45+
* Slurm: [docs](https://mlr-org.github.io/batchtools/reference/makeClusterFunctionsSlurm), [implementation](https://github.com/mlr-org/batchtools/blob/master/R/clusterFunctionsSlurm.R)
46+
* TORQUE/OpenPBS: [docs](https://mlr-org.github.io/batchtools/reference/makeClusterFunctionsTORQUE), [implementation](https://github.com/mlr-org/batchtools/blob/master/R/clusterFunctionsTORQUE.R)
4747

4848

4949
To use the package with the socket cluster functions, you would call the respective constructor [makeClusterFunctionsSocket()](https://mlr-org.github.io/batchtools/reference/makeClusterFunctionsSocket):
@@ -109,7 +109,7 @@ The development of [BatchJobs](https://github.com/tudo-r/BatchJobs/) and [BatchE
109109
* batchtools does not use SQLite anymore.
110110
Instead, all the information is stored directly in the registry using [data.tables](https://cran.r-project.org/package=data.table) acting as an in-memory database. As a side effect, many operations are much faster.
111111
* Nodes do not have to access the registry.
112-
[submitJobs()](https://mlr-org.github.io/batchtools/reference/submitJobs) stores a temporary object of type [JobCollection](https://mllg.github.io/batchtools/reference/JobCollection) on the file system which holds all the information necessary to execute a chunk of jobs via [doJobCollection()](https://mllg.github.io/batchtools/reference/doJobCollection) on the node.
112+
[submitJobs()](https://mlr-org.github.io/batchtools/reference/submitJobs) stores a temporary object of type [JobCollection](https://mlr-org.github.io/batchtools/reference/JobCollection) on the file system which holds all the information necessary to execute a chunk of jobs via [doJobCollection()](https://mlr-org.github.io/batchtools/reference/doJobCollection) on the node.
113113
This avoids file system locks because each job accesses only one file exclusively.
114114
* `ClusterFunctionsMulticore` now uses the parallel package for multicore execution.
115115
* `ClusterFunctionsSSH` can still be used to emulate a scheduler-like system which respects the work load on the local machine.
@@ -234,15 +234,15 @@ getStatus()
234234
```
235235
The resulting output includes the number of jobs in the registry, how many have been submitted, have started to execute on the batch system, are currently running, have successfully completed, and have terminated due to an R exception.
236236
After jobs have successfully terminated, we can load their results on the master.
237-
This can be done in a simple fashion by using either [`loadResult()`](https://mlr-org.github.io/batchtools/reference/loadResult), which returns a single result exactly in the form it was calculated during mapping, or by using [`reduceResults()`](https://mllg.github.io/batchtools/reference/reduceResults), which is a version of `Reduce()` from the base package for registry objects.
237+
This can be done in a simple fashion by using either [`loadResult()`](https://mlr-org.github.io/batchtools/reference/loadResult), which returns a single result exactly in the form it was calculated during mapping, or by using [`reduceResults()`](https://mlr-org.github.io/batchtools/reference/reduceResults), which is a version of `Reduce()` from the base package for registry objects.
238238
```{r}
239239
waitForJobs()
240240
mean(sapply(1:10, loadResult))
241241
reduceResults(function(x, y) x + y) / 10
242242
```
243243

244244
If you are absolutely sure that your function works, you can take a shortcut and use *batchtools* in an `lapply` fashion using [`btlapply()`](https://mlr-org.github.io/batchtools/reference/btlapply).
245-
This function creates a temporary registry (but you may also pass one yourself), calls [`batchMap()`](https://mlr-org.github.io/batchtools/reference/reduceResultsList), wait for the jobs to terminate with [`waitForJobs()`](https://mllg.github.io/batchtools/reference/waitForJobs) and then uses [`reduceResultsList()`](https://mllg.github.io/batchtools/reference/reduceResultsList) to return the results.
245+
This function creates a temporary registry (but you may also pass one yourself), calls [`batchMap()`](https://mlr-org.github.io/batchtools/reference/reduceResultsList), wait for the jobs to terminate with [`waitForJobs()`](https://mlr-org.github.io/batchtools/reference/waitForJobs) and then uses [`reduceResultsList()`](https://mlr-org.github.io/batchtools/reference/reduceResultsList) to return the results.
246246

247247
```{r, R.options=list(batchtools.verbose=FALSE)}
248248
res = btlapply(rep(1e5, 10), piApprox)
@@ -393,7 +393,7 @@ If everything turns out fine, we can proceed with the calculation.
393393

394394
## Submitting and Collecting Results
395395

396-
To submit the jobs, we call [`submitJobs()`](https://mlr-org.github.io/batchtools/reference/submitJobs) and wait for all jobs to terminate using [`waitForJobs()`](https://mllg.github.io/batchtools/reference/waitForJobs).
396+
To submit the jobs, we call [`submitJobs()`](https://mlr-org.github.io/batchtools/reference/submitJobs) and wait for all jobs to terminate using [`waitForJobs()`](https://mlr-org.github.io/batchtools/reference/waitForJobs).
397397
```{r}
398398
submitJobs()
399399
waitForJobs()
@@ -464,12 +464,12 @@ After you have submitted jobs and suspect that something is going wrong, the fir
464464
getStatus()
465465
```
466466
The status message shows that two of the jobs could not be executed successfully.
467-
To get the IDs of all jobs that failed due to an error we can use [`findErrors()`](https://mlr-org.github.io/batchtools/reference/findJobs) and to retrieve the actual error message, we can use [`getErrorMessages()`](https://mllg.github.io/batchtools/reference/getErrorMessages).
467+
To get the IDs of all jobs that failed due to an error we can use [`findErrors()`](https://mlr-org.github.io/batchtools/reference/findJobs) and to retrieve the actual error message, we can use [`getErrorMessages()`](https://mlr-org.github.io/batchtools/reference/getErrorMessages).
468468
```{r}
469469
findErrors()
470470
getErrorMessages()
471471
```
472-
If we want to peek into the R log file of a job to see more context for the error we can use [`showLog()`](https://mlr-org.github.io/batchtools/reference/showLog) which opens a pager or use [`getLog()`](https://mllg.github.io/batchtools/reference/showLog) to get the log as character vector:
472+
If we want to peek into the R log file of a job to see more context for the error we can use [`showLog()`](https://mlr-org.github.io/batchtools/reference/showLog) which opens a pager or use [`getLog()`](https://mlr-org.github.io/batchtools/reference/showLog) to get the log as character vector:
473473
```{r}
474474
tail(getLog(id = 9))
475475
```
@@ -484,21 +484,21 @@ grepLogs(pattern = "simple", ignore.case = TRUE)
484484

485485
## On the Local System
486486

487-
1. Create a Registry with [`makeRegistry()`](https://mlr-org.github.io/batchtools/reference/makeRegistry) (or [`makeExperimentRegistry()`](https://mllg.github.io/batchtools/reference/makeExperimentRegistry)) or load an existing from the file system with [`loadRegistry()`](https://mllg.github.io/batchtools/reference/loadRegistry).
488-
2. Define computational jobs with [`batchMap()`](https://mlr-org.github.io/batchtools/reference/batchMap) or [`batchReduce()`](https://mllg.github.io/batchtools/reference/batchReduce) if you used [`makeRegistry()`](https://mllg.github.io/batchtools/reference/makeRegistry) or define with [`addAlgorithm()`](https://mllg.github.io/batchtools/reference/addAlgorithm), [`addProblem()`](https://mllg.github.io/batchtools/reference/addProblem) and [`addExperiments()`](https://mllg.github.io/batchtools/reference/addExperiments) if you started with [`makeExperimentRegistry()`](https://mllg.github.io/batchtools/reference/makeExperimentRegistry).
487+
1. Create a Registry with [`makeRegistry()`](https://mlr-org.github.io/batchtools/reference/makeRegistry) (or [`makeExperimentRegistry()`](https://mlr-org.github.io/batchtools/reference/makeExperimentRegistry)) or load an existing from the file system with [`loadRegistry()`](https://mlr-org.github.io/batchtools/reference/loadRegistry).
488+
2. Define computational jobs with [`batchMap()`](https://mlr-org.github.io/batchtools/reference/batchMap) or [`batchReduce()`](https://mlr-org.github.io/batchtools/reference/batchReduce) if you used [`makeRegistry()`](https://mlr-org.github.io/batchtools/reference/makeRegistry) or define with [`addAlgorithm()`](https://mlr-org.github.io/batchtools/reference/addAlgorithm), [`addProblem()`](https://mllg.github.io/batchtools/reference/addProblem) and [`addExperiments()`](https://mllg.github.io/batchtools/reference/addExperiments) if you started with [`makeExperimentRegistry()`](https://mllg.github.io/batchtools/reference/makeExperimentRegistry).
489489
It is advised to test some jobs with [`testJob()`](https://mlr-org.github.io/batchtools/reference/testJob) in the interactive session and with `testJob(external = TRUE)` in a separate R process.
490490
Note that you can add additional jobs if you are using an [`ExperimentRegistry`](https://mlr-org.github.io/batchtools/reference/makeExperimentRegistry).
491491
3. If required, query the data base for job ids depending on their status, parameters or tags (see [`findJobs()`](https://mlr-org.github.io/batchtools/reference/findJobs)).
492-
The returned tables can easily be combined in a set-like fashion with data base verbs: union ([`ojoin()`](https://mlr-org.github.io/batchtools/reference/JoinTables) for outer join), intersect ([`ijoin()`](https://mllg.github.io/batchtools/reference/JoinTables) for inner join), difference ([`ajoin()`](https://mllg.github.io/batchtools/reference/JoinTables) for anti join).
492+
The returned tables can easily be combined in a set-like fashion with data base verbs: union ([`ojoin()`](https://mlr-org.github.io/batchtools/reference/JoinTables) for outer join), intersect ([`ijoin()`](https://mlr-org.github.io/batchtools/reference/JoinTables) for inner join), difference ([`ajoin()`](https://mlr-org.github.io/batchtools/reference/JoinTables) for anti join).
493493
4. Submit jobs with [`submitJobs()`](https://mlr-org.github.io/batchtools/reference/submitJobs). You can specify job resources here.
494494
If you have thousands of fast terminating jobs, you want to [`chunk()`](https://mlr-org.github.io/batchtools/reference/chunk) them first.
495-
If some jobs already terminated, you can estimate the runtimes with [`estimateRuntimes()`](https://mlr-org.github.io/batchtools/reference/estimateRuntimes) and chunk jobs into heterogeneous groups with [`lpt()`](https://mllg.github.io/batchtools/reference/chunk) and [`binpack()`](https://mllg.github.io/batchtools/reference/chunk).
495+
If some jobs already terminated, you can estimate the runtimes with [`estimateRuntimes()`](https://mlr-org.github.io/batchtools/reference/estimateRuntimes) and chunk jobs into heterogeneous groups with [`lpt()`](https://mlr-org.github.io/batchtools/reference/chunk) and [`binpack()`](https://mlr-org.github.io/batchtools/reference/chunk).
496496
5. Monitor jobs. [`getStatus()`](https://mlr-org.github.io/batchtools/reference/getStatus) gives a summarizing overview.
497-
Use [`showLog()`](https://mlr-org.github.io/batchtools/reference/showLog) and [`grepLogs()`](https://mllg.github.io/batchtools/reference/grepLogs) to investigate log file.
497+
Use [`showLog()`](https://mlr-org.github.io/batchtools/reference/showLog) and [`grepLogs()`](https://mlr-org.github.io/batchtools/reference/grepLogs) to investigate log file.
498498
Run jobs in the currently running session with [`testJob()`](https://mlr-org.github.io/batchtools/reference/testJob) to get a `traceback()`.
499499
6. Collect (partial) results. [`loadResult()`](https://mlr-org.github.io/batchtools/reference/loadResult) retrieves a single result from the file system.
500500
[`reduceResults()`](https://mlr-org.github.io/batchtools/reference/reduceResults) mimics `Reduce()` and allows to apply a function to many files in an iterative fashion.
501-
[`reduceResultsList()`](https://mlr-org.github.io/batchtools/reference/reduceResultsList) and [`reduceResultsDataTable()`](https://mllg.github.io/batchtools/reference/reduceResultsList) collect results into a `list` or `data.table`, respectively.
501+
[`reduceResultsList()`](https://mlr-org.github.io/batchtools/reference/reduceResultsList) and [`reduceResultsDataTable()`](https://mlr-org.github.io/batchtools/reference/reduceResultsList) collect results into a `list` or `data.table`, respectively.
502502

503503
```{r,echo=FALSE}
504504
knitr::include_graphics("function_overview.png", auto_pdf = TRUE)
@@ -507,7 +507,7 @@ knitr::include_graphics("function_overview.png", auto_pdf = TRUE)
507507
## On Multiple Systems
508508

509509
Most users develop and prototype their experiments on a desktop box in their preferred IDE and later deploy to a large computing cluster.
510-
This can be done by prototyping locally ([`testJob()`](https://mlr-org.github.io/batchtools/reference/testJob) or submit subsets via [`submitJobs()`](https://mllg.github.io/batchtools/reference/submitJobs)).
510+
This can be done by prototyping locally ([`testJob()`](https://mlr-org.github.io/batchtools/reference/testJob) or submit subsets via [`submitJobs()`](https://mlr-org.github.io/batchtools/reference/submitJobs)).
511511
To deploy to the cluster, just copy the file directory (as reported by `reg$file.dir`) to the remote system.
512512
Next, log in on the cluster (typically via `ssh`), `cd` to the copied directory and call `loadRegistry("<file.dir.on.remote">, "<work.dir.on.remote>", writeable = TRUE)`.
513513
This function will (a) source the local configuration file so that you can talk to the cluster (verify by checking the output of `reg$cluster.functions`) and (b) adjust the paths to the new system if argument `update.paths` is set.

0 commit comments

Comments
 (0)