---
title: "Generating caches on a cluster"
author: "Nathan Sheffield"
date: "`r Sys.Date()`"
vignette: >
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteIndexEntry{Generating caches on a cluster}
output: knitr:::html_vignette
---

# Generating caches in parallel using batchtools

By default, `simpleCache` creates caches in the R session you use to call it. If you need to make lots of caches, or very large caches, you may want instead to sub these as jobs to a cluster resource manager (like SLURM). simpleCache can do this using functionality from the `batchtools` package.

This vignette is unevaluated because it relies on the `batchtools` package and a cluster environment.

To do this, first, create a `batchtools` registry. You can follow more detailed documentation in the `batchtools` package, but here's some code to get you started:

```{r Try it out, eval=FALSE}
library(simpleCache)
setCacheDir(tempdir())

registry = batchtools::makeRegistry(NA)
templateFile = system.file("templates/slurm-advanced.tmpl", package = "simpleCache")
registry$cluster.functions = batchtools::makeClusterFunctionsSlurm(
  template = templateFile)
registry
```

Notice that I'm using a custom slurm template here. With a registry in hand, we next need to define the resources this cache job will require:

```{r}
resources = list(ncpus=1, memory=1000, walltime=60, partition="serial")
```

Then, we simply add these as arguments to `simpleCache()` like so:
```{r, eval=FALSE}
simpleCache("testBatch", {
  rnorm(1e7, 0, 1)
  }, batchRegistry=registry, batchResources=resources)
```

This will now create and submit a job script to the cluster. That job script will have R code to create your `testBatch` cache by calling the code in your `simpleCache` call, `rnorm(1e7, 0, 1)`. Next time you run this function, it will just load the cache without recreating it, as you would expect simpleCache to do. Now there's a bunch of other stuff you can use `batchtools` to do with these jobs:

```{r, eval=FALSE}
batchtools::getJobTable(reg=registry)
batchtools::getJobPars()
batchtools::getStatus()

batchtools::getJobTable(reg=registry)
batchtools::getJobPars(1, reg=registry)
batchtools::loadResult(1, reg=registry)
# batchtools::testJob(1, reg=registry)
# killJobs()
```

When you're done, you may want to remove your temporary registry:
```{r, eval=FALSE}
batchtools::removeRegistry(reg=registry)
```

See `batchtools` documentation for more details on using registries.