# Error handling

`dreamlet()` evaluates precision-weighted linear (mixed) models on each gene that passes standard filters. The linear mixed model used by `dream()` can be a little fragile for small sample sizes and correlated covariates. `dreamlet()` runs `variancePartition::dream()` in the backend for each cell cluster. `dream()` reports model failures for each cell cluster and `dreamlet()` reports these failures to the user. `dreamlet()` returns all successful model fits to be used for downstream analysis.

See details from `variancePartition` error page.

# Errors with random effects

Due to a recent bug in the dependency `Matrix` package, all random effects models may fail for technical reasons. If your random effects analysis is failing for all genes in cases with no good explanation, this bug may be responsible. This case can be detected and resolved as follows:

    library(lme4)

    # Fit simple mixed model
    lmer(Reaction ~ (1 | Subject), sleepstudy)
    # Error in initializePtr() :
    #  function 'chm_factor_ldetL2' not provided by package 'Matrix'

This error indicates incompatible installs of `Matrix` and `lme4`. This can be solved with

    install.packages("lme4", type = "source")

followed by restarting R.

# Errors at the assay- and gene-level

The most common issue is when `dreamlet()` analysis succeeds for most genes, but a handful of genes fail in each cell cluster. These genes can fail if the iterative process of fitting the linear mixed model does not converge, or if the estimated covariance matrix that is supposed be positive definite has an eigen-value that is negative or too close to zero due to rounding errors in floating point arithmetic.

In these cases, `dreamlet()` stores a summary of these failures for all cell clusters that is accessible with `details()`. Specific failure messages for each cell cluster and gene can be extracted using `seeErrors()`

Here we demonstrate how `dreamlet()` handles model failures:

    library(dreamlet)
    library(muscat)
    library(SingleCellExperiment)

    data(example_sce)

    # create pseudobulk for each sample and cell cluster
    pb <- aggregateToPseudoBulk(example_sce,
      assay = "counts",
      cluster_id = "cluster_id",
      sample_id = "sample_id",
      verbose = FALSE
    )

    # voom-style normalization for each cell cluster
    res.proc <- processAssays(
      pb[1:300, ],
      ~group_id
    )

    # Redundant formula
    # This example is an extreme example of redundancy
    # but more subtle cases often show up in real data
    form <- ~ group_id + (1 | group_id)

    # fit dreamlet model
    res.dl <- dreamlet(res.proc, form)
    ##  B cells...7.9 secs
    ##  CD14+ Monocytes...10 secs
    ##  CD4 T cells...9 secs
    ##  CD8 T cells...4.4 secs
    ##  FCGR3A+ Monocytes...11 secs
    ##
    ## Of 1,062 models fit across all assays, 96.2% failed

    # summary of models
    res.dl
    ## class: dreamletResult
    ## assays(5): B cells CD14+ Monocytes CD4 T cells CD8 T cells FCGR3A+ Monocytes
    ## Genes:
    ##  min: 3
    ##  max: 11
    ## details(7): assay n_retain ... n_errors error_initial
    ## coefNames(2): (Intercept) group_idstim
    ##
    ## Of 1,062 models fit across all assays, 96.2% failed

    # summary of models for each cell cluster
    details(res.dl)
    ##               assay n_retain                    formula formDropsTerms n_genes n_errors error_initial
    ## 1           B cells        4 ~group_id + (1 | group_id)          FALSE     201      190         FALSE
    ## 2   CD14+ Monocytes        4 ~group_id + (1 | group_id)          FALSE     269      263         FALSE
    ## 3       CD4 T cells        4 ~group_id + (1 | group_id)          FALSE     216      207         FALSE
    ## 4       CD8 T cells        4 ~group_id + (1 | group_id)          FALSE     118      115         FALSE
    ## 5 FCGR3A+ Monocytes        4 ~group_id + (1 | group_id)          FALSE     258      247         FALSE

- `assay`: cell type
- `n_retain`: number of samples retained
- `formula`: regression formula used after variable filtering
- `formDropsTerms`: whether a variable was dropped from the formula for having zero variance following filtering
- `n_genes`: number of genes analyzed
- `n_errors`: number of genes with errors
- `error_initial`: indicator for assay-level error

## Assay-level errors

Before the full dataset is analyzed, `dreamlet()` runs a test for each assay to see if the model succeeds. If the model fails, its does not continue analysis for that assay. These assay-level errors are reported above in the `error_initial` column, and details are returned here.

    # Extract errors as a tibble
    res.err = seeErrors(res.dl)
    ##   Assay-level errors: 0
    ##   Gene-level errors: 1038

    # No errors at the assay level
    res.err$assayLevel

    # the most common error is:
    "Some predictor variables are on very different scales: consider rescaling"

This indicates that the scale of the predictor variables are very different and can affect the numerical stability of the iterative algorithm. This can be solved by running `scale()` on each variable in the formula:

    form = ~ scale(x) + scale(y) + ...

## Gene-level errors

A model can fail for a single gene if covariates are too correlated, or for other numerical issues. Failed models are reported here and are not included in downstream analysis.

    # See gene-level errors for each assay
    res.err$geneLevel[1:2,]
    ## # A tibble: 2 × 3
    ##   assay   feature  errorText
    ##   <chr>   <chr>    <chr>
    ## B cells ISG15    "Error in lmerTest:::as_lmerModLT(model, devfun, tol = tol):…
    ## B cells AURKAIP1 "Error in lmerTest:::as_lmerModLT(model, devfun, tol = tol):…

    # See full error message text
    res.err$geneLevel$errorText[1]
    "Error in lmerTest:::as_lmerModLT(model, devfun, tol = tol): (converted from warning)
    Model may not have converged with 1 eigenvalue close to zero: 1.4e-09\n"

This message indicates that the model was numerically unstable likely because the variables are closely correlated.

# Session Info

    ## R version 4.5.2 (2025-10-31)
    ## Platform: x86_64-pc-linux-gnu
    ## Running under: Ubuntu 24.04.4 LTS
    ##
    ## Matrix products: default
    ## BLAS:   /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3
    ## LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.26.so;  LAPACK version 3.12.0
    ##
    ## locale:
    ##  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C
    ##  [3] LC_TIME=en_US.UTF-8        LC_COLLATE=C
    ##  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8
    ##  [7] LC_PAPER=en_US.UTF-8       LC_NAME=C
    ##  [9] LC_ADDRESS=C               LC_TELEPHONE=C
    ## [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
    ##
    ## time zone: Etc/UTC
    ## tzcode source: system (glibc)
    ##
    ## attached base packages:
    ## [1] stats4    stats     graphics  grDevices utils     datasets  methods
    ## [8] base
    ##
    ## other attached packages:
    ##  [1] GO.db_3.22.0                org.Hs.eg.db_3.22.0
    ##  [3] AnnotationDbi_1.73.0        zenith_1.13.0
    ##  [5] muscData_1.25.0             scater_1.39.2
    ##  [7] scuttle_1.21.0              ExperimentHub_3.1.0
    ##  [9] AnnotationHub_4.1.0         BiocFileCache_3.1.0
    ## [11] dbplyr_2.5.2                muscat_1.25.2
    ## [13] dreamlet_1.9.1              SingleCellExperiment_1.33.0
    ## [15] SummarizedExperiment_1.41.1 Biobase_2.71.0
    ## [17] GenomicRanges_1.63.1        Seqinfo_1.1.0
    ## [19] IRanges_2.45.0              S4Vectors_0.49.0
    ## [21] BiocGenerics_0.57.0         generics_0.1.4
    ## [23] MatrixGenerics_1.23.0       matrixStats_1.5.0
    ## [25] variancePartition_1.41.3    BiocParallel_1.45.0
    ## [27] limma_3.67.0                ggplot2_4.0.2
    ## [29] BiocStyle_2.39.0
    ##
    ## loaded via a namespace (and not attached):
    ##   [1] bitops_1.0-9              httr_1.4.8
    ##   [3] RColorBrewer_1.1-3        doParallel_1.0.17
    ##   [5] Rgraphviz_2.55.0          numDeriv_2016.8-1.1
    ##   [7] tools_4.5.2               sctransform_0.4.3
    ##   [9] backports_1.5.0           utf8_1.2.6
    ##  [11] R6_2.6.1                  metafor_4.8-0
    ##  [13] mgcv_1.9-4                GetoptLong_1.1.0
    ##  [15] withr_3.0.2               prettyunits_1.2.0
    ##  [17] gridExtra_2.3             fdrtool_1.2.18
    ##  [19] cli_3.6.5                 sandwich_3.1-1
    ##  [21] labeling_0.4.3            slam_0.1-55
    ##  [23] sass_0.4.10               KEGGgraph_1.71.0
    ##  [25] SQUAREM_2021.1            mvtnorm_1.3-3
    ##  [27] S7_0.2.1                  blme_1.0-7
    ##  [29] mixsqp_0.3-54             parallelly_1.46.1
    ##  [31] invgamma_1.2              RSQLite_2.4.6
    ##  [33] shape_1.4.6.1             gtools_3.9.5
    ##  [35] dplyr_1.2.0               Matrix_1.7-4
    ##  [37] metadat_1.4-0             ggbeeswarm_0.7.3
    ##  [39] abind_1.4-8               lifecycle_1.0.5
    ##  [41] yaml_2.3.12               edgeR_4.9.2
    ##  [43] mathjaxr_2.0-0            gplots_3.3.0
    ##  [45] SparseArray_1.11.10       grid_4.5.2
    ##  [47] blob_1.3.0                crayon_1.5.3
    ##  [49] lattice_0.22-9            beachmat_2.27.2
    ##  [51] msigdbr_25.1.1            annotate_1.89.0
    ##  [53] KEGGREST_1.51.1           sys_3.4.3
    ##  [55] maketools_1.3.2           pillar_1.11.1
    ##  [57] knitr_1.51                ComplexHeatmap_2.27.1
    ##  [59] rjson_0.2.23              boot_1.3-32
    ##  [61] corpcor_1.6.10            future.apply_1.20.2
    ##  [63] codetools_0.2-20          glue_1.8.0
    ##  [65] data.table_1.18.2.1       vctrs_0.7.1
    ##  [67] png_0.1-8                 Rdpack_2.6.6
    ##  [69] gtable_0.3.6              assertthat_0.2.1
    ##  [71] cachem_1.1.0              zigg_0.0.2
    ##  [73] xfun_0.56                 rbibutils_2.4.1
    ##  [75] S4Arrays_1.11.1           Rfast_2.1.5.2
    ##  [77] reformulas_0.4.4          iterators_1.0.14
    ##  [79] statmod_1.5.1             nlme_3.1-168
    ##  [81] pbkrtest_0.5.5            bit64_4.6.0-1
    ##  [83] filelock_1.0.3            progress_1.2.3
    ##  [85] EnvStats_3.1.0            bslib_0.10.0
    ##  [87] TMB_1.9.19                irlba_2.3.7
    ##  [89] vipor_0.4.7               KernSmooth_2.23-26
    ##  [91] colorspace_2.1-2          rmeta_3.0
    ##  [93] DBI_1.2.3                 DESeq2_1.51.6
    ##  [95] tidyselect_1.2.1          bit_4.6.0
    ##  [97] compiler_4.5.2            curl_7.0.0
    ##  [99] httr2_1.2.2               graph_1.89.1
    ## [101] BiocNeighbors_2.5.4       DelayedArray_0.37.0
    ## [103] scales_1.4.0              caTools_1.18.3
    ## [105] remaCor_0.0.20            rappdirs_0.3.4
    ## [107] stringr_1.6.0             digest_0.6.39
    ## [109] minqa_1.2.8               rmarkdown_2.30
    ## [111] aod_1.3.3                 XVector_0.51.0
    ## [113] RhpcBLASctl_0.23-42       htmltools_0.5.9
    ## [115] pkgconfig_2.0.3           lme4_1.1-38
    ## [117] sparseMatrixStats_1.23.0  lpsymphony_1.39.0
    ## [119] mashr_0.2.79              fastmap_1.2.0
    ## [121] rlang_1.1.7               GlobalOptions_0.1.3
    ## [123] DelayedMatrixStats_1.33.0 farver_2.1.2
    ## [125] jquerylib_0.1.4           IHW_1.39.0
    ## [127] zoo_1.8-15                jsonlite_2.0.0
    ## [129] BiocSingular_1.27.1       RCurl_1.98-1.17
    ## [131] magrittr_2.0.4            Rcpp_1.1.1
    ## [133] babelgene_22.9            viridis_0.6.5
    ## [135] EnrichmentBrowser_2.41.0  stringi_1.8.7
    ## [137] MASS_7.3-65               plyr_1.8.9
    ## [139] listenv_0.10.0            parallel_4.5.2
    ## [141] ggrepel_0.9.6             Biostrings_2.79.4
    ## [143] splines_4.5.2             hms_1.1.4
    ## [145] circlize_0.4.17           locfit_1.5-9.12
    ## [147] buildtools_1.0.0          reshape2_1.4.5
    ## [149] ScaledMatrix_1.19.0       BiocVersion_3.23.1
    ## [151] XML_3.99-0.22             evaluate_1.0.5
    ## [153] RcppParallel_5.1.11-1     BiocManager_1.30.27
    ## [155] nloptr_2.2.1              foreach_1.5.2
    ## [157] tidyr_1.3.2               purrr_1.2.1
    ## [159] future_1.69.0             clue_0.3-67
    ## [161] scattermore_1.2           ashr_2.2-63
    ## [163] rsvd_1.0.5                broom_1.0.12
    ## [165] xtable_1.8-8              fANCOVA_0.6-1
    ## [167] viridisLite_0.4.3         truncnorm_1.0-9
    ## [169] tibble_3.3.1              lmerTest_3.2-0
    ## [171] glmmTMB_1.1.14            memoise_2.0.1
    ## [173] beeswarm_0.4.0            cluster_2.1.8.2
    ## [175] globals_0.19.0            GSEABase_1.73.0