DoIntegrate python strategies error

[kingstonaj]

Hi!
I have been trying to implement your package and, while it works beautifully for me with R-based integrations, I am unable to implement any python Integrations successfully. They produce the following error: (Produced using the code and data from your 'Introduction' vignette

Warning: `bbknnIntegration`: failed, because “MultisessionFuture (<none>) failed to receive message results from cluster RichSOCKnode #1 (PID 46032 on localhost ‘localhost’). The reason reported was ‘error reading from connection’. 
Post-mortem diagnostic: No process exists with this PID, i.e. the localhost worker is no longer alive. Detected a non-exportable reference (‘externalptr’) in one of the globals (‘method_expr’ of class ‘call’) used in the future expression. 
The total size of the 5 globals exported is 214.77 MiB. The three largest globals are ‘method_expr’ (214.77 MiB of class ‘call’), ‘caller_env’ (629 bytes of class ‘function’) and ‘eval_tidy’ (623 bytes of class ‘function’)”
 -> Skipping this integration 

This error is produced regardless of the Python integration strategy used, and occurs when future is set to use sequential, multicore or multisession processing. It also appears to occur regardless of the size of the test dataset, or if I run any of the python-based algorithms through IntegrateLayers individually . Setting use.future to FALSE causes R to encounter a fatal error.
I was wondering if you might be able to suggest any fixes.
Thanks!
Alex

> sessionInfo()
R version 4.4.0 (2024-04-24)
Platform: aarch64-apple-darwin20
Running under: macOS Sonoma 14.5

Matrix products: default
BLAS:   /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib 
LAPACK: /Library/Frameworks/R.framework/Versions/4.4-arm64/Resources/lib/libRlapack.dylib;  LAPACK version 3.12.0

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

time zone: Europe/London
tzcode source: internal

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] future_1.34.0            pbmcsca.SeuratData_3.0.0 pbmcref.SeuratData_1.0.0 SeuratData_0.2.2.9001    SeuratIntegrate_0.4.0    Seurat_5.2.1             SeuratObject_5.0.2       sp_2.2-0                

loaded via a namespace (and not attached):
  [1] RcppAnnoy_0.0.22            splines_4.4.0               later_1.4.1                 batchelor_1.22.0            tibble_3.2.1                polyclip_1.10-7             XML_3.99-0.18              
  [8] fastDummies_1.7.5           lifecycle_1.0.4             edgeR_4.4.2                 globals_0.16.3              lattice_0.22-6              MASS_7.3-65                 backports_1.5.0            
 [15] magrittr_2.0.3              limma_3.62.2                plotly_4.10.4               remotes_2.5.0               httpuv_1.6.15               glmGamPoi_1.18.0            sctransform_0.4.1          
 [22] spam_2.11-1                 spatstat.sparse_3.1-0       reticulate_1.42.0           cowplot_1.1.3               pbapply_1.7-2               DBI_1.2.3                   RColorBrewer_1.1-3         
 [29] ResidualMatrix_1.16.0       pkgload_1.4.0               abind_1.4-8                 zlibbioc_1.52.0             Rtsne_0.17                  GenomicRanges_1.58.0        purrr_1.0.4                
 [36] BiocGenerics_0.52.0         rappdirs_0.3.3              sva_3.54.0                  GenomeInfoDbData_1.2.13     IRanges_2.40.1              S4Vectors_0.44.0            ggrepel_0.9.6              
 [43] irlba_2.3.5.1               listenv_0.9.1               spatstat.utils_3.1-3        genefilter_1.88.0           goftest_1.2-3               RSpectra_0.16-2             spatstat.random_3.3-3      
 [50] annotate_1.84.0             fitdistrplus_1.2-2          parallelly_1.43.0           DelayedMatrixStats_1.28.1   codetools_0.2-20            DelayedArray_0.32.0         scuttle_1.16.0             
 [57] tidyselect_1.2.1            UCSC.utils_1.2.0            farver_2.1.2                ScaledMatrix_1.14.0         matrixStats_1.5.0           stats4_4.4.0                spatstat.explore_3.4-2     
 [64] jsonlite_2.0.0              BiocNeighbors_2.0.1         progressr_0.15.1            ggridges_0.5.6              survival_3.8-3              tools_4.4.0                 ica_1.0-3                  
 [71] Rcpp_1.0.14                 glue_1.8.0                  gridExtra_2.3               SparseArray_1.6.2           mgcv_1.9-1                  MatrixGenerics_1.18.1       GenomeInfoDb_1.42.3        
 [78] dplyr_1.1.4                 withr_3.0.2                 fastmap_1.2.0               digest_0.6.37               rsvd_1.0.5                  R6_2.6.1                    mime_0.13                  
 [85] colorspace_2.1-1            scattermore_1.2             tensor_1.5                  spatstat.data_3.1-6         RSQLite_2.3.9               RhpcBLASctl_0.23-42         tidyr_1.3.1                
 [92] generics_0.1.3              data.table_1.17.0           httr_1.4.7                  htmlwidgets_1.6.4           S4Arrays_1.6.0              uwot_0.2.3                  pkgconfig_2.0.3            
 [99] gtable_0.3.6                blob_1.2.4                  lmtest_0.9-40               SingleCellExperiment_1.28.1 XVector_0.46.0              htmltools_0.5.8.1           dotCall64_1.2              
[106] scales_1.3.0                Biobase_2.66.0              png_0.1-8                   harmony_1.2.3               spatstat.univar_3.1-2       rstudioapi_0.17.1           reshape2_1.4.4             
[113] curl_6.2.2                  nlme_3.1-167                zoo_1.8-13                  cachem_1.1.0                stringr_1.5.1               KernSmooth_2.23-26          parallel_4.4.0             
[120] miniUI_0.1.1.1              AnnotationDbi_1.68.0        pillar_1.10.1               grid_4.4.0                  vctrs_0.6.5                 RANN_2.6.2                  promises_1.3.2             
[127] BiocSingular_1.22.0         beachmat_2.22.0             xtable_1.8-4                cluster_2.1.8.1             locfit_1.5-9.12             cli_3.6.4                   compiler_4.4.0             
[134] rlang_1.1.5                 crayon_1.5.3                future.apply_1.11.3         labeling_0.4.3              plyr_1.8.9                  forcats_1.0.0               stringi_1.8.7              
[141] viridisLite_0.4.2           deldir_2.0-4                BiocParallel_1.40.1         Biostrings_2.74.1           munsell_0.5.1               lazyeval_0.2.2              spatstat.geom_3.3-6        
[148] Matrix_1.7-3                RcppHNSW_0.6.0              patchwork_1.3.0             sparseMatrixStats_1.18.0    bit64_4.6.0-1               ggplot2_3.5.1               statmod_1.5.0              
[155] KEGGREST_1.46.0             shiny_1.10.0                SummarizedExperiment_1.36.0 ROCR_1.0-11                 igraph_2.1.4                broom_1.0.8                 memoise_2.0.1              
[162] bit_4.6.0

[fspecque]

Hi!

Thanks for your interest in the package and for reporting. I'm unable to reproduce this issue and I've never experienced it either, so I might be of little help here.

After some search, my guess is that it's probably an error with R/reticulate and python/conda (but I cannot be sure).
Let's make some tests (here with the conda environment for BBKNN). If you didn't use the default name for the environment, replace "SeuratIntegrate_bbknn" by the name you chose in the lines below.

• first check if the error happens outside of RStudio (i.e. from a terminal).
• if it does, check that reticulate can initialize the conda environment in an R session:

reticulate::py_config()  # load default python just to check if it works

# Close R and in a new session:
reticulate::use_condaenv("SeuratIntegrate_bbknn", required = TRUE)
reticulate::py_config()  # any error here?

np <- reticulate::import("numpy", convert=FALSE)         # any error here?
anndata <- reticulate::import("anndata", convert=FALSE)  # any error here?
bbknn <- reticulate::import("bbknn", convert=FALSE)      # any error here?
sc <- reticulate::import("scanpy", convert=FALSE)        # any error here?
scipy <-  reticulate::import('scipy', convert = FALSE)   # any error here?

• try to install numpy (and/or any of the faulty import()) with pip after conda activate SeuratIntegrate_bbknn and try the integration again in a fresh R session.

Note that you can create a backup of a conda environment before making any change to it if you want to.

Let me know if any of these suggestions works, and if not, tell me where it fails and I'll try to come up with some more.
Florian

[kingstonaj]

Hi Florian,

Thanks so much for the response! Unfortunately nothing much seems to be amiss with the Reticulate/Conda environments that I can spot.

To address your suggestions:

• The error is replicated when R is run in terminal, with a slightly different error message than previous (shown with bbkn for brevity but identical with scanorama and scanvi)

Integration 1 in 2: integrating using 'bbknnIntegration'
0%   10   20   30   40   50   60   70   80   90   100%
[----|----|----|----|----|----|----|----|----|----|
**************************************************|

 *** caught segfault ***
address 0x540, cause 'invalid permissions'
Warning: `bbknnIntegration`: failed, because "Failed to retrieve the result of MulticoreFuture (<none>) from the forked worker (on localhost; PID 74927).
Post-mortem diagnostic: No process exists with this PID, i.e. the forked localhost worker is no longer alive. The total size of the 5 globals exported is 213.51 MiB. 
The three largest globals are 'method_expr' (213.51 MiB of class 'call'), 'caller_env' (629 bytes of class 'function') and 'eval_tidy' (623 bytes of class 'function')"
 -> Skipping this integration

• The Conda environment is initialised with no errors using reticulate in an R session (in both terminal and Rstudio) and all the modules seem to import fine. Outputs are as follows:

> reticulate::py_config()
python:         /Users/alex/Library/Caches/org.R-project.R/R/reticulate/uv/cache/archive-v0/7CuHLUQLhjpOEvByKlr3Q/bin/python3
libpython:      /Users/alex/Library/Caches/org.R-project.R/R/reticulate/uv/python/cpython-3.11.11-macos-aarch64-none/lib/libpython3.11.dylib
pythonhome:     /Users/alex/Library/Caches/org.R-project.R/R/reticulate/uv/cache/archive-v0/7CuHLUQLhjpOEvByKlr3Q:/Users/alex/Library/Caches/org.R-project.R/R/reticulate/uv/cache/archive-v0/7CuHLUQLhjpOEvByKlr3Q
virtualenv:     /Users/alex/Library/Caches/org.R-project.R/R/reticulate/uv/cache/archive-v0/7CuHLUQLhjpOEvByKlr3Q/bin/activate_this.py
version:        3.11.11 (main, Mar 17 2025, 21:33:08) [Clang 20.1.0 ]
numpy:          /Users/alex/Library/Caches/org.R-project.R/R/reticulate/uv/cache/archive-v0/7CuHLUQLhjpOEvByKlr3Q/lib/python3.11/site-packages/numpy
numpy_version:  2.2.4

NOTE: Python version was forced by py_require()

> reticulate::use_condaenv("SeuratIntegrate_bbknn", required = TRUE)
> reticulate::py_config()  
python:         /opt/anaconda3/envs/SeuratIntegrate_bbknn/bin/python
libpython:      /opt/anaconda3/envs/SeuratIntegrate_bbknn/lib/libpython3.13.dylib
pythonhome:     /opt/anaconda3/envs/SeuratIntegrate_bbknn:/opt/anaconda3/envs/SeuratIntegrate_bbknn
version:        3.13.2 | packaged by conda-forge | (main, Feb 17 2025, 14:02:48) [Clang 18.1.8 ]
numpy:          /opt/anaconda3/envs/SeuratIntegrate_bbknn/lib/python3.13/site-packages/numpy
numpy_version:  2.1.3

NOTE: Python version was forced by use_python() function

I also tried the classic uninstall/reinstall of the packages and Conda environments through terminal, but the result was unchanged. Let me know if this sheds any light on the situation, v happy to provide more info if needed!

Thanks,
Alex

[fspecque]

Hi!

Thank you for the detailed feedback!

Given that you get a segfault and that none of the conda environments works, I'm starting to wonder whether this is an error with conda binaries... (see here and there for instance).

I also tried the classic uninstall/reinstall of the packages and Conda environments through terminal

Did you try installing (at least numpy) with pip install rather than conda install inside "SeuratIntegrate_bbknn"? That was suggested in this issue

Regarding the putative conda binaries incompatibility, here are a few suggestions:

• it might be helpful to try something else with the same conda environment (let's stick to "SeuratIntegrate_bbknn"). For example, you could try to RunUMAP() with umap-learn package after you've generated a graph with FindNeighbours() to see if at least that works. Lunch it from a terminal and don't use future preferably, it provides more troubleshooting information.

reticulate::use_condaenv("SeuratIntegrate_bbknn", required = TRUE)

obj <- RunUMAP(obj, graph = "RNA_snn",   # or "SCT_snn" if normalised with SCTransform
               umap.method = "umap-learn", n.epoch = 400L)

• if my intuition is correct, a possible fix would be to create another conda environment with R and necessary packages, install SeuratIntegrate in the environment (using R, for now SeuratIntegrate is not installable with conda), and lunch the integration with R installed in the environment. Not ideal, but it might work according to an aforementioned issue

• in the meantime (or alternatively), you can open an issue from the reticulate repository, they surely are more used to this kind of segmentation fault.

Hope that helps! Please let me know how it goes, I'd be happy to hear what outcomes you get and assist further if I can.
Florian