Analysis flow¶

Here, we’ll track typical data transformations like subsetting that occur during analysis.

If exploring more generally, read this first: Project flow.

# !pip install 'lamindb[jupyter,bionty]'
!lamin init --storage ./analysis-usecase --schema bionty

import lamindb as ln
import bionty as bt
from lamin_utils import logger

→ connected lamindb: testuser1/analysis-usecase

Register an initial dataset¶

Here we register an initial artifact with a pipeline script register_example_file.py.

!python analysis-flow-scripts/register_example_file.py

Pull the registered dataset, apply a transformation, and register the result¶

Track the current notebook:

ln.context.uid = "eNef4Arw8nNM0000"
ln.context.track()

→ notebook imports: bionty==0.49.0 lamin_utils==0.13.4 lamindb==0.76.2

→ created Transform('eNef4Arw8nNM0000') & created Run('2024-08-30 11:17:39.746265+00:00')

artifact = ln.Artifact.get(description="anndata with obs")
artifact.describe()

Artifact(uid='CffQZcJBHRAo4awe0000', is_latest=True, description='anndata with obs', suffix='.h5ad', type='dataset', size=46992, hash='IJORtcQUSS11QBqD-nTD0A', n_observations=40, _hash_type='md5', _accessor='AnnData', visibility=1, _key_is_virtual=True, updated_at='2024-08-30 11:17:38 UTC')
  Provenance
    .created_by = 'testuser1'
    .storage = '/home/runner/work/lamin-usecases/lamin-usecases/docs/analysis-usecase'
    .transform = 'register_example_file.py'
    .run = '2024-08-30 11:17:29 UTC'
  Labels
    .tissues = 'kidney', 'liver', 'heart', 'brain'
    .cell_types = 'my new cell type', 'T cell', 'hematopoietic stem cell', 'hepatocyte'
    .diseases = 'chronic kidney disease', 'liver lymphoma', 'cardiac ventricle disorder', 'Alzheimer disease'
  Features
    'cell_type' = 'my new cell type'
    'cell_type_id' = 'T cell', 'hematopoietic stem cell', 'hepatocyte'
    'disease' = 'chronic kidney disease', 'liver lymphoma', 'cardiac ventricle disorder', 'Alzheimer disease'
    'tissue' = 'kidney', 'liver', 'heart', 'brain'
  Feature sets
    'var' = 'TSPAN6', 'TNMD', 'DPM1', 'SCYL3', 'FIRRM', 'FGR', 'CFH', 'FUCA2', 'GCLC', 'NFYA', 'STPG1', 'NIPAL3', 'LAS1L', 'ENPP4', 'SEMA3F', 'CFTR', 'ANKIB1', 'CYP51A1', 'KRIT1', 'RAD52'
    'obs' = 'cell_type', 'cell_type_id', 'tissue', 'disease'

Get a backed AnnData object¶

adata = artifact.open()
adata

AnnDataAccessor object with n_obs × n_vars = 40 × 100
  constructed for the AnnData object CffQZcJBHRAo4awe0000.h5ad
    obs: ['_index', 'cell_type', 'cell_type_id', 'disease', 'tissue']
    var: ['_index']

Subset dataset to specific cell types and diseases¶

cell_types = artifact.cell_types.all().lookup(return_field="name")
diseases = artifact.diseases.all().lookup(return_field="name")

Create the subset:

subset_obs = adata.obs.cell_type.isin(
    [cell_types.t_cell, cell_types.hematopoietic_stem_cell]
) & (adata.obs.disease.isin([diseases.liver_lymphoma, diseases.chronic_kidney_disease]))

adata_subset = adata[subset_obs]
adata_subset

AnnDataAccessorSubset object with n_obs × n_vars = 20 × 100
  obs: ['_index', 'cell_type', 'cell_type_id', 'disease', 'tissue']
  var: ['_index']

adata_subset.obs[["cell_type", "disease"]].value_counts()

cell_type                disease               
T cell                   chronic kidney disease    10
hematopoietic stem cell  liver lymphoma            10
Name: count, dtype: int64

Register the subsetted AnnData:

curate = ln.Curate.from_anndata(
    adata_subset.to_memory(), 
    var_index=bt.Gene.ensembl_gene_id, 
    categoricals={
        "cell_type": bt.CellType.name, 
        "disease": bt.Disease.name, 
        "tissue": bt.Tissue.name,
    },
    organism="human"
)

curate.validate()

artifact = curate.save_artifact(description="anndata with obs subset")

artifact.describe()

Artifact(uid='IXUeZJXVaP7lgspR0000', is_latest=True, description='anndata with obs subset', suffix='.h5ad', type='dataset', size=38992, hash='RgGUx7ndRplZZSmalTAWiw', n_observations=20, _hash_type='md5', _accessor='AnnData', visibility=1, _key_is_virtual=True, updated_at='2024-08-30 11:17:40 UTC')
  Provenance
    .created_by = 'testuser1'
    .storage = '/home/runner/work/lamin-usecases/lamin-usecases/docs/analysis-usecase'
    .transform = 'Analysis flow'
    .run = '2024-08-30 11:17:39 UTC'
  Labels
    .tissues = 'kidney', 'liver'
    .cell_types = 'T cell', 'hematopoietic stem cell'
    .diseases = 'chronic kidney disease', 'liver lymphoma'
  Features
    'cell_type' = 'T cell', 'hematopoietic stem cell'
    'disease' = 'chronic kidney disease', 'liver lymphoma'
    'tissue' = 'kidney', 'liver'
  Feature sets
    'var' = 'TSPAN6', 'TNMD', 'DPM1', 'SCYL3', 'FIRRM', 'FGR', 'CFH', 'FUCA2', 'GCLC', 'NFYA', 'STPG1', 'NIPAL3', 'LAS1L', 'ENPP4', 'SEMA3F', 'CFTR', 'ANKIB1', 'CYP51A1', 'KRIT1', 'RAD52'
    'obs' = 'cell_type', 'cell_type_id', 'tissue', 'disease'

Examine data flow¶

Query a subsetted .h5ad artifact containing “hematopoietic stem cell” and “T cell”:

cell_types = bt.CellType.lookup()

my_subset = ln.Artifact.filter(
    suffix=".h5ad",
    description__endswith="subset",
    cell_types__in=[
        cell_types.hematopoietic_stem_cell,
        cell_types.t_cell,
    ],
).first()

my_subset

Artifact(uid='IXUeZJXVaP7lgspR0000', is_latest=True, description='anndata with obs subset', suffix='.h5ad', type='dataset', size=38992, hash='RgGUx7ndRplZZSmalTAWiw', n_observations=20, _hash_type='md5', _accessor='AnnData', visibility=1, _key_is_virtual=True, created_by_id=1, storage_id=1, transform_id=2, run_id=2, updated_at='2024-08-30 11:17:40 UTC')

Common questions that might arise are:

What is the history of this artifact?
Which features and labels are associated with it?
Which notebook analyzed and registered this artifact?
By whom?
And which artifact is its parent?

Let’s answer this using LaminDB:

print("--> What is the history of this artifact?\n")
artifact.view_lineage()

print("\n\n--> Which features and labels are associated with it?\n")
logger.print(artifact.features)
logger.print(artifact.labels)

print("\n\n--> Which notebook analyzed and registered this artifact\n")
logger.print(artifact.transform)

print("\n\n--> By whom\n")
logger.print(artifact.created_by)

print("\n\n--> And which artifact is its parent\n")
display(artifact.run.input_artifacts.df())

--> What is the history of this artifact?

_images/c833f271436584381f7a558eef035e4ad466ef57c3a4d4fd7ed62b596ec5686d.svg

--> Which features and labels are associated with it?

  Features
    'cell_type' = 'T cell', 'hematopoietic stem cell'
    'disease' = 'chronic kidney disease', 'liver lymphoma'
    'tissue' = 'kidney', 'liver'
  Feature sets
    'var' = 'TSPAN6', 'TNMD', 'DPM1', 'SCYL3', 'FIRRM', 'FGR', 'CFH', 'FUCA2', 'GCLC', 'NFYA', 'STPG1', 'NIPAL3', 'LAS1L', 'ENPP4', 'SEMA3F', 'CFTR', 'ANKIB1', 'CYP51A1', 'KRIT1', 'RAD52'
    'obs' = 'cell_type', 'cell_type_id', 'tissue', 'disease'

  Labels
    .tissues = 'kidney', 'liver'
    .cell_types = 'T cell', 'hematopoietic stem cell'
    .diseases = 'chronic kidney disease', 'liver lymphoma'

--> Which notebook analyzed and registered this artifact

Transform(uid='eNef4Arw8nNM0000', is_latest=True, name='Analysis flow', key='analysis-flow.ipynb', type='notebook', created_by_id=1, updated_at='2024-08-30 11:17:39 UTC')

--> By whom

User(uid='DzTjkKse', handle='testuser1', name='Test User1', updated_at='2024-08-30 11:17:25 UTC')

--> And which artifact is its parent

	uid	version	is_latest	description	key	suffix	type	size	hash	n_objects	n_observations	_hash_type	_accessor	visibility	_key_is_virtual	storage_id	transform_id	run_id	created_by_id	updated_at
id
1	CffQZcJBHRAo4awe0000	None	True	anndata with obs	None	.h5ad	dataset	46992	IJORtcQUSS11QBqD-nTD0A	None	40	md5	AnnData	1	True	1	1	1	1	2024-08-30 11:17:38.781215+00:00