get the peaks that shared in multiple samples

bwa aln or bwa mem for short reads (36bp)

My ChIP-seq data are 36bp single end reads. I usually use bowtie1 for mapping ChIP-seq reads, but bowtie1 does not handle indels. Since I want to call mutations on the ChIP-seq reads, I have to use another aligner BWA, the most popular mapper written by Heng Li.

The github page says if reads < 70bp, bwa aln should be used. Otherwise, bwa mem should be used.
bwa mem is a more recent algorithm (should be better?).

I searched on biostar, and found When and why is bwa aln better then bwa mem?

I did a simulation test using Teaser using default setting for each aligner.

The results are shown below:

The mapping rate:

Memory usage:

Run time:

Indeed, BWA aln is a little better than BWA mem for short reads.

For a real data set, the samtools flagstat results are shown below:

bwa aln:
282967631 + 0 in total (QC-passed reads + QC-failed reads)
0 + 0 secondary
0 + 0 supplementary
18963259 + 0 duplicates
240660130 + 0 mapped (85.05% : N/A)
0 + 0 paired in sequencing
0 + 0 read1
0 + 0 read2
0 + 0 properly paired (N/A : N/A)
0 + 0 with itself and mate mapped
0 + 0 singletons (N/A : N/A)
0 + 0 with mate mapped to a different chr
0 + 0 with mate mapped to a different chr (mapQ>=5)

bwa mem:
282967631 + 0 in total (QC-passed reads + QC-failed reads)
0 + 0 secondary
0 + 0 supplementary
18332921 + 0 duplicates
236558306 + 0 mapped (83.60% : N/A)
0 + 0 paired in sequencing
0 + 0 read1
0 + 0 read2
0 + 0 properly paired (N/A : N/A)
0 + 0 with itself and mate mapped
0 + 0 singletons (N/A : N/A)
0 + 0 with mate mapped to a different chr
0 + 0 with mate mapped to a different chr (mapQ>=5)

Indeed, bwa aln has a moderate higher mapping rate and a shorter run time for short 36bp reads.

How to deal with paired samples for snakemake

I am currently experimenting snakemake for pipelines. We will have paired samples when processing data. e.g. ChIP-seq (IP vs Input), DNA-seq (Normal vs Tumor). How to specify them in snakemake?
The trick is to use a python dictionary to pair samples and a function to define Input files.

I will show an toy example with ChIP-seq processing.

My first ever minimal working ChIP-seq pipeline using snakemake

Why using snakemake

Snakemake is a python3 based pipeline building tool (a python variant of GNU make) specialized for bioinformatics. I put my notes managing different versions of python here. You can write any python codes inside the Snakefile. Using snakemake is to simplify the tedious pre-processing work for large genomic data sets and for the sake of reproducibility. There are many other tools you can find here for this purpose.

Key features of snakemake

• Snakemake automatically creates missing directories.
• wildcards and Input function

To access wildcards in a shell command: {wildcards.sample}

{wildcards} is greedy (.+): {sample}.fastq could be matching sampleA.fastq if there is no sub-folder anymore, but evenwhateverfolder/sampleA.fastq can be matched as well.

One needs to think snakemake in a bottom-up way: snakemake will first look for the output files, and substitue the {wildcards} with the file names, and look for which rule can be used to creat the output, and then look for input files that are defined by the {wildcards}.

Read the following

flexible bioinformatics pipelines with snakemake
Build bioinformatics pipelines with Snakemake
snakemake ChIP-seq pipeline example
submit all the jobs immediately
snakemake-parallel-bwa
RNA-seq snakemake example
functions as inputs and derived parameters
snakemake FAQ
snakemake tutorial from the developer

examples

https://github.com/slowkow/snakefiles/blob/master/bsub.py
https://github.com/broadinstitute/viral-ngs/tree/master/pipes

A working snakemake pipeline for ChIP-seq

The folder structure is like this:

├── README.md
├── Snakemake
├── config.yaml
└── rawfastqs
    ├── sampleA
    │   ├── sampleA_L001.fastq.gz
    │   ├── sampleA_L002.fastq.gz
    │   └── sampleA_L003.fastq.gz
    ├── sampleB
    │   ├── sampleB_L001.fastq.gz
    │   ├── sampleB_L002.fastq.gz
    │   └── sampleB_L003.fastq.gz
    ├── sampleG1
    │   ├── sampleG1_L001.fastq.gz
    │   ├── sampleG1_L002.fastq.gz
    │   └── sampleG1_L003.fastq.gz
    └── sampleG2
        ├── sampleG2_L001.fastq.gz
        ├── sampleG2_L002.fastq.gz
        └── sampleG2_L003.fastq.gz

There is a folder named rawfastqs containing all the raw fastqs. each sample subfolder contains multiple fastq files from different lanes.

In this example, I have two control (Input) samples and two corresponding case(IP) samples.

CONTROLS = ["sampleG1","sampleG2"]
CASES = ["sampleA", "sampleB"]

putting them in a list inside the Snakefile. If there are many more samples, need to generate it with pythonprogrammatically.

## dry run
snakemake -np

## work flow diagram
snakemake --forceall --dag | dot -Tpng | display

To Do:

• Make the pipeline more flexiable. e.g. specify the folder name containing raw fastqs, now it is hard coded.
• write a wrapper script for submitting jobs in moab. Figuring out dependencies and --immediate-submit

Downsampling bam files to a certain number of reads

When compare different ChIP-seq data sets or analyze a set of ChIP-seq data sets together (e.g. ChromHMM analysis), it is desirable to subsample the deeply sequenced ones to a certain number of reads (say 15million or 30 million).

In paper Integrative analysis of 111 reference human epigenomes:

To avoid artificial differences in signal strength due to differences in sequencing depth, all consolidated histone mark data sets (except the additional histone marks the seven deeply profiled epigenomes, Fig. 2j) were uniformly subsampled to a maximum depth of 30 million reads (the median read depth over all consolidated samples). For the seven deeply profiled reference epigenomes (Fig. 2j), histone mark data sets were subsampled to a maximum of 45 million reads (median depth). The consolidated DNase-seq data sets were subsampled to a maximum depth of 50 million reads (median depth). These uniformly subsampled data sets were then used for all further processing steps (peak calling, signal coverage tracks, chromatin states).

After reading several posts here and here. It seems samtools and sambamba are the tools to use, but they both output a proportion number of reads.

time samtools view -s 3.6 -b my.bam -o subsample.bam
real    6m9.141s
user    5m59.842s
sys 0m8.912s

time sambamba view -f bam -t 10 --subsampling-seed=3 -s 0.6 my.bam -o subsample.bam
real    1m34.937s
user    11m55.222s
sys 0m29.872s

-s 3.6 set seed of 3 and 60% of the reads by samtools. Using multiple cpu with sambamba is much faster and an index file is generated on the fly.

If one wants to get say 15 million reads, one needs to do samtools flag stat or samtools idxstats to get the total number of reads, and then calculate the proportion by: 15 million/total = proportion.

samtools idxstats is much faster when the bam is sorted and indexed:

Retrieve and print stats in the index file. The output is TAB delimited with each line consisting of reference sequence name, sequence length, # mapped reads and # unmapped reads.

Total number of reads: samtools idxstats example.bam | cut -f3 | awk 'BEGIN {total=0} {total += $1} END {print total}'

Finally, feed the proportion to -s flag. One might want to remove the unmapped the reads and the duplicated reads in the bam file before downsampling. One might also need to sort the subsampled bam file again and index it.

library size normalization for ChIP-seq

I have discussed how to use DESeq2 to do differential binding for ChIP-seq at here.

I am experimenting DiffBind to do the same thing, which internally uses EdgR, DESeq and DESeq2. The author Rory Stark is very responsive on the bioconductor support site and has answered several of my questions.

Today, I am going to keep a note here for normalizing the ChIP-seq data. If one compares ChIP-seq versus RNA-seq data, they are in the end all counts data. For RNA-seq, we usually get a read count table for the counts in the exons (union of them is for a gene); for ChIP-seq, we get a read count table for counts within the peaks. The peaks have to be identified by other tools such as MACS first. The counts data follow a (negative) binomial distribution. That's why tools such as DESeq2, which was developed for RNAseq is used for ChIP-seq.

After we get a count table, it comes to the normalization problem. If you are interested, read this paper Beyond library size: a field guide to NGS normalization. In the DiffBind package, the counts table is obtained by a function ?dba.count.

There are several ways to specify how the counts are normalized for the binding affinity matrix:

score   
which score to use in the binding affinity matrix. Note that all raw read counts are maintained for use by dba.analyze, regardless of how this is set. One of:
DBA_SCORE_READS raw read count for interval using only reads from ChIP
DBA_SCORE_READS_FOLD    raw read count for interval from ChIP divided by read count for interval from control
DBA_SCORE_READS_MINUS   raw read count for interval from ChIP minus read count for interval from control
DBA_SCORE_RPKM  RPKM for interval using only reads from ChIP
DBA_SCORE_RPKM_FOLD RPKM for interval from ChIP divided by RPKM for interval from control
DBA_SCORE_TMM_READS_FULL    TMM normalized (using edgeR), using ChIP read counts and Full Library size
DBA_SCORE_TMM_READS_EFFECTIVE   TMM normalized (using edgeR), using ChIP read counts and Effective Library size
DBA_SCORE_TMM_MINUS_FULL    TMM normalized (using edgeR), using ChIP read counts minus Control read counts and Full Library size
DBA_SCORE_TMM_MINUS_EFFECTIVE   TMM normalized (using edgeR), using ChIP read counts minus Control read counts and Effective Library size
DBA_SCORE_TMM_READS_FULL_CPM    same as DBA_SCORE_TMM_READS_FULL, but reporrted in counts-per-million.
DBA_SCORE_TMM_READS_EFFECTIVE_CPM   same as DBA_SCORE_TMM_READS_EFFECTIVE, but reporrted in counts-per-million.
DBA_SCORE_TMM_MINUS_FULL_CPM    same as DBA_SCORE_TMM_MINUS_FULL, but reporrted in counts-per-million.
DBA_SCORE_TMM_MINUS_EFFECTIVE_CPM   Tsame as DBA_SCORE_TMM_MINUS_EFFECTIVE, but reporrted in counts-per-million.

DBA_SCORE_TMM_READS_FULL vs DBA_SCORE_TMM_READS_EFFECTIVE:

Diffbind let's you to choose use full library size or effective library size for trimmed mean of M values(TMM) normalization which was proposed by Mark D Robinson for RNAseq.

Full library size is the number of reads in the bam files.
Effective library size is the number of reads mapped in the exons or within the peaks. It is the column sums for the matrix.

Note that effective library size (bFullLibrarySize =FALSE) may be more appropriate for situations when the overall signal (binding rate) is expected to be directly comparable between the samples.

If one wants to subtract the input reads, one can use DBA_SCORE_TMM_MINUS_FULL and DBA_SCORE_TMM_MINUS_EFFECTIVE

No matter what score you choose, for differential binding analysis in Diffbind, it is always the raw counts is used for the binding matrix. Diffbind (by default) subtract the input raw reads for subsequent analysis. Whether or not this is good was discussed here.

For example, if one uses DESeq2, the details are as follows:

For each contrast, a separate analysis is performed. First, a matrix of counts is constructed for the contrast, with columns for all the samples in the first group, followed by columns for all the samples in the second group. The raw read count is used for this matrix; if the bSubControl parameter is set to TRUE (as it is by default), the raw number of reads in the control sample (if available) will be subtracted. Next the library size is computed for each sample for use in subsequent normalization. By default, this is the total number of reads in peaks (the sum of each column). Alternatively, if the bFullLibrarySize parameter is set to TRUE, the total number of reads in the library (calculated from the source BAM/BED file) is used. The first step concludes with a call to DESeq2’s DESeqDataSetFromMatrix function, which returns a DESeqDataSet object. If bFullLibrarySize is set to TRUE, then sizeFactors is called with the number of reads in the BAM/BED files for each ChIP sample, divided by the minimum of these; otherwise, estimateSizeFactors is invoked. estimateDispersions is then called with the DESeqDataSet object and fitType set to local. Next the model is fitted and tested using nbinomWaldTest

estimateSizeFactors in DESeq2:

Given a matrix or data frame of count data, this function estimates the size factors as follows: Each column is divided by the geometric means of the rows. The median (or, ir requested, another location estimator) of these ratios (skipping the genes with a geometric mean of zero) is used as the size factor for this column.

Upset plot for overlapping ChIP-seq peaks: an alternative to Venn diagram

ChIP-seq-upset

Ming Tang

January 25, 2016

I am going to demonstrate how to make Upset plot using UpSetR package for ChIP-seq peaks. After running a peak-calling method for 3 samples, you want to compare how many peaks are overlapped in two samples and in three samples. People may draw venn diagram for this kind of problem, but when you have more than 3 samples, the venn diagram is hard to read.

# install it if you have not 
# install.packages("UpSetR")
library(UpSetR)
dummy<- data.frame(peak=c("peak_1", "peak_1", "peak_1","peak_2","peak_2","peak_3","peak_3","peak_4"),
                   sample=c("sample_1","sample_2","sample_3","sample_2","sample_3","sample_2","sample_3","sample_1"))

dummy

##     peak   sample
## 1 peak_1 sample_1
## 2 peak_1 sample_2
## 3 peak_1 sample_3
## 4 peak_2 sample_2
## 5 peak_2 sample_3
## 6 peak_3 sample_2
## 7 peak_3 sample_3
## 8 peak_4 sample_1

To get the format that UpsetR accepts, we have to “spread” the sample column to multiple columns. It is like to change a long format data to wide format. See here as well.

library(dplyr)

## 
## Attaching package: 'dplyr'

## The following objects are masked from 'package:stats':
## 
##     filter, lag

## The following objects are masked from 'package:base':
## 
##     intersect, setdiff, setequal, union

library(tidyr)

peaks<- dummy %>% mutate(value =1) %>% spread(sample, value, fill =0 )
peaks

##     peak sample_1 sample_2 sample_3
## 1 peak_1        1        1        1
## 2 peak_2        0        1        1
## 3 peak_3        0        1        1
## 4 peak_4        1        0        0

I noticed that one has to convert peaks(local data frame) back to data.frame for UpsetR to work. In this example, I did not convert the peaks to local data frame by tbl_df(peaks).

upset(peaks, order.by = "freq")