order a matrix by a pre-defined group and then by the rowSums in R

I will lay down the R code below

#create a matrix

> mat<- matrix(sample.int(10, size=30, replace=T), ncol=3)
> mat
      [,1] [,2] [,3]
 [1,]    4    7    6
 [2,]    5    6    1
 [3,]    4    3    1
 [4,]    2    2    9
 [5,]    4    5    4
 [6,]    8    9    8
 [7,]    5    9    7
 [8,]    8    6    7
 [9,]    3    8    5
[10,]    4    3    6

# column bind the mat and the sum of the rows together 

> mat1<- cbind(mat, rowSums(mat))
> mat1
      [,1] [,2] [,3] [,4]
 [1,]    4    7    6   17
 [2,]    5    6    1   12
 [3,]    4    3    1    8
 [4,]    2    2    9   13
 [5,]    4    5    4   13
 [6,]    8    9    8   25
 [7,]    5    9    7   21
 [8,]    8    6    7   21
 [9,]    3    8    5   16
[10,]    4    3    6   13
# order the matrix by the rowSums(mat)
> mat[order(rowSums(mat)),]
      [,1] [,2] [,3]
 [1,]    4    3    1
 [2,]    5    6    1
 [3,]    2    2    9
 [4,]    4    5    4
 [5,]    4    3    6
 [6,]    3    8    5
 [7,]    4    7    6
 [8,]    5    9    7
 [9,]    8    6    7
[10,]    8    9    8
# compare with the order by looking at the last column
> mat1[order(rowSums(mat)),]
      [,1] [,2] [,3] [,4]
 [1,]    4    3    1    8
 [2,]    5    6    1   12
 [3,]    2    2    9   13
 [4,]    4    5    4   13
 [5,]    4    3    6   13
 [6,]    3    8    5   16
 [7,]    4    7    6   17
 [8,]    5    9    7   21
 [9,]    8    6    7   21
[10,]    8    9    8   25
# order the mat matrix in a decreasing order by adding a minus sign 
> mat[order(-rowSums(mat)),]
      [,1] [,2] [,3]
 [1,]    8    9    8
 [2,]    5    9    7
 [3,]    8    6    7
 [4,]    4    7    6
 [5,]    3    8    5
 [6,]    2    2    9
 [7,]    4    5    4
 [8,]    4    3    6
 [9,]    5    6    1
[10,]    4    3    1
# label the rows into different groups
> rownames(mat)<- c("f","a","d","c","b","d","e","a","f","f")
> mat
  [,1] [,2] [,3]
f    4    7    6
a    5    6    1
d    4    3    1
c    2    2    9
b    4    5    4
d    8    9    8
e    5    9    7
a    8    6    7
f    3    8    5
f    4    3    6
> rownames(mat1)<- c("f","a","d","c","b","d","e","a","f","f")
> mat1
  [,1] [,2] [,3] [,4]
f    4    7    6   17
a    5    6    1   12
d    4    3    1    8
c    2    2    9   13
b    4    5    4   13
d    8    9    8   25
e    5    9    7   21
a    8    6    7   21
f    3    8    5   16
f    4    3    6   13
> groups<- factor(rownames(mat))
> groups
 [1] f a d c b d e a f f
Levels: a b c d e f
# order the matrix by group first, and then within the subgroups, order by the rowSums
> mat1[order(groups,rowSums(mat)),]
  [,1] [,2] [,3] [,4]
a    5    6    1   12
a    8    6    7   21
b    4    5    4   13
c    2    2    9   13
d    4    3    1    8
d    8    9    8   25
e    5    9    7   21
f    4    3    6   13
f    3    8    5   16
f    4    7    6   17

It is extremely useful when I want to order the matrix by a pre-defined way and then plot the heatmap with heatmap2 in gplots package.

rehead a bam file

I downloaded some ChIP-seq bam files from the UCSC genome browser http://hgdownload.cse.ucsc.edu/goldenPath/hg19/encodeDCC/wgEncodeHaibTfbs/

and I found some bam files headers  are without chrY information

samtools view -H  my.sorted.bam
@SQ SN:chr1 LN:249250621
@SQ SN:chr2 LN:243199373
@SQ SN:chr3 LN:198022430
@SQ SN:chr4 LN:191154276
@SQ SN:chr5 LN:180915260
@SQ SN:chr6 LN:171115067
@SQ SN:chr7 LN:159138663
@SQ SN:chr8 LN:146364022
@SQ SN:chr9 LN:141213431
@SQ SN:chr10 LN:135534747
@SQ SN:chr11 LN:135006516
@SQ SN:chr12 LN:133851895
@SQ SN:chr13 LN:115169878
@SQ SN:chr14 LN:107349540
@SQ SN:chr15 LN:102531392
@SQ SN:chr16 LN:90354753
@SQ SN:chr17 LN:81195210
@SQ SN:chr18 LN:78077248
@SQ SN:chr19 LN:59128983
@SQ SN:chr20 LN:63025520
@SQ SN:chr21 LN:48129895
@SQ SN:chr22 LN:51304566
@SQ SN:chrX LN:155270560
@SQ SN:chrM LN:16571

for some downstream analysis, I have to rehead this bam file:

samtools view -H my.bam | sed '24 a @SQ SN:chrY LN:59373566' | samtools reheader  -  my.bam > my.reheaded.bam

sed to append a line "@SQ SN:chrY LN:59373566" at  the 24th line

ctr+v+tab to insert a tab in the terminal.

liftover wig file

I was looking at a public ChIP-seq data set, but the author did not put the raw sequence data in the GEO database. Instead, they uploaded a wig file of the ChIP-seq signal file and it was mapped to human hg18 reference genome.

All the other data sets I am analyzing are mapped to hg19, so I have to liftover this wig file to hg19 also.

Solution: CrossMap! http://crossmap.sourceforge.net/

Convert Wiggle/BigWig format files

Wiggle (WIG) format is useful for displaying continuous data such as GC content and reads intensity of high-throughput sequencing data. BigWig is a self-indexed binary-format Wiggle file, and has the advantage of supporting random access. This means only regions that need to be displayed are retrieved by genome browser, and it dramatically reduces the time needed for data transferring (Kent et al., 2010). Input wiggle data can be in variableStep (for data with irregular intervals) or fixedStep (for data with regular intervals). Regardless of the input, the output will always in bedGraph format. bedGraph format is similar to wiggle format and can be converted into BigWig format using UCSC wigToBigWig tool. We export files in bedGraph because it is usually much smaller than file in wiggle format, and more importantly, CrossMap internally transforms wiggle into bedGraph to increase running speed.

If an input file is in BigWig format, the output is BigWig format if UCSC’s ‘wigToBigWig‘ executable can be found; otherwise, the output file will be in bedGraph format.

Typing command without any arguments will print help message:

$ python2.7 CrossMap.py  wig

Screen output:

Usage:
  CrossMap.py wig input_chain_file input_wig_file output_prefix

Description:
  "input_chain_file" can be regular or compressed (*.gz, *.Z, *.z, *.bz, *.bz2,
  *.bzip2) file, local file or URL (http://, https://, ftp://) pointing to remote
  file.  Both "variableStep" and "fixedStep" wiggle lines are supported. Wiggle
  format: http://genome.ucsc.edu/goldenPath/help/wiggle.html

Example:
  CrossMapy.py wig hg18ToHg19.over.chain.gz test.hg18.wig test.hg19

It is very easy to use, after install it following the instruction I did:
CrossMap.py wig hg18ToHg19.over.chain.gz my.wig my_hg19

it generates a bigwig file, a sorted bedgraph and a unsorted bedgraph file.

it took me 36mins to convert a 1.4Gb wig file on my desktop with 4Gb ram.

qseq to fastq conversion

I was looking at some old ChIP-seq data with raw file in qseq format. I want to convert them to fastq file for mapping with bowtie.

A quick google I found:
http://www.biostars.org/p/6682/

the qseqtofastq C++ program http://www.dna.bio.keio.ac.jp/~krisp/qseq2fastq/ has to be compiled by scons, and I encountered some compiling problem. I gave it up and used this java program http://sourceforge.net/projects/snpeff/files/qseq2fastq.jar/download

Usage:

zcat myfile.qseq.tgz | java -jar qseqtofastq.jar -phred64 > myfile.fastq

it took me around 1 hour to finish the conversion of a 600MB tgz file on the computing cluster (single cpu 4GB ram).

I did have a problem at the end:

Exception in thread "main" java.lang.RuntimeException: java.lang.ArrayIndexOutOfBoundsException: 8
        at ca.mcgill.mcb.pcingola.Qseq2Fastq.main(Qseq2Fastq.java:51)
Caused by: java.lang.ArrayIndexOutOfBoundsException: 8
        at ca.mcgill.mcb.pcingola.Qseq2Fastq.main(Qseq2Fastq.java:43)

I counted the lines of the fastq file (divided by 4) and the original qseq file  , and they are equal.  So, I went ahead and mapped the fastq file with bowtie.

Several NGS bioinformatics training materials

1.  Next Generation sequencing wiki book http://en.wikibooks.org/wiki/Next_Generation_Sequencing_(NGS)

2. UCDavis training course https://training.bioinformatics.ucdavis.edu/documentation/

3. MSU training course by Titus Brown http://ged.msu.edu/angus/index.html

4. GOBLET http://www.mygoblet.org/training-portal

5.  more http://ged.msu.edu/angus/bioinformatics-courses.html

6. a list maintained by Stephen Turner in UVA http://stephenturner.us/edu.html

7. GVL practial protocols for ChIP-seq, RNA-seq, variant calling etc https://genome.edu.au/wiki/GVL

These will keep me busy for a while...I feel I do not have enough time to learn :)

Unix sort except the first line

Many times I have a file with a header  needs  to be sorted, but I do not want to sort the header.
I saw it on Twitter:

 Retweeted by Ming Tang

razor ‏@razZ0r  Mar 1

todays fav linux oneliner: command | (read -r; printf "%s\n" "$REPLY"; sort) > output sort everything except first line.

REPLY is the default variable for the read command.

one more general use of the "body" function
http://stackoverflow.com/questions/9281449/unix-skip-header-bash-function
http://unix.stackexchange.com/questions/11856/sort-but-keep-header-line-in-the-at-the-top

# print the header (the first line of input)
# and then run the specified command on the body (the rest of the input)
# use it in a pipeline, e.g. ps | body grep somepattern
body() {
    IFS= read -r header
    printf '%s\n' "$header"
    "$@"
}

IFS is the bash Shell Bourne variable: A list of characters that separate fields used by the shell to split text strings.

$#    Stores the number of command-line arguments that 
      were passed to the shell program.
$?    Stores the exit value of the last command that was 
      executed.
$0    Stores the first word of the entered command (the 
      name of the shell program).
$*    Stores all the arguments that were entered on the
      command line ($1 $2 ...).
"$@"  Stores all the arguments that were entered
      on the command line, individually quoted ("$1" "$2" ...).

Or an awk solution awk 'NR == 1; NR > 1 {print $0 | "sort -n"}'
It is very handy to use.

Test:
tommy@tommy-ThinkPad-T420:~$ cat body_sort.txt 
header
4
6
9
7
14
8

tommy@tommy-ThinkPad-T420:~$ cat body_sort.txt |( read -r; printf "%s\n" "$REPLY"; sort -n)

header

4

6

7

8

9

14

interactive visualization packages for R and python

I've been collecting potential packages for (interactive) visualization of Genomic data.
Here are the ones so far I've collected:

python:
Vispy http://vispy.org/gallery.html
pretty plotlib https://github.com/olgabot/prettyplotlib
D3py https://github.com/mikedewar/d3py
Yhat ggplot for python http://blog.yhathq.com/posts/ggplot-for-python.html
galry https://github.com/rossant/galry
Glue http://www.glueviz.org/en/latest/
Seaborn https://github.com/mwaskom/seaborn
Bokeh https://github.com/ContinuumIO/Bokeh

R:
Click me https://github.com/nachocab/clickme
plotly https://plot.ly/

others:
http://selection.datavisualization.ch/
Gpviz http://icbi.at/software/gpviz/gpviz.shtml
Jheatmap http://jheatmap.github.io/jheatmap/
Caleydo http://caleydo.github.io/caleydo-doc/3.1/index.html#!index.md

There are too many out there, I will need to focus on  one or two.

hosting bigwig by dropbox for UCSC visualization

First, you need to get an idea of what is a bigwig file: https://genome.ucsc.edu/goldenPath/help/bigWig.html

It is the binary form of wig file and allows UCSC genome browser to fetch only the data in the current window. I usually get wig file by MACS1.4 peak calling ChIP-seq data.
MACS2  now does not have the -w option any more. https://github.com/taoliu/MACS/

see a discussion in the google group:
https://groups.google.com/forum/#!searchin/macs-announcement/bedgraph$20ucsc$20track/macs-announcement/LBhAtmC-Zho/uZuxU8ZaqdEJ

MACS2 only creates a bedgraph file http://genome.ucsc.edu/goldenPath/help/bedgraph.html
and one can convert the bedgraph to bigwig:
https://github.com/taoliu/MACS/wiki/Build-Signal-Track
https://gist.github.com/taoliu/2469050
http://hgdownload.cse.ucsc.edu/admin/exe/linux.x86_64/

if you have a bam file, you can produce a bedgraph file by bedtools  genomeCoverageBed:
http://bedtools.readthedocs.org/en/latest/content/tools/genomecov.html
http://www.biostars.org/p/64495/#64680

you need to add a track line (--trackline option) for UCSC genome browser.
https://groups.google.com/forum/#!searchin/bedtools-discuss/bedtools$20bedgraph$20ucsc$20track/bedtools-discuss/3CibmlqIdWA/PW_bhgWQfVMJ

I followed the instructions https://genome.ucsc.edu/goldenPath/help/bigWig.html

To create a bigWig track from a wiggle file, follow these steps:

1. Create a wig format file following the directions here. Note that when converting a wig file to a bigWig file, you are limited to one track of data in your input file; you must create a separate wig file for each data track. Note that this is the file that is referred to asinput.wig in step 5 below.
2. Remove any existing 'track' or 'browser' lines from your wig file so that it contains only data.
3. Download the wigToBigWig program from the directory of binary utilities.
4. Use the fetchChromSizes script from the same directory to create the chrom.sizes file for the UCSC database you are working with (e.g. hg19). Note that this is the file that is referred to as chrom.sizes in step 5 below.
5. Create the bigWig file from your wig file using the wigToBigWig utility like so: wigToBigWig input.wig chrom.sizes myBigWig.bw
   (Note that the wigToBigWig program also accepts a gzipped wig input file.)
6. Move the newly created bigWig file (myBigWig.bw) to a http, https, or ftp location.
7. Construct a custom track using a single track line. The most basic version of the track line will look something like this:
   track type=bigWig name="My Big Wig" description="A Graph of Data from My Lab" bigDataUrl=http://myorg.edu/mylab/myBigWig.bw
   Optional values can include:

     autoScale         <on|off>                             # default is on
     alwaysZero        <on|off>                             # default is off
     gridDefault       <on|off>                             # default is off
     maxHeightPixels   <max:default:min>                    # default is 128:128:11
     graphType         <bar|points>                         # default is bar
     viewLimits        <lower:upper>                        # default is range found in data
     viewLimitsMax     <lower:upper>                        # suggested bounds of viewLimits, but not enforced
     yLineMark         <real-value>                         # default is 0.0
     yLineOnOff        <on|off>                             # default is off
     windowingFunction <mean+whiskers|maximum|mean|minimum> # default is maximum, mean+whiskers is recommended
     smoothingWindow   <off|[2-16]>                         # default is off
     transformFunc     <NONE|LOG>                           # default is NONE

   For further information on custom bigWig track settings, see the Track Database Definition Document. For further information on how bigWig settings are used in native Browser tracks, see the Configuring graph-based tracks page.
8. Paste this custom track line into the text box in the custom track management page.

I've encountered a coordinates problem mentioned here https://groups.google.com/forum/#!topic/macs-announcement/yPSPlKdTOwo when I tried to convert wig to bigwig.

the -clip option in the wigToBigWig program seems to resolve the problem, but it still gives warning messages. The best way is to use bedClip program here first http://hgdownload.cse.ucsc.edu/admin/exe/linux.x86_64/

Now, I need to use some public ftp or http to host the resulted bigwig file. I decided to use dropbox as I have used it for a while and I have 15G store space. The UCSC genome browser has problem to accept my link. I found a answer here:
http://bergmanlab.smith.man.ac.uk/?p=1989

"The first problem with the Share Link function is that the URL automatically generated by Dropbox cannot be read by the UCSC Genome Browser. For example, the link generated to the file “test.bed” in my Dropbox folder is “https://www.dropbox.com/s/7sjfbknsqhq6xfw/test.bed”, which gives an “Unrecognized format line 1″ error when pasted into the UCSC Browser.  This can easily be fixed if you just want to load a single custom track  to the UCSC Browser using Dropbox by simply replacing “www.dropbox” in the URL generated by Dropbox with “dl.dropboxusercontent”. In this example, the corrected path to the file would be “https://dl.dropboxusercontent.com/s/7sjfbknsqhq6xfw/test.bed”, which can be loaded by the UCSC Genome Browser automatically."

Finally I had the data visualized in UCSC!
lsd1 binding at the oct4 locus:

Next time, I will get a bedgraph file from the MACS1.4 or MACS2, and use the bdg2bw program https://gist.github.com/taoliu/2469050
" conversion to bedgraph is necessary to reduce the final bw size (up to 70%)"

=========================================
I got an email from dropbox, this was mentioned in the blog http://bergmanlab.smith.man.ac.uk/?p=1989
I guess I need to find another place to host my bigwig files...

Hi Ming, This email is an automated notification from Dropbox that your Public links have been temporarily suspended for generating excessive traffic. Your Dropbox will continue to function normally with the exception of Public links. For more information on suspended links, please visit the Help Center. If this is your first suspension, you may remove the suspension by visiting your account page.

how to get a genome-wide motif bed file

Someone was asking this question on Seqanswers http://seqanswers.com/forums/showthread.php?t=40762&highlight=genome+motif+bed

For motif analysis, the most popular program is MEME http://meme.nbcr.net/meme/
There are a bunch of tools in the suites including some useful ones for ChIP-seq
I also saw RAST http://rsat01.biologie.ens.fr/rsa-tools/index.html and oPOSSUM were mentioned http://opossum.cisreg.ca/oPOSSUM3/

I analyze ChIP-seq data a lot. Usually, one gets a bed file containing the positions of the peaks. However, many motif analysis programs require fasta file as input.

There are many ways to get fasta file based on coordinates.
See:
http://www.biostars.org/p/7481/
and my previous post http://crazyhottommy.blogspot.com/2013/04/batch-converting-coordinates-to.html

Going back to the question, I've found several ways to get a bed file containing the coordinates for the motifs.

1. Homer homepage
http://homer.salk.edu/homer/
at the very bottom, there are several links for human and mouse .The file contains all the known motif coordinates for the whole genome. Files are big (several Gb). one can get the specific motifs occurrences by grep.

2.  from UCSC software http://genome.ucsc.edu/ENCODE/analysisTools.html
ENCODE-motifs  http://compbio.mit.edu/encode-motifs/
at the bottom of the page:

only the human data are available.

3. Motif-map http://motifmap.ics.uci.edu/
click motif search

Several more organisms are supported

search the motif you want ( I use CTCF as an example)

Click save on the bottom right.

Click my motifs on the upper right.

export to bed or other format files.

several tools for Hi-C data, ChIP-seq and methylation data

See the links below:
http://omictools.com/differential-peak-calling/
http://omictools.com/chip-seq-and-beyond/3c-4c-5c-hi-c/hicup-s1473.html
http://omictools.com/analytical-pipelines8/
http://omictools.com/dmr/

There are too many tools out there for established high-throughput sequencing  based techniques.
1. you need to know what are out there. never try to re-invent the wheel
2. choose the right one to fit your own analysis. (only if none of them does what you want to do, write a script by yourself).  choose one with good documentation and is being actively developed.