install shellcheck in atom

First, download and install Atom.

then  install shellcheck  and linter:

brew install shellcheck

apm install linter

apm install linter-shellcheck

You may need to edit the config file to specify the shellcheck path:

Then, re-open Atom, and write some shell scripts:

See the warnings down in the bottom. Pretty cool!

linux command join with header

See the ipython notebook here 

https://github.com/crazyhottommy/scripts-general-use/blob/master/Shell/Join_command.ipynb

linux commands are very powerful.

They are small but can do small things well.
Today, I am going to demonstrate a very useful linux command join.
join is used to join two files based on a common "ID" column.
This is a common task when you have information stored in two files
and have a common column to link them together.

I first created two dummy tab delimited files: A.txt and B.txt
Let's have a look at the files:

In [1]:

cat A.txt

Jack TX 5
John CA 3
Luke WA 4
Tammy FL 2
Tommy FL 1

In [2]:

cat B.txt

1 450
2 300
3 400
5 150
6 600

column 3 in file A and column 1 in file B are common IDs that link these two files together.
Let's do an inner join:

In [3]:

join -1 3 -2 1 A.txt B.txt

5 Jack TX 150

-1 means file 1, which is A file. 3 means the third column.
-2 means file 2, which is B file. 1 means the first column.
The output is not we expected. A file and B file should have 4 rows linked together.
**It turns out files need to be sorted first according to the common
field for join to work.** Let's sort them first and join them:

In [4]:

sort -k3,3 A.txt > A.sorted.txt
sort -k1,1 B.txt > B.sorted.txt
join -1 3 -2 1 A.sorted.txt B.sorted.txt

1 Tommy FL 450
2 Tammy FL 300
3 John CA 400
5 Jack TX 150

we can also do right outer join (-a 1) :

In [5]:

join -1 3 -2 1 -a 1 A.sorted.txt B.sorted.txt

1 Tommy FL 450
2 Tammy FL 300
3 John CA 400
Luke WA 4
5 Jack TX 150

The default input seprator is space or tab, and output seprator is space.
we can change the output seprator to tab:

In [6]:

join -t$'\t' -1 3 -2 1 -a 1 -o auto A.sorted.txt B.sorted.txt

1 Tommy FL 450
2 Tammy FL 300
3 John CA 400
4 Luke WA 
5 Jack TX 150

Let's do a left outer join (-a 2) and fill the empty field with NA.
Note that only GNU join has the auto flag.

In [7]:

join -t$'\t' -e "NA" -1 3 -2 1 -a 2 -o auto A.sorted.txt B.sorted.txt

1 Tommy FL 450
2 Tammy FL 300
3 John CA 400
5 Jack TX 150
6 NA NA 600

You may think the intermediate sorted file is annoying. One can avoid them using
process substitution:

In [8]:

join -1 3 -2 1 -t $'\t' <(sort -k3,3 A.txt) <(sort -k1,1 B.txt)

1 Tommy FL 450
2 Tammy FL 300
3 John CA 400
5 Jack TX 150

Wow! It is so magic! Using process substitution can speed up your workflow.

So far, the files we have are without headers. What if they contain headers?
Let's make some dummy files first:

In [9]:

printf "name\tstate\tid\n" | cat - A.txt > A_header.txt
printf "id\tsalary\n" | cat - B.txt > B_header.txt

In [10]:

cat A_header.txt

name state id
Jack TX 5
John CA 3
Luke WA 4
Tammy FL 2
Tommy FL 1

In [11]:

cat B_header.txt

id salary
1 450
2 300
3 400
5 150
6 600

Let's join them together:

In [12]:

join -1 3 -2 1 -t $'\t' <(sort -k3,3 A_header.txt) <(sort -k1,1 B_header.txt)

1 Tommy FL 450
2 Tammy FL 300
3 John CA 400
5 Jack TX 150
id name state salary

Unfortunately, after sorting, the header went to the bottom of the file. Many times, the header may just go to the middle of the file. We can always delete the header first,
join the file and then add the header back, but we will need to have intermediate files.
I will use body function which will ignore the header.

In [13]:

join -1 3 -2 1 -t $'\t' <(cat A_header.txt | body sort -k3,3) <(cat B_header.txt | body sort -k1,1)

id name state salary
1 Tommy FL 450
2 Tammy FL 300
3 John CA 400
5 Jack TX 150

change the mac terminal color scheme and use oh my zsh!

I just got my lab mac pro. I wanted to change the color scheme of the default terminal, which is just black and white.

I googled around, and found solarized .

you can download it and unzip it.
Then, open your terminal (you can search "terminal" at the upper right corner)

go to the preferences:

and import the color scheme for terminals in the folder osx-terminal.app-colors-solarized, which is in the unzipped the folder solarized. I preferred the Solarized Dark ansi.terminal , and set it as Default.

Now,  download oh my zsh! by:

$ curl -L https://raw.github.com/robbyrussell/oh-my-zsh/master/tools/install.sh | sh

You will have to install git first for the upper command to work.
Now, when you fire up your terminal, it looks much more prettier! (there are many other schemes for oh my zsh, I found the default is good. You can change it by modifying the .zshrc file in your home directory.)

My first Software Carpentry workshop as an instructor

I just came back from the software-carpentry workshop held on April 2 and 3 at the University of Miami. The workshop link is here http://xuf12.github.io/2015-04-02-umiami/
and the Etherpad:https://etherpad.mozilla.org/2015-04-02-umiami

I taught Shell and R on the morning sessions and I enjoyed it very much. Most of the participants (biologists) are beginners for Shell and R, so I taught the very basics and keep my pace slow. It looks like they also enjoyed learning with me. It was my first time teaching to such a group of (around 15) people and I got myself refreshed on the basics.

The organizer Sawsan is a very nice lady and we enjoyed talking about teaching and (data) science. I met the other three instructors: Xu Fei, Matthew and Ashiwin. They are all wonderful instructors and I learned a lot from them. I also leaned a lot from the Git and SQL sessions. I think I should at least finish the online instructor training by Greg Wilson. I was in a job transition and missed the last session.  I wish I could attend more such workshops to refine my teaching skills.

Despite it was my first time teaching, I got encouraged by the students comments:

I still have a lot to improve though. I will need to speak better english (As a non-native speaker, it sometimes can be quite challenging). I am happy that I made most of the students understood :)

speed up grep

I had a list of gene names in txt file. There are around 500 genes with one gene name in one line, and I want to filter the gtf file from ensemble Homo_sapines.GRCh37.74gtf.gz

the gtf file contains 2244857 lines. I used grep to do it, but it takes very long (~1 hour).

what I used:

zcat Homo_sapines.GRCh37.74gtf.gz | grep -f gene_names.txt -w > my_genes.gtf

I searched on line, and found several posts in stackoverflow to speed up grep:
http://stackoverflow.com/questions/14602963/faster-grep-function-for-big-27gb-files
http://stackoverflow.com/questions/13913014/grepping-a-huge-file-80gb-any-way-to-speed-it-up
http://stackoverflow.com/questions/9066609/fastest-possible-grep

options to speed up:

1) Prefix your grep command with LC_ALL=C to use the C locale instead of UTF-8.

2) Use fgrep because you're searching for a fixed string, not a regular expression

I then used:

zcat Homo_sapines.GRCh37.74gtf.gz | LC_ALL=C fgrep -f gene_names.txt -w > my_genes.gtf

It runs much faster!

make a soft link

I got to know the ln linux command when I was reading The linux command line. I did not realize why I need this command to make links to the same data or file.

One situation I faced is that I have a lot of bam files  with long names in one folder, and when I execute a program in another folder, I have to pass the program a full path to the bam files.

while you think the TAB auto-completion helps most of the time,  it is still too much to specify the full path + long names (ugly). So I want to make a copy of the bam files in the current working director to save me from typing the full path. This is where ln command comes to rescue.

The other scenario is when you have multiple executable files in one folder ~/myprogram/bin, you want to execute the program from anywhere, you could link it to /usr/local/bin, or you can just copy the executable to /usr/local/bin or you can add ~/myprogram/bin to $PATH. see a link here http://unix.stackexchange.com/questions/116508/adding-to-path-vs-linking-from-bin

Let me demonstrate the usage of ln 

look at the man page

man ln

# I have two files in the playground directory and make two directories 

mkdir dir1 dir2
ls

output: tommy@tommy-ThinkPad-T420[playground] ls                              [ 2:32PM]
dir1  dir2  genes_h.txt  genes.txt

I then move the two txt files into dir1 and redirect to dir2 and make soft links of files in dir1 to dir2

mv *txt dir1
cd dir2
ln -s ../dir1/*txt .
ls

if we go to dir1 and make a softlink from there

cd ../dir1
ln -s ./*txt  ../dir2

The links will not work (too many levels of symbolic levels error message if you want to use the link ), instead you need to specify the full path

ln -s $PWD/*txt  ../dir2

see links here http://www.thegeekstuff.com/2010/10/linux-ln-command-examples/
and http://www.thegeekstuff.com/2010/10/linux-ln-command-examples/

linux commands basics

I am attending the NGS course at MSU. This is a great course with great instructors and friendly colleagues.
I highly recommend this course to everyone. http://bioinformatics.msu.edu/ngs-summer-course-2014

This morning, we learned SNP calling by samtools and sam file specification (I will write another blog for the SNP calling) .in the night , TA Elijah gave an awesome introduction to linux commands.
personally, I think this should be taught in the first day of the course. ( I am already pretty familiar with basic linux commands, but it does cause a lot of frustrations for beginners).

I took the notes, and put the commands that taught in a gist, see below and enjoy linux commands!


linux basics by Tommy Tang is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.