Timeline for Counting k-mers
Current License: CC BY-SA 3.0
23 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Mar 2, 2015 at 22:46 | vote | accept | CommunityBot | moved from User.Id=9206 by developer User.Id=3572 | |
| Feb 12, 2015 at 11:02 | history | bounty ended | CommunityBot | ||
| Feb 6, 2015 at 22:14 | comment | added | FUZxxl | @Lembik Doesn't seem too slow for me. Maybe they're just bad programmers. | |
| Feb 6, 2015 at 22:13 | comment | added | user9206 | It's something that people really want to do in bioinformatics. But they are always complaining it is too slow and uses too much memory :) I will leave the bounty awarding until the end of the period if that is OK. | |
| Feb 6, 2015 at 22:12 | comment | added | FUZxxl | @Lembik Well, I guess that bounty is mine then :-). Let's see if someone finds a better solution. How did you actually get the idea for this challenge? | |
| Feb 6, 2015 at 22:11 | history | edited | FUZxxl | CC BY-SA 3.0 |
added 9 characters in body
|
| Feb 6, 2015 at 22:11 | comment | added | FUZxxl | @Lembik that's because the file only has about that many entries (go figure). It's the reason why the compressed representation is so small; the difference is all zeros at the end. (they actually all decrement by one, but that one is not part of the difference representation). | |
| Feb 6, 2015 at 21:43 | comment | added | user9206 | Well.. it's very impressive! I get to 4000 at least. Also I notice that the number of kmers goes up very slowly at this point. I wonder if there is some way of taking advantage of this. For example, the last 4 values up to k = 4000 are 240581805 240581816 240581827 240581838 | |
| Feb 6, 2015 at 21:15 | comment | added | FUZxxl |
@Lembik Do not use cat. Use shell redirection like in ./dsskmer <~/Downloads/chr2.fs. The code needs to know how long the input file is and that's not possible with a pipe.
|
|
| Feb 6, 2015 at 21:08 | comment | added | user9206 | OK.. I had to do sudo ldconfig as well.Now I get cat ~/Downloads/chr2.fa |./dsskmer Cannot seek stdin: Illegal seek | |
| Feb 6, 2015 at 20:25 | comment | added | FUZxxl | and that has to be the last argument. | |
| Feb 6, 2015 at 20:24 | history | edited | FUZxxl | CC BY-SA 3.0 |
added 39 characters in body
|
| Feb 6, 2015 at 20:24 | comment | added | FUZxxl |
Ah sorry, you have to compile with -ldivsufsort. I forgot to say that.
|
|
| Feb 6, 2015 at 20:00 | comment | added | user9206 |
I always get gcc -Wall -O3 -o dsskmer -DICAP=64 dsskmer.c /tmp/ccsuEc2x.o: In function main': dsskmer.c:(.text.startup+0xf9): undefined reference to divsufsort' . How exactly are you compiling it? I just followed the instructions in the file INSTALL to install libdivsufsort.
|
|
| Feb 6, 2015 at 15:58 | history | edited | FUZxxl | CC BY-SA 3.0 |
added 500 characters in body
|
| Feb 6, 2015 at 15:53 | comment | added | FUZxxl | Yeah, it works. I didn't bother checking before. Wait a minute as I update the post. | |
| Feb 6, 2015 at 15:41 | comment | added | FUZxxl | @randomra It should indeed be possible to cap the highest k we observe and I was able to cut the runtime to under one minute with such an approach. Let me check. | |
| Feb 6, 2015 at 15:23 | comment | added | randomra | @FUZxxl did you plot the distribution of the maximum p-values to see how much you can speed up step 3 by cutting of if p>k (counting only to k-mers)? although if step 2 takes 20 secs there might not be much room there. great explanation btw | |
| Feb 6, 2015 at 14:59 | comment | added | FUZxxl |
If you like, you can place strategic fprintf(stderr, "doing step n\n"); statements to figure out how long the steps take.
|
|
| Feb 6, 2015 at 14:49 | comment | added | FUZxxl | step one takes about three seconds, step two takes about 20 seconds, step three takes the entire rest. The time taken by step four is negligible. | |
| Feb 6, 2015 at 14:45 | comment | added | user9206 | This is very interesting. Can you break down the time taken by each part? It seems constructing the suffix array is less than 10% of the time. | |
| Feb 6, 2015 at 13:13 | history | edited | FUZxxl | CC BY-SA 3.0 |
remove stray prototype and change compression format to bzip2
|
| Feb 6, 2015 at 2:07 | history | answered | FUZxxl | CC BY-SA 3.0 |