Today I had to debug a crash to my application that happened in a really weird place. I examined the core dump and I was getting a segmentation fault at the time() function, which is part of the standard library.

Evidently, the chances that there was a bug in time() were pretty slim, so the problem must have been elsewhere, and merely manifested itself as a crash in time().

Opening the core in gdb, the GNU Debugger, and checking the backtrace command showed that the stack was corrupted: instead of getting a nice backtrace leading all the way to main (or to the clone() call that created my thread), I had about 6 levels of proper stack and then over 900 levels of “??” below that. Another obvious hint was that the hexadecimal addresses of functions in those invalid stack levels were completely different from the numbers seen in proper stack levels.

A stack corruption can only mean one thing: someone wrote something over the stack and filled the stack pointer address with garbage instead.

I then proceeded to look at the stack contents, hoping to find from which point did values start to look odd. In gdb, I ran the backtrace full command. This shows all local variables as well.

From there on, it was easy to spot a char[] buffer at the lowermost valid stack level that was being updated by functions higher up in the stack. If that buffer had overflowed, it would certainly make everything from there on in the stack invalid.

There were other pointers in that stack level right after the suspicious buffer. Using the up command, I went up, up, up until I reached that stack level, and then I could check the pointers using the print command. Indeed, gdb replied “cannot reach memory address” for their values — the pointers were invalid.

With the down command I went down the stack, right to the function that was manipulating that buffer. A quick look at the code, combined with checking the values of local variables with print confirmed my suspicions. An off-by-one error made my loop go beyond the end of the buffer, corrupting the stack and causing the crash.

As for time()? It didn’t really crash there. Its return value was being assigned to an address that was made invalid by stack corruption, and gdb couldn’t tell the difference between the crash happening at time() or at its return value, probably due to compiler optimizations.

It dawned on me that, decades from now, when we get nostalgic about the 2000’s, it won’t be whatever songs were playing on the radio that will instantly transport us to that era. But these.

Part I - The Originals

1. “Bed Intruder” - Gregory Brothers feat. Anthony Dodgson

The Gregory Brothers were the redemption to auto-tune. 95 million views.

2. “Chocolate Rain” - Tay Zonday

This song is perhaps the symbol of the golden era user-generated content on Youtube, circa 2006, before Vevo and design changes that moved the focus away from communities and into corporate content. 74 million views.

3. “What What (In The Butt)” - Samwell

What can I say. 44 million views and even a South Park parody.

Part II - The Loopers

4. Nyan cat

Ah, the loopers. The internet is not complete without them. Nyan cat got over 50 million views.

5. “The Llama Song” - Burton Earny

After “Hampsterdance” in the late 90s, I think this was one of the earliest major loopers. Originally from the Albino Blacksheep website, I’m posting here a Youtube upload for your convenience.

6. “Ievan Polkka” - Loituma

Made popular by leekspin.com, this is actually part of a Finnish song performed by a folk local group. Here’s a link to a full performance (it’s quite good!), but the one we’ll remember most is this:

Part III - The Subversions

7. “Dragostea tin dei” - O-Zone

Sometimes one fan can make the fame of one song. In 2004 a guy posted a video of himself… oh boy, Wikipedia can describe it better than I ever could: …”wearing headphones and lipsyncing to the audio of the original O-Zone track whilst moving his head, shoulders and arms gesticulating to the music in an animated and earnest manner.”

And thus was born “Numa Numa“, the second most watched viral video of all time: over seven hundred million views.

8. “Я очень рад, ведь я, наконец, возвращаюсь домой” (”I Am Glad, Cause I’m Finally Returning Back Home”) - Eduard Khil

It all started with this website: http://trololololololololololo.com/. Perhaps the most appropriate domain name in history.

9. “Tunak Tunak Tun” - Daler Mehndi

The song was originally released in 1998, but the video made waves through the internet during the 2000s. The first of many Indian music videos and snippets from Bollywood movies that amused the world:

10. “Never Gonna Give You Up” - Rick Astley

And finally, of course, the song that left a mark in two separate decades, and for very different reasons. Ladies and gentlemen, the one and only:

A piece from a private message I posted in a forum, which I thought it could be useful to share.

As advice for somebody just starting out writing system utils, what resources have you found the most useful/valuable over the years? What is worth spending lots of time to understand? Best books? Advice that you wish you knew 8 years ago?

Oh, good questions. I can only think of general pieces of advice; you may already know some, most or all of them, but let’s see:

• If you want to write system utils, do it in C. Yes, sometimes it’s annoying, pointers, cumbersome string handling, etc. I like other languages better too. But it’s the most portable option and the only thing that’s guaranteed to stay around in Unix in the long run. You’re writing code today but you may end up using it 15 years from now. Today’s Python may be the 90’s Tcl. I do write stuff in other languages (actually, my research work was in the Programming Languages field, so I’m all for language diversity) but if you’re doing system stuff which you want to be as widely available as possible, do it in C.
  • The so-called “pitfalls” of C are more a matter of discipline than anything else. Whenever you allocate a structure to start passing pointers of it around, define which one pointer will be the “owner” of that structure; the one responsible for deleting it. Make sure other pointers don’t outlive the owner and make sure the owner frees the memory. It’s no black magic.

  • One of my motivations for writing htop was actually improving my C skills, which were weak (and some of the early parts of htop still show it, look at the amount of asserts in Vector.c), but nowadays I’m work as a C coder.

  • If you’re writing for Unix only, C99 is fine (and much nicer to write than ANSI C). If you’re planning to support Visual Studio, stick to ANSI C. Depending on what you’re doing you can get away with C99 and using GNU compilers on Windows, too.

• Learn the GNU Autotools. Same thing: it’s even more annoying than C, but in the end it’s what provides the best experience to end users. You do get a lot of features for free, such as support for cross-compiling, etc. Things you may not need now, but that will pop up as feature requests once you have a user base. And to grow a user base, you want your installation to be just “type in ./configure; make; sudo make install” and not “install this-and-that build tool”. Distro packagers will thank you, and cooperation with them is fundamental to get your app out there.
  • A neat side effect of learning Autotools as a developer is that you develop skills in troubleshooting problems when installing other packages. Autotools is ubiquitous, so being proficient at it is another professionally useful skill.

• Similarly, if you’re writing libraries, learn Pkgconfig. If you’re using libraries, learn how to use Pkgconfig from within Autotools. Pkgconfig is nice and will make your life much simpler, the more you use it.

• I haven’t used many physical books, but there are tons of online resources that will help you with the above. The “Autobook” is something you’ll quickly end up in right after your first Google searches for Autotools stuff.
  • man-pages are your friends. You’ll code more efficiently by refering to them (when you need to check the order of the arguments for strncmp for the 1000th time) instead of relying on online search for everything (which is more distracting).

  • I use pinfo as my man reader (with “alias man=pinfo -m” in my shell) because it makes hyperlinks between man-pages. Very efficient.

• You don’t need to use an IDE, but that doesn’t mean you shouldn’t rely on debugging tools.
  • Valgrind is your best friend. In comparison, C programming was miserable before it.

  • Build with “-g”, run with “ulimit -c unlimited” in your shell at all times. Learn to load core files with gdb and examine backtraces for post-mortem analysis when your program

  • If you decide to open the can of worms of multithreading, you can set custom thread names with the prctl command (and then examine them with nice names in htop)

  • Other tools I use often: strace, lsof.

That’s all I can think of for the moment, I hope that helps!

Happy hacking!

These are some “notes to self”, in case I ever need to reconfigure a terminal to get all keys working properly on Dit:

Command line for urxvt

urxvt -cr green -fn '*-lode-*' -fb '*-lode-*' -fi '*-lode-*' -fbi '*-lode-*' \
-bg black -fg '#c7c7c7' -sb -sr +st -sl -1 -b 0 -tn rxvt

Setting Ctrl-Shift on urxvt

Add this to ~/.Xdefaults:

urxvt.saveLines: 10000
urxvt.keysym.C-S-Up: \033[1;6A
urxvt.keysym.C-S-Down: \033[1;6B
urxvt.keysym.C-S-Right: \033[1;6C
urxvt.keysym.C-S-Left: \033[1;6D
urxvt.iso14755: false
urxvt.iso14755_52: false
urxvt.colorBD: #fff
urxvt.colorIT: #ddf

The “saveLines” is unrelated, but it’s nice to have. :)

Force Ctrl-H to generate ^H (0×08)

Try:

infocmp | grep kbs

If it says “kbs=^H”, there’s your problem. Run:

infocmp > rules.txt

Edit rules.txt to make sure it says “kbs=177”. Then recompile/reinstall the file with:

sudo tic rules.txt

Running it with “sudo” will make it sure it reinstalls to the proper systemwide place, or else it will install to ~/.terminfo and you’ll have to configure the $TERMINFO variable properly.

(Terminfo information from this page.)