mod_top can help you find the needle in the haystack when searching for the cause of production outages, slowness or system overutilization.

Screenshot 1

Find Bottlenecks

An admin had complaints of slow pages. By sorting by TIME they were able to find that this lookupCustomerProfile function is taking a lot of time. We see in the same row that it is not taking a lot of CPU time, suggesting that it is waiting on something—probably the database. Also very little MEMORY allocated during that routine rules out large data transfers over the network.

Screenshot 2

Discover Unexpected Production Issues

This server was chewing up CPU, so the admin used mod_top sorted by CPU time and discovers a lot of CPU being consumed by this getNextRowOfCustomerProfile operation. Checking with the developer, it doesn't seem like that could be possible because the function doesn't do anything except load a database row and parse it. However by looking at the number of calls we realize this is being called very frequently by Profile::processCustomer and realize that in aggregate under production load this small amount of work adds up! It looks like Profile::processCustomer is calling getNextRowOfCustomerProfile many many times per second and that is accounting for the CPU burn. When the developer wrote this perhaps they didn't realize there could be so many rows and figured it wouldn't hurt to load them one at a time.

Screenshot 3

Uncover Memory Hogs

We noticed that each Apache instance was using 45 MB of RAM, and yet the application was a simple forms-processing app. Something didn't seem right. By using mod_top sorted by MEMORY consumption we discover that the library that was used for the processing is a pig and seems to create all these different versions of the data in memory before it stores any of it. After a little investigation, there was a configuration parameter in the forms validator that could be set to make it not do all this loading and cross-checking. That reduced the memory footprint and let there be more concurrency on a machine we didn't really think we needed to upgrade.