In order to verify the quality of software, you have to use a lot of different tools, including static and dynamic analyzers. In this article, we'll try to figure out why only one type of analysis, whether static or dynamic, may not be enough for comprehensive software analysis and why it's preferable to use both.
Our team writes a lot about the usefulness of static analysis and the benefits it brings to your projects. We like to run our tool on various open-source projects to find possible bugs, which is our way to popularize the static code analysis method. In its turn, static analysis helps to make programs more high-quality and reliable and reduce the number of potential vulnerabilities. Perhaps everyone who is directly involved in work on source code has that feeling of satisfaction at having bugs fixed. But even if the process of successfully spotting (and fixing) bugs doesn't trigger your endorphins, you surely enjoy the thought of having development expenses reduced thanks to the static analyzer, which has helped your programmers use their time more effectively and efficiently. To find out more about how you can benefit from the use of static analysis in terms of money, see this article. It gives an approximate estimate for PVS-Studio, but those results can be extrapolated to other static analysis tools available on market.
All said above seems to suggest that the purpose of static analysis is to find bugs in the source code as early as possible, thus reducing the expenses on bug fixing. But why do we need dynamic analysis then, and why sticking only to one of the two techniques may be insufficient? Let's give more formal and clear definitions of static and dynamic analyses and try to answer these questions.
Static code analysis is the process of detecting errors and code smells in software's source code. To analyze a program, you don't need to execute it; the analysis will be performed on the available code base. The closest analogy to static analysis is the so called code review except that static analysis is an automated version of code review (i.e. performed by a bot program).
The main pros of static analysis:
The objective cons of static analysis:
It should be noted that static analyzers don't focus exclusively on bug catching. For instance, they can provide recommendations on code formatting. Some tools allow you to check your code for compliance with the coding standard your company sticks to. This includes indentation of various constructs, the use of space/tabulation characters, and so on. In addition, static analysis can be helpful for measuring metrics. A software metric is a quantitative measure of the degree to which a program or its specifications possess some property. See this article to learn about other uses of static analysis.
Dynamic code analysis is the analysis performed on a program at execution time. This means you must have your source code converted into an executable file first. In other words, code containing compilation or build errors can't be checked by this type of analysis. The check is done with a set of input data fed to the program under analysis. That's why the effectiveness of dynamic analysis directly depends on the quality and quantity of the test input data. It is this data that determines the extent of code coverage at the end of the test.
With dynamic testing, you can get the following metrics and warnings:
The main pros of dynamic analysis:
The cons of dynamic analysis:
Dynamic analysis is particularly useful in those areas where program reliability, response time, or resources consumed are the primary concern. A real-time system managing a critical production sector or a database server are some examples of such systems. Any error in these areas can be critical.
Getting back to the question why sticking only to one of the two types of analysis may not be sufficient, let's take a look at a couple of quite trivial examples of bugs that one analysis method has no problems diagnosing while the other is not fit to detect, and vice versa.
The following example is taken from the Clang project:
MapTy PerPtrTopDown;
MapTy PerPtrBottomUp;
void clearBottomUpPointers() {
PerPtrTopDown.clear();
}
void clearTopDownPointers() {
PerPtrTopDown.clear();
}
A static analyzer would point out that the bodies of the two functions are identical. Of course, two functions having identical bodies aren't necessarily a definite sign of a bug, but it is very likely that they have resulted from using the copy-paste technique combined with carelessness on the programmer's side - and that leads to unexpected behavior. In this case, the clearBottomUpPointers method should call the PerPtrBottomUp.clear method. Dynamic analysis wouldn't notice anything wrong in this example because it's an absolutely legitimate piece of code from its point of view.
Another example. Suppose we have the following function:
void OutstandingIssue(const char *strCount)
{
unsigned nCount;
sscanf_s(strCount, "%u", &nCount);
int array[10];
memset(array, 0, nCount * sizeof(int));
}
In theory, a static analyzer could suspect there's something wrong with this code, but implementing such a diagnostic is a very difficult and pointless task. The example is taken from this article, which also elaborates on why it's a bad idea to teach static analyzers how to diagnose errors like that. In brief, static analyzers are very bad at figuring out that a call of the memset function may result in indexing beyond array bounds as they cannot foresee what number will be read from the strCount string; and if the value of strCount is read from a file, it becomes an impossible task for static analysis altogether. On the other hand, a dynamic analyzer would have no trouble noticing and pointing out the memory handling error in this code (given that the program is fed the right data).
This article doesn't aim at comparing static and dynamic analyses. There's no single technique that could diagnose the whole variety of software defects. Neither type of analysis can completely replace the other. To improve the quality of your programs, you'll have to use different types of tools so that they complement each other. I hope the examples shown above are persuading enough.
I don't wish to look too biased toward static analysis, but it is this technique that's being most spoken of and, more importantly, included by companies into their CI processes lately. Static analysis acts as one of the steps of the so called quality gates to building a reliable and high-quality software product. We believe static analysis is going to become a standard software development practice in a couple of years, just like unit testing once did.
To wrap up, I'd like to point out once again that dynamic analysis and static analysis are just two different methods, which complement each other. In the end, all these techniques serve the single purpose of increasing software quality and reducing development expenses.
References: