Anti Virus Software: Norton, McAfee, Trend Micro, or Hauri?
By: Relevant Technologies

Executive Summary

During the last several years, viruses have become increasingly more sophisticated. At the same time, the Internets ever growing popularity and the steady adoption of always on broadband technologies have allowed viruses to spread quickly. Now more than ever, it is important to defend every computer in an organization against viruses in the most effective manner possible. The problem is that there are a number of available anti virus products available, each with their own strengths and weaknesses. To determine which anti virus product is the most effective, weve tested four leading anti virus products against each other. In this paper, we will describe our testing methods and will present you with the results of each test.
Types of Anti Virus Software
For our test, we are comparing anti virus software from Norton (Symantec), McAfee, Trend Micro, and Hauri. Each of these companies manufacturers multiple anti virus products, each intended for different purposes. For the tests described in this paper, we chose to use the version intended for desktop computers. This means that our test results are valid for large businesses, small businesses, and for home users alike. The tests were conducted on January 22, 2003. During these tests, the latest virus definition files were downloaded for each product.

Product Being Tested

At the time that this paper was written, there were four major players in the anti virus. This list includes Norton, McAfee, Trend Micro, and Hauri. The comparisons are made in a random order and dont reflect our preferences or test results.
Test 1; Basic Detection and Repair
For our first test, we placed seven infected files onto a test PC. The files were infected with viruses such as Nimda, Klez, and Fun Love. The idea behind the test was simply to determine each products effectiveness at detecting and cleaning common viruses. For this test, we disabled each anti virus programs automatic scanning engine, copied the infected files to a folder, and then manually scanned the system. We began our tests using McAfee. McAfee had no trouble detecting our infected files. As you can see in Figure A, after detecting the infected files, McAfee asks the user to clean the infected files, and if the clean fails, to delete the files. Figure A
McAfee detects the infection and asks the end user to take action.
The end result was that McAfee was able to detect all seven of the infections, but was unable to clean any of them. You can see the test results in Figure B. Figure B
McAfee detected all 7 infected files but was unable to clean any of them.
Next, we placed the same seven infected files onto the test machine and used Norton Antivirus try to detect and repair them. As you can see in Figure C, Norton claimed to detect nine infections, even though only seven files were actually infected. Figure C
Norton AntiVirus detected nine infections instead of seven.
Like McAfee, Norton AntiVirus asked the user to click a button to begin the repair process. After clicking the Repair button, Norton reported that it was unable to repair any of the seven files. Norton then recommended that the files be quarantined. You can see these test results in Figures D and E. Figure D
Norton AntiVirus detected nine infected files instead of seven, and was unable to repair any of them.

Figure E

Norton AntiVirus was unable to repair the infected files.
Next, we attempted to detect and repair the same nine infections using Trend Micros PCcillin. As you can see in Figure F, the Trend Micro product detected seven infections, and was unable to clean any of them. The Trend Micro product automatically quarantined the files that it was unable to clean. Figure F
Trend Micros PC-cillin detected seven infections and quarantined them rather than cleaning them.
Finally, we tested Hauris ViRobot Expert. ViRobot was able to detect all seven infections and was able to repair them faster than we could blink. You can see these test results in Figure G. Figure G
Hauris ViRobot detected all seven viruses and was able to repair them automatically.
Test 2; Integrity of Repaired Files
In Test 1, Hauris ViRobot was the only anti virus program that was able to repair the infected files. For our second test, we acquired some infected files that any anti virus program should be able to repair. We infected a system DLL file and a system level executable with Nimda, and Fun Love respectively. As you can see in Figure H, we began with a file named CreateCDDA.DLL and a file named WIN32F~3.EXE. In Figure H, pay close attention to the file sizes, date / time stamp, and to the files icons. Figure H

Pay close attention to the file names, icons, sizes, and date time stamps. We began this test by running McAfee against the two infected files. Upon doing so, McAfee detected the virus and prompted us for what action to take. We clicked the Clean button, and McAfee reported that the files had been cleaned, as shown in Figure I. However, if you look at the files in the figure, youll notice that the file sizes have changed. This is normal since viral code has been deleted from the file. Youll also notice though that the date / time stamp has changed and that the WIN32F~3.EXE files icon has changed to a generic icon.
At first, having an altered date / time stamp and an altered icon may not seem like a big deal. However, its very important to preserve date / time stamps. For example, many times when you contact Microsoft for Technical support, they will ask you for the date / time stamp on various system files, in order to determine the files version. If the date / time stamp has been altered, its impossible to tell at a glance if the file is the correct version. Likewise, if an executable files icon has changed, it could possibly mean that the file has lost some of its integrity, and that more may have been removed than just viral code. Figure I
McAfee cleaned the files, but altered the date / time stamp and the icons.
Next, we tried to disinfect the same set of viruses using Nortons. Nortons detected the infection with no problems. When we clicked the Repair button, We received a message that the infected DLL file was repaired, but that the repair failed on the WIN32F~3.EXE file, which was infected with Fun Love. The strange thing about this is that the CreateCDDA.DLL file was infected with Nimda. Nimda is basically a virus that built on Fun Love. Therefore, it seems strange that Nortons could fix Nimda, but not Fun Love. You can see the test results in Figure J. After Nortons completed, the files icons were preserved, but the date / time stamp was reset, even for the file that couldnt be repaired. You can see this in Figure K. Figure J
Nortons cleaned Nimda, but not Fun Love.

Figure K

Nortons reset the files date / time stamp.
After completing our testing with Nortons, we tested PC-cillin. The Trend Micro product detected four viruses even though there were only two files. As you can see in Figure L, PC-cillin misidentified the viruses and simply quarantined the viruses. Figure L
PC-cillin misidentified and miscounted the viruses.
Finally, we tested Hauris ViRobot against the same two infected files. As you can see in Figure M, ViRobot not only repaired the infected files, but also managed to preserve the date / time stamp and the icon. We should point out though that while the results were very obvious with the other three products, we had to perform a manual screen refresh by pressing F5 to see what ViRobot had done with the files. Figure M

ViRobot repaired the files and left the icons and date / time stamps intact.
Test 3; Detecting Viruses In Memory
When we approached Hauri about our tests, they claimed that their ViRobot products could actually detect viruses in memory and could even clean individual executing processes. Hauri provided us with a utility that is designed to test a systems memory for the existence of the Klez virus. Because this utility is a closely guarded trade secret, our non disclosure agreement with Hauri prevents us from revealing the name of the utility. In the screen shots that youll see in this section, file names and commands have been blurred, for legal reasons. However, the screen shots have not been doctored in any other way. Needless to say, we were immediately suspicious of this utility since it was provided to us by one of the anti virus manufacturers under such secrecy. However, rigorous independent testing has confirmed that the utility that well be using for Test 3 is indeed trustworthy.
For this test, we used the SQL Server client configuration utility as a test executable. For the test, we infected the utility with the Klez virus. For the test, we verified that the virus was not present in memory, ran the executable to infect the system, verified that the virus was present in memory, disinfected the virus, and then checked the systems memory to see if the memory was still infected. You can see an example of this process shown in Figure N. In this figure, you can see where we tested the memory, infected the system by loading CLICONFG, and tested the memory again. Figure N
This is how we test to see if a systems memory is infected.
We began by testing McAfee. When McAfee ran, it detected the virus right away. McAfee then closed the infected process (The SQL client configuration utility), and then reported that the system was clean. However, as you can see in Figure O, the systems memory was still infected. Figure O
The systems memory was still infected, even after the virus was cleaned by McAfee.
Next, we repeated the test with Norton AntiVirus. Norton AntiVirus detected the virus, but was unable to repair it. Norton left the infected process, and the systems memory remained infected, as shown in Figure P. Figure P

Norton was unable to disinfect the virus.
For the next test, we attempted to scan for the virus with PC-cillin. As you can see in Figure Q, PC-cillin detected the virus, but was unable to clean it. The virus was also still present in memory, and the infected process continued to run. Figure Q
PC-cillin detected the virus, but could not repair it.
Finally, we repeated the test using ViRobot. Like the other antivirus products, ViRobot had no trouble detecting the infection. However, ViRobot then displayed the message shown in Figure R. This message indicated that the infected file was presently running. ViRobot then gave us a chance to save any documents that might have been open, before closing the infected process. Figure R
ViRobot detected the infection and asked to close the infected process.
After closing the infected process, ViRobot disinfected the file and the systems memory, and then reopened the process. If you look at Figure S, you can see that the memory was completely clean after ViRobot finished cleaning the system. Figure S
ViRobot was able to remove the infection from memory.

Test 4; Performance

For our final test, we wanted to benchmark each products performance during a full system scan. We performed this test because the more processor time that a product uses during a system scan, the less responsive that the PC will be. Therefore, we were checking to see which product has the lowest processor utilization. For this series of tests, we closed all running applications except for the product that we were testing. We then initiated a full system scan, and opened the Windows Task Manager to watch the Performance tab. We waited until a representative amount of activity had occurred prior to taking the screen shots. We began the process by performing a full system scan with McAfee. As you can see in Figure T, although there were spikes in the processor utilization level, McAfees overall CPU utilization was relatively low, averaging around 30%. Figure T
McAfee had around a 30% CPU utilization.
Next, we repeated the same test using Norton AntiVirus. As you can see in Figure U, Norton AntiVirus had nearly a 100% CPU utilization during the scanning process. The period of low activity that you see in the figure prior to the heavy activity was generated by us simply loading the Norton AntiVirus console The full system scan began at the point in the graph where the activity increased so dramatically. Figure U

Norton AntiVirus held the processor at near 100% utilization during the scan.
At this point, we tested PC-cillin. PC-cillin performed very well in the processor utilization test. The average processor utilization was well under 20%, as shown in Figure V. Figure V
PC-cillin had very low processor overhead.
For our final test, we measured the processor utilization while ViRobot was scanning the system. As you can see in Figure W, ViRobot sustained an extremely low level of activity, well below 10% CPU utilization. The spikes that you see in CPU activity at the beginning of this chart were from when we loaded the ViRobot console. Figure W
ViRobot had extremely low CPU usage.

The Results

Although the table below outlines the good and bad points of each product, determining the best product isnt as simple as counting to see which product has the most points. The reason for this is that some features are more important than others and therefore disserve stronger consideration. The chart below is a weighted comparison of the products based on which features are the most important. In this analysis, each product has been given between one and five points for each area of comparison, with five representing the highest possible score. The products score in each area is multiplied by its weight to determine the total number of points for the feature. At the end, all of the points are tallied together to determine the results.
Weight (Default) 50% 20% 20% Feature and Possible Points Virus Detection and Cleansing (50x5=250 possible points) Ability to Repair Viruses Completely (20x5=100 possible points) Ability to Detect and Repair Infections in Memory (20x5=250 possible points) Performance (10x5=50 Possible Points) Total Value of Possible Points: 500 McAfee 3 (150 points) 4 (80 points) 3 (60 points) 3 (30 points) 320 points Norton 3 (150 points) 3 (60 points) 1 (20 points) 1 (10 points) 240 points Trend Micro 4 (200 points) 1 (20 points) 1 (20 points) 4 (40 points) 280 points Hauri 5 (250 Points) 5 (100 points) 5 (100 points) 5 (50 points) 500 points

10% 100%

Conclusion
As you can see from the charts above, our absolute favorite product was newcomer Hauris ViRobot, which earned a perfect score. Our second favorite was McAfee, followed by Trend Micro, with Norton in last place. As you view these results, remember that Relevant Technologies is an independent security research firm, and that we have provided screen shots of the actual tests, to validate our findings. If you would like more information about any of the products that we have discussed in this paper, you can contact each respective company via their Web site. The addresses are as follows:

Contact Information

McAfee: http://www.mcafee.com Norton: http://www.symantec.com Trend Micro: http://www.trendmicro.com Hauri: http://www.hauriusa.net You may also visit Relevant Technologies on the Web at http://www.relevanttechnologies.com
Copyright 2003, Relevant Technologies All Rights Reserved