Your Daily Source for Apache News and Information

The Netcraft Web Server Survey is a survey of Web Server software usage on Internet connected computers. We collect and collate as many hostnames providing an http service as we can find, and systematically poll each one with an HTTP request for the server name.
In the June 2001 survey we received responses from 29,302,656 sites.

Microsoft is the sum of sites running Microsoft-Internet-Information-Server, Microsoft-IIS, Microsoft-IIS-W, Microsoft-PWS-95, & Microsoft-PWS.

Platform groupings are here.

Counting computers running the Web

Introduction

One of the common observations made about the Web Server Survey is that it counts hostnames rather than physical computers, and so is not a suitable metric to indicate hardware installed base or license sales. Technically sophisticated hosting companies can run several thousand sites on a single computer, and the great majority of the world's web sites are located at hosting and co-location companies rather than on peripheral networks.

Building on the operating system detection techniques used by the What's that site running? query and Netcraft's commercial research, we have attempted to address this. Netcraft has developed a technique that, with an error margin, can give an indication of the numbers of actual computers we find on the Web, together with the operating system and web server software used.

By arranging for a number of IP addresses to send packets to us near simultaneously, low level TCP/IP characteristics can be used to work out, within an error margin, if those packets originate from the same computer, by checking for similarities in a number of TCP/IP protocol header fields. To build up sufficient certainty that IP addresses on the same computer have been identified many visits to the sites in the Web Server Survey are necessary, which takes place over a period of over a month.

Round robin DNS, reverse web proxies, some load balancing/failover products like Cisco LocalDirector and BIG-IP and some connection level firewalls hide a number of web servers behind a hostname. A limitation of the technique is that only a single "front" web server will be counted. Additionally with some of these products the operating system detected is that of the "front" device rather than the web server behind.

Error Margins

• Despite making multiple visits, there is still a low probability that two computers will be considered the same by chance similarities in low-level TCP/IP protocol header fields; this leads to under-counting.
• Some IP addresses do not respond on enough visits for the technique to be applied. This is mainly due to computers or networks being down or badly overloaded on several of the visits, in which case there are uncounted computers; by extrapolating using the IP address/computer ratio for sites running similar software we can roughly correct for this error.
• If a system changes or upgrades operating system during the course of the survey, which takes over a month to run, and because of some other detailed issues, a computer may be counted more than once; this leads to over-counting, but generally this should be a fairly small effect.

It is difficult to determine a reasonable error bracket for the computer count numbers, especially as the two major errors are in opposite directions, so cancel to some extent. One useful piece of evidence that suggest there are not really large levels of error, is that the average ratio of sites to computers on hosting company networks, is over 10, whereas the ratio of self hosted sites to computers is about 2.

Considering the technique in the abstract we think that error margins world-wide are in the order of ± 10% on IP addresses allocated to hosting companies, where the greatest number of successful comparisons needs to be made by the technique, and in the order of ± 5% on self hosted networks. Note this is in addition to the limitation that we only identify at most one computer per load-balanced website; we cannot quantify the numerical effect of this limitation, but would expect only a minority of web server computers world-wide to use load-balancers at this time, so not causing large-scale distortion of the results.

Netcraft has been performing this survey since February 1999, generally four times a year. The trends since then have been very smooth suggesting there is only a small amount of "random error" in this survey. There could be significant "systematic error" affecting particular groups of web servers more than others, but there are no strong reasons to suppose this would affect particular operating system groups or types of web server significantly more than others world-wide. Studying the quarterly trend results in detail does give us confidence that the error margins in the results are well within the stated ± 10%.

Results

Microsoft Windows has a significantly higher share of the web when one counts by computer, rather than by host, as in the conventional Web Server Survey. The survey shows 49% of the computers running the web are Windows based; a little more than all of the Unix-like operating systems combined. As some of the 3.6% of computers not identified by Netcraft operating system detector will in reality be Windows systems, it would be fair to say about 50% of public Web Servers world-wide are run on Microsoft operating systems Although Apache running on various Unix systems runs more sites than Windows, Apache is heavily deployed at hosting companies and ISPs who strive to run as many sites as possible on a single computer to save costs. Windows is most popular with end-user and self hosted sites, where the host to computer ratio is much smaller.

Linux is the second most commonly used operating system. Linux has been consistently gaining share since this survey started, but interestingly not significantly to Windows detriment. Operating systems which have lost share have been Solaris and other proprietary operating systems, and to a small degree BSD.

One could characterise this process as Solaris being continually chased further and further up market by Intel based operating systems, with Sun in turn progressively eliminating the other proprietary Unix operating systems. Intel enjoys both the benefits of the boom in freely available Unix software and the ascent of Windows, with competing processors correspondingly marginalised in the web server market.

Sun would reasonably point out that this analysis simply counts the number of computers rather than their cost, and that a $1K Intel machine would count the same as a $1M E10K system, and that while Windows matains its share in Fortune 500 companies, the relative position between Linux and Solaris is approximately reversed in these companies.

The analysis also gives some quantification of the rate at which sites migrate to Windows 2000 from NT. In March 2001, a little over a year after the introduction of the operating system 25% of the computers running Microsoft operating systems are running Windows 2000.

Regional Variations

The results summarised above are from a world-wide perspective and significant variations can occur in regional analyses. Out of 32 countries with at least 0.1% of sites on the web each, Windows computers outnumber Unix-like computers in 22 of them. The countries with the largest proportions of Windows web servers are China, South Africa and Singapore. Countries in which Unix-like operating systems maintain the strongest lead are Poland, Hungary, Japan, Russia and Germany, with Linux strong in Poland and Hungary, and BSD in Russia and Japan. Linux leads Windows in Poland, Hungary, the Czech Republic and Germany, while in Finland, home country of Linus Torvalds, Windows has a tiny lead over Linux!

Fair use, Copyright

Excerpts from this survey may be reproduced if Netcraft and the url http://www.netcraft.com/survey/ are attributed.

Related Stories:
The May Netcraft survey is out: it's an Apache world!(Jun 01, 2001)
The April Netcraft survey is out: Apache on top(Apr 18, 2001)
Apache 2.0.16 released as beta(Apr 09, 2001)
Apache 2.0.15 released as an alpha(Mar 28, 2001)

All times are recorded in UTC. Linux is a trademark of Linus Torvalds. Powered by Linux 2.4, Apache 1.3, and PHP 4 Copyright 2002 INT Media Group, Incorporated All Rights Reserved. Legal Notices,  Licensing, Reprints, & Permissions,  Privacy Policy.