Your Daily Source for Apache News and Information |
Breaking News | Preferences | Contribute | Triggers | Link Us | Search | About |
\n'); } if ( plugin ) { document.write(' '); } else if (!(navigator.appName && navigator.appName.indexOf("Netscape")>=0 && navigator.appVersion.indexOf("2.")>=0)){ document.write(''); } //-->
|
By Stas Bekman Sharing MemoryAs we have learned in the previous article, sharing memory helps us to save memory with mod_perl, which gives us a huge speed up but we pay with price of big memory foot print. I presented a few techniques to save memory by trying to share more of it. In this article, we will see other techniques allowing you to save even more memory. Preloading Registry Scripts at Server StartupWhat happens if you find yourself stuck with Perl CGI scripts and you cannot or don't want to move most of the stuff into modules to benefit from modules preloading, so the code will be shared by the children. Luckily you can preload scripts as well. This time the For example to preload the script /perl/test.pl which is in fact the file /home/httpd/perl/test.pl you would do the following: use Apache::RegistryLoader (); Apache::RegistryLoader->new->handler("/perl/test.pl", "/home/httpd/perl/test.pl"); You should put this code either into But what if you have a bunch of scripts located under the same directory and you don't want to list them one by one. Take the benefit of Perl modules and put them to a good use. The The following code walks the directory tree under which all use File::Find qw(finddepth); use Apache::RegistryLoader (); { my $scripts_root_dir = "/home/httpd/perl/"; my $rl = Apache::RegistryLoader->new; finddepth ( sub { return unless /\.pl$/; my $url = "$File::Find::dir/$_"; $url =~ s|$scripts_root_dir/?|/|; warn "pre-loading $url\n"; # preload $url my $status = $rl->handler($url); unless($status == 200) { warn "pre-load of `$url' failed, status=$status\n"; } }, $scripts_root_dir); } Note that I didn't use the second argument to A simple sub mytrans { my $uri = shift; $uri =~ s|^/perl/|/home/httpd/perl/|; return $uri; } You can easily derive the right translation by looking at the Alias /perl/ /home/httpd/perl/ After defining the URI to filename translation function you should pass it during the creation of the my $rl = Apache::RegistryLoader->new(trans => \&mytrans); I won't show any benchmarks here, since the effect is absolutely the same as with preloading modules. Modules Initializing at Server StartupWe have just learned that it's important to preload the modules and scripts at the server startup. It turns out that it's not enough for some modules and you have to prerun their initialization code to get more memory pages shared. Basically you will find an information about specific modules in their respective manpages. I will present a few examples of widely used modules where the code can be initialized. Initializing DBI.pmThe first example is the You probably know already that under mod_perl you should use the Just as with modules preloading, our goal is to find the startup environment that will lead to the smallest ``difference'' between the shared and normal memory reported, therefore a smaller total memory usage. And again in order to have an easy measurement I will use only one child process, therefore I will use this setting in httpd.conf: MinSpareServers 1 MaxSpareServers 1 StartServers 1 MaxClients 1 MaxRequestsPerChild 100 I'm going to run memory benchmarks on five different versions of the startup.pl file. I always preload these modules: use Gtop(); use Apache::DBI(); # preloads DBI as well
Here is the preload_dbi.pl -------------- use strict; use GTop (); use DBI (); my $dbh = DBI->connect("DBI:mysql:test::localhost", "", "", { PrintError => 1, # warn() on errors RaiseError => 0, # don't die on error AutoCommit => 1, # commit executes # immediately } ) or die "Cannot connect to database: $DBI::errstr"; my $r = shift; $r->send_http_header('text/plain'); my $do_sql = "show tables"; my $sth = $dbh->prepare($do_sql); $sth->execute(); my @data = (); while (my @row = $sth->fetchrow_array){ push @data, @row; } print "Data: @data\n"; $dbh->disconnect(); # NOP under Apache::DBI my $proc_mem = GTop->new->proc_mem($$); my $size = $proc_mem->size; my $share = $proc_mem->share; my $diff = $size - $share; printf "%8s %8s %8s\n", qw(Size Shared Diff); printf "%8d %8d %8d (bytes)\n",$size,$share,$diff; The script opens a connection to the database 'test' and issues a query to learn what tables the databases has. When the data is collected and printed the connection would be closed in the regular case, but The server was restarted before each new test. So here are the results of the five tests that were conducted, sorted by the Diff column:
Now what do we conclude from looking at these numbers. First we see that only after a second reload we get the final memory footprint for a specific request in question (if you pass different arguments the memory usage might and will be different). But both tables show the same pattern of memory usage. We can clearly see that the real winner is the startup.pl file's version where the MySQL driver was installed (1). Since we want to have a connection ready for the first request made to the freshly spawned child process, we generally use the second version (2) which uses somewhat more memory, but has almost the same number of shared memory pages. The third version only preloads the driver which results in smaller shared memory. The last two versions having nothing initialized (4) and having only the To remind you why do we look for the smallest value in the column diff, recall the real memory usage formula: RAM_dedicated_to_mod_perl = diff * number_of_processes + the_processes_with_largest_shared_memory Notice that the smaller the diff is, the bigger the number of processes you can have using the same amount of RAM. Therefore every 100K difference counts, when you multiply it by the number of processes. If we take the number from the version version (1) vs. (4) and assume that we have 256M of memory dedicated to mod_perl processes we will get the following numbers using the formula derived from the above formula: RAM - largest_shared_size N_of Procs = ------------------------- Diff 268435456 - 2609152 (ver 1) N = ------------------- = 309 860160 268435456 - 2469888 (ver 5) N = ------------------- = 262 1011712 So you can tell the difference (17% more child processes in the first version). Initializing CGI.pm
use CGI; CGI->compile(':all'); where you should replace the tag group I'm going to compare the shared memory foot print by using the script which is back compatible with mod_cgi. You will see that you can improve performance of this kind of scripts as well, but if you really want a fast code think about porting it to use So here is the preload_cgi_pm.pl ----------------- use strict; use CGI (); use GTop (); my $q = new CGI; print $q->header('text/plain'); print join "\n", map {"$_ => ".$q->param($_) } $q->param; print "\n"; my $proc_mem = GTop->new->proc_mem($$); my $size = $proc_mem->size; my $share = $proc_mem->share; my $diff = $size - $share; printf "%8s %8s %8s\n", qw(Size Shared Diff); printf "%8d %8d %8d (bytes)\n",$size,$share,$diff; The script initializes the As usual I are going to use a single child process, therefore I will use this setting in httpd.conf: MinSpareServers 1 MaxSpareServers 1 StartServers 1 MaxClients 1 MaxRequestsPerChild 100 I'm going to run memory benchmarks on three different versions of the startup.pl file. I always preload this module: use Gtop();
The server was restarted before each new test. So here are the results of the five tests that were conducted, sorted by the Diff column:
The first version shows the results of the script execution when By looking at the version one of the second table we can conclude that, preloading adds about 20K of shared size. As I have mention at the beginning of this section that's how But even in our very simple case using the same formula, what do we see? (assuming that I have 256MB dedicated for mod_perl) RAM - largest_shared_size N_of Procs = ------------------------- Diff 268435456 - 2134016 (ver 1) N = ------------------- = 223 1191936 268435456 - 2445312 (ver 3) N = ------------------- = 331 802816 If I preload I've heard that the 3.x generation will be less bloated. But it's in a beta state as of this writing. Increasing Shared Memory With mergemem
It looks like it could be run periodically on your server to find and merge duplicate pages. It won't halt your httpds during the merge, this aspect has been taken into consideration already during the design of mergemem: Merging is not performed with one big system call. Instead most operation is in userspace, making a lot of small system calls. Therefore blocking of the system should not happen. And, if it really should turn out to take too much time you can reduce the priority of the process. The worst case that can happen is this: This software comes with a utility called References
Related Stories: |
|
|
About Triggers | Media Kit | Security | Triggers | Login |
All times are recorded in UTC. Linux is a trademark of Linus Torvalds. Powered by Linux 2.4, Apache 1.3, and PHP 4 Legal Notices, Licensing, Reprints, & Permissions, Privacy Policy. |