Optimizing a shared hosting MageBridge server
Saturday, 17 September 2011
For some time now, we have been hosting a great bunch of Joomla! sites and Magento sites for demo purpose of our MageBridge extension-suite. But those sites became slower and slower. That's why we decided to migrate those sites to a new server, and tune that server with a couple of tricks. Results are fast demo-sites plus this extensive blog telling you how we did this.
The problem with the old server
The main problem was that the old server reached its end-of-life. Dating back to an age where we (Yireo) still provided a hosting service (six years ago), the server had a good amount of memory (4GB RAM) but its harddisks were obsolete, and therefor terribly slow to write on.
Memory vs disk
When setting up any webserver for Magento, you should realize that Magento is a heavy application, and you therefor need to tune system resources. When tuning, the quick conclusion is always that memory-access is faster than disk-access, so if you can swap disk-space for memory-space do so. Tricks includes working with tmpfs-filesystems (filesystems based in memory) and MySQL tuning.
However, our webserver was hosting more than 200 Joomla! sites and 40 Magento sites, and no matter what, that's far too many sites to tune with just an old bunch of hardware.
New server at XLS Hosting
So the first step was to get a new server. Because hardware is not our thing anymore, we decided to get a VPS-server with the provider XLS Hosting. Simply by migrating to a new server, disk access times increased and one of the major problems with performance tuning on the old server was solved.
The VPS now used for hosting all MageBridge demos is shipped with some disk space (unimportant when dealing with performance tuning) and 4GB RAM. You should note that the amount of RAM we got on this new server did not differ with the RAM of the old machine. More important than how much RAM we had, was what we did with the RAM in the first place: Configure the webserver to actually use it.
Shared server for demo-sites
So there we are: A new server with plenty of RAM and fast disk access. Now this VPS is still hosting about 120 Joomla! sites and 20 Magento sites: It's a lot, but MageBridge covers such a fast area of ecommerce-development so demos are appropriate. Ofcourse, a slow demo is bad, so we needed to optimize this server for speed.
Now, the first thing to note is that setting up a shared server for speed is very hard to do. There are hosting providers like Byte that setup shared clusters, but for standalone single servers resources are limited. However, when a shared server is only dedicated to runnning Joomla! and Magento things are a bit easier: The basic idea is also tha t Magento needs much more tuning than Joomla!, and whatever tricks you apply to speed up Magento, also speeds up Joomla!.
Tuning MySQL
Joomla! and Magento both run on MySQL databases, and while Joomla! makes per webrequest between 3 and 30 database queries on average, Magento makes about 300 database queries per webrequest. If you setup a standard Linux server, the MySQL server is optimized for a physical server with maybe 128MB RAM - in other words, it is far from optimized when having 4GB RAM. The MySQL configuration needs to be modified.
With our 4Gb RAM we decided to spend 2GB of RAM on the MySQL server, so 2GB on the rest (operating system, webserver, etcetera). Within the MySQL configuration (in our case, located in /etc/my.cnf) we added the following values:
[mysqld] ... key_buffer_size=64M query_cache_type=1 query_cache_size=64M query_cache_limit=2M sort_buffer_size=8M table_cache=4092 innodb_buffer_pool_size=1024M innodb_flush_log_at_trx_commit=2 innodb_thread_concurrency=2 thread_cache_size=4
We're not going to cover all of these values in depth, but if you read on the web about Magento tuning, you frequently will encounter these values for key_buffer_size, query_cache_type, query_cache_size, query_cache_limit and sort_buffer_size. A good resource is also the official Magento whitepaper Maximizing Performance and Scalability with Magento Enterprise Edition.
Specific MySQL values
With a single Magento site, the table_cache could be set to about 200 or 300, but we're dealing with a shared environment here with up to (120 Joomla! sites x 30 tables, 20 Magento sites x 300 tables ~= ) 10.000 database tables, so a value of 4092 is quite moderate.
Of high importance is also the value for innodb_buffer_pool_size. MySQL makes use of various storage engines, and the MyISAM storage engine is popular with Joomla! while the InnoDB storage engine is used throughout the Magento database. Now, InnoDB is able to cache things in a memory-buffer and the more memory you reserve for this, the faster InnoDB data will be accessed over time. Because this is so powerful, database tuning with Magento boils down specifically for this value. Some webresources tell you to set this to half of your database memory (which is 2GB / 2 = 1GB in our case), but we are already experimenting with increasing this value to 2GB.
Setting up PHP APC
So that's half of the memory. Now, let's deal with the other half - we still have 2GB left for tuning. While you will need to reserve memory for the operating system itself, the remainder of memory goes primarily to PHP. With PHP, you can speed up things tremeduously by using a PHP-accelerator. There are many PHP-accelerators out there (XCache, memcache, eAccelerator) but we setlle for APC. Note that installing 2 accelerators on the same server probably gives conflicts (for instance, APC and XCache can't work simultaneously).
Installing APC through PECL
How you want to setup APC is up to you: You can compile it from source, you can install it through your favorite package-manager (apt-get or yum), but we just did it the quick and dirty way:
pecl install apc
After this, we needed to add the new APC extension apc.so to the PHP-configuration php.ini:
extension=apc.so
Tuning PHP APC
So, APC is installed. But we're not done yet. As with all webserver-related things, installing the software is only part of the job - you also need to configure it. The following therefor was also added to our php.ini configuration:
[APC] apc.enabled = 1 apc.shm_segments = 1 apc.shm_size = 1024M apc.num_files_hint = 20000 apc.stat = 1
Looking at this configuration, apc.enabled and apc.shm_segments are just default values. The main trick here is the apc.shm_size which is set to 1GB of RAM. When PHP-scripts are being ran, the APC extension will place the compiled bytecode (opcode) in memory, so that the PHP-functionality is executed faster. The more PHP-files can be placed in memory, the less the webserver needs to read those files from disk.
Now our shared environment contains about 11GB of data in total (all Magento and Joomla! sites combined) and probably the total amount of PHP-files exceeds the 4GB. Unfortunately we don't have that much memory, so we had to settle for 1GB. Note that the default value for apc.shm_size is 32MB which ridiculous for any modern webserver.
APC stat
Another vital option is apc.stat. When switched on, this makes APC check the original PHP-file to see whether it has been modified. Let's say you have a PHP-script that contains 10 lines of code. The PHP-interpreter will convert those 10 lines of code to bytecode, and APC will store that bytecode in memory. But when you add another 2 lines of code, APC needs to be told that the bytecode needs to be updated as well.
When setting apc.stat to 1, this will make sure that the webserver will check the original file every time the equivalent bytecode will be fetched from memory. If you have a lot of files, this will actually degrade your performance, because of all the files still needing to be accessed. But the benefit is that you can modify all those PHP-scripts without worries.
We actually have set apc.stat to 0, meaning that the original PHP-scripts will not be checked and the performance is optimal. However, every time we upgrade a MageBridge extension, or when we modify one of the Magento or Joomla! PHP-files, we need to restart the webserver:
apachectl graceful
A bit annoying, but because we don't work continuously on this webserver, best performance overrules convenience.
Tuning other PHP settings
Before we leave the PHP-configuration to it, there are two more PHP-variables that are best modified:
realpath_cache_size = 32k
realpath_cache_ttl = 7200
The realpath cache is a kind of PHP-buffer remembering the location of various PHP-scripts on the filesystem. The more locations can be remembered, the faster the file access will become.
Tuning Magento
It is very important to note that - so far - we have configured PHP APC in general. But there are a few tricks that can be applied to Magento specifically, in regards with APC. Besides caching PHP-scripts in memory, APC can also be used by Magento to store cache-files in memory. This makes it lightning fast to read-in the Magento cache. For this the Magento files app/etc/local.xml needs to be modified:
<cache>
<backend>apc</backend>
<prefix>magebridge_mage1501_</prefix>
</cache>
The prefix needs to be configured for every Magento instance to prevent that one Magento application is using the cache-files of another Magento application.
TwoLevel cache in Magento
However, there's one slight problem here. Magento caches files in segments, but there are extra attributes that need to be stored for every cache-segment: A cache-description, a timestamp but also a specific tag (which places the cache-segment in a specific tagging-group, for instance BLOCK_HTML or CONFIG). Now, APC is not capable of this tagging - meaning that the tag is lost when caching.
Because this leads to unwanted issues, Magento automatically enables a thing called the TwoLevel cache, wherein APC will be used as primary fast cache, but where also another secondary cache is used to store tagging. If this secondary cache is terrible slow, it will slow down your primary cache as well - resulting in a slow Magento cache. The default store-mechanism for this secondary cache (also called the slow_backend) is files. Worse, when the slow_backend is set to files, it seems that Magento automatically also stores the actual cached-data in files (the folder MAGENTO/var/cache).
So once you become aware of this TwoLevel cache, you realize that using APC as primary backend but not defining a better slow_backend is actually worse, than just using files as primary backend. But, when using the database as slow_backend (while we already paid attention to tuning the MySQL server itself), pays off:
<cache>
<backend>apc</backend>
<slow_backend>database</slow_backend>
<slow_backend_store_data>0</slow_backend_store_data>
<auto_refresh_fast_cache>0</auto_refresh_fast_cache>
<prefix>magebridge_mage1501_</prefix>
</cache>
Magento caching and the Compiler
Now the only things remaining are common standard in Magento: Enable Magento caching on all sites, except development sites. Another common trick is to enable the Magento Compiler. Normally Magento searches in total 5 folders when trying to locate PHP-files, but this can brought down to only 2 folders. However, PHP APC already caches all those PHP-files in memory, so enabling the Magento Compiler is not needed when using the other steps in this tutorial.