Faster Magento 2 Integration Tests
Testing is important. While unit testing and MFTF should play a vital role in Magento 2, let's not forget about integration tests. When you first run the Magento 2 Integration Tests suite of the Magento core, you will be waiting a long time for things to complete. And slow test runs will keep you from making use of them. So let's see how we can optimize Integration Tests to run as fast as possible.
I have to admit, I'm a knowledgable developer but not a TDD-expert. I actually write code most of the time without tests, only to realize afterwards that tests would have been useful and sometimes badly needed because of some bug. Maybe it is also due to the fact that I deal with a lot of legacy code.
That being said, I am currently in the flow of testing: I use TDD on numerous microservices I develop, I use TDD for refactoring and module development. And I like it. But once you get to that point of running tests constantly, tests that take ages to run will break your flow. This needs to be optimized.
Obvious points first
Let's deal with the obvious points first - assuming that you want to write your own code with Integration Tests: Don't run all of the Integration Tests of the core. Instead, copy the original
phpunit.xml.dist, possibly remove the existing testsuites and add your own.
What I personally do a lot now is add a suite
YireoApp (for code in
app/code) and a suite
YireoVendor (for code in
vendor/). But because these suites might run slow, because of dozens of Yireo extensions, I either add a
YireoCustom suite for a specific extension or add a specific PHPStorm configuration. When I want to be in the flow, I only run tests of my own extension.
Also, make sure to know the trick of
TESTS_CLEANUP. With the setting being enabled, every test run will reinstall Magento 2 in your test database. With the setting being disabled, the setup is reused. Google for tips on this.
What is slow?
So now that we know we are only going to run our own testing suite, let's see how much time this takes. I'm assuming the following: Magento 2.3; PHP 7.2 with XDebug disabled and OPCache enabled; MySQL 5.7; SSD drive; one single test
HelloWorldTest with a dummy test like
In an unoptimized environment, this is rather slow. As soon as the Integration Test suite kicks in, the setup scripts take more than a minute. Once the setup is completed, I toggle the
TESTS_CLEANUP constant to
disabled. If I rerun the tests, with the existing database (so without rerunning the setup), it still takes more than 6-8 seconds.
This breaks my flow.
Remove unneeded dependencies
One of the reasons why things are slow is that Magento is big. Making it slimmer will definitely help. Following the excellent IntegerNet blog Make Magento 2 Small Again, we can use the composer
replace trick to remove all packages that are not needed.
I do this usually for production sites. But I also do this for development. If I have documented the dependencies of my own module carefully (and I can easily track my dependencies with the Yireo ExtensionChecker), I can assume that my code and my tests will run without unneeded dependencies being installed. No GraphQL, no MSI, no third-party modules.
Still, it is needed to test things with a full Magento 2 installation. This is not something I do on-demand, but in a Continuous Integration workflow using GitLab CI: Then it is fine for the tests to take up more time.
Boost the database
Another important tip is to make the database bloody fast. With a local setup, you can work on tuning MySQL parameters (buffering, query cache). However, I'm favoring the usage of a Docker image with a MySQL 5.7 server. And with a little trick, you can run the entire (!) MySQL installation in
tmpfs. It can't be faster than that!
In the past, I've been writing about this in the following blog: Boost MySQL speed of Magento 2. The Docker engine will drop the database once restarted, but this is fine for running integration tests.
With the composer
replace trick in place and with MySQL in
tmpfs, it is time to review how fast things are now. My personal results are that I can run the installation of Magento 2 itself within 20-25 seconds (
TESTS_CLEANUP=enabled). And if I rerun the tests with
TESTS_CLEANUP=disabled, my sample test completes within 2 seconds.
This is great. A re-installation no longer requires a coffee break. And 2 seconds is kind of enough to keep me in the flow (and keep my tests in the green). But it still can be improved.
ReachDigitals quick integration tests
The Dutch company ReachDigital (formerly, H&O), has developed an even better approach: In the past, I hacked around in the PHP files of the Integration Testing Framework and discovered that various things could be improved there.
The guys of ReachDigital must have thought the same thing. And they released a drop-in alternative, the Quick Integration Testing Framework, that applies various tricks to boost performance. Once installed via composer, you point your tests to make use of their folder
dev/tests/quick-integration and you're done.
With all results in place, my test now runs in 100ms to 200ms, which is blazing fast. When I rerun the entire setup (
TESTS_CLEANUP=enabled), things are still at 20-25 seconds, so no improvement there - but that's not what ReachDigital tried to accomplish.
One thing I found annoying was that I needed to toggle the value
TESTS_CLEANUP again and again by modifying the
phpunit.xml file. I now fixed this by creating a
bin/magento yireo_devhacks:toggle_testscleanup command (shipped with the module Yireo DevHacks which will bundle more of my personal developer tricks soon).
I'm keen to experiment things with MySQL 8 - it is not yet supported but adds additional performance. However, because my database engine is already completely in memory, I'm not sure if this will shave off much performance.
You should be able to run your own Integration Test in 100ms to 200ms, assuming the Magento 2 setup has completed and you are re-running things. You should be able to install the Magento 2 test application in less than half a minute.