Blog Was wir machen. Wie wir denken. Wofür wir brennen.
This is a summary of my talk for the TYPO3 developer days in Malmö «How we keep EXT:solr uptodate with the TYPO3 core».
TYPO3 8 LTS was announced to be released on the 4th April 2017. Our plan and challenge were to release our Apache Solr for TYPO3 extensions 2 days later. There are several side effects that are important for us to keep in mind:
Conclusion: In the end, everything needs to perform together to have a running search in TYPO3.
To cover the previously mentioned side effects we did the following:
To implement the technical mechanism above we use GitHub pull requests for every change (no direct push to any branch). Every pull request is checked by Travis-Ci.org and Scrutinizer-ci.com to help us to find problems with a change before it get’s merged.
Important: The goal is not to blame anyone's changes. The goal is to keep EXT:solr running with all combinations!
So if you want to start contributing to EXT:solr you can create a fork and enable travis-ci.org and scrutinizer-ci.com on your fork, and run these things even before creating a pull request.
Travis-ci.org is a continuous integration platform that can run builds configured with a simple YAML file. A build contains checks on the software that make sure that your software works as expected. In the case of EXT:solr we currently run:
The goal should be for sure, to cover as much as you can with unit tests because they run much quicker. We will come back later on that topic when we will have a look on the refactoring. But first, we will have a closer look on travis-ci.org
language php
This defines the language of your project. In our case php.
addons:
apt:
packages:
- parallel
In the add-ons section, you can define additional things that should be available in your testing container. In our case, we install the parallel package that we use to run the PHP linter in parallel.
env:
global:
- TYPO3_DATABASE_NAME="typo3_ci"
- TYPO3_DATABASE_HOST="127.0.0.1"
- TYPO3_DATABASE_USERNAME="root"
- TYPO3_DATABASE_PASSWORD=""
In the env section, you can define environment variable that you need during the test. We define the environment variables here, that we use for the TYPO3 testing framework. We use them in our bootstrap.sh script to set them for the testing framework.
matrix:
fast_finish: true
include:
- php: 5.5
env: TYPO3_VERSION=^7.6
- php: 5.6
env: TYPO3_VERSION=^7.6
- php: 7.0
env: TYPO3_VERSION=^7.6
- php: 7.1
env: TYPO3_VERSION=^7.6
- php: 7.1
env: TYPO3_VERSION=^8.7
- php: 7.0
env: TYPO3_VERSION=^8.7
In the matrix section, we define the combinations of PHP and TYPO3 that we want to use when we execute our builds. Each combination will create an own test container on Travis-ci. As we develop for dev-master, we can add this here as well.
before_install:
- composer self-update
install:
- Build/Test/bootstrap.sh
script:
- Build/Test/cibuild.sh
after_script:
- Build/Test/publish_coverage.sh
There are several build steps that we use to do different things.
cache:
directories:
- $HOME/.composer/cache
Last but not least we store the composer cache in a Travis cache to avoid the download in every build.
Scrutinizer-ci.com is a continuous integration tool that helps us to measure and visualize code quality:
The configuration also happens in a simple YAML file. In our project we use the following parts:
checks:
php:
code_rating: true
duplication: true
We want to get a code rating and check for duplications.
filter:
paths:
- 'Classes/*'
We evaluate the classes below Classes since the other PHP files return only typo related stuff where the structure is fixed.
tools:
php_cpd:
enabled: true
php_code_sniffer:
enabled: true
config:
standard: TYPO3CMS
# we do this on travis
php_cs_fixer:
enabled: false
php_mess_detector:
enabled: true
config:
controversial_rules:
superglobals: false
php_pdepend:
enabled: true
# coverage pushed from travis-ci.org
external_code_coverage:
# two runs unit and integration
runs: 2
timeout: 1800
In the tools section, the underlying PHP QA tools can be configured. In our case this means
build_failure_conditions:
- 'patches.label("Doc Comments").count > 0'
- 'patches.label("Spacing").count > 0'
- 'issues.label("coding-style").count > 0'
- 'issues.severity(>= MAJOR).count > 0'
Build failure conditions allow us to define a boundary for a certain metrics that causes the build to fail. This is very handy when you e.g. want to make sure that the quality of an imported project will not get worse over time.
There are a lot of services available around GitHub that allow you to run automated tests and code checks for free. This helps us a lot for EXT:solr to make sure the extension still works with the latest TYPO3 core versions.
We started with adding these tests and improve the ci process over time. This enables us to integrate bigger features faster and reduces the effort for manual testing.
The first lines of EXT:solr are 8 years old now :) Since the way how PHP applications and TYPO3 extension are build was improved a lot, we need to continuously transform the code to keep it update. This means we need to change the code continuously while keeping it doing, what it does before. Automated tests help us to change the code while making sure, it works as before.
In the ideal developer world, the automated tests look like a pyramid
The execution path is from bottom to top. Then the tests fail as quickly as possible and give you a feedback on that during the development.
To develop this way you need to full fill some requirements:
On the other hand, this also means:
To improve these automated tests we currently first add integration tests, that cover bigger parts of the application.
Afterwards we restructure the code and also add unit tests for the new components:
This has the following advantages:
Currently, we have around 400 unit tests that take 10 seconds to run, the coverage here is still less but should be improved over time. The integration tests take around 15 Minutes but cover bigger parts of the application.
Together around 65% are tested automatically. This is a step in the right direction but still improvable. But nevertheless, we need to keep in mind, that doing all this manual would take days.
With the tools that we described below, we achieved that EXT:solr 6.1.0 could be released as planned on the 6. April, two days after TYPO3 8 LTS. At the same time, we could merge around 160 pull requests from 14 contributors and the trends go into the right direction :)
By the end of the month we plan to release EXT:solr 7.0.0 with the following highlights:
For the second half of the year we have the following things that could potentially be done (as always depending on the budget):
Q3 - Focus: UX improvements:
Q4 - Focus: Latest Apache Solr Version :
For 2018 there is no real roadmap yet but we have several ideas that could potentially be done
As shortly mentioned before dkd is working on a semantic service platform nreach.io that provides semantic services for TYPO3. To show you how simple these services can be integrated into TYPO3 and what powerful features are possible,
we’ve implemented a PoC that implements a picture search based on nreach, solr and solrfal. It allows you to search for objects on pictures and also facet on attributes extracted by nreach.
The following video shows how a fal object can be enriched by nreach with semantic data:
<iframe frameborder="0" height="451" width="640" src="https://player.vimeo.com/video/225846429"></iframe>
And then we use the frontend to search and filter for properties that have been extracted by nreach automatically:
<iframe frameborder="0" height="396" width="640" src="https://player.vimeo.com/video/225846487"></iframe>
Here is a collection of links of tools and resources mentioned in the post above:
As you see there are several things we did to provide a release for TYPO3 8 LTS and we have also quite some ideas for the future. If you want to support us. Go to https://typo3-solr.com or call +49 69 2475218-0 and become a partner today!