Apache Solr for TYPO3 review and outlook for 2019

The year 2018 is ending soon and most of us will take a break and celebrate Christmas and the new year 2019. We hope you will have and have a nice and relaxing time!

For us, this is also a chance to look back and to think about the next year and the future of  Apache Solr for TYPO3.

2019 will also be special in some ways since we will celebrate the 10th birthday of our Apache Solr integration for TYPO3.

Review

The year 2018 was an exciting year for TYPO3, Apache Solr for TYPO3 and for me as well :-) In the following paragraphs I want to summarize some highlights of our releases in the last year.

Q1: EXT:solr 8.0.0 - UX Improvements

The focus of the release was the improvement of the user experience in the TYPO3 backend and the search frontend. With 8.0.x we provided

• A new autosuggest, that supports top results and grouping
• With the support of JSON faceting, we use new Apache Solr features and build new features like facet metrics on top of that
• Grouping of facet options allows you to render facets with many options in a flexible way
• Filterable options facets allow filtering option facets based on a search string
• With the migration to bootstrap of our default template you are now able to kickstart a mobile variant of your search in a few minutes
• The index inspector in the backend is now part of our custom backend modules. Besides that, new features like re-queuing of documents have been added.

Besides the usability features a lot of other features (e.g. classification, Apache Solr Phrase support, ...) have been shipped with that release. 

Q2: EXT:solr 8.1.0 - New addons and maintenance

With the release of EXT:solr 8.1.0 we shipped the API preparations for new addon's that are accessible for our partners.

The addon solrdebugtools allows you the visually analyze, why a search result has a certain score and is on its current position of the search result list.

With solrconsole you have the possibility to automate tasks for EXT:solr on the command line. This can help you during deployments of maintenance task of your search index.

Q3: Solarium API and new Apache Solr Server 

Our goal was to release EXT:solr 9.0.0 at the and of the 3 quarter, before the release of TYPO3 9. By the end of Q3 the following tasks have been completed:

• We've migrated EXT:solr and all add-ons (EXT:solrfal, EXT:solrfluidgrouping, EXT:solrconsole, EXT:tika) to use the PHP API solarium. This was also the most wanted feature in our survey from 2017 (http://www.tricider.com/brainstorming/3K311n0M0pp). Since we also want to give something back to the solarium project we've worked on the core admin API (https://github.com/solariumphp/solarium/pull/625) that now allows everybody how uses solarium to maintain the solr cores with the PHP API.
• All addons are compatible to Apache Solr 7.6
• It is now possible to use different connections for reading and writing to solr. This enables you to implement scenarios like "live and shadow cores" or "master and slave replications"

Unfortunately, we needed to reduce the amount of work that we can put into the development of EXT:solr at the end of Q3.

This has several reasons:

• The amount of sponsor decreased from 2017 to 2018.
• On the other hand, we had several project requests and needed to decide what to do

This has several consequences:

• We could not release EXT:solr 9 as planned before the release of TYPO3 9 and it is not released yet
• The support of TYPO3 9 LTS is only finished partly and not all features are working

As before, we always tried to be close to the development of the TYPO3 core and do the things as early as possible. Therefore the following parts are done:

• The table "pages_language_overlay" has been dropped in TYPO3 9 and is now handled as other tables in the TYPO3 core with overlays in the same table. This required several changes in EXT:solr and they are done.
• Apply required changes for FLUID

but as mentions before some things are not done:

• SiteHandling is not fully supported
• The might be issues with the routing since this was added to the core after we needed to reduce our work.
• There are several violations shown in the extension scanner. They need to be fixed too.

Just to get me right: This is not the fault of the TYPO3 core or somebody else. I highly appreciate the work and changes there, because we all need them! We should keep the pace and I like to thank everybody who has worked on the TYPO3 LTS release.

So what does this mean for the future and when will I be able to use EXT:solr with TYPO3 9 LTS

Outlook 

As mentioned before EXT:solr 9 will not fully be ready for TYPO3 9 LTS. Nevertheless, we want to release the current state for everyone who is using TYPO3 8 LTS or wants to test EXT:solr 9 with TYPO3 9 LTS. We also know that some of you already use the current master state with TYPO3 9 LTS and this works for their use case.

By the mid of february we want to release EXT:solr 9 with the above mentioned limitations. Beside that we will release the following addons close to that:

• EXT:solr 9.0.0
• EXT:tika 4.0.0
• EXT:solrfal 6.0.0
• EXT:solrfluidgrouping 2.0.0
• EXT:solrmlt 3.0.0
• EXT:solrconsole 2.0.0
• EXT:solrdebugtools 1.1.0

These releases will also provide some other new features:

• EXT:solrconsole will support solfal

After EXT:solr 9 

After the release of EXT:solr 9.0.0 we want to focus on the integration into TYPO3 9. To reduce the effort on that, we will drop the Support of TYPO3 8 LTS in the release after EXT:solr 9.0.0. Our goal is then:

• Integrate into the SiteHandling and make EXT:solr configurable along with your sites in the site module
• Fix violations that are shown in the extension scanner, wherever it is possible
• Check were we can make use of route enhancers and the new PSR-middleware

So this will be our goal for the second quarter of 2019.

Roadmap for 2019 

By now we do not have a strict roadmap, but several ideas:

• Support integration of learning to rank in order to improve the search results
• Improve the integration with the solarium API to reduce redundancy and share a larger codebase with other PHP projects
• Support indexing of other data source

In 2019 we also want to provide the add-on's with private packagist that you are able to install them with composer. We will publish a pricing model for that in 2019.

Besides these topics we also want to know, what are your ideas and requirements for EXT:solr and which features do you want to see? 

Please take the time and throw in your ideas and vote on the ideas. This helps us to propose a roadmap and set the priorities that fit the most user needs:

http://www.tricider.com/brainstorming/2vvCvqAblcF

Thanks

The things that we achieve in the last year are only possible with the support of several people and we want to use this chance to say thank you:

• Thanks to all partners and premium partners that have supported us in 2018
• Thanks to all contributors in the slack channel, github or by mail
• Thanks to the solarium team for the support on the integration into EXT:solr
• Thanks to the private packagist team for our participants in the beta program and support
• Thanks to the TYPO3 core team for the amazing work on TYPO3 9

How to support 

All of the work is only possible with your support. You can support our project by:

• Becoming an EB partner, that supports the project with money and allows you to access additional add-ons
• Pay us to implement features into EXT:solr or fix bugs that you need in your projects
• Help others on GitHub (https://github.com/TYPO3-Solr/ext-solr) or in our slack channel(https://typo3.slack.com/messages/ext-solr/) to get started with Apache Solr for TYPO3

Support us in 2019 by becoming an EB partner:

http://www.typo3-solr.com/en/contact/

or call: 

+49 (0)69 - 2475218 0