2017 was a very interesting and successful year for "Apache Solr for TYPO3" and we were able to deliver a few cool features.

Just to mention some highlights: 

• Public availability of FLUID Templating in EXT:solr
• Support of TYPO3 8 LTS
• Own TYPO3 backend modules

Thanks to all contributors and sponsors in 2017 helped us to improve EXT:solr!

Some months have passed since the last bigger feature release and I would like to use this change to summarize what we are working on in EXT:solr 8.0.0. Depending on the financing we plan to do the next release in the middle of February. Read on about the upcoming releases for Apache Solr for TYPO3.

What will be new at EXT:solr 8.0.0

If you are interested to see the current state in action feel free to use the master branch in your development system. We are happy to get your feedback.

New suggest with top results

We've replaced the old jQuery UI based autosuggest with a new suggest (https://github.com/devbridge/jQuery-Autocomplete). The advanced suggest can not only show the suggestions, it can also show a configurable amount of top search results. 

When the User clicks on the result, he can directly jump to the result page without opening the search results page.

Related pull request: https://github.com/TYPO3-Solr/ext-solr/pull/1638

Thanks:

• Frans Saris and beech.it for sharing the codebase of the initial patch!

JSON faceting for option facets

Apache Solr offers a json API for faceting since several versions. Starting with the options facet we've added the support to use this JSON faceting API in EXT:solr.

The support of the JSON API, in general, allows us to build new features on top of that API, that was impossible before. With the first implementation we've added the following features:

By now an option was simply the value and the count, that reflects the number of documents that belong to that option. At EXT:solr 8.0.0 we've added a typoscript option that is called "metrics", that allows us to collect and show several metrics from documents that belong to a facet option. Examples of metrics are e.g "sum of downloads", "average price",... These metrics will be available in Option model in the FLUID template and can also be used to sort the facet options.

The following example shows an configured options facet with a configured metric:

plugin.tx_solr.search.faceting.facets.type.metrics {
    newest = max(created)
    oldest = min(created)
}

In the FLUID template you could use the following code in the facet partial to render those metrics:

<span>
   newest: {option.metrics.newest -> f:format.date(format: 'Y-m-d H:i:s')}
</span>
<span>
   oldest: {option.metrics.oldest -> f:format.date(format: 'Y-m-d H:i:s')}
</span>

Related pull request: https://github.com/TYPO3-Solr/ext-solr/pull/1764

Thanks:

• Thanks to Jens Jacobsen and UEBERBIT for sponsoring Jens work on that feature at our code sprint.

Since we'replaced the whole internal communication from EXT:solr to Apache Solr when options facets are used we are very happy to get your feedback and bug reports when you use the options facets with EXT:solr

Group facet options by prefix

When you have option facets with a lot of options, it would be nice to group those options by a prefix. An example is that you group all options by the starting letter to organize them in tabs:

With EXT:solr 8 we ship the following components that allow grouping your facet options to arrange them as you need them in your template:

• LabelFilterViewHelper: Can be used to filter options based on a prefix of the label.
• LabelPrefixesViewHelper: Can be used to access all available prefixes of the facet options.
• TypoScript example template "(Example) Options grouped by prefix" that configures a grouped facet on the author field

Related pull request: https://github.com/TYPO3-Solr/ext-solr/pull/1717

Thanks:

• This feature was sponsored by https://www.linnearad.no/

Filterable options facet

In the previous section, the facets get grouped by prefix to organize a large number of options. Another way that you also often see on the web is to allow to filter the options with an additional input box above the facet.

The implementation of that feature is possible just with a partial and a few JavaScript components. To simplify the integration of that feature in a project we ship

• Example FLUID partial that uses the filter for options
• Example JavaScript that implements the filter functionality
• Example TypoScript "Search - (Example) Options filterable by option value" that uses the partials and javascript for a facet

Related pull request:

• https://github.com/TYPO3-Solr/ext-solr/pull/1741

Default templating with bootstrap CSS

The old templating was created with custom CSS that was shipped with the extension. Since we want to decrease the effort that is required to create a mobile search and many integrators use bootstrap.css we decided to ship bootstrap templates by default. If you want to use another framework or your own custom CSS you are still able to do that with custom templates. 

Nevertheless, the mobile search in a TYPO3 introduction installation with bootstrap is much better than before and your effort to adopt it should be reduced.

Related pull request: https://github.com/TYPO3-Solr/ext-solr/pull/1738

Performance improvements

In EXT:solr 7.x and below a ping request was done before each search. In EXT:solr 8.0.0 we just catch a failed search and handle the unavailability. This saves up to 30% time because we just need one HTTP request to Apache Solr instead of 2.

Related pull request: https://github.com/TYPO3-Solr/ext-solr/pull/1660

Improved index inspector

In the previous versions, we've introduced own backend modules that can also be used by regular TYPO3 users to perform several tasks. With EXT:solr 8.0.0 the index inspector will be moved from the common info module to our info module:

Besides the move, we also added the functionality to ReQueue a single document from the index inspector when you have permissions on the index queue module.

Related pull request: https://github.com/TYPO3-Solr/ext-solr/pull/1763

SOLR_CLASSIFICATION cObject

When you index a lot of documents you might want to create facets based on patterns that occur in the content.

The cObject SOLR_CLASSIFICATION allows you to do a lightweight classification based on regex patterns that you configure in the index configuration.

The following example shows how SOLR_CLASSIFICATION can be used to map patterns on classes that are indexed into a Solr field that could be used for faceting:

plugin.tx_solr.index.queue.pages.businessarea_stringM = SOLR_CLASSIFICATION
plugin.tx_solr.index.queue.pages.businessarea_stringM {
   field = __solr_content
   classes {
      automotive {
         patterns = car,jeep,SUV
         class = automotive
      }
      pharma {
         patterns = pharma,doc,medicine
         class = pharma
      }
   }
}

With the configuration above Solr documents get the value "automotive" assigned in the Solr field "businessarea_stringM" when the content contains the term "car", "jeep" or "SUV"

Related pull request: https://github.com/TYPO3-Solr/ext-solr/pull/1723

Thanks: Thanks to http://www.bibus.ch who sponsored the implementation of this feature.

Phrase support (phrase, bigram, trigram)

With plugin.tx_solr.search.query.(phrase/bigramPhrase/trigramPhrase).fields you can control what is passed to Solr with the ps,ps2 and ps3 value.

With these phrase fields, you can boost documents where phrases occur in close proximity. This can be very handy when you want to tune your search in terms of relevancy.

Related links:

• lucene.apache.org/solr/guide/6 6/the-dismax-query-parser.html#TheDisMaxQueryParser-ThepfPhraseFields_Parameter

Related pull request: https://github.com/TYPO3-Solr/ext-solr/pull/1735

Tie parameter support

With plugin.tx_solr.search.query.tieParameter you can now configure the tie value that is passed to Apache Solr.

This value allows you to configure the impact of low scoring fields to the overall score. 0.0 means, that only high score fields will matter, 0.99 means that all fields have the same impact

Related links: 

• https://solr.pl/en/2012/02/06/what-can-we-use-dismax-tie-parameter-for/
• lucene.apache.org/solr/guide/6 6/the-dismax-query-parser.html#TheDisMaxQueryParser-ThetieTieBreaker_Parameter

Related pull request: https://github.com/TYPO3-Solr/ext-solr/pull/1690

Thanks: Thanks to Marcus Schwemer and in2code that sponsored and shared that feature.

Doctrine ready

TYPO3 8 introduced Doctrine DBAL for database queries and the old API will be removed in TYPO3 9. Since we've used a lot of repositories with custom SQL queries, we had to rewrite a lot of queries.

In EXT:solr we've used the chance to restructure the SQL related code and move them to repositories whenever this was possible. 

With EXT:solr 8 every usage of the old database API is removed and we are prepared in that way to be ready for TYPO3 9.

Fluent API for Queries with the QueryBuilder 

Many parts of the code of EXT:solr deal with queries for Apache Solr that's no surprise :). The corresponding parts in the code especially the Query class had grown over time and reached a huge complexity.

This has several drawbacks:

• It is hard to integrate new features (e.g the tiebreaker or bigram features)
• TYPO3 specific logic and common Apache Solr logic is mixed and makes it hard to switch to frameworks like e.g. Solarium
• The Query class does multiple things: Build the query, initialize the query from the configuration,... This could be split into multiple components.

To get better in that regards our goal is to split the Query into: 

• Query: Aggregate that is responsible to build the Solr query string based on the options
• QueryBuilder: Builder class that is responsible to build an initialized Query object e.g. based on TypoScript configuration and user input.

With the current state the QueryBuilder does the following to build a Query from the user input: 

$query = $queryBuilder->newSearchQuery($rawQuery)
 ->useResultsPerPage($resultsPerPage)
 ->useReturnFieldsFromTypoScript()
 ->useQueryFieldsFromTypoScript()
 ->useInitialQueryFromTypoScript()
 ->useFiltersFromTypoScript()
 ->useFacetingFromTypoScript()
 ->useVariantsFromTypoScript()
 ->useGroupingFromTypoScript()
 ->useHighlightingFromTypoScript()
 ->usePhraseFieldsFromTypoScript()
 ->useBigramPhraseFieldsFromTypoScript()
 ->useTrigramPhraseFieldsFromTypoScript()
 ->getQuery();

Finally, this allows us to:

• Integrate new features faster
• Allow devs to compose own queries that use or ignore several aspects of EXT:solr
• Simplify the switch or integration of a generic Solr API that is independent of TYPO3 (e.g. Solarium)

On the way to TYPO3 9 

With EXT:solr 8.0.0 we will not officially support TYPO3 9 since it is not an LTS release! Nevertheless, we want to stay close to the TYPO3 core and allow the usage in 9 already.

By now we mainly fix Doctrine and Composer related issues and support the dropped "pageslanguageoverlay" table.

So to sum up... EXT:solr 8.0.0 will mainly support TYPO3 8 LTS and we will support TYPO3 9.x a good as we can without losing the backward compatibility to TYPO3 8 LTS.

New features in add-ons (EB only) 

All feature mentioned above is part of EXT:solr and available for everybody. Besides those features, we have some very nice features that are only available for you, when you support us with an EB subscription.

Grouping with EXT:solrfluidgrouping

For EXT:solr with the marker based templating there is an extension in the TER that allows you to group documents based on a field or a query. 

This allows you to e.g split the results into pages and news and separate them in the template.

EXT:solrgrouping had some parts that could be improved:

• It is not compatible with the Fluid templating and no domain classes to handle groups are available
• Pagination in groups was not possible

Since we've rewritten the functionality from scratch and also support pagination for groups now, we decided to offer this component as addon only.

In the following screenshot you can see groups and as you can see each of them can be paginated to an own page of the results:

Backend module for EXT:solrfal 

Solrfal uses an own indexing queue for files that are located in the database table txsolrindexqueue_file. By now it was not possible to see a detailed status of the items in the queue. With the new backend module of solrfal you can: 

• See the state of the queue
• See and release errored queue items to allow a reindex of that item
• Clear the index queue

How can you support?

As you see we have a huge list of useful features that we will make available with EXT:solr 8.0.0

Develop / Test / Document 

As you have seen we've added a lot of new features and changed a lot in the code. Therefore we appreciate your help! Feel free to test and create issues and pull requests on GitHub.

Become an EB partner

The development on EXT:solr would not be possible without our sponsors. If you want to support us in 2018 again, please sign up to our EB. This year you can choose between a basic and a premium option that contains several advantages for you.

2018 basic

• Development Funding in 2018
• Access to exclusive Add-Ons
• one hour of support (worth 110 EUR)
• 5 % discount on hourly rates
• priority to support inquiries
• Your logo and link on typo3-solr.com

Price: (ask for a quote!)

2018 premium

• Development Funding in 2018
• Access to exclusive Add-Ons
• 10 % discount on hourly rates
• more priority to support inquiries
• 1 Solr Training (worth 300 EUR)
• 2 h of expert level support included (worth 240 EUR)
• Your logo and link on typo3-solr.com
• Special Badge link on TYPO3-Solr.com
• Extra Section with your name in our Release Notes, Presentations and Webinars

Price: (ask for a quote!)

End of year Special 

Order your 2018 EB and do something good with us.

Depending on the EB you order, the Solr Team will donate 50 / 100 EUR to a charitable organization. We will take all orders into account that reach us until end of January.

We have chosen Techettes e.V, Aktion Deutschland Hilft, and Tierheim Wetterau e.V.

Find out more about them here: https://www.typo3-solr.com/en/blog/details/news/get-your-eb-2018/ or call: +49 (0)69 - 2475218 0