Blog Was wir machen. Wie wir denken. Wofür wir brennen.
2017 was a very interesting and successful year for "Apache Solr for TYPO3" and we were able to deliver a few cool features.
Just to mention some highlights:
Thanks to all contributors and sponsors in 2017 helped us to improve EXT:solr!
Some months have passed since the last bigger feature release and I would like to use this change to summarize what we are working on in EXT:solr 8.0.0. Depending on the financing we plan to do the next release in the middle of February. Read on about the upcoming releases for Apache Solr for TYPO3.
If you are interested to see the current state in action feel free to use the master branch in your development system. We are happy to get your feedback.
We've replaced the old jQuery UI based autosuggest with a new suggest (https://github.com/devbridge/jQuery-Autocomplete). The advanced suggest can not only show the suggestions, it can also show a configurable amount of top search results.
When the User clicks on the result, he can directly jump to the result page without opening the search results page.
Related pull request: https://github.com/TYPO3-Solr/ext-solr/pull/1638
Thanks:
Apache Solr offers a json API for faceting since several versions. Starting with the options facet we've added the support to use this JSON faceting API in EXT:solr.
The support of the JSON API, in general, allows us to build new features on top of that API, that was impossible before. With the first implementation we've added the following features:
By now an option was simply the value and the count, that reflects the number of documents that belong to that option. At EXT:solr 8.0.0 we've added a typoscript option that is called "metrics", that allows us to collect and show several metrics from documents that belong to a facet option. Examples of metrics are e.g "sum of downloads", "average price",... These metrics will be available in Option model in the FLUID template and can also be used to sort the facet options.
The following example shows an configured options facet with a configured metric:
plugin.tx_solr.search.faceting.facets.type.metrics {
newest = max(created)
oldest = min(created)
}
In the FLUID template you could use the following code in the facet partial to render those metrics:
<span>
newest: {option.metrics.newest -> f:format.date(format: 'Y-m-d H:i:s')}
</span>
<span>
oldest: {option.metrics.oldest -> f:format.date(format: 'Y-m-d H:i:s')}
</span>
Related pull request: https://github.com/TYPO3-Solr/ext-solr/pull/1764
Thanks:
Since we'replaced the whole internal communication from EXT:solr to Apache Solr when options facets are used we are very happy to get your feedback and bug reports when you use the options facets with EXT:solr
When you have option facets with a lot of options, it would be nice to group those options by a prefix. An example is that you group all options by the starting letter to organize them in tabs:
With EXT:solr 8 we ship the following components that allow grouping your facet options to arrange them as you need them in your template:
Related pull request: https://github.com/TYPO3-Solr/ext-solr/pull/1717
Thanks:
In the previous section, the facets get grouped by prefix to organize a large number of options. Another way that you also often see on the web is to allow to filter the options with an additional input box above the facet.
The implementation of that feature is possible just with a partial and a few JavaScript components. To simplify the integration of that feature in a project we ship
Related pull request:
The old templating was created with custom CSS that was shipped with the extension. Since we want to decrease the effort that is required to create a mobile search and many integrators use bootstrap.css we decided to ship bootstrap templates by default. If you want to use another framework or your own custom CSS you are still able to do that with custom templates.
Nevertheless, the mobile search in a TYPO3 introduction installation with bootstrap is much better than before and your effort to adopt it should be reduced.
Related pull request: https://github.com/TYPO3-Solr/ext-solr/pull/1738
In EXT:solr 7.x and below a ping request was done before each search. In EXT:solr 8.0.0 we just catch a failed search and handle the unavailability. This saves up to 30% time because we just need one HTTP request to Apache Solr instead of 2.
Related pull request: https://github.com/TYPO3-Solr/ext-solr/pull/1660
In the previous versions, we've introduced own backend modules that can also be used by regular TYPO3 users to perform several tasks. With EXT:solr 8.0.0 the index inspector will be moved from the common info module to our info module:
Besides the move, we also added the functionality to ReQueue a single document from the index inspector when you have permissions on the index queue module.
Related pull request: https://github.com/TYPO3-Solr/ext-solr/pull/1763
When you index a lot of documents you might want to create facets based on patterns that occur in the content.
The cObject SOLR_CLASSIFICATION allows you to do a lightweight classification based on regex patterns that you configure in the index configuration.
The following example shows how SOLR_CLASSIFICATION can be used to map patterns on classes that are indexed into a Solr field that could be used for faceting:
plugin.tx_solr.index.queue.pages.businessarea_stringM = SOLR_CLASSIFICATION
plugin.tx_solr.index.queue.pages.businessarea_stringM {
field = __solr_content
classes {
automotive {
patterns = car,jeep,SUV
class = automotive
}
pharma {
patterns = pharma,doc,medicine
class = pharma
}
}
}
With the configuration above Solr documents get the value "automotive" assigned in the Solr field "businessarea_stringM" when the content contains the term "car", "jeep" or "SUV"
Related pull request: https://github.com/TYPO3-Solr/ext-solr/pull/1723
Thanks: Thanks to http://www.bibus.ch who sponsored the implementation of this feature.
With plugin.tx_solr.search.query.(phrase/bigramPhrase/trigramPhrase).fields you can control what is passed to Solr with the ps,ps2 and ps3 value.
With these phrase fields, you can boost documents where phrases occur in close proximity. This can be very handy when you want to tune your search in terms of relevancy.
Related links:
Related pull request: https://github.com/TYPO3-Solr/ext-solr/pull/1735
With plugin.tx_solr.search.query.tieParameter you can now configure the tie value that is passed to Apache Solr.
This value allows you to configure the impact of low scoring fields to the overall score. 0.0 means, that only high score fields will matter, 0.99 means that all fields have the same impact
Related links:
Related pull request: https://github.com/TYPO3-Solr/ext-solr/pull/1690
Thanks: Thanks to Marcus Schwemer and in2code that sponsored and shared that feature.
TYPO3 8 introduced Doctrine DBAL for database queries and the old API will be removed in TYPO3 9. Since we've used a lot of repositories with custom SQL queries, we had to rewrite a lot of queries.
In EXT:solr we've used the chance to restructure the SQL related code and move them to repositories whenever this was possible.
With EXT:solr 8 every usage of the old database API is removed and we are prepared in that way to be ready for TYPO3 9.
Many parts of the code of EXT:solr deal with queries for Apache Solr that's no surprise :). The corresponding parts in the code especially the Query class had grown over time and reached a huge complexity.
This has several drawbacks:
To get better in that regards our goal is to split the Query into:
With the current state the QueryBuilder does the following to build a Query from the user input:
$query = $queryBuilder->newSearchQuery($rawQuery)
->useResultsPerPage($resultsPerPage)
->useReturnFieldsFromTypoScript()
->useQueryFieldsFromTypoScript()
->useInitialQueryFromTypoScript()
->useFiltersFromTypoScript()
->useFacetingFromTypoScript()
->useVariantsFromTypoScript()
->useGroupingFromTypoScript()
->useHighlightingFromTypoScript()
->usePhraseFieldsFromTypoScript()
->useBigramPhraseFieldsFromTypoScript()
->useTrigramPhraseFieldsFromTypoScript()
->getQuery();
Finally, this allows us to:
With EXT:solr 8.0.0 we will not officially support TYPO3 9 since it is not an LTS release! Nevertheless, we want to stay close to the TYPO3 core and allow the usage in 9 already.
By now we mainly fix Doctrine and Composer related issues and support the dropped "pageslanguageoverlay" table.
So to sum up... EXT:solr 8.0.0 will mainly support TYPO3 8 LTS and we will support TYPO3 9.x a good as we can without losing the backward compatibility to TYPO3 8 LTS.
All feature mentioned above is part of EXT:solr and available for everybody. Besides those features, we have some very nice features that are only available for you, when you support us with an EB subscription.
For EXT:solr with the marker based templating there is an extension in the TER that allows you to group documents based on a field or a query.
This allows you to e.g split the results into pages and news and separate them in the template.
EXT:solrgrouping had some parts that could be improved:
Since we've rewritten the functionality from scratch and also support pagination for groups now, we decided to offer this component as addon only.
In the following screenshot you can see groups and as you can see each of them can be paginated to an own page of the results:
Solrfal uses an own indexing queue for files that are located in the database table txsolrindexqueue_file. By now it was not possible to see a detailed status of the items in the queue. With the new backend module of solrfal you can:
As you see we have a huge list of useful features that we will make available with EXT:solr 8.0.0
As you have seen we've added a lot of new features and changed a lot in the code. Therefore we appreciate your help! Feel free to test and create issues and pull requests on GitHub.
The development on EXT:solr would not be possible without our sponsors. If you want to support us in 2018 again, please sign up to our EB. This year you can choose between a basic and a premium option that contains several advantages for you.
Price: (ask for a quote!)
Price: (ask for a quote!)
Order your 2018 EB and do something good with us.
Depending on the EB you order, the Solr Team will donate 50 / 100 EUR to a charitable organization. We will take all orders into account that reach us until end of January.
We have chosen Techettes e.V, Aktion Deutschland Hilft, and Tierheim Wetterau e.V.
Find out more about them here: https://www.typo3-solr.com/en/blog/details/news/get-your-eb-2018/ or call: +49 (0)69 - 2475218 0
<tmpopup style="top: 5513px; left: 320.15px;"><tmpopupcolor id="tmpopupcolor--2" style="background: rgb(246, 118, 255) none repeat scroll 0% 0%;"></tmpopupcolor><tmpopupcolor id="tmpopupcolor--3" style="background: rgb(108, 246, 255) none repeat scroll 0% 0%;"></tmpopupcolor><tmpopupcolor id="tmpopupcolor--m" style="background: rgb(45, 237, 255) none repeat scroll 0% 0%;"></tmpopupcolor><tmpopupcolor id="tmpopupcolor--v" style="background: rgb(43, 247, 255) none repeat scroll 0% 0%;"></tmpopupcolor></tmpopup><tmpopup style="top: 5513px; left: 320.15px;"><tmpopupcolor id="tmpopupcolor--2" style="background: rgb(246, 118, 255) none repeat scroll 0% 0%;"></tmpopupcolor><tmpopupcolor id="tmpopupcolor--3" style="background: rgb(108, 246, 255) none repeat scroll 0% 0%;"></tmpopupcolor><tmpopupcolor id="tmpopupcolor--m" style="background: rgb(45, 237, 255) none repeat scroll 0% 0%;"></tmpopupcolor><tmpopupcolor id="tmpopupcolor--v" style="background: rgb(43, 247, 255) none repeat scroll 0% 0%;"></tmpopupcolor></tmpopup>
