highlight module

The highlight module contains classes and functions for displaying short excerpts from hit documents in the search results you present to the user, with query terms highlighted.

The highlighting system has four main elements.

  • Fragmenters chop up the original text into __fragments__, based on the locations of matched terms in the text.
  • Scorers assign a score to each fragment, allowing the system to rank the best fragments by whatever criterion.
  • Order functions control in what order the top-scoring fragments are presented to the user. For example, you can show the fragments in the order they appear in the document (FIRST) or show higher-scoring fragments first (SCORE)
  • Formatters turn the fragment objects into human-readable output, such as an HTML string.

See How to create highlighted search result excerpts for more information.

See how to highlight terms in search results.

Manual highlighting

class whoosh.highlight.Highlighter(fragmenter=None, scorer=None, formatter=None, always_retokenize=False, order=<function FIRST>)
whoosh.highlight.highlight(text, terms, analyzer, fragmenter, formatter, top=3, scorer=None, minscore=1, order=<function FIRST>, mode='query')

Fragmenters

class whoosh.highlight.Fragmenter
fragment_matches(text, matched_tokens)

Yields Fragment objects based on the text and the matched terms.

Parameters:
  • text – the string being highlighted.
  • matched_tokens – a list of analysis.Token objects representing the term matches in the string.
fragment_tokens(text, all_tokens)

Yields Fragment objects based on the tokenized text.

Parameters:
  • text – the string being highlighted.
  • all_tokens – an iterator of analysis.Token objects from the string.
must_retokenize()

Returns True if this fragmenter requires retokenized text.

If this method returns True, the fragmenter’s fragment_tokens method will be called with an iterator of ALL tokens from the text, with the tokens for matched terms having the matched attribute set to True.

If this method returns False, the fragmenter’s fragment_matches method will be called with a LIST of matching tokens.

class whoosh.highlight.WholeFragmenter(charlimit=32768)

Doesn’t fragment the token stream. This object just returns the entire entire stream as one “fragment”. This is useful if you want to highlight the entire text.

Note that even if you use the WholeFragmenter, the highlight code will return no fragment if no terms matched in the given field. To return the whole fragment even in that case, call highlights() with minscore=0:

# Query where no terms match in the "text" field
q = query.Term("tag", "new")

r = mysearcher.search(q)
r.fragmenter = highlight.WholeFragmenter()
r.formatter = highlight.UppercaseFormatter()
# Since no terms in the "text" field matched, we get no fragments back
assert r[0].highlights("text") == ""

# If we lower the minimum score to 0, we get a fragment even though it
# has no matching terms
assert r[0].highlights("text", minscore=0) == "This is the text field."
class whoosh.highlight.SentenceFragmenter(maxchars=200, sentencechars='.!?', charlimit=32768)

Breaks the text up on sentence end punctuation characters (“.”, “!”, or “?”). This object works by looking in the original text for a sentence end as the next character after each token’s ‘endchar’.

When highlighting with this fragmenter, you should use an analyzer that does NOT remove stop words, for example:

sa = StandardAnalyzer(stoplist=None)
Parameters:maxchars – The maximum number of characters allowed in a fragment.
class whoosh.highlight.ContextFragmenter(maxchars=200, surround=20, charlimit=32768)

Looks for matched terms and aggregates them with their surrounding context.

Parameters:
  • maxchars – The maximum number of characters allowed in a fragment.
  • surround – The number of extra characters of context to add both before the first matched term and after the last matched term.
class whoosh.highlight.PinpointFragmenter(maxchars=200, surround=20, autotrim=False, charlimit=32768)

This is a NON-RETOKENIZING fragmenter. It builds fragments from the positions of the matched terms.

Parameters:
  • maxchars – The maximum number of characters allowed in a fragment.
  • surround – The number of extra characters of context to add both before the first matched term and after the last matched term.
  • autotrim – automatically trims text before the first space and after the last space in the fragments, to try to avoid truncated words at the start and end. For short fragments or fragments with long runs between spaces this may give strange results.

Scorers

class whoosh.highlight.FragmentScorer
class whoosh.highlight.BasicFragmentScorer

Formatters

class whoosh.highlight.UppercaseFormatter(between='...')

Returns a string in which the matched terms are in UPPERCASE.

Parameters:between – the text to add between fragments.
class whoosh.highlight.HtmlFormatter(tagname='strong', between='...', classname='match', termclass='term', maxclasses=5, attrquote='"')

Returns a string containing HTML formatting around the matched terms.

This formatter wraps matched terms in an HTML element with two class names. The first class name (set with the constructor argument classname) is the same for each match. The second class name (set with the constructor argument termclass is different depending on which term matched. This allows you to give different formatting (for example, different background colors) to the different terms in the excerpt.

>>> hf = HtmlFormatter(tagname="span", classname="match", termclass="term")
>>> hf(mytext, myfragments)
"The <span class="match term0">template</span> <span class="match term1">geometry</span> is..."

This object maintains a dictionary mapping terms to HTML class names (e.g. term0 and term1 above), so that multiple excerpts will use the same class for the same term. If you want to re-use the same HtmlFormatter object with different searches, you should call HtmlFormatter.clear() between searches to clear the mapping.

Parameters:
  • tagname – the tag to wrap around matching terms.
  • between – the text to add between fragments.
  • classname – the class name to add to the elements wrapped around matching terms.
  • termclass – the class name prefix for the second class which is different for each matched term.
  • maxclasses – the maximum number of term classes to produce. This limits the number of classes you have to define in CSS by recycling term class names. For example, if you set maxclasses to 3 and have 5 terms, the 5 terms will use the CSS classes term0, term1, term2, term0, term1.
class whoosh.highlight.GenshiFormatter(qname='strong', between='...')

Returns a Genshi event stream containing HTML formatting around the matched terms.

Parameters:
  • qname – the QName for the tag to wrap around matched terms.
  • between – the text to add between fragments.

Utility classes

class whoosh.highlight.Fragment(text, matches, startchar=0, endchar=-1)

Represents a fragment (extract) from a hit document. This object is mainly used to keep track of the start and end points of the fragment and the “matched” character ranges inside; it does not contain the text of the fragment or do much else.

The useful attributes are:

Fragment.text
The entire original text from which this fragment is taken.
Fragment.matches
An ordered list of objects representing the matched terms in the fragment. These objects have startchar and endchar attributes.
Fragment.startchar
The index of the first character in the fragment.
Fragment.endchar
The index of the last character in the fragment.
Fragment.matched_terms
A set of the text of the matched terms in the fragment (if available).
Parameters:
  • text – the source text of the fragment.
  • matches – a list of objects which have startchar and endchar attributes, and optionally a text attribute.
  • startchar – the index into text at which the fragment starts. The default is 0.
  • endchar – the index into text at which the fragment ends. The default is -1, which is interpreted as the length of text.