Enable Jekyll's LSI for related posts with fast build speed

Enable Jekyll's LSI for related posts with fast build speed

boost build speed with gsl

Jekyll’s site.related_posts by default just shows 10 recent posts. If you set the lsi to true in _config.yml, the related_posts can really work as it describes. However, enable lsi will certaily slow down the build speed. Especially for posts that written in Chinese, seems like the latent semantic indexing (LSI) will never stop and the Jekyll build process will last for hours and hours…

So, I end up with a manually programmed related posts by targetting the posts with the same tag/category using Liquid in Jekyll’s template. Details can be found in an old post: Related posts in Jekyll.

What is LSI?

Why enable lsi will lead to such slow build speed? We need to have a general feeling on what is the LSI.

Latent semantic indexing, sometimes referred to as latent semantic analysis, is a mathematical method developed in the late 1980s to improve the accuracy of information retrieval. It uses a technique called singular value decomposition to scan unstructured data within documents and identify relationships between the concepts contained therein.

In essence, it finds the latent relationships between words (semantics) in order to improve information understanding (indexing). It provided a significant step forward for the field of text comprehension as it accounted for the contextual nature of language.

So, it needs aditional calculations among different posts for finding the related posts.

Speed up LSI

It’s great to have lsi enabled for accurate related posts, and things become easier with rb-gsl that speed up LSI immensely.

However, rb-gsl requires gsl (GNU Scientific Library) as the runtime dependency, you need to install gsl locally on your build environment.

On Mac OSX, that’s easy with Homebrew:

brew install gsl

On Ubuntu/Debian:

sudo apt-get -y install libgsl-dev

Then, install these two gems or add them to your Gemfile then install them with bundler:

gem install classifier-reborn
gem install gsl

Now, you can safely enable lsi to build related posts with super fast speed.

In case you’re testing your site that doesn’t care about the related posts, you can set lsi to false in _config.yml and build related posts to production environment with bundle exec jekyll build --lsi.

Note that GitHub Pages doesn’t support lsi

林宏

Frank Lin

Hey, there! This is Frank Lin (@flinhong), one of the 1.4 billion 🇨🇳. This 'inDev. Journal' site holds the exploration of my quirky thoughts and random adventures through life. Hope you enjoy reading and perusing my posts.

YOU MAY ALSO LIKE

Using Liquid in Jekyll - Live with Demos

Web Notes

2016.08.20

Using Liquid in Jekyll - Live with Demos

Liquid is a simple templating language that Jekyll uses to process pages on your site. With Liquid you can output an modify variables, have logic statements inside your pages and loop over content.

Syntax highlight with Rouge in Jekyll

Tutorials

2017.03.18

Syntax highlight with Rouge in Jekyll

By default, Jekyll 3 and above versions integrated with Rouge, a pure Ruby syntax highlighter which supports over 100 languages. Since Rouge themes are compatible with Pygments’s stylesheets, it’s nice for us to choose a favourable style.

Related posts in Jekyll

Web Notes

2016.07.13

Related posts in Jekyll

Jekyll features a simple “Related posts” variable per post page with site.related_posts, which contains the 10 most recent posts in default. However, it only works perfectly when lsi (latent semantic indexing) option was enabled. The Liquid tags might be helpful here.

TOC / 目录

TRENDING