Developing SMUI

The SMUI sources are hosted on the corresponding GitHub repository.

For development and building, you will need

Note: As SMUI currently build on top of Play 2.7, only Java 8 and 11 are supported (see https://discuss.lightbend.com/t/play-sample-on-windows-java-lang-illegalstateexception-unable-to-load-cache-item/8663). The latest sbt might come with a Java 17. One solution is to downgrade sbt accordingly, e.g. (Mac OS with brew):

brew info sbt
# Might output: "sbt: stable 1.6.1 (bottled)"
brew info openjdk
# Might output: "openjdk: stable 17.0.1 (bottled) [keg-only]" (WARN: Version too recent)

# Uninstall sbt and openjdk
brew uninstall sbt
brew uninstall openjdk

# Unfortunately, openjdk at a downgraded version might not be available (e.g. for Mac M1 chipset) or broke when building, so openjdk@11 should be installed
brew install openjdk@11
# Maybe the downgraded sbt & openjdk version got not linked properly:
brew unlink sbt
brew link sbt@0.13
brew unlink openjdk
brew link openjdk@11
# Stick to the advise: `echo 'export PATH="/opt/homebrew/opt/openjdk@11/bin:$PATH"' >> ~/.zshrc`

Building SMUI

The code repository provides a Makefile for creating the SMUI docker image. In the project root, run

make docker-build-only

This will build both the backend and frontend components in a dockerized environment and create and tag the following images: querqy/smui:latest and querqy/smui:$VERSION, where $VERSION is the version given in the build.sbt file.

If you want to only build the fat jar itself, e.g. to build your own docker image, you may do so by running sbt:

sbt assembly

Running SMUI locally

When working on the SMUI sources, building a docker image on each change would be very time-consuming. Therefore, you may run a local development server by running:

make serve

The SMUI frontend will then be available on http://localhost:4200 , and the backend on http://localhost:9000. make serve assumes a MySQL/MariaDB instance, as outlined in the SMUI quickstart documentation. Alternatively, you may customize the database configuration. See Development configuration below.

The development server is hot-reloading, i.e. will recompile on changes in the source files.

In case more control is needed over the SMUI configuration sbt can also being used directly, e.g.:

Here are some frequently used command:

  • compile: Compiles the Scala/Play backend.

  • run -Dconfig.file=./smui-dev.conf: Runs SMUI with a dev config (see below).

  • tasks -v: Returns a complete list over all sbt commands available.

  • test: Runs all backend tests.

sbt "run -Dconfig.file=./smui-dev.conf"

Running SMUI with SSL locally

For local development that requires accessing SMUI using a HTTPS URL (e.g. to implement external authentication) we recommend using the Caddy reverse proxy.

caddy reverse-proxy --internal-certs --from https://localhost --to :9000

Development configuration

For developing new features and testing the application with different types of configuration, it is recommended to create a local development configuration of the application (instead of the productive one described above). There is the smui-dev.conf being excluded from version control through the .gitignore, so that you can safely create a local development configuration in the project’s root (naming it smui-dev.conf). Here is an example being used on a local development machine adjusting some features:

include "application.conf"

db.default.url="jdbc:mysql://localhost/smui?autoReconnect=true&useSSL=false"
db.default.username="local_dev_db_user"
db.default.password="local_dev_db_pass"

smui2solr.SRC_TMP_FILE="/PATH/TO/LOCAL_DEV/TMP/FILE.tmp"
smui2solr.DST_CP_FILE_TO="PATH/TO/LOCAL_DEV/SOLR/CORE/CONF/rules.txt"
smui2solr.SOLR_HOST="localhost:8983"

toggle.ui-concept.updown-rules.combined=true
toggle.ui-concept.all-rules.with-solr-fields=true
toggle.rule-deployment.log-rule-id=true
toggle.rule-deployment.split-decompound-rules-txt=true
toggle.rule-deployment.split-decompound-rules-txt-DST_CP_FILE_TO="/PATH/TO/LOCAL_DEV/SOLR/CORE/CONF/decompound-rules.txt"
toggle.rule-deployment.pre-live.present=true
toggle.rule-deployment.custom-script=true
toggle.rule-deployment.custom-script-SMUI2SOLR-SH_PATH="/PATH/TO/LOCAL_DEV/smui2solr-dev.sh"
toggle.rule-tagging=true
toggle.predefined-tags-file="/PATH/TO/LOCAL_DEV/predefined-tags.json"

...

play.http.secret.key="<generated local play secret>"

# smui.authAction = controllers.auth.BasicAuthAuthenticatedAction
# smui.BasicAuthAuthenticatedAction.user = smui_dev_user
# smui.BasicAuthAuthenticatedAction.pass = smui_dev_pass

As you can see, for development purposes you are recommended to have a local Solr installation running as well.

For running The SMUI application locally on your development machine pass the above config file when starting the application in sbt, e.g.:

sbt run -Dconfig.file=./smui-dev.conf 9000

Furthermore, above’s configuration points to an alternative development version of the smui2solr.sh-script. The file smui2solr-dev.sh is as well excluded from the version control. The following example provides a simple custom deployment script approach, that basically just delegates the script call to the main smui2solr.sh one:

echo "In smui2solr-dev.sh - DEV wrapper for smui2solr.sh, proving custom scripts work"

BASEDIR=$(dirname "$0")
$BASEDIR/conf/smui2solr.sh "$@"
exit $?

It can be used as a basis for extension.

Note

Remember to make the script executable (chmod +x).

Testing SMUI

Backend unit tests (specifications)

To execute a specific unit test, you can call the corresponding scalatest with sbt, e.g.:

sbt "testOnly models.querqy.QuerqyExactMatchRulesGeneratorSpec"

NOTE: The testOnly instruction needs to be in quotations (otherwise all tests will be executed).

You can also run through the whole specification using the sbt test command.

Useful commands: Docker powered MariaDB (with local persistence)

Assuming a folder smui_runtime_data exists.

# start MySQL
docker run --name smui-mysql -p 3306:3306 -v /LOCAL/PATH/TO/smui_runtime_data:/var/lib/mysql -e MYSQL_ROOT_PASSWORD=smui -e MYSQL_USER=smui -e MYSQL_PASSWORD=smui -e MYSQL_DATABASE=smui -d mysql
# stop
docker stop smui-mysql
# list and remove eventually
docker container ls -a
docker container rm <CONTAINER_ID>

Note: Setting the MySQL root password is only for making potential root access easy (if necessary at some point).

Also note: When developing with an Apple Silicon (M1 based) device, there does not seem to exist a suitable arm image for MySQL (as of Jan 2022). Therefore, the x86 architecture needs to be specified explicitly: docker run --platform linux/x86_64 --name smui-mysql [...]

Developing git deployment method

SMUI offers the possibility to deploy rules.txt (files) to a git repository. For doing so in a local development setup, it might therefore be necessary to operate a local git instance. The following section describes how that can be achieved.

Bootstrap a local git server (docker)

For the local git server, the dockerhub image jkarlos/git-server-docker will be used, see (command line):

# create a private/public (SSH) key
# e.g. ssh-keygen -t rsa -C "yourself@YourComputer.local"
# create repo folder and provide (public) key
mkdir <SMUI_GIT_ROOT>/keys
mkdir <SMUI_GIT_ROOT>/repos
# TODO better symlink?
cp ~/.ssh/id_rsa.pub <SMUI_GIT_ROOT>/keys/
# start the container (and provide public key)
docker run -d -p 22:22 -v <SMUI_GIT_ROOT>/keys:/git-server/keys -v <SMUI_GIT_ROOT>/repos:/git-server/repos jkarlos/git-server-docker
# NOTE: Your local development user must have permission to access information of your local git user (in case they differ)

Init the git repository

You can run the following script (preferred as git test user itself) to init the repo (command line):

# from within the git server docker container
# NOTE: open shell in container:
docker exec -it <CONTAINER_ID> /bin/sh
# (docker ps will give you the CONTAINER_ID)
cd <SMUI_GIT_ROOT>/repos
mkdir smui_rulestxt_repo
cd smui_rulestxt_repo
git init --shared=true
git add .
git commit -m "my first commit"
cd ..
git clone --bare smui_rulestxt_repo smui_rulestxt_repo.git
# initial manual checkout (on the host machine)
# make sure, there exists an (at least empty) common rules.txt file on the master branch (clone it somewhere and create a master branch)
touch rules.txt
git add rules.txt
git commit -m "empty rules.txt commit"
git push

To configure and start SMUI using a git deployment see “Deploy rules.txt to a git target“.

Developing github actions for SMUI

Build and deployment of SMUI’s official docker image on DockerHub is realised through a github action, which is located under: .github/workflows.

Testing SMUI relies on Ryuk test containers within the sbt test build step. Unfortunately, there seems to be an issue with local build containers used by act (https://github.com/nektos/act), so that adjustments to the deployment workflow can only be tested while triggering the build on the github infrastructure (master push) and not tested locally with act.

This problem is described in the following issue: https://github.com/nektos/act/issues/501.

Anyway, the workflow performs well on the github action container infrastructure.

Releasing SMUI

Every release of SMUI should come with its own version incrementing at least the build number of the semver. The release number is maintained under:

smui/build.sbt

The SmuiVersionSpec enforces version number increments in the test automation.

Have fun coding SMUI!!