Developing SMUI¶
The SMUI sources are hosted on the corresponding GitHub repository.
For development and building, you will need
docker with BuildKit capabilities (i.e. version 18.09 or higher)
docker-compose (optional)
Java Runtime Environment (version 11 is recommended)
NodeJS (version 12 is recommended)
Note: As SMUI currently build on top of Play 2.7, only Java 8 and 11 are supported (see https://discuss.lightbend.com/t/play-sample-on-windows-java-lang-illegalstateexception-unable-to-load-cache-item/8663). The latest sbt might come with a Java 17. One solution is to downgrade sbt accordingly, e.g. (Mac OS with brew):
brew info sbt
# Might output: "sbt: stable 1.6.1 (bottled)"
brew info openjdk
# Might output: "openjdk: stable 17.0.1 (bottled) [keg-only]" (WARN: Version too recent)
# Uninstall sbt and openjdk
brew uninstall sbt
brew uninstall openjdk
# Unfortunately, openjdk at a downgraded version might not be available (e.g. for Mac M1 chipset) or broke when building, so openjdk@11 should be installed
brew install openjdk@11
# Maybe the downgraded sbt & openjdk version got not linked properly:
brew unlink sbt
brew link sbt@0.13
brew unlink openjdk
brew link openjdk@11
# Stick to the advise: `echo 'export PATH="/opt/homebrew/opt/openjdk@11/bin:$PATH"' >> ~/.zshrc`
Building SMUI¶
The code repository provides a Makefile for creating the SMUI docker image. In the project root, run
make docker-build-only
This will build both the backend and frontend components in a dockerized environment and create and tag the following
images: querqy/smui:latest
and querqy/smui:$VERSION
, where $VERSION is the version given in the build.sbt file.
If you want to only build the fat jar itself, e.g. to build your own docker image, you may do so by running sbt:
sbt assembly
Running SMUI locally¶
When working on the SMUI sources, building a docker image on each change would be very time-consuming. Therefore, you may run a local development server by running:
make serve
The SMUI frontend will then be available on http://localhost:4200 , and the backend on http://localhost:9000.
make serve
assumes a MySQL/MariaDB instance, as outlined in the SMUI quickstart
documentation. Alternatively, you may customize the database configuration. See Development configuration
below.
The development server is hot-reloading, i.e. will recompile on changes in the source files.
In case more control is needed over the SMUI configuration sbt
can also being used directly, e.g.:
Here are some frequently used command:
compile
: Compiles the Scala/Play backend.run -Dconfig.file=./smui-dev.conf
: Runs SMUI with a dev config (see below).tasks -v
: Returns a complete list over allsbt
commands available.test
: Runs all backend tests.
sbt "run -Dconfig.file=./smui-dev.conf"
Running SMUI with SSL locally¶
For local development that requires accessing SMUI using a HTTPS URL (e.g. to implement external authentication) we recommend using the Caddy reverse proxy.
caddy reverse-proxy --internal-certs --from https://localhost --to :9000
Development configuration¶
For developing new features and testing the application with different types
of configuration, it is recommended to create a local development
configuration of the application (instead of the productive one
described above). There is the smui-dev.conf
being excluded from
version control through the .gitignore
, so that you can safely
create a local development configuration in the project’s root (naming
it smui-dev.conf
). Here is an example being used on a local
development machine adjusting some features:
include "application.conf"
db.default.url="jdbc:mysql://localhost/smui?autoReconnect=true&useSSL=false"
db.default.username="local_dev_db_user"
db.default.password="local_dev_db_pass"
smui2solr.SRC_TMP_FILE="/PATH/TO/LOCAL_DEV/TMP/FILE.tmp"
smui2solr.DST_CP_FILE_TO="PATH/TO/LOCAL_DEV/SOLR/CORE/CONF/rules.txt"
smui2solr.SOLR_HOST="localhost:8983"
toggle.ui-concept.updown-rules.combined=true
toggle.ui-concept.all-rules.with-solr-fields=true
toggle.rule-deployment.log-rule-id=true
toggle.rule-deployment.split-decompound-rules-txt=true
toggle.rule-deployment.split-decompound-rules-txt-DST_CP_FILE_TO="/PATH/TO/LOCAL_DEV/SOLR/CORE/CONF/decompound-rules.txt"
toggle.rule-deployment.pre-live.present=true
toggle.rule-deployment.custom-script=true
toggle.rule-deployment.custom-script-SMUI2SOLR-SH_PATH="/PATH/TO/LOCAL_DEV/smui2solr-dev.sh"
toggle.rule-tagging=true
toggle.predefined-tags-file="/PATH/TO/LOCAL_DEV/predefined-tags.json"
...
play.http.secret.key="<generated local play secret>"
# smui.authAction = controllers.auth.BasicAuthAuthenticatedAction
# smui.BasicAuthAuthenticatedAction.user = smui_dev_user
# smui.BasicAuthAuthenticatedAction.pass = smui_dev_pass
As you can see, for development purposes you are recommended to have a local Solr installation running as well.
For running The SMUI application locally on your development machine
pass the above config file when starting the application in sbt
,
e.g.:
sbt run -Dconfig.file=./smui-dev.conf 9000
Furthermore, above’s configuration points to an alternative development
version of the smui2solr.sh
-script. The file smui2solr-dev.sh
is
as well excluded from the version control. The following example
provides a simple custom deployment script approach, that basically just
delegates the script call to the main smui2solr.sh
one:
echo "In smui2solr-dev.sh - DEV wrapper for smui2solr.sh, proving custom scripts work"
BASEDIR=$(dirname "$0")
$BASEDIR/conf/smui2solr.sh "$@"
exit $?
It can be used as a basis for extension.
Note
Remember to make the script executable (chmod +x).
Testing SMUI¶
Backend unit tests (specifications)¶
To execute a specific unit test, you can call the corresponding scalatest
with sbt
, e.g.:
sbt "testOnly models.querqy.QuerqyExactMatchRulesGeneratorSpec"
NOTE: The testOnly
instruction needs to be in quotations (otherwise all tests will be executed).
You can also run through the whole specification using the sbt
test
command.
Useful commands: Docker powered MariaDB (with local persistence)¶
Assuming a folder smui_runtime_data
exists.
# start MySQL
docker run --name smui-mysql -p 3306:3306 -v /LOCAL/PATH/TO/smui_runtime_data:/var/lib/mysql -e MYSQL_ROOT_PASSWORD=smui -e MYSQL_USER=smui -e MYSQL_PASSWORD=smui -e MYSQL_DATABASE=smui -d mysql
# stop
docker stop smui-mysql
# list and remove eventually
docker container ls -a
docker container rm <CONTAINER_ID>
Note: Setting the MySQL root password is only for making potential root access easy (if necessary at some point).
Also note: When developing with an Apple Silicon (M1 based) device, there does not seem to exist a suitable arm image for MySQL (as of Jan 2022). Therefore, the x86 architecture needs to be specified explicitly: docker run --platform linux/x86_64 --name smui-mysql [...]
Developing git deployment method¶
SMUI offers the possibility to deploy rules.txt (files) to a git repository. For doing so in a local development setup, it might therefore be necessary to operate a local git instance. The following section describes how that can be achieved.
Bootstrap a local git server (docker)¶
For the local git server, the dockerhub image jkarlos/git-server-docker will be used, see (command line):
# create a private/public (SSH) key
# e.g. ssh-keygen -t rsa -C "yourself@YourComputer.local"
# create repo folder and provide (public) key
mkdir <SMUI_GIT_ROOT>/keys
mkdir <SMUI_GIT_ROOT>/repos
# TODO better symlink?
cp ~/.ssh/id_rsa.pub <SMUI_GIT_ROOT>/keys/
# start the container (and provide public key)
docker run -d -p 22:22 -v <SMUI_GIT_ROOT>/keys:/git-server/keys -v <SMUI_GIT_ROOT>/repos:/git-server/repos jkarlos/git-server-docker
# NOTE: Your local development user must have permission to access information of your local git user (in case they differ)
Init the git repository¶
You can run the following script (preferred as git test user itself) to init the repo (command line):
# from within the git server docker container
# NOTE: open shell in container:
docker exec -it <CONTAINER_ID> /bin/sh
# (docker ps will give you the CONTAINER_ID)
cd <SMUI_GIT_ROOT>/repos
mkdir smui_rulestxt_repo
cd smui_rulestxt_repo
git init --shared=true
git add .
git commit -m "my first commit"
cd ..
git clone --bare smui_rulestxt_repo smui_rulestxt_repo.git
# initial manual checkout (on the host machine)
# make sure, there exists an (at least empty) common rules.txt file on the master branch (clone it somewhere and create a master branch)
touch rules.txt
git add rules.txt
git commit -m "empty rules.txt commit"
git push
To configure and start SMUI using a git deployment see “Deploy rules.txt to a git target“.
Developing github actions for SMUI¶
Build and deployment of SMUI’s official docker image on DockerHub is realised through a github action, which is located under: .github/workflows.
Testing SMUI relies on Ryuk test containers within the sbt test
build step. Unfortunately, there seems to be an issue with local build containers used by act
(https://github.com/nektos/act), so that adjustments to the deployment workflow can only be tested while triggering the build on the github infrastructure (master push) and not tested locally with act
.
This problem is described in the following issue: https://github.com/nektos/act/issues/501.
Anyway, the workflow performs well on the github action container infrastructure.
Releasing SMUI¶
Every release of SMUI should come with its own version incrementing at least the build number of the semver. The release number is maintained under:
smui/build.sbt
The SmuiVersionSpec
enforces version number increments in the test automation.
Have fun coding SMUI!!