When a Moodle scheduled task crashes mid-run, the lock it held can persist indefinitely, blocking every subsequent run. This guide explains how Moodle's task locking works, how to identify a stuck task, and how to clear it without disrupting active tasks.
How Moodle task locking
works
Every scheduled task and ad-hoc task acquires a lock before it runs.
By default Moodle uses a file-based lock factory: it creates a lock file
inside moodledata/lock/ named after the task. While the
lock file exists, no other process can start the same task.
When a task finishes normally it removes the lock file. When it
crashes — OOM kill, PHP fatal, server restart — the lock file stays. The
next scheduled run finds the lock taken and either waits until a timeout
or skips the task entirely. This repeats on every subsequent run. From
the admin panel everything looks normal: the task is enabled and appears
in the list. It just never runs.
Identifying the stuck task
The scheduled task log is the first place to check:
Site administration > Server > Tasks > Scheduled task log
Filter by the task name and look for a run that started but has no
matching completion entry, or check the last run time — if it is hours
older than the configured schedule, the task is likely locked.
From the database:
SELECT classname, lastruntime, nextruntime, timestarted
FROM mdl_task_scheduled
WHERE timestarted > 0
ORDER BY timestarted DESC;
For ad-hoc tasks (queued jobs from enrolments, messaging, etc.):
SELECT classname, timestarted, timecreated, faildelay
FROM mdl_task_adhoc
WHERE timestarted > 0
ORDER BY timestarted;
A row with a timestarted value and no corresponding
completion means the task is either still running or was abandoned
mid-run. Cross-check against running PHP processes:
ps aux | grep php
If there is no matching process for the task, the lock is stale.
Finding the lock file
Lock files live in $CFG->dataroot/lock/. The
filenames are derived from the task class name. For example, the lock
for \core\task\send_new_user_passwords_task will be a file
in the lock/ directory with a name based on that class.
ls -la /var/moodledata/lock/
Files older than your expected task duration with no matching running
process are safe to remove.
Clearing a stale lock
For a file-based lock, removing the file is sufficient:
rm /var/moodledata/lock/task_<lockname>
If your installation uses a database lock factory
($CFG->lock_factory = '\core\lock\db_record_lock_factory'),
locks are stored in mdl_lock_db:
SELECT * FROM mdl_lock_db;
DELETE FROM mdl_lock_db WHERE resourcekey LIKE '%classname%';
After clearing the lock, trigger the task manually to confirm it
runs:
Some tasks — H5P content sync, Turnitin submission checks, large
report generation — can run for several minutes and hold a lock for the
entire duration. When they share a cron invocation with fast tasks, a
slow run blocks or delays everything that follows.
Run heavy tasks on their own dedicated CLI invocation, separate from
the general cron:
# In a separate cron entry:
php /var/www/moodle/admin/cli/scheduled_task.php \
--execute='\mod_turnitintooltwo\task\submission_scores_sync'
This keeps the general cron moving and gives you independent control
over the frequency and timeout of the heavy task.
Persistent ad-hoc task
backlogs
Ad-hoc tasks that repeatedly fail accumulate a faildelay
value and are retried with increasing backoff. A large backlog combined
with a lock issue can cause the queue to grow faster than it drains.
Check the queue depth:
SELECT classname, COUNT(*) AS queued, MAX(faildelay) AS max_delay
FROM mdl_task_adhoc
GROUP BY classname
ORDER BY queued DESC;
Tasks with a high faildelay and a large count are
candidates for investigation. If the failure is environmental (a
third-party API that was down, a missing file), clearing
faildelay and resetting timestarted lets them
retry immediately:
UPDATE mdl_task_adhoc
SET faildelay = 0, timestarted = 0
WHERE classname = '\mod_assign\task\cron_task'
AND faildelay > 0;
Only do this once you have resolved the underlying cause.
When running a multilingual Moodle site with SAML2 single sign-on via the auth_saml2 plugin, you may need the login button label to appear in different languages based on the user’s interface language. Two approaches that look like they should work, do not:
Putting <span class="multilang" lang="en">...</span> tags into the “IdP label override” setting has no effect. The multilang filter only runs on content rendered through Moodle’s format_text() or format_string(). The auth_saml2 plugin passes the label directly from the database to the template without a filter pass.
Using the Language customization tool (Site administration > Language > Language customization) to override auth_saml2 language strings will not help for a custom label either. That tool only modifies strings defined in language files, not values stored as admin settings in the database.
The root cause: auth_saml2 stores the IdP label as a plain string in mdl_config_plugins and renders it via {{name}} in the login form template with no language awareness.
The correct approach: theme template override
The reliable solution is to override core/loginform.mustache in your Moodle theme and replace the dynamic {{name}} rendering with Moodle language strings defined in the theme.
Moodle’s {{#str}} Mustache helper always resolves against the current user’s interface language. By moving the label from a database-stored admin setting into theme language files, you get full multilang support through the standard Moodle mechanism.
Prerequisites
Template overrides must live inside a theme — there is no other mechanism in Moodle for overriding Mustache templates. If you are not already on a custom theme, create a minimal child theme before proceeding. Editing a third-party theme directly will get overwritten on the next theme update.
Drop the theme directory into theme/ and visit the Moodle notifications page to register it. You do not need to activate it as the default theme yet — do that once the template override is in place.
Step 1: Override the login form template
Copy lib/templates/core/loginform.mustache from Moodle core into your theme at:
theme/yourtheme/templates/core/loginform.mustache
If your parent theme already overrides this template, copy from the parent theme instead of from core, so you preserve its customizations.
In the template, find the block that renders identity providers. The default Moodle core template renders the button label as:
For multi-IdP setups where each provider needs a distinct translated label, the loop approach breaks down because there is no per-provider identifier in the template context. In that case, bypass the generic loop and hard-code one button per provider, each pointing to a known URL and using a dedicated string key. This makes the template more tightly coupled to a specific IdP configuration, but gives you full label control per provider.
Step 2: Add language files
Create a language file for each language your site supports:
Each file defines the same string keys with translated values.
lang/en/theme_yourtheme.php:
<?php
defined('MOODLE_INTERNAL') || die();
$string['saml_login_label'] = 'Log in with your organization account';
lang/nl/theme_yourtheme.php:
<?php
defined('MOODLE_INTERNAL') || die();
$string['saml_login_label'] = 'Inloggen met uw organisatieaccount';
Moodle will automatically use the file matching the user’s current interface language. If no file exists for the user’s language, Moodle falls back to English.
Step 3: Purge caches
After deploying your changes, purge Moodle’s theme and template caches: Site administration > Development > Purge all caches.
Caveats
This approach fully decouples the login button label from the auth_saml2 admin setting. Once you override the template, the “IdP label override” field in the plugin settings has no effect on sites using your theme. You own the label entirely from the theme side.
If you later reconfigure auth_saml2 (add a new IdP, change metadata), you will need to update the template and lang files manually to match. This is a minor maintenance trade-off for a clean, built-in multilang solution.
Moodle cron fails silently more often than you'd expect. This guide explains how to detect those failures reliably using an external heartbeat monitor, and walks through the code that implements it as a Moodle plugin.
Why Moodle’s
built-in cron status is not enough
Moodle shows a “Last cron run” timestamp in Site
administration > Server > Tasks > Scheduled tasks.
That value is written when a cron run starts, not when it
finishes. A run that hits a PHP fatal error, exhausts available
memory, or gets stuck on a locked task still updates the timestamp
before it stops.
The result: the admin panel shows a recent timestamp, everything
looks normal, but no tasks have completed for hours.
The dead man’s switch pattern
An external cron monitor works as a dead man’s switch:
At the end of a successful cron run, your server sends a signal — a
simple HTTP GET request — to a URL that the monitoring service
provides.
The service expects to receive that signal on a schedule you
define.
If the signal does not arrive within the expected window, the
service sends an alert.
The monitoring service has no access to your server. It only waits
for the ping.
Why a scheduled
Moodle task doesn’t work here
The obvious approach would be to add a scheduled task that runs last
and fires the ping. Moodle doesn’t support that. There is no “cron
finished” event, and you cannot configure a task to run after all
others.
More fundamentally: if cron is broken, a scheduled task inside that
cron cannot reliably fire. You need the monitor to live outside of
Moodle’s task scheduler.
The approach: a wrapper
script
The solution is to replace the direct cron invocation with a
pipe:
The 2>&1 merges stderr into stdout so that PHP
errors are captured alongside normal output. The wrapper script receives
everything on stdin, inspects it, and fires the appropriate ping.
The key check
Moodle’s cron script writes one predictable string when it completes
successfully:
Cron script completed correctly
The wrapper reads stdin and looks for that string:
If the string is absent — because cron crashed, was killed, or timed
out — the failure branch runs: it fires the failure ping and emails the
full cron log to the configured recipients.
Sending the ping
The ping itself is a GET request to a configurable URL. Inside a
Moodle CLI script you have the full Moodle stack available, so you can
use Moodle’s own curl class:
require_once($CFG->libdir . '/filelib.php');
$curl = new curl();
$curl->get($url);
Having the URL configurable from within Moodle admin — rather than
hardcoded in a shell script — means an administrator can change the
monitoring endpoint without touching the server.
Storing the URL as a Moodle
setting
A single admin settings page handles all configuration. The success
and failure URLs are stored as plugin config values:
$settings->add(new admin_setting_configtext(
'tool_heartbeat/cron_monitor_success_url',
new lang_string('cron_monitor_success_url', 'tool_heartbeat'),
new lang_string('cron_monitor_success_url_desc', 'tool_heartbeat'),
'',
PARAM_URL
;
You configure a monitor on any dead man’s switch service that accepts
HTTP pings. Set the expected interval to match your cron schedule and
add a grace period long enough to cover normal load variation. When
pings stop arriving, the service sends an alert through whatever channel
you choose — email, SMS, webhook.
Edge cases
OOM kills: If PHP is terminated by the operating
system before producing output, the pipe closes without the success
string. The monitoring service will not receive a ping within the
expected window and will alert. The email log in this case will be empty
or partial — check dmesg for OOM killer entries.
Parallel cron runs: The wrapper applies only to the
CLI invocation managed by the system cron job. If you run multiple
parallel cron processes or use a cron daemon, each invocation needs its
own pipe.
Reading stdin:
file_get_contents('php://stdin') works in most setups, but
on some systems the stream is not immediately available. A fallback
using fopen and fgets is worth adding for
robustness:
We implemented this as a Moodle admin tool plugin
(tool_heartbeat), compatible with Moodle 3.9 and later. It
includes the wrapper CLI script, all admin settings, optional email
notifications with the full cron log on failure, and a master on/off
toggle. Contact Solin if you want to use
it directly.
Moodle's AMD modules load JavaScript asynchronously and isolate dependencies using RequireJS. This guide covers module structure, injecting Moodle services, localization, and building production-ready minified output using Grunt.
SOP: Writing and Building AMD Modules in Moodle
This guide will walk you through creating, localizing, and building asynchronous module definition (AMD) JavaScript modules in Moodle, as well as how to pull in core Moodle services (like the notification system) from your AMD code.
1. What is AMD in Moodle?
Moodle uses RequireJS to load JavaScript modules asynchronously. An AMD module is a JavaScript file that:
Declares its dependencies via a define([...], function(...) { … }) call.
Exports one or more functions (often a single init entry point).
Can be minified and concatenated into production-ready code.
By using AMD, modules only load when needed and avoid polluting the global namespace.
2. Folder Layout
Inside your plugin (e.g. mod/assessmentform/), you should have:
mod/assessmentform/
├── amd/
│ ├── src/ ← Your unbuilt source files
│ │ └── survey_init.js
│ └── build/ ← Moodle’s build step will place minified files here
│ └── survey_init.min.js
└── view.php ← Calls your AMD module via js_call_amd()
amd/src/Place your human-readable modules here.
amd/build/After running the build, minified code will live here. Commit the build outputs so your plugin works without requiring Node on the production server.
core/strOffers str.get_string(key, component) for on-the-fly translations.
js_call_amd() in PHPPass an associative array of arguments from PHP into your JS entry-point.
4. Injecting Moodle Services
Just list them in your dependency array:
define([
'jquery',
'core/notification',
'core/str',
'core/ajax',
// …your other libraries…
], function($, notification, str, Ajax) {
// You now have `notification`, `str` and `Ajax.call()` available.
});
Moodle provides a Grunt setup out of the box. At your plugin root:
Install tools (one-time per project):
npm install --global grunt-clinpm install
Build all AMD modules (including yours):
grunt amd
This reads every amd/src/*.js in all plugins, minifies them, and writes to amd/build/*.min.js.
You can also target just your plugin:
grunt amd:mod_assessmentform
Commit the contents of amd/build/ into your plugin’s repository.
If you prefer a standalone build, you can drop a Gruntfile.js at your plugin root that only references your AMD folder. Moodle’s core Gruntfile already handles this; there’s no extra setup needed.
7. Summary Checklist
Create amd/src/yourmodule.js with a define([...], ...).
Build with npm install && grunt amd and commit amd/build/*.min.js.
Test on your dev site, then deploy.
With this workflow in place, you can rapidly iterate your JS in amd/src/ and then generate production-ready, minified AMD modules via Moodle’s standard Grunt toolchain.
Monitoring a Moodle web server requires both datacenter-level alerting and application-specific checks. This guide covers configuring your hosting provider's monitoring tools, adding site-specific metrics, and tracking trends over time.
Goal of Monitoring
The goal of monitoring is to ensure that the system, i.e. the web server and all software on it, is working properly and within established parameters. If at any time a website or a subsystem on the web server stops functioning, a signal should be sent out to the sysop, who maintains the system.
In addition, it should also be possible to examine trends over time, or historic data, to evaluate whether or not the system’s resources should be expanded (or scaled back) in the future.
You will notice that we are relying on two monitoring systems now: one provided by the data center, and a monitoring system based on Webmin, which is an administrative system for (web) servers. The reason for adding Webmin’s monitoring is that the data center does not allow you to monitor specific websites, but Webmin does.
1. Check Monitoring Settings of the Data Center
The data center may have its own monitoring that comes pre-installed and configured with a new web server (VPS). Just make sure that everything is set up correctly.
For instance, for HostEurope, do the following. Sign in to HostEurope’s KIS website: https://kis.hosting-provider.de/ and click on the appropriate type of server: either Virtual Server 10+ or Virtual Server. In this guide we show the first type.
In the following screen, click on the login button, under the Contract column:
This will open a new browser window (or tab). Here you see the current usage:
The following metrics should not exceed 80%:
CPU cores
RAM
And Disk space should not exceed 95%.
If the system is not used to send out email, then the SMTP relays metric is typically 0.
Ideally, Uptime monitoring is 100%, but may decrease slightly to 99.91% over time.
Now click on the Monitoring tab, which should take you to the next screen:
Here, make sure all the settings for Manage Email Alerts are switched on.
This monitor will send out an email to the owner of the KIS account with an alert if either CPU, Disk or RAM usage exceeds 80%.
External Monitoring
It is also recommended to add an external monitor. An external monitor is a monitor that resides on another system. For instance, you can use cronitor.io for free to perform a GET request every five minutes to a website on the server you want to monitor. Don’t forget to add your email address so you will receive notifications when the monitor fails.
Using an external monitor ensures you get alerted if the server goes down even if the entire data center goes down with it.
Heartbeat Monitor
We have a custom plugin, tool_heartbeat, which can be used to send out an “I’m alive” signal to cronitor.io (or a comparable service). Use this tool to make sure Moodle’s (or Totara’s) cron is still working.
Here's how it works:
The Moodle or Totara site stops telling Cronitor "I'm alive!" for whatever reason. (The Heartbeat plugin does this, hence the name.)
Cronitor notices Totara is no longer alive, waits 5 minutes just in case, and then sends out an alert "Type: Alert" ("Event not received on schedule").
If (when) Totara is reanimated, Cronitor sends out an alert "Type: Recovery".
So, in the email messages from Cronitor, "Alert" means there's a problem, and "Recovery" means it's fixed.
Installation and configuration
Place the contents of this directory inside the /admin/tool/heartbeat folder relative to your Moodle or Totara install path.
Configure the cron job to * * * * * php /path_to_your_moodle/admin/cli/cron.php | php /path_to_your_moodle/admin/tool/heartbeat/cli/cron.php > /dev/null
Plugins settings
Cron monitor: Enable the monitor and add the url of the external cron monitor service
Email settings: Enable the email notifications, add the email subject and body, select recipients that get the email.
2. Make sure Webmin is Installed
Our standard procedure is to install Webmin, an administrative system for web servers. So Webmin should be installed and accessible, typically through the hostname and the 10000 port, e.g.: https://vps2.example.internal:10000/.
If it is not installed, please see the installing a new Moodle website.
3. Configure Webmin to Monitor Critical Systems and Websites
Go to Webmin and open the Tools >System and Server Status section:
We need to add five types of monitors:
Load average: what is the average usage of the system in during the last 15 minutes
Disk space: how much is left on the disk (typically an SSD drive)
Apache web server: is the web server up and running?
Free memory: how much free memory do we have left?
MySQL database server: is the database server up and running?
To add a new monitor in Webmin, use the select box next to the button Add monitor of type and then click the button.
Settings for All New Monitors
For all new monitors, do not forget to add a Description that includes your users’s name (or main website), and fill out the field “Also send email for this service to” with the address of the person in the sysop role for this server. Set the field “Failures before reporting” to 1. (See the screenshots below for some examples of where to find these fields.)
Load Average Monitor
The average load is the usage of the system (mainly CPU usage) during the past 5, 10 and 15 minutes. To get a good perspective, we set this monitor to 15 minutes, under Load average to check.
The Maximum load average value is critical: it should not exceed 80%. The actual value to fill in, is based on the number of CPU cores. This is the computation:
n cores x .8
For instance, 1 core is 0.8, and 4 cores gives you a value of 3.2.
The number of cores can be retrieved from Webmin as well. Simply go to Webmin’s homepage and look for Processor information. There you find the number of cores:
This is pretty straightforward: just fill in 5%. This should send out an alert if the disk is over 95% capacity. Filesystem to check is /.
Apache Web Server Monitor
The defaults for this monitor should be fine.
Free Memory Monitor
For this monitor, two values are critical:
Minimum free real memory: we want 20% to be free (or max 80% used)
Minimum free virtual memory: we want this set equal to the amount of physical RAM.
To compute the 20% minimum free RAM, we need to know the total available real memory. You can find this on the “homepage” of Webmin:
Webmin reports the total memory in Gigibytes (GiB). But the Free Memory monitor uses megabytes (MB). To convert the free memory from GiB to MB, use the following formula:
MB = 1073.74 x n GiB
For instance, if we have 7.77 GiB that gives us 8342.9598 MB. Of this number, we take 20% to fill in for the minimum free real memory, and 25% of the virtual memory as the “Minimum free virtual memory”.
MySQL Database Server Monitor
The defaults for this monitor are fine. Make sure that the “Failures before reporting” field is set to 1 and that the “Also send email for this service to” field is filled in.
4. Add a “Remote HTTP Service” Monitor to Another Webmin
What happens if the entire web server is out or can no longer be reached? In that case, all the monitors we added in the section above will no longer run, or if they are still running, their email alerts may not reach you.
To counter this, we add a “Remote HTTP Service” monitor to a Webmin installation on another web server entirely:
As you can tell from the Status history, this check is performed every 5 minutes.
Set the field “Connection timeout” to 10 seconds. This should also notify you if the loading times for the Moodle website get unacceptable (i.e. more than 10 seconds).
5. Test the Monitoring
Testing should only be done on a completely new system that is not in use yet. The monitors are typically working – they consist of proven, well tested software. So we will not be testing that the monitoring software works, but mainly that we have configured it correctly.
The most critical monitor is the one for the actual Moodle website. We test this by simply turning off the web server. This can be done in Webmin.
Go to Servers > Apache and click the stop button, but only on a new system that is not in use yet:
If you have configured the Remote HTTP Service monitor correctly, you should receive an email very soon.
Restart the Apache web server by clicking on the play button.
You can also stop and start Apache on the command line:
If you do not receive any email, make sure that you have used the correct email address, and the correct url (including the port: nowadays almost always 443).
6. Install a New Munin Node on the Web Server
Munin is a logging tool which consists of a server and a node. The node is installed on the system that you want to monitor. The server is where you login to view the historical data. We already have the server in place.
If you login to monitoring.example.internal, you will see an overview of the systems that we are currently monitoring through Munin. Click on a specific system to view the details. Here is an example of the history of the load average:
To install the node on a new web server:
Make sure that the library libparse-http-useragent-perl is installed, e.g.:
sudo apt-get install libparse-http-useragent-perl
Install munin:
apt-get install munin
apt-get install munin-node
Make sure that the Apache’s server-status module is enabled. (You can do this through Webmin.)
Add the ip address of the Munin server (i.e. the “master”) to /etc/munin/munin-node.conf:
allow ^xxx\.xxx\.xxx\.xxx$
Configure the munin plugins.
Configuring The Munin Plugins
The default plugins for the node (so, on your Munin “client” web server) are in /usr/share/munin/plugins/. They appear in your munin website if they're symlinked in /etc/munin/plugins. For instance:
In /etc/munin/plugins, add symlinks to the apache plugins:
You must also configure them in the file /etc/munin/plugin-conf.d/munin-node. In that file, if you want to configure multiple plugins at once, use an asterisk notation. E.g.:
[apache*]
This addresses all apache plugins, which are by default:
apache_accesses
apache_processes
apache_volume
Usually when you look at the source code of the plugins (they're mostly perl scripts), you will find configuration instructions. For instance, the apache plugins need access to Apache's server status, so you have to configure Apache (i.e. httpd.conf):
<Location /server-status>
SetHandler server-status
Order deny,allow
Deny from all
Allow from 127.0.0.1
</Location>
ExtendedStatus on
We should also mention here that some plugins seem to exclude each other. For instance, the apache_average_time_last_n_requests plugin (not installed by default) seems to exclude the other (default) apache plugins.
Finally, restart the node:
/etc/init.d/munin-node restart
And open the firewall for port 4949.
Please note: if any of the Munin plugins fail, you will not see any date from that Munin node on the server (monitoring.example.internal)!
Configure The Munin Server
Finally, you also have to tell the Munin server to start polling the newly added node. Add the ip address of the node server to the file /etc/munin/munin.conf:
[ArbitraryServerName] # Apparently, you can't use spaces in this name
address xxx.xxx.xxx.xxx
use_node_name yes
The Munin server (the 'master') will read the new values within 5 minutes (the default polling interval).
Detailed Monitoring
If you run into any trouble with a VPS, you can add more detailed monitoring.
Performance Monitoring
The following is a monitoring script based on an email exchange with Hosting Provider, May 19th 2022 about the website outages on their VS10 Linux VPS (search for 198.51.100.43 #HE-DE:2ad1f7b4109530473 in the email history).
date >> /var/log/custom-monitoring.log; top -n 1 -b >> /var/log/custom-monitoring.log; lsof -ni >> /var/log/custom-monitoring.log
This log will contain detailed performance information which you can use to identify which particular application is causing high load, for instance.
Explanation:
date: current date and time
top: display linux processes;
-n 1: Specifies the maximum number of iterations, or frames, top should produce before ending.
-b: Starts top in Batch mode, which could be useful for sending output from top to other programs or to a file. In this mode, top will not accept input and runs until the iterations limit you've set with the `-n' command-line option or until killed.
lsof: lists on its standard output file information about files opened by processes
-i: selects the listing of files any of whose Internet address matches the address specified in i. If no address is specified, this option selects the listing of all Internet and x.25 (HP-UX) network files.
-n: selects the listing of files any of whose Internet address matches the address specified in i. If no address is specified, this option selects the listing of all Internet and x.25 (HP-UX) network files.
Log File Rotation
This type of monitoring generates a lot of data, so put it in log file rotation, see Webmin > System > Log File Rotation (the one for /var/log/letsencrypt/*.log is a good example).
Use the default settings, except for:
Rotation schedule: Daily
Number of old logs to keep: 31, so you will always have at least a month's worth of data.
Compress old log files?: Yes.
Slow Query Monitoring for MySQL
MySQL has a slow query log which records all queries which took longer than 10 seconds (by default) to execute. For Moodle, 10 seconds is not realistic because many queries take longer than that, so 30 seconds is probably better.
To activate slow query logging:
Login using the mysql client: sudo mysql -uroot -p
set global slow_query_log = 'ON';
set global slow_query_log_file ='/var/log/mysql/slow-query.log';
set global long_query_time = 30;
Confirm the changes are active by re-entering the MySQL shell (this reloads the system variables) and running the following command: show variables like '%slow%';
Make sure the slow-query.log is in log rotation (see subsection Log File Rotation).
Incident Response
If you receive an alert from either monitoring system, take the following steps:
Verify the alert
If normal usage was impeded, i.e. there was an actual outage, notify your users, with an estimated time to fix if possible
Fix the issue
Take steps to prevent this from happening again (and document them in a relevant SOP)
If there was an outage, notify your users that the issue is now fixed and what you have done, or will do in the very short term, to prevent a recurrence of the incident.
Appendix – Health Monitoring on Servers Without Webmin
Purpose
This section describes how basic server health monitoring is implemented on systems where Webmin is not installed or not permitted.
Instead of relying on a web-based administration interface, monitoring is achieved using:
a lightweight Bash script
systemd timers
standard Unix tooling (mail, logrotate)
This approach minimizes attack surface, avoids additional services, and is fully auditable.
Rationale (Why No Webmin)
Webmin provides convenient monitoring and administration features but:
introduces an additional web-facing service
increases maintenance and patching requirements
is not always allowed under security policies
For these reasons, this server uses a script-based monitoring approach that:
requires no open ports
has no daemon processes
depends only on standard OS components
provides clear alerting and diagnostics
Monitoring Scope
The health check verifies the following:
Disk usage on the root filesystem (/)
System load (1-minute average, normalized per CPU core)
Available memory (MemAvailable)
Required services:
apache2
postgresql
Local HTTP availability via http://127.0.0.1/
On failure:
a diagnostics snapshot is appended to a log file
an alert email is sent
On success:
a single “OK” line is written to the log
no email is sent
Installation
Prerequisites
Ensure mail utilities are installed:
apt update
apt install mailutils
Postfix is already present on this system.
Script Installation
Create the monitoring script:
vim /usr/local/sbin/healthcheck.sh
Insert the full script source provided below.
Set permissions:
chmod 0755 /usr/local/sbin/healthcheck.sh
Create the state directory:
mkdir -p /var/lib/healthcheck
systemd Configuration
Create the service unit:
vim /etc/systemd/system/healthcheck.service
[Unit]
Description=Basic server health check
[Service]
Type=oneshot
ExecStart=/usr/local/sbin/healthcheck.sh
Create the timer unit:
vim /etc/systemd/system/healthcheck.timer
[Unit]
Description=Run healthcheck every 5 minutes
[Timer]
OnBootSec=2min
OnUnitActiveSec=5min
AccuracySec=30s
[Install]
WantedBy=timers.target
To prevent alert emails from being classified as spam or overlooked:
Create a mail filter or rule in the mail client:
Match subject containing:[ALERT][Totara][ubuntu]
Always deliver to inbox (or mark as important)
Optionally apply a label such as “Server Monitoring”
This ensures alerts remain visible while avoiding unnecessary inbox noise.
Script Source Code
/usr/local/sbin/healthcheck.sh
#!/usr/bin/env bash
set -euo pipefail
HOSTNAME_SHORT="$(hostname -s)"
HOSTNAME_FQDN="$(hostname -f 2>/dev/null || hostname)"
NOW="$(date -Is)"
# -----------------------------
# CONFIG (defaults, overridable via environment)
# -----------------------------
: "${ALERT_EMAIL:=onno@solin.co Lee@teaching4business.com nikki@teaching4business.com}"
: "${MAIL_FROM:=monitoring@solin.co}"
: "${DISK_MAX_PCT:=95}"
: "${LOAD_PER_CORE_MAX:=1.50}"
: "${MEM_AVAIL_MIN_MB:=512}"
: "${HTTP_URL:=http://127.0.0.1/}"
: "${ALERT_COOLDOWN_SECONDS:=1800}"
: "${STATE_DIR:=/var/lib/healthcheck}"
SERVICES=("apache2" "postgresql")
# -----------------------------
log_line() {
echo "[$NOW] $*" >> /var/log/healthcheck.log
}
send_alert() {
local subject="$1"
local body="$2"
printf "%s\n" "$body" | mail -a "From: ${MAIL_FROM}" -s "$subject" ${ALERT_EMAIL} || true
}
rate_limited() {
local key="$1"
local stamp="${STATE_DIR}/${key}.stamp"
local now
now="$(date +%s)"
mkdir -p "$STATE_DIR"
if [[ -f "$stamp" ]]; then
local last
last="$(cat "$stamp" || echo 0)"
now - last < ALERT_COOLDOWN_SECONDS && return 0
fi
echo "$now" > "$stamp"
return 1
}
fail() {
local key="$1"
local msg="$2"
log_line "FAIL ${HOSTNAME_SHORT}: ${msg}"
{
echo "----- failure snapshot ($NOW) -----"
uptime
echo
df -h
echo
free -m
echo
top -b -n1 | head -n 60
echo
ss -tulpn
echo
systemctl --failed
echo "----------------------------------"
} >> /var/log/healthcheck.log
rate_limited "$key" && exit 1
send_alert "[ALERT][Totara][${HOSTNAME_SHORT}] healthcheck failed: ${key}" \
"Time: $NOW
Host: ${HOSTNAME_FQDN}
Reason:
${msg}
See /var/log/healthcheck.log for diagnostics."
exit 1
}
touch /var/log/healthcheck.log
disk_pct="$(df -P / | awk 'NR==2{gsub("%","",$5); print $5}')"
[[ "$disk_pct" -lt "$DISK_MAX_PCT" ]] || fail disk "Disk usage ${disk_pct}%"
cores="$(nproc)"
load_1m="$(awk '{print $1}' /proc/loadavg)"
awk -v l="$load_1m" -v c="$cores" -v t="$LOAD_PER_CORE_MAX" 'BEGIN{ exit !l/c)<=t) }' \
|| fail load "Load ${load_1m} on ${cores} cores"
mem_avail_mb="$(awk '/MemAvailable/ {print int($2/1024)}' /proc/meminfo)"
[[ "$mem_avail_mb" -ge "$MEM_AVAIL_MIN_MB" ]] \
|| fail memory "MemAvailable ${mem_avail_mb}MB"
for svc in "${SERVICES[@]}"; do
systemctl is-active --quiet "$svc" \
|| fail "service-${svc}" "Service not active: ${svc}"
done
curl -fsS --max-time 10 "$HTTP_URL" >/dev/null \
|| fail http "Local HTTP check failed"
log_line "OK ${HOSTNAME_SHORT}"
exit 0
Active Directory SSO with SAML2 in Moodle requires installing an authentication plugin, exchanging metadata with your AD administrator, and careful mapping of claim types to user fields. This guide covers setup, testing against SAMLtest.id, and troubleshooting common claim mismatches.
Assumptions
This SOP makes the following assumptions:
The customer has an AD (Active Directory) based system.
They want their users to be able to access Moodle or Totara without logging in (or at the very least they should be able to use their ‘current’ username and password).
Their AD system supports SAML2
Make sure to check these assumptions with your users!
Terminology
Service Provider (SP): In our scenario, Moodle (or Totara) is the service provider – the application that provides the service the user wants to get access to.
Identity Provider (IdP): The customer’s system where the user is authenticated
Claims (or ClaimTypes): user attributes (properties of the user, i.e. information about the user)
Install SAML2 Plugin
Moodle does not support SAML2 out of the box (and neither does Totara). You have to install an authentication plugin: SAML2 Single sign on. If you don’t have access to the web server, try to install the plugin through the upload form for plugins: Site administration > Plugins > Install plugins. That should land you on this url: /admin/tool/installaddon/index.php.
Activate the SAML2 Authentication Plugin
Go to Site Administration > Plugins > Manage Authentication. You should end up on this page: /admin/settings.php?section=manageauths.
‘Enable’ the plugin by clicking on the eye icon (or something similar).
From this screen, you can also directly access the SAML2 plugin’s configuration settings by clicking on the Settings link.
Exchange Metadata with the Customer
Get IdP metadata xml or a public xml URL from the Customer
In order to configure the plugin, you need to exchange metadata with your users. Ask them for the IdP metadata xml or a public xml URL. This should be filled out here:
Sometimes your users will tell you what claims they’re providing (user attributes). In my experience, this information may or may not be accurate. Keep in mind that your users’s IdP metadata is authoritative. If a ClaimType isn’t mentioned there, it means it won’t be made available in Moodle either, through SAML2.
Please note that the reverse is sometimes also true: not all claim types that are mentioned in the IdP metadata are always automatically available. Apparently, the administrator of the IdP system has to ‘turn on’ the claim types or something like that.
Provide the Customer with the SP metadata
After you have done that, you should provide them with the SP (Service Provider) metadata, which can be obtained here:
Copy the url from the ‘View Service Provider Metadata’ link and give it to your users. They should know what to do with it, but just in case they ask you for it:
The Identifier (Entity ID) can be found in the attribute entityID of the SP Metadata.
For instance, in the following snippet, I have highlighted the entityID attribute:
N.B.: if you have trouble generating (accessing) the SP Metadata, wait till you have completed the remaining configuration (see the next section), then try again.
Configure the SAML2 Plugin
To configure the SAML2 plugin, take a close look at your users’s IdP metadata xml. What you need to extract from it, are the fields you need to map in Moodle.
These are the fields that you need at minimum: uid, email address, first name, and last name.
In my experience, these are typically called:
uid: uid, upn or objectidentifier (see subsection below)
But if you set ‘Simplify attributes’ to ‘Yes’ (the default), then you can use the much shorter names:
The defaults for the remaining configuration options are pretty sensible, although you might want to set ‘Auto create users’ to ‘Yes’, depending on whether the users already exist in Moodle or Totara. For testing purposes, set it to ‘Yes’.
Mapping IdP: uid, upn or objectidentifier
If you link accounts in two different systems (which is basically what SSO comes down to), then you need a way to uniquely identify a user in both systems. In SAML2, this typically done with one of the following attributes:
uid
upn (which stands for User Principal Name)
objectidentifier
The default in Moodle (and Totara) is to use uid, but this won’t work if the attribute (claimtype) is not actually present in your users’s IdP metadata xml.
So, find the appropriate attribute, and fill it in here, in Mapping IdP:
Attention: the claim types seem to be case sensitive. For instance, the DisplayName may contain capital letters which are missing in the actual ClaimType value. You should always use the actual ClaimType value.
Mapping to Custom Profile Fields
SAML2 fully supports mapping claim types to custom profile fields. In fact, if you have custom profile fields, they will automatically show up in the configuration screen of the plugin: /admin/settings.php?section=authsettingsaml2 (Site Administration > Plugins > Authentication > SAML2). The resulting field name will be something like: auth_saml2 | field_map_profile_field_isDocentTrainer.
Test the SAML2 Based SSO
All the SAML2 configuration is done and tested on staging. Ask your users for a test AD account, and try to login through SAML2.
If necessary, visit the /auth/saml2/test.php once you’ve got a saml2 based session going. You should see the details of the info the IdP is sending over.For instance, visiting https://www.contoso-learning.example/auth/saml2/test.php with my Microsoft test account (provided to me by the client’s IT staff) yields:
This is useful information, especially if the Claim Types the client is giving you are not actually working. The screen above shows the right names to use for mapping the IdP fields to Moodle fields.
From Staging to Production
Once the SAML2 configuration has been successfully tested, we have to move it to production. There are two procedures here: one for a new customer and one for an existing customer.
New Customer
In case of a new customer, after testing we then copy the staging environment to production. Because the domain for will be different, we have to provide your users with the new SP metadata url.
Existing Customer
For an existing customer, the procedure is:
Test the SAML2 configuration on staging.
Copy the settings from staging to production.
Provide your users with the SP metadata url for production.
Troubleshooting
Turn on the development and debugging mode. If you’ve reached a point where you can’t realistically test with your users’s AD system (e.g. because you’re not allowed to go in there), use this testing system:
Mapping IdP: uid – Which IdP attribute should be matched against a Moodle user field
Mapping Moodle: ID number (this is the mdl_user.idnumber attribute)
Data mapping (First name): givenName
Data mapping (Surname): sn
Data mapping (Email address): mail
Data mapping (ID number): uid
Please keep in mind that the IdP attribute will also be used to create the username. In the example above, the uid will be used as the username, and it will be used as the ID number.
You can test this by having a SAMLtest.id test user login, and then change their Moodle username afterwards – they will still be able to login through SAMLtest.id (because the SAMLtest.id login matches with the ID number, given the settings above).
In most scenarios, it’s superfluous to set the ‘Mapping Moodle’ to anything other than username, since the ‘Mapping IdP’ will be stored there anyway.
However, if you want to link existing Moodle accounts to external accounts, it may be easier to set ‘Mapping Moodle’ to idnumber. That way you only need to supply the external value of Mapping IdP to idnumber. You don’t need to change the existing Moodle username – which can be confusing to users.
‘Common’ Errors
You may have encountered this message on the login screen (/login/index.php):
Error: “Exception – Call to a member function export_for_template() on string”
We do not recommend this since it would be a core hack, but you could ask a developer to go to line 731 of /lib/authlib.php and comment out this part:
The reason behind this bug: when the IdP provides an icon, the variable $idp['icon'] is set to a string. But Moodle (or Totara) expects an instance of pix_icon there.
Troubleshooting
You can use an IdP service like https://idp.ssocircle.com/ to test saml2. But please keep in mind that you should only do that if you think the saml2 plugin is somehow not working properly (e.g. you just installed a new version of the plugin).
Moodle's backup and restore API is powerful but complex. This guide covers the required context setup, database operations, and workflows for programmatic backups and targeted course/section restores, including critical pitfalls around controller cleanup.
This guide aims to help developers programmatically interact with Moodle's robust backup and restore (B&R) system. While powerful, the API can be complex, and certain scenarios, like targeted section restores, require precise handling.
This guide is based on the Moodle 4.1 backup & restore API.
I. General Principles & Best Practices
Moodle Environment is Key:
Configuration: Always include Moodle's main config.php at the beginning of your script.
CLI Context: If the script is intended for command-line execution, define define('CLI_SCRIPT', true); before including config.php.
Required Libraries: Include essential Moodle libraries. Common ones for B&R include:
Most B&R operations require a valid Moodle user context, typically an administrator, to ensure appropriate permissions.
Fetch an admin user using get_admin():global$USER; // Ensure $USER is global if not already
$USER = get_admin();
Error Handling & Logging:
Exception Handling: Moodle's B&R API can throw various exceptions (e.g., moodle_exception, backup_exception, restore_controller_exception). Wrap B&R operations in try...catch blocks to handle potential errors gracefully.
Detailed Logging: Implement a robust logging mechanism to track the script's progress, key variable states, and any errors encountered. This is invaluable for debugging complex B&R workflows. For CLI scripts, cli_writeln() can be used for console output, supplemented by logging to a file.
Resource Management – The destroy() Method:
CRITICAL PITFALL:backup_controller and restore_controller objects, along with their associated backup_plan and restore_plan objects, create complex internal structures. They may hold database connections, temporary file references, and significant amounts of data in memory.
Best Practice:ALWAYS call the $controller->destroy() method after you are completely finished with the controller object. This is typically done in a finally block to ensure cleanup even if errors occur. The destroy() method on the controller will also handle the destruction of its associated plan.
Consequences of Neglect: Failure to call destroy() can lead to memory leaks, PHP timeouts, database connection exhaustion, and persistent temporary data (e.g., "existing temptables found" errors when Moodle attempts its own cleanup later).
Temporary Directories:
Moodle's Temp Space: Moodle B&R operations use a temporary directory, typically located at $CFG->dataroot . '/temp/backup/'.
Path Relativity (Restore): When providing a path to extracted backup contents for a restore_controller, this path must be relative to Moodle's main backup temporary directory (i.e., $CFG->dataroot/temp/backup/).
Creating Temp Dirs: Use Moodle's make_backup_temp_directory() function to create uniquely named subdirectories within this temporary space. This function returns an absolute path.
Cleanup:
Moodle's controller destroy() methods are responsible for cleaning up Moodle's internal temporary files and database records related to that specific B&R operation.
Your script is responsible for cleaning up any additional temporary directories or files it creates (e.g., a directory where you manually copy an MBZ file or extract its contents before passing a relative path to the restore_controller). Use fulldelete() for this.
Database Context (If Working with Multiple Databases):
If your script interacts with a separate database for sourcing backup data (e.g., a dedicated backup Moodle instance's database), ensure you manage the global $DB object context correctly.
Switch to the backup database connection before performing operations against it, and restore the original live Moodle database connection immediately afterward.global$DB;
$original_live_db_connection = $DB;
// Assume $backup_db_conn is a Moodle database object for your backup DB
$DB = $backup_db_conn;
// ... perform operations against the backup database ...
$DB = $original_live_db_connection; // Switch back to live Moodle DB
Moodle Version Awareness:
The B&R API, like other Moodle APIs, can evolve between Moodle versions. Code written for one version might require adjustments for another. Always test thoroughly against your target Moodle version.
Consult the Source Code:
When API behavior is unclear or documentation is sparse, the Moodle source code (primarily in backup/controller/, backup/moodle2/, and related task/plan files) is the definitive reference. PHPDoc comments within the code can also be very helpful.
Iterative Development & Debugging:
Programmatic B&R can be complex. Start with simpler operations and incrementally add complexity. Use systematic debugging techniques and the logging you've implemented.
These are the central orchestrators for backup and restore operations.
Instantiation Examples:
new backup_controller($type, $id, $format, $interactive, $mode, $userid)
new restore_controller($tempdir_relative_path, $courseid, $interactive, $mode, $userid, $target)
Common Parameters:
$type (Backup): The scope of the backup, e.g., backup::TYPE_1COURSE, backup::TYPE_1SECTION.
$id (Backup): The Moodle ID of the item to be backed up (e.g., course ID, section ID from the source database).
$format: The backup format, typically backup::FORMAT_MOODLE.
$interactive: For scripts, this should be backup::INTERACTIVE_NO.
$mode: The purpose/context of the operation, e.g., backup::MODE_GENERAL.
$userid: The Moodle user ID of the user performing the operation.
$tempdir_relative_path (Restore): The path to the directory containing the extracted backup contents, relative to $CFG->dataroot/temp/backup/.
$courseid (Restore): The Moodle ID of the target course for the restore operation.
$target (Restore): Specifies how the restore interacts with the target course, e.g., backup::TARGET_NEW_COURSE, backup::TARGET_CURRENT_ADDING.
Typical Lifecycle: Instantiate Controller -> Get Plan -> (Optionally Adjust Plan Settings/Tasks) -> (For Restore: Execute Prechecks) -> Execute Plan -> (Optionally Get Results) -> Destroy Controller.
Plans (backup_plan, restore_plan):
The detailed blueprint of the B&R operation, generated by its controller.
Contains:
Settings: High-level configuration options for the entire operation (e.g., whether to include users, logs, files). Accessed via $plan->get_setting('setting_name').
Tasks: An ordered list of individual operations that will be performed (e.g., restore course structure, restore a specific section's content, restore an activity module). Accessed via $plan->get_tasks().
The plan is obtained from the controller using $controller->get_plan().
Represent discrete units of work within a plan. Each task is responsible for a specific part of the B&R process.
Tasks can have their own settings, specific to that unit of work (e.g., whether to include user data for a particular activity).
Importance for Targeted Operations: For specific operations like restoring a single section into a particular place, understanding and correctly configuring the relevant task (e.g., restore_section_task) is crucial.
Settings (backup_setting):
Objects that represent configurable options within the B&R process.
They exist at both the overall plan level and within individual tasks.
Access settings using $plan->get_setting('name') or $task->get_setting('name').
Set their values using $setting_object->set_value(...).
Challenge: Discovering the exact name of a setting and determining whether it's a plan-level or task-level setting often requires inspecting Moodle source code (e.g., the define_settings() methods in task classes or how settings are added in plan builder classes).
Backup Types & Restore Targets (Constants):
These constants, defined in backup/util/defines.php (which is usually included via backup_includes.php), dictate the scope and destination of B&R operations.
Backup Types (Scope Examples):
backup::TYPE_1COURSE: Backs up an entire course.
backup::TYPE_1SECTION: Backs up a single section from a course.
backup::TYPE_1ACTIVITY: Backs up a single activity module.
Restore Targets (Destination Examples):
backup::TARGET_NEW_COURSE: Restores the backup into a brand new Moodle course.
backup::TARGET_CURRENT_ADDING: Adds content from the backup into the course specified by $courseid (passed to restore_controller), merging with any existing content.
backup::TARGET_CURRENT_DELETING: Deletes existing content from the target $courseid before restoring content from the backup.
(Others like TARGET_EXISTING_ADDING, TARGET_EXISTING_DELETING are for restoring into a different existing course, selected during UI workflows or configured programmatically).
Key Interaction: The interplay between the backup TYPE_... and the restore TARGET_... significantly influences how settings and tasks behave. For instance, restoring a TYPE_1SECTION backup using TARGET_CURRENT_ADDING requires careful targeting of the restore_section_task to place the content correctly.
III. Core Operations: Step-by-Step with Examples
A. Programmatically Backing Up a Single Section
This example demonstrates backing up a specific section from a Moodle course.
// Assumed variables:
// $source_db_connection: Moodle DB connection object for the source database (if different from live).
// $USER: An admin user object.
// $source_course_id: The ID of the course containing the section.
// $source_section_id: The ID of the section record to back up (from the source DB).
global $DB; // Live Moodle DB
$original_live_db = $DB;
$copied_mbz_path = null;
$backup_controller = null;
$script_controlled_mbz_temp_dir = null;
try {
// Switch DB context if source is different
if (isset($source_db_connection) && $source_db_connection !== $original_live_db) {
$DB = $source_db_connection;
cli_writeln("Switched DB context to source for backup.");
}
cli_writeln("Backing up section ID {$source_section_id} from course {$source_course_id}");
$backup_controller = new backup_controller(
backup::TYPE_1SECTION,
$source_section_id,
backup::FORMAT_MOODLE,
backup::INTERACTIVE_NO,
backup::MODE_GENERAL,
$USER->id
);
$backup_plan = $backup_controller->get_plan();
// Configure general plan settings (e.g., exclude user data)
$general_plan_settings = $backup_plan->get_settings();
if (isset($general_plan_settings['users']) && $general_plan_settings['users'] instanceof backup_setting) {
$general_plan_settings['users']->set_value(0);
}
// Add other general settings as needed (e.g., role_assignments, grade_histories to 0)
// Configure settings for the specific section task (e.g., ensure activities and files are included)
$backup_tasks = $backup_plan->get_tasks();
foreach ($backup_tasks as $task) {
if ($task instanceof backup_section_task && $task->get_sectionid() == $source_section_id) {
// Ensure activities and files within this section are backed up
if (method_exists($task, 'get_setting' { // Good practice to check
$activities_setting = $task->get_setting('activities');
if ($activities_setting instanceof backup_setting) {
$activities_setting->set_value(1); // 1 for include
}
$files_setting = $task->get_setting('files');
if ($files_setting instanceof backup_setting) {
$files_setting->set_value(1); // 1 for include
}
}
break; // Found the relevant section task
}
}
$backup_controller->execute_plan();
$backup_status = $backup_controller->get_status();
if ($backup_status != backup::STNorthwind UniversityS_FINISHED_OK && $backup_status != backup::STNorthwind UniversityS_FINISHED_WARN) {
throw new moodle_exception("Backup failed. Controller status: " . $backup_status);
}
$backup_results = $backup_controller->get_results();
$moodle_managed_backup_file = $backup_results['backup_destination'] ?? null;
if (!$moodle_managed_backup_file instanceof stored_file) {
throw new moodle_exception('Backup destination file (stored_file object) not found in backup results.');
}
// Copy the Moodle-managed backup file to a script-controlled temporary location
$script_controlled_mbz_temp_dir = make_backup_temp_directory('my_script_section_backup_' . $source_section_id . '_' . uniqid(;
$copied_mbz_path = $script_controlled_mbz_temp_dir . DIRECTORY_SEPARATOR . $moodle_managed_backup_file->get_filename();
if (!$moodle_managed_backup_file->copy_content_to($copied_mbz_path {
throw new moodle_exception('Failed to copy MBZ file to script-controlled location: ' . $copied_mbz_path);
}
cli_writeln("Section backup MBZ successfully created at: " . $copied_mbz_path);
} catch (Exception $e) {
cli_writeln("ERROR during backup operation: " . $e->getMessage(;
if ($e instanceof moodle_exception && !empty($e->debuginfo {
cli_writeln("Moodle Exception Debug Info: " . $e->debuginfo);
}
// Further error handling or re-throwing as appropriate
} finally {
// Restore original DB context if it was changed
if (isset($source_db_connection) && $source_db_connection !== $original_live_db && $DB !== $original_live_db) {
$DB = $original_live_db;
cli_writeln("Restored DB context to live Moodle DB.");
}
// Destroy the backup controller to clean up Moodle's internal temp files and DB records
if ($backup_controller instanceof backup_controller) {
try {
$backup_controller->destroy();
} catch (Exception $e_destroy) {
cli_writeln("Warning: Exception during backup_controller destroy: " . $e_destroy->getMessage(;
}
}
// The $copied_mbz_path is now available for the restore step.
// Cleanup of $script_controlled_mbz_temp_dir should happen after $copied_mbz_path is no longer needed.
}
// The $copied_mbz_path variable now holds the path to the generated .mbz file.
// Remember to clean up $script_controlled_mbz_temp_dir later using fulldelete().
B. Programmatically Restoring a Single Section into a Specific Location in an Existing Course
This scenario requires creating an empty placeholder section in the target course first, then instructing the restore process to populate this specific placeholder.
// Assumed variables:
// $target_live_course_id: ID of the live Moodle course to restore into.
// $target_insertion_section_number: The desired section *number* for the new content.
// $USER: An admin user object.
// $path_to_section_mbz: Absolute path to the .mbz file created in the backup step.
// $original_section_backup_details: (Optional) An object/array holding metadata (name, summary, visibility)
// from the original section in the backup, to apply after restore.
global $DB; // Live Moodle DB
$restore_controller = null;
$script_controlled_extraction_dir = null; // For script-managed extraction path (absolute)
$placeholder_section_id_in_live_course = null;
try {
// 1. Create a placeholder section in the live target course
cli_writeln("Creating placeholder section at position {$target_insertion_section_number} in course {$target_live_course_id}...");
$placeholder_section_record = course_create_section($target_live_course_id, $target_insertion_section_number);
if (!$placeholder_section_record || !isset($placeholder_section_record->id {
throw new moodle_exception("Failed to create placeholder section in target course.");
}
$placeholder_section_id_in_live_course = $placeholder_section_record->id;
cli_writeln("Placeholder section created in live course: ID {$placeholder_section_id_in_live_course}, Number {$placeholder_section_record->section}");
// 2. Extract the .mbz file to a Moodle temporary directory (for restore_controller)
// The path passed to restore_controller must be relative to $CFG->dataroot/temp/backup/
$extraction_dir_relative_name = 'my_script_restore_extraction_' . uniqid();
$script_controlled_extraction_dir = make_backup_temp_directory($extraction_dir_relative_name); // Gets absolute path
if (!$script_controlled_extraction_dir) {
throw new moodle_exception("Could not create script-controlled extraction directory.");
}
$packer = get_file_packer('application/vnd.moodle.backup'); // Standard Moodle MBZ packer
if (!$packer) {
throw new moodle_exception('Could not get file packer for MBZ.');
}
if (!$packer->extract_to_pathname($path_to_section_mbz, $script_controlled_extraction_dir {
$packer_errors = $packer->get_errors();
throw new moodle_exception('Failed to extract .mbz file: ' . implode('; ', $packer_errors;
}
cli_writeln("MBZ successfully extracted to: " . $script_controlled_extraction_dir . " (Relative path for controller: " . $extraction_dir_relative_name . ")");
// Ensure moodle_backup.xml exists in the extraction
if (!file_exists($script_controlled_extraction_dir . DIRECTORY_SEPARATOR . 'moodle_backup.xml' {
throw new moodle_exception('moodle_backup.xml not found in extraction directory.');
}
// 3. Instantiate the restore_controller
$restore_controller = new restore_controller(
$extraction_dir_relative_name, // Relative path to extracted content
$target_live_course_id, // Target course ID
backup::INTERACTIVE_NO,
backup::MODE_GENERAL,
$USER->id,
backup::TARGET_CURRENT_ADDING // Adding content to the current (target) course
);
// 4. (Optional but recommended) Execute prechecks
if ($restore_controller->get_status() == backup::STNorthwind UniversityS_NEED_PRECHECK) {
cli_writeln("Executing restore prechecks...");
if (!$restore_controller->execute_precheck( {
// Handle precheck failures/warnings from $restore_controller->get_precheck_results()
// For example, log them and decide whether to proceed.
$precheck_results = $restore_controller->get_precheck_results();
cli_writeln("Restore prechecks indicated issues: " . var_export($precheck_results, true;
// Depending on severity, you might throw an exception here.
}
cli_writeln("Restore prechecks completed. Status: " . $restore_controller->get_status(;
}
// 5. Get the restore plan
$restore_plan = $restore_controller->get_plan();
if (!$restore_plan) {
throw new moodle_exception("Failed to retrieve restore plan from controller.");
}
// 6. CRITICAL: Configure the restore_section_task to target the placeholder
// This is essential for TYPE_1SECTION restores into a specific slot.
if ($restore_controller->get_type() == backup::TYPE_1SECTION) {
$restore_tasks = $restore_plan->get_tasks();
$section_task_configured_for_target = false;
foreach ($restore_tasks as $task) {
if ($task instanceof restore_section_task) {
// This task will process the single section from the backup.
// We instruct it to use our placeholder section's ID as its target in the live course.
cli_writeln("Configuring restore_section_task (Task Name: {$task->get_name()}) to target live section ID: {$placeholder_section_id_in_live_course}");
$task->set_sectionid($placeholder_section_id_in_live_course);
$section_task_configured_for_target = true;
break; // Assuming only one restore_section_task for a TYPE_1SECTION backup
}
}
if (!$section_task_configured_for_target) {
throw new moodle_exception("Could not find or configure the restore_section_task within the plan for TYPE_1SECTION restore.");
}
} else {
// This script's specific targeting logic is primarily for TYPE_1SECTION.
// Handling other backup types (e.g., TYPE_1COURSE) would require different plan/task configuration.
cli_writeln("WARNING: The backup type is not TYPE_1SECTION (actual: {$restore_controller->get_type()}). The specific section targeting logic used here may not apply or work as expected.");
}
// 7. Adjust other general plan settings (e.g., exclude user data, roles)
$general_restore_plan_settings = $restore_plan->get_settings();
if (is_array($general_restore_plan_settings {
foreach ($general_restore_plan_settings as $setting) {
if ($setting instanceof backup_setting) {
if (in_array($setting->get_name(), ['users', 'role_assignments', 'grade_histories', 'course_module_completion'] {
$setting->set_value(0); // 0 for exclude
}
}
}
}
// 8. Execute the restore plan
if ($restore_controller->get_status() != backup::STNorthwind UniversityS_AWAITING) {
throw new moodle_exception("Restore controller is not in AWAITING state before execute_plan. Current status: " . $restore_controller->get_status(;
}
cli_writeln("Executing restore plan...");
$restore_controller->execute_plan();
$restore_status = $restore_controller->get_status();
if ($restore_status != backup::STNorthwind UniversityS_FINISHED_OK && $restore_status != backup::STNorthwind UniversityS_FINISHED_WARN) {
throw new moodle_exception("Restore plan execution failed. Controller status: " . $restore_status);
}
cli_writeln("Restore plan execution finished with status: " . $restore_status);
// 9. (Optional but recommended) Update metadata of the now-populated section
// The restore_section_structure_step populates the section. This step ensures
// metadata like name, summary, visibility exactly matches the source backup object.
$final_live_section_obj = $DB->get_record('course_sections', ['id' => $placeholder_section_id_in_live_course]);
if ($final_live_section_obj && isset($original_section_backup_details {
$update_params = new stdClass();
$update_params->id = $final_live_section_obj->id;
$update_params->name = $original_section_backup_details->name;
$update_params->summary = $original_section_backup_details->summary;
$update_params->summaryformat = $original_section_backup_details->summaryformat;
$update_params->visible = $original_section_backup_details->visible;
$update_params->availability = $original_section_backup_details->availability; // If applicable
$DB->update_record('course_sections', $update_params);
cli_writeln("Metadata (name, summary, etc.) updated for restored section ID {$final_live_section_obj->id}.");
} else if (!$final_live_section_obj) {
cli_writeln("WARNING: Could not re-fetch section ID {$placeholder_section_id_in_live_course} after restore to update metadata. It might not have been correctly populated.");
}
cli_writeln("Section content successfully restored into live section ID {$placeholder_section_id_in_live_course}.");
// 10. Rebuild the course cache for the target course
cli_writeln("Rebuilding course cache for course ID {$target_live_course_id}...");
rebuild_course_cache($target_live_course_id, true);
cli_writeln("Course cache rebuilt.");
} catch (Exception $e) {
cli_writeln("ERROR during restore operation: " . $e->getMessage(;
if ($e instanceof moodle_exception && !empty($e->debuginfo {
cli_writeln("Moodle Exception Debug Info: " . $e->debuginfo);
}
// Further error handling
} finally {
// Destroy the restore controller
if ($restore_controller instanceof restore_controller) {
try {
$restore_controller->destroy();
} catch (Exception $e_destroy) {
cli_writeln("Warning: Exception during restore_controller destroy: " . $e_destroy->getMessage(;
}
}
// Clean up the script-controlled extraction directory
if ($script_controlled_extraction_dir && is_dir($script_controlled_extraction_dir {
fulldelete($script_controlled_extraction_dir);
cli_writeln("Cleaned up script-controlled extraction directory: " . $script_controlled_extraction_dir);
}
// The original .mbz file ($path_to_section_mbz) might also need cleanup if it was temporary.
}
IV. Stumbling Blocks & Pitfalls (Summary)
Targeting TYPE_1SECTION Restores into Existing Courses:
The Challenge: Ensuring the single section's content from the backup is placed into a specific, pre-determined slot (placeholder section) within the target course.
Solution: Create an empty placeholder section using course_create_section(). Then, during the restore process, find the restore_section_task in the restore_plan and call its set_sectionid() method, passing the ID of your placeholder section. This directs the task to populate that specific section record.
Distinguishing Plan-Level vs. Task-Level Settings:
Settings can apply globally to the plan or be specific to individual tasks. Knowing where a setting resides is crucial for configuring it correctly.
Identifying Correct Setting Names:
Setting names are not always obvious. Inspecting define_settings() methods in task classes or related Moodle source code is often necessary.
Essential destroy() Calls:
Always call $controller->destroy() to prevent resource leaks and cleanup issues.
Correct Paths for restore_controller:
The $tempdir parameter for new restore_controller(...) must be the path to the directory containing the extracted backup contents, relative to $CFG->dataroot/temp/backup/. It is not the path to the .mbz file itself.
V. Debugging Tips
Comprehensive Logging: Log controller statuses, plan details (settings and tasks), key variable values, and any error messages.
Inspect Objects: Use var_dump() or print_r() on controller, plan, task, and setting objects to understand their structure and current state (e.g., $plan->get_settings(), $task->get_settings()).
Moodle Developer Debugging: Enable developer-level debugging in Moodle's site administration to get more verbose error messages and stack traces from Moodle itself.
Server and Moodle Error Logs: Check the web server's error log and Moodle's configured error log file for PHP errors or other Moodle-specific error details.
Incremental Testing: Start with the simplest possible B&R operation and gradually add complexity to isolate where issues arise.
backup/util/defines.php: Refer to this file (usually included via backup_includes.php) for the definitions of all backup:: constants.
Examine moodle_backup.xml: This file, found within an extracted MBZ, describes the structure and contents of the backup. Reviewing it can provide insights into relevant settings and tasks.
OAuth2 in Moodle simplifies SSO by delegating authentication to an external identity provider. This guide covers issuer configuration, custom provider setup, and working around Totara's email verification requirement for non-standard OAuth2 services.
Issues
OAuth2 is very hard to troubleshoot, but basically you should only need the 3 end points mentioned above. If your users delivers you a couple of urls which do not work, see if they point to pages which themselves contain urls. Try these urls instead. We’ve had an example of a customer who sent us a kind of metadata url which contained the actual urls to use.
Issue with Email Verification in OAuth2 for Custom Services in Totara
Description of the Issue
When configuring OAuth2-based SSO in Totara, administrators may encounter an issue where the system enforces email verification for custom OAuth2 providers. Unlike predefined providers such as Google, Microsoft, and Facebook, custom services do not offer the option to disable the "Require email verification" setting in the user interface. This behavior results in user accounts being marked as "pending email confirmation," preventing successful logins.
Observations
This restriction does not apply to predefined OAuth2 services, where the "Require email verification" setting can be toggled.
In Moodle 4.5, this limitation has been addressed, allowing custom OAuth2 providers to disable email verification.
The issue stems from a default database configuration that requires email verification for custom services.
Example Scenario
Upon authentication via a custom OAuth2 provider:
A record is created in the oauth2_linked_login table with confirmed set to 0.
Users remain unable to log in until email verification is completed.
Suggested Remedy by Totara
Totara HQ has provided an unsupported workaround involving a direct database query. The query modifies the oauth2_issuer table to disable the email verification requirement for a specific OAuth2 service:
UPDATE [prefix]_oauth2_issuer
SET requireconfirmation = 0
WHERE name = "name_of_issuer_here";
Important Notes:
Replace [prefix] with the database prefix used in the Totara installation (e.g., ttr or mdl).
Ensure that the name_of_issuer_here matches the exact name of the custom OAuth2 service.
Risks and Limitations
Totara does not support this approach as it bypasses a core security measure.
Directly modifying the database introduces a risk of unintended consequences and may compromise system security.
Any issues arising from this change will not be supported by Totara HQ.
Recommendations
Evaluate whether disabling email verification is essential for the specific use case.
If email verification must be disabled, proceed with the query cautiously, ensuring a backup of the database before execution.
Report the requirement to Totara HQ to encourage future support for this feature in the user interface.
By documenting this issue and its resolution, administrators can better understand the limitations of OAuth2 custom services in Totara and make informed decisions about implementing workarounds.
Diagnosing Moodle problems systematically requires checking error logs, database integrity, cron execution, and plugin conflicts. This guide walks through the troubleshooting workflow and specific checks for common failure modes.
List the Symptoms
Get access to Moodle and server
Ask your users for:
Moodle url
Moodle admin username
Moodle admin password
Web server url or ip address
Web server username
Web server password
If the Moodle website is hosted on one of our own servers, we should already have this information – don’t bother your users in that case, but ask the lead engineer instead.
Find the Root Cause
Checking and Enabling X-Sendfile for Large File Delivery (Moodle)
Overview
When Moodle serves large files (video, H5P assets, SCORM, large images), the default behaviour is for PHP-FPM to stream the file through pluginfile.php. This can block PHP workers for minutes and cause sitewide slowdown. Apache’s mod_xsendfile allows Apache to serve files directly from moodledata, bypassing PHP entirely. This section describes how to enable X-Sendfile and how to test whether it is working.
Symptoms
Slow or hanging requests to pluginfile.php.
PHP-FPM slow logs show stack traces in byteserving_send_file() or readfile_accel().
Many long-running PHP-FPM workers during video playback.
Users report Moodle becoming unresponsive while accessing large videos or files.
Step 1: Verify Apache has mod_xsendfile
On the server, run:
a2enmod xsendfile
If the module is missing, install it:
apt update
apt install libapache2-mod-xsendfile
a2enmod xsendfile
service apache2 restart
Step 2: Add required directives to the vhost
Inside the VirtualHost:
<IfModule mod_xsendfile.c>
XSendFile on
XSendFilePath /path/to/moodledata
XSendFilePath /path/to/moodle/code
</IfModule>
Php-fpm may cause trouble if the number of pm.max_children is set too low, for a specific vhost (this is configured through a ‘pool’ file). You’ll see this mentioned in the log, /var/log/php7.4-fpm.log, e.g.:
[30-Sep-2021 10:08:11] WARNING: [pool www] server reached pm.max_children setting (5), consider raising it
Or:
26-Oct-2023 11:50:13] WARNING: [pool 1692438784256193] server reached pm.max_children setting (16), consider raising it
On Ubuntu (HostEurope), the config file for the web user (typically www-data) is here:
/etc/php/X.Y/fpm/pool.d/www.conf. The other pool files are in the same directory. For instance:
Here, 1692438784256193.conf is the php8.1-fpm configuration file for the user elo. How do we know? Well, virtualmin creates an entry in the vhost conf file, e.g. /etc/apache2/sites-available/portal.blueyonder-coaching.example.conf:
As you can see, the socket number matches with the number used in the conf file name.
In addition, we find the user elo in the output of the php8.1-fpm configuration command:
php-fpm8.1 -tt
Which yields (for this example):
[1692438784256193]
prefix = undefined
user = elo
group = elo
listen = /var/php-fpm/1692438784256193.sock
Again, the numbers match and here the socket is also explicitly mentioned.
So, in the php-fpm conf file that you have found, change the setting for pm.max_children, e.g. from 5 to 25 (this is really just an example, see the spreadsheet below to compute the actual value), and do:
command outputs free and available memory. Use the available memory in your calculations.
The difference between free memory vs. available memory in Linux is, free memory is not in use and sits there doing nothing. While available memory is used memory that includes but is not limited to caches and buffers, that can be freed without the performance penalty of using swap space.
Use something like pstree -c -H 19741 -S 19741 to see the current number of php-fpm7.4 processes, where you find the pid by looking for php-fpm: master process (/etc/php/7.4/fpm/php-fpm.conf) in the output of ps -ef. This tells you how many ‘children’ are currently spun up.
Compute php-fpm process size:
python /usr/local/bin/ps_mem.py | grep php-fpm (grab the Python script here).
A final thing to keep in mind is ‘if these "tuned" values are calculated based on the maximum capacity of yyour server and you put the same values to every site, you'll consume your resources multiple times. Instead, these values should be distributed between the pools so that the sum from all pools is equivalent with the "tuned" value.’
(..) ‘Whatever you do, keep your sites in separate pools with separate user accounts. Otherwise a compromise on a single site can spread across all your sites.’
The latest versions of Virtualmin (writing this 20231026) seem to automatically create a pool file for each new vhost. These articles describes how to do it manually:
From the latter article: “To complete the process, you should repeat the steps for each of your virtual hosts. When you are entirely sure mod_php is not being used anymore you can disable it through
$ sudo a2dismod php8.1
Until you've done this, Apache will still include a PHP process for every request, meaning the memory usage will stay the same and possibly be even higher.”
Checking php-fpm Configuration
Simply use this command to see the configuration parameters that are currently being used:
php-fpm8.1 -tt
And use the following command to test whether the configuration is correct:
php-fpm8.1 -t (or php-fpm8.1 --test)
As an aside, you can also check the configuration for Apache:
apachectl configtest
Checking the php-fpm status
Go to /etc/php/X.Y/fpm/pool.d/www.conf (e.g. /etc/php/7.4/fpm/pool.d/www.conf) and look for pm.status_path. Uncomment it and add /phpXY-fpm/status, e.g.:
pm.status_path = /php74-fpm/status
(Do NOT insert a dot here)
Then, in your /etc/apache2/apache2.conf file, add:
Obviously, the paths must match exactly. Restart Apache and php-fpm, and go to your web server’s location, e.g.:
http://213.165.72.180/php81-fpm/status
This should give you something like:
pool: www
process manager: dynamic
start time: 26/Oct/2023:13:37:45 +0000
start since: 13
accepted conn: 1
listen queue: 0
max listen queue: 0
listen queue len: 0
idle processes: 31
active processes: 1
total processes: 32
max active processes: 1
max children reached: 0
slow requests: 0
Please note: this only shows the status for that specific pool. In the example above, that’s www.
Sluggish Moodle Or Totara Site
Check the images that themes like Adaptle allow you to upload for the frontpage, header and background. We once had a Moodle site that was loading two essentially the same images of 7 MB each. Needless to say, this makes loading the site quite sluggish.
The scheduled tasks may add up, especially if you have many users and many automatic cohort syncs (audiences, in Totara). Please note that Totara (and Moodle prior to version 3.7) does not have the task_scheduled table. No serious logging takes place for scheduled tasks.
MySQL Connection Timeout Or Lost Connection
From your-vps server, /etc/mysql/mysql.conf.d/mysqld.cnf:
## TEMP 20230322 the lead engineer - successfully imported a 2.8 GB dump with these settings - main thing is innodb_buffer_pool_size which must be SMALLER
## See https://dba.stackexchange.com/questions/124964/error-2013-hy000-lost-connection-to-mysql-server-during-query-while-load-of-my
#innodb_lock_wait_timeout = 60
#net_read_timeout = 28800
#net_write_timeout = 28800
#connect_timeout = 28800
#wait_timeout = 28800
#delayed_insert_timeout = 28800
#innodb_buffer_pool_size = 4294967296
Timeout Issues
mod_fcgid: read data timeout in 40 seconds
This error may pop up if you’re using fcgi and you’re trying to upload a large scorm file (say 400M) that requires quite some processing time:
mod_fcgid: read data timeout in 40 seconds
(This error may be disguised as a HTTP 500 error, but the Apache log file should contain the actual error message.)
The solution is to increase the value of FcgidIOTimeout in your vhost configuration, e.g.:
FcgidIOTimeout 600
This directive can be put directly inside the VirtualHost part of the Apache configuration file for the website, e.g.:
<VirtualHost 11.22.33.44:443>
FcgidIOTimeout 600
Don’t forget to save the conf file and restart the webserver afterwards, with /etc/init.d/apache2 graceful.
Varnish Timeout
Usually it’s enough to increase max_execution_time in php.ini to solve any timeout issues. However, I recently (20250206) encountered a timeout issue in an AWS EC2 instance that turned out to be using Varnish, which is a caching tool.Varnish was set (in /etc/varnish/.default.vcl) to 30s. After resetting set bereq.first_byte_timeout to 300s (notice the ‘s’ by the way: Varnish completely crashes the site if you leave it out), I had to restart varnish:sudo systemctl restart varnish
And that solved the issue (together with the max_execution_time increase, of course).
Common But Hard to Spot Issues
Spaces in PHP Files
If you inadvertently introduce a space before the <?php opening tag, especially in config.php, things will go wrong. This will pollute the output buffer in cases where output is created and sent to the browser through php, e.g. pluginfile.php or theme/yui_combo.php. Examples are images, css, and javascript. If any output, such as a space, is sent before the headers, the results are catastrophic, especially for all binary files such as files (often, css and javascript is zipped before being sent to the browser, making these resources binary too).
Symptoms: images do not load, no layout (css) is applied and menus (requiring javascript) do not work.
Configuration Issues
Unexpected Name Showing as Sender in Course Welcome Emails
We encountered a case where Moodle’s course welcome emails were being sent with the name of an unrelated user as the sender. This was confusing, since the person whose name appeared had nothing to do with the specific course.
Root cause By default, the setting Send course welcome message – From in the Manual enrolments plugin (/admin/settings.php?section=enrolsettingsmanual) was configured as From the course contact. Moodle interprets “course contact” as the first user it finds with a role listed in the coursecontact configuration (commonly the editingteacher role). Importantly, this lookup also includes role assignments at the system or category level, not only at the course level. Because a user had the editingteacher role at system level, their name was used as the sender in welcome emails across the site.
Resolution We changed the configuration in: Site administration Plugins Enrolments Manual enrolments
Setting: Send course welcome message – FromFrom the no-reply address
This ensures that all course welcome emails are now sent from the neutral noreply address (e.g. noreply@example.com), and no personal user names appear unexpectedly as the sender.
Best practice Always configure Send course welcome message – From to From the no-reply address unless there is a strong reason to display the course contact’s name. This avoids confusion and ensures consistency across all courses.
Case Study: Extremely Slow Course Creation (ICM 2025)
Symptoms
Creating a new course in Moodle 4.1 took minutes to complete, whereas on the previous host it completed in under 30 seconds. Logging in via SAML2 also appeared slower.
Root Cause
The issue was caused by inefficient MySQL write performance, specifically InnoDB’s default behavior of flushing transaction logs to disk after every single write operation. This configuration is safe but can be extremely slow on systems handling many small transactions (like Moodle’s course creation process).
In this case, the new hosting environment used SSD storage but was still using conservative, HDD-era MySQL defaults:
innodb_flush_log_at_trx_commit = 1
innodb_io_capacity = 200
innodb_io_capacity_max = 2000
This caused excessive fsync operations (waiting for every commit to fully write to disk).
Resolution
The hosting partner (Kaliber) optimized MySQL for modern SSD storage and Moodle’s workload. The following key changes were applied:
Increasing innodb_io_capacity allows MySQL to perform more I/O operations per second, aligning with SSD capabilities.
Setting innodb_flush_log_at_trx_commit = 2 reduces log flushes to once per second instead of every transaction — a safe compromise that greatly reduces latency.
Increasing I/O threads improves concurrency on SSDs with multiple queues.
Testing on the development server confirmed that even the most aggressive options (O_DIRECT_NO_FSYNC) only gave minimal additional gain; the 1 fsync per second configuration was the “golden mean.”
After implementing these settings, the “Create new course” operation returned to expected performance levels (seconds rather than minutes).
Lessons Learned / Best Practice
Moodle’s “create course” is a write-heavy operation; database write latency dominates performance.
Always review and tune MySQL’s InnoDB write path after migrations.
On SSDs, conservative defaults such as innodb_flush_log_at_trx_commit=1 and low I/O capacity settings can cause severe slowdowns.
This issue occurred on the ICM production system (2025) after migrating Moodle 4.1 to a new host. It was resolved by Bart (Kaliber) following performance analysis and parameter tuning.
Moodle upgrades involve sequencing: checking compatibility, updating code before running the database wizard, testing customizations, and validating the result. This guide covers the full workflow for both major and minor upgrades using Git.
Preamble: Minor vs. Major Upgrades
What is the difference between a minor upgrade and a major upgrade? A minor upgrade is usually a security patch which can be deployed by overwriting the core code base. By contrast, a major upgrade typically also requires you to upgrade parts of the LAMP stack Moodle is running on, as well as the theme (in case you have customized the theme for the client).
The following is mostly about performing a major upgrade. We will include a separate section on executing a minor upgrade at the end of the guide. If you just need to perform a minor upgrade, you can skip to that section right away.
Familiarize yourself with the Moodle upgrade procedure
Before you get started, check out the Upgrading page on Moodle’s website.
Upgrading in a nutshell
Backup everything beforehand, specifically:
The database
The moodledata directory
The entire code base (can usually be found in public_html, htdocs, or a subdirectory called moodle)
Replace the code base with the new version
Put the config.php file of the old version in the code base of the new version
Add the new versions of the 3rd party plugins
Get access to Moodle and server
Ask your users for:
Moodle url
Moodle admin username
Moodle admin password
Web server url or ip address
Web server username
Web server password
If the Moodle website is hosted on one of our own servers, we should already have this information – don’t bother your users in that case, but ask the lead engineer instead.
Check if server meets requirements
As stated in the moodle.org section on upgrading, make sure that your users’s server meets the minimum requirements for the new Moodle version. The most important software to check is:
MySQL (or whatever database they’re currently running Moodle on)
In the case of upgrades from 2.9 to 3.1 or later, check whether the MySQL / MariaDB database can be converted from Antelope to Barracuda and the new character set & collation for.
PHP – including the required extensions
If the server does not meet the requirements, make sure to check with the sysop whether the system can be updated. Do not perform the actual system updates yet, because that may disrupt the current Moodle version.
Installing a Separate PHP Version Using PHP-FPM under Ubuntu
If you are the sysop, or devop, you can also install the correct version of PHP yourself. (See also this site.)
Now add a directive to the vhost configuration file of the Moodle website, right before the ‘Directory’ part, e.g:
<FilesMatch \.php> # Apache 2.4.10+ can proxy to unix socket
SetHandler "proxy:unix:/var/run/php/php7.4-fpm.sock|fcgi://localhost/"
</FilesMatch>
<Directory /home/ocwdemo/public_html>
In some cases, it may be necessary to activivate the proxy_fcgi mod at this point:
sudo a2enmod proxy_fcgi proxy
Also don’t forget to restart the webserver gracefully:
/etc/init.d/apache2 graceful
And if you're upgrading to a new php-fpm version, check the php.ini for that version, e.g.:
/etc/php/7.3/fpm/php.ini
And restart after changing any values (like max_post_size etc.) there:
/etc/init.d/php7.3-fpm restart
Please note: php-fpm also requires performance tweaking. This is outside the scope of this guide.
Upgrade Cron Job
If the php version has changed and you’re using multiple php versions on the same server, make sure that the command for the cron job execution is updated too:
E.g. if your php version was 7.0 for the previous version of Moodle, and your cron command is this:
For most vhosts, we have a custom backup script running: Backupvhost Script. This script relies on the php version Moodle is using, so if the php version has changed for the Moodle upgrade, make sure to call the backup script with the new php version too. In addition, the location of the config.php may change from one version of Totara to another.
Take inventory of all ‘Additional’ (i.e. 3rd party) plugins
Visit /admin/plugins.php and note all ‘Additional’ plugins. These are 3rd party plugins for which you need to get the new source code separately, usually through https://moodle.org/plugins/.
Here’s an example:
(Since your users’s Moodle version was very old, in this example, we had to perform two subsequent upgrades: first to 3.1 and then to 3.4.)
Pay special attention to the theme
The theme is usually the only part that’s really been customized. In many cases, the theme will not survive any major upgrades, unless it’s a standard Moodle theme that has only had some configuration for an uploaded logo, some color customizations, and maybe a favicon added.
If the site looks very bad after the upgrade, it may be necessary to ask a Moodle theme developer to fix it. This must go through sales first (ask the lead engineer).
Look for any non-Moodle content which is stored in the Moodle directory
Anything that is not part of the Moodle code base (including plugins), such as external images, Flash files, movies, etc., should obviously not be stored in the Moodle directory (the wwwroot folder). But customers do this nonetheless. So be sure to copy that content and put it back in place after an upgrade.
Copy All Additional Plugins
For testing & development it may be convenient to simply copy over all ‘additional’ plugins from the old code base to the new. Here’s a script to do this:
#!/bin/bash
clean="/home/engineer/temp/moodle41-clean"
modified="/home/engineer/temp/moodle41-modified"
target="/home/engineer/php/example-site/public_html"
# Find directories in modified that are missing in clean, excluding hidden directories
find "$modified" -mindepth 1 -type d ! -path "*/.*" | while read dir; do
relative_path="${dir#$modified/}" # Get relative path
if [ ! -d "$clean/$relative_path" ]; then
echo "Copying: $relative_path"
rsync -av "$dir" "$target/${relative_path%/*}/"
fi
done
Check if there are any core hacks
It’s very unusual to find any core hacks on a customer’s Moodle website. Just ask them and if there’s any doubt, diff against the original code base for the exact same version, e.g.:
diff -qr moodle1/ moodle2
This compares two directories recursively (-r) and only reports the differences (-q). For example:
diff -qr multitenant-argononsand.20240213/ multitenant-v17test.20240213/
Files multitenant-argononsand.20240213/multitenant/lib.php and multitenant-v17test.20240213/multitenant/lib.php differ
Or, ignoring the .git directory:diff -qr --exclude='.git' moodle1/ moodle2
How to Find The Exact Moodle Build
If you want to compare the client’s Moodle code base with the ‘official’ code base, you need the exact same build. Here’s how to find it.
Switch to the stable branch and update it
In your local clone of the ‘official’ Moodle code base:
This ensures your local MOODLE_401_STABLE branch contains every version bump and bugfix up to the latest commit.
Search for the version bump in version.php
git log -S "20250131" -- version.php
The -S "20250131" flag looks for commits that add or remove the exact string 20250131 in version.php. The top result is the commit where Moodle’s maintainers changed:$release = '4.1.15+ (Build: 20250131)';
$version = 2022112815.06;
Check out that specific commit
Suppose the git log output shows a commit hash of 59a1f3e4a2b8c5d7e8f9a0123b4c5d6e7f8a9b0c. You can then do:git checkout 59a1f3e4a2b8c5d7e8f9a0123b4c5d6e7f8a9b0c
Verify that version.php matches the desired buildgrep '\$version\|\$release' version.php
You should see exactly:$version = 2022112815.06;
$release = '4.1.15+ (Build: 20250131)';
At this point, your working copy is pinned to the precise Moodle build (4.1.15+ (Build: 20250131)) that matches the client’s code base.
Perform a test upgrade for your users to approve
Put the test upgrade on a website under our domain. For instance, if your users’s Moodle site is https://www.contoso-learning.example, create a domain contoso-learning.staging.example.
Then you need to create the actual virtual host for the test website. Essentially this comes down to Installing a New Moodle Website on a VPS.
But instead of creating a new Moodle website, upgrade your users’s.
Set Right Expectations for Test Version
In previous upgrade projects, we’ve encountered customers who mistakenly believed that the configuration changes they made to the test-version would be retained in the live-upgrade.
To prevent this, inform your users that any changes they make to the test-version will not be transferred to the live version of the upgrade.
Also, advise customers to test the customizations primarily. There is no real reason to test Moodle core or any of the much used 3rd party plugins since these will have been tested by the larger user community.
Get The Upgraded Code In The Repository
We deploy the upgrade using a Git branch, let’s call it upgrade. What’s more, the upgrade branch is going to be based on the upstream Moodle repository. In your local repo, do the following. Add the official Moodle repo as the upstream repo:
Get a specific branch. Here, we’re pulling the branch containing Moodle 4.1 (the upstream branch name can be found in the upstream repo through git remote show upstream):
git fetch upstream MOODLE_401_STABLE
Create a local branch based on the upstream branch:
Optionally, if you want to store the 4.1 version in your remote repo:
git push -u origin moodle41
Create the ‘upgrade’ branch and check it out:
git checkout -b upgrade moodle41
Finish by adding, committing and pushing the new code:
git commit -a -m "Adding code for Moodle 4.1"
git push --set-upstream origin upgrade
Upgrading the website
Put the following items on the test website:
The original database
The original moodledata directory
Checkout the git branch storing the source code for the upgrade (i.e. the new Moodle version)
Add the new versions of the 3rd party plugins
Copy the config-dist.php file from the source code to config.php and make the adjustments for the proper directory references and database access details.
Check if there are any additional language packs that need upgrading
Go to the /admin/ url of the test website to trigger the actual upgrade process. Take a good look at the information on the ‘Current release information’ page that comes up just before you start the upgrade. Do not click on continue.
Are the Server Checks and Other Checks okay? (You can ignore any warnings about the site not being https if you’re on a test site).
Pay attention to collation issues: It is not enough to just change the collation in config.php! You should always run the following cli script if there are any collation issues:
(Check Moodle documentation linked to in the /admin page.)
Once everything is ready for the actual upgrade, go to the command line interface. Then:
Check that the cli version of php is the same as the ‘web version’
Create a Screen session (this means the upgrade will proceed even if your connection to the server is suddenly broken).
Navigate to /admin/cli/ and perform the following command:
php upgrade.php
Please use the correct PHP version here, e.g. php5.6 upgrade.php.
We don’t want to use the web interface for upgrading, because the upgrade can take a long time. You might run into timeout issues somewhere along the way.
Basic Tests
After the upgrade, perform a few basic tests:
Test if Moodle is still sending out email
Make sure the cron job is still running properly
Contact your users
Once the test upgrade is done, contact your users with the details of the test website, specifically the url.
Ask your users to approve the upgrade.
Also send the details of the test installation, such as the Moodle admin account (username and password) to our lead developer (the lead engineer, at the time of writing), including the MySQL password, the Unix password and the url. Be sure to do that in a secure way.
Perform the live upgrade
Prepare the Git repository for the deployment of the upgraded code. In your local repo, do the following:
Re-add the new versions of all the missing plugins if necessary. (And then commit and push in Git).
Then:
Put the Moodle website in maintenance mode: visit /admin/settings.php?section=maintenancemode to do that.
Create a complete backup of:
The moodledata directory
The database
The entire code base (can usually be found in public_html, htdocs, or a subdirectory called moodle)
Alternatively, if the hosting environment supports it, you may instead create a snapshot of the entire system. (Hosting Provider supports this through their KIS panel.)
Update the system (i.e. the web server) if necessary – this may disrupt the current (old) Moodle installation, so make absolutely sure Moodle is in maintenance mode first.
Also make sure there are no other websites that depend on the old version of php or the database (or whatever component you need to update on the system)
Make sure that the settings of the current stack are retained. For example, if php.ini for php 7.1 has upload_max_filesize = 500M, then make sure that your new php 7.2 version has the same value in the new php.ini file. Idem ditto for post_max_size and max_execution_time.
Replace the source code with the new Moodle version by replacing the local master branch with the newly updated remote master branch:
Copy the config.php file from the backup of the code base
Go to the /admin/ url of the Moodle website to check if there are any issues with the plugins
Go to the command line and start a Screen session
Perform the actual upgrade through php admin/cli/upgrade.php
Do some basic tests:
Test if Moodle is still sending out email
Make sure the cron job is still running properly (this can probably be done only after completing the next step)
In some cases, it may be necessary to copy backend settings from the test server (e.g. if a theme was upgraded, make sure to copy any custom css). Use the Site admin presets tool for this: /admin/tool/admin_presets/index.php. However, beware that this will take the site out of maintenance mode if you import from a site that has the setting disabled.
Take the Moodle website out of maintenance mode on the /admin/settings.php?section=maintenancemode page.
Once the upgrade is done successfully, contact your users.
Performing a Minor Upgrade
Minor upgrades are typically performed when Moodle HQ issues a security alert. The email has the subject “Moodle Security Alerts” and the sender is securityalerts@moodle.org. The upgrade will typically be a security patch.
In such a case, it is good practice to do a minor upgrade. For some customers, there is even a contractual obligation to perform minor upgrades.
Preparing The Git Repository
Manual minor-upgrade Git operations are no longer the standard process.
For websites that should receive automatic Moodle 4.5 minor upgrades, onboarding is now done by configuration on the automation hosts:
Secret: must match SOLIN_WEBHOOK_SECRET in coordinator env
Enable timer if not already enabled:
solin-deploy-gitbot-<repo>-nightly.timer
Steps To Do The Minor Upgrade
Once a site is onboarded, the recurring minor upgrade is handled automatically by the coordinator + runner stack:
Nightly Git bot fetches upstream Moodle tags and selects latest eligible patch tag (v4.5.*).
Git bot merges that upstream tag into the configured site branch(es) and pushes.
Bitbucket webhook creates a deploy job in the coordinator.
Runner pulls the target commit, enables maintenance mode, runs admin/cli/upgrade.php --non-interactive, disables maintenance mode, and performs health checks.
On deployment failure, automatic rollback is executed with backup + optional Git commit restore, and an immediate rollback alert email is sent.
Why third-party plugins are not overwritten in minor upgrades
The automated minor upgrade uses a Git merge of upstream Moodle tag into the existing site branch. It does not replace the branch with a clean upstream tree.
As long as third-party plugins are committed in the site repository branch, they remain in place after the merge. In other words: plugin loss risk comes from branch history/content mistakes (for example, force-rewriting a branch without carrying custom plugin directories), not from the normal automated minor-upgrade merge process itself.
Troubleshooting
SSL / HTTPS Issue
During an upgrade I ran into the following issue: a test php script was running just fine, but whenever I tried to access Moodle itself, I was redirected to the server’s default site. In addition, https was enforced.
It turned out that the site had this setting:
This was a Moodle 3.1 site, from back when only the login pages had https.
This situation may arise when you run Moodle without an SSL certificate, for example because you’re using a temporary domain name and you want to add the certificate once you have attached the actual domain name to the same vhost.
If you find yourself in a similar situation, go to the script lib/setuplib.php and look for the initialise_fullme() function.
In this code block:
if (!empty($CFG->overridetossl {
if (strpos($CFG->wwwroot, 'http://') === 0) {
$CFG->wwwroot = str_replace('http:', 'https:', $CFG->wwwroot);
} else {
unset_config('overridetossl');
}
}
Copy unset_config('overridetossl'); to the top:
unset_config('overridetossl');
if (!empty($CFG->overridetossl {
if (strpos($CFG->wwwroot, 'http://') === 0) {
$CFG->wwwroot = str_replace('http:', 'https:', $CFG->wwwroot);
} else {
unset_config('overridetossl');
}
}
And now there won’t be a forced https redirect anymore.
H5p – execute task "Download available H5P content types from h5p.org" by setting it to every 1 minute (and then resetting to default settings).
