Setting Up Automated HR Import in Totara

Totara's HR Import automates user provisioning from CSV files via SFTP, supporting delta or complete imports of users, organizations, positions, and job assignments. This guide covers setup, custom profile field mapping, and transitioning from manual testing to fully automated imports.

Assumptions

This SOP makes the following assumptions:

• The customer has an external system from which they want to import data into Totara.
• They can provide csv files through SFTP in an automated way (we do not cover any other data sources for the HR Import in this guide yet).

Terminology

• Element: the elements determine which type of data can be imported – Job assignment; Organisation; Position; and User.
• Source: for each element, you can configure a source for the data. For example, information for the User element could come from a csv file, or from an external database.
• SFTP: Secure FTP (File Transfer Protocol), used to copy files from one computer (e.g. a web server) to another over the internet.
• Public – Private Key(pair): to provide secure easy access to the SFTP server (typically the web server where Totara lives), we ask your users for their public key (also known as ssh public key). The public key must be added to the web server by a system administrator. The customer's private key is automatically used to encrypt the SFTP connection.

Provide Sample CSV Files and Documentation

Most customers ask for an example of the csv files. Use spreadsheets to list the column headers and provide a row of sample data. Also add a clearly delineated row of documentation (explaining the data types, uniqueness requirements, and so on).

If the site is using any custom profile fields, then you must adapt the sample csv files and documentation for their benefit. Here is an example: Kiwa – HR Import Samples Plus Instructions.

The file names for automated HR import should be the following:

Users: user.csv

Job assignments: jobassignment.csv

Organisations: org.csv

Positions: pos.csv

Ask for a Public SSH Key for SFTP

In this scenario, the site is going to set up an automated process to upload their csv files to the Totara web server, using SFTP. To make this easy and safe, ask your users to send over their public key, and add it to .ssh/authorized_keys file (or ask the system administrator to do this).

Since we don’t know anything about your users’s system (and don’t want to), we’re not going to explain how to set up an automated process for uploading, nor how to generate a public – private key pair.

Configure HR Import

In this section, we’re going to configure HR Import specifically for your users. There’s one caveat: we don’t want to automate the HR Import yet. We’re going to test manually first (see the section Test the Customer’s CSV Files), and then switch to automation (see the section Automate the HR Import).

Delta vs. Complete Import

• It is worth understanding what happens when you set "Source contains all records" to "Yes". This is not spelled out explicitly in the documentation, but combining the various descriptions, the behavior is as follows:
• Users who are present in Totara but missing from the source file will be removed.
• What removed means depends on your "Allowed HR Import actions" > "Delete" setting: keep, suspend, or full delete. A full delete removes all user data, including associated records (grades, appraisals, etc.).
• It is recommended to choose suspend here rather than full delete.
• If you are only importing a delta via HR Import, include a deleted column in your source file (0 for retained users, 1 for users to be removed).

The following is documented: for the source user, there is a timemodified field. The documentation states:

If you want records to be updated every time an import is performed and not take the Unix timestamp value into account, you can specify a zero (0) value for this field. This will effectively force HR import to import the data every time and ignore changes in Unix timestamp values.

Check the Customer's SFTP Upload of the CSV Files

The goal here is twofold:

• Make sure that your csv files actually got transferred to the correct directory on the web server (a directory that must also be readable by Totara).
• Check the format of the csv files.

Format of the CSV Files

Check the uploaded csv files against the csv documentation and samples you provided earlier. Keep a sharp lookout for custom profile fields. These must have a column header in the following format: customfield_[shortname-of-the-custom-profile-field] (minus the square brackets).

Test the Customer’s CSV Files

If the csv files are present on the web server, and their formatting is alright (see the previous section), we should test the actual import. Use a test version of Totara to do this.

Automate the HR Import

To schedule the automated HR import the task \totara_core\task\tool_totara_sync_task is used.

A specific folder structure is required:

path-to-totara/csv/ready

In the settings of the HR import you should add at files directory only the path-to-totara (without csv/ready)

With the first HR import run using the files directory, a new folder will be created called Store. If the system has no permission to create a folder, you can also manually create this folder.

The csv files should be added to the ready folder.

After the HR-import run, the csv files are moved to the store folder. The names of the files are changed by adding an unix timestamp.

Discuss with customer how many files should be stored