How to Set Up Automated HR Imports in Totara

Assumptions

This SOP makes the following assumptions:

• The customer has an external system from which they want to import data into Totara.

• They can provide csv files through SFTP in an automated way (we do not cover any other data sources for the HR Import in this SOP yet).

Terminology

• Element: the elements determine which type of data can be imported – Job assignment; Organisation; Position; and User.

• Source: for each element, you can configure a source for the data. For example, information for the User element could come from a csv file, or from an external database.

• SFTP: Secure FTP (File Transfer Protocol), used to copy files from one computer (e.g. a web server) to another over the internet.

• Public – Private Key(pair): to provide secure easy access to the SFTP server (typically the web server where Totara lives), we ask the customer for their public key (also known as ssh public key). The public key must be added to the web server by a system administrator. The customer’s private key is automatically used to encrypt the SFTP connection.

Provide Sample CSV Files and Documentation

Most customers ask for an example of the csv files. Use spreadsheets to list the column headers and provide a row of sample data. Also add a clearly delineated row of documentation (explaining the data types, uniqueness requirements, and so on).

The file names for automated HR import should be the following:

Users: user.csv

Job assignments: jobassignment.csv

Organisations: org.csv

Positions: pos.csv

Ask for a Public SSH Key for SFTP

In this scenario, the customer is going to set up an automated process to upload their csv files to the Totara web server, using SFTP. To make this easy and safe, ask the customer to send over their public key, and add it to .ssh/authorized_keys file (or ask the system administrator to do this).

Since we don’t know anything about the customer’s system (and don’t want to), we’re not going to explain how to set up an automated process for uploading, nor how to generate a public – private key pair.

Configure HR Import

In this section, we’re going to configure HR Import specifically for the customer. There’s one caveat: we don’t want to automate the HR Import yet. We’re going to test manually first (see the section Test the Customer’s CSV Files), and then switch to automation (see the section Automate the HR Import).

Delta vs. Complete Import

In het kader van het project voor de provincie Antwerpen, zou ik nog uitzoeken wat er gebeurt bij een keuze voor import met alle gebruikers. Het staat nergens expliciet in de documentatie, maar als je verschillende beschrijvingen combineert, dan gebeurt er het volgende:

– Als je bij “Source contains all records” voor “Yes” kiest, dan worden ontbrekende gebruikers die in de bron ontbreken, maar wel in Totara aanwezig zijn, verwijderd.

Wat verwijderd inhoudt, hangt dan weer af van je keuze bij “Allowed HR Import actions” > “Delete”: keep, suspend of full delete. Bij een full delete worden alle gebruikersgegevens verwijderd, incl. de geassocieerde (grades, appraisals, etc.).

Het advies is om hier de optie suspend te kiezen.

Als je alleen een delta importeert via HR Import, dan moet je de kolom deleted meegeven (met een 0 voor blijvende, en een 1 voor te verwijderen gebruikers).

The following is documented: for the source user, there is a timemodified field. The documentation states:

If you want records to be updated every time an import is performed and not take the Unix timestamp value into account, you can specify a zero (0) value for this field. This will effectively force HR import to import the data every time and ignore changes in Unix timestamp values.

Check the Customer’s SFTP Upload of the CSV Files

The goal here is twofold:

• Make sure that the customer’s csv files actually got transferred to the correct directory on the web server (a directory that must also be readable by Totara).

• Check the format of the csv files.

Format of the CSV Files

Check the uploaded csv files against the csv documentation and samples you provided earlier. Keep a sharp lookout for custom profile fields. These must have a column header in the following format: customfield_[shortname-of-the-custom-profile-field] (minus the square brackets).

Test the Customer’s CSV Files

If the csv files are present on the web server, and their formatting is alright (see the previous section), we should test the actual import. Use a test version of Totara to do this.

Automate the HR Import

To schedule the automated HR import the task \totara_core\task\tool_totara_sync_task is used.

A specific folder structure is required:

path-to-totara/csv/ready

In the settings of the HR import you should add at files directory only the path-to-totara (without csv/ready)

With the first HR import run using the files directory, a new folder will be created called Store. If the system has no permission to create a folder, you can also manually create this folder.

The csv files should be added to the ready folder.

After the HR-import run, the csv files are moved to the store folder. The names of the files are changed by adding an unix timestamp.

Discuss with customer how many files should be stored