SLURM Rest API¶

SLURM provides a REST API daemon which allows to submit and manage jobs.

Rest API server¶

the Maxwell cluster and the solaris sub-cluster have separate API server:

maxwell: https://max-slurm-rest.desy.de/sapi
solaris: https://max-sol-portal.desy.de/sapi

Authorization is handled through tokens. This can either be a JWKS keycloak token or a JWT SLURM token

Using JWT SLURM token¶

On Maxwell/Solaris you will need two tokens:

a revocable portal token. It is generated with username and password, and comes without expiry, but can be revoked.
a slurm token. It is generated with the portal token, but has a maximum lifetime of 24 hours. It can't be revoked.

BE AWARE: whoever knows your token has access to all your files on maxwell! We therefore disabled the ability to generate unlimited tokens using scontrol.

Token handling on Maxwell¶

on Maxwell nodes like display nodes, you can generate tokens with some convenience scripts. On Maxwell you could simply use sbatch to submit a job, so there is usually not much of an advantage in using the API. Anyhow to generate tokens

portal_token   # full path: /software/tools/bin/portal_token

generates the portal token and stores it in ~/.maxwell/portal.token.

slurm_token -l 84400

generates a slurm token valid for 24 hours.

export $(slurm_token -l 84400

generates a slurm token valid for 24 hours and stores it in the variable SLURM_TOKEN.

Token handling elsewhere¶

the REST API are not exposed outside the DESY network, but can be reached from any machine in the DESY network. To generate tokens you will usually not have the convenience commands, but it's easy enough to use simple curl commands or python scriplets.

Portal Token ... Using CURL

# creating a portal token
curl -u user https://max-slurm-rest.desy.de/reservation/get_new_token?cli_api=4.6.0

#it will ask for the password of the user.








.

... Using PYTHON

# creating a portal token
import requests
import getpass

user="user"
pw = getpass.getpass()

session = requests.Session()
session.auth = (user, pw)

url = "https://max-slurm-rest.desy.de/reservation/get_new_token?cli_api=4.6.0"
portal_token = session.get(url)
print(portal_token)

SLURM Token ... Using CURL

# creating a slurm jwt token
url = "https://max-slurm-rest.desy.de/reservation/get_new_slurm_token?cli_api=4.6.0&lifespan=86400&stu=hpcguest"
export SLURM_TOKEN=$(curl -u user $url | jq '.token' | tr -d '"')

# it will ask for the password, use the portal token! 
# Like before the slurm token with be stored in SLURM_TOKEN








.

... Using PYTHON

# creating a slurm jwt token
import requests
import getpass
import json

user="user"
# use portal token as password
portal_token = getpass.getpass()

session.auth = (user, portal_token)
url = f"https://max-slurm-rest.desy.de/reservation/get_new_slurm_token?cli_api=4.6.0&lifespan=86400&stu={user}"
response = session.get(url)
slurm_token = json.loads(response.text)['token']

print(slurm_token)

Job submission¶

To submit a job, the job-script has to be embedded into a json string. A very simple example for a job script:

Using CURL

# a sample file named job.json:

{
  "job":{
     "partition": "maxcpu", 
     "name":"testapi",
     "time_limit": {"set": True, "number": 1000},
     "current_working_directory":"/home/user",
     "environment":["PATH=/bin:/usr/bin/:/usr/local/bin/",
                    "LD_LIBRARY_PATH=/lib/:/lib64/:/usr/local/lib"]
   },
  "script":"#!/bin/bash -l\n
            srun hostname\n
            sleep 300\n
           "
}

# submit the job
curl -L -H "Content-Type: application/json" \
     -H X-SLURM-USER-NAME:user \
     -H X-SLURM-USER-TOKEN:$SLURM_TOKEN \
     -X POST https://max-slurm-rest.desy.de/sapi/slurm/v0.0.40/job/submit \
     -d@job.json















.

Using PYTHON

import requests
import json
from requests.auth import HTTPBasicAuth

SLURM_REST_API_URL = "https://max-slurm-rest.desy.de/sapi/slurm/v0.0.40/job/submit"
user = "user"  
SLURM_TOKEN = os.getenv(SLURM_TOKEN)

JOB_SCRIPT_FILE = "job_script.sh"
with open(JOB_SCRIPT_FILE, 'r') as file:
    job_script = file.read()

headers = {
    "Content-Type": "application/json",
    "Accept": "application/json",
    "X-SLURM-USER-NAME": user,
    "X-SLURM-USER-TOKEN": SLURM_TOKEN
}

payload = {
  "job":{
     "partition": "allcpu", 
     "name":"testapi",
     "time_limit": {"set": True, "number": 1000},
     "current_working_directory":"/home/hpcguest",
     "environment":["PATH=/bin:/usr/bin/:/usr/local/bin/",
                    "LD_LIBRARY_PATH=/lib/:/lib64/:/usr/local/lib"]
      },
    "script": job_script
}


session = requests.Session()

response = session.post(SLURM_REST_API_URL,
            headers=headers,
            data=json.dumps(payload)
        )

print(json.loads(response.text))

To convert a batch-job into a suitable script for curl:

cat job.script | sed 's|"|\\"|g' | sed ':a;N;$!ba;s|\n|\\n|g'

Job information¶

# all jobs
curl -L -s -H "Content-Type: application/json" -H X-SLURM-USER-NAME:$(whoami) -H X-SLURM-USER-TOKEN:$SLURM_TOKEN \
     -X GET https://max-slurm-rest.desy.de/sapi/slurm/v0.0.40/jobs

# to extract information about individual jobs use json parser like jq:
curl -L -s -H "Content-Type: application/json" -H X-SLURM-USER-NAME:$(whoami) -H X-SLURM-USER-TOKEN:$SLURM_TOKEN \
     -X GET https://max-slurm-rest.desy.de/sapi/slurm/v0.0.40/jobs > jobs.json
cat jobs.json | jq '.jobs[]| select(.job_id == 7752003)'
cat jobs.json | jq '.jobs[]| select(.user_name == "username")'  # replace username by a real username

# specific running or pending job
curl -L -s -H "Content-Type: application/json" -H X-SLURM-USER-NAME:$(whoami) -H X-SLURM-USER-TOKEN:$SLURM_TOKEN \
     -X GET https://max-slurm-rest.desy.de/sapi/slurm/v0.0.40/job/7750418

# a job already removed from the queue will report  `"error": "_handle_job_get: unknown job 7751560"`

# retrieve information about finished jobs:
curl -L -H "Content-Type: application/json" -H X-SLURM-USER-NAME:$(whoami) -H X-SLURM-USER-TOKEN:$SLURM_TOKEN \
     -X GET https://max-slurm-rest.desy.de/sapi/slurmdb/v0.0.40/job/7740309

Using Keycloak tokens - JWKS¶

stubs!

Simple sample:

KC_SERVER_URL="https://keycloak.desy.de/auth"
KC_REALM="production"
KC_CLIENT_ID="desy-public"

curl -fsSL -X POST "${KC_SERVER_URL}/realms/${KC_REALM}/protocol/openid-connect/token" \
  --header "Content-Type: application/x-www-form-urlencoded" \
  --data-urlencode "password=${KC_USER_PASSWORD}" \
  --data-urlencode "username=${KC_USER_USERNAME}" \
  --data-urlencode "client_id=${KC_CLIENT_ID}" \
  --data-urlencode "scope=openid sun" \
  --data-urlencode "grant_type=password" | jq

provides access_token & refresh_token.

export SLURM_TOKEN="keycloak access_token"
curl -H "Content-Type: application/json" -H X-SLURM-USER-TOKEN:$SLURM_TOKEN      
     -X POST https://max-portal.desy.de/sapi/slurm/v0.0.40/job/submit --data-binary @job.json

and the job gets submitted. SLURM requires username or sun to be included in the token https://slurm.schedmd.com/jwt.html, so scope=sun is needed (openid is not).

Generating JWT as root¶

As a privileged user it's possible to create token for arbitrary users on maxwell:

#!/usr/bin/env python3
import sys
import os
import pprint
import json
import time
from datetime import datetime, timedelta, timezone

from jwt import JWT
from jwt.jwa import HS256
from jwt.jwk import jwk_from_dict
from jwt.utils import b64decode,b64encode

if len(sys.argv) != 3:
    sys.exit("generate_jwt.py [user name] [expiration time (seconds)]");

jwt_key = os.environ.get('JWT_KEY', '/etc/slurm/jwt_hs256.key')

with open(jwt_key, "rb") as f:
    priv_key = f.read()

signing_key = jwk_from_dict({
    'kty': 'oct',
    'k': b64encode(priv_key)
})

message = {
    "exp": int(time.time() + int(sys.argv[2])),
    "iat": int(time.time()),
    "sun": sys.argv[1]
}

a = JWT()
compact_jws = a.encode(message, signing_key, alg='HS256')
print("SLURM_TOKEN={}".format(compact_jws))

python3 generate_jwt.py user 3600 would generate the token - if you have a copy of the "secret key". Using keycloak tokens to submit jobs on behalf of users is a much better choice!

Documentation¶

topic	url
General information about SLURMs REST API	https://slurm.schedmd.com/rest.html
SLURM REST API reference	https://slurm.schedmd.com/rest_api.html
Information about JSON web tokens	https://slurm.schedmd.com/jwt.html
slides for the SLUG 2020 talk	https://slurm.schedmd.com/SLUG20/REST_API.pdf
slides for the SLUG 2019 talk	https://slurm.schedmd.com/SLUG19/REST_API.pdf