User grid-like submission to CEs¶

Submitting jobs to a CondorCE authenticated/authorized by a plain SSL user or robot certificate or WLCGToken.

Preface¶

With the phase out of Globus support, i.e., grid user proxy support, the job submission to a Grid CE is assumed to be only through factories. User submission is not really intended and works only with a number of workarounds.

The scope is to submit from a user context with condor_submit to a CondorCE in the grid. The CondorCE has to have the user's DN (for SSL authz) or WLCGToken client ID (for token authz) prepared in the CE's user mappings.

Caveats¶

In the user context, the $X509_USER_PROXY environment variable has to be set and pointing to a grid proxy certificate. The proxy itself is not used in the actual submission, but since the path for SSL authz has dependencies on the proxy code we need a proxy present (can in principle also invalid beyond its lifetime)

For the submission, negotiator/collector/scheduler daemons has to be present on the submission node as to handle the local job shadows.

Set Up¶

User Environment¶

As your user on a node with negotiator/collector/scheduler daemons present, you will need to prepare your user environment.

We have prepared an exprimental environment script, that should prepare most of the necessary environment variables

source /cvmfs/grid.desy.de/etc/profile.d/grid-usersubmission.sh

Details¶

Proxy Preparation¶

If you do not have a proxy present, source your experiments environment set up or our generic set up based on the preliminary Alma9 grid ui.

source /cvmfs/grid.desy.de/etc/profile.d/scripts/glite-ui-env.sh

Request a proxy from your VOMS

voms-proxy-init  -voms foo

Condor SSL Preparations¶

Prepare the necesasary Condor config ads. Here it is assume, that your grid user certificate and key are located in your $HOME directory under ${HOME}/.globus/user{cert,key}.pem

export _condor_AUTH_SSL_CLIENT_KEYFILE=~/.globus/userkey.pem
export _condor_AUTH_SSL_CLIENT_CERTFILE=~/.globus/usercert.pem
export _condor_SEC_CLIENT_AUTHENTICATION_METHODS=SSL
export _condor_AUTH_SSL_CLIENT_CERTFILE=~/.globus/usercert.pem
export _condor_AUTH_SSL_CLIENT_CADIR=/cvmfs/grid.cern.ch/etc/grid-security/certificates/
export _condor_AUTH_SSL_SERVER_CADIR=/cvmfs/grid.cern.ch/etc/grid-security/certificates/
export _condor_AUTH_SSL_USE_CLIENT_PROXY_ENV_VAR=false
export _condor_AUTH_SSL_SERVER_CAFILE=''
export _condor_AUTH_SSL_CLIENT_CAFILE=''
export X509_CERT_DIR=/cvmfs/grid.cern.ch/etc/grid-security/certificates/
export X509_VOMS_DIR=/cvmfs/grid.cern.ch/etc/grid-security/vomsdir/

NAF specific set up¶

As NAF workgroup servers use a remote submission, a grid-site submission would fail as condor_submit would not find a local scheduler. To use temporarily a locally running scheduler/negotiator/collector (ensure that your system administrator has these daemons started), set the following environment variables so that Condor will use the local daemons

export _condor_SCHEDD_HOST=${HOSTNAME}
export _condor_CREDD_HOST=${HOSTNAME}
export _condor_CONDOR_HOST=${HOSTNAME}
export _condor_SEC_CREDENTIAL_PRODUCER=""

Submission¶

After setting up the necessary environment, a job can be submitted to a remote CondorCE with something like

condor_submit -verbose  -spool -pool grid-htc-preprod-ce01.desy.de:9619 -name grid-htc-preprod-ce01.desy.de myjob.submit

where the remote CE has to be set as central manager (-pool) and access point (-pool) with the correct port.