1616612 : job failing

Created: 2026-07-13T09:47:45Z - current status: new

Summary

A user reports performance issues with job submissions on the Maxwell system for a specific proposal (anonymized). Initially, jobs were assigned to a dedicated reservation (upex_[PROPOSAL]_damnit) with 12 CPU nodes, but many jobs took abnormally long to complete. After switching to a general CPU partition (allcpu), some jobs—particularly those involving XPCS-based calculations—still exhibited severe slowdowns: - Reading data from .h5 files took 10× longer than expected. - Matrix calculations via CPUs took 50–300× longer compared to prior benchmarks.

Logs (attached) show discrepancies between runtime performance during the beamtime and later test runs. Benchmark results suggest hardware/node-specific issues may be affecting certain computations.

Anonymized Details

  • Proposals affected: [PROPOSAL_A], [PROPOSAL_B] (originally named).
  • Reservation: upex_[REDACTED]_damnit (12 CPU nodes).
  • Environment path: /gpfs/exfel/exp/[YEAR]/[MONTH]/p[PROPOSAL]/usr/Software/analysis_env/
  • Benchmark commands: bash sbatch run_pipeline.sh [PROPOSAL] [RUN_NO] [SAMPLE]
  • Files referenced: Logs: r[RUN]-p[PROPOSAL].out, AL_job_r[RUN]_[JOB_ID].out. Scripts: run_pipeline.sh, pipeline_args.py.

---

Possible Causes & Solution Steps

1. Node-Specific Performance Issues

  • Hypothesis: Some nodes in the reservation or allcpu partition may have degraded storage/I/O performance (e.g., HDF5 reads) or CPU throttling.
  • Action:
    • Check system metrics (CPU load, disk I/O latency) for nodes used during problematic runs (listed in logs).
    • Compare against nodes used in successful benchmarks.
    • Run iostat, dstat, or nmon during test jobs to identify bottlenecks.

2. Filesystem Latency

  • Hypothesis: Slow .h5 file reads suggest potential issues with GPFS (or other shared filesystem) performance under high concurrency.
  • Action:
    • Test reading/writing .h5 files locally vs. over the network.
    • Monitor GPFS server loads during job execution.

3. Software Environment Differences

  • Hypothesis: The analysis environment (/gpfs/.../analysis_env/) might have dependency conflicts or non-optimal library versions (e.g., HDF5, NumPy).
  • Action:
    • Verify consistency of the Conda/Python environment across nodes.
    • Re-run benchmarks with verbose logging to capture library paths and versions.

4. Resource Contention

  • Hypothesis: Other jobs on shared nodes may have consumed resources (e.g., memory bandwidth, cache).
  • Action:
    • Isolate jobs on exclusive nodes for testing.
    • Use tools like htop or Slurm’s sacct to analyze resource usage patterns.

5. Code/Algorithm Changes

  • Hypothesis: Unintended changes in the pipeline code (e.g., unoptimized loops in matrix operations) could cause slowdowns.
  • Action:
    • Diff the run_pipeline.sh/pipeline_args.py against known-good versions.
    • Profile the code (e.g., Python’s cProfile or line_profiler).
  1. Reproduce the Issue:
  2. Run the provided benchmarks on suspect nodes using: bash sbatch run_pipeline.sh [PROPOSAL] [RUN_NO] nosample

  3. Time the "read" and "calc" phases explicitly.

  4. Compare Nodes:

  5. Submit identical jobs to different partitions/nodes and compare timings.

  6. Inspect Logs:

  7. Analyze attached logs for errors (e.g., I/O wait times, OOM kills) near the TTCF calculation done in or CANCELLED AT lines.

  8. Escalate Hardware Checks:

  9. If node-specific, involve sysadmins to test disk/CPU health on flagged nodes.

Note

All personal/company identifiers (e.g., proposal numbers beyond [PROPOSAL], usernames, exact paths) should remain redacted in public discussions. Share detailed logs/files only with authorized personnel.