Make Kubernetes Job side effects safe to retry

A single requested completion does not establish exactly-once application execution.

Source facts

Kubernetes documents that the same program can sometimes start twice even when a Job requests one completion, one parallel Pod, and restartPolicy Never. A Job’s application must tolerate temporary files, locks, and incomplete output from earlier runs. Pod failures normally count toward the configured backoff limit, while additional policies can refine failure handling.

Source: Kubernetes documentation — Jobs.

Applicability

Review Job retry and duplicate-start behavior before running a task with external side effects.

DSE recommendation

DSE recommends choosing an application-level operation identifier and a safe duplicate-handling strategy before scheduling side-effecting batch work. Separate work acquisition, progress recording, and final commit so another attempt can determine what already happened. Define cleanup for partial files and expired locks, and retain logs that connect every Pod attempt to the logical operation. Review the retry budget alongside the cost or risk of another attempt. Disabling container restarts alone should never be the stated control preventing duplicate business changes.

Verification

Run a disposable job against a nonproduction target, interrupt it between preparation and completion, and allow another attempt. Inspect the resulting records and artifacts for duplicates or unfinished state. Confirm that the application’s recovery logic, rather than an assumed execution count, protects correctness. Use a harmless logical operation identifier and list the expected target records before the interruption. Have the task owner reconcile every attempted execution with that single intended outcome, including any deliberate no-op retry.

Official reference

Kubernetes documentation — Jobs. Reviewed October 20, 2026.