Bottom line: Distinguish a known non-retriable application failure from an infrastructure disruption before tuning a Job retry budget. Keep failure classification separate from the application’s ability to handle repeated execution safely.
\nThe Handling retriable and non-retriable pod failures with Pod failure policy documentation is the primary reference.
\nThe tutorial labels Pod failure policy stable since Kubernetes 1.31 and demonstrates a FailJob rule matching exit code 42 from a container named main. Its disruption example uses Ignore with the DisruptionTarget Pod condition to avoid incrementing the retry counter toward backoffLimit. For the FailJob example, the controller records FailureTarget when the Job is considered failed and Failed after all its Pods terminate.
The source’s example exit code is not a universal classification for applications. This article does not establish exactly-once execution, approve unlimited retries or define a production node-drain procedure. Keep duplicate-side-effect protection, workload cleanup and maintenance authorization in their own reviewed procedures.
\nDSE recommends a failure-class worksheet owned jointly by the application and cluster operators. For each class, record the evidence that identifies it and the intended next action. Require the application owner to define the meaning of an exit status instead of copying the tutorial’s simulated bug code into an unrelated workload.
\nFor a proposed non-retriable class, review the exact container selector and status match with the workload definition. Name who will investigate the failed Job and preserve the necessary evidence. Keep an unmatched application failure visible in the procedure so it is not mistakenly reported as a recognized permanent defect.
\nReview disruption handling separately. Identify the approved maintenance or infrastructure case being considered, the retry-budget requirement and the observation that will classify the Pod. Do not use a failure-policy exercise as authorization to drain a shared production node. Keep the laboratory fixture and maintenance process distinct.
\nRetain the existing application retry-safety review. Use the policy decision to describe controller handling, not to replace duplicate detection or incomplete-output recovery. Have the operator reconcile the final Job state with the specific matching condition before closing an incident, including any Pods still awaiting termination.
\nProposed verification, not performed: use an isolated Job with harmless output and a deliberately chosen non-retriable test exit. Record the expected matching container, policy action and condition sequence, then inspect the actual Job and Pod statuses.
\nIn a separate authorized disruption exercise, predict the retry-counter treatment before collecting its result. Keep each case’s manifest, observed exit or condition, counter and Job state. Include an unmatched test exit to examine the intended fallback independently. Do not report a controller action as evidence that the application performed its external work only once.
\nSource review completed October 29, 2026. Recheck the official documentation and deployment-specific applicability before a production change.
\n", "content_text": "Bottom line: Distinguish a known non-retriable application failure from an infrastructure disruption before tuning a Job retry budget. Keep failure classification separate from the application’s ability to handle repeated execution safely.\nSource fact: documented behavior\nThe Handling retriable and non-retriable pod failures with Pod failure policy documentation is the primary reference.\nThe tutorial labels Pod failure policy stable since Kubernetes 1.31 and demonstrates a FailJob rule matching exit code 42 from a container named main. Its disruption example uses Ignore with the DisruptionTarget Pod condition to avoid incrementing the retry counter toward backoffLimit. For the FailJob example, the controller records FailureTarget when the Job is considered failed and Failed after all its Pods terminate.\nWhat the source does not establish\nThe source’s example exit code is not a universal classification for applications. This article does not establish exactly-once execution, approve unlimited retries or define a production node-drain procedure. Keep duplicate-side-effect protection, workload cleanup and maintenance authorization in their own reviewed procedures.\nApplicability questions\n\nWhich application exit status is genuinely non-retriable?\nWhich container should the exit-code rule inspect?\nWhich disruption case should not consume this Job’s retry budget?\nWhich Job and Pod status observations will confirm the chosen action?\n\nDSE recommendation: document the failure classes before the policy\nDSE recommends a failure-class worksheet owned jointly by the application and cluster operators. For each class, record the evidence that identifies it and the intended next action. Require the application owner to define the meaning of an exit status instead of copying the tutorial’s simulated bug code into an unrelated workload.\nFor a proposed non-retriable class, review the exact container selector and status match with the workload definition. Name who will investigate the failed Job and preserve the necessary evidence. Keep an unmatched application failure visible in the procedure so it is not mistakenly reported as a recognized permanent defect.\nReview disruption handling separately. Identify the approved maintenance or infrastructure case being considered, the retry-budget requirement and the observation that will classify the Pod. Do not use a failure-policy exercise as authorization to drain a shared production node. Keep the laboratory fixture and maintenance process distinct.\nRetain the existing application retry-safety review. Use the policy decision to describe controller handling, not to replace duplicate detection or incomplete-output recovery. Have the operator reconcile the final Job state with the specific matching condition before closing an incident, including any Pods still awaiting termination.\nVerification and evidence\nProposed verification, not performed: use an isolated Job with harmless output and a deliberately chosen non-retriable test exit. Record the expected matching container, policy action and condition sequence, then inspect the actual Job and Pod statuses.\nIn a separate authorized disruption exercise, predict the retry-counter treatment before collecting its result. Keep each case’s manifest, observed exit or condition, counter and Job state. Include an unmatched test exit to examine the intended fallback independently. Do not report a controller action as evidence that the application performed its external work only once.\nOfficial references\n\nHandling retriable and non-retriable pod failures with Pod failure policy — Kubernetes.\n\nSource review completed October 29, 2026. Recheck the official documentation and deployment-specific applicability before a production change.", "content_markdown": "Bottom line: Distinguish a known non-retriable application failure from an infrastructure disruption before tuning a Job retry budget. Keep failure classification separate from the application’s ability to handle repeated execution safely.\n\n## Source fact: documented behavior\n\nThe [Handling retriable and non-retriable pod failures with Pod failure policy](https://kubernetes.io/docs/tasks/job/pod-failure-policy/) documentation is the primary reference.\n\nThe tutorial labels Pod failure policy stable since Kubernetes 1.31 and demonstrates a FailJob rule matching exit code 42 from a container named main. Its disruption example uses Ignore with the DisruptionTarget Pod condition to avoid incrementing the retry counter toward backoffLimit. For the FailJob example, the controller records FailureTarget when the Job is considered failed and Failed after all its Pods terminate.\n\n## What the source does not establish\n\nThe source’s example exit code is not a universal classification for applications. This article does not establish exactly-once execution, approve unlimited retries or define a production node-drain procedure. Keep duplicate-side-effect protection, workload cleanup and maintenance authorization in their own reviewed procedures.\n\n## Applicability questions\n\n- Which application exit status is genuinely non-retriable?\n\n- Which container should the exit-code rule inspect?\n\n- Which disruption case should not consume this Job’s retry budget?\n\n- Which Job and Pod status observations will confirm the chosen action?\n\n## DSE recommendation: document the failure classes before the policy\n\nDSE recommends a failure-class worksheet owned jointly by the application and cluster operators. For each class, record the evidence that identifies it and the intended next action. Require the application owner to define the meaning of an exit status instead of copying the tutorial’s simulated bug code into an unrelated workload.\n\nFor a proposed non-retriable class, review the exact container selector and status match with the workload definition. Name who will investigate the failed Job and preserve the necessary evidence. Keep an unmatched application failure visible in the procedure so it is not mistakenly reported as a recognized permanent defect.\n\nReview disruption handling separately. Identify the approved maintenance or infrastructure case being considered, the retry-budget requirement and the observation that will classify the Pod. Do not use a failure-policy exercise as authorization to drain a shared production node. Keep the laboratory fixture and maintenance process distinct.\n\nRetain the existing application retry-safety review. Use the policy decision to describe controller handling, not to replace duplicate detection or incomplete-output recovery. Have the operator reconcile the final Job state with the specific matching condition before closing an incident, including any Pods still awaiting termination.\n\n## Verification and evidence\n\nProposed verification, not performed: use an isolated Job with harmless output and a deliberately chosen non-retriable test exit. Record the expected matching container, policy action and condition sequence, then inspect the actual Job and Pod statuses.\n\nIn a separate authorized disruption exercise, predict the retry-counter treatment before collecting its result. Keep each case’s manifest, observed exit or condition, counter and Job state. Include an unmatched test exit to examine the intended fallback independently. Do not report a controller action as evidence that the application performed its external work only once.\n\n## Official references\n\n- [Handling retriable and non-retriable pod failures with Pod failure policy](https://kubernetes.io/docs/tasks/job/pod-failure-policy/) — Kubernetes.\n\nSource review completed October 29, 2026. Recheck the official documentation and deployment-specific applicability before a production change." }, "json_ld": { "@context": "https://schema.org", "@graph": [ { "@type": "Organization", "@id": "https://dsesecurity.com/#organization", "name": "Detection Systems & Engineering", "alternateName": "DSE Security", "url": "https://dsesecurity.com/", "logo": { "@type": "ImageObject", "url": "https://update.dsesecurity.com/assets/dse-logo-20260812.png?v=1.8.20" } }, { "@type": "Organization", "@id": "https://update.dsesecurity.com/#editorial-team", "name": "DSE Security Editorial Team", "url": "https://update.dsesecurity.com/", "parentOrganization": { "@id": "https://dsesecurity.com/#organization" } }, { "@type": "WebSite", "@id": "https://update.dsesecurity.com/#website", "name": "DSE Updates", "alternateName": "DSE Security Knowledge Hub", "url": "https://update.dsesecurity.com/", "inLanguage": "en-US", "publisher": { "@id": "https://dsesecurity.com/#organization" }, "potentialAction": { "@type": "SearchAction", "target": { "@type": "EntryPoint", "urlTemplate": "https://update.dsesecurity.com/?q={search_term_string}" }, "query-input": "required name=search_term_string" } }, { "@type": "WebPage", "@id": "https://update.dsesecurity.com/updates/separate-job-exit-code-failures-from-kubernetes-disruptions/", "url": "https://update.dsesecurity.com/updates/separate-job-exit-code-failures-from-kubernetes-disruptions/", "isPartOf": { "@id": "https://update.dsesecurity.com/#website" }, "lastReviewed": "2026-10-29" }, { "@type": "BreadcrumbList", "@id": "https://update.dsesecurity.com/updates/separate-job-exit-code-failures-from-kubernetes-disruptions/#breadcrumbs", "itemListElement": [ { "@type": "ListItem", "position": 1, "name": "DSE Updates", "item": "https://update.dsesecurity.com/" }, { "@type": "ListItem", "position": 2, "name": "Business Continuity", "item": "https://update.dsesecurity.com/topic/business-continuity/" }, { "@type": "ListItem", "position": 3, "name": "Separate Job exit-code failures from Kubernetes disruptions", "item": "https://update.dsesecurity.com/updates/separate-job-exit-code-failures-from-kubernetes-disruptions/" } ] }, { "@type": [ "Article", "TechArticle" ], "@id": "https://update.dsesecurity.com/updates/separate-job-exit-code-failures-from-kubernetes-disruptions/#article", "identifier": "https://update.dsesecurity.com/updates/separate-job-exit-code-failures-from-kubernetes-disruptions/", "url": "https://update.dsesecurity.com/updates/separate-job-exit-code-failures-from-kubernetes-disruptions/", "headline": "Separate Job exit-code failures from Kubernetes disruptions", "description": "The Pod failure policy tutorial demonstrates FailJob for a known exit code and Ignore for DisruptionTarget.", "abstract": "The Pod failure policy tutorial demonstrates FailJob for a known exit code and Ignore for DisruptionTarget. Review those failure classes separately from application duplicate-handling logic.", "articleBody": "Bottom line: Distinguish a known non-retriable application failure from an infrastructure disruption before tuning a Job retry budget. Keep failure classification separate from the application’s ability to handle repeated execution safely.\nSource fact: documented behavior\nThe Handling retriable and non-retriable pod failures with Pod failure policy documentation is the primary reference.\nThe tutorial labels Pod failure policy stable since Kubernetes 1.31 and demonstrates a FailJob rule matching exit code 42 from a container named main. Its disruption example uses Ignore with the DisruptionTarget Pod condition to avoid incrementing the retry counter toward backoffLimit. For the FailJob example, the controller records FailureTarget when the Job is considered failed and Failed after all its Pods terminate.\nWhat the source does not establish\nThe source’s example exit code is not a universal classification for applications. This article does not establish exactly-once execution, approve unlimited retries or define a production node-drain procedure. Keep duplicate-side-effect protection, workload cleanup and maintenance authorization in their own reviewed procedures.\nApplicability questions\n\nWhich application exit status is genuinely non-retriable?\nWhich container should the exit-code rule inspect?\nWhich disruption case should not consume this Job’s retry budget?\nWhich Job and Pod status observations will confirm the chosen action?\n\nDSE recommendation: document the failure classes before the policy\nDSE recommends a failure-class worksheet owned jointly by the application and cluster operators. For each class, record the evidence that identifies it and the intended next action. Require the application owner to define the meaning of an exit status instead of copying the tutorial’s simulated bug code into an unrelated workload.\nFor a proposed non-retriable class, review the exact container selector and status match with the workload definition. Name who will investigate the failed Job and preserve the necessary evidence. Keep an unmatched application failure visible in the procedure so it is not mistakenly reported as a recognized permanent defect.\nReview disruption handling separately. Identify the approved maintenance or infrastructure case being considered, the retry-budget requirement and the observation that will classify the Pod. Do not use a failure-policy exercise as authorization to drain a shared production node. Keep the laboratory fixture and maintenance process distinct.\nRetain the existing application retry-safety review. Use the policy decision to describe controller handling, not to replace duplicate detection or incomplete-output recovery. Have the operator reconcile the final Job state with the specific matching condition before closing an incident, including any Pods still awaiting termination.\nVerification and evidence\nProposed verification, not performed: use an isolated Job with harmless output and a deliberately chosen non-retriable test exit. Record the expected matching container, policy action and condition sequence, then inspect the actual Job and Pod statuses.\nIn a separate authorized disruption exercise, predict the retry-counter treatment before collecting its result. Keep each case’s manifest, observed exit or condition, counter and Job state. Include an unmatched test exit to examine the intended fallback independently. Do not report a controller action as evidence that the application performed its external work only once.\nOfficial references\n\nHandling retriable and non-retriable pod failures with Pod failure policy — Kubernetes.\n\nSource review completed October 29, 2026. Recheck the official documentation and deployment-specific applicability before a production change.", "datePublished": "2026-10-29T17:40:53+00:00", "dateModified": "2026-10-29T17:41:39+00:00", "mainEntityOfPage": { "@id": "https://update.dsesecurity.com/updates/separate-job-exit-code-failures-from-kubernetes-disruptions/" }, "inLanguage": "en-US", "isAccessibleForFree": true, "author": { "@type": "Organization", "name": "DSE Security Editorial Team", "url": "https://update.dsesecurity.com/#editorial-team" }, "publisher": { "@id": "https://dsesecurity.com/#organization" }, "image": { "@type": "ImageObject", "@id": "https://update.dsesecurity.com/updates/separate-job-exit-code-failures-from-kubernetes-disruptions/#primaryimage", "url": "https://update.dsesecurity.com/assets/editorial/continuity-recovery-social-v2.jpg?v=1.8.20", "contentUrl": "https://update.dsesecurity.com/assets/editorial/continuity-recovery-social-v2.jpg?v=1.8.20", "width": 1200, "height": 630, "caption": "Separate Job exit-code failures from Kubernetes disruptions" }, "articleSection": [ "Business Continuity", "IT" ], "keywords": [ "Business Continuity", "IT", "Guide", "Advisory priority" ], "genre": "Guide", "about": [ { "@type": "Thing", "name": "Business Continuity", "url": "https://update.dsesecurity.com/topic/business-continuity/" }, { "@type": "Thing", "name": "IT", "url": "https://update.dsesecurity.com/topic/it/" } ], "wordCount": 517, "timeRequired": "PT3M", "publishingPrinciples": "https://update.dsesecurity.com/updates/dse-updates-editorial-methodology/", "usageInfo": "https://update.dsesecurity.com/usage/", "copyrightHolder": { "@id": "https://dsesecurity.com/#organization" }, "copyrightNotice": "Copyright © 2026 Detection Systems & Engineering. All rights reserved.", "citation": { "@type": "CreativeWork", "name": "Kubernetes — Handling retriable and non-retriable pod failures with Pod failure policy", "url": "https://kubernetes.io/docs/tasks/job/pod-failure-policy/" } } ] } }