Skip to content

CKS: fix NPE when remove a failed external node #12407

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
weizhouapache wants to merge 2 commits into
base: 4.22
Choose a base branch
from
Open

CKS: fix NPE when remove a failed external node #12407

weizhouapache wants to merge 2 commits into from
+1 −1

Conversation

@weizhouapache
Copy link
Member

@weizhouapache weizhouapache commented Jan 12, 2026

Description

This PR fixes #11581

Types of changes

  • Breaking change (fix or feature that would cause existing functionality to change)
  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (improves an existing feature and functionality)
  • Cleanup (Code refactoring and cleanup, that may add test cases)
  • Build/CI
  • Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

  • Major
  • Minor

Bug Severity

  • BLOCKER
  • Critical
  • Major
  • Minor
  • Trivial

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

@weizhouapache weizhouapache force-pushed the 4.22-cks-remove-failed-external-node branch from 9750c2a to 415ebb1 Compare January 12, 2026 12:22
@github-actions
Copy link

github-actions bot commented Jan 12, 2026

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

@weizhouapache weizhouapache changed the base branch from 4.22 to 4.20 January 12, 2026 12:22
@weizhouapache weizhouapache added this to the 4.20.3 milestone Jan 12, 2026
@weizhouapache
Copy link
Member Author

weizhouapache commented Jan 12, 2026

@blueorangutan package

@blueorangutan
Copy link

blueorangutan commented Jan 12, 2026

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@DaanHoogland DaanHoogland linked an issue Jan 12, 2026 that may be closed by this pull request
CKS: NPE when trying to remove a external node from a cks cluster #11581
Open
@weizhouapache weizhouapache force-pushed the 4.22-cks-remove-failed-external-node branch from 415ebb1 to fb18a23 Compare January 12, 2026 12:44
@github-actions
Copy link

github-actions bot commented Jan 12, 2026

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

@weizhouapache weizhouapache changed the base branch from 4.20 to 4.22 January 12, 2026 12:44
@weizhouapache weizhouapache reopened this Jan 12, 2026
@blueorangutan
Copy link

blueorangutan commented Jan 12, 2026

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 16336

@codecov
Copy link

codecov bot commented Jan 12, 2026 •
edited
Loading

Codecov Report

❌ Patch coverage is 0% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 17.59%. Comparing base (2399edd) to head (e166703).
⚠️ Report is 8 commits behind head on 4.22.

Files with missing lines Patch % Lines
...bernetes/cluster/KubernetesClusterManagerImpl.java 0.00% 1 Missing ⚠️
Additional details and impacted files 
@@            Coverage Diff            @@
##               4.22   #12407   +/-   ##
=========================================
  Coverage     17.59%   17.59%           
+ Complexity    15600    15599    -1     
=========================================
  Files          5910     5910           
  Lines        529733   529755   +22     
  Branches      64719    64724    +5     
=========================================
+ Hits          93218    93227    +9     
- Misses       426023   426035   +12     
- Partials      10492    10493    +1
Flag Coverage Δ
uitests 3.60% <ø> (+0.02%) ⬆️
unittests 18.66% <0.00%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@weizhouapache
Copy link
Member Author

weizhouapache commented Jan 13, 2026

@blueorangutan package

@blueorangutan
Copy link

blueorangutan commented Jan 13, 2026

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan
Copy link

blueorangutan commented Jan 13, 2026

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 16345

@weizhouapache
Copy link
Member Author

weizhouapache commented Jan 13, 2026

@blueorangutan test

@blueorangutan
Copy link

blueorangutan commented Jan 13, 2026

@weizhouapache a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

@blueorangutan
Copy link

blueorangutan commented Jan 14, 2026

[SF] Trillian test result (tid-15176)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 55022 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr12407-t15176-kvm-ol8.zip
Smoke tests completed. 148 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test Result Time (s) Test File
test_03_deploy_and_scale_kubernetes_cluster Failure 28.67 test_kubernetes_clusters.py

vishesh92
vishesh92 approved these changes Jan 14, 2026
View reviewed changes
Copy link
Member

@vishesh92 vishesh92 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clgtm

@vishesh92
Copy link
Member

vishesh92 commented Jan 14, 2026

@blueorangutan package

@blueorangutan
Copy link

blueorangutan commented Jan 14, 2026

@vishesh92 a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan
Copy link

blueorangutan commented Jan 14, 2026

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 16359

@weizhouapache
Copy link
Member Author

weizhouapache commented Jan 15, 2026

@blueorangutan test

@blueorangutan
Copy link

blueorangutan commented Jan 15, 2026

@weizhouapache a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

@blueorangutan
Copy link

blueorangutan commented Jan 15, 2026

[SF] Trillian Build Failed (tid-15188)

@DaanHoogland
Copy link
Contributor

DaanHoogland commented Jan 15, 2026

@weizhouapache the issue and PR are marked for 20.3 but the branch is based off of 4.22. What should happen?

@DaanHoogland DaanHoogland modified the milestones: 4.20.3, 4.22.1 Jan 16, 2026
@vladimirpetrov vladimirpetrov self-assigned this Jan 28, 2026
@weizhouapache weizhouapache marked this pull request as ready for review January 28, 2026 09:27
kiranchavala
kiranchavala reviewed Jan 28, 2026
View reviewed changes
Copy link
Contributor

@kiranchavala kiranchavala left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@weizhouapache

Issue is still present

  1. Create a cks cluster

  2. Add a external node to the cks cluster

  3. The external node fails to add due to space issues

Screenshot 2026-01-28 at 3 18 19 PM

  1. Cluster running state

  2. Try to remove the external node

Exception

Screenshot 2026-01-28 at 3 52 35 PM

logs

[root@ref-trl-10752-k-Mol8-kiran-chavala-mgmt1 ~]# cat  /var/log/cloudstack/management/management-server.log  |grep -i "job-47"
2026-01-28 10:27:01,000 INFO  [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-2:[ctx-7c702c5d, job-47]) (logid:9e9c5bd4) Add job-47 into job monitoring
2026-01-28 10:27:01,003 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (qtp1390913202-17:[ctx-0b3912d3, ctx-14f879bd]) (logid:97df2239) submit async job-47, details: AsyncJob {"accountId":2,"cmd":"org.apache.cloudstack.api.command.user.kubernetes.cluster.RemoveNodesFromKubernetesClusterCmd","cmdInfo":"{\"response\":\"json\",\"ctxUserId\":\"2\",\"sessionkey\":\"mnK0SnDiHU-7yH8r_GeAGoLWmVk\",\"httpmethod\":\"POST\",\"ctxStartEventId\":\"207\",\"id\":\"e2889f42-b5f7-4732-864a-b27c5525c88b\",\"ctxDetails\":\"{\\\"interface com.cloud.vm.VirtualMachine\\\":\\\"9dfc270b-4924-4114-83d2-4cf9117435c5\\\",\\\"interface com.cloud.kubernetes.cluster.KubernetesCluster\\\":\\\"e2889f42-b5f7-4732-864a-b27c5525c88b\\\"}\",\"ctxAccountId\":\"2\",\"uuid\":\"e2889f42-b5f7-4732-864a-b27c5525c88b\",\"nodeids\":\"9dfc270b-4924-4114-83d2-4cf9117435c5\",\"cmdEventType\":\"KUBERNETES.CLUSTER.NODES.REMOVE\"}","cmdVersion":0,"completeMsid":null,"created":null,"id":47,"initMsid":32985365610879,"instanceId":1,"instanceType":"KubernetesCluster","lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":2,"uuid":"dbee196a-4267-4a76-9a50-2eb1cfa82a9b"}
2026-01-28 10:27:01,004 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (API-Job-Executor-2:[ctx-7c702c5d, job-47]) (logid:dbee196a) Executing AsyncJob {"accountId":2,"cmd":"org.apache.cloudstack.api.command.user.kubernetes.cluster.RemoveNodesFromKubernetesClusterCmd","cmdInfo":"{\"response\":\"json\",\"ctxUserId\":\"2\",\"sessionkey\":\"mnK0SnDiHU-7yH8r_GeAGoLWmVk\",\"httpmethod\":\"POST\",\"ctxStartEventId\":\"207\",\"id\":\"e2889f42-b5f7-4732-864a-b27c5525c88b\",\"ctxDetails\":\"{\\\"interface com.cloud.vm.VirtualMachine\\\":\\\"9dfc270b-4924-4114-83d2-4cf9117435c5\\\",\\\"interface com.cloud.kubernetes.cluster.KubernetesCluster\\\":\\\"e2889f42-b5f7-4732-864a-b27c5525c88b\\\"}\",\"ctxAccountId\":\"2\",\"uuid\":\"e2889f42-b5f7-4732-864a-b27c5525c88b\",\"nodeids\":\"9dfc270b-4924-4114-83d2-4cf9117435c5\",\"cmdEventType\":\"KUBERNETES.CLUSTER.NODES.REMOVE\"}","cmdVersion":0,"completeMsid":null,"created":null,"id":47,"initMsid":32985365610879,"instanceId":1,"instanceType":"KubernetesCluster","lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":2,"uuid":"dbee196a-4267-4a76-9a50-2eb1cfa82a9b"}
2026-01-28 10:27:05,108 ERROR [c.c.k.c.a.KubernetesClusterRemoveWorker] (API-Job-Executor-2:[ctx-7c702c5d, job-47, ctx-a5f8ed50]) (logid:dbee196a) Error trying to remove node 9dfc270b-4924-4114-83d2-4cf9117435c5 from Kubernetes Cluster e2889f42-b5f7-4732-864a-b27c5525c88b: Error during SCP transfer. com.cloud.utils.exception.CloudRuntimeException: Error during SCP transfer.
2026-01-28 10:27:05,121 ERROR [o.a.c.a.c.u.k.c.RemoveNodesFromKubernetesClusterCmd] (API-Job-Executor-2:[ctx-7c702c5d, job-47, ctx-a5f8ed50]) (logid:dbee196a) Failed to remove node(s) from Kubernetes cluster ID: 1 due to: Failed to remove node(s) from Kubernetes cluster ID: 1 org.apache.cloudstack.api.ServerApiException: Failed to remove node(s) from Kubernetes cluster ID: 1
2026-01-28 10:27:05,122 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-2:[ctx-7c702c5d, job-47]) (logid:dbee196a) Complete async job-47, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to remove node(s) from Kubernetes cluster ID: 1 due to: Failed to remove node(s) from Kubernetes cluster ID: 1"}
2026-01-28 10:27:05,123 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-2:[ctx-7c702c5d, job-47]) (logid:dbee196a) Publish async job-47 complete on message bus
2026-01-28 10:27:05,123 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-2:[ctx-7c702c5d, job-47]) (logid:dbee196a) Wake up jobs related to job-47
2026-01-28 10:27:05,123 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-2:[ctx-7c702c5d, job-47]) (logid:dbee196a) Update db status for job-47
2026-01-28 10:27:05,124 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-2:[ctx-7c702c5d, job-47]) (logid:dbee196a) Wake up jobs joined with job-47 and disjoin all subjobs created from job- 47
2026-01-28 10:27:05,128 DEBUG [c.c.a.ApiServer] (API-Job-Executor-2:[ctx-7c702c5d, job-47]) (logid:dbee196a) Retrieved cmdEventType from job info: KUBERNETES.CLUSTER.NODES.REMOVE
2026-01-28 10:27:05,130 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (API-Job-Executor-2:[ctx-7c702c5d, job-47]) (logid:dbee196a) Done executing org.apache.cloudstack.api.command.user.kubernetes.cluster.RemoveNodesFromKubernetesClusterCmd for job-47
2026-01-28 10:27:05,130 INFO  [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-2:[ctx-7c702c5d, job-47]) (logid:dbee196a) Remove job-47 from job monitoring

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kiranchavala kiranchavala kiranchavala left review comments

@DaanHoogland DaanHoogland DaanHoogland left review comments

@vishesh92 vishesh92 vishesh92 approved these changes

Assignees

@kiranchavala kiranchavala

Labels

component:agent component:build component:integration component:kubernetes component:kvm component:primary-storage component:storage component:UI

Projects

None yet

Milestone

4.22.1

Development

Successfully merging this pull request may close these issues.

CKS: NPE when trying to remove a external node from a cks cluster

6 participants

@weizhouapache @blueorangutan @vishesh92 @DaanHoogland @kiranchavala @vladimirpetrov