VMware vCloud Director Stuck While Canceling Task

I have a customer who called me yesterday for having a task in vCloud Director that has been stuck in the cancelling stage for over a month. The problem has started after a power outage they had in their datacenter. vCloud Director is operating just fine, just that particular task was doing nothing and in reality it was cancelled but in the vCloud GUI it kept saying “Cancelling Copy” as shown in the below screenshot.

vCloud Director Stuck on Canceling Copy

My customer was bugged by this and wanted to get rid of this message in his vCloud Director GUI, which we were able to do successfully by following the below procedure. Please note the following procedure will take your vCloud Director Portal down for few minutes, so you probably want to schedule it during an outage.  At a high level what is the below procedure will be doing is gracefully shutting down the vCD Cells then bringing them up, which should clean up any stuck cancellation tasks. Below is a step by step of how to do so:

  1. Log in to the target vCloud Director cells using SSH as root.
  2. Navigate to $VCLOUD_HOME/bin/ (typically /opt/vmware/cloud-director /bin/ ).
  3. Retrieve the current job status. This cell-management-tool command line supplies system administrator credentials and returns the count of running jobs (i.e. Job count 3, Is Active = true):

 ./cell-management-tool -u administrator -p Pa55w0rd cell –status

  1. Stop the task scheduler to quiesce the cell.Use this cell-management-tool command:
            ./cell-management-tool -u administrator -p Pa55w0rd cell –quiesce true

 

Note: This command prevents new jobs from being started. Existing jobs continue to run until they complete or are cancelled. To cancel a job, use the vCloud Director Web Console or the REST API.

 

  1. When the Job count is 0 and Is Active is false, it is safe to shut down the cell. Keep executing the command in step 4 till these conditions are met then use the below command to stop the Cell:

service vmware-vcd stop

Note: You can run “service vmware-vcd status” to ensure that the vcd cell was stopped properly.

  1. Repeat the above steps to other vCloud Directors Cells involved in your setup
  2. Start your vCD Cells again using the following command (Repeat this for all of your vCD Cells): service vmware-vcd start

Now you should be able to login back to your vCloud Director Cells and should have all these Stucked While Canceling tasks vanished from your web interface. Hope you find this useful!

Comments

  1. Hi there, what is the best way to automatically schedule a shutdown and restart of vCD management server layer (vCD 5.5, vShield, and vCenter)? I understand the shutdown order. This is regarding the vCD management servers are all associated to a single MS SQL server. There is a requirement to have ms updates applied to the micorsoft the SQL server.

Speak Your Mind

*