VMware Cloud Director 10.3.x OS Configuration Phase has failed error

While I was installing VMware Cloud Director 10.3 in my home lab, I have received the error “OS Configuration Phase has failed.”. This happened as I tried to access the configuration page at https://<My VCD IP:5480 at the first reboot after deploying the appliance from an OVA.

Here is a screenshot of the actual error:

VMware Cloud Director 10.3 OS Configuration Phase has failed

Checking the /opt/vmware/var/log/firstboot as instructed in the error, and I found:

2021-10-27 17:33:16 | Updating /etc/hosts file ...
2021-10-27 17:33:16 | System hostname is: photon-machine
2021-10-27 17:33:16 | [ERROR] Hostname was not updated via dns reverse-lookup. Failing OS configuration phase.
2021-10-27 17:33:16 | [ERROR] Fix and verify DNS server info and other related networking settings, and redeploy this appliance again.

Here is a screenshot of the file output:

From the error, it was easy to spot it was a DNS issue. Checking my DNS server and I have found out that I have missed configuring the DNS record for my VCD instance. You will need to add a forward and reverse DNS records for you VCD Cells to your DNS server and this should fix the issue.

Though to share in case others face the problem as they get sloppy configuring their home lab late at night :).… Read More

VMWorld 2021 announced projects Summary

VMworld 2021 is here, and as you would expect many big announcements are in the making. As it’s easy to get lost between Arctic, Ensemble, Capitola, Cascade, Monterey, Radium projects announcements, so here is small summary about each one. It can help as well in case you missed one of them or looking for more details:-

VMware Project Arctic and Project Cascade announced at VMworld

Project Arctic:-

Project Arctic will bring multi-cloud to the fingertips of vSphere customers, by natively integrating cloud connectivity into vSphere – making vSphere cloud-aware, and making hybrid cloud the “default” operating model.  Millions of IT Admins who use vCenter on-premises will be able to leverage their domain expertise and benefit from expanded cloud capabilities. Customers would be able to instantly access VMware Cloud capacity and deploy VMware Cross-Cloud Services – for example, they would be able to enable disaster recovery to the cloud with just a few clicks, or leverage security services for threat detection and ransomware protection.


Project Cascade:-

Project Cascade will provide a unified Kubernetes interface for both on-demand infrastructure (IaaS) and containers (CaaS) across VMware Cloud – available through an open command line interface (CLI), APIs, or a GUI dashboard.  Project Cascade will address the needs of developers and DevOps in a multi-cloud world.… Read More

VMware Tanzu Mission Control (TMC) is now officially GA for VMware Cloud Provider Partners

On June 4, 2021 VMware Tanzu Mission Control (TMC) has become available to our VCPP Cloud Provider. It is a quite exciting news to many of our Managed Service Providers that are trying to offer Kubernetes added services across Multi-Cloud.

Tanzu Mission Control will help unlocks new opportunities for VMware cloud providers to offer Kubernetes (K8s) managed services for a multi-cloud. TMC can manage multi-tenant K8s clusters created on VMware Cloud Director (VCD) in addition to any CNCF conformant K8s clusters on public clouds as well. This includes the native K8s implementation at hyperscaler such as EKS, AKS, and GKE.

VMware Tanzu Mission Control

Here is a summary of the top features TMC will help our VMware Cloud Providers offer to their tenants:

  • Cluster lifecycle management: Provision, scale, upgrade and delete Tanzu Kubernetes Grid clusters via Tanzu Mission Control across environments.
  • Attaching clusters: Attach any conformant Kubernetes clusters running in other environments—either on-prem or in public clouds—to Tanzu Mission Control for centralized management.
  • Centralized policy management: Apply consistent policies, such as access, security, and even custom policies to a fleet of clusters and namespaces at scale.
  • Observability and diagnostics: Gain global observability of the health of clusters and workloads across clouds for quick diagnostics and troubleshooting.
Read More

VMware Cloud Director Cannot verify the Kubernetes API endpoint certificate on this Supervisor Cluster Error

While trying to Connect my VMware Cloud Director to my Tanzu Kubernetes Grid environment (TKGS) in vSphere 7 update 2, I kept hitting a certificate error. The error presented itself during the step to configure the Kubernetes policy for my Provider VDC. After following the wizard at:

Resources ==> Cloud Resources ==> Provider VDCs ==> Kubernetes.

The Wizard goes properly till I get to the Machine Classes tab which is where VMware Cloud Director get to check on the certificate of the supervisor cluster. That when the below error present itself:

“VMware Cloud Director cannot verify the Kubernetes API endpoint certificate on this Supervisor Cluster. It might be a vSphere generated default self-signed certificate or another invalid certificates. For the steps to install your own certificate, see Change the Kubernetes API Endpoint Certificate. For trusting the certificate, see https://kb.vmware.com/s/article/80996”

Kubernetes Policies in VCD 10.2 with vCenter 7.0 and later Tanzu are non-functional

The cause of the issue is the certificate structure of Tanzu Kubernetes in vCenter, The certificate of the Supervisor Cluster is not automatically trusted by VCD. Calls made to the Supervisor Cluster by VCD fail due to SSL errors. While I have this issue with VCD 10.2.2, it actually affected previous versions as well, especially when using self-signed certificates.

KB80996 article on how to fix this need to be updated with step marked in red below, but for those who needs to resolve the issue today, I wanted to document the steps in here.… Read More

VMware Cloud Providers Feature Fridays

My colleague Guy Bartram had been leading over 30 VMware Cloud Providers focused sessions where he host a different expert on each. I was hosted previously on the Bitnami and App Launchpad Feature Friday previously.

Feature Friday Bitnami and App Launchpad

I highly recommend every VMware Cloud Provider to take a look at these sessions and subscribe to the VMware Feature Fridays Youtube Channel.

Below are the list of feature Friday available as of the day this article is written. Please refer to the above link for the VMware Feature Fridays Youtube Chanel link above to keep updated with newer sessions:

Feature Friday Episode 33 – Terraform vCloud Director Provider 3.1 and NSX T

Terraform vCloud Director Provider 3.1 delivers infrastructure as code for VMware Cloud Director partners and their customers. Learn what is available in the new 3.1 release and particularly the support for NSX-T.

Feature Friday Episode 32 – VMware Cloud Director APIs

In this video I’m joined by Benoit Serratrice, Staff Cloud Solutions Architect at VMware to discuss VMware Cloud Director APIs and SDKs. Watch this session to understand some of the capabilities of the APIs available and the corresponding SDK coverage of the APIs, so you can make the right decision and save time programmatically automating your cloud solution.… Read More

What’s new in NSX Migration for VMware Cloud Director 1.2

As most of you are aware by now, VMware announced the sunsetting of NSX for vSphere (NSX-V), and the current end of general support is targeted for January 2022, while the end of technical guidance will be in January 2023. It is important that Cloud Providers migrate from NSX-V to NSX-T as soon as possible. As the process for larger providers will take a good amount of prep work and testing, the earlier they start the better their experience will be. The good news, the migration tools available this time is very helpful to streamline the process.

There are multiple methods to migrate from NSX for vSphere to NSX-T Data Center. They are listed below in the order of their usability in a VMware Cloud Director environment:

  • VMware NSX Migration for VMware Cloud Director can migrate the workload VMs and other organization VDC objects to the same vCenter Server instance managed by VMware Cloud Director. This is the option you really want to use if you have VMware Cloud Director in your environment. Here is the link for download. Here is the link as well for features supported by the migration tool.
  • Coexistence – New workloads are deployed on NSX-T, and the older workloads are allowed to die over time.
Read More

VMware Cloud Director Service Initial Availability

We have just announced the Cloud Director Service Initial Availability, where the first region would be VMC on AWS US West for now. There is a plan to expand this globally in the near future. Stay tune for a Cloud Director Service to be available in VMC on AWS region near you!!

So what is this all about? We are bringing the same Cloud Director our Cloud Providers and End customers love to VMC on AWS for now and other hyperscalers in the future. This will mainly be offered through our VMware Cloud Providers to their customers as a manged service. Here is few use cases I can see for this:

1- Asset Light Cloud Providers: New cloud providers who don’t want to operate or own their own datacenters, but still want to offer services based on our leading Cloud Provider stack. Now they can easily do that and have the infrastructure installed and managed by VMware!!

2- Geo Expansion: Cloud Providers who want to expand in a Geo where they don’t operate their own infrastructure. Again they can use Cloud Director Service in there to expand in that Geo. You can as well federate between your local Cloud Director and Cloud Director Service setups, so your customers don’t hub portals or have to worry where their service is coming form.… Read More

Cloud VMware Cloud Flix Series for Cloud Providers

The More you Know, the Greater your Impact! The VMware Cloud Provider Team have created a series of 60-minute business and technical sessions that will enable you to capture business opportunities in a Multi-Cloud World. During these webinars, you will gain information and context to better understand the rapidly evolving Cloud Services market and how VMware can help you be successful in delivering compelling new services to your end customers.

To learn more please join our VMware CloudFlix series, see specific topics listed below. There are 10 insightful sessions coming up where our VMware experts will deep dive into the new technical Cloud capabilities.

VMware Cloud Flix educational sessions for VMware Cloud Providers

The initial list is below, I will keep adding to it as we add more sessions:

– May, 12th : Extending SDWAN services into your cloud platform with VeloCloud – Registration -> https://lnkd.in/eDmGp6Y

– May, 14th : How to move from NSX-V to NSX-T using the migration tool – Registration -> https://lnkd.in/ePJ6E6w

– May, 19th : VCD 10.1 What’s New – Registration -> https://lnkd.in/eB29mSM

– May, 26th: Cloud Director Service – Registration -> https://lnkd.in/eFXdTAJ

– May, 28th: CloudFlix Webinar – Container Service Overview for Service Providers​ – Registration -> http://tiny.cc/o23spz

– June, 2nd: vSAN reset – Registration -> http://tiny.cc/8chtozRead More

Cloud Director Kubernetes as a Service with CSE 2.6.x Demo

If you have been following our VMware Cloud Provider Space for a while, you have probably been introduced to our Cloud Director Kubernetes as a Service offering based on VMware Container Service Extension. In the past, Container Service Extension used to be command line only, where a nice UI was introduced in CSE 2.6.0. Here is a demo of what the new UI of Cloud Director Container Service Extension look like out of the box:

If you are curious of how to install CSE 2.6.1, you can follow my earlier post: VMware Container Service Extension 2.6.1 Installation step by stepRead More

VMware Container Service Extension 2.6.1 Installation step by step

One of the most requested feature with previous versions of the VMware Container Service Extension (CSE) is to add a native UI to it. As of CSE 2.6 we have added a native UI to CSE, which is adding to the friendliness of CSE and will make it much more appealing to many of our cloud providers. At just few clicks, our customers can deploy a K8S clusters at our Cloud Providers with filling few easy to understand fields.

Kubeconfig file will be auto generated as well and can be handed out right away to the developer limiting the efforts required by the tenant operation team/cloud providers administrators. Here is a quick screenshot teaser of what CSE look like. You can find a nice demo of CSE 2.6.1 UI at my following blog post: Cloud Director Kubernetes as a Service with CSE 2.6.x Demo

For more info on what is new with CSE 2.6.x please check my following blog post: vCloud Director Container Service Extension 2.6.x is here


Container Service Extension Create New Cluster


In this post, I am assuming you have an existing vCloud Director environment and AMQP already configured. To start the installation of CSE 2.6.1, you will need a supported OS. In my case, I have decided to go with CentOS 8.1.Read More

Running vCD Cli fail with the following error: ModuleNotFoundError: No module named ‘_sqlite3’

After installing the VMware Container Service Extension, which install the vCD CLI in the process, vCD CLI kept failing to start and complaining about not finding sqlite3 module as showing below.  I was installing on CentOS 8.1, but even then it sounds like the sqlite version included with CentOS is out of date for what vCD CLI require to be installed. Below is what the error looked like.

# vcd --help
Traceback (most recent call last):
  File "/home/rohan/.local/lib/python3.7/site-packages/vcd_cli/browsercookie/__init__.py", line 18, in <module>
    from pysqlite2 import dbapi2 as sqlite3
ModuleNotFoundError: No module named 'pysqlite2'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/rohan/.local/bin/vcd", line 6, in <module>
    from vcd_cli.vcd import vcd
  File "/home/rohan/.local/lib/python3.7/site-packages/vcd_cli/vcd.py", line 121, in <module>
    from vcd_cli import login  # NOQA
  File "/home/rohan/.local/lib/python3.7/site-packages/vcd_cli/login.py", line 24, in <module>
    from vcd_cli import browsercookie
  File "/home/rohan/.local/lib/python3.7/site-packages/vcd_cli/browsercookie/__init__.py", line 20, in <module>
    import sqlite3
  File "/usr/local/lib/python3.7/sqlite3/__init__.py", line 23, in <module>
    from sqlite3.dbapi2 import *
  File "/usr/local/lib/python3.7/sqlite3/dbapi2.py", line 27, in <module>
    from _sqlite3 import *
ModuleNotFoundError: No module named '_sqlite3'

While the steps below is mainly focused on how to fix the issue to be able to install vCD or CSE, the same steps will be relevant to anyone having a similar ModuleNotFoundError: No module named ‘_sqlite3’ error in Python to install any other application.… Read More

CSE 2.6.1 Error: Default template my_template with revision 0 not found. Unable to start CSE server.

While trying to run my Container Service Extension 2.6.1 after a successful installation. I kept getting the following error when trying to run CSE “Default template my_template with revision 0 not found. Unable to start CSE server.”

To fix this you will need to:

  1. Edit your CSE config.yaml file to include the right name of the default template and revision number. (Much more on this below)
  2. Encrypt your CSE config file again
  3. Re-Run CSE with your encrypted config file

In this post I will explain this in a bit more details for those hitting the same issue, as the resolution is quite simple but might not be as obvious if you are doing this for the first time.

For a start, here is what the exact error look like:

[root@vtcse01 Python-3.7.3]# cse run --config encrypted-config.yaml
Required Python version: = 3.7.3
Installed Python version: 3.7.3 (default, May  4 2020, 15:36:31)
[GCC 8.3.1 20190507 (Red Hat 8.3.1-4)]
Password for config file decryption:
Decrypting 'encrypted-config.yaml'
Validating config file 'encrypted-config.yaml'
Connected to AMQP server (VTAMQP01.vt.com:5672)
InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised.
Connected to vCloud Director (vcd.vt.com:443)
Connected to vCenter Server 'vtvc01' as 'administrator@vsphere.local' (vtvc01.vt.com:None)
Config file 'encrypted-config.yaml' is valid
Loading k8s template definition from catalog
Found K8 template 'ubuntu-16.04_k8-1.15_weave-2.5.2'
Read More

Cloud Director App Launchpad Demo

Cloud Director App Launchpad enables Service Providers to offer a marketplace of applications within VMware Cloud Director. App Launchpad is a free plug-in for VMware Cloud Director that provides a user interface to easily access and launch applications from VMware Cloud Director content catalogs. Using App Launchpad, developers and DevOps engineers can launch applications to VMware Cloud Director in seconds.

 Bitnami supply over 120 pre-packaged Open Source applications that can be pushed to the App Launchpad marketplace in no time. It will allow VMware Cloud Providers to offer over 120 Open Source pre-packaged applications to their customers in no time.  

In this video, I will take you through a full demo of what the end tenant experience with the App Launchpad market place look in Cloud Director 10.1 look like, how to push a Bitnami application to App Launchpad, as well conclude with going through the Cloud Provider interface of the App Launchpad. 

If you are interested in how to deploy Cloud Director App Launchpad to the stage required before you can publish Bitnami applications to it as shown in the video, you can follow my previous post for a step by step instruction on how to deploy Cloud Director App Launchpad at: VMware Cloud Director App Launchpad 1.0Read More

VMware Cloud Architecture Toolkit 5.0 is here!!

Thanks to the great efforts of my colleague Martin Hosken, VMware Cloud Architecture Toolkit (Known as vCAT) is updated to version 5.0. vCAT has always been a great guide to our Cloud Providers on how to best architect/build their Cloud Providers stack. vCAT 5.0 is no exception and include tons valuable information for our Cloud Providers. Highly recommended!

In this new VMware Cloud Provider Platform guide, you will learn how to take full advantage of VMware Cloud Provider Platform, understand how it works and how it can be designed with this comprehensive eBook. You can access vCAT 5.0 for free at the following link: vCAT 5.0 download or by clicking the image of the book cover below.

Read More

VMware Cloud Director App Launchpad 1.0 Step by Step Installation

As the VMware Cloud Director has been evolving and adding features to help cloud providers better serve their customers, and help them target new markets, we are introducing Cloud Director App Launchpad bundled as a free extension with Cloud Director 10.1. App Launchpad will allow the Cloud provider to offer a Curated catalog of applications for vCD tenants, where they can deploy applications at a single click. The end user will be able to deploy these applications without having to understand the infrastructure it hosts it, networking configuration, firewall, Public IP or so on, as all of it will be automated in the back-end.

As VMware had acquired Bitnami (The leader of Packaged Open Source Applications) a bit over a year ago, it makes no surprise to anyone that Bitnami provides over ~130 Open Source Applications ready to serve to end tenants in the Cloud Director App Launchpad. That’s not all, as beyond the Bitnami, Cloud Director App Launchpad support another two sources for applications:

  • ISV apps from VMware Cloud Marketplace
  • In-house apps

This will allow our cloud providers to have a nicely populated catalog from day one as well the freedom to add their own custom offering to it. Before starting on the installation instructions, the below screenshot demonstrate what the marketplace of App Launchpad look like:

You can check out my following post for a nice detailed demo of VMware Cloud Director App Launchpad at: Cloud Director App Launchpad Demo

Now you know how nice it looks like, let’s get started on how to get it up and running.Read More