VMware Cloud Director 10.3 Remote Console Common issues and how to resolve it

After upgrading my home lab to VMware Cloud Director 10.3, I started having Remote Console issues. While searching online, it sounds to affect many others and though I will share the two issues I have and how to resolve them.

The first one, I kept getting the following error when trying to access VMware Remote Console “Connection error: could not negotiate SSL.”, as shown below:

VMware Remote Console Connection error could not negotiate SSL

For those who are more of a Web Console users, I just kept getting a disconnected error every time I try to connect using it as shown below.

VMware Cloud Director Web Console Disconnected error

Alright, so now you have the same problem. How to resolve it? It sounds like in the later versions of VMware Cloud Director, it is becoming more restrictive on SSL Certificates and without a signed certificates the Remote Console does not seem to work properly. This was done to enhance security.

As I have found a very well written and detailed a blog post on how to replace your certs with Lets Encrypt Wildcard SSL certificates by Marc Roeleveld, I have decided against writing my own on the subject and point you to his excellent post that can be found at the link below. By the way, it’s worth mentioning that Lets Encrypt certificates are free of charge, and quite easy to get, so no excuse not to install a signed certificate even for your home lab.… Read More

VMware Cloud Director 10.3.2 is now GA

The release of VMware Cloud Director 10.3.2 brings some exciting new features for our Cloud providers. Here I will highlight the most anticipated ones, which I think many of my Cloud Providers will find appealing.

VMware Cloud Director GPU as a Service (GPUaaS)

Nvidia GPU as a Service in VMware Cloud Director

VMware Cloud Director GPU as a Service (GPUaaS) allows customers to use the processing power of NVIDIA managed vGPU by leveraging the VMware Cloud Director without the need of purchasing dedicated high-end GPUs. The capability is focused on general purpose AI/ML with compatible Nvidia hardware. Cloud Director multi-tenanted vGPU as a Service allows cloud providers to gain more margin using Nvidia MIG (Multi-instance GPU) which achieves multitenancy boundaries between workloads at the physical level inside a single device and is a big deal for multi-tenant environment.

Providers can offer vApp templates pre-configured with all the necessary sizing policies, placement policies, GPU Profiles with the VM and guest OS enabled for vGPU and configured in the templates. This will enable tenants to consume vGPU templates in a self-service fashion.

Cloud Provider admins can meter vGPU consumption per tenant for tenant billing via API and tenants are given usage visibility and management over their vGPU derived workloads. Note that Cloud Providers must own and manage their NVIDIA GPU estate, licensing, management application install and configurations.… Read More

VMware Cloud Director 10.3.x OS Configuration Phase has failed error

While I was installing VMware Cloud Director 10.3 in my home lab, I have received the error “OS Configuration Phase has failed.”. This happened as I tried to access the configuration page at https://<My VCD IP:5480 at the first reboot after deploying the appliance from an OVA.

Here is a screenshot of the actual error:

VMware Cloud Director 10.3 OS Configuration Phase has failed

Checking the /opt/vmware/var/log/firstboot as instructed in the error, and I found:

2021-10-27 17:33:16 | Updating /etc/hosts file ...
2021-10-27 17:33:16 | System hostname is: photon-machine
2021-10-27 17:33:16 | [ERROR] Hostname was not updated via dns reverse-lookup. Failing OS configuration phase.
2021-10-27 17:33:16 | [ERROR] Fix and verify DNS server info and other related networking settings, and redeploy this appliance again.

Here is a screenshot of the file output:

From the error, it was easy to spot it was a DNS issue. Checking my DNS server and I have found out that I have missed configuring the DNS record for my VCD instance. You will need to add a forward and reverse DNS records for you VCD Cells to your DNS server and this should fix the issue.

Though to share in case others face the problem as they get sloppy configuring their home lab late at night :).… Read More

VMWorld 2021 announced projects Summary

VMworld 2021 is here, and as you would expect many big announcements are in the making. As it’s easy to get lost between Arctic, Ensemble, Capitola, Cascade, Monterey, Radium projects announcements, so here is small summary about each one. It can help as well in case you missed one of them or looking for more details:-

VMware Project Arctic and Project Cascade announced at VMworld

Project Arctic:-

Project Arctic will bring multi-cloud to the fingertips of vSphere customers, by natively integrating cloud connectivity into vSphere – making vSphere cloud-aware, and making hybrid cloud the “default” operating model.  Millions of IT Admins who use vCenter on-premises will be able to leverage their domain expertise and benefit from expanded cloud capabilities. Customers would be able to instantly access VMware Cloud capacity and deploy VMware Cross-Cloud Services – for example, they would be able to enable disaster recovery to the cloud with just a few clicks, or leverage security services for threat detection and ransomware protection.

https://blogs.vmware.com/vsphere/2021/10/how-innovations-in-vsphere-are-redefining-infrastructure-to-run-future-apps.html

Project Cascade:-

Project Cascade will provide a unified Kubernetes interface for both on-demand infrastructure (IaaS) and containers (CaaS) across VMware Cloud – available through an open command line interface (CLI), APIs, or a GUI dashboard.  Project Cascade will address the needs of developers and DevOps in a multi-cloud world.… Read More

VMware Tanzu Mission Control (TMC) is now officially GA for VMware Cloud Provider Partners

On June 4, 2021 VMware Tanzu Mission Control (TMC) has become available to our VCPP Cloud Provider. It is a quite exciting news to many of our Managed Service Providers that are trying to offer Kubernetes added services across Multi-Cloud.

Tanzu Mission Control will help unlocks new opportunities for VMware cloud providers to offer Kubernetes (K8s) managed services for a multi-cloud. TMC can manage multi-tenant K8s clusters created on VMware Cloud Director (VCD) in addition to any CNCF conformant K8s clusters on public clouds as well. This includes the native K8s implementation at hyperscaler such as EKS, AKS, and GKE.

VMware Tanzu Mission Control

Here is a summary of the top features TMC will help our VMware Cloud Providers offer to their tenants:

  • Cluster lifecycle management: Provision, scale, upgrade and delete Tanzu Kubernetes Grid clusters via Tanzu Mission Control across environments.
  • Attaching clusters: Attach any conformant Kubernetes clusters running in other environments—either on-prem or in public clouds—to Tanzu Mission Control for centralized management.
  • Centralized policy management: Apply consistent policies, such as access, security, and even custom policies to a fleet of clusters and namespaces at scale.
  • Observability and diagnostics: Gain global observability of the health of clusters and workloads across clouds for quick diagnostics and troubleshooting.
Read More

VMware Cloud Director Cannot verify the Kubernetes API endpoint certificate on this Supervisor Cluster Error

While trying to Connect my VMware Cloud Director to my Tanzu Kubernetes Grid environment (TKGS) in vSphere 7 update 2, I kept hitting a certificate error. The error presented itself during the step to configure the Kubernetes policy for my Provider VDC. After following the wizard at:

Resources ==> Cloud Resources ==> Provider VDCs ==> Kubernetes.

The Wizard goes properly till I get to the Machine Classes tab which is where VMware Cloud Director get to check on the certificate of the supervisor cluster. That when the below error present itself:

“VMware Cloud Director cannot verify the Kubernetes API endpoint certificate on this Supervisor Cluster. It might be a vSphere generated default self-signed certificate or another invalid certificates. For the steps to install your own certificate, see Change the Kubernetes API Endpoint Certificate. For trusting the certificate, see https://kb.vmware.com/s/article/80996”

Kubernetes Policies in VCD 10.2 with vCenter 7.0 and later Tanzu are non-functional

The cause of the issue is the certificate structure of Tanzu Kubernetes in vCenter, The certificate of the Supervisor Cluster is not automatically trusted by VCD. Calls made to the Supervisor Cluster by VCD fail due to SSL errors. While I have this issue with VCD 10.2.2, it actually affected previous versions as well, especially when using self-signed certificates.

KB80996 article on how to fix this need to be updated with step marked in red below, but for those who needs to resolve the issue today, I wanted to document the steps in here.… Read More

VMware Cloud Providers Feature Fridays

My colleague Guy Bartram had been leading over 30 VMware Cloud Providers focused sessions where he host a different expert on each. I was hosted previously on the Bitnami and App Launchpad Feature Friday previously.

Feature Friday Bitnami and App Launchpad

I highly recommend every VMware Cloud Provider to take a look at these sessions and subscribe to the VMware Feature Fridays Youtube Channel.

Below are the list of feature Friday available as of the day this article is written. Please refer to the above link for the VMware Feature Fridays Youtube Chanel link above to keep updated with newer sessions:

Feature Friday Episode 33 – Terraform vCloud Director Provider 3.1 and NSX T

Terraform vCloud Director Provider 3.1 delivers infrastructure as code for VMware Cloud Director partners and their customers. Learn what is available in the new 3.1 release and particularly the support for NSX-T.

Feature Friday Episode 32 – VMware Cloud Director APIs

In this video I’m joined by Benoit Serratrice, Staff Cloud Solutions Architect at VMware to discuss VMware Cloud Director APIs and SDKs. Watch this session to understand some of the capabilities of the APIs available and the corresponding SDK coverage of the APIs, so you can make the right decision and save time programmatically automating your cloud solution.… Read More

What’s new in NSX Migration for VMware Cloud Director 1.2

As most of you are aware by now, VMware announced the sunsetting of NSX for vSphere (NSX-V), and the current end of general support is targeted for January 2022, while the end of technical guidance will be in January 2023. It is important that Cloud Providers migrate from NSX-V to NSX-T as soon as possible. As the process for larger providers will take a good amount of prep work and testing, the earlier they start the better their experience will be. The good news, the migration tools available this time is very helpful to streamline the process.

There are multiple methods to migrate from NSX for vSphere to NSX-T Data Center. They are listed below in the order of their usability in a VMware Cloud Director environment:

  • VMware NSX Migration for VMware Cloud Director can migrate the workload VMs and other organization VDC objects to the same vCenter Server instance managed by VMware Cloud Director. This is the option you really want to use if you have VMware Cloud Director in your environment. Here is the link for download. Here is the link as well for features supported by the migration tool.
  • Coexistence – New workloads are deployed on NSX-T, and the older workloads are allowed to die over time.
Read More

VMware Cloud Director Service Initial Availability

We have just announced the Cloud Director Service Initial Availability, where the first region would be VMC on AWS US West for now. There is a plan to expand this globally in the near future. Stay tune for a Cloud Director Service to be available in VMC on AWS region near you!!

So what is this all about? We are bringing the same Cloud Director our Cloud Providers and End customers love to VMC on AWS for now and other hyperscalers in the future. This will mainly be offered through our VMware Cloud Providers to their customers as a manged service. Here is few use cases I can see for this:

1- Asset Light Cloud Providers: New cloud providers who don’t want to operate or own their own datacenters, but still want to offer services based on our leading Cloud Provider stack. Now they can easily do that and have the infrastructure installed and managed by VMware!!

2- Geo Expansion: Cloud Providers who want to expand in a Geo where they don’t operate their own infrastructure. Again they can use Cloud Director Service in there to expand in that Geo. You can as well federate between your local Cloud Director and Cloud Director Service setups, so your customers don’t hub portals or have to worry where their service is coming form.… Read More

Cloud VMware Cloud Flix Series for Cloud Providers

The More you Know, the Greater your Impact! The VMware Cloud Provider Team have created a series of 60-minute business and technical sessions that will enable you to capture business opportunities in a Multi-Cloud World. During these webinars, you will gain information and context to better understand the rapidly evolving Cloud Services market and how VMware can help you be successful in delivering compelling new services to your end customers.

To learn more please join our VMware CloudFlix series, see specific topics listed below. There are 10 insightful sessions coming up where our VMware experts will deep dive into the new technical Cloud capabilities.

VMware Cloud Flix educational sessions for VMware Cloud Providers

The initial list is below, I will keep adding to it as we add more sessions:

– May, 12th : Extending SDWAN services into your cloud platform with VeloCloud – Registration -> https://lnkd.in/eDmGp6Y

– May, 14th : How to move from NSX-V to NSX-T using the migration tool – Registration -> https://lnkd.in/ePJ6E6w

– May, 19th : VCD 10.1 What’s New – Registration -> https://lnkd.in/eB29mSM

– May, 26th: Cloud Director Service – Registration -> https://lnkd.in/eFXdTAJ

– May, 28th: CloudFlix Webinar – Container Service Overview for Service Providers​ – Registration -> http://tiny.cc/o23spz

– June, 2nd: vSAN reset – Registration -> http://tiny.cc/8chtozRead More

Cloud Director Kubernetes as a Service with CSE 2.6.x Demo

If you have been following our VMware Cloud Provider Space for a while, you have probably been introduced to our Cloud Director Kubernetes as a Service offering based on VMware Container Service Extension. In the past, Container Service Extension used to be command line only, where a nice UI was introduced in CSE 2.6.0. Here is a demo of what the new UI of Cloud Director Container Service Extension look like out of the box:

If you are curious of how to install CSE 2.6.1, you can follow my earlier post: VMware Container Service Extension 2.6.1 Installation step by stepRead More

VMware Container Service Extension 2.6.1 Installation step by step

One of the most requested feature with previous versions of the VMware Container Service Extension (CSE) is to add a native UI to it. As of CSE 2.6 we have added a native UI to CSE, which is adding to the friendliness of CSE and will make it much more appealing to many of our cloud providers. At just few clicks, our customers can deploy a K8S clusters at our Cloud Providers with filling few easy to understand fields.

Kubeconfig file will be auto generated as well and can be handed out right away to the developer limiting the efforts required by the tenant operation team/cloud providers administrators. Here is a quick screenshot teaser of what CSE look like. You can find a nice demo of CSE 2.6.1 UI at my following blog post: Cloud Director Kubernetes as a Service with CSE 2.6.x Demo

For more info on what is new with CSE 2.6.x please check my following blog post: vCloud Director Container Service Extension 2.6.x is here

 

Container Service Extension Create New Cluster

 

In this post, I am assuming you have an existing vCloud Director environment and AMQP already configured. To start the installation of CSE 2.6.1, you will need a supported OS. In my case, I have decided to go with CentOS 8.1.Read More

Running vCD Cli fail with the following error: ModuleNotFoundError: No module named ‘_sqlite3’

After installing the VMware Container Service Extension, which install the vCD CLI in the process, vCD CLI kept failing to start and complaining about not finding sqlite3 module as showing below.  I was installing on CentOS 8.1, but even then it sounds like the sqlite version included with CentOS is out of date for what vCD CLI require to be installed. Below is what the error looked like.

# vcd --help
Traceback (most recent call last):
  File "/home/rohan/.local/lib/python3.7/site-packages/vcd_cli/browsercookie/__init__.py", line 18, in <module>
    from pysqlite2 import dbapi2 as sqlite3
ModuleNotFoundError: No module named 'pysqlite2'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/rohan/.local/bin/vcd", line 6, in <module>
    from vcd_cli.vcd import vcd
  File "/home/rohan/.local/lib/python3.7/site-packages/vcd_cli/vcd.py", line 121, in <module>
    from vcd_cli import login  # NOQA
  File "/home/rohan/.local/lib/python3.7/site-packages/vcd_cli/login.py", line 24, in <module>
    from vcd_cli import browsercookie
  File "/home/rohan/.local/lib/python3.7/site-packages/vcd_cli/browsercookie/__init__.py", line 20, in <module>
    import sqlite3
  File "/usr/local/lib/python3.7/sqlite3/__init__.py", line 23, in <module>
    from sqlite3.dbapi2 import *
  File "/usr/local/lib/python3.7/sqlite3/dbapi2.py", line 27, in <module>
    from _sqlite3 import *
ModuleNotFoundError: No module named '_sqlite3'

While the steps below is mainly focused on how to fix the issue to be able to install vCD or CSE, the same steps will be relevant to anyone having a similar ModuleNotFoundError: No module named ‘_sqlite3’ error in Python to install any other application.… Read More

CSE 2.6.1 Error: Default template my_template with revision 0 not found. Unable to start CSE server.

While trying to run my Container Service Extension 2.6.1 after a successful installation. I kept getting the following error when trying to run CSE “Default template my_template with revision 0 not found. Unable to start CSE server.”

To fix this you will need to:

  1. Edit your CSE config.yaml file to include the right name of the default template and revision number. (Much more on this below)
  2. Encrypt your CSE config file again
  3. Re-Run CSE with your encrypted config file

In this post I will explain this in a bit more details for those hitting the same issue, as the resolution is quite simple but might not be as obvious if you are doing this for the first time.

For a start, here is what the exact error look like:

[root@vtcse01 Python-3.7.3]# cse run --config encrypted-config.yaml
Required Python version: = 3.7.3
Installed Python version: 3.7.3 (default, May  4 2020, 15:36:31)
[GCC 8.3.1 20190507 (Red Hat 8.3.1-4)]
Password for config file decryption:
Decrypting 'encrypted-config.yaml'
Validating config file 'encrypted-config.yaml'
Connected to AMQP server (VTAMQP01.vt.com:5672)
InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised.
Connected to vCloud Director (vcd.vt.com:443)
Connected to vCenter Server 'vtvc01' as 'administrator@vsphere.local' (vtvc01.vt.com:None)
Config file 'encrypted-config.yaml' is valid
Loading k8s template definition from catalog
Found K8 template 'ubuntu-16.04_k8-1.15_weave-2.5.2'
Read More

Cloud Director App Launchpad Demo

Cloud Director App Launchpad enables Service Providers to offer a marketplace of applications within VMware Cloud Director. App Launchpad is a free plug-in for VMware Cloud Director that provides a user interface to easily access and launch applications from VMware Cloud Director content catalogs. Using App Launchpad, developers and DevOps engineers can launch applications to VMware Cloud Director in seconds.

 Bitnami supply over 120 pre-packaged Open Source applications that can be pushed to the App Launchpad marketplace in no time. It will allow VMware Cloud Providers to offer over 120 Open Source pre-packaged applications to their customers in no time.  

In this video, I will take you through a full demo of what the end tenant experience with the App Launchpad market place look in Cloud Director 10.1 look like, how to push a Bitnami application to App Launchpad, as well conclude with going through the Cloud Provider interface of the App Launchpad. 

If you are interested in how to deploy Cloud Director App Launchpad to the stage required before you can publish Bitnami applications to it as shown in the video, you can follow my previous post for a step by step instruction on how to deploy Cloud Director App Launchpad at: VMware Cloud Director App Launchpad 1.0Read More