CSE 2.6.1 Error: Default template my_template with revision 0 not found. Unable to start CSE server.

While trying to run my Container Service Extension 2.6.1 after a successful installation. I kept getting the following error when trying to run CSE “Default template my_template with revision 0 not found. Unable to start CSE server.”

To fix this you will need to:

  1. Edit your CSE config.yaml file to include the right name of the default template and revision number. (Much more on this below)
  2. Encrypt your CSE config file again
  3. Re-Run CSE with your encrypted config file

In this post I will explain this in a bit more details for those hitting the same issue, as the resolution is quite simple but might not be as obvious if you are doing this for the first time.

For a start, here is what the exact error look like:

[root@vtcse01 Python-3.7.3]# cse run --config encrypted-config.yaml
Required Python version: = 3.7.3
Installed Python version: 3.7.3 (default, May  4 2020, 15:36:31)
[GCC 8.3.1 20190507 (Red Hat 8.3.1-4)]
Password for config file decryption:
Decrypting 'encrypted-config.yaml'
Validating config file 'encrypted-config.yaml'
Connected to AMQP server (VTAMQP01.vt.com:5672)
InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised.
Connected to vCloud Director (vcd.vt.com:443)
Connected to vCenter Server 'vtvc01' as 'administrator@vsphere.local' (vtvc01.vt.com:None)
Config file 'encrypted-config.yaml' is valid
Loading k8s template definition from catalog
Found K8 template 'ubuntu-16.04_k8-1.15_weave-2.5.2'
Read More

Cloud Director App Launchpad Demo

Cloud Director App Launchpad enables Service Providers to offer a marketplace of applications within VMware Cloud Director. App Launchpad is a free plug-in for VMware Cloud Director that provides a user interface to easily access and launch applications from VMware Cloud Director content catalogs. Using App Launchpad, developers and DevOps engineers can launch applications to VMware Cloud Director in seconds.

 Bitnami supply over 120 pre-packaged Open Source applications that can be pushed to the App Launchpad marketplace in no time. It will allow VMware Cloud Providers to offer over 120 Open Source pre-packaged applications to their customers in no time.  

In this video, I will take you through a full demo of what the end tenant experience with the App Launchpad market place look in Cloud Director 10.1 look like, how to push a Bitnami application to App Launchpad, as well conclude with going through the Cloud Provider interface of the App Launchpad. 

If you are interested in how to deploy Cloud Director App Launchpad to the stage required before you can publish Bitnami applications to it as shown in the video, you can follow my previous post for a step by step instruction on how to deploy Cloud Director App Launchpad at: VMware Cloud Director App Launchpad 1.0Read More

VMware Cloud Architecture Toolkit 5.0 is here!!

Thanks to the great efforts of my colleague Martin Hosken, VMware Cloud Architecture Toolkit (Known as vCAT) is updated to version 5.0. vCAT has always been a great guide to our Cloud Providers on how to best architect/build their Cloud Providers stack. vCAT 5.0 is no exception and include tons valuable information for our Cloud Providers. Highly recommended!

In this new VMware Cloud Provider Platform guide, you will learn how to take full advantage of VMware Cloud Provider Platform, understand how it works and how it can be designed with this comprehensive eBook. You can access vCAT 5.0 for free at the following link: vCAT 5.0 download or by clicking the image of the book cover below.

Read More

VMware Cloud Director App Launchpad 1.0 Step by Step Installation

As the VMware Cloud Director has been evolving and adding features to help cloud providers better serve their customers, and help them target new markets, we are introducing Cloud Director App Launchpad bundled as a free extension with Cloud Director 10.1. App Launchpad will allow the Cloud provider to offer a Curated catalog of applications for vCD tenants, where they can deploy applications at a single click. The end user will be able to deploy these applications without having to understand the infrastructure it hosts it, networking configuration, firewall, Public IP or so on, as all of it will be automated in the back-end.

As VMware had acquired Bitnami (The leader of Packaged Open Source Applications) a bit over a year ago, it makes no surprise to anyone that Bitnami provides over ~130 Open Source Applications ready to serve to end tenants in the Cloud Director App Launchpad. That’s not all, as beyond the Bitnami, Cloud Director App Launchpad support another two sources for applications:

  • ISV apps from VMware Cloud Marketplace
  • In-house apps

This will allow our cloud providers to have a nicely populated catalog from day one as well the freedom to add their own custom offering to it. Before starting on the installation instructions, the below screenshot demonstrate what the marketplace of App Launchpad look like:

You can check out my following post for a nice detailed demo of VMware Cloud Director App Launchpad at: Cloud Director App Launchpad Demo

Now you know how nice it looks like, let’s get started on how to get it up and running.Read More

The VMware Cloud Director appliance deployment fails when you enable the setting to expire the root password upon the first login

As I have been trying to deploy the vCloud Director 10.1 in my home lab, I have faced the below errors, and I wanted to share with others the resolution in case you are facing it and have missed it in the release notes.

“No nodes found in cluster, this likely means PostgreSQL is not running on this node. Consult the management UI from another node where PostgreSQL is running. Otherwise, check /opt/vmware/var/log/vcd/vcd_ova_ui_app.log if you think this is an error.”

The following error in the /opt/vmware/var/log/vcd/vcd_ova_ui_app.log file:

————–/opt/vmware/var/log/vcd/vcd_ova_ui_app.log Start———————

2020-04-19 13:23:52,026 | ERROR | uWSGIWorker1Core0 | ERROR: Command ‘[‘sudo’, ‘-n’, ‘-u’, ‘postgres’, ‘/opt/vmware/appliance/bin/api/replicationClusterStatus.py’]’ returned non-zero exit status 255.

2020-04-19 13:23:52,027 | ERROR | uWSGIWorker1Core0 | Return code: 255

2020-04-19 13:23:52,027 | DEBUG | uWSGIWorker1Core0 | Got cluster status: {}

2020-04-19 13:24:16,968 | DEBUG | uWSGIWorker1Core0 | Appliance is of type primary or standby

2020-04-19 13:24:16,969 | DEBUG | uWSGIWorker1Core0 | getting cluster status: [‘/opt/vmware/appliance/bin/api/replicationClusterStatus.py’]

2020-04-19 13:24:17,098 | ERROR | uWSGIWorker1Core0 | ERROR: Command ‘[‘sudo’, ‘-n’, ‘-u’, ‘postgres’, ‘/opt/vmware/appliance/bin/api/replicationClusterStatus.py’]’ returned non-zero exit status 255.

2020-04-19 13:24:17,099 | ERROR | uWSGIWorker1Core0 | Return code: 255

2020-04-19 13:24:17,099 | DEBUG | uWSGIWorker1Core0 | Got cluster status: {}

———–/opt/vmware/var/log/vcd/vcd_ova_ui_app.log End————————

Usually there is two common names for the above issue with vCloud Director 10.1… Read More

How to Change VMware NSX-T Manager IP Address

VMware NSX-T logoThere is often the situation where you need to change the IP addresses of your NSX-T Managers. For example, you might be changing your IP schema as I am doing currently in my home lab. NSX-T does not have a field to change the IP address of it’s NSX Managers, but you will need to add new NSX Managers with the new desired IP, then gradually delete the old ones. Luckily the process is easy and straight forward as documented below.

Note: While I only have a single NSX-T Manager in my environment as it is a small home lab, usually in a production environment, you always want to maintain 3 NSX Managers active to sustain your NSX-T availability. Try to follow one of the below two approaches to maintain that.

  • Scenario A:
    • Manager A has IP address
    • Manager B has IP address
    • Manager C has IP address
    • Add Manager D with a new IP address, for example,
    • Remove Manager A.
    • Add Manager E with a new IP address, for example,
    • Remove Manager B.
    • Add Manager F with a new IP address, for example,
    • Remove Manager C.
  • Scenario B:
    • Manager A has IP address
Read More

VMware Cloud Director 10.1 is here!

VMware Cloud Director 10.1 has just been released and ready for you to try! As we have got used to in the past few releases of vCloud Director, the amount of features added in each new release is substantial even in minor releases. VMware Cloud Director 10.1 is no different and comes with plenty of features than our Cloud Providers will be thrilled with.

As a start, you might have noticed the name change. vCloud Director has gone under a huge rebranding and is now called VMware Cloud Director, as our Cloud Director Service across the hyperscalers is on its way as well. While some of our documentations might take some time to get updated and where you might still find some old references on the web of it being called vCloud Director, going forward it will be named VMware Cloud Director. Please welcome “VMware Cloud Director”, as it brought a huge amount of features with it.

App Launchpad

Many of our Cloud Providers had often asked about adding applications market place to our Cloud Director Portal in the past. Now with App Launchpad they get a very modern looking market place within the Cloud Director portal, where they can publish applications from different sources including Bitnami Community Catalog VM images, ISV apps, and even in-house developed applications.… Read More

Google Cloud joins Azure and AWS in offering VMware Cloud

VMware had been working hard lately on executing their vision: “Build, Run, Manage, Connect and Protect Any App on Any Cloud on Any Device.”

VMware Cloud everywhere, Amazon AWS, Microsoft Azure, and Google GCP

If you have spoke to any of my colleagues lately or attended VMworld or any other VMware event, I am sure you have heard it or a slightly modified version of it.

Most of you have gotten with VMware Cloud on AWS (VMC on AWS) by now, as it was the first offering of VMware Cloud on hyperscaler. It is the offering of Native VMware stack on a hyperscaler, where the environment gives you the same features and manageability you have used to on-prem in addition to the integration of the hyperscaler services in an environment where you don’t have to worry about maintenance, upgrade, patching, while enjoying all the new features as new releases hit the market.

The idea had appealed to a large number of customer, where thousands of VMC on AWS hosts were spin up over the past couple of years. Microsoft had followed Amazon AWS suit and started offering Azure VMware Solutions lately. I have a feeling that it will pickup quite quickly as both VMware and Microsoft has a strong presence in the enterprise world.… Read More

vCloud Director Container Service Extension 2.6.x is here!

As more and more of our Cloud Providers are being asked to support providing K8s and Container services to their customers in a self-service and Multi-tenant fashion, we have released Container Service Extension over 30 months ago.

The goal of Container Service Extension was to offer an Open Source plugin for vCloud Director that gives our Cloud Providers the capabilities to spin and scale Kubernetes Cluster with ease and minimum knowledge of Kubernetes by the Infrastructure team. It is an easy service that our Cloud Providers can add to their catalog at no extra cost specific to CSE. For more information about CSE, I would suggest you take a look at: https://vmware.github.io/container-service-extension/INTRO.html

VMware Container Service Extension

Our Container Service Extension had been evolving quickly and we just released CSE 2.6 . CSE 2.6 beta has been in testing for sometime by several partners. In this release, we had a lot of great enhancements coming including a native UI.

Here is a summary of features included with Container Service Extension 2.6:

  • New Templates with updated Kubernetes and Weave
  • In place Kubernetes upgrade for clusters
    • CSE offers the new capability to do in place upgrade of Kubernetes related software in Native clusters.
Read More

VMware Cloud Provider Pod 1.5.0.x Update is here

Product Versions Deployed with Cloud Provider Pod had been updated to include vCloud Director 10.0. Below is the full Version changes:

  • VMware ESXi and vCenter Server are updated to 6.7 Update 3
  • VMware NSX for vSphere is updated to 6.4.6
  • VMware vCloud Director is updated to 10
  • VMware vRealize Network Insight is updated to 5.0
  • VMware vRealize Operations Manager is updated to 8.0
  • VMware vRealize Operations Manager Multi-Tenant App is updated to 2.3
  • VMware vRealize Operations Manager for NSX 
  • VMware vRealize Operations Manager – Cloud Pod Management Pack is updated to 3.0

·  You must get a new deployment guide from the Cloud Provider Pod designer and follow the instructions included in it. The instructions are changed, now you must manually upload the products.json in the Cloud Provider Pod appliance to get the new product versions.

·  There is a critical fix in the Drivers and Tools section of the Cloud Provider Pod page to resolve an issue with the download of required deployment software. See the known issues for more details.

As there is plenty of improvements and new features in vCloud Director 10.0, if you were waiting to stand up your setup with CPOD, now you can do it with vCloud Director 10.0… Read More

VMware Octant

VMware Octant an Open Source Project

As I have been discovering more with K8s, I have been growing to be a fan of VMware Octant. As defined by the open source project:

A highly extensible platform for developers to better understand the complexity of Kubernetes clusters.

Octant is a tool for developers to understand how applications run on a Kubernetes cluster. It aims to be part of the developer’s toolkit for gaining insight and approaching complexity found in Kubernetes. Octant offers a combination of introspective tooling, cluster navigation, and object management along with a plugin system to further extend its capabilities.

The reason I like Octant is that it allows me to visualize my K8s environment very easily. I can see all of my deployments, Replica Sets, Daemon sets, Pods, Replication Controllers, jobs and almost every other K8s Construct in a visual presentation. It can be of a great value as well to K8s newbies to understand K8s constructs. As it allows them to visualize the effects and changes completed by running a particular Kubectl command.

Here is few examples of the useful screens that you can find in Octant. Here is the Octant overview tab listing a summary of Deployments, Replica Sets, Jobs, Pods, Services, Ingress Controllers and more.… Read More

Kubernetes as a Service utilizing Nirmata and VMware vCloud Director

Over two years ago, VMware had released vCloud Director Container Service Extension (CSE). The idea back then was to allow service providers to spin Kubernetes Clusters for their customers at ease with a single command line. CSE has as well allowed our VMware Cloud Providers to offer Kubernetes Clusters as a self service, where Tenants can request, create and delete their own clusters within few minutes and with a very minimal interaction.

Even better CSE had two methods of integrations, where CSE was able to deploy Kubernetes clusters directly into vCloud Director as a vAPP that contains all masters and workload nodes or to integrate with Enterprise PKS and deploy on top of PKS benefiting from PKS capabilities and features.

While CSE had been very capable Open Source plugin that was originated by VMware, it was missing few features that many Cloud Providers desired. Nirmata had seen the need, and created Nirmata Plugin for vCloud Director which had fulfilled the below desired enhancements:

Nirmata for Kubernetes Logo

Graphical User Interface (GUI)

While vCloud Director Container Service Extension (CSE) had been very capable as far as deploying and managing Kubernetes clusters goes, it lacked a nice Graphical User Interface (GUI). Actually out of the box CSE had no GUI at all.… Read More

vCloud Director 10.0 is here!!

VMware vCloud Director 10

For those who were following VMware vCloud Director Evolution lately, it has been developing quite fast and new features have been added continuously at a none matched speed. The development vCloud Director had over the past couple of years just amaze me as has surpassed what was delivered in 6 years development cycle previously.

vCloud Director 10 Evolution over time

vCloud Director 10.0 is here! Below is a summary of what it’s bringing to the table. Tons of great features and enhancement and more to come as we progress.

Modernizing Cloud Operations and Improving Core Efficiency

  • VCD as Central Point of Management: added Admin and Tenant UI, listing of vCenter inventory
  • Events API
  • VCD Appliance: Enhancements in migration, HA, certificate mgmt. These are very important for those who are keen to move to the vCD Appliance which is the direction where vCloud Director is going in the future.
  • VRO Plugin: Enable Multi-tenancy in VRO plugin and VRO cluster support. This will take vCD extensibility to the next level.
  • Enhanced support for VM storage placement with SDRS: added support for intra-SPOD placement; improved support for: inter-SPOD placement, datastore + SPOD hybrid placement, named disks and linked clones’ placement
  • Upgrading the appliance using the default VMware repository
  • Template based deployments, New Template Repository.
Read More

How to force delete a PKS Cluster

There are times when you want to delete a PKS Cluster, but the deletion with the usual cPKS delete-cluster command fails.

pks delete-cluster <PKS Cluster Name>

This is usually due to issues with that cluster either it failed for some reason during deployment or it was alternate into a way that destabilize it. No worry, there is a way to still force delete it and here is what I will focus on in this post. Please note you should always try to delete pks clusters using the pks delete-cluster command first and only resort to BOSH deployment delete when that does not work.

Please note in this post, I am assuming you have already setup the BOSH CLI and ready to use it. If you don’t have that setup already, I would suggest you follow the instructions at the following link.

Get your Bosh Credentials:

1-  Login to PCF OPS Manager Interface

Login to PCF OPS Manager Interface

2- Click on the Installation Dashboard

3- Click on BOSH Director for vSphere

4- Click Credentials tab

5- Click on the “Link to Credentials” link next to Director Credentials

PKS Get Bosh Director Credentials

6- Keep a copy of the Director Password, as you will need it later. It will look something like below:

PKS BOSH Director Password

Force Delete PKS Cluster using Bosh:

1- SSH to your Operations Manager Appliance

2- Run the following Command to login to BOSH

 $ bosh -e pks login

Use the username and Password collected earlier from BOSH Director for vSphere.… Read More

Installation of NSX 6.4 VIB on ESXi 6.7 host failed

I have often got to interact with customers who had an issue getting the NSX VIB installed on their ESXi host. Most of the time, it is a tedious configuration issue or a step that they have forgotten. I have hit a similar issue today in my lab with me missing a simple step and wanted to share the error and the fix with others just in a hope it helps others recover from the same error quicker.

I was getting the following error every time I tried to install the NSX 6.4.5 VIB on my ESX 6.7U2 host, and similar error as well when I try to run the resolve button. The error stated “Unable to access agent VIB module at (_NSX_87_VTRES01_VMware Network Fabric). A screen shot of the errir is below.

Unable to access agent VIB Module at vxlan.zip

There was a more detailed error on my NSX screen, which unfortunately I seem to have lost the screenshot for, but it stated something like below:

vtesxi01.vt.com: Unable to access agent offline bundle at
Cause : <esxupdate-response>
<error errorClass=”MetadataDownloadError”>
<errorDesc>Failed to download metadata.</errorDesc>
<msg>(‘https://vtvc01.vt.com:443/eam/vib?id=ecf4a884-c9f5-406c-b57e-75a6613a3651’, ‘/tmp/tmpjnw369p9’, ‘[Errno 14] curl#6 – “Couldn\’t resolve host \’vtvc01.vt.com\'”‘)</msg>

As I have seen this one before, I was immediately able to spot that the fix is more than likely I have forgotten to setup Forward or Reverse DNS record or configuration for one of my setup component being ESXi, vCenter or NSX.… Read More