VMware innovations in the automation of VMware vSphere virtual infrastructure in 2023

This year, VMware has taken a serious approach to refining its automation frameworks for operations in virtual infrastructure. This was motivated by the fact that historically, VMware has accumulated a large number of automation tools. These tools were once tied to specific products and interfaces. Over time, these products evolved and merged into various product lines, but the old interfaces continued to linger with different solutions. Managing these solutions can still be done in several different ways, and when considering various platforms and auxiliary services, there becomes an overwhelming number of ways to do so.

This leads to confusion among administrators who don’t understand which automation path is recommended and how to determine whether the scripts developed today will work in the virtual infrastructure a year from now. Let’s try to figure this out.

So, in which direction regarding the reforming of automation frameworks is VMware currently heading:

1. New VI/JSON protocol for vSphere infrastructure management 

VMware vSphere has a rich set of APIs that allow management of the platform, which is simultaneously one of its strengths and weaknesses. These APIs, covering everything from managing virtual machines to virtualizing storage and networks, provide flexibility in platform configuration and the ability to automate processes.

Virtual management interfaces (VI API), introduced about 15 years ago, have found widespread use. Thanks to these APIs, a multitude of products and scripts have been created, and developers are familiar with their methodology. Based on SOAP and XML, the VI API interface, despite its functionality, has complexities related to security and the use of XML. At the same time, when working with automation interfaces, developers increasingly prefer JSON.

To facilitate integration into JSON-compatible environments, including web browsers and modern programming languages like Go, VMware introduced a new protocol similar to REST and based on JSON – VI/JSON. This feature became available starting with VMware vSphere 8 Update 1.

The new VI/JSON protocol represents a modern replacement for the outdated SOAP, offering the same VI API, a modern security model, and minimal changes when transitioning from SOAP.

This protocol provides direct access to managed properties, several hundreds of record types, reflecting the state of elements managed by vSphere, from virtual machines to networks and clusters. Previously, these properties were accessible only through the Property Collector, but now they can be retrieved with a simple HTTP GET request.

Unlike SOAP, in VI/JSON resources and operations are encoded in the request’s URL address, not in the body of the message. This simplifies integration at the firewall level, eases traffic analysis, and scriptwriting. Encoding data based on JSON makes development easier for VI API users.

Security has also been improved: instead of using HTTP cookies for session identification, a special HTTP header is used, which prevents Cross-Site Request Forgery when the threat comes from a third party.

Here’s how simple it looks to turn on a virtual machine using the new protocol:

https://{vcenter-host}/sdk/vim25/{release}/VirtualMachine/{moId}/PowerOnVM

2. Improvements in automation tools: API and OpenAPI

As we noted earlier, over time VMware vSphere has become a more complex product, offering various types of API. The most commonly used today are:

• vSphere Web Services API, using the SOAP protocol
• vSphere Automation API, applying REST principles

Having multiple APIs increases the complexity of managing and operating large environments, and this complexity is reflected in various aspects – from the need to have different tools and skills to different endpoints and authentication mechanisms for different APIs.

Each of the APIs has its own documentation with its own guides and code examples, which can lead to confusion among administrators. They might assume that they need to choose between REST API and SOAP API, whereas they should use both since they complement each other.

Today, VMware is actively working on improving APIs, focusing on simplifying API use, optimizing documentation, enhancing the developer experience, and unifying protocols and tools. You can find the latest news about this from the following links:

• vSphere Automation REST API Programming Guide
• vSphere Web Services Programming Guide
• Understanding the vSphere Automation API
• Code Samples

If we go back to the VI/JSON protocol, the same structure, managed objects, and method names from the aforementioned APIs were retained so that clients could transition their existing automation solutions to the new JSON protocol with minimal changes.

Lastly, VMware offers OpenAPI 3.0 specifications in addition to the documentation on the virtual infrastructure’s JSON API. VMware is also working on merging the JSON API documentation with the Automation API documentation where all REST services are documented. This brings closer the moment of creating a unified resource for clients to search for documentation.

Here, three main directions of improvements can be highlighted, on which work is currently being conducted:

• In the long term, VMware plans to transform the remaining public SOAP interfaces (such as SPBM, SMS, and others) to be compatible with new formats, including providing OpenAPI 3.0 specifications for each of them and integrating them with the REST API documentation.
• The next transformation will be the introduction of a common identification mechanism for the REST API and the new JSON protocol instead of the SOAP API. This will simplify the development of automated solutions and provide unified management for all vSphere services within a single session.
• The final step in the improvement process will be to ensure access to all services through a single endpoint, allowing administrators to simply interact with all of vSphere’s APIs. This will greatly simplify the work and enhance the quality of documentation and developer comfort.

3. Cross-Platform PowerCLI Engine

Using PowerShell/PowerCLI cmdlets for automation offers several advantages, including easy installation across various operating systems and a powerful scripting language that supports commands specifically designed for VMware products.

PowerCLI seamlessly integrates (and arguably even better than other tools) into the VMware ecosystem, providing harmonious interaction with products such as vSphere, vSAN, SRM, and others. Although PowerCLI operates independently with its release schedule, VMware aims to align it with official vSphere versions. This allows PowerCLI users to quickly access new features and capabilities delivered in new versions of VMware vSphere.

Many associate PowerShell with Microsoft Windows, which is understandable. However, the multi-platform core of PowerShell Core is gaining popularity on Linux and MacOS platforms. VMware is striving to support these platforms on par with Windows.

As VMware vSphere and other VMware products evolve over time, PowerCLI must also undergo changes, and this needs to happen quickly to remain current with product line innovations. This accelerated pace shouldn’t compromise the usability and quality of PowerCLI cmdlets. Therefore, VMware has set the task of expanding PowerCLI functionality for products in shorter terms while maintaining the necessary quality of modules and cmdlets.

When PowerCLI 13.0 was introduced in November 2022, it became the first PowerCLI fully compatible with PowerShell Core, running on Microsoft Windows, Linux, and Apple MacOS. This allows the execution of code samples and full scenarios on any of the mentioned platforms, and, for instance, schedule their execution using cron on Linux. Such flexibility makes it a highly efficient automation tool, expanding orchestration capabilities.

This implies that modules, including VMware.ImageBuilder and VMware.DeployAutomation, had to ensure compatibility with Apple MacOS and Linux. New commands were also introduced for managing vSAN ESA cluster disks, and the VMware.VimAutomation. Cloud module was updated to support the VMware Cloud Director 10.4 API functionality.

In PowerCLI 13.1, two new modules were introduced: VMware.Sdk.VR, which supports the VMware vSphere Replication REST API, and VMware.Sdk.Srm, responsible for supporting the VMware Site Recovery Manager REST API. Additionally, new cmdlets were added for working with Lifecycle Manager offline repositories, remote data store management of vCenter Server, direct vSAN disk management, cluster shutdown, mounting remote vSAN data stores from extended vSAN clusters, and much more.

Furthermore, starting from the PowerCLI 12.4 release, cmdlet generation for the vSphere Automation API was automated, released as part of a new module called VMware.SDK.vSphere. This approach allows for rapid creation of cmdlets and modules to support new features, ensuring quality and consistency. In subsequent releases, VMware continued to introduce new PowerCLI modules based on the auto-generation approach (e.g., modules VMware.Sdk.Nsx.Policy and VMware.Sdk.SRM).

4. SDK Improvements

vSphere Software Development Kits (SDKs) are development automation tools for various VMware products based on different programming languages and frameworks.

SDKs implement various bindings for the vSphere API. For instance, the vSphere Automation SDK provides bindings for the following REST services in vSphere:

• VMware vSphere Automation Endpoint in vCenter Server
• VMware Cloud on AWS Console API
• VMware NSX-T on VMware Cloud API
• NSX VMware Cloud on AWS Integration API
• VMware Cloud on AWS Site Recovery API

For the SOAP protocol, bindings have been implemented in the vSphere Management SDK for the following services:

• VMware vSphere Web Services API
• VMware vSphere ESXi Agent Manager API
• VMware Storage Policy API
• Virtual Storage Lifecycle Management API
• VMware vCenter Server Storage Monitoring Service API
• VMware vCenter Server Single Sign-On

One of the primary challenges until recently was the vast number of SDKs for automated workflow management in the Software-Defined Data Center (SDDC). There are two types of SDK bindings available for each programming language. As an example, pyvmomi is designed for access to the vSphere SOAP API, and vSphere Automation Python SDK is for working with the vSphere REST API.

There were some issues here – the Automation SDK was in VMware’s GitHub repository, whereas the Management SDK was hosted on the developer.vmware.com site. Pyvmomi was available in the PyPI repository, but the Automation SDK for Python was not found there. In general, there was no single point for developers to access the needed SDKs. What’s more, some companies don’t allow the use of open-source software from GitHub repositories.

Another issue is that some VMware partners need early access to SDKs to test their solutions and address arising issues promptly. However, SDKs aren’t part of VMware’s beta programs, so they don’t receive preview versions before the official General Availability release. This hinders early feedback and complicates software development for new vSphere releases.

Changes made this year or planned for the near future:

• VMware has placed all its SDKs on the developer.vmware.com/sdks page (you can also find packages like pyvmomi, vSphere Automation SDK for Java, and vSphere Automation SDK for Python there).

• VMware is beginning to listen to the developer community (for example, the auto-completion feature in pyvmomi 8.0.1.0 can be cited).
• VMware is increasing the update frequency of its SDKs so that developers can timely access new functionalities.
• VMware’s next step will be the launch of a unified SDK that integrates bindings for both SOAP and REST-based APIs.
• There will be a centralized place for developers to access all necessary libraries – documentation and download sources will be located in public package repositories such as PyPI and Maven Central.
• Bindings for the new JSON protocol will be added.
• VMware plans to provide early access to SDKs for clients and partners.

Conclusions

This year, VMware has done a lot not only to enhance its automation tools for the vSphere virtual infrastructure and other products but also to thoroughly reorganize them. Indeed, in the past, developers complained that resources for APIs, frameworks, and SDKs were scattered in different places, and many tools seemed to duplicate one another, making it unclear which exactly to use. As a result, VMware has taken several steps to address these issues, and they have also made the PowerCLI interface cross-platform, greatly easing the lives of developers and virtual infrastructure administrators. The introduction of the VI/JSON protocol for infrastructure management indicates that VMware is keeping pace with the times and aligning with the needs of developers and current technological trends.

Undoubtedly, these are not all the positive updates – after all, the VMware Explore 2023 conference is ahead, where many additional innovations and improvements in terms of automation tools for virtual infrastructure will be announced.

 

Related materials:

• Major release of VMware Cloud Foundation 5.0 architecture – what’s interesting about it?
• VMware Skyline Health Diagnostics 4.0 – Simplifying VMware Infrastructure Monitoring and Troubleshooting

from StarWind Blog https://bit.ly/3Z6RToI
via IFTTT