Wednesday, November 22, 2017

Run Kubernetes Locally with Hyper-V on Windows Server 2016

Recently I have been working on a project which requires the use of Docker containers and the ability to scale out with services. Docker itself offers the ability to do this using Docker swarm. However, the development community overwhelmingly supports containerization using Kubernetes. Kubernetes is Google’s containerization platform that offers a rich set of features for scaling containerized applications. I like to do all my development on a multiple Windows virtual machines using Hyper-V. Some days I can have up to three virtual machines running with three different projects . Doing Docker development required Windows Server 2016 and I had no problem testing Docker swarm across multiple virtual machines. Trying to develop with Kubernetes locally on a Windows Server 2016 virtual machine was a different story. There is not a lot of information on how to accomplish this, in fact I could not come across any. Most of the information was doing it using Kubernetes Minikube on Windows 10 but not in a virtual machine. This post will give you all the steps you need to get Kubernetes Minikube to run on a Windows Server 2016 Hyper-V virtual machine.

Minikube

Google’s Minikube is the Kubernetes platform you can use to develop and learn Kubernetes locally. You can basically do anything with Minikube as you could if you were running it in Azure except load balance and scale out across nodes. Learn more about Minikube here. Minikube is run in its own virtual machine that runs Linux. So when you want to use it on a windows virtual machine you must enable Hyper-V. You are nesting virtual machines.  So here are the steps you need to take to get Minikube to run reliably on your Windows Server 2016 Hyper-V virtual machine.

 

Steps

Configure Nested Virtualization

The first thing you must do is enable nested virtualization on the Windows Server 2016 virtual machine you want to run Minikube on. You must do this from the host using PowerShell.

Set-VMProcessor -VMName  -ExposeVirtualizationExtensions $true 

Install Hyper-V Feature on the Windows Server 2016 Virtual Machine

Download the Latest Version of KubeCtl and Minikube for Windows

KubeCtl

Minikube

Copy both to the root C drive or the root drive you will be running PowerShell from. Rename the Minikube executable to Minikube. This will make it easier for you to execute Minikube commands in PowerShell.

Create a Virtual Switch for Minikube

Create a virtual switch to be used with the Minikube virtual machine using PowerShell

New-VMSwitch -Name VmNAT -SwitchType Internal

 

Assign an IP Address to the Virtual Switch Adapter

Get-NetAdapter "vEthernet (VmNat)" | New-NetIPAddress -IPAddress 192.168.137.1 -AddressFamily IPv4 -PrefixLength 24

 

Configure the Ethernet Microsoft Hyper-V Network Adapter

Internet Protocol 4 enabled
Hyper-V Extensible Virtual Switch disabled
Internet Protocol 4 has 8.8.8.8 as the preferred DNS server

Share this adapter with the virtual switch adapter setup for Minikube. This is found in the sharing tab.

Configure the Virtual Switch Ethernet Adapter

Internet Protocol 6 disabled

Reboot the Virtual Machine

Start Minikube

Start Powershell in administrator mode. Change the directory to the Root drive where Minikube.exe is located. Run the following Powershell.
 

.\minikube start --kubernetes-version="v1.8.0" --vm-driver="hyperv" --hyperv-virtual-switch="VmNAT" --v=7 --alsologtostderr

If all works you will see the message “kubectl is now configure to use the cluster”

To make sure all is working execute the following powershell

 

.\kubectl get pods --all-namespaces

If all is working you should see something like this.

Minikube and Windows Containers

So once this is all up an going you should be happy. Unfortunately, Minikube does not work with Windows containers yet. You can still run other linux containers to test and learn. Kubernetes and Windows containers are only supported in Azure. Another aggravation is when you either stop and restart Minikube or reboot the virtual machine, then Minikube stops working. It can no longer obtain an IPv4 address. This is from a recent bug in Windows Server 2016. You must disable and re-enable sharing the Ethernet adapter with the virtual switch adapter. Big pain. Below is a PowerShell script you can use at startup or just execute it after one of these scenarios. So I hope you find this useful. It took me a long time to get this to work consistently. I can only hope Minikube will support windows containers and this bug gets fixed.

# Register the HNetCfg library (do only once)
regsvr32 hnetcfg.dll


$m = New-Object -ComObject HNetCfg.HNetShare
$c1 = $m.EnumEveryConnection |? { $m.NetConnectionProps.Invoke($_).Name -eq "Ethernet" }
$c2 = $m.EnumEveryConnection |? { $m.NetConnectionProps.Invoke($_).Name -eq "vEthernet (VmNAT)" }
$config1 = $m.INetSharingConfigurationForINetConnection.Invoke($c1)
$config2 = $m.INetSharingConfigurationForINetConnection.Invoke($c2)
$config1.DisableSharing()
$config2.DisableSharing()
$config1.EnableSharing(0)
$config2.EnableSharing(1)