Recently I have been working on a project which requires the use of Docker containers and the ability to scale out with services. Docker itself offers the ability to do this using Docker swarm. However, the development community overwhelmingly supports containerization using Kubernetes. Kubernetes is Google’s containerization platform that offers a rich set of features for scaling containerized applications. I like to do all my development on a multiple Windows virtual machines using Hyper-V. Some days I can have up to three virtual machines running with three different projects . Doing Docker development required Windows Server 2016 and I had no problem testing Docker swarm across multiple virtual machines. Trying to develop with Kubernetes locally on a Windows Server 2016 virtual machine was a different story. There is not a lot of information on how to accomplish this, in fact I could not come across any. Most of the information was doing it using Kubernetes Minikube on Windows 10 but not in a virtual machine. This post will give you all the steps you need to get Kubernetes Minikube to run on a Windows Server 2016 Hyper-V virtual machine.
Minikube
Google’s Minikube is the Kubernetes platform you can use to develop and learn Kubernetes locally. You can basically do anything with Minikube as you could if you were running it in Azure except load balance and scale out across nodes. Learn more about Minikube here. Minikube is run in its own virtual machine that runs Linux. So when you want to use it on a windows virtual machine you must enable Hyper-V. You are nesting virtual machines. So here are the steps you need to take to get Minikube to run reliably on your Windows Server 2016 Hyper-V virtual machine.
Steps
Configure Nested Virtualization
The first thing you must do is enable nested virtualization on the Windows Server 2016 virtual machine you want to run Minikube on. You must do this from the host using PowerShell.
Set-VMProcessor -VMName-ExposeVirtualizationExtensions $true
Install Hyper-V Feature on the Windows Server 2016 Virtual Machine
Download the Latest Version of KubeCtl and Minikube for Windows
Copy both to the root C drive or the root drive you will be running PowerShell from. Rename the Minikube executable to Minikube. This will make it easier for you to execute Minikube commands in PowerShell.
Create a Virtual Switch for Minikube
Create a virtual switch to be used with the Minikube virtual machine using PowerShell
New-VMSwitch -Name VmNAT -SwitchType Internal
Assign an IP Address to the Virtual Switch Adapter
Get-NetAdapter "vEthernet (VmNat)" | New-NetIPAddress -IPAddress 192.168.137.1 -AddressFamily IPv4 -PrefixLength 24
Configure the Ethernet Microsoft Hyper-V Network Adapter
Internet Protocol 4 enabled
Hyper-V Extensible Virtual Switch disabled
Internet Protocol 4 has 8.8.8.8 as the preferred DNS serverShare this adapter with the virtual switch adapter setup for Minikube. This is found in the sharing tab.
Configure the Virtual Switch Ethernet Adapter
Internet Protocol 6 disabled
Reboot the Virtual Machine
Start Minikube
Start Powershell in administrator mode. Change the directory to the Root drive where Minikube.exe is located. Run the following Powershell.
.\minikube start --kubernetes-version="v1.8.0" --vm-driver="hyperv" --hyperv-virtual-switch="VmNAT" --v=7 --alsologtostderr
If all works you will see the message “kubectl is now configure to use the cluster”
To make sure all is working execute the following powershell
.\kubectl get pods --all-namespaces
If all is working you should see something like this.
Minikube and Windows Containers
So once this is all up an going you should be happy. Unfortunately, Minikube does not work with Windows containers yet. You can still run other linux containers to test and learn. Kubernetes and Windows containers are only supported in Azure. Another aggravation is when you either stop and restart Minikube or reboot the virtual machine, then Minikube stops working. It can no longer obtain an IPv4 address. This is from a recent bug in Windows Server 2016. You must disable and re-enable sharing the Ethernet adapter with the virtual switch adapter. Big pain. Below is a PowerShell script you can use at startup or just execute it after one of these scenarios. So I hope you find this useful. It took me a long time to get this to work consistently. I can only hope Minikube will support windows containers and this bug gets fixed.
# Register the HNetCfg library (do only once)
regsvr32 hnetcfg.dll
$m = New-Object -ComObject HNetCfg.HNetShare
$c1 = $m.EnumEveryConnection |? { $m.NetConnectionProps.Invoke($_).Name -eq "Ethernet" }
$c2 = $m.EnumEveryConnection |? { $m.NetConnectionProps.Invoke($_).Name -eq "vEthernet (VmNAT)" }
$config1 = $m.INetSharingConfigurationForINetConnection.Invoke($c1)
$config2 = $m.INetSharingConfigurationForINetConnection.Invoke($c2)
$config1.DisableSharing()
$config2.DisableSharing()
$config1.EnableSharing(0)
$config2.EnableSharing(1)
