Whilst looking to spin up a new CoreOS VM to run Docker containers in, I noticed that they had added support for configuring cloud-config via VMware’s Guestinfo interface. In short, instead of putting your cloud-config in to an ISO an attaching it to the VM, you can embed it within the VM’s configuration file. To me, this seems like a far cleaner way of configuring CoreOS and potentially far simpler.

Unfortunately, it’s very easy to mess up this approach and it can be difficult to figure out where you’ve gone wrong if you’re not familiar with it, so read carefully! CoreOS’s documentation on this is useful, but not always the most intuitive.

The instructions below were used to use the Guestinfo Interface to pass the cloud-config file to a VM running CoreOS 899.15.0. Hopefully you’ll find them useful if you’re giving this method a try!

Instructions

First of all, download the latest CoreOS Stable OVA from http://stable.release.core-os.net/amd64-usr/current/coreos_production_vmware_ova.ova. Then, using your vSphere client, use the Deploy OVF wizard and follow the steps. Do not opt to have the VM powered on once the wizard completes.

Then, create your cloud-config file as normal. Here’s mine that i’ll be using an an example;

#cloud-config
ssh_authorized_keys:
  - "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAkC7zRhZnhj6R1kZq9a7fQw1r/35LN4pBhlfgWR6nPkhuMnm3RUXNbG+JuqONAxExzK+e5wxuFWue9jPSi4J+wnaAtqTSP/bJ+kos03oeeOPcjsN6lwqaujR4wonCJTbKGDsgdRWpRpI6pyqFaOZUuaOUU6SktLUtw7XSqROG97HiyJ3BB/QQysF6irTZLiTCesCFIzD0Ffq/9jo5l1lbwzqEFdp0qQ8kHWN7UtzcoUzh3qR3NfuCSJSU/HH0kZfZ/kzOKiFWAF0GiVTQtHX1q83di4hDP8hqtyGXoB9PIwpzZEdjb4NK9T5bvBmXS6L1xMgvmrkXCfZ1aFcJVVTT+Q== rsa-key-20160410"
hostname: "docker"

Notably, compared to CoreOS’s documentation on this, your SSH key must be embedded in quotes as above, otherwise it isn’t parsed correctly! I’ll be raising this as an issue with them shortly.

As noted in the documentation, Guestinfo expects the cloud-config data to be encoded using base64, so use base64 -w0 cloud-config && echo  to take your cloud-config file, encode it to base64 and output it to your terminal. For example;

user@host:~$ base64 -w0 cloud-config && echo
I2Nsb3VkLWNvbmZpZwpzc2hfYXV0aG9yaXplZF9rZXlzOgogIC0gInNzaC1yc2EgQUFBQUIzTnphQzF5YzJFQUFBQUJKUUFBQVFFQWtDN3pSaFpuaGo2UjFrWnE5YTdmUXcxci8zNUxONHBCaGxmZ1dSNm5Qa2h1TW5tM1JVWE5iRytKdXFPTkF4RXh6SytlNXd4dUZXdWU5alBTaTRKK3duYUF0cVRTUC9iSitrb3MwM29lZU9QY2pzTjZsd3FhdWpSNHdvbkNKVGJLR0RzZ2RSV3BScEk2cHlxRmFPWlV1YU9VVTZTa3RMVXR3N1hTcVJPRzk3SGl5SjNCQi9RUXlzRjZpclRaTGlUQ2VzQ0ZJekQwRmZxLzlqbzVsMWxid3pxRUZkcDBxUThrSFdON1V0emNvVXpoM3FSM05mdUNTSlNVL0hIMGtaZlova3pPS2lGV0FGMEdpVlRRdEhYMXE4M2RpNGhEUDhocXR5R1hvQjlQSXdwelpFZGpiNE5LOVQ1YnZCbVhTNkwxeE1ndm1ya1hDZloxYUZjSlZWVFQrUT09IHJzYS1rZXktMjAxNjA0MTAiCmhvc3RuYW1lOiAiZG9ja2VyIgo=

Now we need to edit the VM’s configuration file to set the relevant configuration parameters. If you’re using a vSphere ESXi server like me, SSH to the CLI and use vi /vmfs/volumes/DATASTORE/VMNAME/VMNAME.vmx to edit the file. Jump to the bottom of the file and add the following lines;

guestinfo.coreos.config.data = "I2Nsb3VkLWNvbmZpZwpzc2hfYXV0aG9yaXplZF9rZXlzOgogIC0gInNzaC1yc2EgQUFBQUIzTnphQzFYzJFQUFBQUJKUUFBQVFFQWtDN3pSaFpuaGo2UjFrWnE5YTdmUXcxci8zNUxONHBCaGxmZ1dSNm5Qa2hTW5tM1JVWE5iRytKdXFPTkF4RXh6SytlNXd4dUZXdWU5alBTaTRKK3duYUF0cVRTUC9iSitrb3MwM29ZU9QY2pzTjZsd3FhdWpSNHdvbkNKVGJLR0RzZ2RSV3BScEk2cHlxRmFPWlV1YU9VVTZTa3RMVXR3N1hcVJPRzk3SGl5SjNCQi9RUXlzRjZpclRaTGlUQ2VzQ0ZJekQwRmZxLzlqbzVsMWxid3pxRUZkcDBxUThSFdON1V0emNvVXpoM3FSM05mdUNTSlNVL0hIMGtaZlova3pPS2lGV0FGMEdpVlRRdEhYMXE4M2RpNGhUDhocXR5R1hvQjlQSXdwelpFZGpiNE5LOVQ1YnZCbVhTNkwxeE1ndm1ya1hDZloxYUZjSlZWVFQrUT0IHJzYS1rZXktMjAxNjA0MTAiCmhvc3RuYW1lOiAiZG9ja2VyIgo="
guestinfo.coreos.config.data.encoding = "base64"

The first line is our base64 encoded cloud-config file and the second line tells CoreOS that the configuration data is encoded in base64 (as it is possible to pass the data in a number of other formats). Save the configuration file and quit.

After editing the VM’s configuration, we need to inform vSphere that the file has changed and that it should reload it. From the ESXi’s server CLI again, run the command vim-cmd vmsvc/getallvms  . Find the name of your VM in the list and note down its Vmid. Then, run the command vim-cmd vmsvc/reload Vmid, replacing Vmid with the Vmid from the previous command. As a word of warning, do not edit the VM’s configuration file or reload the VM’s configuration while the VM is powered on, as your changes will likely be overwritten. You must do these steps with the VM powered off.

Success! CoreOS has read our cloud-config data and set the hostname correctly.

Finally, power on the VM and watch the console output. If successful, the VM should boot and at the login screen the hostname you supplied in your cloud-config file should be visible instead of the default ‘localhost’ hostname. Try connecting over SSH to the IP displayed on the console and logging in using the username core. If the cloud-config file was loaded successfully (and you have your SSH key loaded locally), you should be able to connect to CoreOS’s CLI.

Troubleshooting

As mentioned earlier, it’s easy to make a mistake using this method and CoreOS isn’t exactly helpful at telling you what went wrong – at first!

Bypassing the CoreOS Login

In order to troubleshoot why the cloud-config configuration isn’t being loaded, we first need to get in to CoreOS’s CLI. Of course, without a working cloud-config configuration this usually wouldn’t be possible. Like all good Linux systems though, it’s possible to bypass the login locally when booting up!

Reboot the VM, and as soon as the GRUB bootloader is displayed, quickly press e to edit the bootloader – you will need to be very quick as GRUB will timeout after about a second and continue booting as usual. Once in the bootloader, move down to the 4th line and append coreos.autologin . If done correctly, it should look similar to the screenshot below;

A correctly altered kernel boot parameter for enabling automatic login.

Once edited, press F10 to continue booting. This time, the login will be bypassed and you’ll be dumped at the standard CoreOS CLI. Note that this will only work locally – you won’t be able to login over SSH.

Checking that the Guestinfo parameters have been set

To check that the Guestinfo has been correctly set in the VM’s configuration file and read by the hypervisor, run the following commands to show the contents of the various Guestinfo configuration parameters;

/usr/share/oem/bin/vmtoolsd --cmd "info-get guestinfo.coreos.config.data"
/usr/share/oem/bin/vmtoolsd --cmd "info-get guestinfo.coreos.config.data.encoding"

If the parameters have been set correctly, you should see the exact same data as you set in the VM’s configuration file. If they differ or if the command returns “No value found”, shut down the VM, edit the VM’s configuration file again and check that both guestinfo.coreos.config.data  and guestinfo.coreos.config.data.encoding  have been set correctly and then reload the VM’s configuration. Then, power on the VM and try again.

Manually running the cloud-config configuration

From the CoreOS CLI, run coreos-cloudinit –from-vmware-guestinfo . This will pull the Guestinfo configuration from the hypervisor and try to parse it – this is a great way to discover if you’ve made an error in your cloud-config file or the encoding of it. If the data is all correct, you should see the following lines;

2016/04/10 19:48:21 Set hostname to docker
2016/04/10 19:48:21 Authorized SSH keys for core user

If the data has an error in it, you should see a message with the error along with which line of your cloud-config file caused the error. You can also try using CoreOS’s cloud-config validator tool to check the validity of your cloud-config configuration.