Using VMware’s Guestinfo Interface to Configure cloud-config on a CoreOS VM

Whilst looking to spin up a new CoreOS VM to run Docker containers in, I noticed that they had added support for configuring cloud-config via VMware’s Guestinfo interface. In short, instead of putting your cloud-config in to an ISO an attaching it to the VM, you can embed it within the VM’s configuration file. To me, this seems like a far cleaner way of configuring CoreOS and potentially far simpler.

Unfortunately, it’s very easy to mess up this approach and it can be difficult to figure out where you’ve gone wrong if you’re not familiar with it, so read carefully! CoreOS’s documentation on this is useful, but not always the most intuitive.

The instructions below were used to use the Guestinfo Interface to pass the cloud-config file to a VM running CoreOS 899.15.0. Hopefully you’ll find them useful if you’re giving this method a try!

Instructions

First of all, download the latest CoreOS Stable OVA from http://stable.release.core-os.net/amd64-usr/current/coreos_production_vmware_ova.ova. Then, using your vSphere client, use the Deploy OVF wizard and follow the steps. Do not opt to have the VM powered on once the wizard completes.

Then, create your cloud-config file as normal. Here’s mine that i’ll be using an an example;

Notably, compared to CoreOS’s documentation on this, your SSH key must be embedded in quotes as above, otherwise it isn’t parsed correctly! I’ll be raising this as an issue with them shortly.

As noted in the documentation, Guestinfo expects the cloud-config data to be encoded using base64, so use base64 -w0 cloud-config && echo  to take your cloud-config file, encode it to base64 and output it to your terminal. For example;

Now we need to edit the VM’s configuration file to set the relevant configuration parameters. If you’re using a vSphere ESXi server like me, SSH to the CLI and use vi /vmfs/volumes/DATASTORE/VMNAME/VMNAME.vmx to edit the file. Jump to the bottom of the file and add the following lines;

The first line is our base64 encoded cloud-config file and the second line tells CoreOS that the configuration data is encoded in base64 (as it is possible to pass the data in a number of other formats). Save the configuration file and quit.

After editing the VM’s configuration, we need to inform vSphere that the file has changed and that it should reload it. From the ESXi’s server CLI again, run the command vim-cmd vmsvc/getallvms  . Find the name of your VM in the list and note down its Vmid. Then, run the command  vim-cmd vmsvc/reload Vmid, replacing Vmid with the Vmid from the previous command. As a word of warning, do not edit the VM’s configuration file or reload the VM’s configuration while the VM is powered off, as your changes will likely be overwritten. You must do these steps with the VM powered off.

Success! CoreOS has read our cloud-config data and set the hostname correctly.

Success! CoreOS has read our cloud-config data and set the hostname correctly.

Finally, power on the VM and watch the console output. If successful, the VM should boot and at the login screen the hostname you supplied in your cloud-config file should be visible instead of the default ‘localhost’ hostname. Try connecting over SSH to the IP displayed on the console and logging in using the username core. If the cloud-config file was loaded successfully (and you have your SSH key loaded locally), you should be able to connect to CoreOS’s CLI.

Troubleshooting

As mentioned earlier, it’s easy to make a mistake using this method and CoreOS isn’t exactly helpful at telling you what went wrong – at first!

Bypassing the CoreOS Login

In order to troubleshoot why the cloud-config configuration isn’t being loaded, we first need to get in to CoreOS’s CLI. Of course, without a working cloud-config configuration this usually wouldn’t be possible. Like all good Linux systems though, it’s possible to bypass the login locally when booting up!

Reboot the VM, and as soon as the GRUB bootloader is displayed, quickly press e to edit the bootloader – you will need to be very quick as GRUB will timeout after about a second and continue booting as usual. Once in the bootloader, move down to the 4th line and append  coreos.autologin . If done correctly, it should look similar to the screenshot below;

A correctly altered kernel boot parameter for enabling automatic login.

A correctly altered kernel boot parameter for enabling automatic login.

Once edited, press F10 to continue booting. This time, the login will be bypassed and you’ll be dumped at the standard CoreOS CLI. Note that this will only work locally – you won’t be able to login over SSH.

Checking that the Guestinfo parameters have been set

To check that the Guestinfo has been correctly set in the VM’s configuration file and read by the hypervisor, run the following commands to show the contents of the various Guestinfo configuration parameters;

If the parameters have been set correctly, you should see the exact same data as you set in the VM’s configuration file. If they differ or if the command returns “No value found”, shut down the VM, edit the VM’s configuration file again and check that both  guestinfo.coreos.config.data  and  guestinfo.coreos.config.data.encoding  have been set correctly and then reload the VM’s configuration. Then, power on the VM and try again.

Manually running the cloud-config configuration

From the CoreOS CLI, run  coreos-cloudinit --from-vmware-guestinfo . This will pull the Guestinfo configuration from the hypervisor and try to parse it – this is a great way to discover if you’ve made an error in your cloud-config file or the encoding of it. If the data is all correct, you should see the following lines;

If the data has an error in it, you should see a message with the error along with which line of your cloud-config file caused the error. You can also try using CoreOS’s cloud-config validator tool to check the validity of your cloud-config configuration.

Enabling OpenWRT’s NTP Server

Normally, I configure my systems to synchronize their clocks over NTP to one of the NTP Pool servers. However, due to the load-balanced nature of the NTP Pool, clients must use fully qualified domain names in order to synchronize with NTP Servers. Most of the time, this isn’t a problem – most NTP clients fully support using FQDNs as opposed to IPs. However, a minority of clients, usually those in embedded hardware, only allow IPs to be used.

I don’t want to hardcode an external NTP server IP in to these systems, so I need a local NTP server on my network. OpenWRT’s NTP client already synchronizes with the NTP pool, so making it also act as a NTP server for internal clients is ideal. OpenWRT’s default client is the busybox NTP client, which can also act as a server – no need to install additional packages! However, by default OpenWRT only enables it as a client and not a server, but activating the server is rather simple.

  1. Edit /etc/config/system . Unfortunately, the NTP server isn’t configurable via the web interface so you’ll have to use the command line.
  2. Find the  config timeserver 'ntp' section. Underneath the existing NTP server configuration add the configuration option  option enable_server '1' . Save the file and quit.
  3. Restart the NTP service using the command  /etc/init.d/sysntpd restart .
  4. Done! OpenWRT will now act as a local timeserver.

To verify that the server really is active, you can grep  the output of  ps for ntpd . If the configuration was applied successfully, ntpd  should have be running with the -l  flag. Using another system, you could also query it using  ntpdate -q hostname , which will query the server and tell you the current offset of your local clock.

Although this won’t be as accurate as synchronizing directly with one of the NTP Pool servers, it should be more than accurate enough for most needs – with sub-second accuracy.

This was tested on OpenWRT Chaos Calmer 15.05, but should work with OpenWRT Backfire 10.3.1 onwards as per the documentation.

cu: unknown host

I’ll be honestly amazed if anyone reads this post – the number of people wanting to access a serial port from a FreeBSD system must be incredibly small. However, there’s only a few references to this problem out there so I thought it worth writing about.

When running  cu to connect to a serial port I was receiving the error “cu: unknown host”, followed by the hostname of my machine. cu consults /etc/remote  for port names, and an entry for my machine’s hostname wasn’t present (and I hadn’t passed an alternative hostname to cu ). The easy way to fix this was to set the host to an existing friendly name for the port, which in my case is uart0. The command to connect to the serial port then becomes;

cu -l cuau0 uart0

And presto! A connection to the serial port is established, no need to edit /etc/remote  either. The default settings use a baud rate of 9600, which can be overridden using the -s parameter.

Intel NUC Internal CEC Adapter by Pulse Eight Review

Pulse Eight recently released a new internal CEC adapter designed especially for use in Intel NUCs. I’ve used their external one for some time so that I can use my TV’s remote to control Kodi on the NUC and automatically switch HDMI inputs – it’s super convenient. However, it does consume an extra USB port, so I leaped at the chance to convert to using their new internal adapter instead. Postage was quick, I had the adapter in my hands the very next day and set about installing it.

The Pulse Eight Internal CEC Adapter for Intel NUCs, with screwdriver for scale. It's really tiny!

The Pulse Eight Internal CEC Adapter for Intel NUCs, with screwdriver for scale. It’s really tiny!

The adapter works by connecting to the following headers on the NUC’s motherboard;

  • The USB headers – for power and data.
  • The Front Panel header – to be able to trigger a press of the power button to switch on the NUC.
  • The Custom Solutions header – to be able to communicate via CEC over the HDMI cable.

Pulse Eight provide a wiring loom to connect the adapter to the NUC, with each individual cable having a separate connector so that they can reach each of the different headers. The downside of this however is that you need to consult the NUC’s documentation to find out exactly which part of the header is, for example, the USB data+ channel. Thankfully, for my NUC (D54250WYK) Intel have published a Detailed Product Specification document which lists all of the header pinouts. The documentation included with the adapter is unfortunately somewhat sparse, simply stating (to paraphrase) “connect these coloured cables to this header”. It would have been very useful if they had included instructions on exactly which connector needed to be connected to each part of the header, rather than forcing users to try and find the header pinouts. However, i’m guessing this was not done so that one package could be used across the (many) models of NUCs out there. The documentation I received also had a small error in it, with the USB and Custom Solutions headers being labelled the wrong way around. A small note included with the adapter points this out, and Pulse Eight have confirmed that the correct documentation is being reprinted at the moment.

On my NUC, the USB and Front Panel headers are on the top of the motherboard, which means it’s reasonably simple to connect them. However, the Custom Solutions header is on the underside of the motherboard, which is not so easy to get to. After removing the cover, two more screws need to be removed from the motherboard. While the motherboard is now loose, it’s really hard to remove. I found that I needed to apply some force to the Ethernet port whilst simultaneously flexing the rear of the case – it won’t flex much but it only needs to flex a small amount in order for the motherboard to pivot upwards.

USB and Front Panel Header Connections.

USB (r) and Front Panel (l) Header connections.

Connections on the Custom Solutions Header.

Custom Solutions Header connections.

Although not hugely problematic, the cables are extremely tight. An extra centimeter or two of length would have made it much easier to route the cables between the various connectors. The suggested mounting for the adapter is on top of the Ethernet port shield, however I found that I had to mount it slightly off it so that the bottom of the case would fit on correctly again.

Once connected and reassembled, Ubuntu and Kodi immediately picked up the new adapter and it worked flawlessly in conjunction with my TV just as the old external adapter did.

Conclusions

Whilst installation can be somewhat fiddly and difficult, once installed the adapter performs extremely well without needing to consume any external USB ports and keeping the sleek looks of the NUC intact. The installation could be made simpler if the documentation was more comprehensive (and correct), but with the correct documentation, Intel’s Detailed Product Specification document and the tips above about removing the motherboard, it should be reasonably simple for anyone to install.

The internal adapter is £10 cheaper than the external adapter so unless you’re looking to be really lazy I would strongly recommend the internal adapter over the external one. The functionality between them is exactly the same.

The adapter can currently be purchased directly from Pulse Eight’s website for £20, with free postage in the UK.

Zen and IPv6

I’ve run IPv6 at home for a while now using a tunneled connection provided by SIXXS. However, my ISP (Zen) recently announced that they are trialing native IPv6. Naturally, I signed up as soon as I could – tunneled IPv6 is good, but native IPv6 is better! I was activated on the trial today and i’m currently writing this post using a fully native IPv6 connection.

Configuration

My router runs OpenWRT, so I was expecting setup to be reasonably simple. In the end, it wasn’t as simple as I was expecting but that was due to a lack of knowledge anywhere on how to get this going. The simple configuration is below, with an explanation afterwards. Don’t start applying this configuration until;

  1. Zen has confirmed you’re on the IPv6 trial, and has sent you an email with your IPv6 addresses.
  2. You have rebooted your FTTC modem. IPv6 isn’t enabled until your PPPoE session has restarted!

First, you’ll need to edit your /etc/config/network  file so that it has the following entries;

Then, make sure the new wan6  interface is present in your firewall zone configuration;

Finally, commit the configuration changes using  uci commit and then reload the network configuration by running the command  /etc/init.d/network reload . If you run ifconfig pppoe-wan, you should see that you have a shiny new /64 IPv6 address as well as your existing IPv4 address, hurrah!

Testing

The easiest test is to run a ping test against Google using ping6 ;

Prefix Delegation

Zen give you two prefixes;

  • /64 Neighbour Discovery – This is the address assigned to your WAN interface, for communication with Zen’s upstream network.
  • /48 Delegation Prefix – This is assigned to hosts on your LAN network.

OpenWRT’s DHCPv6 implementation will automatically grab both prefixes, assign them to the appropriate interfaces and act as a DHCPv6 server on the LAN interface using the delegated prefix. Any LAN devices that use automatic address assignment for IPv6 should quickly pick up an address from the delegated prefix.

Explanation

So, why do two interfaces need to be configured when the existing WAN connection has an IPv6 option? The existing PPPoE WAN connection is a virtual point-to-point (hint: PPP!) connection that runs over your physical (e.g. eth0) ethernet connection (hint: oE). Therefore, we need to create a separate virtual interface that will use DHCPv6 over the existing PPPoE connection, which is where the wan6 connection comes in. It is absolutely vital that the ifname configuration for the wan6 interface uses @wan (a variable that refers to the virtual PPPoE connection) instead of the physical connection, otherwise an IPv6 address won’t be obtained.

Conclusion

So far, i’ve only had this up and running for a number of hours, but everything seems rock solid so far. Latency is a few milliseconds lower compared to the tunneled connection, and being able to run at a “native” MTU rather than a reduced one for tunneling is also advantageous.

Hopefully this helps if you’re also getting started on Zen’s IPv6 trial. Good luck!