New http_directory setting seems to break boot_wait

[HariSekhon]

When enabling http_directory in #75 it seems to break boot_command - the keystrokes aren't sent through to control the boot for 30 seconds which is too late, despite boot_wait = "5s" being set.

I wonder if it's because of this line:

https://github.com/cirruslabs/packer-plugin-tart/pull/58/files#diff-ca2528fec399d3e12f23b8a1dbe42423edb6ac4e0b6f414d101cb2b14497a562R55

[edigaryev]

This is likely caused by this branch of code:

packer-plugin-tart/builder/tart/step_run.go

Lines 124 to 134 in 9ddd019

	func typeBootCommandOverVNC(
	ctx context.Context,
	state multistep.StateBag,
	config *Config,
	ui packersdk.Ui,
	tartRunStdout *bytes.Buffer,
	) bool {
	if config.HTTPDir != "" || len(config.HTTPContent) != 0 {
	ui.Say("Detecting host IP...")
	hostIP, err := detectHostIP(ctx, config)

If you use http_directory or http_content, we need to figure out the IP-address on which the VM will be able to reach the HTTP server running on host, and this requires a VM to obtain an IP address first, which takes time.

You can set http_bind_address manually to a value other than 0.0.0.0 to override this behavior, see also #63.

[HariSekhon]

That would seem to explain the behaviour I'm seeing, but I've used http_directory setting with the packer builder plugin for VirtualBox and it didn't have this delay in sending keystrokes, which implies it didn't have to wait such a long time to determine the host VM as I was passing it this line:

ds=nocloud-net;s=http://{{.HTTPIP}}:{{.HTTPPort}}/

so I think either I'm missing something or there is something more that needs to be done here.

Is it possible this is just some logic or optimization problem in the tart packer plugin itself since the VirtualBox packer plugin seems to have solved this?

Also, isn't the host IP usually the same IP each time, usually the first or second IP on the VM network. For example, I had created an alternative config where I was running my own web server locally and passing this to the tart VM:

'ds=nocloud-net;s=http://192.168.64.1:8000/'

which I'd reverse engineered from tty2 console on a tart VM itself and this worked consistently for me...

I just tried setting in the pkr.hcl:

http_bind_address = "192.168.64.1"

and the whole VM just hangs for 18 minutes so far:

$ packer build --force ubuntu-arm64.tart.http.pkr.hcl
ubuntu-22.04.tart-cli.ubuntu: output will be in this color.

<no_output>

[HariSekhon]

After leaving it for over an hour it was still stuck.

Do I commented out the http_bind_address line and tried again:

packer build --force ubuntu-arm64.tart.http.pkr.hcl
ubuntu-22.04.tart-cli.ubuntu: output will be in this color.

==> ubuntu-22.04.tart-cli.ubuntu: Starting HTTP server on port 8798
==> ubuntu-22.04.tart-cli.ubuntu: Creating virtual machine...
==> ubuntu-22.04.tart-cli.ubuntu: Starting the virtual machine for installation...
==> ubuntu-22.04.tart-cli.ubuntu: Detecting host IP...

<stuck here for over 30 seconds - missed the boot loader's 30 second timeout>
<eventually continued but boot is hosed by that point>

==> ubuntu-22.04.tart-cli.ubuntu: Host IP is assumed to be 192.168.64.1
==> ubuntu-22.04.tart-cli.ubuntu: Waiting for the VNC server credentials from Tart...
==> ubuntu-22.04.tart-cli.ubuntu: Retrieved VNC credentials, connecting...
==> ubuntu-22.04.tart-cli.ubuntu: Connected to the VNC!
==> ubuntu-22.04.tart-cli.ubuntu: Typing the commands over VNC...
==> ubuntu-22.04.tart-cli.ubuntu: Waiting for the install process to shutdown the VM...

It looks like it failed to detect the host IP after a long period and missing the boot loader countdown

==> ubuntu-22.04.tart-cli.ubuntu: Host IP is assumed to be 192.168.64.1

perhaps this is why it's hanging and not sending the boot_command?

[edigaryev]

and the whole VM just hangs for 18 minutes so far:

This is probably because the interface with 192.168.64.1 address doesn't exist yet at the time of HTTP server creation and Packer's plugin SDK does not discern between bind: address already in use and bind: address family not supported by protocol family errors when starting the HTTP server, so it automatically re-tries the same operation over and over, but with another port and without logging any errors because it thinks that it's due to the occupied port.

I think you can fix this by binding to the IP-address that your macOS machine got from your LAN/WLAN, for example:

http_bind_address = "192.168.0.50"

Note that this will make your HTTP server available from the LAN too.

[edigaryev]

Regarding the root cause, do I understand correctly that even if you increase the GRUB_TIMEOUT in your Ubuntu autoinstall configuration, the VM will only get it's IP address via DHCP after booting?

[HariSekhon]

So I tested setting the http_bind_address to the same IP my laptop has on the LAN and yes that does actually work around issue of not sending keystrokes in time for the bootloader and also the permanent hang from combining http_directory and http_bind_address.

The only downside is that being a DHCP address, this is not a permanent solution to packer configs on github which will need to run across machines of unknown IP addresses, run by people on the internet or shared with colleagues etc.

I mean I could easily script a search and replace of the local machine's IP address but it'd be such an ugly hack... aside from exposing the autoinstaller containing hashed passwords on the LAN...

[HariSekhon]

How would I set GRUB_TIMEOUT in a packer build?

Ubuntu's AutoInstall isn't triggered because the keystrokes aren't sent and besides grub bootloader happens before autoinstaller...

Apologies if I'm missing something simple there.

[Sn0wCrack]

This still seems to be an issue when using from_iso.

Trying to fetch the IP address of the newly created Tart VM before Packer timesout seems to just return no IP address found.

Tart doesn't appear to assign IPs to VMs, instead solely relying on DHCP to do this, this means getting an IP address to perform cloud-init from is not possible, at least on Ubuntu installation media.

[edigaryev]

Tart doesn't appear to assign IPs to VMs, instead solely relying on DHCP to do this, this means getting an IP address to perform cloud-init from is not possible, at least on Ubuntu installation media.

We provision Ubuntu with Cloud Init in github.com/cirruslabs/linux-image-templates quite successfully, see cloud-init.pkr.hcl for example.

Also, under the hood from_iso is simply converted to --disk command-line arguments to tart run.

If you could tell us a bit more about your requirements and setup, for example, how your Packer templates look like, we’d be happy to help.

[Sn0wCrack]

We provision Ubuntu with Cloud Init in github.com/cirruslabs/linux-image-templates quite successfully, see cloud-init.pkr.hcl for example.

Also, under the hood from_iso is simply converted to --disk command-line arguments to tart run.

If you could tell us a bit more about your requirements and setup, for example, how your Packer templates look like, we’d be happy to help.

Thanks for the tip, I actually ended up finding this myself the other day and applying it. Everything seems to work fine after doing that.

Was more or less confusion on my part due to this plugin working a bit differently to how most other standard packer plugins work in addition to some of the networking differences inherent to Apple's virtualization framework.