I’m honestly not sure the world needs another “here’s how I solved my problem with Crossbow blog post – it’s been covered pretty well already by Nicholas, Chris, Ben, Joerg and likely many others, but still – adding to the collective knowledge of your search engine of choice, here’s my version:


Original image by ToastyKen
Original image by ToastyKen

Introduction

Support for
virt-install to initiate PV and HVM installs of OpenSolaris guests

using OpenSolaris AI (Automated Installer) is on it’s way once our 3.3 wad integrates, and for the most part, I did
that work on a standalone network of two machines. This meant that I had to go into
the office every time I wanted to try something out. Of course, with Crossbow, I
needn’t have gone to that much trouble – so, in case anyone else finds themselves in
the same situation, here’s how to do it.

Setting up an AI server for testing on a network where you don’t own the dhcp
server can be a pain. However, given a .vdi that already
houses the AI server, all you need is the right network configuration in
your host to make testing AI + xVM easy – thanks Crossbow! I now just need to ship
around a domU .vdi with my AI server on it, and the corresponding virsh xml definition
and I can have it running on any machine in a jiffy.

The things you want are:

  1. a dedicated etherstub in dom0 that you can create vnics from
  2. vnics on that etherstub, each with
    a local network address, one of which is plumbed in your host
  3. ipnat rules in dom0 to allow that local network out
    through your primary interface, and ssh into the guest
    on defined ports
  4. ipfilter rules in dom0 to block stuff you don’t want reaching the
    outside
  5. ( and visa versa, perhaps providing port redirection into your guest )
Implementation

Here’s what you need to run:

# dladm create-etherstub timswitch0
# dladm set-linkprop -p mtu=1500 timswitch0
# dladm create-vnic -l timswitch0 timnic0
# ifconfig timnic0 plumb 192.168.1.1 netmask 255.255.255.0 up
# (if necess.) ifconfig timnic1 plumb
otherwise, pass virt-install timswitch0, and the vif-vnic script
will create a vnic on top of it to plug guests into that etherstub
# routeadm -u -e ipv4-forwarding

then having edited /etc/ipf/ipf.conf and /etc/ipf/ipnat.conf (see below)

# svcadm enable ipfilter
# ipnat -f /etc/ipf/ipnat.conf

This gives us the following configuration:

# dladm show-link
LINK        CLASS    MTU    STATE    OVER
ath0        phys     1500   up       --
atge0       phys     1500   unknown  --
timswitch0  etherstub 1500  unknown  --
timnic1     vnic     1500   up       timswitch0
timnic0     vnic     1500   up       timswitch0
# ifconfig -a
lo0: flags=2001000849 mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
ath0: flags=1104843 mtu 1500 index 15
inet 10.0.0.10 netmask ffffff00 broadcast 10.0.0.255
ether 0:15:af:6a:64:ec
atge0: flags=201100843 mtu 1500 index 16
inet 0.0.0.0 netmask ff000000 broadcast 0.255.255.255
ether 0:1e:8c:bb:8e:7a
timnic0: flags=1100843 mtu 1500 index 18
inet 192.168.1.1 netmask ffffff00 broadcast 192.168.1.255
ether 2:8:20:7a:17:4f
lo0: flags=2002000849 mtu 8252 index 1
inet6 ::1/128
ath0: flags=2004841 mtu 1500 index 15
inet6 fe80::215:afff:fe6a:64ec/10
ether 0:15:af:6a:64:ec

Here’s what you need in /etc/ipf. Replace 'ath0' with your primary interface.
In ipnat.conf we redirect port 9022 to the ssh port in our guest on 192.168.1.2 (the guest
containing the AI server) so to ssh in to the guest, do:

# ssh -p 9022 <IP address of dom0>
----------- /etc/ipf/ipnat.conf --------------
map ath0 192.168.1.0/24 -> 0/32 portmap tcp/udp auto
map ath0 192.168.1.0/24 -> 0/32
rdr ath0 0.0.0.0/0 port 9022 -> 192.168.1.2 port 22
--

Various IP filter rules applied in dom0. At the moment, we’re logging
packets that aren’t allowed through timnic0. Run ‘ipmon’ in dom0 to watch
this log – useful when debugging (and choosing what other rules to add)

----------- /etc/ipf/ipf.conf -------------------
#
# ipf.conf
#
# IP Filter rules to be loaded during startup
#
# See ipf(4) manpage for more information on
# IP Filter rules syntax.
# allow ssh into our guest from to anywhere
pass out quick on timnic0 from any to any port = 22 keep state
# allow dns, http, ssh in from timnic0, remember we're using NAT here,
# and we're a router, so this is actually traffic from the guest network
pass in quick on timnic0 proto udp from any to any port = 53 keep state
pass in quick on timnic0 proto tcp from any to any port = 22 keep state
pass in quick on timnic0 proto tcp from any to any port = 80 keep state
# allow anything at all inside our guest network
pass in quick on timnic0 from 192.168.1.0/24 to 192.168.1.0/24
pass out quick on timnic0 from 192.168.1.0/24 to 192.168.1.0/24
# allow nothing else through timnic0
block in log quick on timnic0
block out log quick on timnic0
--

In a guest, we see:

root@opensolaris:/tmp# ifconfig -a
lo0: flags=2001000849 mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
xnf0: flags=1004843 mtu 1500 index 2
inet 192.168.1.45 netmask ffffff00 broadcast 192.168.1.255
ether 0:16:36:5f:7f:c5
lo0: flags=2002000849 mtu 8252 index 1
inet6 ::1/128
root@opensolaris:/tmp# netstat -rn
Routing Table: IPv4
Destination           Gateway           Flags  Ref     Use     Interface
-------------------- -------------------- ----- ----- ---------- ---------
default              192.168.1.1          UG        1         76 xnf0
192.168.1.0          192.168.1.45         U         1         31 xnf0
127.0.0.1            127.0.0.1            UH        1         28 lo0
Routing Table: IPv6
Destination/Mask            Gateway                   Flags Ref   Use    If
--------------------------- --------------------------- ----- --- ------- -----
::1                         ::1                         UH      1       0 lo0

Update: had a typo in one of the commands above, thanks for spotting it seanmcg

Advertisements