##### ##### Configuration suggestions for Cisco switches ##### The purpose of this document is to aid in designing and setting up the control and experimental networks on other emulabs. First a bit of background: On the the control network will be your control nodes (boss, ops, any tipservers, etc.), control hardware (SNMP-controllable devices such as power controllers and switch IP interfaces), your connection to the outside world, and the control interfaces of your nodes. So far, we haven't tried to distribute the control net across multiple switches, but this should be theorically possible. You'll need to set up the VLANs, etc. on the control net by hand. The experimental network will consist of one or more (this, we have tested) switches connected with trunk (802.1q, or proprietary, like Cisco ISL) lines. Most configuration of these switches will be taken care of by our software. (See setup-cisco.txt for some configruation options you many want to apply to these switches if they are Ciscos.) ##### Splitting up the control net We basically have 5 VLANs on the control network: 'external' contains our connection to the outside world 'private' contains the boss node, and our backup server 'public' contains our ops node 'control' contains the control net interfaces of all experimental nodes 'control-hardware' contains all IP-controllable devices (namely, power controllers and switch IP interfaces, as well as a second interface on the boss node. This VLAN uses private IP addresses, and does NOT contain a router interface. This is done for security - we route (using a module in our control-net switch) between these VLANs, and do some firewalling between each of them. The main goals are: 1) Protect both control and experimental nodes from the outside world (and vice-versa - we don't want people attacking the outside world from our nodes) 2) Protect the control nodes from the experimental nodes 3) Protect the control hardware (power controllers, etc.) from nodes and the outside world 4) Protect the boss node (which is _not_ publically accessible) from the ops node (which all experimenters have shells on.) Now, it's entirely possible to combine these VLANs into one big one - this is what we've done on our mini-testbed here. But, there are some serious security implications with doing it this way - namely, that the nodes can theoretically impersonate each other, power cycle each other, and all kinds of nasty things. At the very least, you should have a firewall between your testbed and the outside world, to satisfy #1. It is also a good idea to separate the nodes' control net into a separate VLAN, which satisfies #2 and #3. After all, you are giving people root access to the experimental nodes. In situations where you are only giving acess to a small number of trusted people, this is probably not too big a deal, but once access gets outside the small circle of your friends, or if you are allowing students access, then taking these precautions are a very good idea. If possible, putting boss and ops on separate networks is also a good idea (#4), but is probably the least important part to worry about. Basically, it's up to you to decide how much security you want to worry about. I'd recommend at least going after #1, #2, and #3. We've put a lot of thought into this, so if you're wondering how your choice here will affect other aspects of security, ask Utah, and we can probably help you. Since our firewall rules change frequently as we add new services, it's best to contact Utah and ask us for the current set. ##### Connecting the contol net to the experimental net In order to be able to control the experimental switches (ie. create new VLANs, etc.) you need a way to talk to them. So, you'll need one line that goes from your control net to the experimental net. However, you will need to be _very_ careful that no experimental traffic can leak across this line, and that it cannot be used as a 'back door' into the private net by nodes. On the control switch, it should be put into the 'control-hardware' VLAN. On the experimental switch, it should be in the same VLAN as the switch's IP interface. The best thing to do is create a new VLAN, and move the switch's IP interface into it. On CatOS, you accomplish this with: set interface sc0 You will also need to give the experimental switch(es) IP addresses. With CatOS, you do it like this: set interface sc0 set interface sc0 up ##### DHCP through the router If your boss node is on a seperate VLAN from the node control net, you'll need to make sure that DHCP traffic can get from the control net to your boss node, since normally, DHCP is not forwarded through routers. On Cisco routers, this is done with the 'ip helper-address'. For example, here, the name of the router's interface in the node control net is 'Vlan3'. So, I'd log into the router, and run the following: configure terminal interface Vlan3 ip helper-address 155.101.128.70 Of course, replace 'Vlan3' with the name of your router's node control-net interface, and replace the IP address with that of your boss node. ##### IGMP snooping on the control net In order for multicast to work correctly, you need to make sure that IGMP snooping is enabled on the control switch. This is needed for frisbee, our disk-loading system. It's up to you wheter you want to enable this on the experimental switches. On CatOS, the command is: set igmp enable #### VTP domains If you have only one Cisco experimental-net switch, you don't need to have it using VTP, which is a mechanism for keeping VLANs in sync across multiple switches. You do this (in CatOS) with: set vtp mode transparent If you have multiple experimental switches connected by trunk lines, you should use FTP. Pick a domain name (we call ours simply 'Testbed',) and run the following on all of your switches: set vtp domain Pick one switch to be the master - it doesn't really matter which one. (See the switch setup insturctions in setup-db.txt, and make sure you name the stack after the master switch.) On this one, run set vtp mode server On all the others, run set vtp mode client