Some notes on how to diagnose problems on Cisco switches running CatOS 1. Looking at switch stats/status for a node interface. Mac may have a wrapper for this, but here is the guts of what you have to do. To look at the switch port for a particular enet card, first figure out what switch port it is connected to! mysql tbdb select * from wires where node_id1='tbpcXX'; What you care about are node_id2/card2/port2. Node_id2 tells you which cisco: "cisco" is the testbed cisco (``tip test'') and "cisco2" is the control net cisco (``tip control''). card2/port2 give you the info you need for the cisco-style card/port name (e.g., 3/39). [Note: There are also a couple of other easier ways to do this: Rob has created a command called if2port: 108 paper:~> if2port Usage: /usr/testbed/sbin/if2port 109 paper:~> if2port tbpc06 +--------+-----+--------+-----+-----+ |node_id1|card1|node_id2|card2|port2| +--------+-----+--------+-----+-----+ |tbpc06 | 0|cisco | 3| 42| |tbpc06 | 1|cisco | 3| 44| |tbpc06 | 2|cisco | 3| 46| |tbpc06 | 3|cisco | 3| 48| |tbpc06 | 4|cisco2 | 3| 11| +--------+-----+--------+-----+-----+ 5 rows processed Another useful way is with 'snmpit -l -debug'. This will give all the tbpcXX:Y <==> ciscoport translations for any port currently in use, as well as listing the VLANs currently set up: 111 paper:~> snmpit -l -debug DEBUG MODE ON: Set to level 1 Command line was: snmpit -l -debug Use of uninitialized value at /usr/testbed/bin/snmpit line 1281. READING TRANSLATIONS Opening SNMP session to 155.101.128.175...Succeeded Getting VLAN info... Got default vtpVlanName 1.1 (1) default Got tact-reserve2-l0-0 vtpVlanName 1.2 (2) tact-reserve2-l0-0 Got brandeis-BuddyCache-l0-0 vtpVlanName 1.3 (3) brandeis-BuddyCache-l0-0 Got tact-reserve2-l0-1 vtpVlanName 1.4 (4) tact-reserve2-l0-1 ... Got 3 vlanPortVlan 6.34 3 ('6.34' == tbpc23:0) Got 19 vlanPortVlan 6.41 19 ('6.41' == tbpc21:0) Got 19 vlanPortVlan 6.42 19 ('6.42' == tbpc24:0) Got 19 vlanPortVlan 7.25 19 ('7.25' == tbpc25:0) Got 19 vlanPortVlan 7.34 19 ('7.34' == tbpc29:0) Got 3 vlanPortVlan 7.41 3 ('7.41' == tbpc27:0) Got 19 vlanPortVlan 7.42 19 ('7.42' == tbpc30:0) Got 19 vlanPortVlan 8.33 19 ('8.33' == tbpc32:0) Got 5 vlanPortVlan 8.42 5 ('8.42' == tbpc36:0) Got 8 vlanPortVlan 9.33 8 ('9.33' == tbpc38:0) ID Name Members of VLAN -------------------------------------------------- 1 default 2 tact-reserve2-l0-0 tbpc04:0 tbpc10:0 3 brandeis-BuddyCache-l0-0 tbpc20:0 tbpc22:1 tbpc23:0 tbpc27:0 4 tact-reserve2-l0-1 tbpc04:1 tbpc13:0 5 agile-test001-l0 tbpc36:0 6 janos-moab-l0 tbpc02:0 tbpc03:1 7 janos-moab-l1 tbpc01:0 tbpc03:0 8 agile-afreenet-l0 tbpc06:0 tbpc38:0 12 magi-test1-l0 tbpc07:0 tbpc11:0 18 _mylan tbpc15:0 tbpc16:0 tbpc18:0 tbpc19:0 19 __mylan tbpc21:0 tbpc24:0 tbpc25:0 tbpc29:0 tbpc30:0 tbpc32:0 The interesting translations are just above the vlan table, in the far right hand columns. For example, the last line before the table indicates that in vlan 8 is port 9.33, which belongs to tbpc38:0. ] Armed with this info, tip to the correct cisco, login and enable. Then you can do: show port status card/port or show port card/port for everything. If, say, the port is disabled, you can do: set port enable card/port 2. Checking on the firewall rules You have to login to the "control" Cisco and then "session 15" to connect to the Router module. While at the Router> prompt, you will get any "access denied" type messages that the router produces, ala: 23w2d: %SEC-6-IPACCESSLOGP: list control-shark denied \ udp 0.0.0.0(0) -> 255.255.255.255(0), 602 packets If you suspect that some rule is preventing your traffic from getting through, then try generating your traffic while you are connected to the router and see if you get errors. To see the whole lists in all their ugliness, type: show ip access-lists The rules are pretty straightforward. First match wins. Netmasks are bass-ackwards (intead of 255.255.255.0, you would use 0.0.0.255). Each list is applied both on entrance and exit to the like-named control network VLAN. 3. Changing the firewall rules ROB: Working on this part 4. Finding MAC address information To find which port a given MAC address is on type (on the switch console): show cam where MAC is colon-seperated, like 08:00:2b:81:62:d3. To show all MAC addresses in a given VLAN, type: show cam dynamic where VLAN is the number, not the name. 5. Deleting a "sticky" ARP entry If you should ever be so unfortunate as to have to replace a faulty shark, in addition to recording the new MAC address in the DB and DHCP config file, you may also need to clear it from the router module. If you fire up a new shark, and it says that it cannot get its DHCP info, this is likely the problem. To find out, login to the control Cisco and "session 15" to get to the router module. You should start seeing periodic 24w2d: %IP-3-STCKYARPOVR: Attempt to overwrite Sticky ARP entry: \ 155.101.130.73, hw: 0800.2b81.62d3 by hw: 0800.2b81.611b messages. To clear the arp entry (actually the whole cache), enable at the Router> prompt and then do "clear arp". 6. Replacing a node/NIC If you replace a node, you'll need to change the secure MAC address for that port. The following command should work: Console> (enable) set port security 3/1 enable 01-02-03-04-05-06 Of course, use the real port number and MAC address (noting the funky MAC syntax). Note that you will proably also need to use the 'Sticky ARP Entry' clearing procedure covered above. 7. Checking on port security To find out what MAC address(es) are associated with a given port. use: Console> (enable) show port security To find out if a given port has been disabled for being a Bad Boy (tm) w/ respect to MAC addresses, use: Console> (enable) show port - the state will be 'disabled', and you should see some information on the security violation To re-enable a port after it has been disabled due to security violations: Console> (enable) set port enable To disable security for a port: Console> (enable) set port security disable 8. Manual VLAN configuration - From the switch command line To see a list of all configured VLANs, use: Console> (enable) show vlan On the control net, all of the VLAN names should be self-explanatory Adding a port to a VLAN is very easy. Just type: Console> (enable) set vlan ... where is the number of the VLAN, and is the port (you can use the 'if2port' script to get the port number) To 'remove' a port from a VLAN, set it to VLAN 1. To create a new VLAN, use: Console> (enable) set vlan name ... where is some unused VLAN number (use 'show vlan' to find one), and is some descriptive string To delte a VLAN use: Console> (enable) clear vlan ... where is the VLAN number (duh!) NOTE: This puts all of the VLANs ports back into VLAN 1, and disables them. Use 'set port enable ' to re-enable it 9. Cloning all traffic from a port or VLAN to another port Pick a port to recieve the traffic - let's call it To forward the traffc from one port: Console> (enable) set span To forward the traffic from an entire VLAN: Console> (enable) set span NOTE: You might want to append 'rx' or 'tx' to the VLAN command line, or you'll get doubles of everything (incoming and outgoing both) NOTE2: If you want to be able to send traffic from the monitor port, you'll need to append 'inpkts enable' ala: Console> (enable) set span 18 3/9 rx inpkts enable To stop cloning: Console> (enable) set span disable NOTE: as of 11/5/01 we have fxp1 on plastic attached to the control net so that it can be used for cloning. Its cisco2 port is 3/23, use that for . Also note that you will have to have fxp1 configured in order to run tcpdump on it. I just do: ifconfig fxp1 inet 192.168.1.1 netmask 0xffffff00 and then: ifconfig fxp1 down when I am done. 10. Good reference for making nodes boot quickly (ensuring that port spanning, trunking, etc. are off on the switch port): http://www.cisco.com/warp/public/473/12.html The gist is that we should use the convenient: set port host To turn on fastport (disable spanning tree) and turn off channeling (combining multiple ports to make a fast link) and trunking (a single port serving multiple VLANs).