EMULAB Setup in Orbit Project

[Installation & Setup |  Components & Function | Analysis of Design | Patch & BugFix | Reference]

Working Group: 

Basic install procedures

  1. Install FreeBSD on the control nodes ( Boss and Ops)  and test nodes.
  2. Configure Network
  3. Install sourcecode from EMULAB
  4. Test basic functions.

Extra Solution Packages

Reference:

Acronyms
Test Node Diagram 
Disk Partition in FreeBSD
It is required that in Boss machine, \var  partitions must be larger than 100MB and /usr partition should be at least 2G. Using "df" command to check.
#df
Filesystem  1K-blocks    Used    Avail Capacity  Mounted on
/dev/ad0s1a    128990   76198    42474    64%    /
/dev/ad0s1f    257998    1262   236098     1%    /tmp
/dev/ad0s1g  74232440 1536862 66756984     2%    /usr
/dev/ad0s1e    257998    9624   227736     4%    /var
procfs              4       4        0   100%    /proc

Enable SSH service in Boss && Ops
To start up this, just add an line in  \etc\rc.conf   
        sshd_program="/usr/sbin/sshd"
Configure IP addresses
  a) DHCP
        ifconfig_em0='dhcp'
       dhcp_flags ='yes'  (for boss server)
  b) fixed IP address configuration
        ifconfig_fxp0=" inet 10.0.0.2 netmask 255.255.255.0"
        defaultrouter =" 10.0.0.1"
c) edit etc/hosts and etc/resolv.conf files

To make the KDE display look better, it is better to configure XFree86 with two extra lines:
defaultdepth = 24
.......
Modes = "1024x768"

To start KDE when "startx", create a file .xinitrc in /root with 1 line code "startkde".
FreeBSD Kernel Configuration file is loacted in /usr/src/sys/i386/conf directory

Serial Console  is a basic method to diagnoses the system when Ethernet port does not work.
Basically, default console is "internal" means "screen"(of monitor).

1. Purpose of Boot loader
During initialization, the loader will probe for a console and for disks, and figure out what disk it is booting from. It will set variables accordingly, and an interpreter is started where user commands can be passed from a script or interactively.
The loader will then read /boot/loader.rc, which by default reads in /boot/defaults/loader.conf which sets reasonable defaults for variables and reads /boot/loader.conf for local changes to those variables. loader.rc then acts on these variables, loading whichever modules and kernel are selected.

In FreeBSD, You have to make a file called boot.config containing /boot/loader -h. All this does is pass a flag to the bootloader to boot into a serial console. 
# echo "/boot/loader -h" > boot.config
And, the other way is to specify the boot loader and the kernel to use the serial console by writing just one line in /boot/loader.rc:
set console=comconsole
This will take effect regardless of the settings in the boot block discussed in the previous section.
To make a login prompt on serial console, you have to
Open the file /etc/ttys with an editor and locate the lines: 
ttyd0 "/usr/libexec/getty std.9600" unknown off secure
ttyd1 "/usr/libexec/getty std.9600" unknown off secure
ttyd2 "/usr/libexec/getty std.9600" unknown off secure
ttyd3 "/usr/libexec/getty std.9600" unknown off secure
ttyd0 through ttyd3 corresponds to COM1 through COM4. Change off to on for the desired port.
 

More sophisticated technique for serial console: If you want all messages re-direcred to SERIAL port, the BIOS of motherboard must support it. So, generally, I do folowing on test node1 to get a Console both on COM1 and Video console.

My notes on getting a serial console at 115200

-must be com1
-com1 must be at port 0x3F8 irq 4
-in bios set the port and irq as above
-in bios set serial redirection to com1
-in bios set baud rate 115200
-in bios set RTS/CTS flow control (optional)
-edit (or create) /etc/make.conf to add these lines:
BOOT_COMCONSOLE_PORT=   0x3F8
BOOT_COMCONSOLE_SPEED=  115200
-cd /sys/boot
-make clean
-make
-make install
-fdisk -B
No im not kidding.  Part of the boot knowing baud rate loader lives in
the main disk boot block.
-cd /boot
-edit loader.conf
-add a line:
console=comconsole
-edit /boot.config make it read (with a return after it):
-Dh
(the above is minus D h return, thats 4 characters)
-cd /usr/src/sys/i386/conf
-edit GENERIC (or whatever your kernconf is called)
-add:
options               CONSPEED=115200         # Console Redirection
-cd /usr/src
-make buildkernel KERNCONF=GENERIC
-make installkernel KERNCONF=GENERIC
-open /etc/ttys
-change ttyd0 "/usr/libexec/getty std.9600" unknown off secure" to "115200" "vt100" and "on"
-reboot



Configure Switches:
1. Connect new Cisco 2950 switch to a serial console
FreeBSD COM port is named as "sio0" which is mapped to /dev/ttyd0
Using minicom and a serial cabel connected to the CONSOLE port of the back panel of the CISCO Switch. Basic serial port setting in minicom is 8N1, 9600bps and No hw and Sw flow control.
Then it will prompt you the configuration questions:

And, then configure management  IP Address as:  (display in minicom console)
Enter interface name used to connect to the
management network from the above interface summary: vlan1
Configuring interface Vlan1:
  Configure IP on this interface? [yes]: yes
    IP address for this interface: 10.0.0.254
    Subnet mask for this interface [255.0.0.0] :
    Class A network is 10.0.0.0, 8 subnet bits; mask is /8
Would you like to enable as a cluster command switch? [yes/no]: no

After that,  the switch will build a initial configuration. And, then we can connect any port of the switch with a Ethernet cable, to a terminal port with address in same IP segment.
For example, I am using "10.8.0.1" in fxp0 of boss node, then "10.8.0.254" will work.
The hostname of the switch is set to "data0"

2. About Control Switch ( Cisco 2900)
Enable Secret Password: ?????
Enable Password :  **23
Terminal Password *******3

Enable PortFast in Control Switch



System Diagram && Hardware in Network:

10.0.0.1  ------ D-link Firewall.  (192.168.120.131)
10.0.0.2  ------ Boss Server.       (192.168.120.132)
10.0.0.3  ------- Ops Server.      (192.168.120.133)
10.0.0.10 ----- 10.0.0.254 range (divided to small ranges)
10.0.0.254 ----- Control Switch ( Cisco -2900XL)
Begin with 10.1.0.0.....  DMZ range for data switch

VLAN on Control Switch?
Instructions from Utah, attached as setup-network.txt
##### Splitting up the control net

We basically have 5 VLANs on the control network:
'external' contains our connection to the outside world
'private' contains the boss node, and our backup server
'public' contains our ops node
'control' contains the control net interfaces of all experimental nodes
'control-hardware' contains all IP-controllable devices (namely, power
	controllers and switch IP interfaces, as well as a second
	interface on the boss node. This VLAN uses private IP addresses,
	and does NOT contain a router interface.

This is done for security - we route (using a module in our control-net switch)
between these VLANs, and do some firewalling between each of them. The main
goals are:
1) Protect both control and experimental nodes from the outside world (and
   vice-versa - we don't want people attacking the outside world from our nodes)
2) Protect the control nodes from the experimental nodes
3) Protect the control hardware (power controllers, etc.) from nodes
   and the outside world
4) Protect the boss node (which is _not_ publically accessible) from the ops
   node (which all experimenters have shells on.)

Now, it's entirely possible to combine these VLANs into one big one - this is
what we've done on our mini-testbed here. But, there are some serious security
implications with doing it this way - namely, that the nodes can theoretically
impersonate each other, power cycle each other, and all kinds of nasty things.

At the very least, you should have a firewall between your testbed and the
outside world, to satisfy #1.

So, vlan is used to separate one switch into several independent switches.
Router or ( A router-like machine with multiple ports) is across those VLANs.
As  we have no routing module in 2900XL. we don't need create those vlans.

Configure D-Link Firewall
username:  admin
password: *****523
LAN zone:
External to Internet:
DMZ zone:
Disable DHCP service (Important!)
Mapping 192.168.120.132  ----> 10.0.0.2
Mapping 192.168.120.133  ----> 10.0.0.3
Service Setup:
Outgoing: Any for all options
Incoming: HTTP & HTTPS for 192.168.120.132 and SSH for 192.168.120.133.
So, with this configuration, the ping service is not provided to outsiders. And, in internal machines, you will not hear any reply if you ping "www.google.com" or other websites.

Configure DHCP server on Boss
"isc-dhcp-3.0" is installed with the FreeBSD package. as there is already a file "2.dhcpd.sh" in the /usr/local/etc/rc.d/ directory, we only need to create a "dhcpd.conf" file in /usr/local/etc/ directory, set IP range as 10.0.0.5.-10.0.0.15.  First we change  a templete file. Then, generate the real conf file with following command:
/usr/testbed/sbin/dhcpd_makeconf dhcpd.conf.template > dhcpd.conf
The port 67 will be used for DHCP and proxydhcp will occupy this port if it get loaded first. So, be sure to start dhcp first. (Refer to the PXEBOOT ). This is done by the differnent .sh files with a number perfix in/usr/loca/etc/rc.d directory. The DNS is running first with the "1.named.sh" and DHCP is second as "2.dhcpd.sh". When a new test node is added to Testbed,  the web script will change the dhcpd.conf file to adding the new IP addr as a fixed IP for the new node. So, "chmod +w  dhcpd.conf ", So it could be aceessed by the administor of Orbit-lab ( "orbitadm").


Terminal Server
IP address has to be set through the "webpage" method if it is already configured an IP address. new IP address is set as 10.0.0.253.And the username is root, password is w*****. Connecting to serial ports, a "null modem" cable must be used. The ports are configured as "realport","115200 8N1". The command to use each port is "connect x". x is the port number. Use" kill x" to close connection. And "Ctrl-[ + Enter" to exit if the session is dead.

Configure and Install Ops:

Making directories: /usr/testbed, /usr/proj, /usr/users/ ,/usr/groups/, /share. Basically, /usr partition in Ops is very large as 75G, we are going to mapping all /proj/ and /users/ directory in this partition. Make symbolic links:
ln -s /usr/users /users
ln -s /usr/proj /proj
ln -s /usr/groups /groups

The  defs files
They incudes some description of set-up environment. You can put it anywhere. but include the whole path when you do "configure" .
So, I copy the defs-example from sourcecode ( the snapshot) to  /usr/users/zhibinwu/testbed/ directory and then modified it.
Thare are tow major parts in defs-example. One is about the hostnames, change them to ".......orbit-lab.org". WWW server is named as "www.example.orbit-lab.org". And the other is of file systems. Give the whole path of some file system we created above in this file. And rename it as defs-orbit.
The information in defs file are used to generate scripts and shell programmers with those parameters. such as the domain names.

Build and Install
 use the ~/tbobj directory to do my builds in. This is the "object directory". And all sourcecode   are placed in the /users/zhibinwu/testbed/ directory. So do configure as:
ops# cd ~/tbobj
ops# /users/zhibinwu/testbed/configure --with-TBDEFS=/users/zhibinwu/testbed/defs-orbit
install:
ops# cd install
ops# chmod 744 ops-iustall
ops# ./ops-install
A lot of problems encountered in the ops-install process. The ops-install script is just quite straight forward to do a series of tasks. Some tasks are not proceeded smoothly.

Comment files of port-install
When it proceeds to install ports. There is going to copy the testbed/install/ports/ * to /usr/ports/misc/ (under default port directory of FreeBSD). However, there is a incompatible issue. In the "snapshots" from Emulab,  the variable COMMENT is undeclared in the Makefile. And there is  a separate file including the content of comment as named "pkg-comment". This is  no longer valid now. So, I get  new "portmakefiles" from Emulab..(  put in /common directory of WINLAB). Update those Makefile and remove "pkg-comment" files.

And I need to get two additional source files from EMULAB of "elvind4.0.3.tar.gz" and "libelvin4.0.3.tar.gz"
Those should be stored locally in /usr/ports/distfiles/

Then, the ops-install would be succeed like this:
Creating admin group                              [ Skipped   ]
Setting directory permissions
| /usr/testbed                                    [ Skipped   ]
| /users                                          [ Skipped   ]
| /proj                                           [ Skipped   ]
| /groups                                         [ Skipped   ]
| /share                                          [ Skipped   ]
+-----------------------------------------------> [ Skipped   ]
Installing ports
| Copying ports into place                        [ Skipped   ]
| Installing ports (may take a while)             [ Succeeded ]
+-----------------------------------------------> [ Succeeded ]
Adding testbed content to rc.conf                 [ Succeeded ]
Configuring sendmail
| Setting up /etc/mail/local-host-names           [ Succeeded ]
| Setting up mailing lists
| | Creating /etc/mail/lists                      [ Succeeded ]
| | Creating mailing list files
| | | testbed-ops                                 [ Succeeded ]
| | | testbed-logs                                [ Succeeded ]
| | | testbed-www                                 [ Succeeded ]
| | | testbed-approval                            [ Succeeded ]
| | | testbed-audit                               [ Succeeded ]
| | | testbed-stated                              [ Succeeded ]
| | | testbed-testsuite                           [ Succeeded ]
| | +-------------------------------------------> [ Succeeded ]
| | Adding lists to /etc/mail/aliases             [ Succeeded ]
| | Running newaliases                            [ Succeeded ]
| +---------------------------------------------> [ Succeeded ]
+-----------------------------------------------> [ Succeeded ]
Setting up exports
| Creating /etc/exports.head                      [ Succeeded ]
| HUPing mountd                                   [ Skipped   ]
+-----------------------------------------------> [ Succeeded ]
Setting up syslog
| Editing /etc/syslog.conf                        [ Succeeded ]
| Creating /var/log/tiplogs                       [ Succeeded ]
| Creating log files
| | /var/log/logins                               [ Succeeded ]
| | /var/log/tiplogs/capture.log                  [ Succeeded ]
| | /var/log/mountd.log                           [ Succeeded ]
| +---------------------------------------------> [ Succeeded ]
| Setting up /etc/newsyslog.conf                  [ Succeeded ]
+-----------------------------------------------> [ Succeeded ]
Adding cron jobs
| Editing /etc/crontab                            [ Succeeded ]
| HUPing cron                                     [ Succeeded ]
+-----------------------------------------------> [ Succeeded ]
Editing /usr/local/etc/sudoers                    [ Succeeded ]
Allowing root ssh                                 [ Succeeded ]
Setting up capture
| Creating /etc/rc.local                          [ Succeeded ]
| Creating /usr/testbed/etc                       [ Succeeded ]
| Creating empty /usr/testbed/etc/rc.capture      [ Succeeded ]
+-----------------------------------------------> [ Succeeded ]
----------------------------------------------------------------------
Installation completed succesfully!
Please reboot this machine before proceeding with boss setup
Local mailing lists have been created, with no members, in
/etc/mail/lists . Please add members to the following lists:
testbed-ops@orbit-lab.org
testbed-logs@orbit-lab.org
testbed-www@orbit-lab.org
testbed-approval@orbit-lab.org
testbed-audit@orbit-lab.org
testbed-stated@orbit-lab.org
testbed-testsuite@orbit-lab.org

Install sourcecode ( EMulab software) on Ops:
The GNUmakefile in the object directory is used to make Emulab software, for example. it compiles the sourcecode into some directories as:
ops-install:
        @$(MAKE) -C tbsetup control-install
        @$(MAKE) -C security control-install
        @$(MAKE) -C tip control-install
#       @$(MAKE) -C os control-install
        @$(MAKE) -C db control-install
        @$(MAKE) -C tbsetup control-install
        @$(MAKE) -C utils control-install
        @$(MAKE) -C event control-install

Correspondingly, the tasks of "control-install" is defined in GNUmakefile in each directory

Boss Installation: Step by Step
                   It has been proved that ports install is a rocky process with FreeBSD 4.9
Database Filling:

Install Nodes

                 
Power Controller Install:

   Running Experiment

  1. First simple 2-node ns script:
    set ns [new Simulator]
    source tb_compat.tcl

    set node0 [$ns node]
    set node1 [$ns node]

    set link0 [$ns duplex-link $node0 $node1 100Mb 0ms DropTail]

    $ns rtproto Static

    set udp0 [new Agent/UDP]
    $ns attach-agent $node0 $udp0
    set cbr0 [new Application/Traffic/CBR]
    $cbr0 attach-agent $udp0

    set null0 [new Agent/Null]
    $ns attach-agent $node1 $null0

    $ns connect $udp0 $null0
    $ns at 1.0 "$cbr0 start"

    $ns at 5.0 "$cbr0 stop"

    $ns run
    2 pcs will be assigned to this experiment. After tb-swap_in succeeded. The new ip address assigned to the experiment interfaces of test nodes are 10.1.1.2 (pc1) and 10.1.1.3/ However, as default netmask is 255.255.255.0. Those nodes could not talk to testswitch (10.1.0.254) directly, but through the eth0 and control switch. To fix this problem, The experiment initially failed before we fixed the testswitch. Then it seems OK. But the real problem is that experimetns cannot be run as normal as ns scirpt. We need monotor the traffic with tcpdump ourselves.
    It's better to have an NS-script to run just define the topology and re-loading all the nodes. Then experimenters can run an End-to-End applicaiton and tcpdump.
    To check the experiment status go to /usr/testbed/expwork/ directory.

    Another simple loop ns-script example to do this is :
    # This is a simple ns script that demonstrates loops.
    	set ns [new Simulator]                  
    	source tb_compat.tcl

    	set maxnodes 2
    	set lanstr ""

    	for {set i 1} {$i <= $maxnodes} {incr i} {
    	    set node($i) [$ns node]
    	    set name node-${i}
    	    append lanstr "$name "
    	    tb-set-node-os $node($i) FBSD-STD
    	}

    	# Put all the nodes in a lan
    	set big-lan [$ns make-lan "$lanstr" 100Mb 0ms]

    	$ns run
   Test with delay Node

         snmpset testswitch private vmVlan.9 integer 42

 


Test-node Diagram:



Acronyms

APOD
Authenticated ICMP Ping of Death
IPOD
ICMP Ping of Death
GRUB
Grand Unified Bootloader
PXE 
Preboot eXecution Environment. is, in one persons words, Intel's attempt at making PCs work more like SUNs ( ref). PXE outlines a protocol for enabling the BIOS to retrieve the operating system over standard network protocols.
TFTP
Trivial File Transfer Protocol, a simple form of the File Transfer Protocol (FTP). TFTP uses the User Datagram Protocol (UDP) and provides no security features. It is often used by servers to boot diskless workstations, X-terminals, and routers.
 

   

  Last modified Apr.6 2004, Zhibin Wu.