VMware Tools & Commands

Taupo Guide to VMware

In the time we have been working with VMware products we have learned many commands, hints and tips along the way. This page started as an in-house command line reference document for ESX server. However it has grown into a knowledge base for all aspects of deploying on the VMware vPlatform. This is very much a living document and updates, suggestions and corrections are all very welcome. Please email me on vmware@taupoconsulting.co.uk. If you find this document useful, feel free to link to it. I now maintain the most up to date version of this ESX server guide on a dedicated website for virtualisation technologies called B2V. You can find the latest version of the guide at www.b2v.co.uk/b2vguide2vmware.htm.		Why the name Taupo? Microsoft Infrastructure Virtualisation B2V Links Contact Home t:+44 131 466 3466
Boot Environment
/etc/lilo.conf	LILO is the boot loader (LILO=Linux Loader) used for ESX server (remember the service console is based on a modified version of Red Hat Linux 7.2). Lilo.conf is the text file to describe the OS to boot. This text file is compiled into a binary file that is actually used by LILO. If you are troubleshooting the APPEND line, then use vmkpcidivy tool. You should not have to revert to manually editing this file. If you ever do edit this file, then you need to compile it with lilo –c to write the binary file. Append= The pci device mask specified in the append line of lilo.conf is actually an include, not a mask out. The important thing to remember is the append line defines the hardware visible to the service console. Don’t replace LILO with another boot loader, e.g. GRUB. ESX manages allocation of PCI devices between service console and VM kernel with the expectation of LILO. You can also view PCI device allocation using the MUI, found under Startup Options Alternatively, you can use the legacy web interface using the URL http://esxserver/pcidivy
Boot Order	LILO loads the service console kernel, e.g. /boot/initrd-2.4.9-vmnix2.img This kernel obeys what is stored in the file /etc/inittab
/etc/inittab	This file is read by the init process and specifies the run level to be used by the service console. The line that states the run level will look something like this. id:3:initdefault: The init process then works through the start up scripts in the appropriate diectory. For run level 3, this directory would be /etc/rc.d/rc3.d The file also starts up the virtual terminals on the ESX server mingetty tty2 through mingetty tty5. The mingetty process is minimum getty.
/etc/rc.d/rc.local	A text file used most commonly when we need to share the service console NIC with the VM kernel. This may be required in say a blade server which has only 2 NICs, and we need to dedicate 1 NIC to Vmotion. insmod vmxnet_console devName=vmnic0 ifup eth0
/etc/rc.d/rc2.d	Contains start-up scripts for run level 2
/etc/rc.d/rc3.d	Contains start-up scripts (logical links) for run level 3. Run level 3 is used most of the time as it is command line full multi-user mode. The ones we are interested in are shown below: S00vmkstart S12syslog S55sshd S56xinetd S90vmware S91httpd.vmware
chkconfig --list	This service console tool displays a table showing which daemons are enabled for the run levels for the vmnix operating system. ntpd 0:off 1:off 2:on 3:on 4:on 5:off 6:off syslog 0:off 1:off 2:on 3:on 4:on 5:off 6:off snmpd 0:off 1:off 2:off 3:off 4:off 5:off 6:off If we wanted to change a service so that it is enabled for a particular run level, then we can use chkconfig –level. chkconfig --level 1 ntpd on The above command would turn on ntpd for run level 1, this would not affect the run levels that ntpd was already set for. So in this example, the ntpd run levels would be ntpd 0:off 1:on 2:on 3:on 4:on 5:off 6:off
service --status-all	This lists all the service daemons and their status. We can find running services by looking for the running status service –-status-all \|grep running would produce an output similar to the following: crond (pid 1423) is running httpd (pid 1486 1482 1479) is running syslogd (pid 1136) is running sshd (pid 1208) is running To avoid unnecessarily rebooting an ESX server after making certain configuration changes, we can frequently just restart the appropriate daemon. For example we could restart the Apache web server for the MUI with the command: service httpd.vmware restart and we can also check a named service running status with service httpd.vmware status
S12syslogd	The centralised logging system. When ESX is running, both the service console and VMkernel log messages through it.
/etc/ssh	Secure shell settings Quick way to restart the ssh daemon. Use full path to do this. /etc/init.d/sshd restart /etc/ssh/ssh_config Text file with the configuration of SSH client. /etc/ssh/sshd_config Text file with the configuration file for SSH daemon. An important setting in this file is PermitRootLogin=Yes/No. You can quickly check this with a grep on the file. grep permit /etc/ssh/sshd_config ~/.ssh/known_hosts A text file that stores the RSA keys for known hosts. If we rebuild a host and then reconnect to it over SSH we may be prevented if the known_hosts file has cached the old key. I think SCP (secure copy) uses RSA keys as well as SSH sessions. Note this file is maintained on a per-user basis.
/etc/xinetd.conf	The config file for Xinetd, the network services daemon? The daemon itself is stored in /usr/sbin/xinetd This launches the daemons that are bound to it on demand
vmware-authd	This is the authentication daemon. This daemon authenticates users of the management interface (MUI) and remote consoles using the username/password database defined in /etc/passwd. This service binds via the xinetd daemon and so the configuration file that specifies the listening port is in /etc/xinetd.d/vmware-authd This text file contains the settings for the VMware remote access authentication daemon. This file specifies the TCP:902 port used by remote console. If this port was changed here, it must also be changed in the file /etc/vmware/config. If we wanted to add Kerberos off-box authentication for MUI access, then its in the pluggable authentication module configuration file that corresponds to this daemon that we would make a change. This file is found at /etc/pam.d/vmware-authd We would need to change the current "auth required" to "auth sufficient" and add a last line of "auth required" using the Kerberos authentication module.
S90vmware	This script performs the following actions: vmnix driver Loaded by modprobe vmnixmod.o VMkernel Loaded by vmkloader Logger VMkdump Any dump will be copied to /root Starts VMs Performed by vmware-serverd
vmware-serverd	This daemon runs on demand in the service console to provide information to any VMware service that needs it. This performs actions in the service console on behalf of the VMware Remote Console and the web based MUI. It is started at boot time to do any VM autostarts. This process is replaced with vmware-ccagent if Virtual Center is installed.
S91httpd.vmware	This script starts the Apache web server which provides the ESX Server MUI. Configuration is stored in /usr/lib/vmware-mui/apache/conf/httpd.conf This process communicates with vmware-serverd for backend data. Remember a refresh in the browser is only a refresh to Apache, to get new data, click on the refresh button to get new kernel data. Remember if the httpd.vmware service starts and then stops immediately, check your service console disk space.
vmware-snmpd
vmware-snmptrap
/usr/sbin/vmkstatus	This is a perl script which runs every x seconds on tty1.
lsmod	Lists drivers loaded for the service console linux. Remember this command differs from the vmkload_mod utility which shows the modules loaded for the VMkernel. Module Size Used by Tainted: PF vmnixmod 177056 121 e1000 68456 0 (unused) usb-uhci 21220 0 (unused) usbcore 50112 1 [usb-uhci] megaraid2 32928 6 The same information can be found by cat /proc/modules
Starting & Stopping ESX Server
init 0	Instructing a halt
init 1	Instructing run level 1
esx 1	If this is entered at the LILO boot prompt we can get a root shell. We are instructing the vmnix kernel to execute at run level 1 (single user mode).
esx 3	Again at the LILO boot prompt, this time, we are instructing the vmnix kernel to execute at run level 3 (the default). If we suspected the run level was wrong we can use this to get back up and running without having to revert to booting Linux on its own.
linux rescue	Used when you boot the ESX server with a Red Hat Linux CD
chroot	Change root directory to new directory specified as a parameter
shutdown	Brings down the system in a secure way. shutdown -h now Halt after shutdown shutdown -r now Restart after shutdown
linux -s	At the LILO boot loader, the default options are esx Normal ESX boot linux Linux SMP kernel, no VMkernel load linux-up Linux Uni-processor kernel, no VMkernel load If we use the cursor key at the LILO screen to select one of the three default choices, the boot prompt (displayed below the menu) changes to reflect this. This allows us to augment the boot command with an option switch. boot: linux –s In this case, the –s instructs Linux to boot in single user mode. A critical security point here is that in single user mode, Linux automatically logs on as root! Once in single user mode if we wish to continue into multi-user mode then we type either exit or CTRL-D
RPM Packages
rpm	As ESX service console is based on modified Red Hat Linux, we can use the RPM package installation method. rpm -qa
Networking
/etc/modules.conf	Identifies interface eth0 and aliases alias eth0 e1000 alias eth1 e1000 options e1000 Speed=100
Ifup eth0	Brings interface “eth0” up
Ifdown eth0	Takes interface “eth0” down.
Ifconfig	Prints a list of the network interfaces. A quick way of getting IP and MAC of interfaces Can also be used to bring interfaces up or down ifconfig eth0 up ifconfig lo down
mii-tool	Interface tool. Doesn’t work correctly with some network cards, including Intel 1000 Pro copper NICs.
service network restart	Great way to do orderly restart of vmnix network configuration. Another way would be to do ifdown eth0; ifup eth0
/etc/hosts	Host lookup file, just like windows. The ESX server MUST have an entry for itself in its own HOSTS file.
/etc/resolv.conf	The DNS name server settings, This file does not need to be present if you are not using DNS.
/etc/sysconfig/network	Text file containing hostname and default gateway.
/etc/sysconfig/network-scripts/ifcfg-eth0	IP, mask and device name. The text file contains IP configuration for interface eth0, which is reserved for the service console.
route	Modify or print routing table route Prints routing table route del –net default Deletes the default gateway route add –net default gw w.x.y.z Adds a new default gateway
netconfig	Network configuration setup program
/etc/issue	File which shows ESX and vmnix version VMware ESX Server 2.1.2 Kernel 2.4.9-vmnix2 on an i686
uname -a	Outputs details of the server as shown Linux esx1.taupoconsulting.net 2.4.9-vmnix2 #1 Fri Aug 6 04:38:44 PDT 2004 i686
/etc/fstab	Local and remote file systems which are mounted at boot. This file is explained in more detail in the mount section.
/proc/net/NICfamily/eth0.info	A text file that can be checked to see what the service console NIC is doing. The speed of the console NIC specified in the file modules.conf can be confirmed by this file.
User Administration
id	Displays the UID for the currently logged on user or if supplied with a parameter can be used to display the UID of a named user.
passwd	Used to change the logged on users password or for a named account passwd <user> Adds a user to /etc/passwd By default a user added with default options has a group created of the same name. We can add the user to another group, e.g. have a group membership of say ESXUsers useradd alistair –G vmsqladmins The above command adds a user called alistair whose primary group (-g) is alistair and other group (-G) membership is vmsqladmins We can add additional parameters to more fully specify the account. useradd financeflagshipuser –g Finance –s /bin/false In the above example the users’ primary group is Finance and the shell is specified. In this case the shell is /bin/false which is a bogus shell which would prevent logon as this user. By default in the service console vmnix, this would be /bin/bash for the Bourne-again shell. It appears the only other shell shipped with vmnix is csh (the C shell).
groupadd	Adds a group to /etc/group groupadd esxadmins In the above example, a new group called esxadmins is created and therefore a new line appears in /etc/group.
gpasswd	The best tool for modifying the /etc/group file. The following command adds greg to the ESXUsers secondary group. gpasswd –a greg ESXUsers Group removal is simple with the –d switch: gpasswd –d tony ESXUsers
useradd	This command is used to modify a user. Be very careful with this command as when used with –G to set the users group membership, it is not adding the user to a group but is actually setting the list of secondary groups a user belongs to. Therefore in the following example if bill had secondary group list of ESXUsers and SQLadmin, then after entering: usermod –G ali bill then bill would only have a secondary group of ali and nothing else!
groupmod	Modify a group
su	Switch user, super user and used without parameters expects root. If we restrict root from logging in over SSH, then we force remote users to authenticate as themselves and then SU to run privileged commands if need be, thus leaving a decent audit trail.
sudo	Allows delegation of administration in terms of certain commands that normally only a particular user can execute (usually root).
visudo	This is just the "vi" text editor, but it automatically opens the /etc/sudoers file. The point of visudo is to ensure we always edit the right file as the location of the sudoers file differs between nix distributions, but this command is constant and will utilise the right sudoers file for the distro being used.
/etc/sudoers	The text file that contains the sudo users and the rules that apply to them.
/etc/group	List of security groups. kirsten:x:505:kirsten ESXUsers:x:507:kirsten,flagship flagship:x:508:flagship vpxuser:x:511: adminaccount:x:512: JohnSmith:x:513: This may look like a list of users, but it is groups. As the service console (vmnix) is a modified version of Red Hat Linux, the Linux security configuration is the same as Red Hat. One feature of Red Hat no found in all Linux distributions is that of the user private group (UPG). Whenever you create a user, a group of the same name is created also and the user is made a member. The format of the file is: groupname:x:user1,user2 so when we see groups like JohnSmith:x:513 we can assume the 513 is the UID for the user JohnSmith and this is his UPG.
/etc/passwd	List of security users. When we add a user with either the MUI or a command line tool such as useradd, we are adding to this text file. ali:x:500:500:Alistair Sutherland:/home/ali:/bin/bash sara:x:501:501:Sara Daniels:/home/sara:/bin/bash bill:x:502:502::/home/bill:/bin/bash andy:x:504:504::/home/andy:/bin/bash the format of the file is username:x:userID:groupID:fullname:homedirectory:shell Normally the group ID will match the user ID.
/etc/shadow	Password file
/etc/skel	Skeleton directory – new home directories take copies of the files stored here.
/var/log	This directory stores key log files for both the service console and the VMkernel. Of note are the vmkernel, vmkwarning & messages file logs. These logs can be viewed with the more, cat, head and tail command line tools. We can also access these logs via the MUI via the following link in the Options tab. If you use the sudo tool to run a command under a different security context then the log file /var/log/secure will contain the audit trail for such activity. Check the file /etc/syslog.conf for logging settings.
netstat	Shows currently active network connections
lsof	List open files Pipe the results into grep to check for open ports lsof \|grep IPv4.\*LISTEN
pam	Pluggable Authentication Module
free -m	Shows free memory in the service console. The "-m" switch specifies to display the results in megabytes.
fdisk	Disk partitioning tool
du	Disk usage. Great for finding out which folders are using disk space du –h /home/ali/vmare du –h ~ du –s summary
df -h	Print disk partitions (with human readable switch)
vdf -h	Print disk partitions with knowledge of VMFS partitions (type FB) (with human readable switch)
dd	Disk dump utility common to Linux. Copy a file while converting and formatting. This can be a quick and dirty way of making an ISO. This could be done in the service console with dd if=/dev/cdrom of=/vmimages/new.iso bs=32k This tool could also be used to go from ASCII to EBCDIC etc. This tool can be used to create an additional swap file. For example, if we did not allocate a big enough swap partition for the service console during ESX installation, we can create one now in a file of 64MB. dd if=/dev/zero of=/swapfile bs=1M count=64 If we did add a swap file, we would need to make sure it is started when ESX starts. Therefore, an entry in /etc/fstab would be needed as this file describes the local and remote file systems to mount at boot. The total amount of service console swap space is the sum of the swap partition and any swap files that are active.
mkswap	A command that must be run against a newly created swap file in order to activate it. Think of creating a swap file with dd like creating a partition, then mkswap is like formatting that partition. The swapon command then turns it on when needed.
swapon	Enables swap file for service console
swapoff	Disables swap file for service console
/proc/swaps	A text file that can be checked to see what swap the service console is using. The output contains a priority which shows which swap device will be used first before the other(s). Useful to determine if swap space is getting used and if there is more than 1 swap. Remember this is vmnix (service console) swap, not VMkernel. The VMkernel swap is in one or more files on a VMFS volume (hence the strong recommendation that even when using a SAN, a vmfs volume is created on direct attached storage to allow local swap). Filename Type Size Used Priority /dev/sda3 partition 554232 0 -1 /swapfile file 65528 0 -2
Basic File Commands
touch <filename>	When used with a non-existent filename, this tool creates an empty file of name filename. However, this can be used to touch an existing file and update its last modified or last accessed attributes. This could be scripted if required. Watch for running touch against any file stored on a VMFS as there appears to be a problem. Remember that not all Linux tools are modified for VMFS awareness. The VMFS is not an ext3 partition but is a mount point, it’s the VMkernel that accesses any VMFS volumes.
cat > file	Enter text at command line and then CTRL-D to write to file.
echo blah > file	Writes the text following echo command to file. This could be good for quickly creating files echo modprobe usb-uhci > S92usb echo modprobe usb-ohci >>S92usb Another great use of this technique is to make changes to the ESX server configuration via the /proc hierarchy, e.g. changing the number of shares for a VM echo 2500 > /proc/vmware/vm/nnn/cpu/shares would change the VM CPU shares to 2500. However such a change would only exist for the duration of the world created for that VM. After the VM is powered off this in memory structure is lost. To make such a change persistent, we would need to add the line sched.cpu.shares = "2476" to the VMX file of the virtual machine.
head filename	By default, the head command prints the first 10 lines of the specified file. We can choose how many lines we want instead of 10 by specifying the –n switch. This is good for looking at the file /proc/vmware/vmhba:x:x:x/0:0 with the –n 22 switch. Also good for using with the file command to determine whether a virtual disk is in ESX format or COW format. head server.dsk \| file - The “–“ is crucial to making the above command work. For an ESX virtual disk we would expect to see something like standard input: x86 boot sector
tail filename	Prints the last 10 lines of the specified file. Just like the head command, there is a –n switch that can be specified to list the last n lines of the named file.
sort	Can alphabetically or numerically sort redirected command output or files. sort fruit or to sort a basic score sheet sort –g –k 2 scores.txt
grep	Group regular expression, used to string search the files or command outputs. You can use grep –i to indicate search with case insensitive.
find	The find utility is used much in the same way as many Windows people used the DIR command. If you know roughly what files you are looking for, then this is the tool. The ls tool simply lists, whereas the find tool will find according to one or more criteria, a common one being find files modified in the last day using the –mtime switch as shown in the table. –mount used to ensure it doesn't traverse to remote file systems -size obvious -mtime modified in the last n*24 -mmin modified in the last n minutes -ls use output format as if ls were used -name name the file you are looking for (you just don’t know where it is!) find –mmin -30 files modified in last 30 minutes find –mtime 1 files modified in last 24 hours find –size +10000 files in excess of 10,000 bytes find –mount –size +10000 -ls files on non-remote file system find –name “hosts” -ls file called hosts
vi	Standard UNIX text editor I ESC :wq :q!
nano	Another text editor, more friendly but you should use –w to avoid word wrap.
setup	Allows changing of NIC, region, firewall, mouse, keyboard
ls -a	List files in a directory including hidden (also known as dot files due to their prefix) files.
ls -dl */	List directories in long format (does not display files). Could add as a shell alias, say lsd.
ll	Same as ls –al
less	Scrollable command line, great for piping large output into
more	Exactly the same as DOS and Windows, also great for piping large output into.
chown	Changes file ownership. If only 1 user name is specified then the user ownership is set only and the group ownership if left unchanged as shown in the solaris.vmx example below. However rather than having to use chown and then chgrp straight after it, you can set user and group ownership in one operation by specifying the username(s) separated by a colon as in the netware5.vmx example shown. #chown ali solaris.vmx #chown ali:ali netware5.vmx
chgrp	Changes the group owner for a file, leaving the user owner unchanged. #chgrp ali file.txt
chmod	Changes file permission. We can use either letters or numeric equivalency when setting permissions. We set permission for 3 principals, the user, the group and others (ugo). If you are from a Windows background then don’t confuse “o” with owner. #chmod u+rx,g+r,o+r file.txt Note that using + or – indicates we are adding to or removing from existing permission. If we wish to reset the permission we use “=” to explicitly set the object permissions. #chmod u=rx,g=r,o=r file.txt yields r-xr—-r-- Sometimes you will see a chmod using “a” to specify all (user, group & other), so we could quickly set read permissions by #chmod a-wx,a+r file.txt yields r—-r-—r-- A more common way to set permission is using chmod is using numeric equivalent values (4,2,1 for r,w,x) and permutations thereof. chmod 777 windows2k.vmx set perms to rwxrwxrwx chmod 754 windows2k.vmx set perms to rwxr-xr-- (default) Watch for chmod commands with 4 digits, e.g. chmod 0754. This refers to additional attributes Sticky bit SUID (Set User ID) SGID (Set Group ID) The sticky bit can be set on executables which tells Linux to keep the application in memory. The reason for this is to improve load times for other users who wish to run the same executable. This relates to the multi-user nature of UNIX/Linux. Given the speed of memory and disk access nowadays the need to keep applications in memory is much less important and so the sticky bit needed so much. chmod Set User ID is used on an executable so that when it is run, it is run under the security context of the file owner not the current user. That means in case I have an application whose owner is ' root ' and it has its SUID bit set, then when I run this application as a normal user, that application would still run as root. Since the SUID bit tells Linux that the the User ID root is set for this application and whenever this application executes it must execute as if root was executing it (since root owns this file). Set Group ID. Just like SUID, setting the SGID bit for a file sets your group ID to the file's group while the file is executing To set any of these 3 attributes, we use a 4th digit preceding the usual 3 used with chmod. That digit is set using the following: 4 Set user ID (s) 2 Set group ID 1 Set sticky bit (t) So if we want to set a file with permission rwxr-xr-x and set the user ID bit we could use chmod 4755
umask	Another permissions feature you may encounter is that of umask.
VMware Tools
vmkpcidivy	A VMware tool. Used to divide up the PCI resources between the service console and VMs or if they are shared. E.g. 1 NIC to SC, 1 SCSI adapter to share between SC and VMs, 1 SCSI adapter to VMs. This tool is stored in /usr/sbin/vmkpcidivy This is a command line tool which asks a series of questions. Use with the –i switch for interactive. This is also used to refreshnames and –q vmhba_devs For example, if I had a SAN LUN of vmhba1:0:25 and lets say I removed the VMFS from this LUN and wished to use it from the service console, I’d refresh names and then query the vmhba_devs to find out what device name the service console was going to use for this LUN, e.g. /dev/sda. A very useful feature of this tool is the ability to create a new profile. This adds a new boot option to the LILO boot menu that will have its own allocation of memory and PCI devices. If you are unsure about the changes you are making, then create a new profile e.g. esx (modified)
vmkfstools	The vmkfstools utility is the tool for managing virtual disks. Remember that to copy into a VMFS could have an adverse affect on other VMs with virtual disks on the same LUN. We always want to avoid using copy to populate a VMFS. Copy operations will update the volume in 16k blocks causing unnecessary SCSI reservations to update vmfs metadata. The switches that can be used with the command are listed below: -i to import a virtual disk to VMFS -e to export a virtual disk from VMFS -s to re-scan for new LUNs -S to set vmfs metadata volume label -X 6000M ./file.dsk to extend an existing DSK to 6GB -c 4000M ./file.dsk to create a new empty virtual disk -C to create a new vmfs volume -l vmfsname to list virtual disks on specified vmfs -F to set the access mode e.g. public/shared -k to create a VMkernel swap file -w to activate a VMkernel swap file -y to deactivate a VMkernel swap file -T to convert a vmfs1 volume to vmfs2 Remember that the vmfs parameter always goes last on this command parameter set for vmkfstools. This can be confusing for the beginner as the source and target order is different for imports and exports. If we want to simply list the files on a vmfs volumes we use the -l switch. vmkfstools –l /vmfs/vmhba0:0:0:8 or if we wish to use the volume label; vmkfstools –l <vmfs-metadatalabel> would produce an output similar to the following Name: VMFS2-VOL1 (public) Capacity 129465874944 (123461 file blocks * 1048576) avail Permission Uid Gid Attr Bytes (Blocks) Last Modified Filename rw------- 0 0 swap 2146435072 ( 2047) Nov 18 18:25 Swap.vswp rw------- 500 500 disk 4194304000 ( 4000) Nov 16 14:12 VM1.dsk rw------- 500 500 disk 6291456000 ( 6000) Nov 23 22:19 VM2.dsk rw------- 500 500 disk 2621440000 ( 2500) Nov 17 23:09 VM3.dsk rw------- 500 500 disk 4194304000 ( 4000) Nov 24 18:11 VM4.dsk To create a new virtual disk we use the -c switch vmkfstools –c 2048M /vmfs/vmhba0:0:0:8:newdisk.dsk This command would create a new virtual disk (monolithic) on the specified vmfs volume. Remember it is always better to use the VMFS name as this will not change even if your hardware does. To import a virtual disk into the VMFS we use vmkfstools with the -i switch. This will take a virtual disk in COW format into monolithic format without causing excessive SCSI reservations on the LUN holding the target VMFS vmkfstools –i /vmimages/file.vmdk /vmfs/vmhba1:0:25:1/virtdisk.dsk As always with this command, the parameter specifying the VMFS location is always the last parameter.
vmware-cmd	A command line tool to perform VM operations. This tool is found in /usr/bin vmware-cmd –s register /home/ali/vmware/newVM/newVM.vmx vmware-cmd –s unregister /<path to VM> vmware-cmd –l vmware-cmd /home/user/vmware/server/server.vmx stop soft vmware-cmd /home/user/vmware/server/server.vmx getheartbeat vmware-cmd /home/user/vmware/server/server.vmx getstate vmware-cmd /home/user/vmware/server/server.vmx getid vmware-cmd /home/user/vmware/server/server.vmx getpid The stop type can be soft, try soft or hard.
vmkdump	This is used to manage the VM kernel core dump partition. We can change the partition used if required. This tool is also needed if the core dump partition had been removed because ESX expects it to be there when starting up, so we need to tell ESX that it has gone. vmkdump -q Query the VM kernel for which partition it will use vmkdump -p vmhba0:0:0:3 Set VM kernel dump partition to partition 3 vmkdump –p none:0:0:0 Set VM kernel dump partition to none Remember the vmkcore partition does not have a mount point in the service console and is not specified as ext3. We can use the fdisk -l command to view where the core dump partition is in relation to the disk layout.
vmkload_mod	Allows viewing with the –l switch, loaded and unloaded VMkernel modules. This command differs from lsmod which lists the modules loaded for the service console. This is a very good way of differentiating what modules the kernel is using versus the ones used by vmnix. #vmkload_mod -l Name R/O Addr Length R/W Addr Length ID Loaded vmklinux 0x4de000 0xf000 0x12516b0 0x53000 1 Yes nfshaper 0x4ed000 0x1000 0x12a81b0 0x1000 2 Yes e1000 0x4ee000 0xf000 0x12a91b8 0x6000 3 Yes megaraid2 0x4fd000 0x6000 0x12f6008 0x3000 4 Yes bond 0x503000 0x2000 0x138a158 0x2000 5 Yes # Note, the -l parameter can also be specified as --list
Devices
/etc/modules.conf	Not only lists eth0 (as discussed in network section) but in fact this text file sometimes describes the devices that are assigned to the service console by vmkpcidivy.
modinfo	This tool takes a service console driver and displays the options it supports. For example modinfo e1000 Would produce a list of flow control settings for the Intel gigabit NIC. modinfo cciss Would produce the file details and version of the HP Smart Array controller.
lspci	Great tool for listing pci devices. Could be used to demo what the VM is presenting to the guest OS. lspci –v verbose output lspci –t print tree of PCI devices (use –tv for verbose tree) You may wish to examine /proc/pci also in order to correctly identify PCI devices and their slot configurations. One point to note is that when you are faced with PCI slot numbers is that not all hardware vendors number their slots in a straight forward left to right configuration. Make sure you know your slot numbers and their layout!
lsusb	A tool to list USB devices. Gives out way more info that is actually required. Remember that USB devices cannot be presented to virtual machines in ESX Server. If you wish to use a USB device in ESX, then you will have to use a USB over IP device and install the appropriate driver software into your guest OS for this. The most common USB over IP device is companyX
kudzu	A Red Hat tool to detect and configure hardware, can be dangerous with VMware.
/etc/vmware/netmap.conf	This text file maps ESX virtual switch names to device names. It is a network map config file as opposed to the more generic devices map config file (devnames.conf). network0.name = "SecuredGigabit" network0.device = "vmnic2" network1.name = "VirtualSwitch1" network1.device = "vmnet_1" network2.name = "InternetSwitch" network2.device = "vmnic1" Remember that if a virtual switch has no physical adapters, then it is vmnet_x. If a virtual switch has only 1 physical adapter is is vmnic_x. If there are two or more physical adapters assigned to a virtual switch, then the device is bondx. A bond can be in one of three modes, out-mac (default), out-ip and standby out-mac A VM virtual NIC is assigned to a pNIC in the bond and it uses only that out-ip A VM TCP conversation is placed on an available pNIC standby A VM will only use one NIC until a failure, then the other is used
/etc/vmware/devnames.conf	This text file maps device names (example above) to modules and their PCI addresses. Note that the devnames.conf file contains SCSI devices and NIC devices. 002:14.0 megaraid vmhba0 005:04.1 nic vmnic0 011:07.0 nic vmnic1 012:08.0 nic vmnic2
/etc/vmware/vmware-devices.map	Appears to be like a hardware compatibility list. Watch out for creating your own device map, devices.local The /etc/vmware/vmware-devices.map file contains a list of devices supported by ESX Server. This release includes support for a local version of this file, /etc/vmware/vmware-devices.map.local. Modify the vmware-devices.map.local to select different device drivers. This file is not modified during an ESX Server upgrade, preserving your customizations. The vmware-devices.map.local is read when the VMkernel is loaded: Any changes to the vmware-devices.map.local file require a reboot, or at least an unload/reload of the VMkernel to take effect. Entries in the vmware-devices.map.local files are used in addition to the entries in the vmware-devices.map file. The vmware-devices.map.local file does not need to mirror the vmware-devices.map file. Any vmware-devices.map.local file entries that correspond to the vmware-devices.map file entries supercede the vmware-devices.map file entries.
/etc/vmware/config
/etc/vmware/hwconfig	Loads of information in this text file. Useful for finding which nic is in which team. If using alongside devnames.conf and netmap.conf use the following command grep –i vmnic /etc/vmware/hwconfig
/dev/fd0	How to address the floppy disk drive
/dev/sda	SCSI device
/proc/vmware/pci	Text file with discovered PCI devices Bus:Sl.F Vend:Dvid Subv:Subd Type Vendor ISA/irq/Vec P M Module Name Spawned bus 000:00.0 8086:3590 1028:016e Host/PCI Intel C 000:02.0 8086:3595 0000:0000 PCI/PCI Intel 001 C 000:03.0 8086:3596 0000:0000 PCI/PCI Intel 004 C 000:04.0 8086:3597 0000:0000 PCI/PCI Intel 007 C 000:05.0 8086:3598 0000:0000 PCI/PCI Intel 010 C 000:06.0 8086:3599 0000:0000 PCI/PCI Intel 013 C 000:29.0 8086:24d2 1028:016e USB Intel 11/ 16/0x69 A C 000:29.1 8086:24d4 1028:016e USB Intel 10/ 19/0x71 B C 000:29.2 8086:24d7 1028:016e USB Intel 7/ 18/0x79 C C 000:29.7 8086:24dd 1028:016e USB Intel 3/ 23/0x81 D C 000:30.0 8086:244e 0000:0000 PCI/PCI Intel 016 C 000:31.0 8086:24d0 0000:0000 PCI/ISA Intel C 000:31.1 8086:24db 1028:016e IDE Intel 0/ / A C 001:00.0 8086:0330 0000:0000 PCI/PCI Intel 002 C 001:00.2 8086:0332 0000:0000 PCI/PCI Intel 003 C 002:14.0 1028:0013 1028:016e RAID Dell 7/ 38/0x91 A S vmhba0 004:00.0 8086:0329 0000:0000 PCI/PCI Intel 005 C 004:00.2 8086:032a 0000:0000 PCI/PCI Intel 006 C 005:04.0 8086:1010 8086:1012 Ethernet Intel 7/ 58/0x99 A C 005:04.1 8086:1010 8086:1012 Ethernet Intel 10/ 59/0xa1 B V e1000 vmnic0 010:00.0 8086:0329 0000:0000 PCI/PCI Intel 011 C 010:00.2 8086:032a 0000:0000 PCI/PCI Intel 012 C 011:07.0 8086:1076 1028:016d Ethernet Intel 11/ 72/0xa9 A V e1000 vmnic1 012:08.0 8086:1076 1028:016d Ethernet Intel 11/ 73/0xb1 A V e1000 vmnic2 016:13.0 1002:5159 1028:016e Display ATI 7/ 18/0x79 A C
findnic	This tool is also known as “The VMkernel Network Card Locator”. It locates a physical NIC in an ESX server by using pings. It is left to the operator to check which NIC is sending the echo requests by either unplugging network cables or inspecting flickering lights on the NIC or the switch. The useful feature of this tool is we can ping based on the device name that the VMkernel uses to access the NIC. We should remember that each physical NIC in an ESX server that is allocated to the VMkernel does not itself have an IP address, therefore in to perform an ICMP echo request, we need to temporarily give that NIC an IP; this is specified as the first IP parameter in the command, the second IP parameter being the ICMP destination. findnic –i 5 vmnic2 10.0.0.1 192.168.1.3 The above command will send ICMP echo requests to 192.168.1.3 every 5 seconds. We could also use the –f switch which would flood ping.
File Systems
procfs	The volatile /proc directory hierarchy can be treated as a file system but is held in RAM.
/proc/vmware/sched/cpu	A text file snapshot of cpu scheduling. You could increase CPU shares using this method echo 10000 > /proc/vmware/vm/<number>/cpu/shares
watch	This is a fantastic utility that polls whatever command you supply it with an displays a running changing status. For example, we could use the command watch cat /proc/vmware/mem to obtain a dynamic view of memory usage by VMware. If you are viewing lots of output but can't see what is actually changing we use the -d parameter to specify differences, thus highlighting changes between refreshes. watch –d cat /proc/vmware/mem To specify a different refresh rate, we use the --interval=n parameter
smbclient	This is a redirector type tool to view and connect to SMB (Microsoft networking) hosts. Before updating the /etc/fstab file with remote file system information, check first using smbclient that the share is visible. The following was produced with smbclient –I 192.168.1150 –U username –L computername Sharename Type Comment --------- ---- ------- E$ Disk Default share Microsoft Disk IPC$ IPC Remote IPC D$ Disk Default share NETLOGON Disk Logon server share REMINST Disk Remote Installation Share ADMIN$ Disk Remote Admin SYSVOL Disk Logon server share VPLOGON Disk Symantec AntiVirus C$ Disk Default share VPHOME Disk Symantec AntiVirus To create a mount point to a Microsoft share is very straightforward. Remember, we are allowing the service console to access a remote file system. This is not related to what virtual machines are doing. Further, we need to be careful if we are attempting to do any file operations due to potential limits with 2GB file sizes. 1. Add entry to local hosts (/etc/hosts) for MS host 2. Create directory where you wish the mount point with mkdir 3. Run smbclient –I <ipaddress> -U user –L NetBIOSComputername to check you can see the shares 4. Add an entry to /etc/fstab //server/share /mountdir smbfs ip=ipaddress,username=user,password=pass,noauto 0 0 5. Mount the remote file system with mount /mountdir 6. Change directory into mount point and the directory will no longer be empty but will appear just like a mapped drive. Alternatively, if you just want to map temporarily to a Microsoft host and not have to modify fstab, then use smbclient interactively as shown: smbclient //ipaddress/share –U NThost\NTuser Password: ****** Domain=[TAUPO] OS=[Windows 5.0] Server=[Windows 2000 LAN Manager] smb: \>
NFS	Network File System (NFS) is provided by rpc.nfsd and would normally be launched by an nfs script in /etc/rc.d. To start using NFS to mount directories on other ESX or Linux servers, we can use the following steps: 1. Change the ESX Server which is to be the NFS server to use medium security (using the MUI is easiest for this) 2. Add an entry to the file /etc/exports on the NFS server by either using vi text editor (/vmimages *) or use the exportfs command 3. Check the NFS client is not excluded by the server file /etc/hosts.deny 4. Up to you if you explicitly allow the NFS client by adding the NFS client to /etc/hosts.allow on the NFS server 5. Start the NFS server daemons a. /etc/rc.d/init.d/service portmap start b. /etc/rc.d/init.d/service nfslock start c. /etc/rc.d/init.d/service nfs start 6. Start the NFS client daemons a. /etc/rc.d/init.d/service netfs start b. /etc/rc.d/init.d/service portmap start 7. Mount remote export directory from the client with the command a. mount –t nfs server:/export localdir/localmount
showmount	This command is used by a NFS client to see what directories are being exported by a NFS server. showmount –e nfsserver This command can be specified with the hostname name or IP address of the NFS server holding the exported directories.
exportfs	The exportfs command allows you to selectively export or unexport directories without restarting the various NFS services.
NIS	NIS stands for Network Information Service. Ensures numeric owner IDs are unique across the organisation. This is because numeric owner IDs are used in NFS, so we are often going to have mistaken identity as used 515 on the nfs client will not be the same as user 515 on the nfs server.
vmware-mount.pl	Just like the Windows utility to mount virtual disks when they are powered off so you can check what’s in them. This will mount ext3 and vfat as read/write but NTFS as read only. To find out what file systems are in the virtual disk, use the –p switch. vmware-mount.pl –p /vmfs/VMFS-VOL1/win2k3.dsk -------------------------------------------- VMware for Linux - Virtual Hard Disk Mounter Version: 1.0 build-9638\nCopyright 1998 VMware, Inc. All rights reserved. -- VMware Confidential -------------------------------------------- Nr Start Size Type Id Sytem -- ---------- ---------- ---- -- ------------------------ 1 63 8160957 BIOS 7 HPFS/NTFS If we actually want to mount a partition then we need to be specific and create a directory (or use an existing) to be our mount point. mkdir /myntfs vmware-mount.pl /vmfs/VMFS-VOL1/win2k3.dsk 1 –t ntfs –o ro /myntfs This command will tie up the console window hence you’ll need to spawn a new window first to navigate to /myntfs to view the contents.
mount	Mount file system This command can be used with the –t switch to specify file system type, e.g. nfs, smbfs or iso9660 mount -t nfs [-o options] esx4:/vmimages /root/other_server_vmimages
umount	Un-mount file system. Note it is u-mount and not unmount!
smbmount	A mount –t smbfs passes control across to this utility. We can use this utility directly if we prefer for mounting SMB host file systems.
/etc/fstab	If we have smb mount points permanently, then this file could end up with user credentials in it. This file is readable by everyone so this is not good. We can place the credentials for the smbmount in a hidden secured file in our home folder eg. /root/.smbcreds echo username=user > .smbcreds echo password=pass >> .smbcreds chmod 600 .smbcreds Then in the /etc/fstab file we substitute the username and password for credentials=/root/.smbcreds. Therefore the whole line in the fstab would be //server/share /mountdir smbfs ip=ipaddress,credentials=/root/.smcreds,noauto 0 0
Shell
.bashrc	A hidden file that extends shell script for the BASH shell. This is found in the users home folder.
/etc/bashrc	System-wide bash shell settings
clear	Equivalent of CLS command in MS-DOS and in Windows command prompt.
sum	Prints the 16-bit checksum and size of the specified file.
md5sum	Prints the MD5 checksum for the file supplied as a command parameter. VMware publish md5 checksums for all downloads to allow the end user to confirm that the file had not be corrupted in download.
cal	Display calendar for current month -3 last, current and next month 200x display year cal 2005 \| less to get scrollable calendar
/etc/vmware/vm-list	Auto generated text file listing the VMs on the server. This file should not be manually edited, should match up with output of vmware-cmd –l.
Process Management
PID	PID stands for Process ID. Every running process has a process ID that is valid for the length of process execution.
PPID	PPID stands for Parent Process ID, which is the PID of the process that launched that process.
ps	Show running processes in the service console. ps –A ps –eaf ps –eaf \|grep vmware-serverd ps –efw is useful as the “w” indicates wide format, so we can see the full directory path to the vmx file.
pstree -ph	Show process tree with PIDs and ancestors
esxtop	VMware version of process viewer top
renice	Change process priority. If there were many VMs running concurrently, it may be necessary to temporarily raise the priority of the MUI in order to complete administrative tasks #ps –eaf \|grep vmware #renice –p <pid of vmware-serverd> #renice –p <pid of httpd.vmware> To reset the PID of these processes back to their defaults, use renice again to set the priority to zero.
pidof	Finds the PID of a named process. #pidof vmware-authd
&	We can configure processes to run in the background by adding a “&” suffix when launching the program. #sleep 10 &
jobs	List the process jobs running in the background.
SAN & Disk Management
Disk.MaxLUN=8	This is the default setting, which scans 0 to 7 on start up. Remember if we what to scan up to LUN N we must set Disk.MaxLUN to N+1. This setting is found in the /etc/vmware/vmkconfig file but should be modified via the MUI (Options tab, Advanced Settings).
Disk.MaskLUNs	This parameter controls LUN visibility, again this is in the /etc/vmware/vmkconfig file. LUN masking is only supported on fibre channel HBAs. This overrides the Disk.MaxLUN setting. vmhba0:0:4,6-255 would scan 0,1,2,3,5 i.e. skip 4 and skip 6 through 255 vmhba0:0:3,4,9-255 would scan 0,1,2,5,6,7,8 i.e. skip 3 & 4 and skip 9 through 255
/etc/vmware/vmkconfig	This is a text file which stores vmkernel configuration, like an INI file or registry key. The important fact to note is that this file does not exist until you make an edit away from the default setting.
vmkfstools –s vmhba0	Used to re-scan for new LUNs on specified host bus adapter.
proc/vmware/scsi
WWN	SAN devices are identified by a world wide name, a unique 64-bit address. Remember we can use the perl script wwpn.pl to determine quickly what the WWN is for the installed FC hba.
LUN	A LUN identifies individual units of storage behind a SCSI ID. A LUN could be a single disk or a RAID5 volume.
Zoning	Zoning is either hard (switch port) or soft (WWN controlled)
LUN Masking	LUN Masking is a disk array feature that controls which LUNs are presented to which WWNs
/proc/scsi/driver/number	The WWN for an adapter would be found in this file.
Disk.SupportSparseLUN	The setting Disk.SupportSparseLUN should =1 as LUNs may be discontiguous. For example if there are disk volumes at LUNs 0,1,2 and 6 then we want to be sure that after LUN 2, the VMkernel storage driver does not stop scanning. We want the LUN scanning to reach the last LUN specified in DiskMaxLUN parameter, regardless of whether the visible LUN numbers are contiguous or not.
Disk.RetryUnitAttention=1	This setting tells ESX server to retry SCSI commands as vendor specific status codes may have been received and ESX may think the volume is present but not accessible when in fact its just a message that cache has been upgraded.
Performance
Free	A simple utility to display free memory in the service console.
esxtop	This is the VMware version of top and provides CPU, memory and disk configuration realtime information just like top does, but this time we only see the data relating to the VMkernel so we see worlds instead of processes as resource consumers.
top	Shows the running processes in the service console and lists the top consumers of CPU time.
iostat
vmstat	Don’t get confused by this command, it is a Linux command, not a VMware command. This is meant to view Linux processes, memory and paging. We generally use vmstat with 2 numeric parameters, the first parameter is how frequently the tool should run (specified in seconds). The second parameter specifies how many times the tool should run e.g. vmstat 3 10 would run the tool every 3 seconds for 10 times and then exit.
man cpu	Great info on using the /proc/vmware file system to change share allocation and processor affinity.
/proc/vmware	In a subdirectory called /vm there are subdirectories for each vm labelled by number How do you find out what number corresponds to what VM? grep –i worldid= ~ali/vmware/ISAserver/vmware.log Some other examples of great info that can be extracted are outlined in the following table: /proc/vmware/vm/139/cpu affinity 0,1,2,3 /proc/vmware/vm/139/cpu shares 1000 /proc/vmware/vm/139/cpu status /proc/vmware/vm/138/cpu hyperthreading /proc/vmware/sched/cpu
vmkusage	Generates web page usage reports. Was not automatically installed in previous versions of ESX vmkusagectl install to install the utilisation web pages & setup a cron job vmkusage –graph to generate graph images vmkusage –regroove to wipe the db and start stats again
hstatus	This is a web page like the MUI accessible via http://esxserver/hstatus Not sure if we need to be logged in for this to work, but vmkusage does appear to be required. We get loads of output on this page, similar to running command line tools. This is part of what looks like the legacy interface to ESX server, i.e. it doesn’t look as cool as the MUI of ESX 2.x.
crontab	Lists the cron jobs scheduled for the user when used with the –l (list) switch.
SNMP
snmpwalk	Used to walkthrough SNMP mibs. -M – use MIBSDIR -m all use mibs list instead of default mibs list
Trap Receiver	A utility to display SNMP traps. MIBs can be loaded into this.
Perl Scripts
vmsnap.pl	Backup script to backup a virtual disk whilst the virtual machine is still running. This is achieved by the script creating a REDO file and if necessary a REDO file of a REDO file; thus reducing the changing data and allowing a straight copy of the non-changing file.
vmres.pl	This script is the restore tool for vmsnap.pl
wwpn.pl	Used to determine the world wide name (WWN) for hba.
vmkstatus.pl	This Perl script runs on tty1 of the ESX server.
pbind.pl	Create or delete HBA persistent bindings
vmware-mount.pl	This is used to mount a virtual disk in the service console while the virtual machine is not running. For example, if a configuration file in the guest OS virtual disk is preventing a virtual machine from booting correctly, you could

Please email info@taupoconsulting.co.uk for more information.