Quelle: http://trivore.com/vmware/

Introduction

This page is designed to aid in designing, setting up, and operating the VMware ESX Server v2.5.x platform. Most of this information has been gathered both during working with the product on different production environments, and (together with students) during the several training sessions delivered on the product.

As this information has been gathered mostly for Trivore's VMware ESX Server class students, the representation might not always be the best possible. Also, some information is pretty purely GNU/Linux oriented. This is intentional, as many technical people are still pretty new to the GNU/Linux stuff running on the Service Console.

These hints and tips do not replace neither the excellent VMware ESX Server manual, nor a decent hands on GNU/Linux knowledge, nor a decent, hands-on-oriented VMware ESX Server classroom education. Any Unix knowledge makes your life a lot easier.

This document is a work in progress as long as v2.5.x is the latest version. Soon after that this document is frozen and a new one should be created. If there is something you might want to see here, or you would otherwise send me feedback, please email us. The maintainer of this information is Kari Mattsson of Trivore Corp.

Planning and documenting the system

This chapter focuses on the planning phase of system setup. It emphasises and reasons the importance of the documentation. More planning oriented material is also in the Storage chapter.

The files making up a VM (virtual machine)

Generally about 4 files make a VM: xxx.dsk, xxx.vmx, vmware*.log, and nvram. There could be more than one .dsk file associated to a VM (virtual machine). All files, but the .dsk file(s) are always in the same (sub)directory. It is most convenient to name that subdirectory xxx.

The older .log files are always deleteable at will. The latest .log file can be deleted when the VM is powered off. A new log file will be created during the next VM power-on. As the log files do not take much disk space, most administrators let them be. If for some reason the current log file vmware.log starts growing too rapidly, you can rename+compress it, or even delete it. It is recommended to execute logrotate daily to check for the logs. More on this later.

The nvram file contains CMOS/BIOS settings for the VM. It the file does not exist, it will be (re)created next time the VM powers up. If any changes were made to the nvram via VM's BIOS setup, do not forget to remake those changes. The most common change to the BIOS settings of a VM is to change the boot device order. The default is floppy, hard drive, CD/DVD. You might want to raise the CD/DVD to higher in the list to be able to boot from there.

In addition to the .dsk file, a .dsk.REDO file will exist in the same directory, if non-persistent, undoable, or append disk mode is active for that particular virtual SCSI disk. The .dsk.REDO file can grow up to several gigabytes. How large it will grow, depends only on what file/disk operations are done on the disk.

It is a good idea to place a brief descriptive xxx.txt file next to the xxx.vmx file to document what this VM is, what is its business purpose, what is the OS version and level, what are the main installed applications, and who is/are responsible for the VM. If you absolutely do not want the .txt file, you could place that same information to the comment lines of the .vmx file. Is the information safe there, is untested.

An alternative to the per VM .txt file approach is a single .txt file which contains information on all the VMs on the server. Okay, you can maintain this document anywhere with any tools, but remember, pure text files are most easily accessed and maintained on the ESX Server by the administrator(s).

Naming a VM, and its files is important. It will be crucial once there are tens (dozens for those of you still using legacy, non-metric systems :-) of VMs. It is a recommended practice to:

• Have a short, but descriptive name for the VM without any spaces.
• Have the VM's name to show up consistently in the configuration file directory name (or VMFS partition name).
• Have the VM's name to show up consistently in all the filenames related to the VM.
• Have the VM's name to show up consistently in the one line VM description in the .vmx file.
• Have a brief descriptive .txt file next to the .vmx file to document the VM.

A powered on VM can be suspended, much like a laptop or any of the later Windows'. VMware uses the term "Suspend to disk". "Hibernate to disk" is actually a more accurate term here. When entering the suspend (hibernate) state, a suspend file will be created. It has a filename extension .vmss. It is very practical to direct the suspend file to the same directory where the VM's operating system .dsk file is.

If you have two or more VMware ESX Servers, you might want to try suspending a VM on one ESX Server, and resuming it on another. Practical in many cases...

The .vmss file will be a few megabytes larger than the maximum RAM memory allocated to the VM.

Example list of files associated with virtual machine 'payroll'

This is a realistic example list of files.

/data/payroll/payroll.vmx

The configuration file defining all aspects of the VM.

/data/payroll/nvram

BIOS/CMOS settings of this VM. This file is (re)created if it is missing.

/data/payroll/vmware.log /data/payroll/vmware-0.log /data/payroll/vmware-1.log

Server created log file(s). Can be deleted to save disk space.

/data/payroll/payroll.txt

Small text file documenting this VM. It is strongly recommended to create this file.

/vmfs/public01/payroll-sys.dsk

Boot(/application) disk (first physical disk) of the VM. It could have more than one partition. Windows VMs usually have only one partition, but Linux/BSD/Netware/Solaris VMs have more than one.

/vmfs/public01/payroll-data.dsk

Data disk (second physical disk) of the VM. It could have more than one partition. Partitioning depends on the requirements of the VM.

Planning and documenting

Planning is important with ESX. It is especially important with the storage allocated to VMs. Some preliminary questions to answer are: What are the VMs to be installed on ESX Server? What operating systems will be installed? Are they file/print/database/application servers, or what? What kind of storage allocation will be needed? How much storage is initially reserved to a each VM? Are VMs executed on more than one ESX Server at the same time? What are the CPU requirements per VM? What are the network I/O requirements per VM? What are the disk I/O requirements per VM? What are the memory requirements per VM? How much storage is required for suspended VMs? Will non-persistent, undoable or append disk mode be used?

Documenting is another extremely important issue with ESX Server, as these systems easily become extremely complex. Mainframe and Unix style management attitude is really required. Document everything carefully. It will save your day often. The few line .txt file next to .vmx file is especially nice. An example skeleton .txt file follows:

Customer name....................:  [Customer name in a multi customer server only]
Name of the VM (virtual machine).:  [vm###, or hostname/computername]
Creation date for this file......:  [dd-Mmm-yyyy / creator's name and contacts]
Last update of this file.........:  [dd-Mmm-yyyy / updater's name and contacts]
Business purpose of this VM......:  [a short business-related description]
Responsible person(s) for this VM:  [name, contact (email and/or telephone)]
OS name and version in this VM...:  [OS name, update level]
Installed core application(s)....:  [app1, app2, app3]

Storage: partitioning, filesystems, directories, files, etc.

This chapter covers the issues on storage. Some of this material is very planning oriented.

Background information on partitions

The Service Console on VMware ESX Server is based on RedHat Linux 7.2. All the data in this section relates to that fact.

Maximum number of partitions (which can contain data) per SCSI disk (=logical drive on RAID systems) in Linux kernel 2.4.x based systems is 14. The Service Console is based on kernel version 2.4.9. Of these max 15 partitions 3 are primary (1-3), 1 is extended (4), and 11 are logical (5-15). Extended partitions never contain any actual data. It merely acts as a container for the logical partitions.

This 14 data partition (partitions 1-3, and 5-15) limit could force you to create smaller logical drives than you wanted.

It is a nice practice to use primary partitions for non-vmfs filesystems, and logical partitions for vmfs filesystems. Nothing forces you to do that, it is just a choice.

On previous generation IBM, and other brand servers, there is a 10-100 MB System/Service/Diagnostic Partition accessible with [Alt-F1], [F10], or [F11], and it is normally partition /dev/sda4. VMware ESX Server recognises this partition as "Old VMware Core Dump Partition". Usually you do not want to touch this partiton, so leave it alone. VMware will not touch/use it.

SCSI disk naming and a word about ISO images

SCSI disks are named as /dev/sda (first), /dev/sdb (second), /dev/sdc (third). The first disk is always the boot disk and (practically always) internal to the server. The other disks are usually on an external (usually fibre-connected) disk enclosure. Please note, that many IDE-RAID controllers show up as a SCSI controller, but are currently not supported by VMware ESX Server.

It is strongly recommended to use CD/DVD ISO images, and/or floppy images. Images are very handy for OS and application installations. They are much more reliable and convenient than the actual physical media. Usually some portion of the server internal disk space is reserved for these ISO images. Example commands on how to create these images are given later in this document.

VMFS (vmfs, virtual machine file system) and accesibility modes

Partitions (dubbed as volumes in VMware ESX Server) formatted as vmfs do not support subdirectories. It is one thing given up for the fast speed of vmfs.

VMware ESX Server v2.x uses version 2 of vmfs dubbed VMFS-2. These volumes (partitions) have 2 possible accessibility settings:

• public - This is the default. On a fibre channel or similar SAN, make this volume (partition) visible to more than one ESX Server concurrently. ESX Server locks individual .dsk files so that only one ESX Server can access it at a time. That way file system integrity is ensured.
• shared - Shared is the same as public with one difference: The .dsk files can be accessed by multiple ESX Servers at the same time. This accessibility setting is only recommended for clustered VMs which need to share virtual SCSI disks (.dsk files). The most common example are Netware and Windows fault-tolerant clusters.

VMFS-2 also allows spanning upto 32 extents. Use spanning only if you really have to. It is not as bad as RAID-0, but you still have a major risk of losing it all if one volume has severe problems/fails. VMFS-1 volumes can be converted to VMFS-2 with vmkfstools command.

When you are creating a .dsk file to a shared mode volume, the volume might go to read-only mode. You can enable writing to the volume by entering command "vmkfstools --config writable <vmhba-name>".

A note on .dsk files (which are seen as physical SCSI disks in the VM)

The virtual SCSI disks (.dsk files) have four different modes:

• persistent,
• non-persistent,
• undoable,
• append.

Last two modes are invaluable when upgrading or testing the software in a VM, as you can easily revert to the point you enabled that mode. Effectively you can undo any changes to the file system.

Strategy for placing all the files - VMware ESX Server Service Console files, and the VM files

It is wise to install VMware ESX Server to the local/internal disks (single RAID-1 SCSI logical drive preferably with a hot-spare). Dual 36 GB drives make plenty of space for mid-size systems. Dual 72 GB is actually today more common and scales up to high-end systems. You might want to go for the higher rotating speed drives to get faster ESX Swap.

Place all the VM-related files (guest OS's configs and disks) to external disks (logical drives) on fibre channel (or similar).

Standard: Partitioning the VMware ESX Server local/internal disk (/dev/sda)

This is where the VMware ESX Server and its Service Console will reside. Partitioning examples are for a system with mirrored 36 GB local/internal drives.

Please note, that the hard drive gigabytes are most often salesman gigabytes (10⁹), not true gigabytes (2³⁰). You get less than you thought of.

Server internal hard drive (/dev/sda) partitioning

DeviceName  PartType  Size     MntPoint    FStype  Remarks
/dev/sda1   primary   2500     /             ext3  Service Console root.
/dev/sda2   extended  32500+                       Logical partition container.
/dev/sda5   logical   1000     swap          swap  Service Console swap.
/dev/sda6   logical   ~14500   /local        ext3  See notes below.
/dev/sda7   logical   100      vmcore      vmcore  VMware Core Dump.
/dev/sda8   logical   ~16000   /vmfs/esxswap vmfs  See notes below.

• Note: Your partition numbers are propably different from the example above. You will only have these partition numbers if you do partitioning with fdisk.
• /vmfs/esxswap is of accesibility type "public", and formatted with volume label "esxswap". You do not define the name during install, but later by setting a volume laber for the volume. It is used only for VMware swap, and nothing else. The size of this volume (partition) equals to the size of VMware swap that is created.
• /local is used for CD/DVD ISO images, floppy images, etc. It is also used for backup purposes. You can also export .dsk files faster to this volume, than to an VMFS volume.
• Balance between /local and /vmfs/esxswap. Make /vmfs/esxswap just big enough for the VMware swap. Nothing else goes there. Allocate as big /local as possible.
• Use /local as a backup target of the Service Console with a tar command. More information on that later in this document.
• VMware swap file should be 1 - 2 times the amount of physical memory on the system. The default is 1 time the amount of memory. This size together with the physical RAM amount determines the possible amount of memory over-subscribing. If you think you be adding more memory to the system, you should prepare for that when determining the size of this partition.
• Service Console swap space should be 1 - 2 times the amount of memory reserved for the Service Console. That reserved memory amount depends on how many VMs are powered on and running at the same time. VMware ESX Server manual and the MUI tells how to calculate this memory amount. In the example above, the maximum required amount of Service Console memory, 512 MB is presumed. 2 times 512 MB approximates the 1000 MB swap above. This swap is defined early in the installation and cannot easily be changed. That is why it is a good idea to plan for the largest possible configuration. You don't save a lot of disk space by reducing this swap partition to 400 MB, the smallest recommended.
• The maximum amount of memory to reserve for Service Console is 800 MB. It is required rather rarely. If you install IBM Tivoli Storage Manager, Legato Networker, or similar program to Service Console, and you have 32+ VMs running at the same time, then it might be necessary to reserve 800 MB for Service Console. This affects to the Service Console swap size, too. Maybe you want to go up to 1600 MB for the Service Console swap instead of 1000 MB? In the real life it should not be necessary.

Tuned: Partitioning the VMware ESX Server local/internal disk (/dev/sda)

This is really for experienced GNU/Linux administrators only. Use this partitioning scheme instead of the above one on VMware ESX Servers with higher security requirements. For simplicity, the possible System/Service partition is not shown.

This fine-tuning makes it possible to make the Service Console even more secure. You do not want to apply this plan unless you are very familiar with Linux and the Service Console.

DeviceName  PartType  Size     MntPoint    FStype  Remarks
/dev/sda1   primary   100      /boot         ext3  Service Console kernel is here.
/dev/sda2   primary   500      /             ext3  Service Console root.
/dev/sda3   extended  34000+                       Logical partition container.
/dev/sda5   logical   500      /tmp          ext3  This is really not much used.
/dev/sda6   logical   1000     /var          ext3  All the local log files will be here.
/dev/sda7   logical   1500     /usr          ext3  All standard programs are here.
/dev/sda8   logical   50       /usr/local    ext3  All server local scripts/programs.
/dev/sda9   logical   350      /opt          ext3  Optional 3rd party programs.
/dev/sda10  logical   1000     swap          swap  Service Console swap. See note below.
/dev/sda11  logical   11000    /local        ext3  See notes below.
/dev/sda12  logical   100      vmcore      vmcore  VMware Core Dump.
/dev/sda13  logical   16000    /vmfs/esxswap vmfs  See notes below.

• Important! Please also see the notes above in the standard partitioning example.
• Actual Service Console partition order and numbers might differ a bit from the above example unless you do partitioning manually with fdisk.
• Some further site and/or installation dependent fine-tuning on partition sizes might be necessary.
• For increased security, you can mount /boot, /usr, /usr/local, and maybe /opt (depending on installed software) as read-only when in production. Also, the mounting of /local could be made non-automatic.
• You might be able to umount /boot after system startup.
• /home is not used in VMware ESX Server, and thus not required, nor defined.
• You might need a bit larger /opt in some cases. Check and know your optional 3rd party software.
• Use /local for backuping the Service Console with a tar command. More information on that later in this document.
• About 2000 MB of internal disk space will be left unpartitioned in the extended partition for possible future requirement.

Partitioning and otherwise handling the external disk space for VMs

There are two grand philosophies (schemes) with partitioning the external disk space where VMs reside. You can combine the schemes, if required. Both schemes are presented below:

• "Scheme A": Place all small files (.vmx, .log, .txt, and nvram) belonging to one VM into one ext3 formatted directory (/data/vm001/), and all big .dsk, and .REDO files, and the .vmss files into a single large vmfs formatted partition (/vmfs/public01/).
• "Scheme B": Place all small files (.vmx, .log, .txt, and nvram) belonging to one VM into one ext3 formatted directory (/data/vm001/). Then create a vmfs partition for the VM, which is big enough for all its .dsk, and .REDO files, and the .vmss file. The vmfs partition should be named (during formatting) after the VM (vm001), so it can be accessed as /vmfs/vm001/.

Please note, that both schemes place VM configuration, and other related small files to a subdirectory under the /data directory.

Scheme A is absolutely the most common, and used about 99 % of the time. It is however possible to mix both schemes at the same time. Some organisations use scheme B for the most important VMs.

External disk space partitioning scheme A

This partitioning scheme allows easier growing of .dsk files and generally is more efficient on disk space usage. It is also a bit riskier. Something could happen to the vmfs partition where all the .dsk files are. The filesystem could corrupt or something.

Partitioning example for system with 200 GB fibre channel logical drive. The first formatted filesystem is ext3, and the rest are of type vmfs with accesibility set to either public (no operating system cluster VMs exist in the SAN) or shared (operating system VM quorum, and shared disks exist in the same SAN):

/dev/sdb1   primary       200   ext3 fs       mounted as /data for VM configs and logs
/dev/sdb2   extended  ~200000   volume name   example name: vmhba2:1:0:2
/dev/sdb5   logical   ~160000   public01      /vmfs/public01; accessibility: public
/dev/sdb6   logical     39000   shared01      /vmfs/shared01; accessibility: shared

• If you don't have/plan to have any Netware/Linux/Windows fault-tolerant clusters, then you can safely ignore the "shared01" volume/partition.
• VM cluster shared disks such as the quorum must reside on vmfs volumes set as shared. Their (the cluster VMs) SCSI adapter setting should also be set to virtual (you have a cluster-in-a-box) or physical (cluster nodes are not in the same ESX Server).
• Directory /data contains one subdirectory for every VM defined. Each subdirectory is named after the VM (vm001, vm002, vm003, and so on).

External disk space partitioning scheme B

This partitioning scheme needs more preliminary planning. Even then, you might end up having external disk space allocation headache.

It is important to note, that VMware does not endorse this scheme. They recommend placing just .dsk files to VMFS volumes.

Partitioning example for system with 200 GB fibre channel logical drive; all formatted filesystems are of type vmfs and accesibility is set to either public (only single VMware ESX Server exists in the SAN) or public (also another VMware ESX Server exists in the SAN):

/dev/sdb1   primary       200 ext3 fs       mounted as /data for VM configs and logs
/dev/sdb2   extended  ~200000 volume name   example name: vmhba2:1:0:2
/dev/sdb5   logical     10000 vm001         vm001-os.dsk, 6000
/dev/sdb6   logical     30000 vm002         vm002-os.dsk, 4000 & vm002-data.dsk, 20000
/dev/sdb7   logical     10000 vm003         vm003-os.dsk, 7000
/dev/sdb8   logical     95000 vm004         vm004root.dsk,10000 & vm004home.dsk, 75000

• Please read the notes for scheme A, too.
• Please note, that some free space should always be left on the partition for non-persistent, undoable, append mode .dsk.REDO files, and a suspend file.
• Example: vm001 is a Win2000 infrastructure server with AD, DNS, DHCP, etc. only. There is no user data.
• Example: vm002 is a Win2000 general purpose file server with separate OS+applications in the vm002-os.dsk, and user home directories and common shared directories in vm002-data.dsk. This separation makes many nice things possible...
• Example: vm003 is a Win2000 cluster node with applications. Shared directories and quorum are in /dev/sdc#. /dev/sdc is not present in this document/example.
• Example: vm004 is SuSE GNU/Linux 9.0 server with a single .dsk file for all but /home, and a single huge .dsk file for /home.
• As one VM is placed in one vmfs partition, it is practical to name the partition same as its VM. If you do that, the .vmx file is referred to as /vmfs/vm001/vm001.vmx, or similar. Coherency makes easy administration, you know.

Multiple ESX Servers connected to the same fibre channel infrastructure

On multiple VMware ESX Server configurations with shared fibre channel storage, the following (seemingly a little conflictiong) issues and restrictions apply:

• All the VMs on one logical drive are executed on the same ESX Server at one time.
• If an ESX Server has any local/public files on the fibre channel box, create a server group for it, and a logical drive for it. Make vmfs partitions on that logical drive of accessibility type public if possible. VM clusters might require accessibility type shared (Windows ones do). You could have other file systems such as ext3 for VM configs there, too.
• All storage shared between VMs on different ESX Servers (i.e. storage failing over between cluster nodes, or similar) should be placed on separate dedicated logical drives. This is because logical drive is the smallest possible disk unit to move (fail over and back) between ESX Servers.
• On the fibre channel configuration place all the ESX Servers to the same server group so that they can all potentially access the same logical drives even at the same time, if required.

Network: setup and configuration-related issues

Please consider the following issues:

• It is recommended to allocate the first netcard (usually an onboard netcard) to the Service Console, or you might lose management connection to the ESX Server.
• The sharing resolution for PCI devices is one card. Netcards cannot be shared between Service Console and VMs, but SCSI cards can be shared.
• For multiport NICs the sharing resolution is a single port.
• It is strongly recommended to set the speed and full duplex setting for each physical netcard.
• Service Console netcard settings are set in the Service Console /etc/modules.conf file. Occasionally you might want to change some options for speed or duplexity.
• Best possible performance is obtained if each VM is allocated its own netcard. Usually this is not possible. The next best thing is to limit the network speed allowed for each VM sharing a physical netcard (or port of a card). ...But. Only limit the speed if you need to. Most of the time only busy fileserver VMs eat up the full network bandwidth.
• Remote console uses port 902/tcp by default, so it, and ports 22/tcp, 80/tcp, and 443/tcp should be open to the ESX Server Service Console netcard.

User/group structure on VMware ESX Server Service Console

There are again two schemes you can choose. The first one (A) is the one which is the default. The second one (B) has a quite different approach.

• "Scheme A": This is the default scheme, where there is only one user, which can make changes to a certain VM configuration. This user could be a single named account ('admin', or similar), or it could be the human account ('johndoe', or similar) who created the VM.
• "Scheme B": This scheme has one or more VM administrator group(s), which can make changes to all VM configurations belonging to the group.

No matter which one you choose, you can always convert to the other one rather easily. Change-over propably takes quite a long time, as it requires powering off all the VMs one-by-one for a short period of time.

Service Console user structure scheme A

Using this scheme, you create one userid for each administrator. The are all placed to one primary group "users". When VMs are created, the creator-administrator gets "rwx" rights to the VM, and the group gets "rx" rights. This means only the creator-administrator can make changes to the VM. Depending on your environment, this could, or could not be what you want.

Service Console user structure scheme B

Using this scheme, you create one userid for each administrator. The are all placed to one primary group "adminsNN". If only one group is required, you could name it "vmadmins". If multiple groups are needed, create groups "admins01", "admins02", etc.

If you define multiple groups for administrators on a large installation, it is a good idea to create one dummy admin account for each group (username "admin01", "admin02", etc.) which only belongs to that group and is used to create new VMs belonging to that administrator group. By doing that, you ensure the access rights are right for the newly created VM.

Actual administrator user accounts always belong to one user group primarily, and optionally to the other user groups. There is more discussion on this at the end of this document (Summary of initial tasks...).

When VMs are created, the creator-administrator gets "rwx" rights to the VM, and the group gets "rx" rights. This means only the creator-administrator can make changes to the VM.

The discussion below presumes you only have one administrator group named "vmadmins".

The difference between the two schemes is made by:

• Making the "vmadmins" group owner of the configuration file directory, .dsk file volume configuration (.vmx) file, nvram file, log files, and the .dsk files belonging to the VM.
• Giving "rwx" rights to the above volumes, directories, and to the (.vmx) configuration files.
• Giving "rw" rights to the VMs nvram files, log files, and the .dsk files.

As you can see, the difference between scemes is quite small. It is still very significant. Using scheme B also the "vmadmins" group can make required changes to the VM. Now, depending on your environment, this could, or could not be what you want. The earlier you make your choise between schemes, the easier.

VMware ESX Server tools and documentation

Most, if not all VMware Service Console utilities have a good man page. Enter "man xxx" to get the manual page for command "xxx". Example: man vmkfstools.

VMware ESX Server vmfs partition mount utility is vmware-mount.pl. Parameters -p xxx.dsk will print out the partition table of the .dsk file. Parameters xxx.dsk partition# /mnt/point will mount a particular partition of a .dsk file to a specified mountpoint in the service console. By default public vmfs partitions are mounted under /vmfs/ directory. You have to mount public and shared partitions manually or you could mount them automatically at every boot by placing correct mount-vmfs lines to the end of /etc/rc.d/rc.local file on the Service Console.

General and very comprehensive vmfs filesystem utility is vmkfstools. It can format, convert, and rename volumes. It can shrink and expand .dsk files. It can do many many more important things.

VMware ESX Server performance monitoring is installed by default, but it is not activated. It can be activated with command vmkusagectl install. If you want to cleanup the statistics, you will have to uninstall the monitoring service with command vmkusagectl uninstall. You can then clean the database with command vmkusagectl cleandb, and (re)activate/(re)install it again. The excellent monitoring web pages, created with rrdtool, are at address "http://esx-server-name/vmkusage/".

Command esxtop is like the top command, but it shows the VMkernel processes instead.

vdf command displays the amount of free space on different volumes, including VMFS volumes.

ESX netcard utility for locating correct netcard among many netcards is findnic.

Some other commands are: vmfs-ftp, vmkmultipath, vmklogger, vmware-cmd, vm-support.

Some examples on vmware-cmd usage is given below:

vmware-cmd /data/vm001/vm001.vmx getstate

Retrieve power state of the VM: off, on, suspended, stuck.

vmware-cmd /data/vm001/vm001.vmx start

Power on the VM.

vmware-cmd /data/vm001/vm001.vmx reset trysoft

Reboot the VM. First try a nice shutdown, then if necessary force a shutdown before reboot.

vmware-cmd /data/vm001/vm001.vmx stop trysoft

Shutdown/halt the VM. First try a nice shutdown, then if necessary force a shutdown. Finally power off.

vmware-cmd /data/vm001/vm001.vmx suspend

Suspend a VM.

vmware-cmd /data/vm001/vm001.vmx addredo public01:vm007.vmdk

Add redo log to a powered-on VM.

vmware-cmd /data/vm001/vm001.vmx commit public01:vm007.vmdk 0 0 1

Commit the above created redo log to the disk file. The three numeric parameters are: level (0=.vmdk is in persistent state, 1=otherwise), freeze (0=do not freeze VM during commit, 1=freeze VM during commit), and wait (0=finish the command immediately, 1=wain until the commit is finished, the finish the command).

Most of these tools are properly documented in the VMware ESX Server manual and/or on VMware's website.

VMware ESX Server-related Linux files and directories

There are a couple of files and directories you should know about. The most important ones are listed below.

/etc/modules.conf

This file contains a list of devices in the system available to the Service Console. Usually the devices allocated solely to VMs, but physically existing on the system are also shown here in the commented-out ("#") lines. This is an important file for root and administrators.

/etc/fstab

This file defines the local and remote filesystems which are mounted at ESX Server boot.

/etc/rc.d/rc.local

This file is for server local customisations required at the server bootup. Potential additions to this file are public/shared vmfs mounts.

/etc/syslog.conf

This file configures what things are logged and where. Some examples are given below:

*.crit     /dev/tty12

This example logs all log items at level "crit" (critical) or higher to the virtual terminal at tty12. You can see this log by pressing [Alt]-[F12] on the console.

*.=err     /dev/tty11

This example logs all log items at exactly level "err" (error) to the virtual terminal at tty11. You can see this log by pressing [Alt]-[F11] on the console.

*.=warning     /dev/tty10

This example logs all log items at exactly level "warning" to the virtual terminal at tty10. You can see this log by pressing [Alt]-[F10] on the console.

*.*     192.168.31.3

This example forwards everything (all syslog entries) un-encrypted to another (central) syslog server. Pay attention to that server's security.

/etc/logrotate.conf

This is the main configuration file for log file rotation program. It defines the defaults for log file rotation, log file compression, and time to keep the old log files. Processing the contents of /etc/logrotate.d/ directory is also defined here.

/etc/logrotate.d/

This directory contains instructions service by service for log file rotation, log file compression, and time to keep the old log files. For the three vmk* files, raise "250k" to "4096k", and enable compression.

/etc/inittab

Here you can change the amount of virtual terminals available on the Service Console. Default is 6, but you can go up to 9. I almost always go :-)

/etc/bashrc

The system default $PS1 is defined here. It is a good idea to change "\W" to "\w" here to always see the full path while logged on the Service Console. This is one of my favourites.

/etc/profile.d/colorls.sh

Command "ls" is aliased to "ls --colortty" here. Many admins don't like this colouring. You can comment-out ("#") this line. I always do this one, too.

/etc/init.d/

This directory contains the actual start-up scripts.

/etc/rc3.d/

This directory contains the K(ill) and S(tart) scripts for the default runlevel 3. The services starting with "S" are started on this runlevel, and the services Starting with "K" are killed, i.e. not started..

/var/log/

This directory contains all the log files. VMware's log files start with letters "vm". The general main log file is "messages".

/etc/ssh/

This directory contains all the SSH daemon configuration files, public and public keys. The defaults are both secure and flexible and rarely need any changing. The only exception is a change to /etc/ssh/sshd_config file if you want to restrict logins for root user.

/etc/vmware/

This directory contains the most important vmkernel configuration files.

/etc/vmware/vm-list

A file containing a list of registered VMs on this ESX Server.

/etc/xinetd.conf

This is the main and defaults setting configuration file for xinet daemon. Processing the contents of /etc/xinetd.d/ directory is also defined here.

/etc/xinetd.d/

This directory contains instructions service by service for if and how to start the service. Of the services here, vmware-authd, wu-ftpd, and telnet are most interesting to us. Two of the most interesting parameter lines are "bind =" and "only_from =", which allows limiting service usage.

/etc/ntp.conf

This file configures the NTP daemon. Usable public NTP servers in Finland are fi.pool.ntp.org, elsewhere in Europe europe.pool.ntp.org. You should always place two to four NTP servers to ntp.conf file. Due to the nature of *.pool.ntp.org, you should just have the same line four times in the configuration file. Check www.pool.ntp.org for a public NTP server close to you. Remember to change the service to autostart at runlevel 3.

VMware ESX Server-related Linux commands

There are several commands you should familiarise yourself with. Most of them and some more are listed here. All of them have an online manual page, which you can read with the command "man command-name".

man

Prints the manual page for a command or a configuration file entered as a parameter to this command.

reboot

Does a nice reboot on the system. Does "Force Power Off" for the VMs.

halt

Does a nice halt on the system. Does "Force Power Off" for the VMs.

shutdown

Generic command for shutting down or rebooting the system.

fdisk

Command line disk partitioning program in Linux. It is powerful and has a very simple user interface. Please note, that ext2, and ext3 both use the same partition ID.

fdisk /dev/sdb

On command line, starts fdisk against second available SCSI disk. "sda" is the first SCSI disk, "sdc" is the third SCSI disk etc. VMware ESX Server is installed on /dev/sda, and the external storage is /dev/sdb, and maybe some others too.

p

Fdisk subcommand, prints the current partition table on current disk.

d

Fdisk subcommand, deletes an existing partition. Enter the partition number to delete. It is recommended to printout the current partition table before deleting anything.

n

Fdisk subcommand, creates a new partition. Select partition type (primary, extended, or logical). Almost always you should use the default starting cylinder. For size, enter "+NNNNNm", where NNNNN is the size in megabytes.

t

Fdisk subcommand, change partition type (id). By default fdisk creates ext2/esx3 type partitions. We might also want sometime to use id "fb", the vmfs type, or some other type.

w

Fdisk subcommand, writes the current partition table to disk. If you don't get any errors, you don't have to reboot. If you get errors at this point, the new partition table is used only after next system boot.

mke2fs

This command formats a partition for ext2, or ext3 filesystem.

mke2fs -j /dev/sdb1

Formats /dev/sdb1 using ext3 filesystem.

mke2fs /dev/sdb1

Formats /dev/sdb1 using ext2 filesystem.

mkdir

Makes a directory.

mkdir /data

Creates directory /data for the VM configs.

nano

Edit a file with a bit easier UI that vi.

nano -w /etc/fstab

This is propably the very first file editing command you want/need. "-w" turns word-wrapping off, so you can more easily edit longer lines than about 74 characters.

nano /etc/inittab

nano /etc/bashrc

mount|umount

These commands manually mount/umount CDs, floppies, local partitions, and remote directories to a selected local directory. The local (empty) directory must exist before the mount can succeed. Example mound command would be "mount /dev/sdb5 /data". Permanent mounting is done by editing the /etc/fstab file.

mount

Shows all the active mounts.

mount -a

Remounts everything specified in /etc/fstab file. This is propably the very mount command you will be entering.

mount /dev/cdrom

This command does the default mounting of a CD to the default mountpoint. In Service Console the CD is mounted to /mnt/cdrom directory.

mount /mnt/floppy

Mounts a normal 1440KB floppy (/dev/fd0) to the specified directory.

mount -t iso9660 -o loop /local/w2005srv.iso /mnt/isocd

Mount a CD/DVD ISO image file to the specified directory. This is very useful for testing and other purposes. The mountpoint directory must exist (mkdir /mnt/isocd) before mounting.

umount /mnt/cdrom

Unmount anything mounted to the specified mountpoint. If nothing is mounted, the command does nothing.

rm

Removes files and/or directories.

mv

Moves and/or renames files and/or directories.

kudzu

This is the RedHat's tool to detect and configure hardware: adding new and removing old. When you run kudzu, or system runs it at bootup, be careful. Kudzu might offer to remove hardware you have dedicated solely to the VMs. Know your hardware and configuration. It might be a good idea to refer to /etc/modules.conf file before running kudzu. A safe action to select in kudzu is "Do nothing". Select it when in doubt.

service

RedHat-made tool for daemon (service) starting/stopping/restarting/status querying. Syntax is "service daemonname [start|stop|restart|status]". Alternate to this command, which works with all Linuces is to call the script directly, like /etc/init.d/httpd.vmware restart, /etc/init.d/xinetd restart, or /etc/init.d/sshd restart.

groupadd

Adds a new group to the Service Console. It is recommended to use one non-root group for VM admins and add operator/admin users there. To create that group, enter the following command:

groupadd -g 7777 vmadmins

Create a group with groupid number 7777. This number is an arbitary number. For practical (not explained here) reasons this number should have four digits.

useradd

Adds a new user to the Service Console with status disabled. To create an account for the new admins, enter the following commands:

useradd -c "VMware ESX Server operator" helpdesk

Create a single userid, which will be able to operate all of the VMs.

useradd -g 7777 johndoe

Create a userid, and make groupid 7777 (vmadmins) as its primary group.

useradd -g 7777 -c "Kari Mattsson" mattkar2

Create a userid, and make groupid 7777 (vmadmins) as its primary group.

usermod

Changes settings for a user. Usually used for user group manipulation.

usermod -G wheel mattkar2

passwd

Changes the password for the userid entered as a parameter for the command. Only root can change the password for other users. They can only change their own password with command "passwd". Userids are disabled by default. They are activated by setting a password for them. An example command for root to set a password is the following command:

passwd johndoe

chown

Changes the owner user and optionally owner group of a directory, or a file. Optionally this command works recursively with parameter "-R". The assignment parameter is of type "user.group", or just "user". Some examples are given below:

chown -R helpdesk.vmadmins /vmfs /data

Recursively change the user-owner, and the group-owner of specified files/directories to userid.groupid.

chown helpdesk.vmadmins /vmfs/local/*

chown -R root /data/vmware

chown root.vmadmins /etc/modules.conf

chgrp

Changes the owner group of a directory, or a file. Optionally this command works recursively with parameter "-R". Examples for "chown" apply here, but without the "root." part, as only the group is changed here.

chattr

Change special attribute of a directory, or a file. Immutable attribute is set with parameter "-i".

chmod

This command is the main command for changing file modes. Like chown, it can do things recursively with parameter "-R". Below are some example commands:

chmod -R 0775 /vmfs /data

chmod u=rwx,g=rwx,o=r /vmfs/freebsd462/*

chmod g+rwx /vmfs/vm007/*

chmod -R u+rwx,g=r,o-rwx /var/log/*

chmod u=rw,g=rw,o=r /etc/modules.conf

chmod 664 /etc/modules.conf

chmod u=rw,g=rw /vmfs/*/*.dsk

It appears, that this last example works rather nicely. Note, that those VMs which are powered-on, have their .dsk files locked.

dd

With this 'disk dump' command you can create ISO images and floppy images. You can also use it to create imagefiles of partitions and whole disks. Below are some example commands:

dd if=/dev/cdrom of=/local/suse90pro-dvd1.iso bs=2048

dd if=/dev/cdrom of=/local/w2003srv.iso bs=2048

The above two examples create an ISO image of a CD/DVD. You can safely ignore the error message usually shown at the end of the media.

dd if=/dev/fd0 of=/local/bootfloppy1.img bs=1440k

This command creates a floppy image quickly.

dd if=/dev/fd0 of=/local/bootfloppy2.img bs=512

This might be a bit slower version of the above example.

cat

ConCATenate file from start to standard output (terminal screen by default). Usually takes filename as a parameter.

ls

LiSt files in a directory. -R makes it recursive, and -l shows more information on each item.

stat

Show statistics of a file. This is the most comprehensive directory entry examiner.

tac

Like "cat", but starts from the end of the file (or standard input).

head

Show selected amount of lines from the start of a file.

tail

Like "head", but start from the end of the file. Practical command to follow what is happening with a log file is command like tail -f /var/log/messages.

grep

Search for a string from standard input or from a file. This is a powerful command.

find

Find files by name or many of the other attributes. Another very powerful command. Below are some example commands:

find /vmfs -type f -iname *.dsk

find /data -type f -iname *.vmx

find / -type f -iname *.bak

find . -type d -name sbin

find / -type f -name *

tar

Tape ARchive, a command which combines many files into one for backup purposes. Below are some example commands:

tar -cvjf /local/servcons.tar.bz2 --exclude /proc --exclude /local --exclude /vmfs --exclude /data /

Create a bzip2'ed tar backup file the whole Service Console. Smaller, slower backup.

tar -cvzf /local/servcons.tar.gz --exclude /proc --exclude /local --exclude /vmfs --exclude /data /

Create a gzipped tar backup file the whole Service Console. Faster, bigger backup.

tar -cf /local/vm-configs.tar /data

Create a tar backup file of all files in and under /data directory.

tar -xvzf /local/vm007-config.tar.gz

Extract gzipped tar backup file to current directory.

tar -xvjf /local/vm007-config.tar.bz2 -C /tmp

Extract gzipped tar backup file to /tmp directory.

find / -type f -iname vm007* | tar -cjvf /local/vm007-backup.tar.bz2 -

Find all files starting as 'vm007', and create a compressed backup tar file of them.

bzip2|bunzip2

These commands compress and decompress files. The recommended and default extension is .bz2. The compression is slower that with gzip, but the files are considerably smaller. Decompression is fast.

gzip|gunzip

These commands compress and decompress files. The recommended and default extension is .gz. The compression is quite fast, and files are quite small. Decompression is fast.

more|less

These commands are almost the same, and usually act in a pipe. They are used for file pagination to terminal. Below are some example commands:

zcat /var/log/vmksummary.1.gz | less

more /etc/passwd

ntpdate

This command takes an NTP server as a parameter and synchronises the clock once. This command doesn't work when local NTP daemon is running. Example: ntpdate europe.pool.ntp.org

Install VMware tools to different operating system VMs

These instructions serve as a tiny reminder for administrators on how to do this task. Remember, that this installation has to be done on every VM after an VMware ESX Server upgrade. In the real life, the VMs usually work quite nicely with a downlevel VMware tools, but the combination is not supported, nor recommended.

A hint for VMware: It would make many administrators life a lot easier, if this VMtools upgrade could be done more easily. Think of 2 ESX Servers with 20 VMs on both, and you see what a hassle it can be.

Install VMware tools to GNU/Linux

Many later Linux distributions have a built-in support for VMware's virtual display adapter and have drivers for it. If you have an older distribution, or a custom one, then when you are installing X to the Linux VM, select "Unsupported VGA compatible" as the display adapter.

If you are installing VM to be used as a server, try to do without X.

First you have to open "xterm" or similar CLI. You also could press [Ctrl]+[Alt]+[F2] to switch to TTY2. Login as root there, if not already logged-in. Then follow the instructions.

After selecting in the Remote Console "Settings", and then "VMware Tools Install". Enter the following commands:

cd /tmp
mount /dev/cdrom
cp /media/cdrom/* /tmp               ### See the note below!
umount /dev/cdrom
tar -xzf vmware-linux-tools.tar.gz
cd vmware-tools-distrib
./vmware-install.pl

A note on the 'cp' command above:
On Gentoo CD/DVD is mounted to '/mnt/cdrom'. On SUSE it is mounted to '/media/cdrom', and/or '/media/dvd'. On RedHat-family of Linuces CD/DVD is mounted to '/mnt/cdrom'. On Debian it is mounted to '/cdrom'. So change the 'cp' command above accordingly.

...now during the script execution just accept the defaults most of the time by pressing [Enter]. It is when you ar easked about display resolution for X, when you have to stop for a while and think. Select proper resolution, and you are done with the VMware Tools install.

Install VMware tools to Novell Netware

While logged on as admin on the console, do the following to install VMware tools:

1. On the Remote Console menu, select "VMware Tools Install...".
2. On the console enter the following command: vmtools:\setup
3. Wait for a few seconds, and you can see the tools have been installed.

Install VMware tools to Microsoft Windows

White you are logged-on as an administrator, do the following to install VMware tools:

1. On the Remote Console menu, select "VMware Tools Install...".
2. VMware Tools installation begins automatically, if your VM has autorun enabled. If it doesn't have, you will have to start the installation manually from the VMs just-mounted-CD.
3. Wait for a few seconds, press "Next", and/or "Yes" several times, and you can see the tools have been installed.

The default full installtion also installs VMXnet network adapter driver.

Network: A note on Service Console (Linux) security in production use

This discussion presumes the default high security level is set during the initial VMware ESX Server configuration. In this security level only the following ports are open on the Service Console:

22/tcp

SSH daemon listens to this port for remote connections. By default password authentication is used for logons. RSA/DSA public/private key authentication can be used and it is actually tried first. Userid/password authentication is actually tried second. For higher security and for automated/scripted logons RSA/DSA authentication must be used.

902/tcp

VMware authd, the web management UI (MUI) and remote console authentication daemon (service) for VMware ESX Server uses this port. The daemon is not listening on this port directly, but xinetd does. When someone open connection to port 902, xinetd then launches authd, and the actual authentication starts. Xinetd-related authd security is defined in the file /etc/xinetd.d/vmware-authd.

80/tcp and 443/tcp

The httpd.vmware application web server listens to these ports. With high security on, all connections to port 80 are automatically redirected to port 443.

8222/tcp and 8333/tcp

These ports are used by ESX Server's web UI. They are just forwards to ports 80 and 443 respectively. These ports do not need to be open on the firewalls.

Remember, that sshd is by default always running on the Service Console, so you can always connect to it and do low level management directly to the Service Console files. An example of this kind of management is when the MUI stops responding. Just login using your account via ssh, and enter the following command to restart the webserver responsible for the MUI: su -c "/etc/init.d/httpd.vmware restart". You normally need the root's password to complete this task. You could (should!) also use sudo/visudo to make thing even easier.

Additional tasks to increase security

This is mainly a list of things to do. Not all of this applies to all installations.

1. Disable SSH login for root account
   Edit file /etc/ssh/sshd_config and change line # PermitRootLogin yes to PermitRootLogin no. The restart sshd with command /etc/init.d/sshd restart. From now on you have to use you normal account to login via SSH, and then use su - command to switch to root, or use sudo command to execute commands as a root.
2. Limit access to the su command
   This is done by adding each user to a special group "wheel" with command usermod -G wheel userid. Then edit file /etc/pam.d/su and remove comment mark "#" from the following line: auth required /lib/security/pam_wheel.so use_uid. Now users in "wheel" group can use su command.
3. Enable sudo command for the user group "vmadmins"
   With this setting enabled, all the users in "vmadmins" group can execute commands that require root privileges. When they do, they are asked their own password before the command is executed. That way reasonable security is enforced. This task is easily done by executing the visudo command, and adding the following line to the end of the file: %vmadmins ALL=(ALL) ALL, or rather more specific %vmadmins ALL=(ALL) "/usr/local/vmadmin-scripts". On the latter example, you also have to link all the scripts you want to enable to the specified directory. You could fine-tune this by editing the /etc/sudoers file with visudo.
4. Use PAM to selectively disable access to selected services with pam_listfile.so module
   You can limit what is allowed user by user, service by service. Discussion on this is beyond the scope of this document.

Logrotate script to rotate vmware.log files

The file below should be saved as /etc/logrotate.d/vmware_log, and the rotating will start.

# logrotate script for vmware.log files
# 
/data/*/vmware.log {
    size 1M
    compress
    dateext
    rotate 2
    notifempty
}

Backup VMs to other host with sshd on the Service Console

Please note, that from the "other host", where you have backed up the VMs, the VMs can be backed up to tape, etc.

As you cannot copy the .dsk files while the VP is powered on, it is probable, that the time windows for executing this script on a production server is too small. That is why this script is more a proof-of-concept.

This discussion presumes the following:

• Default high security level is used on the VMware ESX Server.
• A userid "backup" has been created on the Service Console for backup purposes. This userid only needs read rights to the files for backup, and read+execute rights to the directories where these files are.
• SSH daemon must be started. By default it is.
• SSH suite command-line program "scp", or "scp2" will be used on the "other host" where the backuped VMs will be stored.
• RSA/DSA public/private keys have been created for the userid "backup" and they will be used.
• There must be enough disk space, and filesizes above 4 GB must not be a problem on the disk where the VMs are backed up.
• The VM to be backed up must be in power off state.

Now the following shell script can be used on the "other host" to fetch a VM passed as a parameter to the script:

#!/bin/bash
# Fetch files related to VM $1 from VMware ESX Server using userid 'backup'.
export vmname=$1
mkdir /backup/esx01.trivore.com/${vmname} 2>/dev/null
cd /backup/esx01.trivore.com/${vmname}
scp -p backup@esx01.trivore.com:/vmfs/${vmname}/* .

Name the above script "/usr/local/sbin/esx-vm-backup", and execute it with parameter "vm001", or any other VM name. Example command: /usr/local/sbin/esx-vm-backup vm001

Task: Do a full backup of the Service Console

This backup, to ensure all files make it to the backup, should be done in runlevel 1, i.e. in single user state. In that runlevel, network, and vmfs volumes are all unavailable. Also the ESX Server is not up in runlevel 1.

The way to enter single-user Linux mode is:

• Enter "linux single" at the "boot:" prompt when starting up the server. You have to be rather quick at this.

It is presumed here, that a sufficiently large partition exists, and is mounted to directory /local. If not, then some adaptations has to be made. Two working example commands are below to make a compressed tar backup file:

tar -czf /local/service-console-2.5.1-`date -I`.tar.gz \
	--exclude /proc --exclude /local --exclude /data /

tar -cjf /local/service-console-`head -2 /etc/issue|tail -1|cut -d" " -f4`-`date -I`.tar.bz2 \
	--exclude /proc --exclude /local --exclude /data /

...where 2.5.1 is the VMware ESX Server version number, and `date -I` is replaced with an ISO standard format of current date, such as "2005-10-13". Second example will fetch the VMware ESX Server version number automatically. Only the descending single quotes shown in the example do work.

Normal size for the backup file is about 200 MB, and it is created in two or three minutes. The "z" in the first example uses gzip for bigger backup file. The "j" in the second example uses bzip2 for smaller backup file. For compression gzip is faster than bzip2. For decompression, the speed is roughly the same.

If you want to time the backup, or any other command, just begin the whole command-line with an additional command "time ".

It is also recommended to save a current copy of /etc/fstab, your current mounts, and current partitioning. These are all very helpful items in a crisis situation. Just enter the following commands to save this information to /local directory.

cp /etc/fstab /local/current_etc_fstab
mount > /local/current_mount
vdf > /local/current_vdf
fdisk -l > /local/current_fdisk-l

As the above information does not change very often, it is a very doog idea to have the above 4 files printed out next to the server.

For help in restoration you have to contact someone with enough Linux knowledge, as the scenarios, options, and choices on what/how to do it vary so much. Trivore Corp. will obviously be happy to help you - for a moderate fee.

Task: Do a full restore of the Service Console after a catastrofic failure

Too numerous requests have arrived to include this information, so here it is.

Full VMware ESX Server Service Console restore is actually rather simple. Below are the required steps. Note, that depending on how badly corrupted your system is, you might be able to skip some of the steps.

1. Install VMware ESX Server from scratch. Be very careful when partitioning system. If your VM configuration files in the /data directory is safe on the external storage, and your virtual hard disks, are safe, then do not touch the external partitions. Remember to mount the /data during install, but do for format it. If your internal
2. Enter linux single at the boot: prompt when starting up the server. You have to be rather quick at this.
3. As as example, enter tar -xjf /local/service-console-2.5.1-2005-10-13.tar.bz2 -C / to restore the backup. Use options "-xzf" if you used gzip for compression (.tar.gz). Use your backup file name here.
4. Enter lilo to reinstall proper Linux loader.
5. Enter reboot to reboot the system back into production.

The above task can be done in less than 15 minutes on just about any hardware. As you may have noticed, some of the large iron boots for a long time.

Commands (script!) to fix user rights on VM directories and files

Please note, that these commands require, that you have implemented userid plan B.

These commands make a script you should execute regularly/occasionally as a root to make sure you have the proper rights on VM directories and files.

#!/bin/sh
# 
# I am /usr/local/sbin/vmrights.
# 
# Note: You will get an error message on ESX swap file on the 'chgrp' 
#       line below.  Add '2> /dev/null' to the line below to avoid it.
# Note: This script is for single VM admin group setups only.
# Note: If you are using group other than 'vmadmins', change the line 
#       below accordingly.
# 
data=/data
admingrp01=admins01
chgrp -R $admingrp01 $data /vmfs
chmod 0775 $data $data/* /vmfs /vmfs/*
chmod 0660 /vmfs/*/*.dsk /vmfs/*/*.vmdk
find $data -type f -name *.vmx -exec chmod 0770 {} \;
find $data -type f -name nvram -exec chmod 0660 {} \;
find $data -type f -name vmware*.log -exec chmod 0660 {} \;
# 
# If you have one or more VMs, where all the files are in one single 
# VMFS volume (External disk space partitioning scheme B), then 
# you are required to make some changes to this script.

If you make this a script, then please store it (as a root) to /usr/local/sbin as 'vmrights', or similar. Also give root executable rights (chmod u+x [filename]) to this script for easy execution.

chmod u+x /usr/local/sbin/vmrights

Remember, that only root can really execute this script/these commands.

It is a good idea to execute this script automatically to fix the file permissions. That way all administrators can do what they need to do. To make the script execute either daily, or hourly, execute only one of the commands below, not all of them:

ln -s /usr/local/sbin/vmrights /etc/cron.monthly
ln -s /usr/local/sbin/vmrights /etc/cron.weekly
ln -s /usr/local/sbin/vmrights /etc/cron.daily
ln -s /usr/local/sbin/vmrights /etc/cron.hourly

Summary of initial tasks to be done just after installation (...or sometime later :-)

These command could be combined to a script to be executed. Because these are one-off commands, many admins just copy-paste these to a VMware ESX Server Service Condsole ssh session for immediate execution.

##### SECTION: User and group management.
# 
# Only create as many VM admin groups as you need.
# Also below, only create generic admin if you created the group.
groupadd -g 7001 admins01
groupadd -g 7002 admins02
groupadd -g 7003 admins03
#
# These generic admins are only used for creating the VMs
# belonging to a particular admins group (admins01, admins02, ..)
useradd -g 7001 admin01 ; passwd admin01
useradd -g 7002 admin02 ; passwd admin02
useradd -g 7003 admin03 ; passwd admin03
#
# Create an admin beloging primarily to one group, and 
# secondarily to another group. Set pasword.
useradd -g 7001 -G 7003 person1 ; passwd person1
#
# Create an admin beloging primarily to one group only.
# Set pasword.
useradd -g 7002 person2 ; passwd person2
#
# The group numbers 7001+ are arbitrary. Any unique number will do.
# All numbers between 1000 and 65533 are safe here.


##### SECTION: Keep correct time on the system.
# 
# Tasks below: Keep correct time on the server system.
# Configure ntpd, make it autostart, and start it now.
# Replace 'fi' with your own countrycode, or 'europe', or goto www.pool.ntp.org!
echo "# " >> /etc/ntp.conf
echo "# It is good idea to define at least 4 [*.]pool.ntp.org servers." >> /etc/ntp.conf
echo "server fi.pool.ntp.org" >> /etc/ntp.conf
echo "server europe.pool.ntp.org" >> /etc/ntp.conf
echo "server europe.pool.ntp.org" >> /etc/ntp.conf
echo "server pool.ntp.org" >> /etc/ntp.conf
chkconfig --add ntpd
/etc/init.d/ntpd restart


##### SECTION: Command aliases and correct keyboard.
# 
# Create some handy aliases and activate them immediately.
echo "alias nano='nano -W'" >> /etc/profile
echo "alias cd..='cd .." >> /etc/profile
echo "alias ..='cd .." >> /etc/profile
source /etc/profile
# Select your keyboard. The default is US.
setup


##### SECTION: Optional Service Console system logging changes.
# 
# Three lines below: we want to see several more severe log entries on TTYs too.
echo "# " >> /etc/syslog.conf
echo "# Comment-out 3 lines below to protect tty10-tty12" >> /etc/syslog.conf
echo "*.crit      /dev/tty12" >> /etc/syslog.conf
echo "*.=err      /dev/tty11" >> /etc/syslog.conf
echo "*.=warning  /dev/tty10" >> /etc/syslog.conf
# The line below: restart system logging daemon to activate above changes.
/etc/init.d/syslog restart


##### SECTION: A mandatory and an optional system change.
# 
# Make the important changes below:
#   - change 'umask 022' to 'umask 002' to everyone (after the 'else' line).
#   - change '\W' in PS1 environment variable to '\w' to enable seeing full path all the time.
# The first one is mandatory if you are using user acoount scheme B, the second one is optional.
#
nano -W /etc/bashrc


##### SECTION: Creating/formatting/mounting partition for VM configs.
# 
# Note: The tasks below are usually done at the installation phase with VMware
#       ESX Server 2.1+, so you can usually skip these.
#
# Enter the following command to start non-boot disk partitioning: fdisk /dev/sdb
#   ! ensure /dev/sdb is the proper disk to store VMs for you
#   - create 200 MB primary partition 'sdb1' (new, primary, 1)
#   - create rest-of-drive extended partition 'sdb2' (new, extended, 2)
#   - write-out the partition table, reboot server if you get errors on writing
#   - then you can enter the following commands finalise the task:
fdisk /dev/sdb
mke2fs -j /dev/sdb1
mkdir /data
echo "/dev/sdb1  /data  ext3  defaults  0 2" >>/etc/fstab
mount -a
# Now you want to use the Management UI to create/format/label VMFS volumes for VMs.
# Sometimes the MUI requires a server restart to recognise new partitions, 
# such as the /dev/sdb1 created just above.