In this example, we have an existing /data partition of 350 GB. Well, it’s just not big enough, so we’ll add another 100 GB hard drive and give it to the /data partition.

After adding the 100 GB hard drive to the VM through vCenter, connect to the server through SSH or the console. Here’s the existing setup:

# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
7.7G 2.2G 5.2G 30% /
/dev/sda1 99M 24M 71M 26% /boot
tmpfs 1.5G 0 1.5G 0% /dev/shm
/dev/mapper/VolGroup01-LogVol00
345G 15G 313G 5% /data

# cat /proc/scsi/scsi
Attached devices:
Host: scsi0 Channel: 00 Id: 00 Lun: 00
Vendor: VMware Model: Virtual disk Rev: 1.0
Type: Direct-Access ANSI SCSI revision: 02
Host: scsi0 Channel: 00 Id: 01 Lun: 00
Vendor: VMware Model: Virtual disk Rev: 1.0
Type: Direct-Access ANSI SCSI revision: 02

First we tell Linux to find the new drive in the correct location:

# echo “scsi add-single-device 0 0 2 0″ > /proc/scsi/scsi

Now we see the new drive has been detected:

# cat /proc/scsi/scsi
Attached devices:
Host: scsi0 Channel: 00 Id: 00 Lun: 00
Vendor: VMware Model: Virtual disk Rev: 1.0
Type: Direct-Access ANSI SCSI revision: 02
Host: scsi0 Channel: 00 Id: 01 Lun: 00
Vendor: VMware Model: Virtual disk Rev: 1.0
Type: Direct-Access ANSI SCSI revision: 02
Host: scsi0 Channel: 00 Id: 02 Lun: 00
Vendor: VMware Model: Virtual disk Rev: 1.0
Type: Direct-Access ANSI SCSI revision: 02

If we check fdisk, we see the new drive is blank like we expect:

# fdisk -l

Disk /dev/sda: 12.8 GB, 12884901888 bytes
255 heads, 63 sectors/track, 1566 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sda1 * 1 13 104391 83 Linux
/dev/sda2 14 1566 12474472+ 8e Linux LVM

Disk /dev/sdb: 375.8 GB, 375809638400 bytes
255 heads, 63 sectors/track, 45689 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sdb1 1 45689 366996861 83 Linux

Disk /dev/sdc: 107.3 GB, 107374182400 bytes
255 heads, 63 sectors/track, 13054 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/sdc doesn’t contain a valid partition table

So let’s give it a partition:

# fdisk /dev/sdc
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won’t be recoverable.

The number of cylinders for this disk is set to 13054.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)
Warning: invalid flag 0×0000 of partition table 4 will be corrected by w(rite)

Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-13054, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-13054, default 13054):
Using default value 13054

Command (m for help): p

Disk /dev/sdc: 107.3 GB, 107374182400 bytes
255 heads, 63 sectors/track, 13054 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sdc1 1 13054 104856223+ 83 Linux

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

Fdisk now confirms that everything looks good:

# fdisk -l

Disk /dev/sda: 12.8 GB, 12884901888 bytes
255 heads, 63 sectors/track, 1566 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sda1 * 1 13 104391 83 Linux
/dev/sda2 14 1566 12474472+ 8e Linux LVM

Disk /dev/sdb: 375.8 GB, 375809638400 bytes
255 heads, 63 sectors/track, 45689 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sdb1 1 45689 366996861 83 Linux

Disk /dev/sdc: 107.3 GB, 107374182400 bytes
255 heads, 63 sectors/track, 13054 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sdc1 1 13054 104856223+ 83 Linux

Verify that LVM can see the drive:

# lvmdiskscan
…skip…
/dev/sdc1 [ 100.00 GB]
…skip…

Create a physical volume within LVM:

# pvcreate /dev/sdc1
Physical volume “/dev/sdc1″ successfully created

Add/Extend this physical volume to the volume group:

# vgextend VolGroup01 /dev/sdc1
Volume group “VolGroup01″ successfully extended

Extend the logical volume to use the full size of the volume group:

# lvm lvextend -l +100%FREE /dev/VolGroup01/LogVol00
Extending logical volume LogVol00 to 449.99 GB
Logical volume LogVol00 successfully resized

Actually resize the partition for the OS to recognize the new size:

# resize2fs -p /dev/mapper/VolGroup01-LogVol00
resize2fs 1.39 (29-May-2006)
Filesystem at /dev/mapper/VolGroup01-LogVol00 is mounted on /data; on-line resizing required
Performing an on-line resize of /dev/mapper/VolGroup01-LogVol00 to 117961728 (4k) blocks.
The filesystem on /dev/mapper/VolGroup01-LogVol00 is now 117961728 blocks long.

And the new drive space is now available to the user, who is already busy filling it:

# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
7.7G 2.2G 5.2G 30% /
/dev/sda1 99M 24M 71M 26% /boot
tmpfs 1.5G 0 1.5G 0% /dev/shm
/dev/mapper/VolGroup01-LogVol00
443G 48G 373G 12% /data

One thing that I have always hated about using a Windows PC is when I pull up a command prompt to telnet to a mail server or something to test.  If you’ve ever tried this you know what I’m talking about – you can’t see what you are typing!  It’s not that bad, if you type perfectly all the time, but I do not.  Well, I was looking something up on MSDN today and happened to stumble upon this gem:  “set localecho”.  You have to specify it within the telnet prompt – you can’t specify it in one line on the command prompt, but at least it’s something.

From the command prompt, type just telnet, instead of the full line:

> telnet

Type in set localecho and then open the server port that you want:

telnet> set localecho
telnet> open mailserver.domainname.com 25

This command removes any files older then 2 years old in the current directory:

find . -type f -mtime +730 | while read filename; do rm -f “$filename”; done

Let’s start off with a simple disk group:

# vxprint
Disk group: tempdg-tmp

TY NAME ASSOC KSTATE LENGTH PLOFFS STATE TUTIL0 PUTIL0
dg tempdg-tmp tempdg-tmp – – – – – -

dm tempdg-tmp02 emcpower1s2 – 733975296 – – – -

v templv-tmp fsgen ENABLED 733975296 – ACTIVE – -
pl templv-tmp-02 templv-tmp ENABLED 733975296 – ACTIVE – -
sd tempdg-tmp02-01 templv-tmp-02 ENABLED 733975296 0 – – -

This shows the disk group’s name is tempdg-tmp, the original disk in the group is called tempdg-tmp02, the logical volume is called templv-tmp, which is made up of one plex called templv-tmp-02, which has one subdisk called tempdg-tmp02-01.

So, let’s add an additional disk tempdg-tmp01 and use it to mirror the data on tempdg-tmp02. If we look first at how the device is setup for the original drive we see that it is using cdsdisk format:

# vxdisk -o alldgs list
DEVICE TYPE DISK GROUP STATUS
emcpower1s2 auto:cdsdisk tempdg-tmp02 tempdg-tmp online

So let’s make our new device cdsdisk formatted as well:

# vxdisk -o alldgs list
DEVICE TYPE DISK GROUP STATUS
emcpower6s2 auto:none – – online invalid

# /etc/vx/bin/vxdisksetup -i emcpower6 format=cdsdisk

# vxdisk -o alldgs list
DEVICE TYPE DISK GROUP STATUS
emcpower6s2 auto:cdsdisk – – online

Now to add the new disk to the disk group. Note that since the original drive was called 02, the one we are adding is 01:

# vxdg -g tempdg-tmp adddisk tempdg-tmp01=emcpower6

Now we can see the new drive in the disk group (and that it isn’t currently doing anything):

# vxprint
Disk group: tempdg-tmp

TY NAME ASSOC KSTATE LENGTH PLOFFS STATE TUTIL0 PUTIL0
dg tempdg-tmp tempdg-tmp – – – – – -

dm tempdg-tmp01 emcpower6s2 – 733975296 – – – -
dm tempdg-tmp02 emcpower1s2 – 733975296 – – – -

v templv-tmp fsgen ENABLED 733975296 – ACTIVE – -
pl templv-tmp-02 templv-tmp ENABLED 733975296 – ACTIVE – -
sd tempdg-tmp02-01 templv-tmp-02 ENABLED 733975296 0 – – -

So let’s tell the disk group to set tempdg-tmp01 up as a mirror of tempdg-tmp02:

# vxassist -g tempdg-tmp mirror templv-tmp tempdg-tmp01 &

I ran it in the background because it can take a long time depending on how big it is (about 2 hours for 350 GB). To watch the mirroring process, use vxtask. (You can also use vxprint to see what it is doing.)

# vxtask list
TASKID PTID TYPE/STATE PCT PROGRESS
177 ATCOPY/R 04.89% 0/733975296/35921920 PLXATT templv-tmp templv-tmp-01 tempdg-tmp

Once it completes, you will see your new drive setup as a mirror:

# vxprint
Disk group: tempdg-tmp

TY NAME ASSOC KSTATE LENGTH PLOFFS STATE TUTIL0 PUTIL0
dg tempdg-tmp tempdg-tmp – – – – – -

dm tempdg-tmp01 emcpower6s2 – 733975296 – – – -
dm tempdg-tmp02 emcpower1s2 – 733975296 – – – -

v templv-tmp fsgen ENABLED 733975296 – ACTIVE – -
pl templv-tmp-02 templv-tmp ENABLED 733975296 – ACTIVE – -
sd tempdg-tmp02-01 templv-tmp-02 ENABLED 733975296 0 – – -
pl templv-tmp-01 templv-tmp ENABLED 733975296 – ACTIVE – -
sd tempdg-tmp01-01 templv-tmp-01 ENABLED 733975296 0 – – -

First, setup the disk inside Veritas, in this example I’ll use the sliced format (emcpower6 is the Veritas Device name – not the PowerPath pseudo name – use vxdisk list to see the name):

# /etc/vx/bin/vxdisksetup -i emcpower6 format=sliced

Initiate the disk group by adding the disk to a new group (tempdg-tmp). The device name is emcpower6 and the name of the disk inside the disk group will be called tempdg-tmp01:

# vxdg init tempdg-tmp cds=off tempdg-tmp01=emcpower6

See how big we can make the disk group:

# vxassist -g tempdg-tmp maxsize
Maximum volume size: 733952000 (358375Mb)

Setup the disk group to the maximum size (templv-tmp is the logical volume name):

# vxassist -g tempdg-tmp -b make templv-tmp 733952000 tempdg-tmp01 layout=nostripe

At this point you can take a look at what you created with vxprint:

# vxprint
Disk group: tempdg-tmp

TY NAME ASSOC KSTATE LENGTH PLOFFS STATE TUTIL0 PUTIL0
dg tempdg-tmp tempdg-tmp – – – – – -

dm tempdg-tmp01 emcpower6s2 – 733952000 – – – -

v templv-tmp fsgen ENABLED 733952000 – ACTIVE – -
pl templv-tmp-01 templv-tmp ENABLED 733952000 – ACTIVE – -
sd tempdg-tmp01-01 templv-tmp-01 ENABLED 733952000 0 – – -

Create a file system on the logical volume, using VXFS:

# /opt/VRTS/bin/mkfs -F vxfs /dev/vx/rdsk/tempdg-tmp/templv-tmp
version 6 layout
733952000 sectors, 366976000 blocks of size 1024, log size 65536 blocks
largefiles supported

Create a mount point, mount it, see it mounted:

# mkdir /u2

# mount -F vxfs /dev/vx/dsk/tempdg-tmp/templv-tmp /u2

# df -h /u2
Filesystem size used avail capacity Mounted on
/dev/vx/dsk/tempdg-tmp/templv-tmp 350G 153M 328G 1% /u2

This setup, although not unusual, does make it more complicated. When a new LUN is connected to the host, by default, Veritas will see it first and claim the drive. This is a bad thing, because we want PowerPath to find the drive, setup the pseudo drive and encapsulate the multipathing.

Previous administrators with this setup were unable to get PowerPath to recognize the drive before Veritas without rebooting the server. They would connect the LUN and reboot the server. If Veritas detected the drive first, they would remove the disk from Veritas and reboot until PowerPath claimed the drive. This, of course, to me was unacceptable, so I have decided to document the procedure I follow for properly connecting a new LUN to a host with this configuration. (Note: If you have already got yourself in this situation, you will still need to reboot to cleanup Veritas’ hold on the drive. The alternative to rebooting would be to allocate another LUN for the host and worry about cleaning up the previous one later.)

So how do you get Veritas to not claim the drive when you connect the LUN to the host? Simple, turn off the service that discovers the drive! If you dig into the init scripts for Veritas (/etc/init.d/vxvm-startup2), you’ll see starting on line 193 the following lines:

# Start Event Source Daemon, to enable dynamic device discovery
# Empty the lock file before starting the daemon
> /etc/vx/.vxesd.lock

vxddladm start eventsource

This dynamic device discovery service must be turned off by executing the following command:

# vxddladm stop eventsource

Of course, if you don’t want dynamic discovery service running, you should also comment out line 197 so that it is never started in the first place. I have found that it is necessary to run the stop command multiple times to make sure it is off. Even if I have stopped it on a particular server before, I run it again before connecting any LUNs, just to make sure.

So here are the steps with sample output if any was generated:

Turn off Veritas dynamic discovery (IMPORTANT!!):

# vxddladm stop eventsource

Rescan your fibre connections, optionally verify that you can see the new LUN:

# /opt/QLogic_Corporation/SANsurferCLI/scli -do all rescan
Driver rescan completed on HBA instance 0.
Driver rescan completed on HBA instance 1.

Tell Solaris you’ve made some changes and let it make the proper devices:

# devfsadm -v -C

Configure PowerPath to work with the new LUN:

# /etc/powercf -q
# /etc/powermt config
# /etc/powermt save

Check that your new LUN is properly configured in PowerPath:

# /etc/powermt display dev=all
– cut –
Pseudo name=emcpower6a
CLARiiON ID=APM00012345678 [Test Storage Group]
Logical device ID=60060160A42611000A2B03D123456789 [LUN - TEMP LUN]
state=alive; policy=CLAROpt; priority=0; queued-IOs=0
Owner: default=SP A, current=SP A
===============================================
—————- Host ————— – Stor – — I/O Path – — Stats —
### HW Path I/O Paths Interf. Mode State Q-IOs Errors
===============================================
1280 pci@9/fibre-channel@2 c4t1d6s0 SP B0 active alive 0 0
1280 pci@9/fibre-channel@2 c4t2d6s0 SP A0 active alive 0 0
1281 pci@9/fibre-channel@2 c5t1d6s0 SP B1 active alive 0 0
1281 pci@9/fibre-channel@2 c5t2d6s0 SP A1 active alive 0 0
– cut –

Now we need to create a label on the drive, use the PowerPath pseudo name:

# format
Searching for disks…done

c4t1d6: configured with capacity of 349.99GB
c4t2d6: configured with capacity of 349.99GB
c5t1d6: configured with capacity of 349.99GB
c5t2d6: configured with capacity of 349.99GB
emcpower6a: configured with capacity of 349.99GB

AVAILABLE DISK SELECTIONS:
0. c1t0d0
/pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w210000008710a5b2,0
– cut –
40. emcpower6a
/pseudo/emcp@6
Specify disk (enter its number): 40
selecting emcpower6a
[disk formatted]
Disk not labeled. Label it now? yes
format> q

Get rid of the default partitions created by the label command:

# fmthard -d 0:00:00:0:0 /dev/rdsk/emcpower6a
# fmthard -d 1:00:00:0:0 /dev/rdsk/emcpower6a
# fmthard -d 6:00:00:0:0 /dev/rdsk/emcpower6a

Verify that everything looks the way you want it:

# format
Searching for disks…done

AVAILABLE DISK SELECTIONS:
0. c1t0d0
/pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w210000008710a5b2,0
– cut –
40. emcpower6a
/pseudo/emcp@6
Specify disk (enter its number): 40
selecting emcpower6a
[disk formatted]
format> verify

Primary label contents:

Volume name = < >
ascii name =
pcyl = 57344
ncyl = 57342
acyl = 2
nhead = 256
nsect = 50
Part Tag Flag Cylinders Size Blocks
0 unassigned wm 0 0 (0/0/0) 0
1 unassigned wm 0 0 (0/0/0) 0
2 backup wu 0 – 57341 349.99GB (57342/0/0) 733977600
3 unassigned wm 0 0 (0/0/0) 0
4 unassigned wm 0 0 (0/0/0) 0
5 unassigned wm 0 0 (0/0/0) 0
6 unassigned wm 0 0 (0/0/0) 0
7 unassigned wm 0 0 (0/0/0) 0

format> q

If you’ve done everything right, Veritas should not see your new drive yet. Verify:

# vxdisk -o alldgs list
DEVICE TYPE DISK GROUP STATUS
emcpower0s2 auto:sliced devdg-erp02 devdg-erp online
emcpower1s2 auto:cdsdisk testdg-erp02 testdg-erp online
emcpower2s2 auto:sliced devdg-dw02 devdg-dw online
emcpower3s2 auto:sliced devdg-tle01 devdg-tle online
emcpower4s2 auto:sliced dbadg-erp01 dbadg-erp online

Now that everything is setup right, go ahead and tell Veritas to look for new drives. Important Note: If it is not setup right at this point, Veritas will claim the raw device and it’s a pain to cleanup!

# vxdctl enable

Now, you will see the pseudo drive in the list. If it’s a raw device name, you messed up:

# vxdisk -o alldgs list
DEVICE TYPE DISK GROUP STATUS
emcpower0s2 auto:sliced devdg-erp02 devdg-erp online
emcpower1s2 auto:cdsdisk testdg-erp02 testdg-erp online
emcpower2s2 auto:sliced devdg-dw02 devdg-dw online
emcpower3s2 auto:sliced devdg-tle01 devdg-tle online
emcpower4s2 auto:sliced dbadg-erp01 dbadg-erp online
emcpower6s2 auto:none – – online invalid

You should now see new devices in /dev/vx/dmp and /dev/vx/rdmp. What’s next? Well, that’s entirely up to you. Now that you’ve got the right drive in Veritas, you can set it up anyway you want – add it to an existing disk group, mirror a disk group for some added redundancy, create a new disk group, etc. In upcoming posts, I’ll detail how to create a new disk group with this LUN and then start digging into dynamically resizing file systems while everything is still online and available to the users.

return ($n < 0) ? 0 : $n;

The biggest problem with trying to figure out what this does if you've never seen it before is that you have no idea what to search for. Do you look up "?:", no, because that actually means something else. Google Code Search has been my friend in the past on things like this, because it can handle all the special characters. I highly recommend trying it out. This time I figured out it was an old C construct, so I just went over and asked a developer what it meant.

So, for those who didn't know already, this is what it means:

($n < 0) - this is the statement that is being checked (like in an if statement).

The part between the ? and the : is what is returned if the statement is true. The part after the : is what is returned if the statement is false.

So in this case, the author was making sure that the number returned was positive. If $n was less than zero (or negative), it would return zero. If it was zero or some positive number, it just returned that number.

Now, why would you do this? Well, it would all depend on how the variable was derived in the subroutine above it. In my case, the author was using a creative method to find the total number of a specific character in a string (in this case the slash):

my $str = shift(@_);
my $n = split('/', $str) - 1;

Now, although this works, Perl will complain about it because you are clobbering the array @_ and consequently your subroutine arguments. So I ended up changing the code to use the translate command "tr", which returns the number of matches:

my $n = ($str =~ tr/\///);

Of course, now the return code is irrelevant, but not causing any harm...

for i in `ps -A | grep -v PID | awk '{print $1}'`; do echo PID:$i; pfiles $i | grep port; done

To break it down:

This gives you all the PIDs of running processes:

ps -A | grep -v PID | awk '{print $1}'

Loop through them with the for loop, and then print the PID and the open ports of that PID to the screen:

echo PID:$i; pfiles $i | grep port;

The interesting thing that I found that I wanted to post about was the corruption of the Solaris 8 superblock that was caused by mounting the partitions within Solaris 9. There were changes made in fsck on Solaris 9 that made the superblock incompatible with Solaris 8. When you boot off of a Solaris 8 partition that has been mounted by Solaris 9, you’ll get the following errors:

The / file system (/dev/rdsk/c0t2d0s0) is being checked.
/dev/rdsk/c0t2d0s0: BAD SUPERBLOCK AT BLOCK 16: BAD VALUES IN SUPER BLOCK
/dev/rdsk/c0t2d0s0: USE AN ALTERNATE SUPERBLOCK TO SUPPLY NEEDED INFORMATION;
/dev/rdsk/c0t2d0s0: e.g. fsck [-F ufs] -o b=# [special ...]
/dev/rdsk/c0t2d0s0: where # is the alternate super block. SEE fsck_ufs(1M).

/dev/rdsk/c0t2d0s0: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.

WARNING – Unable to repair the / filesystem. Run fsck
manually (fsck -F ufs /dev/rdsk/c0t2d0s0). Exit the shell when
done to continue the boot process.

Of course, the drive path may be different, etc. The solution to this is simple, run fsck and restore the superblock from one of it’s backups – in this case I restored from block 32:

fsck -F ufs -y -o b=32 /dev/rdsk/c0t2d0s0

Run this on each of the affected partitions (all the partitions that you mounted) and you are good to go.