This is attempt to dispel some of the voodoo attached with linux and SAN (EMC)Refresh Fibre or scsi cards to find new lunsMaybe one or the other (driver dependent) echo 1 > /sys/class/fc_host/host0/issue_lip echo 1 > /sys/class/fc_host/host1/issue_lip echo '- - -' >/sys/class/scsi_host/host0/scan echo '- - -' >/sys/class/scsi_host/host1/scan echo "scsi-qlascan" > /proc/scsi/qla2xxx/1 echo "scsi-qlascan" > /proc/scsi/qla2xxx/2 echo "- - -" > /sys/class/scsi_host/host1/scan echo "- - -" > /sys/class/scsi_host/host2/scan # cat /proc/scsi/scsi Attached devices: Host: scsi0 Channel: 00 Id: 00 Lun: 00 Vendor: DGC Model: LUNZ Rev: 0428 Type: Direct-Access ANSI SCSI revision: 04 Host: scsi0 Channel: 00 Id: 01 Lun: 00 Vendor: DGC Model: LUNZ Rev: 0428 Type: Direct-Access ANSI SCSI revision: 04 Host: scsi1 Channel: 00 Id: 00 Lun: 00 Vendor: DGC Model: LUNZ Rev: 0428 Type: Direct-Access ANSI SCSI revision: 04 Host: scsi1 Channel: 00 Id: 01 Lun: 00 Vendor: DGC Model: LUNZ Rev: 0428 Type: Direct-Access ANSI SCSI revision: 04 LUNZ are the lun "containers" and will be replaced when you luns are in the storage group and assigned NOW Install Naviagent and start it wait a while and rescan - if you luns are set up the LUNZ paths will replaced : # cat /proc/scsi/scsi Attached devices: Host: scsi0 Channel: 00 Id: 00 Lun: 00 Vendor: DGC Model: RAID 5 Rev: 0428 Type: Direct-Access ANSI SCSI revision: 04 Host: scsi0 Channel: 00 Id: 01 Lun: 00 Vendor: DGC Model: RAID 5 Rev: 0428 Type: Direct-Access ANSI SCSI revision: 04 Host: scsi1 Channel: 00 Id: 00 Lun: 00 Vendor: DGC Model: RAID 5 Rev: 0428 Type: Direct-Access ANSI SCSI revision: 04 Host: scsi1 Channel: 00 Id: 01 Lun: 00 Vendor: DGC Model: RAID 5 Rev: 0428 Type: Direct-Access ANSI SCSI revision: 04 now your ready to install powerpath and use them! Powerpath and Volume creationrpm -ivh EMCpower.LINUX-5.3.1.00.00-111.rhel5.x86_64.rpm Preparing... ########################################### [100%] 1:EMCpower.LINUX ########################################### [100%] All trademarks used herein are the property of their respective owners. NOTE:License registration is not required to manage the CLARiiON AX series array. [root@sabad10955 ~]# powermt config PowerPath modules are not loaded. Ensure that the modules are loaded and try again. # /etc/init.d/PowerPath start Starting PowerPath: done # powermt config Warning: license for CLARiiON storage system sup # emcpreg -add xxxx-yyy-zzz-fff-ooo-bbb-aaa-rrr 1 key(s) successfully added. # powermt config # powermt display Symmetrix logical device count=0 CLARiiON logical device count=1 Hitachi logical device count=0 Invista logical device count=0 HP xp logical device count=0 Ess logical device count=0 HP HSx logical device count=0 ============================================================================== ----- Host Bus Adapters --------- ------ I/O Paths ----- ------ Stats ------ ### HW Path Summary Total Dead IO/Sec Q-IOs Errors ============================================================================== 0 lpfc optimal 2 0 - 0 0 1 lpfc optimal 2 0 - 0 0 # powermt display dev=all Pseudo name=emcpowera CLARiiON ID=CKM00093100497 [sabad10955_test] Logical device ID=60060160450626009E32F3E3783EE011 [TEST sabad10955 testlun] state=alive; policy=BasicFailover; priority=0; queued-IOs=0 Owner: default=SP B, current=SP B Array failover mode: 1 ============================================================================== ---------------- Host --------------- - Stor - -- I/O Path - -- Stats --- ### HW Path I/O Paths Interf. Mode State Q-IOs Errors ============================================================================== 0 lpfc sda SP B1 active alive 0 0 0 lpfc sdb SP A0 active alive 0 0 1 lpfc sdc SP B0 unlic alive 0 0 1 lpfc sdd SP A1 unlic alive 0 0 # powermt set policy=co # powermt display dev=all Pseudo name=emcpowera CLARiiON ID=CKM00093100497 [sabad10955_test] Logical device ID=60060160450626009E32F3E3783EE011 [TEST sabad10955 testlun] state=alive; policy=CLAROpt; priority=0; queued-IOs=0 Owner: default=SP B, current=SP B Array failover mode: 1 ============================================================================== ---------------- Host --------------- - Stor - -- I/O Path - -- Stats --- ### HW Path I/O Paths Interf. Mode State Q-IOs Errors ============================================================================== 0 lpfc sda SP B1 active alive 0 0 0 lpfc sdb SP A0 active alive 0 0 1 lpfc sdc SP B0 active alive 0 0 1 lpfc sdd SP A1 active alive 0 0 Now prepare the lun # fdisk /dev/emcpowera Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel Building a new DOS disklabel. Changes will remain in memory only, until you decide to write them. After that, of course, the previous content won't be recoverable. The number of cylinders for this disk is set to 10240. There is nothing wrong with that, but this is larger than 1024, and could in certain setups cause problems with: 1) software that runs at boot time (e.g., old versions of LILO) 2) booting and partitioning software from other OSs (e.g., DOS FDISK, OS/2 FDISK) Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite) Command (m for help): p Disk /dev/emcpowera: 10.7 GB, 10737418240 bytes 64 heads, 32 sectors/track, 10240 cylinders Units = cylinders of 2048 * 512 = 1048576 bytes Device Boot Start End Blocks Id System Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 1 First cylinder (1-10240, default 1): Using default value 1 Last cylinder or +size or +sizeM or +sizeK (1-10240, default 10240): Using default value 10240 Command (m for help): t Selected partition 1 Hex code (type L to list codes): 8e Changed system type of partition 1 to 8e (Linux LVM) Command (m for help): p Disk /dev/emcpowera: 10.7 GB, 10737418240 bytes 64 heads, 32 sectors/track, 10240 cylinders Units = cylinders of 2048 * 512 = 1048576 bytes Device Boot Start End Blocks Id System /dev/emcpowera1 1 10240 10485744 8e Linux LVM Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. Syncing disks. set up the filter ! edit /etc/lvm/lvm.conf filter = [ "a|/dev/cciss|", "r/sd.*/", "a/emcpower.*/" ] # vgscan Reading all physical volumes. This may take a while... Found volume group "rootvg" using metadata type lvm2 # pvcreate /dev/emcpowera1 Physical volume "/dev/emcpowera1" successfully created # vgcreate datavg /dev/emcpowera1 Volume group "datavg" successfully created # lvcreate -l 100%VG -n datalv datavg Logical volume "datalv" created # mkfs.ext3 /dev/mapper/datavg-datalv mke2fs 1.39 (29-May-2006) Filesystem label= OS type: Linux Block size=4096 (log=2) Fragment size=4096 (log=2) 1310720 inodes, 2620416 blocks 131020 blocks (5.00%) reserved for the super user First data block=0 Maximum filesystem blocks=2684354560 80 block groups 32768 blocks per group, 32768 fragments per group 16384 inodes per group Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 Writing inode tables: done Creating journal (32768 blocks): done Writing superblocks and filesystem accounting information: done This filesystem will be automatically checked every 38 mounts or 180 days, whichever comes first. Use tune2fs -c or -i to override. PowerPath boot fsck painDocumented more verbose here but OS may try to fsck emcpower devices before powerpath has started change the defaults option in /etc/fstab to _netdev , this delays the mount until the network has started which should give enough time for powerpath to start. # Customer filesystems /dev/sapapp/home /home ext3 data=writeback 1 1 /dev/sapapp/sapmnt /sapmnt ext3 data=writeback 1 1 /dev/sapapp/sapmnt_PG2 /sapmnt/PG2 ext3 data=writeback 1 1 will become # Customer filesystems /dev/sapapp/home /home ext3 data=writeback,_netdev 1 1 /dev/sapapp/sapmnt /sapmnt ext3 data=writeback,_netdev 1 1 /dev/sapapp/sapmnt_PG2 /sapmnt/PG2 ext3 data=writeback,_netdev 1 1 MultiPathdViewing pathsecho "show paths" |multipathd -k cil dev dev_t pri dm_st chk_st next_check 0:0:0:2 sda 8:0 40 [active][ready] XX........ 11/40 0:0:1:2 sdb 8:16 40 [active][ready] XX........ 11/40 0:0:2:4 sdc 8:32 40 [active][ready] XX........ 11/40 0:0:3:4 sdd 8:48 40 [active][ready] XX........ 11/40 0:0:4:6 sde 8:64 40 [active][ready] XX........ 11/40 0:0:5:6 sdf 8:80 40 [active][ready] XX........ 11/40 1:0:0:1 sdg 8:96 40 [active][ready] XX........ 11/40 1:0:1:1 sdh 8:112 40 [active][ready] XX........ 11/40 1:0:2:3 sdi 8:128 40 [active][ready] XX........ 11/40 1:0:3:3 sdj 8:144 40 [active][ready] XX........ 11/40 1:0:4:5 sdk 8:160 40 [active][ready] XX........ 11/40 1:0:5:5 sdl 8:176 40 [active][ready] XX........ 11/40 2:0:0:1 sdm 8:192 40 [active][ready] XXXXXXXX.. 32/40 2:0:1:1 sdn 8:208 40 [active][ready] XXXXXXXX.. 32/40 2:0:2:3 sdo 8:224 40 [active][ready] XXXXXXXX.. 32/40 2:0:3:3 sdp 8:240 40 [active][ready] XXXXXXXX.. 32/40 2:0:4:5 sdq 65:0 40 [active][ready] XXXXXXXX.. 32/40 2:0:5:5 sdr 65:16 40 [active][ready] XXXXXXXX.. 32/40 3:0:0:2 sds 65:32 40 [active][ready] X......... 5/40 3:0:1:2 sdt 65:48 40 [active][ready] X......... 5/40 3:0:2:4 sdu 65:64 40 [active][ready] X......... 5/40 3:0:3:4 sdv 65:80 40 [active][ready] X......... 5/40 3:0:4:6 sdw 65:96 40 [active][ready] X......... 5/40 3:0:5:6 sdx 65:112 40 [active][ready] X......... 5/40 2:0:6:2 sdy 65:128 40 [failed][faulty] XXXXXXXXXX 10/10 2:0:7:2 sdz 65:144 40 [failed][faulty] XXXXXXXXXX 10/10 opps got some failures Removing PathsOpps someone switched the fibres and then switched them back , now multipathd thinks it's got more luns that it should have multipathd> hcil dev dev_t pri dm_st chk_st next_check 0:0:0:2 sda 8:0 40 [active][ready] X......... 6/40 0:0:1:2 sdb 8:16 40 [active][ready] X......... 6/40 0:0:2:4 sdc 8:32 40 [active][ready] X......... 6/40 0:0:3:4 sdd 8:48 40 [active][ready] X......... 6/40 0:0:4:6 sde 8:64 40 [active][ready] X......... 6/40 0:0:5:6 sdf 8:80 40 [active][ready] X......... 6/40 1:0:0:1 sdg 8:96 40 [active][ready] X......... 6/40 1:0:1:1 sdh 8:112 40 [active][ready] X......... 6/40 1:0:2:3 sdi 8:128 40 [active][ready] X......... 6/40 1:0:3:3 sdj 8:144 40 [active][ready] X......... 6/40 1:0:4:5 sdk 8:160 40 [active][ready] X......... 6/40 1:0:5:5 sdl 8:176 40 [active][ready] X......... 6/40 2:0:0:1 sdm 8:192 40 [active][ready] XXXXXX.... 27/40 2:0:1:1 sdn 8:208 40 [active][ready] XXXXXX.... 27/40 2:0:2:3 sdo 8:224 40 [active][ready] XXXXXX.... 27/40 2:0:3:3 sdp 8:240 40 [active][ready] XXXXXX.... 27/40 2:0:4:5 sdq 65:0 40 [active][ready] XXXXXX.... 27/40 2:0:5:5 sdr 65:16 40 [active][ready] XXXXXX.... 27/40 3:0:0:2 sds 65:32 40 [active][ready] XXXXXXXXXX 40/40 3:0:1:2 sdt 65:48 40 [active][ready] XXXXXXXXXX 40/40 3:0:2:4 sdu 65:64 40 [active][ready] XXXXXXXXXX 40/40 3:0:3:4 sdv 65:80 40 [active][ready] XXXXXXXXXX 40/40 3:0:4:6 sdw 65:96 40 [active][ready] XXXXXXXXXX 40/40 3:0:5:6 sdx 65:112 40 [active][ready] XXXXXXXXXX 40/40 2:0:6:2 sdy 65:128 40 [failed][faulty] XXXXX..... 5/10 2:0:7:2 sdz 65:144 40 [failed][faulty] XXXXX..... 5/10 2:0:8:4 sdaa 65:160 40 [failed][faulty] XXXXX..... 5/10 2:0:9:4 sdab 65:176 40 [failed][faulty] XXXXX..... 5/10 2:0:10:6 sdac 65:192 40 [failed][faulty] XXXXX..... 5/10 2:0:11:6 sdad 65:208 40 [failed][faulty] XXXXX..... 5/10 Simple , delete them : del path sdad etc. now shows : 0:0:0:2 sda 8:0 40 [active][ready] X......... 6/40 0:0:1:2 sdb 8:16 40 [active][ready] X......... 6/40 0:0:2:4 sdc 8:32 40 [active][ready] X......... 6/40 0:0:3:4 sdd 8:48 40 [active][ready] X......... 6/40 0:0:4:6 sde 8:64 40 [active][ready] X......... 6/40 0:0:5:6 sdf 8:80 40 [active][ready] X......... 6/40 1:0:0:1 sdg 8:96 40 [active][ready] X......... 6/40 1:0:1:1 sdh 8:112 40 [active][ready] X......... 6/40 1:0:2:3 sdi 8:128 40 [active][ready] X......... 6/40 1:0:3:3 sdj 8:144 40 [active][ready] X......... 6/40 1:0:4:5 sdk 8:160 40 [active][ready] X......... 6/40 1:0:5:5 sdl 8:176 40 [active][ready] X......... 6/40 2:0:0:1 sdm 8:192 40 [active][ready] XXXXXX.... 27/40 2:0:1:1 sdn 8:208 40 [active][ready] XXXXXX.... 27/40 2:0:2:3 sdo 8:224 40 [active][ready] XXXXXX.... 27/40 2:0:3:3 sdp 8:240 40 [active][ready] XXXXXX.... 27/40 2:0:4:5 sdq 65:0 40 [active][ready] XXXXXX.... 27/40 2:0:5:5 sdr 65:16 40 [active][ready] XXXXXX.... 27/40 3:0:0:2 sds 65:32 40 [active][ready] XXXXXXXXXX 40/40 3:0:1:2 sdt 65:48 40 [active][ready] XXXXXXXXXX 40/40 3:0:2:4 sdu 65:64 40 [active][ready] XXXXXXXXXX 40/40 3:0:3:4 sdv 65:80 40 [active][ready] XXXXXXXXXX 40/40 3:0:4:6 sdw 65:96 40 [active][ready] XXXXXXXXXX 40/40 3:0:5:6 sdx 65:112 40 [active][ready] XXXXXXXXXX 40/40 References http://www-01.ibm.com/support/docview.wss?uid=isg3T1011985 |
tips >