tips‎ > ‎

Linux and SAN devices

This is attempt to dispel some of the voodoo attached with linux and SAN (EMC)



Refresh Fibre or scsi cards to find new luns 


Maybe one or the other (driver dependent) 

echo 1 > /sys/class/fc_host/host0/issue_lip
echo 1 > /sys/class/fc_host/host1/issue_lip 
echo '- - -' >/sys/class/scsi_host/host0/scan 
echo '- - -' >/sys/class/scsi_host/host1/scan 

echo "scsi-qlascan" > /proc/scsi/qla2xxx/1
echo "scsi-qlascan" > /proc/scsi/qla2xxx/2 
echo "- - -" > /sys/class/scsi_host/host1/scan 
echo "- - -" > /sys/class/scsi_host/host2/scan 


# cat /proc/scsi/scsi 
Attached devices:
Host: scsi0 Channel: 00 Id: 00 Lun: 00
  Vendor: DGC      Model: LUNZ             Rev: 0428
  Type:   Direct-Access                    ANSI SCSI revision: 04
Host: scsi0 Channel: 00 Id: 01 Lun: 00
  Vendor: DGC      Model: LUNZ             Rev: 0428
  Type:   Direct-Access                    ANSI SCSI revision: 04
Host: scsi1 Channel: 00 Id: 00 Lun: 00
  Vendor: DGC      Model: LUNZ             Rev: 0428
  Type:   Direct-Access                    ANSI SCSI revision: 04
Host: scsi1 Channel: 00 Id: 01 Lun: 00
  Vendor: DGC      Model: LUNZ             Rev: 0428
  Type:   Direct-Access                    ANSI SCSI revision: 04



LUNZ are the lun "containers" and will be replaced when you luns are in the storage group and assigned

NOW Install Naviagent and start it
wait a while and rescan - if you luns are set up the LUNZ paths will replaced :

# cat /proc/scsi/scsi
Attached devices:

Host: scsi0 Channel: 00 Id: 00 Lun: 00
  Vendor: DGC      Model: RAID 5           Rev: 0428
  Type:   Direct-Access                    ANSI SCSI revision: 04
Host: scsi0 Channel: 00 Id: 01 Lun: 00
  Vendor: DGC      Model: RAID 5           Rev: 0428
  Type:   Direct-Access                    ANSI SCSI revision: 04
Host: scsi1 Channel: 00 Id: 00 Lun: 00
  Vendor: DGC      Model: RAID 5           Rev: 0428
  Type:   Direct-Access                    ANSI SCSI revision: 04
Host: scsi1 Channel: 00 Id: 01 Lun: 00
  Vendor: DGC      Model: RAID 5           Rev: 0428
  Type:   Direct-Access                    ANSI SCSI revision: 04


now your ready to install powerpath and use them!

Powerpath and Volume creation 

rpm -ivh EMCpower.LINUX-5.3.1.00.00-111.rhel5.x86_64.rpm
Preparing...                ########################################### [100%]
   1:EMCpower.LINUX         ########################################### [100%]
All trademarks used herein are the property of their respective owners.
NOTE:License registration is not required to manage the CLARiiON AX series array.
[root@sabad10955 ~]# powermt config
PowerPath modules are not loaded.
Ensure that the modules are loaded and try again. 
# /etc/init.d/PowerPath start
Starting PowerPath:  done

# powermt config
Warning: license for CLARiiON storage system sup
# emcpreg -add xxxx-yyy-zzz-fff-ooo-bbb-aaa-rrr 
1 key(s) successfully added.

# powermt config

# powermt display
Symmetrix logical device count=0
CLARiiON logical device count=1
Hitachi logical device count=0
Invista logical device count=0
HP xp logical device count=0
Ess logical device count=0
HP HSx logical device count=0
==============================================================================
----- Host Bus Adapters ---------  ------ I/O Paths -----  ------ Stats ------
###  HW Path                       Summary   Total   Dead  IO/Sec Q-IOs Errors
==============================================================================
   0 lpfc                          optimal       2      0       -     0      0
   1 lpfc                          optimal       2      0       -     0      0


# powermt display dev=all
Pseudo name=emcpowera
CLARiiON ID=CKM00093100497 [sabad10955_test]
Logical device ID=60060160450626009E32F3E3783EE011 [TEST sabad10955 testlun]
state=alive; policy=BasicFailover; priority=0; queued-IOs=0
Owner: default=SP B, current=SP B Array failover mode: 1
==============================================================================
---------------- Host ---------------   - Stor -   -- I/O Path -  -- Stats ---
###  HW Path                I/O Paths    Interf.   Mode    State  Q-IOs Errors
==============================================================================
   0 lpfc                      sda       SP B1     active  alive      0      0
   0 lpfc                      sdb       SP A0     active  alive      0      0
   1 lpfc                      sdc       SP B0     unlic   alive      0      0
   1 lpfc                      sdd       SP A1     unlic   alive      0      0



# powermt set policy=co
# powermt display dev=all
Pseudo name=emcpowera
CLARiiON ID=CKM00093100497 [sabad10955_test]
Logical device ID=60060160450626009E32F3E3783EE011 [TEST sabad10955 testlun]
state=alive; policy=CLAROpt; priority=0; queued-IOs=0
Owner: default=SP B, current=SP B Array failover mode: 1
==============================================================================
---------------- Host ---------------   - Stor -   -- I/O Path -  -- Stats ---
###  HW Path                I/O Paths    Interf.   Mode    State  Q-IOs Errors
==============================================================================
   0 lpfc                      sda       SP B1     active  alive      0      0
   0 lpfc                      sdb       SP A0     active  alive      0      0
   1 lpfc                      sdc       SP B0     active  alive      0      0
   1 lpfc                      sdd       SP A1     active  alive      0      0



Now prepare the lun

# fdisk /dev/emcpowera
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.


The number of cylinders for this disk is set to 10240.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
   (e.g., DOS FDISK, OS/2 FDISK)
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): p

Disk /dev/emcpowera: 10.7 GB, 10737418240 bytes
64 heads, 32 sectors/track, 10240 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes

         Device Boot      Start         End      Blocks   Id  System

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-10240, default 1): 
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-10240, default 10240): 
Using default value 10240

Command (m for help): t
Selected partition 1
Hex code (type L to list codes): 8e
Changed system type of partition 1 to 8e (Linux LVM)

Command (m for help): p

Disk /dev/emcpowera: 10.7 GB, 10737418240 bytes
64 heads, 32 sectors/track, 10240 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes

         Device Boot      Start         End      Blocks   Id  System
/dev/emcpowera1               1       10240    10485744   8e  Linux LVM

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

set up the filter !
edit /etc/lvm/lvm.conf
filter = [ "a|/dev/cciss|", "r/sd.*/", "a/emcpower.*/" ]


# vgscan
Reading all physical volumes.  This may take a while...
  Found volume group "rootvg" using metadata type lvm2

# pvcreate /dev/emcpowera1
  Physical volume "/dev/emcpowera1" successfully created

# vgcreate datavg /dev/emcpowera1 
  Volume group "datavg" successfully created
# lvcreate -l 100%VG -n datalv datavg
  Logical volume "datalv" created

# mkfs.ext3 /dev/mapper/datavg-datalv 
mke2fs 1.39 (29-May-2006)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
1310720 inodes, 2620416 blocks
131020 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=2684354560
80 block groups
32768 blocks per group, 32768 fragments per group
16384 inodes per group
Superblock backups stored on blocks: 
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632

Writing inode tables: done                            
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 38 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.



PowerPath boot fsck pain

Documented more verbose here but OS may try to fsck emcpower devices before powerpath has started

change the defaults option in /etc/fstab to _netdev , this delays the mount until the network has started which should give enough time for powerpath to start.

# Customer filesystems
/dev/sapapp/home        /home           ext3 data=writeback     1 1
/dev/sapapp/sapmnt              /sapmnt         ext3 data=writeback     1 1
/dev/sapapp/sapmnt_PG2          /sapmnt/PG2     ext3 data=writeback     1 1

will become

# Customer filesystems
/dev/sapapp/home        /home           ext3 data=writeback,_netdev     1 1
/dev/sapapp/sapmnt              /sapmnt         ext3 data=writeback,_netdev     1 1
/dev/sapapp/sapmnt_PG2          /sapmnt/PG2     ext3 data=writeback,_netdev     1 1





MultiPathd

Viewing paths


echo "show paths" |multipathd -k 

cil    dev dev_t  pri dm_st   chk_st   next_check      
0:0:0:2 sda 8:0    40  [active][ready]  XX........ 11/40
0:0:1:2 sdb 8:16   40  [active][ready]  XX........ 11/40
0:0:2:4 sdc 8:32   40  [active][ready]  XX........ 11/40
0:0:3:4 sdd 8:48   40  [active][ready]  XX........ 11/40
0:0:4:6 sde 8:64   40  [active][ready]  XX........ 11/40
0:0:5:6 sdf 8:80   40  [active][ready]  XX........ 11/40
1:0:0:1 sdg 8:96   40  [active][ready]  XX........ 11/40
1:0:1:1 sdh 8:112  40  [active][ready]  XX........ 11/40
1:0:2:3 sdi 8:128  40  [active][ready]  XX........ 11/40
1:0:3:3 sdj 8:144  40  [active][ready]  XX........ 11/40
1:0:4:5 sdk 8:160  40  [active][ready]  XX........ 11/40
1:0:5:5 sdl 8:176  40  [active][ready]  XX........ 11/40
2:0:0:1 sdm 8:192  40  [active][ready]  XXXXXXXX.. 32/40
2:0:1:1 sdn 8:208  40  [active][ready]  XXXXXXXX.. 32/40
2:0:2:3 sdo 8:224  40  [active][ready]  XXXXXXXX.. 32/40
2:0:3:3 sdp 8:240  40  [active][ready]  XXXXXXXX.. 32/40
2:0:4:5 sdq 65:0   40  [active][ready]  XXXXXXXX.. 32/40
2:0:5:5 sdr 65:16  40  [active][ready]  XXXXXXXX.. 32/40
3:0:0:2 sds 65:32  40  [active][ready]  X......... 5/40 
3:0:1:2 sdt 65:48  40  [active][ready]  X......... 5/40 
3:0:2:4 sdu 65:64  40  [active][ready]  X......... 5/40 
3:0:3:4 sdv 65:80  40  [active][ready]  X......... 5/40 
3:0:4:6 sdw 65:96  40  [active][ready]  X......... 5/40 
3:0:5:6 sdx 65:112 40  [active][ready]  X......... 5/40 
2:0:6:2 sdy 65:128 40  [failed][faulty] XXXXXXXXXX 10/10
2:0:7:2 sdz 65:144 40  [failed][faulty] XXXXXXXXXX 10/10


opps got some failures

Removing Paths

Opps someone switched the fibres and then switched them back , now multipathd thinks it's got more luns that it should have


multipathd> hcil     dev  dev_t  pri dm_st   chk_st   next_check      
0:0:0:2  sda  8:0    40  [active][ready]  X......... 6/40 
0:0:1:2  sdb  8:16   40  [active][ready]  X......... 6/40 
0:0:2:4  sdc  8:32   40  [active][ready]  X......... 6/40 
0:0:3:4  sdd  8:48   40  [active][ready]  X......... 6/40 
0:0:4:6  sde  8:64   40  [active][ready]  X......... 6/40 
0:0:5:6  sdf  8:80   40  [active][ready]  X......... 6/40 
1:0:0:1  sdg  8:96   40  [active][ready]  X......... 6/40 
1:0:1:1  sdh  8:112  40  [active][ready]  X......... 6/40 
1:0:2:3  sdi  8:128  40  [active][ready]  X......... 6/40 
1:0:3:3  sdj  8:144  40  [active][ready]  X......... 6/40 
1:0:4:5  sdk  8:160  40  [active][ready]  X......... 6/40 
1:0:5:5  sdl  8:176  40  [active][ready]  X......... 6/40 
2:0:0:1  sdm  8:192  40  [active][ready]  XXXXXX.... 27/40
2:0:1:1  sdn  8:208  40  [active][ready]  XXXXXX.... 27/40
2:0:2:3  sdo  8:224  40  [active][ready]  XXXXXX.... 27/40
2:0:3:3  sdp  8:240  40  [active][ready]  XXXXXX.... 27/40
2:0:4:5  sdq  65:0   40  [active][ready]  XXXXXX.... 27/40
2:0:5:5  sdr  65:16  40  [active][ready]  XXXXXX.... 27/40
3:0:0:2  sds  65:32  40  [active][ready]  XXXXXXXXXX 40/40
3:0:1:2  sdt  65:48  40  [active][ready]  XXXXXXXXXX 40/40
3:0:2:4  sdu  65:64  40  [active][ready]  XXXXXXXXXX 40/40
3:0:3:4  sdv  65:80  40  [active][ready]  XXXXXXXXXX 40/40
3:0:4:6  sdw  65:96  40  [active][ready]  XXXXXXXXXX 40/40
3:0:5:6  sdx  65:112 40  [active][ready]  XXXXXXXXXX 40/40
2:0:6:2  sdy  65:128 40  [failed][faulty] XXXXX..... 5/10 
2:0:7:2  sdz  65:144 40  [failed][faulty] XXXXX..... 5/10 
2:0:8:4  sdaa 65:160 40  [failed][faulty] XXXXX..... 5/10 
2:0:9:4  sdab 65:176 40  [failed][faulty] XXXXX..... 5/10 
2:0:10:6 sdac 65:192 40  [failed][faulty] XXXXX..... 5/10 
2:0:11:6 sdad 65:208 40  [failed][faulty] XXXXX..... 5/10 


Simple , delete them :

del path sdad etc.

now shows :
0:0:0:2  sda  8:0    40  [active][ready]  X......... 6/40 
0:0:1:2  sdb  8:16   40  [active][ready]  X......... 6/40 
0:0:2:4  sdc  8:32   40  [active][ready]  X......... 6/40 
0:0:3:4  sdd  8:48   40  [active][ready]  X......... 6/40 
0:0:4:6  sde  8:64   40  [active][ready]  X......... 6/40 
0:0:5:6  sdf  8:80   40  [active][ready]  X......... 6/40 
1:0:0:1  sdg  8:96   40  [active][ready]  X......... 6/40 
1:0:1:1  sdh  8:112  40  [active][ready]  X......... 6/40 
1:0:2:3  sdi  8:128  40  [active][ready]  X......... 6/40 
1:0:3:3  sdj  8:144  40  [active][ready]  X......... 6/40 
1:0:4:5  sdk  8:160  40  [active][ready]  X......... 6/40 
1:0:5:5  sdl  8:176  40  [active][ready]  X......... 6/40 
2:0:0:1  sdm  8:192  40  [active][ready]  XXXXXX.... 27/40
2:0:1:1  sdn  8:208  40  [active][ready]  XXXXXX.... 27/40
2:0:2:3  sdo  8:224  40  [active][ready]  XXXXXX.... 27/40
2:0:3:3  sdp  8:240  40  [active][ready]  XXXXXX.... 27/40
2:0:4:5  sdq  65:0   40  [active][ready]  XXXXXX.... 27/40
2:0:5:5  sdr  65:16  40  [active][ready]  XXXXXX.... 27/40
3:0:0:2  sds  65:32  40  [active][ready]  XXXXXXXXXX 40/40
3:0:1:2  sdt  65:48  40  [active][ready]  XXXXXXXXXX 40/40
3:0:2:4  sdu  65:64  40  [active][ready]  XXXXXXXXXX 40/40
3:0:3:4  sdv  65:80  40  [active][ready]  XXXXXXXXXX 40/40
3:0:4:6  sdw  65:96  40  [active][ready]  XXXXXXXXXX 40/40
3:0:5:6  sdx  65:112 40  [active][ready]  XXXXXXXXXX 40/40



References
http://www-01.ibm.com/support/docview.wss?uid=isg3T1011985






Comments