SAN admins here. ask your questions.

Discussion in 'OT Technology' started by trouphaz, Apr 28, 2008.

  1. trouphaz

    trouphaz New Member

    Joined:
    Sep 22, 2003
    Messages:
    2,666
    Likes Received:
    0
    So, based on mike99TA's thread, I figured we have enough SAN admins in this forum to answer any questions that the rest of you may have about SAN or storage in general.

    I have experience with HP Storageworks XP12000, XP1024 and XP512 (which equate to HDS USP/Tagmastore, 9980 and 9960), HDS 9500 series, AMS500 and 9200, EMC DMX3 and Clariion CX300. I've worked with all kinds of Brocade switches from the 2Gbit 8-port switches up to 4Gbit 48000 director class. I also worked with EMC's Celerra NAS device and ADIC i2000 tape libraries (others as well, but most aren't really worth mentioning).

    Anyway, I don't know how successful this will be, but ask away. I figure between Mike99TA, crontab and I, we could provide some info. Maybe even figure out an EDU that would be worth writing based on questions.
     
  2. Mike99TA

    Mike99TA I don't have anything clever to put here right now

    Joined:
    Oct 3, 2001
    Messages:
    4,553
    Likes Received:
    0
    Location:
    Greenville, SC
    My SAN experience is mostly limited to HP and brocade products. As with trouphaz, a lot of stuff from the 2Gbit 8 ports, all the way up to 4Gbit 48000 directors with >200 ports, as well as multiprotocol routers over long distance links using dark fiber (100+km).

    HP Arrays mostly XP12000, XP1024, XP512, and EVA class (4000/6000/6100) arrays, but I've gotten into alot of the grittier features of all of them, including business copies, continuous access across 100+km links, etc. Some NetApp NAS experience, and lots of experience with HP/Quantum Enterprise Tape Libraries and Sepaton Virtual Tape Libraries.

    Anyway, I'm guessing there aren't too many people here interested in this kind of stuff (even if it does pay a boatload of money), but I'll try to answer any questions that come up if I can.
     
  3. midcalbrew

    midcalbrew OT Supporter

    Joined:
    Jun 14, 2006
    Messages:
    2,376
    Likes Received:
    0
    Location:
    MidCal
    It's nice to know I've got someone to go crying to when I have problems ... an EMC CX3-40 was just kind of dumped on our department of three and none of us had any experience at all. We just added a ADIC (Dell-branded, as is the Clarion) tape library into the mix. It's always fun to learn new things.
     
  4. JayC71

    JayC71 Guest

    I've configured and worked with EMC FC4700, CX500/310/320 SANs and Brocade 5000 switches. Maybe not the most knowledgeable here, but I may be able to help.
     
  5. AbortionSurvivor

    AbortionSurvivor Active Member

    Joined:
    Jun 5, 2002
    Messages:
    3,016
    Likes Received:
    0
    Location:
    Nor. Cal
    IN. I'd give you a huge cookie if you can figure this out out.

    Ok. Here's the setup.
    Two Netapp FAS3040's in a cluster running 10GB Ethernet iSCSI.
    Switch is a Foundry 8port 10GbE. Both copper (CX4 cables) and fiber connections are used.

    There are four 10GbE fiber connections (2 fiber per Netapp Filer).

    Here's the problem.
    Lets name the Filers as FilerA & FilerB.

    When both FilerA and FilerB are online, the four 10GbE interfaces are not reachable via ping. In this case, of the four total interfaces, two are available for 10GbE iSCSI connection. -1 interface each are up and running I/O.

    IF I reboot both filers, all four interfaces come online and I am able to ping all four interfaces. iSCSI can also connect to each interface (all four of them).

    After about 20 mins of light I/O. Both filers's interfaces go down and not able to ping anything.
    Oh, if I reboot FilerA first, FilerB never takes over the cluster. FilerB usually times out until FilerA finishes rebooting and takes ownership of it's designated interfaces.

    I have a slight feeling that Ops wired the fiber connections incorrectly.
    That and Netapp got back to us and told us there's a bug in their 10GbE iSCSI code.
    We don't know the specifics or if it's software iSCSI or a TOE card issue.

    Oh, and the hosts servers?
    I can't say cus it's secret Vmware update builds. =P
     
    Last edited: Apr 30, 2008
  6. AbortionSurvivor

    AbortionSurvivor Active Member

    Joined:
    Jun 5, 2002
    Messages:
    3,016
    Likes Received:
    0
    Location:
    Nor. Cal
    Last thing i should mention.
    If i take FilerA offline and reboot FilerB, then FilerB is happy and works.
     
  7. crontab

    crontab (uid = 0)

    Joined:
    Nov 14, 2000
    Messages:
    23,441
    Likes Received:
    12
    We have a FAS3040 cluster and a FAS6030 cluster, but no iscsi or fcp., nor 10g.

    Do these two 10G iscsi interfaces have a vip per filer?

    What if you just configure/enable a single 10g iscsi link per filer. Does the cluster behave?

    The cluster failover on filerA is looking for which interface on filerB? We have channeled/trunked gig interfaces on the filers and it's talks to each other's vip.
     
  8. crontab

    crontab (uid = 0)

    Joined:
    Nov 14, 2000
    Messages:
    23,441
    Likes Received:
    12
    Hmm, Veritas SF guys.

    Any you using SF 5.0 MP1 on Sol10? We have SF 5.0 on RHEL4 and it works fine. We have two solaris boxes with sol10 and sf 5.0 mp1 and vea takes at least 30 minutes to start. Currently have a open case with Sun backlined to Veritas, but just wondering if anyone else has seen this.

    We need the gui for the people who don't know veritas that well. I use the gui myself when i create a single volume and fs in one shot.
     
  9. Mike99TA

    Mike99TA I don't have anything clever to put here right now

    Joined:
    Oct 3, 2001
    Messages:
    4,553
    Likes Received:
    0
    Location:
    Greenville, SC
    Heh, was just going to say - we don't use the GUI here and everyone that manages the *nix servers knows how to do anything they need to do from the command line with SF.

    Also I've noticed some weird bugs in the vea gui even in 4.1 MP4 (trying to grow a volume by using the "MAX" button tries to overallocate, at least on our servers).

    We just keep a document with a list of common SF commands that anyone might need to use and a small description of how to use it, and tell everyone to stay away from the GUI.

    Of course I realize this doesn't help you at all. We also don't use SF 5.0 at all (not available for Linux through our licensing portal yet).
     
  10. Mikey D

    Mikey D New Member

    Joined:
    Oct 13, 2002
    Messages:
    6,666
    Likes Received:
    0
    Location:
    Royal Oak, MI
    I know I'm not as experienced as a lot of the guys in this thread already but I'm primarily an admin for backup and restore purposes.

    Experience with Solaris and AIX as well as some limited experience with HPUX, Linux releases, etc...

    We primarily run IBM Tivoli Storage Manager on Solaris and AIX platforms here in some large environments. ACSLS controlled 9310's and SL 8500's, tape drives vary from 9840A/B/C 9940A/B T10000A 2Gb and 4Gb models as well as the variety of LTO drives. We primarily use SUN/STK FLX series or 65xx series arrarys for TSM uses. Switch gear we have a variety of Cisco SAN gear, 9509/9513's 2Gb/4Gb as well as 92xx and 91xx series.

    Good deal of experience with Emulex HBA's and their associated LPFC drivers as well as SUN's Leadville drivers.

    We just setup an EMC DL4206 Virtual Tape library here with active engine fail over, pretty cool stuff to see and really good performance. We're using this with our Networker solution at the moment and are planning on bringing one in hopefully for our TSM systems in a year or two.

    My storage admins have been starting to train me with all the Cisco SAN gear and the disk subsystems so eventually I'll hopefully get some work with our Symmetix and Clariion systems.
     
  11. crontab

    crontab (uid = 0)

    Joined:
    Nov 14, 2000
    Messages:
    23,441
    Likes Received:
    12
    Ah acsls... I've never used a more rock solid application. That was the only component in my backup infrastructure that did not break ever. I was upset when I had to patch and upgrade. STK was like you must upgrade... I'm like why? It works perfectly, never hiccups, never gets confused horribly in oversubscription. How about a patch to fix these SN6K port hangs? Or give me something to replace these wretched SN2000 hubs.
     
  12. trouphaz

    trouphaz New Member

    Joined:
    Sep 22, 2003
    Messages:
    2,666
    Likes Received:
    0
    man, i have no experience with NetApp Filers. are they any good? all i heard was that they are great when you first deploy them, but that their WAFL filesystem degrades significantly over time as the filesystem fills up and data gets spread all over the place.
     
  13. crontab

    crontab (uid = 0)

    Joined:
    Nov 14, 2000
    Messages:
    23,441
    Likes Received:
    12
    We only use them for NFS/CIFS, so we do use their WAFL fs. We have about ~60TB RAID6. A rough estimate would be 70% utilized overall, some volumes are fuller than others. This has been deployed before my time, but I was charge of the hardware upgrades from one node to two clustered nodes.

    Performance-wise, we are pleased with it's performance. It's just a NFS/CIFS share of home directory type data so we're not expecting any insane amount of iops like the HPC department expects with their fanned NFS servers.

    I'd say my employer is happy with netapp as a NAS product as am I. It's much better than the HP NAS Executor product that I've used that was actually a windows storage OS behind the scene's and choked hard if I farted near it.

    I can't say much about their FC or iscsi performance, but wafl isn't involved. I also don't expect the performance to match or come close to the higher end Hitachi's due to it's architecture. It's definitely a mid-tier to high-tier level storage imho. Maybe closer to EMC storage or some variant of a Xiotech SAN.
     
  14. crontab

    crontab (uid = 0)

    Joined:
    Nov 14, 2000
    Messages:
    23,441
    Likes Received:
    12
    Hmm, On the topic of data getting spread all over the place. Do you not recommend that behavior? Many newer SAN are spreading it's data across all spindles. The EVA does this with their vraid within the disk groups created. Some SAN's spread it across every single spindle.

    Do you prefer this method or the way the Hitachi and some EMC san's do it, with X disk RAID groups with ldev's/volumes carved out?
     
  15. crontab

    crontab (uid = 0)

    Joined:
    Nov 14, 2000
    Messages:
    23,441
    Likes Received:
    12
    More questions.

    What are your thoughts on deduplication? One of our vendors just sent out an alert where a volume panic can occur on a deduplicated volume, but no data is lost. I brought this topic up with my personal IT crew that I chat with, ex-employees or people I've worked with and we all agreed that it's not ready for prime time. It was hot like a year ago for backup d2d2t and some storage vendors, but now you hear nothing of it. No sales guy is pushing it. Anyone use it with success?

    How about iscsi. Do you feel that 10g iscsi will replace 4G, 8G fc? If a dedicated 10G iscsi san infrastructure is cheaper than the fastest fc SAN and infrastructure, why go fc? Windows support for iscsi is definitely there, i don't know about UNIX. I haven't played with it.
     
  16. trouphaz

    trouphaz New Member

    Joined:
    Sep 22, 2003
    Messages:
    2,666
    Likes Received:
    0
    well... my preference was how i had it done on my arrays (HP XP, HDS Thunder and EMC DMX). each lun was pulled from a single raid group, but successive luns came from different groups. then, i would stick with standard sizes and combine them into LUSEs or metas to present to the host. so, the individual lun would be spread across a 7+1 RAID5 and then the LUSE/meta would be striped across some number of luns. each lun would add an additional 8 drives to the spindle count. as an example, we used 15Gb hypers on our DMX3 which we would combine in groups of 4 generally and present a host with a single large "disk." so, a 300Gb "drive" that the host had access to could be spread across 160 disks in the array.


    for NAS, i had only used the EMC Celerra. i was the lead storage admin and someone else just went ahead and bought a couple without involving me. luckily, they worked pretty well and we ran into no issues. :) we used it for NFS, CIFS and iSCSI. it was a standalone unit with its own Clariion storage behind it instead of front-ending SAN storage. our NAS was relagated to tier 2 storage, so nothing that went on it needed high performance.
     
  17. crontab

    crontab (uid = 0)

    Joined:
    Nov 14, 2000
    Messages:
    23,441
    Likes Received:
    12
    So the fact that you are spreading data across multiple spindles with expansion features on the SAN and on the host end, why not let the SAN natively do itself? On these newer type SAN's....

    Basically, you have nothing against data everywhere I take it...
     
  18. trouphaz

    trouphaz New Member

    Joined:
    Sep 22, 2003
    Messages:
    2,666
    Likes Received:
    0
    i've never been wowed by deduplication. i think it is something that looks great on paper, but once you realize the investment of time to really get much out if it, it no longer seems so great. it isn't like you can just drop it into your backup environment and all of a sudden cut back on disk/tape usage by 50%. you have to rearrange your backups to work best with deduplication for it to work well.

    as far as iscsi, i could see that taking off though i still see a place for FC. to properly deploy iscsi in an environment that could beat or match FC in performance, you'd end up spending just as much to build a dedicated network with dedicated network interfaces on the host. i see iscsi taking over on the low to mid range while FC goes from mid range to high end.
     
  19. trouphaz

    trouphaz New Member

    Joined:
    Sep 22, 2003
    Messages:
    2,666
    Likes Received:
    0
    not really. no problems with spreading data around. i just remember using an HP va7410. that thing would auto balance itself on the fly to "optimize" how data was laid out. it might not be exactly what you are referring to, but that thing was a piece of shit.
     
  20. Mike99TA

    Mike99TA I don't have anything clever to put here right now

    Joined:
    Oct 3, 2001
    Messages:
    4,553
    Likes Received:
    0
    Location:
    Greenville, SC
    I prefer the LUN carving we get on our HP XP - this way we can choose exactly how we want to spread our data.

    We have our XPs diced into mostly 20GB LUNs. We don't use LUSE luns very often (only for exchange actually, never for Unix), and when we allocate multiple LUNs for a server we spread them out evenly over a set number of Controller Units.

    The way the EVA does it is fine for lower I/O apps but it doesn't work very well in a large enterprise. With the XP we can force certain apps to be spread in different ways then other apps (ie: apps that use extreme amounts of I/O we can segregate, etc).
     

Share This Page