A Day at HP: SuperDome & HP-UX Live

07 Oct '09 - 17:14 by benr

In an effort to reach out to "new media", HP had a tech-day event at the Cupertino Campus for a variety of invited bloggers. For some reason I was among them, because inviting a Sun zealot to an HP-UX love-fest is always a recipe for a good time. I wore a Sun Microsystems shirt, of course, just to keep things clear. Some of the other attendees included David Adams of OS News, Saurabh Dubey of ActiveWin, and Andy McCaskey of SDR News.

To make a long story short, the event celebrated the 10 Year Anniversary of SuperDome and the 25 Year Anniversary of HP-UX... both, they say, are still on top and going strong.

For my non-enterprise readers you may or may not have heard or know much about HP SuperDome, so here's the scoop. SuperDome is HP's "High End" for the "Integrity" line of servers, which can utilize either Intel Itanium or PA RISC, although the latter has gone the way of the dodo. While competitive offerings, even from Sun, have come and gone during the 10 Years of SuperDome I readily admit that it remains a viable platform due to the simplicity and modularity of its architecture of which HP should be, and is, very proud. So proud in fact that the model they showed off to us was a prototype unit that is still in use to day, simply upgraded to newer processors and used for performance testing and analysis.

SuperDome's architecture is simple. Power at the bottom, cooling at the top with additional cooling elements in the middle. The core of the system is a powerful backplane which connects "cell boards" to each other and to IO expansion. Nothing fancy, just big ass cables from point A to B. Up to 2 SuperDome chassis (or "frames" depending on which audience you address) can be melded together by wiring the backplanes together which is why you see some single cabnet SuperDome's and some that are 2 side-by-side joined at the hip, as it were. Each "Cell Board" contains memory slots and CPU sockets. At present they're offering Quad-Core Itanium, and each chassis supports up to 8 Cell Boards, so you get up to 32 sockets (128 cores) per chassis, but as I said you can cable 2 chassis together increasing the total "single" system capability to 64 sockets.

One of my big nit picks was that when HP SuperDome first arrived it was direct competition to the Sun "StarFire" Enterprise 10,000 ("E10K") which had double the capabilities of the HP! Hardly competition. But time has proven HP out. While the E10K was an excellent HPC Super Computer (in its day) for very large SMP requirements, the SuperDome instead focused on being carved up using partitioning. This is a two fold strategy, first using "nPars" to "hard partition" cell boards into individual systems and then to use the HP-UX vPars to "virtualize" instances of HP-UX within those nPars. The result was a platform that was truly about flexibility and centralization as opposed to being the biggest and baddest gun in the west. Lets not forget that both E10K and E15K both had, I think, the best partitioning technology available, but the markets they appealed to were very different. When you combine SuperDome's simplistic architecture along with its marketing focus of consolidation and flexibility you get a solution with far more staying power.

Of course, the trouble in my mind is that HP-UX is a steaming pile of crap.... but they assure me this is not so. As though little had changed in the last 10 years, HP still seem to think of the real competition being IBM mainframe installations. They hardly acknowledge Sun as competition because there is so little focus by Sun in that competitive space. So the OS war they are fighting isn't HP-UX vs AIX vs Solaris, but rather HP-UX vs IBM z/OS.

HP-UX 11i v3 Update 5 (and you thought Solaris naming was stupid!) just released. The roadmap current charts out to HP-UX 11i v5... so if your wanting for HP-UX 12 you'll wait longer than for Solaris 11.

I had several conversations with important persons of interest regarding Solaris and the jist was generally the same as the market at large. They find DTrace to be annoying, because while people "need it" they don't use it. Furthermore they claim that DTrace like facilities have existed within the HP-UX kernel for years but they never considered it of interest to end customers and therefore appeared to come late the the party. With techonologies like SMF they agreed that the end-user excitement wasn't as great as they expected, and hands down everyone saw ZFS as the really big ticket win in Solaris.

But.... HP-UX vs Solaris is little more than a side-note. In the enterprise space in which they see themselves as relevant OS wars don't exist. CFO's, they think, make the decisions and its not about whether Solaris is better than HP-UX or AIX but whether the hardware provides value and the supplier offers excellent service... the OS is, in that case, simply a glue layer. At least, that's the vibe they put off.

And whats more, I consider Itanium to have run its course... but HP are quite confident that its still the best processor available and are even minorly annoyed at the idea of putting Intel Xeon processors in the SuperDome cell boards. In short, not happening.

The event was fun, I was glad to attend. It was a throwback to the world pre-X86, when servers were big and RAS was the standard. I congratulate HP on producing a fantastic product with so much staying power and wish them continued success in the future.


- - C O M M E N T S - -

Humorous. Make no mistake, people buy Solaris because it’s cheap. It’s easily the worst commercial UNIX in terms of both user-space and administration.

HP-UX on the other hand is very nice in both areas.

(One of many examples) How long has HP-UX had kernel-space, bootable LVM? SVM is a real good effort though. Really good effort. “E for effort” on that one.

Sam (Email) - 07 October '09 - 18:57

I find it interesting that, as far as I can determine, the Superdome is the next generation of the V Class machines, which were basically rebadged Convex Exemplars that HP bought the rights to when the company went belly-up in 1994. Similar to how a lot of the bits and pieces in the Sun high-end machines came from Cray SuperServers and Fujitsu.

I could be wrong, but the last I heard, most of the ccNUMA technology wasn’t HP’s originally.

petrilli (Email) (URL) - 07 October '09 - 19:14

> Make no mistake, people buy Solaris because it’s cheap.
I disagree, they brought in to Solaris because of ISV support! More apps on Sun SPARC than HPUX/AIX/IRIS/DOMAIN OS.

I agree LVM & SAM are nice tools, so is AIX’s smit.
(Veritas/Symantec would could not been a success without Solaris poor disk management and needs for VxFS)

Sun has never been the fastest or the cheapest, but just a good all-rounder back in its day.

Andy Paton (Email) (URL) - 07 October '09 - 20:24

@petrilli: I’m not sure about HP’s legacy, but there is no doubt that Sun’s greatest mid-range/high-end servers came from Dr. Cray in the mid/late 90’s.

@Sam: I would disagree strongly… HP-UX administration is crap personified. SVM, though, I won’t make excuses for; it was and is horrible.

benr - 07 October '09 - 20:26

Really? Ever actually done any of it?

Advantages over Solaris:


  • No slices.

  • No stupid rules about “don’t use slice 2!”

  • Very robust package management with very robust remote installation of packages

  • You can actually USE Hp’s provide patch tools and don’t have to rely on third party tools, unlike solaris where Sun actually suggests you use a third party tool



  • HP’s (free) ‘PV Links’ vs Solaris’ nothing

    Those are the biggies. What’s so crap about it, from your POV?

    Sam (Email) - 07 October '09 - 20:38

    @Sam: I shouldn’t start a pissing match… I haven’t actually used HP-UX in 5 years or more. I have a C110 that I should fire up again with a modern release of HP-UX (assuming I can find one). But here is my recollection of irritations with HP-UX:


    • HP-UX tended toward BSD-isms rather than SysV

    • SAM is required too often, doing things on the CLI was exceedingly painful. I submit that if I was a full-time HP-UX admin for a couple years I’d pick it up, but similar to AIX the CLI was unwelcoming. SMIT/SAM shouldn’t be so essential, imho.

    • Working with Software Depots was a pita.



    • Licensing. One of the chief reasons HP in general sucks is that it seems there is a constant battle to both understand all the required options and licenses and managing them. Solaris has no such concerns.

      Maybe some of the things with HP-UX have been figured out or improved over time, but I tend to doubt it. Its certainly not a hackers platform. Maybe if my first experience with it was in a classroom it’d be different, but it isn’t. LVM blows chunks and things only get decent when you use VxVM/VxFS.

      I’ve requested from HP some way for me to get my hands on HP-UX 11i v3 to really be more objective but so far I don’t have a system to play on. If someone wants to give me ssh access to their box I’d appreciate an opportunity to hack around on it.

      benr - 07 October '09 - 22:08

      Sam:

      slices – actually with ZFS you can forget about them altogether in most installations.

      packaging – have you seen pkg(5)?

      patches – I don’t know but Sun provided smpatch works for me good. Also keep in mind that with pkg the concept of patches is basically gone. It is already the case in Open Solaris.

      PV Links – well, MPxIO is available for free in Solaris for many years. First it was an add-on but since Solaris 10 (or probably before) it became a part of the OS. I’ve been using it from jbods, to midrange and high-end disk arrays from multiple vendors both on SPARC and x86 – just does its job.

      However lets play the game – some Solaris advantages over HP-UX:

      – Solaris runs on x86 hardware, including HP x86 servers, HP-UX doesn’t
      – Solaris is open source in terms of Open Solaris
      – Open Solaris Cluster – not only it is for free but it is also open source, also thanks to Solaris running on cheap x86 servers (including from HP) one can deploy clustered configuration in environments where it was not possible before
      – Solaris ZFS – well, there isn’t really anything in HP-UX to compete here
      – Solaris DTrace -while there are couple of useful tools in HP-UX (so there have been in Solaris forever) there is nothing which is even close to DTrace, and yes DTrace solves real problems in real production systems in case some HP-UX folks doubt it
      – Solaris Crossbow – is there anything close in HP-UX so one can basically accomplish virtual wire? I honestly don’t know and doubt it… btw: crossbow is for free and open source :)
      – Solaris xVM (Xen based virtualization) so one can virtualize Windows, Linux, *BSD, Solaris and other operating systems with all their software on x86. Can HP-UX do it?
      – SSD support in ZFS – this is so big that it is worth mentioning as a separate feature to ZFS. Does HP-UX offer anything which allows to use SSD (or nvram) to trasparently provide read and write cache to all applications?

      ok, that’s good enough I guess :)

      Robert Milkowski (Email) - 07 October '09 - 22:15

      HP-UX is NOT “a steaming pile of crap”, it’s one of the most high-performance, stable, secure UNIX System V systems out there.

      And, he’s very, VERY similar to Solaris. If you look at the manual pages of HP-UX, they’re almost identical in content and structure (minus Solaris and HP-UX specific parts).

      HP-UX is very fast. And has excellent compilers. Which are very similar to Sun Studio compilers, and even have the same facilities (like profile feedback, and ld’s $ORIGIN).

      HP-UX is very much a hackers’ platform in the sense that compiling, linking, and packaging all that freeware, and building a RunTime Platform identical to Solaris is a challenge, and a fun challenge at that!

      And by the way, SD-UX, HP-UX’s software management subsystem, is STILL LIGHTYEARS AHEAD of either System V packaging OR IPS. IPS is not even CLOSE when compared to SD-UX in terms of capabilities, or functionality offered.

      Something to think about.

      So please don’t slam HP-UX. It’s not HP-UX’s fault that management at hp is greedy and incompetent, and won’t open source a truly amazing and excellent UNIX.

      UX-admin (Email) - 08 October '09 - 00:59

      Actually the old SunAP software is a much closer equivalent to pvlinks (with mpxio you don’t have to play the ‘which path is defined first’ game to get load balancing).

      Also, HP’s lvm only became tolerable when it licensed veritas and started using that in place of the old lvm, and even then, at least with SVM you weren’t nickled and dimed on features (oh want to grow online? that’s extra? oh want to mirror? that’s extra).

      Why do I have to have RPC running to install software (and get a horribly non-obvious error when it’s not and you try)?

      I can remember for years having to pay 3x the cost of an enterprise storage array to get an HP badge on it because HP-UX couldn’t support more than 8 luns per target (so you paid a 200% markup over the list price of the array from Hitachi to get a small firmware patch that would remap luns on the array to different targets on the HP-UX box).

      Never saw anything like live upgrade (in fact upgrading was usually a crapshoot, better to just install clean to a new disk).

      Jason (Email) - 08 October '09 - 02:57

      My biggest concern with HP-UX from an admin perspective is that I tend to use a lot of open source which is usually not packaged for HP-UX and often doesn’t compile either.

      To a certain degree this is also true of Solaris, but we’ve had sunfreeware and blastwave for years and with opensolaris things are starting to look really good. Nothing really compared to that in the HP-UX community as far as I’m aware.

      Also I find it annoying I can’t simply install HP-UX in a VM or a spare PC to toy around with.

      Dennis Krul (Email) (URL) - 08 October '09 - 07:31

      Yea the whole BSD vs SySV thing is just taste and over the years I’ve learned that neither tastes any worse than the other.

      In the 10 or so years I’ve been an HP-UX/Solaris/Linux admin, I’ve only used Sam for the rare case of a reboot-required kernel change, and that’s almost always just after install. Set it and forget it.

      Software depots can be a pain, but they’re Functional and do tons of things that pkg(5) can not do. I’d much rather live with the pain of learning a (perhaps overly) complex system than one that’s so simple I can’t do what I need to do. Solaris package management is still RPM before yum. Even with yum, RPM is still the worst Linux package management and Solaris isn’t even up to its state of the art.

      I completely agree that HP is a greedy POS company that has stupid licensing.

      You frequently say things like “LVM blows chunks” but you don’t explain why. I’ve had maybe 5 LVM problems in the history of my career and I was able to recover from every single one of them. Not even close to true of SVM. ZFS is far too immature to even think about putting my data on it.

      Sam (Email) - 08 October '09 - 13:24

      Slices: Can you boot ZFS yet? If so, how long has that been an option? How robust is it? We played with it earlier this year. We presented a SAN volume down two paths and put it in a ZFS hoojamawiki and when we removed both paths the system crashed. Do the same thing under HP-UX and you’ll make syslog work harder, but the system will stay up and recover when the LUNs recover.

      So if you’re not willing to trust the layer-mangling infant from the same folks who can’t get basic mirroring right (in SVM land, there’s still patches from this year for mirror consistency), you’re stuck with slices in 2009. But yea, keep defending it and pointing to your newborn ZFS.

      Yea the patching thing… I’m glad you’ve found something that works. I’m not sure how you sorted through the litany of different suggestions from Sun. I recently watched an surreal and bizarre video made by Sun about the Solaris 10 rec’ed strategy for patching…. LiveUpgrade or something… it was completely insane and suggested breaking your mirror for a path back to sanity should things go wrong. Awesome.

      sam (Email) - 08 October '09 - 13:25

      MPXIO has been around for a couple of years and it still stinks compared to the much older and easier to use Pv Links. We recently transitioned a machine from a Sun SAN to an EMC and after spending hours on the phone with support we couldn’t get MPXIO to front-end both SANs at the same time. In HP-UX, I’d just add both paths to the volume group and be done. No editing undocumented files. No support calls that go nowhere. Just ‘lvextend’.

      x86 hardware still sucks.

      sam (Email) - 08 October '09 - 13:27

      I’ve already said it in other ways, but I’d use LVM over ZFS any day of the week.

      DTrace is the big Solaris win.

      sam (Email) - 08 October '09 - 13:27

      Crossbow/Xen. For my part, virtualization is largely management enter tainment, but HP offers both hardware and software virtualization.

      Since I don’t know/care about it I probably shouldn’t comment, but I’m pretty sure there’s windows for Itanium and I know there’s Linux for it, so the answer to your question is likely yes, but I don’t know. Also, why on earth would you want to run any of those operating systems on real UNIX hardware? Or is this a feature in search of a problem?

      I also don’t know the answer to your SSD question, but again I’d guess that they do.

      sam (Email) - 08 October '09 - 13:30

      Sorry for all of the posts. I thought it was a length thing, since the error message provided no useful info and just said “spam isn’t appreciated.” (I can tell you work for Sun!)

      Turns out you can’t say enter tain ment as one word.

      sam (Email) - 08 October '09 - 13:31

      Sam:

      Unless something has changed recently Ben does not work for Sun neither do I.

      Booting from ZFS – of course it works and it is supported. It was first provided in Open Solaris and then later in Solaris 10.

      When both sides of mirror disappeared by default ZFS will panic the box. However you can change it (for example: zpool set failmode=wait rpool) so it will wait forever for disks to re-appear.

      ZFS and reliability – well I know MANY deployments on zfs from several dozen gigabytes to petabytes in size, in clustered environments and not.

      MPxIO – as often is with technology we could exchange many anecdotes when a given technology failed. For me in most cases MPxIO just works – you enable it with one single command and it just works. Actually I haven’t got any issue with it for some years now and I have deployed many different storage disk arrays in a meantime.

      patching – well, smpatch comes with the OS by default. Anyway the concept is gone in Open Solaris anyway.

      LiveUpgrade – obviously you misunderstood. The way it works is that if you root-fs is running on ZFS then if you want to upgrade your operating system the live upgrade will create a clone of your system, update the clone while the current OS and applications are running and once it’s done and it is convenient for you you can reboot into upgraded OS. If something is broken you can reboot back into previous version which hasn’t been modified at all. If you do not use ZFS then you need to have extra device (mirrored or not) for the OS so usually a second mirror of disks or a mirror on different slices. It is actually amazing technology which makes sysadmin life so much easier.

      Windows on Itanium… why would I bother? If I need to consolidate Windows it is mainly for its applications which very rarely are running on Itanium. Then I don’t see a point wasting money on Itanium… but that’s different story.

      pkg – well it does actually provide most of the goodies of yum and some more. have you actually used it? If I want to install dhcp I will just do: pkg install dhcp, and it will automatically download it, with all dependencies if needed, and get them installed. How easier is it on HP-UX? Then if I want to upgrade entire OS instead of upgrading a package-by-package I can do so called imageupdate when a clone of the OS will be created and updated so I can reboot into it once it’s done. Something is wrong and I don’t have to rollback all updates – I can just boot into unchanged previous environment. Anything like this on HP-UX?

      Don’t get me wrong – HP-UX is a solid UNIX platform. The problem is that it is not open source, it is not for free (??), it doesn’t run on x86 hardware (which is great for many things), it has crippled licensing policy as others mentioned, and it seems to stay behind (nothing to compete with zfs, dtrace, smf, crossbow, ...). It runs only on Itanium (PA-RISC doesn’t count anymore)- and while it is a good CPU on a paper it hasn’t picked up in the market. Then it has an issue Solaris used to have – problem with ready-to use open source applications. Thanks to Open Solaris it has already changed dramatically and is moving in a good direction, HP-UX… what?

      The other problem is that HP-UX is mostly being perceived as a legacy system – if you are already on HP-UX platform you might consider staying on it, but otherwise why bother? Like it or not but it doesn’t have a so called “cool factor” and doesn’t run on popular hardware in a market so except for a shrinking group of HP-UX admins nobody cares to take even interests in it, unless you happen to came across couple of HP-UX boxes from time to time as I do…

      Robert Milkowski (Email) - 08 October '09 - 21:15

      Good coverage there Ben.
      I had the (mis)fortune of having to work on a project (@ a pharma giant) which was converting en-masse to HP/HP-UX from a stable and well performing Sun/Solaris base.

      My biggest grouse with HP, Itanic and HP-UX was their tendency to try and nickel-and-dime the customer for everything.

      For eg:

      nPAR and vPAR is all fine and dandy, but you have to pay $$$ per core to utilize the technology.

      Oh…HPUX has Resource management tools, but you have to pay $$$$ per core to utilize the technology.

      I wasn’t greatly impressed with their mid-range and entry-level Integrity servers (seemed very compaq’ish for some reason…)

      Practically everything that you get from Sun/Solaris at no cost (besides what you pay for buying the hardware) had an associated cost with HP.

      It was hard to see where the ROI was going to come from, given we had a rock-solid Sun/Solaris shop (still Solaris 9 at that) and none of the extra costs that HP was pulling out of it’s bag.

      Not to mention the abysmally clunkly HP ServiceGuard clusterware (shudders…it’s not even really clusterware, it’s just a bunch of scripts thrown together, to put it simplistically)...

      I’m in the process of setting up a virtualized service delivery platform (in another shop btw) right now on Sun’s sturdy T5440s, using Sun Cluster 3.2 and Containers at a fraction of the cost we were looking at with HP’s solution (despite a blanket 40% discount to boot).

      I agree with HP’s observation about CIOs and CFO’s not caring about the technology that drives their business…they don’t. But that shouldn’t prevent technologists from recommending the best product for the job.

      The two nice things about HPUX is the way their device naming is so consistent (unlike Solaris). It is actually easy to figure out where a device is located physically in the chassis using ioscan.
      The other is Ignite (which Sun developed JET to catch up with).

      Cheers!

      implicate_order (Email) (URL) - 09 October '09 - 17:43

      Having worked for large financial companies and doing consulting work for Fortune 500 companies, I can say without a doubt that HP-UX is on the decline more so than Solaris. Over the past 5 years, HP-UX and other UNIX variants, like IRIX have been targeted for retirement in large enterprises. Literally, I’ve seen HP-UX gear tossed in the trash or sitting in data center loading docks for recycling more so than any other UNIX gear. The real battle in the enterprise UNIX space is between Solaris and AIX. Right now IBM is pressing hard on Sun customers during this whole Oracle/Sun acquisition which is taking a long time. Unfortunately, many Sun customers are holding off on purchases until the deal is done, which is hurting Sun big time.

      Speaking from experience, HP really should have taken Tru64 UNIX and Alpha on as their main platform. While there are some nice features in HP-UX, it hasn’t really advanced in years. Most of the advancements have been on the hardware side, not the OS. HP copped out on advancing HP-UX years ago. Look at how they have focused on using Symantec/Veritas products to replace their LVM and HA needs.

      Tru64 had some great features such as Adv-FS, which is the closest thing I’ve worked with to ZFS. It’s already a completely clean 64-bit platform. Has great management tools, clustering (TruCluster), compilers, security, etc. At least by early 2000’s standards. Unfortunately, HP couldn’t see the value in Tru64 and scrapped all its plans to merge the two. Not to mention the Alpha processor was way ahead of the pack, EV8 should have blown everything else away and introduced multi-core/multi-threaded CPU’s years ahead of the others. But HP shelved a lot of the great technology there and sold of the rest to Intel.

      Honestly, if HP had positioned the Tru64 and Alpha technology properly, things might be different. But it’s not a big surprise with HP. They tend to buy products and kill them off because they threaten their long-standing products.

      I think we can all be thankful that Oracle is buy Sun and not IBM, which would gut and kill off Sun’s products. Hopefully Oracle will do a better job of monetizing Sun’s products. I don’t think Sun has had any major mis-steps technologically, just marketing and business wise. When you walk into big shops and want to know where the money is being made, it’s on Solaris/SPARC and AIX/Power.

      Octave Orgeron (Email) - 10 October '09 - 18:11

      Your article is very interesting, I have introduced a lot of friends look at this article, the content of the articles there will be a lot of attractive people to appreciate, I have to thank you such an article.

      supra shoes (Email) (URL) - 14 October '09 - 02:37

      I use to be a strong advocate for HP and HP-UX. I’ve used both Sun and HP technical workstations and servers for over 20 years. HP, with very few exceptions, always seemed faster and just seemed to handle loading better. The engineering of the systems at all levels just oozed quality and reliability.

      But then there is HP-UX. Love it for enterprise tools like Oracle, transaction processing systems, etc. Hate it for the overt lack of Opensource tools and utilities. You know, the ones I use or wish to use in my day to day job. I think think this is what will doom (has doomed?) them in the midrange computing market and despite what anyone says, the high-end (telco/finance/cloud) isn’t enough to maintain the level of innovation they need to stay completive over the next several decades.

      At least Opensolaris will keep the core OS and technology in the hands of the enthusiasts at home and will hopefully keep the user base healthy. HP has no such path and thus I predict many tears for them. Oh and IBM/AIX should just die already.

      Bart (Email) - 27 October '09 - 17:07

      I have to agree with the statement “I would use LVM over ZFS any day”. The combination of HP’s OnlineJFS (their top version of the VxFS) along with LVM is a hell of a combination. ZFS is a memory hog, no multi-pathing, not exactly well-poised for a scale-up environment, and ‘self-healing’ is capability is done by storage arrays. Any filesystem that touts its coolness by saying “128 bits” must have been designed by a marketing person, not an engineer. Compression is better off done by the array. Not to say ZFS is terrible—as it may very well prove to be generally as capable/reliable as HP’s VxFS/LVM combo, it’s just not there yet.

      kdoubleo (Email) - 05 November '09 - 17:20

      Does HP-UX still use the UNIX init system? Both AIX and Solaris use a service management facility. AIX I think always has, Solaris since 10.

      init is a bane of UNIX availability. I do not know how HP-UX can claim to be an alternative to the mainframe if it still uses init.

      Honestly, AIX is far more mainframe like than any other UNIX, in part due to mainframe inspired hardware features like memory keys on POWER6/AIX6.

      Someone asked if Superdome was based on a Convex design. Yes, it was the follow-on to the Exemplar machines. HP took the hardware and ported HP-UX to the platform. No doubt, Superdome is a pretty solid platform. But as it is based on Itanium, it is irrelevant. Today, SGI announced Altix is now powered by Xeon. How long until Superdome shifts to Xeon? What UNIX will the Xeon Superdome run?

      Mark - 17 November '09 - 02:52

      The sale of fake Gucci shoes is now a common event. But, the online shopping platform adds a fresh layer of deception. Individuals buying the fashionable shoes do not understand that the products they are buying are fake.

      gucci women shoes (Email) (URL) - 25 January '10 - 04:32

      fashionable shoes do not understand that the products they are buying are fake.

      javon (Email) (URL) - 25 January '10 - 09:19

      Ed Hardy Hats were later launched with much fanfare. Today, they are perceived as status symbols and extremely popular among the young and old, and even celebrities.
      ed hardy

      ed hardy (Email) (URL) - 03 March '10 - 08:32

      Follow your dreams, you can reach your goals.

      jju (Email) (URL) - 10 March '10 - 02:21

      interesting and professional site!

      links of london (Email) (URL) - 10 March '10 - 08:47

      This selection of colored jewelry is critical. The same time, business wear, based on the clever mix of jewelry, business wear can play a subtle change in the effect of shape; here is the most important jewelry,. The length of necklace,
      links of london

      joejonas (Email) (URL) - 22 April '10 - 07:51

      wir verkaufen[url=[[http://www.gamesavor.com/CDK/wow-cd-ke..]]]] wow key [/url] bei Gamesavor

      xiao (Email) - 04 May '10 - 04:00

      P90x .It really is not expensive if you factor in the cost

      of a gym membership,P90x workout . The cost for P90X is

      about three months of a paid gym membership but you get to

      keep the program foreverP90x . You can try many of the

      online sites, but it will be the same as buying from the

      company or a Beachbody Coach. Make sure you are getting

      original DVD’s. People are selling copies all over. The

      problem is how long will they last, P90x workout ,and you

      truly need the exercise and nutrition guide to even follow

      the program. You can go to any site or you can go to

      www.p90xmall.com and click on products. P90x dvd You can

      order directly from the site,P90x dvd.

      p90x (Email) (URL) - 05 May '10 - 05:35

      Lots of specialists tell that loans aid a lot of people to live the way they want, because they can feel free to buy needed goods. Moreover, different banks offer student loan for different persons.

      GoldenLidia (Email) (URL) - 08 May '10 - 06:16

      hi there, thanks for your great post, it help me so much! now i wonder if you linke the mbt shoes from my site: the url is [[http://www.thesuitshoes.com]]

      mbt shoes (Email) (URL) - 12 May '10 - 06:51

      A good website recommend you:www.offerreplicawatches.com.They sell Chanel Watches, Replica BRM Watches,offer Cartier Watches, Copy Ebel Watches, Replica Hublot Watches, Fake Tag Heuer Watches, wholesale Richard Mille Watches.

      They also sell some very fashion watches, such as Montblanc Watches for sale,Chopard Watches, Dior Watches and Gucci Watches. Rolex watches are hot sell. If u are Armani Watches and Breitling Watches fan. You can still look at Omega Watches and Longines Watches, many new arrivals watches just come here.You can have a try on 58338312391273913. :[url=[[http://www.offerreplicawatches.com]www..]]].

      Replica Chanel Watches,Dvd Boxes (Email) (URL) - 12 May '10 - 07:30

      I love [[http://www.shoppingmbtshoes.com/mbt-wo..]]

      air jordan shoes (Email) (URL) - 12 May '10 - 13:26

      Dell inspiron 9400 Battery [[http://www.adapterlist.com/dell/inspir..]]

      laptop battery (Email) (URL) - 15 May '10 - 06:11

      Toshiba pa3534u-1brs Battery [[http://www.globallaptopbattery.co.uk/t..]]

      laptop batteries (Email) (URL) - 15 May '10 - 06:17

      The newest version of air max 90 are available now.Welcome to see our special nike air max 2009 and air max. They are your must-have nike air max 2010|air max 97|cheap air max.

      air max (Email) (URL) - 20 May '10 - 06:33

      AIX is far more mainframe like than any other UNIX, in part due to mainframe inspired hardware features like memory keys on POWER6/AIX6.

      car dvd (Email) (URL) - 23 May '10 - 09:49

      Buy Nike Air Max 90 Shoes just $45-55 USD in[[http://www.iofferitems.com,]], 40-70% Off. Cheap Air Max 90 Shoes, Free Shipping! Buy Air Max 90 Now!

      nike air max 90 shoes (Email) (URL) - 28 May '10 - 02:56

      Buy Nike Air Max 90 Shoes just $45-55 USD in [[http://www.iofferitems.com,]], 40-70% Off. Cheap Air Max 90 Shoes, Free Shipping! Buy Air Max 90 Now!

      nike air max 90 shoes (Email) (URL) - 28 May '10 - 05:56

      I like writing, hurriedly, not that you are not
      [[http://www.woe-from.com]]
      [[http://www.louis-vuitton-outlet-store...]]

      louis vuitton (Email) (URL) - 01 June '10 - 02:25

      [[http://www.sxhgz.com]] : nike shoes

      puma shoes (Email) (URL) - 21 June '10 - 01:54

      [[http://www.mbtshoeslatest.com]]
      [[http://www.nikeairmaxshoe.com]]

      Nike air max shoe (Email) (URL) - 24 June '10 - 07:49

      The athletic shoes which makes using this technology may the very good local constable convoy mobilization body, Air Max 2009.
      [[http://www.allhotshoes.com/]]

      air max shoes (URL) - 09 July '10 - 05:31

      [[http://www.cheapjerseyschina.com]]
      [[http://www.cheapjerseyschina.com]]

      tanglnog (Email) (URL) - 03 August '10 - 02:01

      very cool article ,thanks for sharing the article!like my cool stuff .very useful.
      uCoolStuff is the leading China wholesaler for [[http://www.ucoolstuff.com]] cool stuff [[http://www.ucoolstuff.com]] cool gifts, unusual gadgets and other unique gift ideas. We provide the very latest cool stuff and cool gifts for you

      cool stuff (Email) (URL) - 25 August '10 - 08:25

      excellent article , I added you to my [[http://www.china-wholesale-directory.c..]] Top China Wholesalers category.. thanks for sharing the article!

      China Wholesale Directory (Email) (URL) - 25 August '10 - 08:33

      [[http://www.macvobconverter.net]] VOB Converter mac is 2010 the hottest software which is very popular among mac users who want to convert VOB to other video formats on mac like MP4, AVI, WMV, MPG, MPEG, MOV, TS, FLV, etc. With super fast conversion speed and high output quality, it can ensure you a fluent process to convert VOB mac. And in the same way, this Mac VOB Converter can convert various video formats to VOB for enjoyment.
      [[http://www.macvobconverter.net/mac_vob..]]

      VOB Converter mac (Email) (URL) - 30 August '10 - 05:47

      11zoe
      cheap ugg boots[[http://www.cheapuggbootssale.co.uk]]
      Timberland boots sale[[http://www.timberlandboots4sale.co.uk]]

      cheap ugg boots (Email) (URL) - 13 December '10 - 03:12

      Personal information





      Remember your information?
      Comment

      Small print: All html tags except <b> and <i> will be removed from your comment. You can make links by just typing the url or mail-address.


^M