Stefan Koopmanschap's RSS

We will be taking the following file-servers down shortly for maintenance tomorrow 01-07 at approximately 1 AM PST. If one of the following file-servers is listed when clicking “Account Status” when logged into the panel this will affect you:

lewzealand, julius, wanda

It also might cause slightly elevated load during the down time on the following servers:

achilles, aero, alameda, alondra, archibald, arrow, atlantic, avalon, aviation, babyruth, barry, bazooka, beaudry, bellflower, blowpop, bonbon, bottlecap, boyd, broadway, burton, butler, catalina, centinela, century, cerritos, chex, cognac, compton, cookie, cordial, cupcake, dingdong, doheny, dorito, drumstick, dumdum, edendale, euclid, fairfax, fletcher, foothill, fundip, garth, garvey, gobstopper, goober, good, gower, grand, grappa, gumdrop, haven, heath, hillhurst, hillmont, hoover, hope, hops, hyperion, jefferson, jurupa, lankershim, larchmont, laurel, lemonhead, limbo-spunky4, linq, locust, look, lucas, ludlow, malt, manchester, mayo, melrose, mento, milliken, mills, monroe, nilla, nix, oakdale, ocean, olive, ord, pico, pikespeak, planters, plenty, pocky, poprock, pretzel, price, pringle, razzle, redondo, ritz, rockyroad, rodeo, rum, sake, saltine, schnapps, selma, skittle, snocap, soto, spree, starburst, sticky, sunset, taffy, tampa, temple, tequila, topanga, vanowen, vermouth, warhead, washington, western, whittier, whopper, wilshire, york

The downtime for each file-server should be in the range of 20-40 minutes during the maintenance.

It looks like Aquaman is having some problems with its raid array and is currently down. Our admins are working on the issue and more information is to come when it is known.

Update 04:51 AM PST: Aquaman is currently being moved over to completely new hardware. We currently trying to copy what we can from the old machine and restore whatever else from backups. Unfortunately this machine is very new and we don’t have many backups available.

More updates to come soon.

Update 05:20 AM PST: So far about 8 of the ~200 GB of data has been copied to the new server. So far it has not had any errors so things are looking good. At this rate we are looking at around 5-6 hours to transfer the rest of the data assuming everything goes well. More updates to come later.

Now that everyone is back to the salt mines for the new year, it’s time for a Ruby on Rails upgrade! 2.2.2 has been released, and we’re not seeing any major complaints about it, so it’s time to upgrade the servers!

We always give our users fair warning because upgrading Ruby on Rails always tends to break currently running applications - but everyone can avoid problems if they freeze their rails! Freezing is way easy. All you have to do is login via to your shell account, change directories into your app, and run:

rake rails:freeze:gems

I’ll be building the packages today and testing them, and if they’re not horrible, I’ll be deploying them across our network slowly as the week progresses. Everything should be done by the end of next week at the very latest!

It would appear that aquaman is currently having issues with serving up some content. Customers have been reporting issues with not being able to see JPG, PNG, GIF, CSS and JS files, amongst others. This means that sites on aquaman will appear to be missing images and theming for the time being — but two members of our admin team are working on fixing this. Updates will come when there’s news of what’s going on.

We are currently experiencing some slowness on the main DreamHost.com site as well as the web panel (where you get things done!) but are currently working on restoring the services to their full and working order! We’ll post updates as they come!

We have started the process of upgrading ruby from 1.8.5 to 1.8.7. This new version is required by some recent ruby software and has been requested several times. In our testing we didn’t discover any issues, but we have now heard about problems with some specific software applications such as RadiantCMS. If you are suddenly having problems with a ruby-based website it may be due to this upgrade and you should check for any updates to the software you are using.

In light of the recent issues with this server the admin team has decided it would be best to rebuild it on new hardware. There was a problem with the machine (but not the drives) so it either had ram or bad raid controller and rather than have our customers suffer while we pin it down we’re getting everyone onto a working server. This should be completed within the next hour.

We had an networking issues with the file server (slapper) connection within this cluster, this has been corrected but unfortunately caused some issues with apache services. Our admins have also corrected this and any sites that had experienced downtime due to this should now have their sites coming up normally, if your are still having issues please contact support.

We thought that the problems with yoshi were through, but it started acting up again earlier today and required a couple of reboots. After the last reboot it didn’t come back up at all. We moved it to new hardware twice now, so we’ve ruled out the possibility of it being a hardware issue. Our admins are in the datacenter and are working on getting this server back up as soon as they can. Please check back here for updates, as we’ll post them as sson as we have them. We’re sorry for the continued issues with this server, but hopefully we’ll be able to find the cause now.

We are consolidating several racks to free up space for new hardware (yay!), consequentially there will be some downtime for the servers involved (boo!). Here’s the list of servers that will be affected:

psmysql1871 psmysql1881 psmysql2460 psmysql2461 psmysql2623 psmysql2624 psmysql2648 psmysql2653 psmysql2660 psmysql2839 psmysql2861 psmysql2865 psmysql2882 psmysql2905 psmysql2934 psmysql3050 psmysql3053 psmysql3054 psmysql3131 psmysql3132 psmysql3159 psmysql3164 psmysql3173 psmysql3226 psmysql3229 psmysql3509 hank stormie redbull brisk milk nastia blogdu bpsites fabsilv flawless foobar fuzzyraygun ps3711 ps3853 ps5130 pssunkist stacyfayehan unadvised vanguardps webon zakidesign algrimur apolo bethndn cihan danjacobs mega mystic ps4088 ps4192 ps4316 ps5119 ps5277 ps5585 ps5965 qikserv001 seventeendots usher wesleybrasil zinkweb alignak ansible arif astore davros digitmidia freewebsites hutchtest protium rawkins tajvps technomatt trashmenagerie appletini calvin fastlane frigate judi krillsc raoul skif albasoun delphae dymweb grafia isnmweb keeeeez sonsuztasarim vejaisso brainsquirt carlosrobles cre8tivegroup filter gannon gravatt igordon izzy lorddjinn nhym pacifico ps5597 ps5902 quagmire reserver sharpie snidely artica binamedia esc ffxiii frances gordini grosbouddha idmo ims intenselighting levkov lion permie pickles sai titania andrewslaterpresents basketmen hpalumni ps3948 ps5861 ps6244 scary webapp erremedia grassroots lerftacular lingolook rocknroll sharktopus skywalker storyhole threesixty goodcow mab pollos ps1 ps4593 ps5661 ps6145 tripter trux webcoot adverstore fanboy loronix nolaflash opiniuj pocapower rightstuff rojax adhoc alpine burning cfinc fishbulb kitchensink maozveg ps4097 ps4158 ps4452 ps4800 quixotic

The majority of the servers listed are Private Servers so there are many of them on a single physical machine. Each physical machine will be down for 5-10 minutes, there are 25 machines in all so we are looking at a window of 2ish/3ish hours. Sorry for any inconveniences!

The web-server named ‘gimli’ has had a PSU failure and is currently being failed over (moved over to new hardware). No customer data is stored on this server so the move will not result in any data loss. The fail-over process typically takes approximately 1 hour. This post will be updated once the fail-over process is completed and all services on gimli are back up.

Update 06:05 AM PST: The fail-over process has been completed and all services should be restored. Please contact support if you are on gimli and having any issues with your sites.

It looks like we are seeing some packetloss on some of the machines in the ‘blingy’ cluster. This is not affecting all servers in this cluster and as far the problem has been verified on the following servers:

agnus, andrewmarc, anhlong, aspen, bayk, boom, bushido, clarke, clearelement, conker, darth, dobby, dstat, electricmousetrap, erinkel, flamenco, gimme, globalaction, grace, heradlc, jak, kewka, klungo, kyrie, log2ch, markbenj, marth, micaiah, mordecai, nero, ness, nukunuku, pichu, pikachu, pirate, pit, pmu, polgarth, ps4194, ps4233, ps4417, ps4619, ps4879, ps5121, ps5203, ps5274, ps5526, ps5776, ps6125, ps6150, richmond, ryoohki, shareit, snake, sol, sothe, strider, tavdogweb, voldo, wellbored, xdpa, zihark

Our Network Admin is currently investigating the cause. Please check here for updates.

Update 5:15 AM PST: It looks like there was a problem with routing on one of our distribution switches which was causing the problem. Routing has been moved to one of our core routers and the problem is fixed. Please contact support if you are having any issues and are on on the blingy cluster.

Customers on greene will be seeing issues as the server has crashed and requires trudging into the datacenter to give it a manual reboot. We apologize for any issues this may be causing the users affected by this, but hopefully the server should be back up shortly. More updates to come as the issue progresses.

Some users on mclean have been reporting service issues recently.  We’ve discovered that the server is currently saturating the port that it is using to communicate with the outside world.  Since we can’t have that, our admin team is in the process of upgrading the server so it handles its traffic a bit better.

The current estimate from the point person on this issue is about an hour from the time of this post.  We reserve the right to run over that estimate a bit — but if that happens, I’ll do my best to keep the folks suffering from this issue updated.

Update - 1:33 PM PST: Wow! The upgrade went way faster than was expected and I’ve already been given the all-clear. If you’re on mclean and continue to see issues, please contact support and we’ll do our best to make sure everything is taken care of.

The webserver yoshi is exhibiting symptoms that hint at impending disk failure, the server is being taken offline and is being moved to new hardware. Estimated downtime is 30 minutes.

We are currently experiencing some networking issues with one of the file servers in our ’spacey’ cluster. If you’re on one of the servers below and are having problems with your website, this should be resolved soon. One of our admins is on his way to the datacenter to work on the problems with this file server and fully restore service.

7up amp aquafina arizona barqs bawls boba brisk cactuscooler calpico cappuccino caprisun chai clamato codered coffee coke crush crystallight culligan dads dasani dietrite drpepper eggnog espresso evian fanta fiji frappe fresca fuze gatorade geyser gingerale grapico hansens hi-c hiball horchata ibc inko izze jolt koolaid lipton martinellis milk minutemaid monster mountaindew mrpibb mug musclemilk nastia nehi nesbitt nestea niagara odwalla oj orangebang ovaltine pellegrino pepsi perrier plentymag pom powerade propel punch redbull refresco riptide rockstar rootbeer schweppes seagull seltzer servidorgva shasta silk simpledollar slice slim smoothie snapple sobe sparkletts sprite squirt stewarts sunkist sunnyd tab tampico tang tea tejava tizer tonic tropicana v8 voss welchs wishserver yoohoo

There was a security issue with the server tang, and to be safe, the server is moving to new hardware.

Digdug’s boot disk is pushing up daisies. We’re going to replace it as fast as possible, but it will take us a couple of hours to get it back up and running. We apologize for the inconvenience.

There’s a brand new version of WordPress available in the advanced one-click installer — and boy is it pretty.  The WP team has spent a lot of time and put quite a bit in the way of resources towards making things both look and work nicer.  This release doesn’t contain any security fixes that we’re aware of, but it definitely feels quite a bit faster.  Well, at least in our rather unscientific tests it does.  Your mileage may vary, of course…

Give your installs an upgrade when you’ve got some free time.  And if something goes totally haywire, our support team is here and totally happy to help.

P.S. Our web panel team assures us that the easy one-click installer will be upgraded shortly.  Keep an eye on this post for updates.

Schilling has lost a NIC, so right now we’re transferring everything to new hardware. This shouldn’t take longer than an hour. We apologize for the inconvenience.