Mobile version (Display Regular Site)

Skip to: Navigation | Content | Sidebar | Footer


Weblog Entry

Unsettling

June 05, 2007

Like watching a random stranger going through your personal belongings in the customs line, it’s a fairly unsettling feeling knowing that someone else has been able to make changes in your own markup under your nose. Here’s what happened, best as I can tell.

A bit more than a week ago, tired and jetlagged, I received an email that my site had a bunch of hidden links at the footer of every page. Links that looked suspiciously like spam. My immediate reaction was incredulity, and sure enough when I looked, no such thing existed. I wrote off that complaint as someone getting caught in a frame, or browsing with a spyware-infested PC, and forgot about it.

Then a few days later, I received a similar message from someone else. One was easy to dismiss, two was curious. I viewed source again, but this time, there they were. A series of a few dozen links to otherwise innocuous sites, with obviously spammy drug-related keywords. All wrapped up in a <u style=display:none>. Presumably the invalid code is how people were stumbling across them.

Example Spam Links

Figure: Screenshot of the server version of a page with the spam links in question (with absolute paths to included files intentionally obscured).

After some digging, here are the bits I’ve managed to piece together:

  • The links the first person saw were different from those the second person and myself saw afterward. That they changed, combined with the fact they only appeared to be visible at certain times of the day, leads me to believe that the links were inserted multiple times. It seems like someone had a script doing this automatically, because…
  • The links were inserted directly into the files sitting on the server. There was a clear pattern: any file named index.php or index.html had them appended to the end, after everything else in the file. The script started from the very root of my user account on my web server, and went four levels deep. Since I host multiple sites on the same server, every single site on this server was affected.

That left two questions. One, how did this happen? And two, how do I fix it? The second question was easy enough: manually recurse through every file affected and delete the links one by one. A huge time-waster, and it killed an entire afternoon last week, but it worked. That bought me some temporary relief, but given the already-observed recurrence of these links, I had to dig deeper and get to the root of the problem.

After a few emails with my host, they were pointing the finger at unspecified PHP security holes. There’s probably no way to tell specifically how this happened, they say; the log files would be a nightmare to sift through.

Since this site does use PHP in a few spots, I only had a few guesses about where I should be looking to fix the problem:

  • First thing I did was change my password, naturally.
  • Then I upgraded to the latest version of Movable Type. Previously I was on 3.2, now I’m up to 3.35
  • And then, armed with tutorials and books I started guessing at where my PHP might be in error. This felt like stabbing in the dark, especially because the user-facing scripts I’ve written already had a bunch of data-checking built in. But there are a few other low impact uses of PHP along the way which didn’t necessarily seem to need the extra scrutiny at the time; those I’ve gone back and tightened up quite a bit.

Cumulatively, that may have worked. A week later I haven’t seen another recurrence. I don’t feel out of the woods quite yet, but I’m crossing my fingers.

So let this be a cautionary tale for anyone using any sort of dynamic code or CMS on their site: security matters. Stay on top of updates. Change passwords regularly. All that stuff they tell you to do, it’s for good reason.

Update: From the comments, it appears that numerous people are experiencing this problem. It also appears that there are two common threads amongst those who have suffered it: they host with Dreamhost, and/or they use Wordpress. It’s not exclusive; some who host with Dreamhost don’t use Wordpress, and some host elsewhere but do use Wordpress.

For those who host with Dreamhost: I received a confirmation email from them at 8:27pm PST on June 5th that yes indeed, something in the neighbourhood of 3,500 FTP accounts have been compromised. If you’re on Dreamhost, time to change all your passwords, and check your sites for mischief. They haven’t posted an official announcement anywhere so I can’t link to it, however they recommend keeping an eye on their status blog for updates as they come in.

So in the end, it looks like there was nothing I could have done to prevent this one aside from host elsewhere. Just goes to show how important backup is, local and remote. Here’s how to use rsync to back up your entire web server, from a few years back.


1
Simon G. says:
June 05, 11h

I’ve seen a few examples of this type of thing pop up on forums I watch in the last few weeks. It generally seems to be old versions of wordpress that’s being attacked, and this is the first I’ve heard of MT being exploited like this.

It’s quite clever - this means that the spammer/cracker is going to get quite a page rank boost for quite a long time.

–Simon

2
June 05, 11h

And one more tip: avoid PHP like the plague. :)

3
Dave S. says:
June 05, 11h

“It’s quite clever - this means that the spammer/cracker is going to get quite a page rank boost for quite a long time.”

If they’d linked to actual spam sites, yes. But they linked to harmless third party sites in this case. (It’s possible the sites in questions had been similarly hacked, but when I checked, the pages linked simply didn’t exist.)

Hard to determine what the point was, other than maybe a probe to see if it’s possible.

So far no adverse hit to my own PageRank, I think, but that’s another possible outcome: Google assassination.

4
June 05, 11h

Yikes! This kind of spamming is far more serious than anything I have previously seen happen to a blog. This makes comment spam look like child’s play. Time for a quick security checkup on all of my sites.

5
Josh Dura says:
June 05, 11h

Dave, I had the exact same thing happen this week. It happened to every single one of my sites (my blog, a few WoW sites I host, and a Poker site). I even have a PunBB forum I host, and they edited one of the forum descriptions to host an iframe that loaded a file from one of their sites that popped up different pages…

I have no clue how this happened, but went through the exact same process you did. I have come to no other conclusion other than maybe Dreamhost’s servers were hacked in some way. Now, I am no server admin, so this might be a total shot in the dark, but I know every one of my forums, blogs, etc has been updated to the most secure release. What I do know, is the day that I changed my control panel password on Dreamhost, was the day that this stopped happening.

6
Dave S. says:
June 05, 11h

@Josh - very interesting. I certainly haven’t ruled out the possibility that it was a Dreamhost security lapse. And to be honest, it felt like they were a little quick in using PHP to shrug off responsibility.

Also there’s the fact that this was root-level; I’m sure some PHP exploits are capable of that type of attack, but if you had the same experience, and I’m running none of the same scripts as you (my PHP is all custom, with one single non-public password-protected exception)… makes you wonder.

Any other Dreamhost who have experienced this?

7
June 05, 12h

Dave — You may want to sift through all the file source code in your public_html directories for all of your sites (assuming you may have additional domains hosted on your account) and look for anything fishy. Sometimes, not only do they change code on your site pages, they also upload or change files completely with things called “shell viewers”, a common one being c99 shell viewer exploits.

It’s good that you updated MT to the latest version. I would also probably look to see what PHP version Dreamhost is using, because that can also have adverse effects on your site. Also, get rid of any old apps that you no longer utilize, archive them if necessary, and take them out of public view.

Here is a great read about some common web application exploits:

http://www.honeynet.org/papers/webapp/

8
June 05, 12h

I had the same thing happen on my blog, which is running Wordpress (PHP) on a non-Dreamhost server.

Seems like it might be a PHP hack that affects several server combinations.

Also, the same issue with seemingly useless pages being linked to.

9
Josh Dura says:
June 05, 12h

The one thing that confused me about it all, was how they edited my database. They put in an iframe in my forum description. Up until then, I was convinced it was just something targeting index.php, but that made me think someone had access to my entire Dreamhost Control Panel (phpmyadmin, file editor, etc).

10
June 05, 12h

That’s super creepy. It’s like you had an invader in your house. Yikes.

11
voyou says:
June 05, 12h

Something very similar happened to the Crooked Timber blog (at http://crookedtimber.org/2007/06/02/wordpress-question/ ); which, as it turns out, is also hosted by Dreamhost.

12
Roy Anger says:
June 05, 12h

Counting this post and the comment from Josh Dura, this is now the fourth time in about a week I’ve heard people mention this problem. The one common element has been Dreamhost. See http://www.adamfortuna.com/2007/05/31/you-could-be-hacked-too/ post.

13
Tom says:
June 05, 12h

The may be a completely idiotic question, but are you using the new Gravatars and if so, could it have been that code that was used as an exlpoit?

14
Josh Dura says:
June 05, 12h

@Tom: I am using old Gravatars on joshdura.com (it BADLY needs an update, I know :D), so I doubt that is it.

15
Dave S. says:
June 05, 12h

@Tom - Not idiotic at all. Yes, I am using the new Gravatars. Yes, it could be that their code is the problem. Some of the other sites mentioned are not though.

But aside from PHP, not all the other sites mentioned have anything else in common (though there’s a strong Dreamhost current running right now). So it might be there’s more than one reason why this is happening. Could be PHP + Dreamhost, could be PHP + Gravatars, could be PHP, could be something else entirely. Hard to know for sure yet.

16
June 05, 12h

Yep, happened to me, and I know a few other Dreamhosters who were also affected. In every case, Dreamhost claim it was probably PHP-related, and suggesting people upgrade Wordpress or whatever other PHP thing they happen to have on their file server (if any).

I’m not particularly satisfied by Dreamhost’s vagueness over this…

17
Ingo Chao says:
June 05, 12h

Searching for the phrase
“<u style=display:none>”
reveals lots of results.

18
Dave S. says:
June 05, 12h

Oh, duh. I just realized that this got all my sites… including some sites that are running Wordpress. So just because this one doesn’t, doesn’t mean that it had to be the attack vector. Could have been any of the other sites.

So put me down for one more on Dreamhost running Wordpress who got affected. Sounds like there’s a pretty common thread now.

19
Dan says:
June 05, 12h

We had a similar problem across our VPS. We had multiple installs of both Movable Type and Joombla, and think we pinned it down to some insecure PHP in Joombla. Now, we’ve got crons running every ten minutes looking for world-writable directories and looking for anything like the c99shell scripts that were used to hack our pages in the first place. This is by far the most aggravating thing I’ve ever dealt with on the web, so you have my sympathy.

20
Runa says:
June 05, 13h

Dave, do all your other sites on the same host use php? Is anyone of the affected an easy static site (just xhtml, css, javascript) or does it use another server side scripting language?
Maybe in that case php wouldn’t be the problem.

21
June 05, 13h

Interesting and scary indeed! I too am with Dreamhost, but haven’t run across this problem yet. I’ve googled around a bit and could it be that you are using plain FTP instead of SFTP? Regular FTP transmits passwords in plain text, so I imagine if someone figured out a way to monitor that on Dreamhost’s servers, then a ton of people would be vulnerable. I’ll have to keep an eye on my sites, but so far, so good.

22
Ian Lloyd says:
June 05, 13h

I had exact same problem - numerous sites attacked. DH told me that it had been accessed by IPs in Russia, Mongolia and US and advised me to change all passwords. They also suggested that I may have not protected myself properly (I responded saying that I had a clean system, never publish FTP passwords, had non-guessable password etc).

As it happens, the only way I knew that it had happened is because the script had processed an index.php file and removed all carriage returns. In doing so, it had mangled a PHP script to cause a script error on the server - it was *that* that alerted me to the problem.

It affected sites that were HTML and PHP, and some had Wordpress installed.

I really hope that you highlighting this will enlighten me as to the cause - I was becoming v suspicious and assumed that DH’s servers had been compromised, but of course they would say if this were the case!

23
Dave S. says:
June 05, 13h

@Aaron - “I’ve googled around a bit and could it be that you are using plain FTP instead of SFTP?”

No. I’m fairly secure, I use SFTP and HTTPS as much as possible.

24
June 05, 13h

It happened to me too, on Dreamhost. I also found that they had somehow uploaded a remote control script called ‘web.php’. Look for it on your servers. It starts out with a huge ASCII picture of a spider. I found evidence in the logs that somebody at 195.5.3.234 was POSTing commands to it. Some of these POSTs were stopped by mod_security, but most made it through.

The one thing I regret is that I immediately deleted the script completely, it probably would have been smarter to keep it and reverse-engineer it.

25
anush says:
June 05, 13h

That’s terrifying! I didn’t think this kind of crafty spamming was prevalent. Does anyone have any advice for not-so-PHP-proficient webmasters to prevent this kind of thing from happening, besides changing passwords regularly and updating blogging software?

26
Frank says:
June 05, 13h

I’ve been around web hosting for quite a while and my first guess (not blaming anyone) would be a compromised server. I’ve helped out many people where every index.* was defaced or altered. Once (if) someone root level access to the server is simple to install a program that modifies every index.*. The fact that multiple of Dave S’s sites were affected would indicated something at a server level.

27
June 05, 13h

I know of another 5-6 people who’ve similarly been hacked and run on Dreamhost. If PHP is the other common vector, are you running on PHP4 or 5?

28
June 05, 14h

I just got nailed with the same hack and I’m also a DreamHost customer, though I’m running WordPress. I got nailed right after I upgraded to 2.2 via the one-click install.

Luckily for me, my pages are served with the application/xhtml+xml MIME type so their junk markup broke the page which tipped me off to it right away. Like you, the junk was appended to the index.php directly in the root, though that was the only place it was placed.

29
June 05, 14h

As well as Jonathan’s question about the version of PHP, is register_globals turned off?

30
Ben Ward says:
June 05, 14h

And count me in as another Dreamhoster suffering from this. Multiple sites, across multiple Shell/FTP accounts.

You can email DH and they’ll send you a list of all the IPs that have accessed the FTP/Shell using a specified user account. If you’ve a hosting account with multiple shell accounts, could be useful to look for crossover.

It’s very concerning that Dreamhost is the common factor here. Certainly software would be a more obvious explanation, but with no reports from any other hosts? That’s very very strange.

31
Meri says:
June 05, 14h

We’re also with Dreamhost and running Wordpress – had the same issue with a selection of our sites. Also a little worried that DH seems to be the common thread here…

32
Kenrick says:
June 05, 14h

@jeff - “And one more tip: avoid PHP like the plague. :)”

its more like, learn the language you are using and youll be fine, don’t tell me this couldn’t happen in python or ruby or any other language.
Statements like that are totally stupid. I’d think you’d be above that.

33
Keith says:
June 05, 14h

The Blue Flavor site got hacked as well. It’s not with Dreamhost (it used to be…) and so, theoretically, this isn’t a Dreamhost-specific issue.

34
June 05, 14h

@Kenrick
It’s not really php that is the problem, more php programmers ;-)

As a language php makes it very easy to write bad code (almost encourages).. Use of globals, not escaping sql, lack of robust server variable handling. I’ve not used php for a while and the reason for this is that it became very difficult to maintain/test larger applications because of the lack of support for good programming paradigms. Python isn’t the solution, nor is Ruby; good programming practises are the solution and these are easier to learn/implement using Python and Ruby.

35
June 05, 15h

Dave, you seem to be using Movable Type, and from what I can tell in your code it looks like the attackers modified your MT templates or some other static file. I’d chalk that up to an issue in the existing version of MT. However, I do see you doing includes that use data from the $_SERVER superglobal array – while I don’t know offhand that the DOCUMENT_ROOT data can be manipulated by the user, other members of the $_SERVER array *are* based on user input, and *should not be considered safe*. Best practice is to treat *everything* in the superglobals as potentially hazardous, and escape all of it. You’d be better off setting a constant at the top of your templates that contains your DocumentRoot, and using that.

My two cents on the “PHP is inherently dangerous” FUD floating around here:

PHP isn’t the problem, in that the PHP interpreter itself does not suffer from a larger number of holes than other interpreters. Nor do I believe that PHP inherently is less secure than other scripting languages.

The state of PHP coding *is* the issue. I see a lot of people mentioning Wordpress, which has a relatively poor security track record. The analysis I did on NIST NVD stats related to PHP apps might prove interesting:

http://blog.funkatron.com/archives/infosec/the-php-app-insecurity-top-20/

PHP is very, very easy to pick up and become productive in quickly. That’s a good and bad thing: good for getting people going quickly, bad for expecting best practices from folks who barely have any programming background. Secure programming is generally done crappily even on the university level, so it’s a lot to expect folks who pick up a little scripting here and there to grok it.

Inherently, I don’t think PHP is less secure that Python or Ruby. By default, none of these languages do things like forcing input filtering or escaping ourput by default, and none of them require parameterized SQL that would, for the most part, eliminate SQL injection issues. Some *frameworks* do, but there are plenty of good frameworks for PHP as well. Rails (built on Ruby), Django (built on Python), .Net (usually C#), and the like all are strong frameworks, but all of them are also open to the usual web app security problems if a programmer isn’t careful.

36
Matthom says:
June 05, 16h

I’m another Dreamhost customer who’s experienced this, although I’m not running any blogging package - rather a self-made system.

The attack consisted of malicious “index.php” files replacing my own versions of index.php.

To work around this, I modified my .htaccess redirect script to point to “index_xxx.php,” instead of index.php. I guess you could call this “security through obscurity,” but it seemed to stop the problem from re-occuring (for now).

37
June 05, 16h

Another one here: Dreamhost, PHP, Wordpress.

Ironically enough, the reason I noticed is that my site is serving its code up as application/xhtml+xml to browsers that support it, so my entire site fell over for Firefox users: hardly pleasant, but it meant the problem got spotted very quickly.

Whatever the problem is, it scares seven shades out of me. I’m not going to be happy until I know what the problem is, how it occurred, and that steps have been taken to fix it. That might just be upgrading to the latest version of Wordpress; on the other hand, it might mean changing to a different CMS, or even changing web host.

38
June 05, 17h

One more thing: it is very, very, very unlikely this is a Dreamhost issue. I’ve used them for *years,* and I’ve used a lot of hosts before them. Dreamhost does a number of things to improve security that the vast majority of web hosts do *not* do (such as forcing users to run PHP as a suexec’d CGI). DH is one of the most secure large hosts out there.

Again, if you’re running Wordpress, that’s almost certainly your issue. If you are running anything before WP 2.2.0, you’re running a version that has known, exploitable security holes in it. You *must* upgrade WP, or switch to something that has a better security record. Textile, Movable Type, ExpressionEngine, and Serendipity have all proven to be more secure, to name a few.

39
June 05, 17h

@Kennrick:

First: It was basically a joke. Tongue-in-cheek, hence the smiley face. Sorry if you were perosnally offended.

Second, and to Tim Parkin’s point: Yes, this could happen in Python, Ruby, or any other language. Likewise, viruses *could* happen on Mac OS X. But you know what? They (pretty much) don’t. And they’re everywhere on Windows. Just the same, PHP is constantly suffering from security holes like this, and they’re rarely heard about with other languages. As Tim points out, it’s not entirely the fault of the language itself. But, no matter who’s at fault, it’s obvious that there is a fault with the security of PHP apps, or stuff like this wouldn’t happen so regularly.

Bottom line – the facts can’t be argued: PHP deals with many more serious exploits than either Python or Ruby.

40
June 05, 19h

Update: turns out a handful of DreamHost FTP accounts got hacked and mine was included, which is why my site got attacked.

41
June 05, 19h

Hmm, interesting. This same attack pattern went through MySiteSpace recently, although the links weren’t hidden in that instance (http://www.webhostingtalk.com/archive/index.php/t-597187.html). The links changed every night and any file named index.* was modified - PHP or not. It also linked to not-so-benign code and URLs, triggering antivirus warnings for site visitors.

The affected site doesn’t run Wordpress, in fact the only PHP in use was a simple frontpage doing some RSS parsing and a random image. No apps, no database.

Still, I replaced that single PHP file with an HTML file just in case; but the attacks continued.

MSS never claimed that the fault was at the users’ end. They claimed to fix the problem by “upgrading their servers to fix the problem once and for all”.

Since then the problem has not recurred, even though the PHP home page has been reinstated.

Make of that what you will :)

42
Tom says:
June 05, 19h

Dave — somewhat unrelated, and I’m sure you’re quite wary of betas so you may not be that curious, but I went out on a limb and installed MT4-beta…

The interface is a big time switcheroo that I find somewhat disorienting. My admin pages and front-end both load pretty slowly now, but the speed isn’t nearly as bad after the pages are cached. Fat lot of good that does any potential visitors… There are also now an obscene amount of plugins than previously came stock. You can now create “pages” and manage files from within MT (which I probably won’t ever do). Comments are broken in my install, FWIW.

43
Sur says:
June 05, 19h

For what it’s worth: Everything here points to the files being modified directly, with MT completely uninvolved. Involvement would be easily verifiable via the code showing up in the templates, which was not mentioned.

This issue has popped up several times in the MT forum, and not a single one of those people has been able to substantiate the involvement of MT.

44
June 05, 20h

One more Dreamhost customer here who got bitten by the same issue…

Best I can figure, the attack vector was an old install of WordPress that I set up via the DH 1-click install tool a while back. I never did anything with it and forgot about it… bad idea, as that meant I had an out of date (i.e. full of vulnerabilities) WP hanging around.

45
June 05, 20h

@Ara - I just received an email from Dreamhost referring to the compromised FTP accounts that you mentioned. Mine was among those hacked, but no changes were made to my files.

According to the email only 20% of the accounts that were accessed had any changes made to them and the files that were altered were either index.php or index.html.

I went ahead and changed all of my passwords on all of my DH accounts, not just the FTP account that they recommended.

46
June 05, 20h

Just read your update, Dave. You say that FTP accounts have been compromised. In that case, this probably had nothing to do with either PHP or WordPress. It’s probably a matter of someone (or, rather, some script) logging into via FTP and making the changes directly to files named index.php and index.html (rather than exploiting security holes in PHP or WP). This probably means that MT users could simply rebuild their site, thus regenerating all (or most of) your .html files and be okay…right (sorry, been a long time since I’ve used MT)?

It would mean that PHP apps like Mint could be affected, since they often have files named index.php (although my Mint on Dreamhost doesn’t seem to be affected). It also means that users of frameworks that abstract URLs from filesystem locations (Django, RoR, etc.) are probably safe, since they are much less likely to have any files named index.html or index.php.

I don’t seem to be affected by any of this, so this is all totally unfounded – but it’s what I would extrapolate from Dave’s info that FTP accounts were compromised and the target is files named index.html and index.php.

47
June 05, 21h

I’m running WP 2.1.2, on DH, and don’t appear to have any issues. I’m still poking around, but so far, so good. Perhaps since I’m still on 2.1.2 that played a part, or perhaps I just got lucky.

48
June 05, 21h

I just got an email too, and for our main shell account. I haven’t found any damage yet…

Dave, Thanks for helping bring this to DH’s attention!

49
Simon G. says:
June 05, 23h

All the dreamhosters who’ve been hacked - what machine are you guys hosted off? I’m on kitkat and haven’t had any problems (touch wood!).

The one thing that *amazes* me here, is that it seems like so many people are NOT running any form of file alteration or intrusion detection software which would have immediately spotted this.

I’ve just finished writing up a quick quide to running a simple file alteration/IDS application on shared hosting here: http://simon.net.nz/articles/simple-webserver-file-alteration-monitoring-using-integrit/

I hope it helps someone.

–Simon

50
June 05, 23h

I’ve actually been considering switching to Dreamhost — this is definitely giving me second thoughts. Although, staying on TextDrive is driving me crazy — my site is down far too often for what I pay a month.

51
Kevin Lawrence says:
June 06, 00h

“One more thing: it is very, very, very unlikely this is a Dreamhost issue… Again, if you’re running Wordpress, that’s almost certainly your issue.”

Man… couldn’t have been more wrong, eh?

52
June 06, 00h

I think the more pressing question is how many of those 3500 accounts contain sensitive data - either people’s personal data, or confidential client data. There are a number of people and companies who hosts web applications at Dreamhost.

Now, we can be naive and hope that this is simple a spammer running a script. But compromising 3500 accounts – at that level – is a serious breach of security. If they could change the files the way you describe, they would be able to look at client and personal data/databases (or simply download them).

I do not know about Mezzoblue, but for those who have been compromised that also store client and/or personal data this is much more serious. If you got a web application every user needs to be notified and told that their password (email addresses etc.) has been compromised. For people who have a section where clients can login to their stuff, they too need to be notified.

It seems like the work of a spammer – but Dreamhost was hacked in the worst way there is (root level it seems).

53
Alex says:
June 06, 01h

@Jeff Croft - It’s naive to think that PHP is less secure, because sites running it get hacked more often than Python or Ruby sites, for example. Hackers aren’t even interested in exploiting Python, because there are hardly any sites using it… Now, PHP on the other hand is the most popular language on the planet.

As a spammer, would you even bother searching to exploit platforms that nobody uses, or would you exploit the most common platform?

As far as Dave’s problem goes, or anyone with Dreamhost, I think it’s a security hole in host’s server configuration.

54
June 06, 01h

It looks like this is a Dreamhost problem and has nothing to do with WordPress - they sent out an e-mail last night saying that 3,500 customer FTP accounts have had their passwords stolen: http://simonwillison.net/2007/Jun/6/dreamhost/

55
Nev says:
June 06, 02h

Thanks for highlighting this issue, Dave. I have sites with Dreamhost and haven’t heard anything about the problem, so I’m hoping that means my sites weren’t compromised. Just to be safe, I’ve changed all my passwords anyway.

56
Gerben says:
June 06, 04h

I had similar problems quite some time ago. It only affected the index.html files. It did put a few links directly after the first paragraph (so after the first ), but it didn’t hide them. They also seemed to point to nonexistent urls. I’m not using dreamhost, WP or even PHP.

I’ve since been setting my file permissions to to read only, which is a bit of extra work but should make it more difficult for most automated scripts. I haven’t had any problems since, but I know that is in no way any guarantee.

57
sck says:
June 06, 04h

For those with Dreamhost accounts that have this issue, are you guys logging in through FTP or SFTP? Dreamhost supports SFTP which is a secure alternative to FTP. I have a feeling someone is possibly sniffing the plaintext FTP passwords. I only connect to my Dreamhost site via SFTP and don’t have this issue.

58
KC says:
June 06, 05h

Something needs to be done to stop these hackers - someone needs to find the root of the problem and give them what they deserve. This kind of foul play makes me sick, how can ppl sleep at night?

59
June 06, 05h

Based on Dave’s article and the first few comments, I would have guessed that this was some cross-site scripting (XSS) attack, based on the exploitation of known vulnerabilities in PHP (and/or applications using PHP, like MT or WordPress).

It actually seemed unlikely to me that it was the result of a compromised server, as later comments suggest… (Since these 3500 accounts are probably not all on the same physical server –I hope!–, *multiple* servers would have to be affected.)

Making sure that you always have the latest security patches installed is probably the best thing you can do against these attacks; also, I think it’s a good idea to subscribe to SANS’ Security Alerts to stay informed!

60
June 06, 05h

I was one of the first DreamHosters to notice this problem. I serve my XHTML as “application/xhtml+xml”, so when the “hack” occurred it broke my website because it wasn’t well-formed. I immediately informed DreamHost about it, and the full scale of the problem began to emerge over the following few days.

61
June 06, 05h

I had an account get hacked in a similar way about a month or so ago. This is not on a Dreamhost server but on an account with Fasthosts in the UK, I’m really not sure how they got into the account although I know my friend had quite a week password so that may have been it. The code they placed in every index page was like this:

<iframe src=’link to page on another website was here’ width=’1’ height=’1’ style=’visibility: hidden;’></iframe>

I changed the password to something more secure and I think it’s been fine since. Scary to see Dreamhost getting compromised on that level though!

62
David Guy says:
June 06, 06h

I just had this happen to me on Media Temple hosting. Exact same problem - index file injected with a hidden div. It seems like this is so widespread! How are they cracking into so many servers? Surely they can’t be guessing passwords. I do have a few instances of MT, Wordpress and Expression Engine installed. Not to mention some older guestbook type scripts on a couple of domains.

We suffered a similar hack a couple of years ago - but instead of spam being injected, the hacker replaced ALL index files with a nasty message. That was a true nightmare. Thank goodness for backups.

63
June 06, 06h

It looks like Dreamhost is to blame. 3500 FTP Passwords Leaked - http://www.caydel.com/dreamhost-leaks-3500-ftp-passwords/.

Dave can you let us know if you received the ill-fated email message?

64
June 06, 07h

@Kevin Lawrence: Yeah, I was totally wrong. Yow. That’s just… bad.

@Marcel Feenstra: Movable Type is written in Perl, not PHP. Also, exploits in PHP itself are rare, comparatively, to application-level exploits. I did some stats analysis earlier this year that deals with this:

http://blog.funkatron.com/archives/phpsecinfo/so-what-is-the-state-of-secure-development-in-php/

65
June 06, 07h

I had one of my customers forward the e-mail they got from Dreamhost about all this though I haven’t found any signs of files being modified. Same for my own site, though I didn’t receive that e-mail directly which either means I’m safe or they haven’t got around to messing with me yet. Guess it’s time to change a few more passwords just to be safe.

Considering all the stability and outage problems they had last year and now this, might be time to start looking around for new hosting.

66
June 06, 07h

I got hit this morning, and got a message from Dreamhost as well. I rebuilt all my pages, and replaced whatever I needed to with local files.

After doing some more looking around, I noticed two files in my javascript folder that weren’t mine (and I’m not sure how long they’d been there)

One file was named “lt.txt”, which appeared to list a bunch of urls, and also a “index2.php” file, which contained… well, a bunch of code I couldn’t tell what it was doing.

Like an idiot, I threw them out, and didn’t hold on to them for anyone else to look at, and compare.

So… my point is… anyone who’s experienced the index.php appending might want to look about for more tomfoolery.

67
June 06, 08h

@Ed Finkler: My bad… Many CMS are written in PHP; I didn’t realize MT uses Perl.

As for DreamHost leaking 3500 passwords, that’s just… incredible! :-(

68
June 06, 09h

“So in the end, it looks like there was nothing I could have done to prevent this one aside from host elsewhere” I must agree with You - right now people who work in SEO have bigger competition so they try new “things” and what I can see that a lot of Turkish hackers work with people from SEO and that’s why Your blog have spam. They search for bugs in the code of most popular blog scripts like wordpress and they try to hack it and put their links. So I also want to thank Dave, for helping bring this to DH’s attention! As You can see i use also wordpress and now i checking if they also infected my blog.

69
June 06, 09h

Another unhappy Dreamhoster here. I think I’ve rooted out all the bad code, but in some cases, on HTML pages rather than PHP, the spam actually REPLACED code, so I’ll have to see if I have backups somewhere. The PHP pages seemed to be easy to fix, since the malicious code was usually just added after the legitimate stuff.

70
David Guy says:
June 06, 10h

Since my server was hacked I went through all the folders and found one that was out of place. Inside was a script. The first line was:
PHPSHELL.PHP BY MACKER (mod. troll32)

If you’ve been hacked you may want to look for a similar file. Mine was inside a /img/ folder an was named sd.php along with a /tar/ folder with tons of spam links.

71
June 06, 10h

Since all of the data points here are of the “Me too, I got hacked too” variety, I thought it might be worth sharing that I run several sites on Dreamhost, including multiple installs of WordPress, and haven’t found any trace of this hack in any of them.

Not that that proves anything, of course, but this thread was shaping up to look as if all Dreamhost/WordPress customers were being hacked.

72
Jonas Flint says:
June 06, 11h

I hate to discourage you folks. But it is not just Dreamhost, and I quickly realized it has nothing to do with someone knowing your passwords.

This happened to several of my sites months ago. I struggled with it for months. Finally I ended up writing a script that eliminates this problem altogether as long as you include the files that are being attacked in the script. After a while they will not just target your index files. But will also target your header and footer files as well. Same content also. Viagra, cialas, all of that jazz.

Anyways, I have a solid solution that works but I hesitate to post it lest it be compromised. If you are interested in the solution, please email me, Dave.

73
Andreas says:
June 06, 12h

I had the same problem with my blog - also spent an afternoon cleaning up installs and changing passwords.

In my case, the links pointed to hidden SEO pages on “Arthurmag,” which then redirect to a spammy medicine search engine.

Very strange stuff, and it’s clear there are many others affected: http://s.technorati.com/www.arthurmag.com?sort=freshness

74
Jonas Flint says:
June 06, 13h

I believe the some of the sites that being linked to are compromised sites as well. So the hackers are doing a pretty good job at hiding themselves.

Look for files on your servers with strange file names like 845356.php etc. They tend to be in upload directories or directories with lower permission settings.

Sad thing is, not only do they put hidden iframes on the site that redirect to other sites, but also installs spyware and virus’s on computers as well…

For a while I took it kinda personal ;) . It was me vs. the hacker. The challenge was on. I would replace the compromised files, and the next morning, sure enough a hidden line of code within the page.

Eventually I wrote a clever little script based on the hackers (or the script he wrote)patterns. I still check those sites now and then, and haven’t had a problem since…

75
Dustin says:
June 06, 13h

The injection is almost certainly due to an unprotected cross-site-scripting vulnerability. If an attacker is able to inject custom php code into an upload or contact us style form it is possible to trick the server into executing the uploaded code unchecked. This can be caught in programming fairly easily, but can be overlooked by novice programmers even easier!

This style of attack is much easier than hacking FTP/SSH passwords and can be easily automated.

I’ve found that many (very popular) scripts available on the net do not correctly sanitize user input. Always treat whatever a user submits as tainted!

76
June 06, 15h

@ Ed Knittel - thanks for the link (http://www.caydel.com/dreamhost-leaks-3500-ftp-passwords/). looks like that was it. Every site I had on my package had it’s index page wiped and replaced with an iframe. I have now altered my methods.

77
sosa says:
June 06, 16h

I’ve must confess a sin: Three weeks ago, I’ve downloaded and used a cracked version of coda (because i had a mac to play with) and while i was using it a paranoid thought come to my head: – What if this unreliable cracked program is uploading my ftp’s login details somewhere?

I stopped using the program (and the mac) and forget about my paranoia until today that somebody logged into my account using my password and replaced every single index.php with some grim iframe.

3,500 curious sinners sounds like a possible number don’t you think?

78
June 06, 16h

Add me to the list. I am doing a complete reinstall of WordPress right now (after deleting everything on my web site, changing all of the passwords, etc.).

What worries me is that two other dreamhost FTP accounts I have did not get a notification e-mail about. The sites appear to be clean (as far as I can tell), but that still scares me that their passwords might be in the hands of someone else too. Guess I gotta change those too.

I never installed Coda (and certainly no crack for it), so scratch that off the list.

The fault is clearly on Dreamhost’s side, not on our individual machines.

79
Jonas Flint says:
June 06, 17h

Ummm, this is not just happening to dreamhost accounts. I don’t have a dreamhost accounts and this very thing happened to me just weeks ago…

Not to be grim, but I’m betting that, you will all change your passwords and replace some installations and will see it again in about a week. How do I know this? It happened to me…

80
Kyle says:
June 06, 18h

Looks like it was a DH security breach:

http://www.dreamhoststatus.com/2007/06/06/security-breach/

81
June 06, 20h

One ‘defence’ I run against this sort of thing is, when ‘common markup’ is included into the html via an include, I put a die() command after my </html> tag.
ie
</html>
<?php die(); ?>

The hackers don’t usually spot that I have done that, and an amazing number do drop their spam links at the bottom of the file.
It doesn’t stop them hacking in, but it does stop a large number of hacks from having an effect.
I do like doing that to them :)

82
Tanya de Fries says:
June 07, 00h

The spamers today are going to be more and more tricky. For some month ago i had a problem with a script that wrote an extra line on all my pages(not at dreamhost) with an advertising redirection-link really crazy and until now i dunno how it could be. But the hardest thing is that if u have hidden links on your site, google rank your site down because of the hidden content. i hope there will be some good anti-spam solutions in future.

83
Big Bean says:
June 07, 01h

I’m pretty sure that this has nothing to do with Dreamhost. It’s most likely one of the hundreds of XSS (Cross Site Scripting) + RCE (Remote Code Execution) exploits which exists on the net.

Best thing to do?

ALWAYS keep the LATEST version of EVERY piece of web software and all of the PLUGINS for the apps that you have installed on your site… even that little test install of XYZ that you “don’t even have linked up” cause the bad guys will find it and exploit it.

It only takes one vulnerable app to allow access to your entire site.

I know I know, it might make your pretty themes and skins break but trust me, you would rather deal with that than a defaced, or worse, deleted website.

I suggest subscribing to the RSS feeds at your favorite app’s websites and upgrading things (backing up of course) ASAP.

Good security sites:
http://securityfocus.com
http://ha.ckers.org/xss.html
http://www.owasp.org/index.php/Main_Page

84
Pirahna says:
June 07, 03h

I had problems with index.html and index.php replaced by hacker but in the end it turned out to be my fault.

Use separate ftp users for every site you own (that’s if you own under 400), check your php scripts for security holes …

It’s the user’s fault these things happen, i don’t think dreamhost had something to do with it.

Think about it, there’s a https connection with the panel, so the only problem is your password and your browser.

85
Composer says:
June 07, 03h

Thanks for highlighting this issue, Dave. I have sites with Dreamhost and haven’t heard anything about the problem, so I’m hoping that means my sites weren’t compromised. Just to be safe, I’ve changed all my passwords anyway.

86
Will says:
June 07, 04h

This happened to me on my previous host. Somehow in the source of one site and iframe had been embedded. It made an element of the design miss-align by 1px, so thats how I noticed it. I switched host…..to Dreamhost. *runs off to check his blogs source*

87
Jonas Flint says:
June 07, 07h

Well lets think about this. It must take a lot of effort to log into each account, access passwords, modify the pages ect right? WRONG. I have come to realize that perhaps these hackers are sending out some kind of detection spider, that will discover certain security holes and run some automated script that does its hacking thing, with little effort from the hacker after the initial code is written.

That being said, I’m not so sure it is a password issue, more so then issues with insecure scripts and permission settings. I would be curious to know, who, actually stopped having these problems after they changed passwords etc.

Seems like a lot of you changed passwords, updated scripts etc. Did this work for you?

88
theratoni says:
June 07, 08h

I feel I need to chime in here too. I personally host a lot of sites on Dreamhost and luckilly I have not been effected by this.

However, I have ran into the same problem described above on some machines at my work within the same time period. Very, very similar things are happening/were happening. I think it’s important that this thread knows that while Dreamhost has been having a ton of problems this year, THIS ISSUE IS NOT SPECIFIC TO JUST DREAMHOST. There are a number of other comments here mentioning the same thing.

One thing I am noticing is the prevalence of Wordpress being a culprit. In our instances at my workplace, I’m leaning toward this, especially with so many other people having the same problem. While this could be a simply undiscovered exploit, I keep thinking about admittance of Wordpress.org being hacked a few months back and their admittance that the software code was edited.

Did they scour their code enough before it was released? Did they clean it enough? Or are these hacks just what’s leftover from the initial intrusion?

89
dt says:
June 07, 09h

Sometimes, the security problem is not on the server side, but right on your computer. Few months ago was reported a virus that steal FTP accounts:

http://www.viruslist.com/en/viruses/encyclopedia?virusid=152836

I have the same problem with two clients for webhosting(iframe added to some pages) and both of them told me they were using an old version of Total Commander FTP client. After checked the FTP server logs, it looks like the attackers connect with the right password on FTP, download some files, infect and put them back online. No brute force attack, no PHP problems.

90
Luke says:
June 07, 10h

After reading the release from Dreamhost linked by Ara I’m under the impression that the attackers exploited a security hole in CPanel. So it is not entirely Dremhost’s fault - I’m not saying they are not responsible here (they are) but it seems like this issue could have hit anyone running CPanel…

91
theratoni says:
June 07, 10h

The post from Dreamhost only mentions a breach into Dreamhost’s User Control Panel. It mentions *nothing* concerning actual files being edited in user’s web accounts.

There is a two discussions going on here. One concerning a large breach of security with Dreamhost, but also a significant number of other folks who do *not* have webhosting with Dreamhost having the same problems.

92
kenrick says:
June 07, 13h

@jeff
no its not that Im personally offended, it just annoying to see those kinds of comments. python or ruby et al. of course wouldnt have as many sites that exploits, they can’t just because there are not as many of them, plus the barrier to entry is much higher than php.

i do think that at least python (ruby was cool at first but now i find it annoying) really tries to encourage the programmer to write good code. Jeff I know on your site you have done a great job at evangelizing the django framework, but really could you have seen yourself learning it if you weren’t in the same room as one of its contributors?

I think the ease of which php programs can be written are both its strength and its weakness (to quote michael scott).

93
June 07, 13h

Last year we left Dreamhost because of so many drop times, not even when they had their huge outage. I’m glad I did - I oversee a few servers for clients (FreeBSD, Ubuntu Server, and OS X servers), and we outsource ours to MediaTemple… this is a disaster. I know their size makes them a huge target, but there’s something fishy about their lackluster history of playing with production servers, and all these gaping issues.

94
Abu says:
June 09, 21h

“Avoiding PHP like the plague” is hardly a solution. Spend more time reading Chris Shifflet’s PHP security blog would be a more constructive form of advice.

With regards to server security, I would also recommend you to choose a host who is honest. Very often, from what we’ve experienced with some of our sites, is that the hosting company is more concerned about volumes and not that security-centric. The IT Security company which we hired (http://www.entylyst.com) to help us solve a breach of security said there was no fault in our code and that the host was to blame. Unfortunately, to protect their own image, they deny it and what can you do. I’d say that, if it happens again, or if others report it who are hosted with the same host, then move host.

95
J Martin says:
June 12, 10h

Great post. I had the same problem on my site. I was using an old mailing list system at the time. My problem was not links on my site but spam mail on my email form (which was not related to the mailing list.) I was receiving about 1500 spam mails a day from my contact form. Now normally I wouldn’t worry about any spam mail but when it is on my contact form I had to open each one to verified if its spam or not. (In general I would have about 50 requests on my form a day and every email would need full attention.) Just opening the emails to verified if its spam or not were killing me. Now, I have 7 websites on 3 different servers and they all got infected the same day. The first step I did was adding filters to block the spam. So if most emails would have on the subject the word “Viagra”, I filtered that word but with in few hours the subject would change to “Lesbian Girls” So for the next week and a half it was a cat and a mouse until I moved to the next step where I created code verification on all the sites. That took a programmer, a few hundred bocks and again a week and a half of waiting. The results of the code verification were nothing. The spam never stopped coming. Finally I found out just like you that there was a hole in the mailing list script. I deleted the mailing list php page code and the spam stopped. Today I’m using Feed Burner until I will have time to search for something better.

96
David J. says:
June 12, 16h

I suspect this is related to the exploits that recently occurred with Hostgator (a few months back, root level cPanel zero day exploit / PHP), IPowerWeb (PHP / Root) & earlier I believe a few other large providers were nailed.

I can’t mention their names though as it was never publicly released that they were exploited. Essentially the same thing appears to have happened at Dreamhost. In this case (from what I can gauge) any folders that were ‘world writable’ were effected.

No matter whether you were using Wordpress or any other CMS: If your folder has permissions of 7xx it will get exploited.

This has continued to occur on a number of providers. I highly doubt it was at all related to FTP passwords being sniffed unless Dreamhost’s backend was exploited. Most users I spoke to who were effected had never used anything but SFTP / SSH / HTTPS.

There’s no reason to use anything else when the provider offers shell! I am sorry to hear about the sheer amount of users that it caused damage to.

97
James says:
June 15, 14h

In my experience these hacks come from the ftp server. I manage multiple machines in a wide range of environments. I’ve had three hacks in the past 10 years (two in the last year… so the kiddies are on the march) and they were all ftp vulnerabilities (proftpd). The most recent one was exactly the one described here. My only recourse for this particular client (who had applications that needed ftp access) was to install vsftpd. It seems to have done the trick. The problem is not sniffing as far as I can tell. It seems to be some sort of error in the way that proftpd handles access ( I had actually set proftpd to chroot all the accounts - locked users into their directories). But, I think, due to an outdated kernel they were able to achieve root on the box and then had free access to upload what they wanted. In my case it was phishing shit.

So in my experience it was not php, nor wordpress. I was proftpd in one case, and a bad kernel in another. The fixes where simple. Reinstall, change ftp server, upgraded kernel. I’m a pretty active admin, but sometimes these things slip.

My 2c

98
Thomas Tallyce says:
July 01, 14h

Don’t forget that shared hosting is, I believe, often running as a single user - the webserver ‘user’. This means that a script on one customer’s host can be used to grab passwords or do other stuff on another entirely separate account. One could trivially write a script to traverse through a directory hierarchy and pick out ‘interesting’ things.

Best to go for CGI/FCGI -based hosting if you are concerned about that. That then runs as your own username, and the only way stuff is then broken into is if you have an unpatched/insecure application/script or your password is stolen.

99
Patrick says:
July 02, 10h

You can store your include files outside the document root. This is usually good practice so someone does not get a hold of sensitive information (like database connection strings).

100
Ron says:
July 18, 14h

I’ve had this happen to me on multiple sites all hosted on Uplinkearth, a shared Windows server. Also happened to some friends of mine. None of my sites on MediaTemple have been affected.

Uplinkearth wrote me an email to tell me they believed my password was hacked using a “dictionary” attack. Due to the fact I observed multiple attacks across different accounts, this seems unlikely.

Its not just Dreamhost or MediaTemple, or Linux / PHP for that matter. Its definitely either a virus on the Host’s side or a local virus affecting FTP software.

101
James says:
July 23, 00h

Hi,

I would personally consider switching hosts - If investigating logs to determine cause of intrusion is too much work, how will they even be able to know how to prevent it from occurirng again? They have no business being in hosting if they’re not proactive and aggressive on security, I would have exploded if they’d tried to fob me off like that.

I host my personal site on a Media Temple (gs) Rails container setup, and I’m ok with paying the $40/odd a month if it means I don’t have to continually patch and maintain system and other software (who has the time, if its not your job?). I figure, they’re reputable enough (they host some big names), and I’m not saving *that* much by going with a cheap as chips provider.

102
spindles says:
August 14, 03h

web hosts, in general, like to use C-panel.
Most web panels are updated by web hosts too infrequently. security holes in the CMS app would be secondary to the holes in C-panel which are separate from holes in the hosts’ system of security. it really sucks when all three become a problem and i am so glad i picked a different hosting farm for this summer! good luck guys/gals. thanks for the article!

103
John says:
August 22, 03h

What an a…….!

I’ll rip the guy a new cylinder.

You’re one of the guy’s I admire Dave, I’ve met you and you just don’t deserve this kind of attack.

There’s really no call for this, find out his address and a contract will be made.

I’m paying,

Yechhh!


John.

104
Rich says:
September 17, 00h

It’s started again. I’ve been hit 3 times in the last week, including twice in the last 24 hours.

Hostrocket.com based in Albany has not posted anything official about their server issues and this vulnerability, but hundreds of sites seem to be affected on multiple servers.

I’m really tired of having to repair this crap…any suggestions on how I could use some kind of tracking tools to see what is appending this code to my site pages?

105
anon says:
October 25, 09h

It appears to have happened again. All the index.* on multiple domains I host with Dreamhost were modified on Oct 10, 2007 at 00:57 BST. A snippet of javascript was added to the bottom of all of them, which included a url-encoded chunk of HTML, which itself was an IFRAME linked to a counter.php file hosted on a server in Russia… Lovely

106
December 12, 17h

Count me in.

I host a small number of sites on my own server, hosted with a very, very good SysAdmin company. I’ve just discovered that a number of index.html, index.php, conf.php, and page files have been modified to include the garbage.

We only run SFTP, so I’m guessing PHP vulnerability. We’re a few versions behind on a CMS called CMS Made Simple.

Keee-rist how annoying. I’ll post more if I find out anything interesting. Thanks, folks, for the tips to look for lt.txt, web.php, and sd.php files.

s.

107
July 03, 16h

First, I am NOT hosted by Dream Host.

One of my blogs was hacked in early April. I had a ton of crons and weird looking traffic in my stats, which tipped me off.

I got fired up (rage fueled), found the hacked file, deleted it, and changed password. A few weeks later, I upgraded to WordPress 2.5.1 using the plugin for doing that automatically, which I kind of regret.

Yesterday when I was routinely looking over info on my various sites in the Dashboard at Google Webmaster Tools… all looked fine and normal there, BUT I had noticed a steep drop in traffic in the past two weeks.

Did a few quick searches using the keywords for which I usually rank high and discovered to my horror that I have been de-indexed at Google.

After 24 hours, I still don’t know if my site is clean. I have tracked down some suspicious stuff and removed a few lines of code and changed my password, plus a few other mods I read about today, and have written Google to request reconsideration. I am so bummed.

I know now that it’s possible my files were permanently compromised from the hack in April and that there’s a chance I brought that nastiness with me when I upgraded to 2.5.1

Man, I wish we could band together and nail this shut. I know Google is working with individuals to try to provide tools and recos., as is WordPress, but this is just sick.

108
Angela says:
November 03, 07h

This just happened to my websites on Dreamhost also. The same things you are describing. Long PHP files of code with the ASCII Spider at the top. They put php files in directories and in my index.php files. They also added index.html files with code. The same list of links that MezzoBlue had with the same hidden CSS code.

I spent hours yesterday going through and deleting the files I could find, and then changing all of my FTP passwords and my control panel password. Most of my sites use Textpattern CMS, so this was probably from the Dreamhost security breach last year, but the crooks are just now getting to mine I guess. (And I did change my password right after Dreamhost told us about this last year, but the files must have been placed before I changed the password or something.)

I hope they don’t come back. This made me lose Google Adsense revenue and may affect my ranking.

109
February 14, 01h

Incredible, I contact Dreamhost to tell them my problem and they send back a generic list of possibilities… They had this problem a year or so ago!

Thanks for posting this. It is making me feel a little better knowing that I am not crazy.