So, you couldn’t tell it by Godaddy’s Status Page, but there is currently an email outage for many of their Cpanel users. One of my clients has been down since Monday at around 4PM and from reading Twitter, it looks like people started reporting similar issues between 10/17 or 10/18.

godaddy_email_down

Talking with Godaddy was not helpful, despite there being a number of similar reports on Twitter, when I initially reported it they said that they were unaware of any issues, but had me do a traceroute for them to help them identify the issue. My client called in several hours later and they told him that they only became aware of the issue before he called in.

It wasn’t clear what the issue was, but emails were being deferred when sent to his address(meaning they were rejected and the sending MTA put them on hold) and logging into webmail to send emails didn’t work, nor did sending via Outlook.

From what I can tell from speaking with them, my client speaking with them, and from reports on Twitter, there is no ETA and they haven’t indicated what the issue is. If you were to look at their status page, you wouldn’t know anything is wrong…however there is definitely an outage.

When the 24 hour mark hit for my client and still no update or fix from Godaddy, I went ahead and switched their email hosting temporarily so that they could get some work done and respond to the emails that have been piling up for the past day. Amusingly this triggered some angst from Godaddy support who in, what my client said was a rude tone, indicated that not only did they not have an ETA, but now he would not be able to tell if his email did start working again. Of course this isn’t true, as sending emails via webmail and SMTP doesn’t work, so all he needs to do to check is login to Godaddy Webmail or try to send an email from Outlook to see that it is still broken.

Update 10/22/2015:  Email started working again yesterday evening. Total downtime for my client was around 3 days…over 72 hours. Of course it was obvious when it started working again, he called me within minutes of it coming up as he stopped getting errors in outlook.

When a Status Page isn’t a Status Page

useless_support_page

To be fair, or I guess pragmatic, given their size maybe it just isn’t practical to keep their Status page updated every time there is an outage. If it was, there would always be something on there, as just from reading twitter, you will almost always see someone complaining of an issue with their website/domain/email. Even if only a small fraction of those are actually Godaddy’s fault, it is probably safe to say that there are at least weekly Godaddy Service Outages that impact a good number of their customers. This just wouldn’t be a good use of time for their employees…surely this Is Par for the Course with Web Companies? /S

And to be even fairer, they do change things from time to time. From browsing the Way Back Machine, they crawled the page 103 times in 2015 thus far and I was able to find two outage reports…well three if you count the two that were on the same day. So, while this page does change from time to time, obviously not every outage warrants an update.

For example on September 7, 2015, there was the below:

Hosting Control Center September 7, 2015 at 11:06 AM
Some customers are unable to login to CPanel, and some CPanel sites are down or running slowly. We are working to correct this issue as quickly as possible and appreciate your patience.

So, the question is what warrants a Godaddy System Alert…perhaps there a certain number of customers that must be impacted. This might explain why it typically says No Significant Issues. There may be some sort of metric where the number of impacted customers/sites must hit a certain point to be considered Significant and they can’t just ignore it.

Is this Par for the Course with Web Companies?

When you compare this to other companies, like Rackspace, Amazon, or Google, the difference is night and day. There are thousands of crawls compared to hundreds, as their status pages change a great deal. They typically provide a much more detailed and useful status page as well, which list out their services, what their current status is, and reports issues as they occur. After all, people depend on their services for business and are their customers, so having this information available is important.

It is possible that Godaddy is just a rock solid host, awesome company, and never has outages, unlike these other companies with fragile infrastructure. Or perhaps they are just a really small company and their IT team is good enough to mitigate issues within minutes when they happen, so there is no need to update the page. However, I think it is much more likely that proper hosts provide proper status pages, while Godaddy only reports items when they his a certain metric of customer impact.

Yesterday, a vulnerability in an old version of Revolution Slider was reported. The vulnerability allows visitors to view arbitrary files on the web server, like wp-config.php, without being logged in. All you need to view any file on the server is to know the location of the file and for the web-server user to have permission to view it.

According to ThemePunch, the plugin developer, the vulnerability was patched 29 versions ago in February, but they decided not to publicize the severity of the issue, aside from a single ‘fixed security issue’ line in their change log. This was because:

“[We were] told not to make the exploit public by several security companies so that the instructions of how to hack the slider will not appear on the web.”

As a result of this negligence and the way that Revolution Slider is Updated and bundled with themes, this simply left any website not running a recent version of Revolution Slider vulnerable for months to an extremely serious file inclusion vulnerability.

Its Your Fault For Not Updating

This seems to be the company line for this issue. After the vulnerability was made public, they have stated:

“You should always keep the slider up to date like any other WordPress component but urgently need to do this when using Version 4.1.4 or below in order to fix the security issue. […] We are sorry for you guys out there whose slider came bundled with a theme and the theme author did not update the slider. Since you cannot use the included autoupdate function please contact your theme author and inform him about his failure!”

And it is true, you should keep your plugins updated.

However, this is a paid plugin and doesn’t allow easy updates like a normal wordpress plugin does. Further, on all of the sites I fixed, they didn’t appear to have a nag telling the user an update was available from the backend. So, unless you are a developer and actively visited the plugin’s website, you wouldn’t even know the plugin needs to be updated, let alone an extremely serious security vulnerability.

As they mention, they sell a developer license that allows developers to include the plugin in their theme. When the plugin is included with themes, you can’t update it without updating the theme. So, any theme that isn’t regularly updated is at risk. And, since some shoddy developers edit the theme directly, rather than making a proper child theme, it isn’t always easy to update the theme. This, of course, isn’t the fault of ThemePunch, nor is it good practices to develop like this, but it does happen and is going to be a legitimate problem for people.

Even if you actually have the premium plugin itself, you can’t just update it. There is no auto-update feature(at least not in the vulnerable versions I saw,) so you can’t update it like you would a regular WordPress plugin. Nor, to my knowledge, is there an update nag on the plugin page telling users they need to update. Instead, you need to download the premium plugin, which requires a login to the site that sells it. 

The catch here is that most website owners aren’t going to have access to the login information needed to update the plugin. Your average website owner isn’t a developer. They probably paid someone to create the theme, who presumably installed a valid copy of the plugin. Unfortunately due to the nature of web design, this simply means that hundreds(thousands?) of sites are silently vulnerable to an extremely serious vulnerability and won’t even know it, unless they have a responsible web developer or host. Again, this isn’t the fault of ThemePunch, but is a fault with the premium plugin model when it doesn’t allow for quick/easy updates.

Negligence Through Security Through Obscurity

According to the plugin author, this vulnerability was fixed in February, but they chose not to report it. It has been reported that this vulnerability was publicly disclosed months ago and regardless, it is safe to say that it was known by some people for the past few months.

By choosing not to report the vulnerability and making site owners aware of this huge security risk, they effectively pushed back the date where we found out about it leaving their customer’s sites vulnerable to a known attack. And, now that is is released and being exploited like mad, we are left scrambling to fix it anyway. So, not reporting it only helped the bad guys.

I understand fully that this is a paid plugin and the need for them to protect it. I get that. And, I understand that you should keep your plugins updated. Nothing in their statements that I have seen is untrue.

However, in the event of a serious vulnerability like this, not making a valid attempt at reporting it, especially when you know that your plugin doesn’t get updated frequently and the vulnerability likely impacts a large number of sites, is negligent.

Updating a Plugin You Can’t Update

I don’t use this plugin on WordPress templates I develop, but it is used by several clients that I host. I found it bundled in two client’s themes and installed as a plugin for two other clients. All 4 had their wp-config.php file downloaded already and all sites on my servers have been scanned for this vulnerability already.

I wrote a quick and dirty patch for the outputImage function, which you can view here. This is only meant as a temporary fix, until you can assess the issue and do a proper update, but since this attack is ongoing and widespread, you should take some sort of action asap.

Mod_Security also appears to block the attack.

ipad_compatability_issue_sRecently, I ran into a weird issue with a 1G iPad and had to figure out a work-around to install apps on it.

One of my clients inherited an old first generation ipad from a friend. Before he gave it to him, it was wiped and factory restored.

After setting it up, when attempting to install certain apps, like Google Maps, Google Chrome, or Netflix, he got the following message: This application requires iOS 6.0 or later. You must update to iOS 6.0 in order to download and use this application.

Of course, the last version of iOS that was supported on this iPad was 5.1.1 and their suggestion is not possible.

After talking to a friend that has an old iPad and doing some reading, it seemed like most people would get a prompt to download the last compatible version of the app. However, even after wiping it again via itunes and making sure everything was setup, it still wouldn’t let us install old apps.

However, after a bit of playing around, I figured out a workaround that let us install both Netflix and Google Chrome on the app.

The Problem

When installing an app on a first generation ipad, a warning stating ‘This application requires iOS 6.0 or later.’ is shown and installation is blocked.

The Workaround

  1. Install iTunes on a computer and Sync iPad
  2. Install desired apps via itunes onto the ipad
  3. Wait until the apps finish downloading, unplug the ipad
  4. The apps will attempt to install, but will hang on the ipad
  5. Delete the apps from the ipad
  6. On the ipad, go into the app store and re-install the app
  7. You will now be prompted to install the last compatabile version of the app

older_version_ios

This worked for both Netflix and Chrome, however Google Maps, which we did NOT install via iTunes first still gave the upgrade needed error.

Why does this work?

I can only guess, but it seems like at some point Apple changed their policy on old devices and started allowing people to install older versions of software on their devices. I found a reddit thread from 2 months ago that discussed the change.

However, we were using a brand new iCloud/iTunes account, which had never installed any apps.

So, presumably, Apple only allows you to install compatible versions of Apps you already own. When I asked my friend, he had no issue installing any app, including Google Maps, which was not already on his 1G iPad. However, he had installed it before on other devices. By installing it first via iTunes, even though it doesn’t actually work, Apple will then allow you to install an older version…

Update 2015: Since the writing of this article, Lowes has taken action to improve the LowesLink situation. They now have an actual web-app that does not require Java and it seems to work better. I will provide a better review of LowesLink when I have more time. They are also using TwoFactor Auth now when you login with a new computer.

I wrote about Loweslink a few months ago, after years of frustration supporting several clients that use Lowe’s Home Improvement Warehouse vendor portal.

You can read through the above post, but in a nut-shell, LowesLink is a Java based portal and, due to both their technical requirements and support recommendations, anyone that uses it is very likely running an old and insecure version of Java in their browser. As a result, most, if not all Lowe’s third party vendors, and likely a good number of their employees, are at a significantly increased risk to virus’s and other computer infections.

Since my writing, Lowe’s has improved their documentation online, as well as removed some of the more criminally negligent advice, like disabling updates to Java and their statements that a 2 year old version of Java is the only supported version.

However, ultimately, it is still the same pig, just with a slightly nicer shade of lipstick.

Main Changes to LowesLink Documentation:

– Removed advice to disable Java Updates

– Updated Documentation to indicate that newer versions of Internet Explorer do work with it

– Direct users to disable insecure/mixed content warning in Internet Explorer

Removed advice to disable Java Updates

In the past, both via their trained telephone support staff and online documentation, LowesLink has directed users to install an old and insecure version of Java, as well as disabling automatic updates.

Since they updated their documentation, Lowes no longer directs the user to disable Java updates on their computer and even mention that a newer version of Java 6 works in some cases.

However, they still recommend installing a version of Java that is no longer supported, stating:

loweslink java requirements

To get the old version, they direct their users to visit oldapps.com to get the old, unsupported version of Java.

What makes this so bad is that Java in the browser is responsible for a large percent of all computer viruses, if not the largest factor. There is almost always a known unpatched Java exploit and that is for the most recent version and doesn’t take into account 0-days. Directing users to install an old version of the Java browser plugin is criminally irresponsible.

Updated Documentation to indicate that newer versions of Internet Explorer Work with LowesLink

This change to the LowesLink documentation is good. I have had Internet Explorer 9 with a new version of Java working for at least a year, if not longer with a client.

In the past, they directed Windows Vista and Windows 7 users to downgrade to Internet Explorer 8, even though as I mentioned, at least for the type of work my vendors do, it has worked okay in Internet Explorer 9 for awhile now.

Direct users to disable insecure/mixed content warning in Internet Explorer

One issue with Loweslink is that instead of serving all of their content via an encrypted SSL connection(HTTPS,) they serve some resources via an un-encrpyted connection. At least on the login page, I believe this was mostly images, but I haven’t looked in awhile.

Most browsers will warn you in some way when this happens, as it opens the user up to several attacks and can compromise the security of the browsing session.

Internet explorer makes this the most obvious, as you get a counter-intuitive popup every time you visit a page that loads content insecurely, warning you.

Rather then fix the issue, which can often just as simple as ensuring all resources are loaded via HTTPS, instead of HTTP, Lowe’s now directs users to disable the warning.

So, the content still gets loaded insecurely on Lowe’s, with the added bonus of removing this warning from ALL other websites as well.

So, for example, if a user goes to their bank website(or an attack/phishing website) and there is a similar issue with mixed content, they won’t see a warning that their account info could potentially be leaked.

loweslink mixed content_warning

The above are the exact instructions, which lead to a document showing how to disable this important, albeit often annoying and counter-intuitive, warning from Internet Explorer.

Conclusion

It is good that Lowe’s has finally updated their documentation. Before it had references to 2011 and I can say that it hadn’t ever been updated in the 2-3 years I have been supporting it. Now, they even have a revision date on some of the documentation.

However, they haven’t really changed anything. It looks a bit nicer and the advice is a bit better.

However, Lowe’s is still getting vendors to install an insecure version of Java, which significantly increases the risk that a vendor will get a computer virus. And, Lowes is still recommending changes to browser settings that further decrease browser security.

Update 2013-08-06: Since writing this article, Lowe’s has updated the documentation on the Loweslink website. See my response to the Loweslink Documentation Changes

Update 2015: Lowes updated their LowesLink system this year to move away from Java and have improved security by forcing users to use TwoFactor Auth whenever they login from a new system.

really_lowes_its_2013I do a good bit of tech support and once every few months, I get a call from a contractor or vendor that works with Lowe’s Home Improvement and needs help getting LowesLink to work on their computer.

Each time I do, I am utterly amazed at how amazingly reckless the LowesLink service is and the state that it leaves a user’s computer in. In regards to computer security anyone who uses LowesLink, even more so if you actually follow their published instructions, has opened a gaping hole in their computer’s security.

For those who are not familiar with it, LowesLink is a web portal that makes invoicing and receiving payments from Lowe’s easier. It has been in my experience mostly been used by independent contractors, but I would imagine it is also heavily used on the corporate side of things too.

The Main Issues Are as Follows:

1) LowesLink discourages people from using a modern browser, stating a requirment of Internet Explorer 7 or 8. They tell IE9 users to downgrade.
2) LowesLink requires people to use the generally insecure Java Web Browser Plugin
3) LowesLink website tells users to download an old version of Java from 2010 – 2011, jre-6u20 (2010) – jre-6u27(2011)
4) LowesLink, through their published documentation and support staff, tell users NOT to update Java and instruct users to disable updates of java.
5) The LowesLink website, while using HTTPS, loads content insecurely. This results in a warning when visiting their page.

Why this is bad:

All of the above is a great way to ensure that their users are running an insecure and vulnerable system. It effectively creates the perfect storm of bad advice and insecure software.

The Java Browser plugin is ridiculously insecure by itself, not even taking into account that their website instructs people to install a 2+ year old version.

It would probably be quicker to point out the days over the last 3 years when there hasn’t been an unpatched java browser vulnerability being actively exploited in the wild. Consistently, Java is top of the list of insecure software that results in computer infections, along with Flash and Adobe Reader.

As a result, telling your users to install a version from 2010-2011 and then disable updates is amazingly reckless and irresponsible.

What makes it even worse is, at least in part, the instructions published on their website are incorrect. Not only is it possible to run it with the latest version of Java 6, but also IE9!

Also, telling users not to update java is insane, Java 6 has already been updated 3 times in 2013, with fixes for around 60 security issues. If you follow their instructions, you would never get these updates, unless preformed manually. Which most users are not going to do.

Event the US Government has come out stating that Java should be disabled in the browser, as it represents such a serious threat to security. And that is the most recent version! Not the version Lowe’s wants you to install!

Loading Mixed Content from a Secure URL

As any web-dev with even a bit of experience can tell you, if you are going to use HTTPS, then you should load ALL resources over HTTPS.

However, Lowe’s not only fails to do this, but because the user is required to use Internet Explorer, they will see an unintuitive warning each time they visit.

Browsers handle insecure content differently and how IE handles it by default is to display a warning about the insecure content each visit to the page. The question is one of those ones that is phrased a little awkwardly, where if you care about security you really should hit yes, rather then no. Hitting yes, which is what most people instinctively do when they encounter a popup, would tell IE to only load the secure content.

However, in this case if you want LowesLink to work and display properly, you would need to probably hit no, which tells IE to load both insecure and secure content.

Fix Your Documentation

Aside from just being horrible advice, the published documentation is actually incorrect.

I have been able to get LowesLink to work using a current version of Java 6 and using Internet Explorer 9. So, I know it works, while still less then optimal.

If Lowe’s isn’t prepared to invest in fixing this mess, at least spend some resources making sure it works on a modern browser stack!

Why I am Writing This

This is one of those posts that I almost write each time I encounter LowesLink, as it is just such overwhelmingly bad advice. Whenever possible, I end up urging the user to use Firefox or Chrome and then ONLY use Internet Explorer for LowesLink, in an effort to reduce the risk of infection.

I am writing in the hopes that Lowe’s will, as their support assures me each time I call, work to update to this system. However, they have been saying that for years and their system has been reducing the security of their users for just as long.

Lowes: Clean Up Your Act!

LowesLink is a disservice to all Lowe’s Users and those who support them. The LowesLink System, especially if you follow their published instructions, by design results in a computer that is vulnerable to infection.

This choice makes Lowe’s Contractors, Vendors, and Employees a very easy group to target and the low hanging fruit of the corporate world.

For years, support has been apologizing and saying they are working on something better, but here it is 2013 and they are still telling users to install a 2 version of Java from 2011 and disable automatic updates.

Further, this isn’t just some Java applet color picker we are talking about. This is a system used for invoicing, bidding, and a ton of other really important and likely sensitive tasks. I understand it costs money to update, but I can’t even wrap my head around the multitude of bad choices that has brought us to this point.

Even my Aunts and Uncles are tech savvy enough to pick up on all the Java related news, but apparently Lowe’s can’t or won’t invest the money to protect their users. Instead, they simply leave their users computers open to infection!

Update 2013-08-06: The Loweslink Documentation has been updated since this post was created. See top of this article for more info.

Over the years, I have learned a lot about computers not just in regards to fixing computers, but also troubleshooting, the right types of questions to ask, how to walk people through simple repairs over the phone, and also how different people use computers. One lesson that it took me some time to learn is that you can’t always setup other people’s computers like your own.

I would classify myself as a power user and have been for some time. I became the computer guy for my friends and family pretty early on and was often called to setup a computer for the first time or figure out why something wasn’t working. Back then, I would lock down their internet browser, tweak security settings, and generally setup the computer just like mine. However, most folks aren’t power users and this often had the effect of making their life more difficult or meaning that they would see a warning due to a security setting and just click through it. It took me a while to figure out that while this type of computer use was preferable for me, most people don’t want or need that type of experience.

Eventually though it clicked and while I still take great care setting up peoples computers, I now try to do it from the perspective of a normal non-technical user, rather than a power user.

The Case of the Locked Down Router

I ran into a great example of this today, while dropping off a laptop.

The client had been having problems getting their work computer to connect to the network. Their laptop and ipad worked fine, but just not their work computer. So, they asked me to take a look at it while I was over there.

The network was saved with the wrong security settings, WEP instead of WPA, so I deleted the saved network profile and re-addded it. It worked right away.

However, I have found that it is always a good idea to restart the computer after making these types of changes (or any changes really), to make sure it still works on reboot. And, sure enough, as soon as I rebooted I could no longer connect to the network. So, I logged into the router to see what was going on.

After a few minutes of playing, I discovered that it was setup to only allow 2 DHCP leases at a time. As a result, unless they manually set an IP address in your network adapter, it would only ever be possible to connect two devices to their router.

In retrospect, me being able to connect right away made sense. I had their personal laptop long enough for its lease to expire, so when I came back to their house and started the work computer, it filled the second slot that their personal laptop would normally have taken.

I set it to a higher limit and problem solved, they were able to connect with multiple devices.

During the work, we talked for a bit and I discovered that the person who setup their router was a friend who works in networking.

In addition to limiting the number of DHCP leases, he also made a few other changes, like setting the SSID to not be broadcast, which were geared at locking down the router. While this is similar to how I would setup a personal network, with a limited number of DHCP leases, MAC filter, reduced subnet, ect, setting up a non-power user’s network like this isn’t generally a good idea, as they would never have thought to check the DHCP limit and didn’t know how to reset their router.

So, I think this ends up being a great example of why you should try to put yourself in the shoes of the user when setting up a computer(or network,) rather than approaching it how you would a personal system.

I was looking for a replacement motherboard for an older Intel computer this morning, specifically one that had firewire + LGA775 for a pentium 4, and came across the below one on amazon.

The price is rather ridiculous at close to 1.5K, but you can save a whopping $6 off the price on Amazon and it is a blazing fast Pentium 4! What a deal ;)

I would imagine its just a decimal/feed error, but still found it a bit amusing…