Embracing Growth: Lessons From a Global IT Outage

/in ,

Embracing Growth: Lessons From a Global IT Outage

By Eric Egolf, CEO

We are a little over 1 week after the CrowdStrike-related incident and the chaos that ensued from it. There are many, many articles that have already been written about this incident, so I don’t want to spend too much time rehashing what’s already out there. But I do want to give a quick synopsis of the situation, share some of the lessons we at CIO Solutions learned, and highlight some of the conversations we are anticipating to see continuing to unfold in the industry.

An Overview of What Happened

At its core, the cause of the incident was simple. This was not an external threat or breach of any kind; it was a software update that CrowdStrike, a leading security software provider, released to their product. This was a specific kind of update- to a driver, not just the software. It’s common for security vendors to do updates like this so customers don’t have control over whether or not they choose to update; pushing an update through on the driver ensures it goes through to everyone and is usually done to keep customers secure.

This particular update involved a bad driver at the kernel level (the heart of what the Windows operating system uses). When the update went through it rendered the system unusable until it could have a manual intervention to roll the update back. The manual nature of the fix required, in many cases, hands on keyboards and IT personnel in front of computers, a key reason it took so long to resolve.

The fallout was huge. The disruption was widespread (estimated at around 8.5 million Microsoft devices) and globally impacted the operations of organizations. The cost of damages is in the billions.

An event like this has never been experienced before. I like to think in terms of what we can learn from it; the insights we at CIO Solutions can gain to enhance our response abilities, the advancements vendors might make from this experience, and the overall industry knowledge that will now shape future conversations.

What We Learned At CIO Solutions

I can tell you that our staff had no idea when they left work on Thursday evening what they would be in for soon. When the issue was detected, our teams were called back in for a 2-day long, round-the-clock sprint of high-octane, high-stress, high-stakes work. That was not an experience they would choose to relive.

As with any first-time event, we uncovered some areas for improvement. Most of these growth opportunities are in the areas of prioritization and documentation.

Given the circumstances, I believe we did a pretty good job prioritizing which systems to focus on first to effectively divide and conquer remediation work. But the prioritization metric was intuitive and reactive, making it more ad-hoc than it would’ve been if we had time to proactively and intentionally plan how to approach it.

Likewise, for an event of this scale, our normal help desk documentation system was not ideal. With thousands of tasks being added to the list and changing rapidly, there are likely other more robust ways we could explore to keep track of the work, progress, and accountability in an incident like this. With the experience of this unique scenario now under our belt, we can continue to explore and evaluate these learning opportunities.

Vendor & Industry Lessons

On a more macro level, there are a lot of lessons to be learned for vendors and the industry overall. One of which is how vendors empower IT Admins. Any vendor that is providing any level of software updates to systems, whether they’re in the security space or not, is going to need to re-think how they provide their IT Admins tools to control this.

Another thing we’ve seen time and time again is vendors who experience a devastating event and come back stronger as a result. Again, we’ve never seen anything at this scale, so the story will continue to play out in a unique way. Regardless, the vendors involved will be rethinking the checks and balances on their quality assurance processes. They will be forced to reexamine how they are testing updates before they go out as well as better ways to stagger updates.

Even broader, questions around secure third-party access will be part of the future conversation. As part of a 2009 EU Commission ruling, Microsoft allowed for interoperability provisions that effectively allowed third parties (in this case, CrowdStrike) access to the “kernel” level. This level of access means third-party security tools like CrowdStrike can affect Windows devices at a deeper operational level. The ability to access this level of Windows devices was a core piece of this perfect storm, and the reason that specifically Microsoft devices were impacted. It’s worth noting that Apple has no such access-level requirement in the EU and operates in a different ecosystem. Whatever this ends up looking like, there will likely be conversations around regulatory requirements, and an evolution in better more secure ways to ensure interoperability and grant third party access.

In Conclusion

The silver lining for us at CIO Solutions is that any team worth its salt comes together in adversity, and we truly got to see that in real-time. This experience connected our team even more and brought to the forefront for everyone a reminder of how agile, capable, and dedicated their colleagues are. This type of event has never been seen before and they worked together under pressure to create the playbook on the fly.  I have to give our team an A+ for teamwork, creativity, and tenacity.

As for how vendors will recover and what new processes and requirements we can expect to emerge in the industry, that’s still unclear at this point. What ultimately shakes out from this event, only time will tell. One thing is for sure, I think the industry overall will continue demanding discussions and answers around these core issues. Hopefully, we will see more solutions that will ensure that IT departments and service providers are given the controls they need, while at the same time ensuring that even mistakes by their own people internally don’t have the unchecked ability to cause such widespread havoc.

Understanding The Enemy + Why Your Antivirus Isn’t Enough

Understanding The Enemy + Why Your Antivirus Isn’t Enough

/in , ,

By Russ Levanway

You probably saw a dominant story in the news a couple of months ago about a major fuel shortage across the eastern seaboard. The pipeline that provides almost half the oil to the northeast and south came under a cyber-attack. Gas pumps ran dry in Tennessee, Georgia, and other states. This happened fast on the heels of other major exploits. Then in the last 2 weeks, tech news has been dominated by a serious vulnerability in management software called Kaseya, with over a million computers encrypted with ransomware as a result.

Ransomware attacks are getting to the point where they are becoming existential threats to organizations and can disrupt entire industries and supply chains.  If it wasn’t serious before, it is now.  Furthermore, hackers are increasingly sophisticated and daring. They’re often backed by foreign governments bent on destabilizing, stealing intellectual property, or just plain old making money via extortion.

The risks of a confidential data leak are higher than they’ve ever been before. It is critical that businesses not only understand how these adversaries operate but also rethink their own approach to security.

How cyber extortion works

Hackers’ typical MO is:

    1. Acquire your passwords or exploit some vulnerability
    2. Log into your device and/or network automatically or manually
    3. Steal a copy of your valuable data (credit card numbers, bank account numbers, social security numbers, intellectual property)
    4. Encrypt everything
    5. Hold it for ransom

If they don’t get what they came for, (you restore the data and can’t (or won’t) pay the ransom), the hackers leak your data all over the internet, selling it to the highest bidder.

Doesn’t my antivirus software protect me?

As someone in the IT field, one of the questions I often get asked is ”what about antivirus software? Doesn’t that protect me?” This is an understandable question. I preach the benefits of installing and maintaining antivirus software all the time. If it’s so important to have this tool installed, shouldn’t that be enough protection?

Unfortunately, no. The truth is, antivirus software stops 95 percent of attacks, so we always have it deployed as a security baseline, bar none. But what is it stopping exactly? Antivirus is preventing known viruses, known threats. When we talk about extortion and data infiltration, we’re not talking about viruses — we’re often talking about other tactics.

Flying under the radar

Threat actors often use phishing techniques to trick you into giving them your password (if they haven’t stolen it elsewhere). Often, a cyber-attack like this begins with an email from “your bank” that asks you to log in to your account to validate information. If you aren’t well versed in how to identify a counterfeit or deceptive email like this, you’ll fall for it and click the link. (No need to be embarrassed by your gullibility: you are in very good company. According to some estimates, a staggering 30 percent of people open phishing emails and 12 percent click on malicious links and/or attachments.) That fateful click leads to a counterfeit of your bank’s website. You put in the username and password, and you’re led to a blank page. You’ve been phished. Now the hackers have your credentials for the bank. All of this is done without using a virus of some kind, mind you.

Alternatively, threat actors may identify a vulnerability in your system. Once this vulnerability is identified, they exploit it by running what may appear to be legitimate software that goes undetected. Again, hacking you and your systems without the use of a virus.

These tactics leverage legitimate credentials and exploit existing vulnerabilities. Because of this, they can, therefore “fly under the radar”. Standard antivirus software can’t prevent this, it can only help stop code it knows to be malicious.

Adjusting your expectations

I talk about hacking all the time, I must seem like a broken record. But cyber-attacks keep happening, both in extreme cases like what we see in the news and for our clients, large and small. I keep hoping that if nothing else, a major event like the fuel shortage can help people understand how prevalent and destructive they really are.

Arming yourself with an understanding of how these threat actors operate is the first step. The second step is realizing that effective cybersecurity isn’t a question of simply having current antivirus installed. As we’ve seen, this tool can only do so much. That’s why the approach needs to shift. Cybersecurity is not one-dimensional and antivirus is not a catchall. In today’s world, antivirus is only one part of what must be a much broader cybersecurity toolset. It’s important that the expectation is adjusted to match the reality.

[ READ: Ditch the Drama: 5 ways to stay ahead of the hackers]

Ditch the Drama: 5 Ways to Stay Ahead of The Hackers

/in , , ,

By Russ Levanway

Ransomware attacks are getting to the point where they are becoming existential threats to organizations and can disrupt entire industries and supply chains.  If it wasn’t serious before, it is now.  Furthermore, hackers are increasingly sophisticated and daring, and are often backed by foreign governments bent on destabilizing, stealing intellectual property, or just plain old making money via extortion.  The risks of a confidential data leak are higher than they’ve ever been before.

One of the questions I get asked regularly is: “What can I do to protect myself from data infiltration?”

The first step is arming yourself with an understanding of how these threat actors operate. The second step is realizing that effective cybersecurity isn’t a question of simply having current antivirus installed. In today’s world, threats are varied in nature, and an effective cybersecurity toolset must be multi-dimensional. [READ: Understanding the Enemy + Why Your Antivirus isn’t Enough.]

Here are the 5 best things you can do to protect your business and stay ahead of the hackers:

#1 Keep learning

As cliché as it is, “knowledge is power”. The most powerful line of defense is prevention and education.

We continually have to remind people of that. Thankfully, at CIO Solutions we have long been offering anti-phishing educational tools to clients. These include a valuable training tool that enables your company to educate users in real-time. Through simulations, training videos, and more this tool can make users aware of phishing and empower them to identify and avoid it. We provide this to most of our customers, but its efficacy is only as good as the business’s willingness to put in the work.

To reap the benefits of a program like this, users have to engage with the orientations and training videos; they don’t work by osmosis. Businesses that embrace these trainings and stress their importance are better off than those that don’t. Often, it’s the companies whose employees skip the trainings that wind up incapacitated by a phishing attack, desperately in need of our help to clean up a mess.

#2 Remember your backups

We were recently engaged by a cybersecurity forensics firm to help a large organization that was mismanaging its backups. Sadly, they had been infected with ransomware and all their data was encrypted, including their backups. The data was not recoverable because of the encryption, and the ransom was beyond what they could afford.

Moral of the story? Backups and protection are key. Never skimp on backups and be sure they are set up properly with an onsite and offsite copy that is firewalled from the regular network.

#3 Invest in cyber liability insurance

We consistently recommend cyber liability insurance. Businesses insure against fire, flood, and theft of property. Based on prevalence, cyber-attacks should now be listed among those sorts of catastrophes.

Cyber liability insurance is extremely valuable and, in the grand scheme of things, pretty affordable. Consider the astronomical cost of getting attacked: loss of business, forced shutdown, frustration, and paying for IT help (not to mention the financial costs incurred by paying a ransom). It can be crippling if your data is encrypted. Several days may pass before you can get your network running again. You may even need forensic help to get back online, investigate whether your data was stolen, and prevent further attacks.

Bottom line: If (or when) that happens, cyber liability insurance is a small price to pay for protection.

#4 Look into Endpoint Detection and Response (EDR)

Don’t confuse EDR with antivirus protection. Antivirus software can detect known threats and prevent the installation or deployment of known viruses. EDR can detect variants to patterns in both software and user behavior.

Let’s say Joe’s computer typically downloads 100MB a day from the internet. One day it reverses and uploads 100MB to the internet. EDR will see that as suspicious and flag it.

In our effort to stay at the forefront of cyber-attack prevention, CIO Solutions now offers CrowdStrike, a very advanced EDR tool. A cybersecurity forensics firm we work closely with thoroughly vetted it as a best of breed solution.  As recently as a year ago, the program was outside most organizations’ budget, but today it’s far more affordably priced. Are you a current client of CIO Solutions with questions about CrowdStrike? Don’t hesitate to ask.

#5 Enable Multi-Factor Authentication (MFA)

You’ve probably gotten used to the number of websites these days that won’t let you in with a plain old password. Your bank probably also texts or emails you a security code. You might even have an application on your phone called an Authenticator app with rolling codes that you have to enter to log in.

These are all examples of MFA.

Your business ought to implement MFA on key applications as well. This tool has quickly become a standard in the evolving security landscape. Even if someone DOES get your password, it is useless without the other authentication factor. The second piece to grant access is the security code that will only come up on your phone (which they don’t have). We highly recommend this.

Don’t put off to tomorrow what you can do today

The bad news: hackers will always be a threat.

The good news: there are effective ways to protect yourself, but you have to deploy them now.

Armed with that information, how will you begin protecting yourself from ransomware, phishing, and data infiltration?  How can we help?

Microsoft Exchange 0-Day Vulnerability Incident

Microsoft Exchange 0-Day Vulnerability Incident

/in ,

As you have likely seen in the news, Microsoft Exchange servers were the target of a 0-day vulnerability attack that resulted in (and continues to produce) thousands of compromised accounts and ransomware infections. What happened in this attack and what steps did CIO Solutions take to protect our clients?

Incident Background

On March 2, 2021, Microsoft publicly reported that they had “detected multiple 0-day exploits being used to attack on-premises versions of Microsoft Exchange Server in limited and targeted attacks. In the attacks observed, the threat actor used these vulnerabilities to access on-premises Exchange servers which enabled access to email accounts and allowed the installation of additional malware to facilitate long-term access to victim environments. Microsoft Threat Intelligence Center (MSTIC) attributes this campaign with high confidence to HAFNIUM, a group assessed to be state-sponsored and operating out of China, based on observed victimology, tactics, and procedures.”

According to cybersecurity firm, Volexity (which Microsoft credits for identifying initial exploits) hackers began stealthily targeting Exchange servers “in early January.”

After publicly reporting the vulnerabilities on March 2, Microsoft released security patches for multiple versions of Exchange. These patches were “out of band”, meaning they were released outside of typical launches in urgent response to this threat.

This was a necessary step for mitigating the impact of the attack. Unfortunately, the exploit was in use before the patch was released. Additionally, this attack was sophisticated and novel. In late February, just before Microsoft publicly released its security patch, security researchers observed an automated second wave of attacks. These targeted victims across industries.

CIO Solutions’ response

Upon determining the severity of this attack and consulting with our Information Security experts, we immediately began implementing measures to mitigate the impact of the attack. The day the security patch was released by Microsoft, we instated emergency patch windows outside of our standard patch schedule. Our engineers worked tirelessly over the next few days and the weekend to ensure that all necessary steps were taken to protect our clients.

Due to the timing and severity of the attack, the patches alone are not able to eliminate compromise. The unique nature of this threat was not yet understood and detected by Antivirus software. In addition to the recommended patching, we also installed additional threat monitoring tools and implemented mass password resets across user, service, and administrator accounts to clients potentially impacted by this vulnerability (Note: Only clients with on-premise Exchange servers were impacted by this incident.  Clients in CIO Cloud and CIO Hosted Email or Office 365 were not). Once we learned that a specific Antivirus software evolved to be capable of detecting the compromise, we also deployed this software.

Support Implications

With any major security threat, maintaining productivity and security is a balancing act. Our team worked diligently to ensure that we were at the forefront of implementing these proactive security measures. Our goal is always to protect our clients while at the same time, continuing to provide the same level of support and responsiveness that our clients expect.

Please be aware that with this increased demand on our teams, our ability to respond to more routine support requests may have been impacted. We appreciate your understanding and cooperation as we continue to work through the challenges posed by the ever-evolving threat landscape.

For additional information on the vulnerability, please see the resources below: