It may not have been someone in city government clicking on something “bad” in their email.

Nor was it likely a “targeted” attack because most ransomware is automated and simply scans for vulnerabilities.

But once the attackers were into Baltimore’s network, they knew what they were dealing with – a dangerously ill-prepared, kludged-together municipal IT system.

“Fixing this is going to require a view across all the city’s agencies and a reckoning with 20 years of crud – information technology crud – that is now there,” says tech journalist and analyst Sean Gallagher.

Gallagher spoke with The Brew about the Robbinhood ransomware attack, which hit two weeks ago, that has shut down email and many other functions of government – including citywide real estate transactions – and raised questions about why the city was so vulnerable to the attack in the first place.

“Things that have not been fixed, things that have not been changed, things that expired long ago, things that have been patched haphazardly,” says Gallagher, IT and national security editor of the online technology journal Ars Technica.

“There are things I’ve looked at across the city’s systems that look like they were summer projects by interns,” says Gallagher, who lives in the city.

Baltimore’s chaotic jumble of operating systems was full of opportunities for Internet criminals.

Gallagher has a perspective on the muddle of different technologies the city uses to pay bills and get services both as an IT expert and as a citizen.

Trying to book a pavilion at Druid Hill Park for his daughter’s graduation party was an exercise in frustration that involved phone calls, two different computer systems showing conflicting information about availability and, ultimately, a drive across town to a city office in Canton.

“I don’t think the city has a full handle on its vulnerability management and patch management and keeping up to date with things,” he says. “I don’t think the city has a full handle on what software is running on its networks.”

For Internet criminals, this chaotic landscape was full of opportunities.

“There are so many different ways this could have come into Baltimore city government,” says Gallagher, who discusses the attack – and the city’s susceptibility to it – in the Q&A that follows.

A couple of major points emerge from Brew editor Fern Shen’s conversation with Gallagher:

• The city’s IT is a mishmash of services running on old operating systems or using unsupported software or using new software that is not well understood across city agencies.

• Maintaining so many Internet-facing servers makes Baltimore highly vulnerable to being scanned by anybody in the world – anybody who has a computer.

• Baltimore spends less than half of what other similar-sized cities spend on IT and very little of the budget is controlled by professionals at BCIT (Baltimore City Information & Technology).

“I don’t blame the BCIT folks because I think they’re overworked and underpaid. And I think the office is dramatically underfunded.”

He points instead to the upper levels of government.

“I think the city leadership is totally clueless about this stuff. I think Mayor Young demonstrated that in both of his press conferences,” he said, noting comments that Young made in the wake of the attack about how, while they are idled, city workers can clean up city streets.

“Such an attitude belies the fact that there are critical services that citizens rely on – revenue-generating services, essential services like real estate transactions, services like fixing housing violations, sewer backups, water main breaks, non-functioning traffic lights – that have been partly or wholly idled by this.”

BREW: So it all started on the DPW website?

GALLAGHER: Probably. This particular malware, Robbinhood, requires something else to already be in the network. It’s not like somebody clicked on something bad in their email or opened a word doc and it spread across the whole network.

This is a piece of ransomware that is built to be spread by someone with administrative access to the network. So it means that somebody had to get into probably one of Baltimore city servers.

“This is a piece of ransomware that is built to be spread by someone with administrative access to the network.”

Given that Public Works was the first to notice something, it was probably DPW. And from there they were able to use the level of access they got from whatever vulnerability they found.

Does that mean that somebody hacked into DPW?

It was probably automated. But, yeah, somebody took advantage of something in the configuration of the server of Public Works or some other back-end server at the city. It’s also possible that someone at DPW had their username and password stolen by a “phishing” attack. I don’t have a lot of insight into the architecture of how the city sets up its server farms, but I get the impression they’re all pretty much on the same network so once you got in, you were in.

Based on what researchers I’ve talked to have told me, I believe somebody would have had to get into the network with some sort of other piece of bad software that got into the network either through phishing or they more likely found a vulnerability in the server or some other Internet-facing server. Baltimore has a lot of stuff that’s directly on the Internet.

Versus what?

Versus being hidden behind a firewall. Because of the structure of the city’s network – Baltimore City owns this giant set of Internet protocol addresses – it’s what they call a Class C address. It’s several hundred thousand Internet addresses that have been provisioned by the ICANN [Internet Corporation for Assigned Names and Numbers], and it’s not clear how many of those addresses are used to directly connect to the Internet.

But the thing is, most of the city’s email, web services and email and things like that are hosted on computers within the city’s network, so they directly connect to the Internet. And that means that they are vulnerable to being scanned for problems by anybody in the world, anybody who has a computer.

Was this a flaw in the set-up?

It’s hard to say. One of the things that’s easy to say is that despite [OIT Director] Frank Johnson saying the city has “had audits and passed with flying colors,” I don’t think the city has a full handle on its vulnerability management and patch management and keeping up to date with things.

I don’t think the city has a full handle on what software is running on its networks. And just from personal experience having to deal with city services, it’s a jumble of different technologies they use to run things like paying your bills online and getting access to service and things like that.

Many of those services, I can’t tell how well they’re supported because it seems like pieces of them are really dated.

We know that the city police department, which just completed its audit mandated by the consent decree, was shown to have a lot of problems in terms of how up-to-date and well maintained its IT systems are. And it is one of the best-funded departments in the city.

How did the ransomware get in?
There are several possible ways that the ransomware got into the city’s network. Two are most likely: Either the attacker found a software bug on a city server as part of a mass scan and used that bug to gain a foothold on the server. Or a city worker’s login information was stolen through a phishing attack and those credentials were used by the attacker to connect to the network. Once the attacker was in, they could have used existing software on the network to spread the ransomware.

Because of the way the ransomware was written, it had to be installed separately on each computer it attacked.  When Robbinhood malware gets onto a computer, the first thing it checks for is if there is an encryption key that’s on the computer.

“Once the attacker was in, they could have used existing software on the network to spread the ransomware.”

That had to be put there by something else. And then when it sees the encryption key, it then runs, trying to turn off any anti-virus software and other security software that would detect what it was doing, and disconnecting the computer from the rest of the network before it encrypts all the files on the computer.

And then, once it’s done that, then it starts encrypting files.  So each individual computer would not have been able to go out and encrypt files in other computers. The malware had to spread to each individual computer.

Based on the information I have so far, It looks like the servers got targeted first.  The communications were hit, and digital payment services and databases were hit. All their email went down.

So it went after the heart of the network immediately. (And because the city runs everything out of Windows, that left the phone system and email and all of these  applications the city uses to do e-billing and things like that, vulnerable.)

I don’t know how far it spread because few details have been shared. There’s been very little transparency in what’s going on.

Any other potential explanations for how this happened?

Another possibility is a thing called a remote desktop protocol [RDP] that allows you to log into a Windows session remotely.

The idea is that I can get remote access to a full Windows session through this software over the Internet that allows me to be on the network of that computer.

If I had to guess, I would say one of the most likely ways that this got onto Baltimore’s network was that there was an RDP server or something like that that was being used for remote maintenance, and there was firewall opened that allowed somebody to get to it.

Somebody found it and got in using simple credentials, then was able to escalate from there and get access to distribute the malware.

What might have repelled the attack or even prevented it?

Two things. First, they would have had to understand what their vulnerabilities were. And second, patch their software to prevent it from happening.

From what I’ve seen with previous ransomware attacks, this is not something that somebody developed “a zero day” to do. In other words, it’s not something where they said, “Oh, I’ve got this new hack that I’m going to use on the city of Baltimore for the first time.” Especially because they’re only asking for like $70,000 to get the keys.

This was not a targeted attack. This is not something that was aimed at Baltimore City because somebody decided they wanted to go after Baltimore City.

So the attackers cast a wide net and we turned out to be vulnerable?

Yes. This is what happened with Greenville, N.C., which got hit with this same malware a few weeks ago. And there was Atlanta a year ago. San Francisco’s BART system had it happen a year and a half ago.

Baltimore’s 911 system got hit last year because they had a firewall disabled for a couple of hours during maintenance. Somebody scanned for vulnerabilities, found them and hit them. It wasn’t even like they had a longstanding vulnerability. They just turned off a firewall setting for four hours and they got hit with ransomware.

This is part of the problem with ransomware. If you don’t have a particularly vigilant security team going through and looking for vulnerabilities in your system and patching them aggressively, you’re going to get hit.

The scammers look for vulnerabilities in different types of servers – there are well-known vulnerabilities – and once they find them, they drop this malware on them. And this is going to continue to be a problem for many small and medium-sized cities across the country – and larger cities.

Should last year’s attack on Baltimore’s 911 been a wake-up call?

You would have thought, yes. The problem is that doing the things that need to be done to keep the city safe from ransomware are labor intensive and expensive.

And the city has many old systems that it just keeps maintaining and running that may even not be supported anymore by the software developers that made them.

There’s a lot of custom software that was put in place to run different software applications across the city. There are things I’ve looked at in the city that look like they were summer projects by interns.

I’ve looked at their infrastructure to see if there was anything indicative of what caused this. There are so many different ways this could have come into Baltimore city government it’s insane.

What was learned from other ransomware attacks?

Look at the one that hit Medstar, the hospital chain that includes Union Memorial in Charles Village.

That happened because there was a piece of software that patients use to access billing and records that was running using an older version of a Java server that was known to be vulnerable. It was something that was built into the software that they got. It might not have been on their list.

They probably were not aware they were vulnerable.  It was a third-party product that used an open source product that Medstar didn’t directly interact with.

This is a reason why one of the things going on in the computer security field is the demand for companies to do big systems to have what amounts to a list of ingredients, a bill of lading of what software is included in their solutions so that they can track these things.

I don’t know if Baltimore City has really done anything to inventory what’s on their servers.

Is the city’s outdated operating systems a big part of the problem?

Sure. Look at the Baltimore Police Department. They use Lotus Notes, which IBM stopped supporting almost a decade ago.

I don’t know what version of Windows is used citywide. I’m assuming they’re up to Windows 7. Which is the baseline for government at this point. Anything older than Windows 7 or Windows server 2012, it’s not supported anymore.

Hasn’t Baltimore been making a push to adopt “Smart City” technology?

Yes, they have been talking about that, putting more and more sensors on the network to track traffic and things like that. But all of those are also potential points of entry for an attack because they have operating systems that may, or may not, be patched up.

In other words, there are many, many doors to get into the city network.

Was there a way to detect this attack sooner?

The question to ask the city is, “What did you audit when you say you got these security patches that passed with flying colors – what exactly was audited? What did you have in place to deal with this as a response?”

If they had a security operations center where they saw, suddenly, these computers trying to shut down a number of services, that should have been a red flag. They would have had error messages.

If they had a security operations center where they saw, suddenly, these computers trying to shut down a number of services, that should have been a red flag.

If they had somebody monitoring the logs and they were aware these critical systems connected to the Internet were all of a sudden trying to shut down – antivirus trying to shut off file shares – that should have been an alert to do something.

Have you tried to discuss these issues with the city?

I’ve tried to talk to the city in the past about its systems. I’ve gotten a lot of dead air because there’s been no leadership to talk to, really. Honestly, I don’t have a high level of confidence in the city’s handle on its IT systems from personal experience.

Here’s an anecdote: My daughter is graduating from high school. I wanted to book a pavilion at Druid Hill Park for a party. So I go online to see how to go about it. The website is a mishmash of things and gave me availability for things like city garden plots, but it wouldn’t give me availability for whether I can apply to get a pavilion.

I finally found it. The search within the site was broken. So I used Google to find out how to get a reservation for a pavilion. It directed me to an email address. I got an email back response that said, “We don’t check emails,” and gave a physical address to go, which turned out to be an address they had moved out of two years ago.

I called them and asked if there a pavilion open on June 1. I’m now gonna drive down to Canton and apply for one if there is, because that’s where their office is now.

So she looked at her system and said, “yes, there are still some available.” So I drove down there and spoke to another woman who told me, looking at a different system, “Oh, no, there’s a 10 miler race in the park that day.” So it was closed.

To summarize, what’s the biggest problem with Baltimore’s IT?

There’s not been a whole lot of thought about the architecture of the systems. I don’t blame the IT folks because I think they’re overworked and underpaid. And I think they’re dramatically underfunded.

But I think the city leadership is totally clueless about this stuff. I think Mayor Young demonstrated that in both of his press conferences. He made comments like, if employees are idled, maybe I’ll just ask ’em to clean up the city streets.

It belies the fact that there are critical services that citizens rely on – revenue-generating services, essential services like real estate transactions, services like fixing housing violations, sewer backups, water main breaks, non-functioning traffic lights – that have been partly or wholly idled by this.

It seems like city leadership hasn’t yet grasped this fundamental fact.

PART 2: What will it take to transform Baltimore’s anemic IT?