Smart Grid: New Views on Countermeasures

The evolution of the Smart Grid, and most notably the introduction and expansion of IP-based internetworking into the traditionally SCADA, and often proprietary, world of utilities is bringing a new series of threats to the Grid, as we have been writing about for some time now. Much of our blogging in the past year has focused on it: on the threats, on the similarities, and on the insufficient attention we believe the problem has received.

Recently, as we have been working to better understand the other side of this equation, the complex and unique SCADA systems that have long managed and monitored the Grid and its components, it has become clear that while these threats are definitely coming, it is not simply going to be a re-application of counter measures from the IP security handbook. The Smart Grid will be enabled by the new technologies, will rely on existing assets and resources, and will require nuanced thinking in terms of counter measures. Here are a few examples of areas where new thinking is necessary, and where existing IP cures could easily kill the Smart Grid patient:

Password Guessing and Identity Spoofing

Traditional IP Countermeasure: System access is blocked after a fixed number of failed login attempts are seen. Just like an ATM machine swallowing a card, or a retailer disabling an account, protection from brute force password guessing involves some out-of-band process to restore legitimate access.

Smart Grid/SCADA Reality: This countermeasure would actually be a powerful attack against a system requiring management or update. As an example, if attackers were attempting to overheat a component, they could first direct a series of expected unsuccessful login attempts, and then execute rapid powercycling or other destabilizing attack. A lock-out security mechanism would then prevent the real manager from taking appropriate actions, and would complicate any form of recovery.
In this example, a more appropriate countermeasure might be to require a second form of authentication after repeated failures, like a token or a biometric, or to throw an alarm while increasing the time between login attempts. A more nuanced behavior is required that would never go so far as to lock-out access to a critical system or service.

Viruses and Malware

Traditional IP Countermeasure: One of the most common security mechanisms that is deployed across a majority of critical systems is antivirus or anti-malware software. These solutions scan systems in their entirety, identifying files, memory, or running objects that look to be dangerous to the system.

Smart Grid/SCADA Reality: Advances in platform capabilities and cost savings have combined to increase the number of standard, off-the-shelf systems that play a role in the provision or management of power operations on the Smart Grid. These systems, however, are not natively any more protected from viruses and malware than a typical commercial or residential system. They do differ, however, in the tight requirements that exist for their performance, and in the likelihood that they are already maxed out in terms of processor load. As a result, it is unlikely that traditional antivirus or anti-malware software could be supported on those systems without negatively impacting other areas of performance.
Antivirus and anti-malware software will likely have to operate at gateways in the new Smart Grid, and behavioral analytics of network behavior may be required to identify the introduction or spread of viruses or malware internal to the provider.

System Breach and Compromise

Traditional IP Countermeasure: Intrusion Detection Systems, either host or network-based are a common means of identifying incursions, permitting both early interdiction and detailed forensic analysis in the case of breaches.

Smart Grid/SCADA Reality: The essence of successful intrusion detection is the capability of identifying well-known patterns of attack and of describing expected acceptable patterns of operations. The complexity of many SCADA systems and their individuality, combined with lesser-known protocols and dynamic configurations, make identifying intrusions very difficult. The traditionally overworked systems are also poor candidates for host-based systems, and the equally overworked utility staffs are not commonly trained in identifying and executing remediation steps for attacks once in process.
Monitoring and intrusion detection are clearly vital to the success and reliability of the Smart Grid, but it is likely that other forms of corruption detection, like executable and data integrity validators, baseline system monitoring, and network anomaly detection, will be less invasive and disruptive solutions.

As we continue down this road towards a new Smart Grid, and an even newer Soft Grid, we will need to establish a relationship between threat and countermeasure that is appropriate to, and designed for, the fragile and vital world of power management. There will be need for new solutions, and for customizing existing favorites, but the first step is getting one foot in each camp, and not locking ourselves rigidly into existing utility or IT security mindsets.

Photos Courtesy of:

http://www.flickr.com/photos/bigmikesndtech/CC BY 2.0

http://www.flickr.com/photos/ghewgill/ / CC BY 2.0

The Smart Grid Security Confidence Game

You may not know Ira Winkler from a hole in the wall. Or know the difference between Ira Winkler and Henry Winkler ... heyyy !!! But you should know this: Winkler, a well-known and generally well respected IT security pundit, recently published an opinion piece in Computerworld called "The hackability of the smart grid". In it, he lists six types of trouble he claims hackers will likely be able to cause to the future "Smart Grid":

• Cutting electricity to homes and businesses
• Overburdening the grid
• Causing brown-outs
• Having smart-grid devices attack the grid itself
• Getting free service
• Undermining confidence

To be sure, while all of the above are plausible and serious, it's the last one, related to confidence, which could ultimately have the biggest impact on Smart Grid operators and other stakeholders.

It seems to me like there's a vacuum out there where only pundits dwell. Wouldn't it be excellent if some of the utilities could be more forward leaning and get out in front of this issue?!? Messages on the thorough measures they're taking to protect the grid and their customers might help. But so far they're not saying much, and until they do, folks like Winkler are the ones exhibiting their confident predictions that rough days are in store for the young Smart Grid and that the utilities are playing marketing defense, not working to shore up their security and privacy weak spots with vigor.

Says Winkler:

The power companies don't like it when people say things like this, as they showed by attacking me after my previous exposé of power-grid vulnerabilities. So far, though, every claim I made has been proved correct by documented attacks or government reports. Sadly, I know that I will be proved right once again.

I hope he's wrong, but you know what they say about hope. Hope is not a strategy ... for protecting the Smart Grid. Let's hear it utilities execs ... we know you're busy working these issues, but please take a minute to tell the public what you are doing to prove Winkler wrong (Ira, not Henry).

Photo Credit: Igor Bespamyatnov @ Flickr

Is International Collaboration in the Cards for the Smart Grid?

There are currently Smart Grid conferences, planning committees and pilot deployments happening on every continent except maybe Antarctica. Yet most everything I've read to date concerns work being done in the US. I can tell you, however, that many of the readers of the Smart Grid Security Blog are from Europe and Asia. I can also relate that after moderating a Smart Grid panel at a recent clean tech conference in Boston, I was approached by a gentleman who wanted to ensure I knew about a big RFP coming out soon to build a Smart Grid for the city-state of Singapore. (Here's a link to a conference that just took place there.)

So, with that said, here's a short post on the international angle: le Smart Grid. Warning: if you favor answers, this post is light on them and chock-a-block full of questions. Here's a few starters to get us started:

• Will the fully deployed Smart Grid have borders?
• In North America, will the Smart Grid eventually transcend the current regional topology of Regional Transmission Operators (RTO's) and Independent Systems Operators (ISO's)?
• While the electrons that constitute my emails transit the continent (heck, most of the globe) with ease, the same cannot be said for the electrons currently bringing my monitor to life. Will the Smart Grid change this?
• Is there anything the US can learn from early international efforts in Europe, where Germany was a first mover?

According to this recent article from Smart Grid News, seems like current thinking, in the US anyway, may not be very collaborative ... at least not as far as security is concerned. Here's a recent statement from a Canadian Electricity Association (CEA) VP on how current Smart Grid security legislation and standards make no mention of working as a team with our partners in the Great White North:

[The US has] got to realize that the North American grid is international, it's interconnected, it's integrated. Consultations, cooperation between governmental authorities on both sides of the border is going to be imperative, otherwise you won't be able to ensure system reliability and you'll probably undermine system reliability.

I realize my understanding of these issues is likely simplistic. Yet the ability to quickly "island off" healthy portions of the grid from unhealthy ones is key functionality every region and every nation is shooting for. But islanding should be an emergency response, not the square one status quo inside the US or among close allies.

Smart Grid Privacy Before Smart Grid Security

Can we have a little privacy, please? The question of how to secure a system isn't fully relevant until you've figured out what needs securing, and that often begins with policy decisions on how to manage sensitive customer data.

Here at the SGS Blog, our relentless quest for more and better info re: the state of security policy and technology for the Smart Grid sometimes has us overlooking things of a less technical, but no less impact-full nature, like privacy. As Katie Fehrenbacher of earth2tech puts it:

"Smart Grid security” is most often discussed in the terms of national security — a hacker develops a worm that can jump across smart meters and black out neighborhoods, for example, or can make a generator blow up remotely. Privacy — keeping personal information in the hands of the consumer and away from advertisers, the utility or any other third party — is an entirely different concern that utilities have to be prepared for with the build-out of the Smart Grid.

Yup. From the maltreated customers' point of view, be they large commercial or industrial enterprise or a simple household, it matters little whether their data is divulged via hacking or poor privacy controls. The simple fact that someone or some organization in a trusted position was less than fully responsible with their financial, health, behavioral (or other) info is more than enough cause to trigger a call to Attorney911. And media reports of privacy debacles will serve to greatly reduce confidence and enthusiasm for wider Smart Grid deployments.

So much depends on the customers' first experience with the Smart Grid and the amount of control over privacy decisions they are given. Here's draft privacy standard verbiage from Rebecca Herold, who in addition to being "The Privacy Professor," doubles as an energetic volunteer on NIST's Smart Grid Privacy Group:

Consent and Choice: The organization must describe the choices available to individuals and obtain explicit consent if possible, or implied consent when this is not feasible, with respect to the collection, use and disclosure of their personal information.

That sounds like a great way to begin a new relationship. Mutual consent. The freedom to say "I do" or "I do not." And why do I say "new" relationship? One thing we've learned in our recent travels through the Smart Grid universe is that the typical US utility has a less than stellar understanding of its customers. And the adverse is true: many utility customers cannot even name the company that supplies their electricity. If Smart Grid dreams do indeed come true, both parties will soon be on a first name basis. They're going to learn things they never knew about each other before. And if it's done right, they will come to trust each other with some very important information and services.

Photo Credit: NotionsCapital on Flickr

Smarter Grid. Struggling SCADA?

In June of this year, the FBI arrested a hacker named Jesse McGraw (aka "GhostExodus") for installing malicious software on a couple of systems at a hospital in Texas. He didn't crack some protocol or breach a server, he allegedly walked around in his security guard uniform and a "hoodie" with a USB drive carrying malware. An ultimate insider.

The entire episode can be found in a very readable account at the website of the somewhat eponymously named Wesley McGrew, who actually located and identified McGraw after a relatively short period of social network mashing, Googling, and just good, old-fashioned rational thinking. ( For those of you with eye-strain from concentrating on the Smart Grid Security Blog, there is also a very good podcast interview with McGrew by Michael Farnum at An Information Security Place.)

The story has been told in multiple places, and was widely covered in local media at the time, but in doing some research today on SCADA vulnerability and exploitability, there were items in the complaint, in the write-up, and in the comments (some of them quite scathing) from the hacker's cohorts to McGrew's account of the events, that made me think of the SCADA security challenges associated with the new Smart Grid environment in some different and more urgent ways.

What Once was Old is Old Again
It is not news that components of SCADA systems can be older and have been designed for reliability and stability on mainly protected networks populated with trusted people. In discussing his motivations for researching the attacker, and for calling the authorities, McGrew cites his current doctoral research in information security, particularly in SCADA security. When he discovered that the attacker had installed botnet software on a hospital HVAC system, his level of urgency shot up. He feared that even modest corruption of that system could cause real danger to patients, at one point referring to SCADA systems of the type as a sort of "rickety ensemble" of old and new pieces, which could not be expected to withstand much tinkering.

He is not alone in this expectation. In a presentation back in 2007, delivered at HITBSecConf2007 Malaysia, called "Hacking Scada", other statements supported this fear, including the fact that ordinary anti-virus software could be expected to crash many SCADA systems due to the increased load, and that simple utlities like "ping" had been shown to bring those assets down.

As an IT person coming to utilities, I had expected vulnerability, but did not expect the real fragility in these important systems.

HMI by DIY
I was also surprised to learn that many of the front-ends ( HMI or Human-Machine Interface systems) of these newer SCADA implementations are actually created on-site. Think of it as a Do-It-Yourself graphical user interface. This is necessary, in as much as most of them are actually doing extremely custom things. The presence of different sensors, different arrangements, different control structures, demand that the interface itself be created in a way that is very much tailored to the environment that is actually going to be managed.

I learned this while researching the new importance of the internet protocol and even web-oriented interfaces, as components in the HMI interfaces of these systems. Packages actually ship with IDEs (Integrated Development Environments) containing libraries and widgets necessary to create useful, functional, and hopefully intuitive representations of the complex system of sensors, RTU's, PLC's, and more. It is not clear how seriously security is regarded in the creation of these custom interfaces, or how simple it can be to enable security controls available through the IDE's. It appears that there exist few standards and fewer tools relating to their certification.

Getting Warm in Here?
As it was with attacks and breaches in the early days of the Internet, the facts surrounding the means of identifying the actual attack and attacker are discouraging.

Based on the reporting from the hospital...which existed in Texas...in summer...it's hot there...the air conditioning system had failed multiple times, and they didn't check for, or find, the remote control software on the HVAC system. Instead, a researcher hundreds of miles away had gotten an unrelated message from a hacker, did some research, and discovered from pictures of the HMI screens that the system had been corrupted.

Admittedly, information security may be relatively new to the traditional SCADA user, but there needs to be better tooling, or better integrity assurance, or just better education and awareness to make some information security analysis more standard.

IT Hacking Ignorance
It could be that the most dangerous reality of this article could be summed up in the uninformed actions of the attacker, and the reactions of others to his arrest. The malicious software that was delivered through a USB drive into an exposed USB port, was a botnet, remote control software, and the attacker was planning a "massive" denial of service attack from all of his controlled machines.

I think it is pretty clear that this guy did not know how unstable this system would become, or how important HVAC is in a hospital in Texas. Operating room environmentals, pharmaceutical storage temperatures, patient recuperation, are all intimately connected to those systems. It is literally life and death. It is hard to imagine from the descriptions of the attacker and his attack that he construed his incursion as being as dangerous as it was. Similarly, the ignorance of many of the comments on his arrest miss this entirely, presenting their view of the attack as being that he "hacked an air conditioner or something".

Whether it be in the minds of the internal resources who do not think about information security and an HVAC system, or external attackers who do not understand the complexity, seriousness, and importance of these newly interconnected SCADA systems, the fundamental disconnect on action and effect need to be made much more visible.

The reliance of SCADA-enabled systems like HVAC on their actual software, and the reliance of the utilities and customers on these SCADA systems is a connection that is becoming obvious as the Smart Grid expands the number and the exposure of these systems to all.

Images Courtesy:

http://www.flickr.com/photos/borkurdotnet/ / CC BY 2.0

and

Wesley McGrew

Seeking a Balanced Perspective: How Cyber Risks to Grid are and are not MAD

As you may suspect by now, Jack and I are not fans of alarmist language. You won't hear us using terms like "Cyber Pearl Harbor" or "Cyber 9/11" unless our purpose is to debunk them, as Jack did quite thoroughly on his former blog, Suitable Security, here. We find that hysteria is not a particularly promising state of mind to be in when one is attempting to make the world better, safer and more secure. And that's the lead-in to this second post re: the recent 60 Minutes feature on ominous trouble in Cyberland.

Oh, one more thing before the post really starts -- I should explain the kitten. This kitten is here to help you relax. OK? Let's begin.

MAD, or Mutually Assured Destruction, is a Cold War-era term which neatly describes why nuclear deterrence works and has so far kept our planet from being reduced to a glowing ember from a massive thermonuclear exchange. You are still relaxed I see ... that's good.

Last week we posted a link to, and a couple comments on, an alarming 60 Minutes episode on cyber security risks to critical US infrastructure. It described how vulnerable the US is to computer hackers and used examples from DOD, the financial sector and the electrical grid. An additional level of disturbing detail was provided by former Director of National Intelligence (DNI) Mike McConnell, who said he's certain that foreign code is resident on national grid systems. Our own anecdotal experience with critical systems in other industries corroborates this. In hacker lingo: we are "owned."

Still relaxed? You should be, because there's ample evidence, in the 60 Minutes material and elsewhere, that even as we are heavily targeted, we also have substantial penetration of our potential adversaries' systems. Hence, the resemblance to MAD. I'm making this comparison preemptively before some journalist or K Street analyst does, because I think it's worth laying a few of the cards on the table and thinking about this in a non-alarmist fashion. Here's a short list of attributes to compare and contrast:

Nuclear characteristics:

• Once underway, nuclear war is for keeps: you're either launching nukes or you're not
• Though some once believed in it, "limited nuclear war" is generally considered unlikely
• While we work to make missile defense a reality, our best defense against nuclear attack has been a good offense (see: deterrence)
• Damage from nuclear exchanges is usually believed to be catastrophic
• With missiles and bombers heading our way, it's fairly easy to discern the origin of attack, and hence, the attacker
• There are currently 9 countries listed as nuclear nations. Others seek to join this group, but it's expensive, complicated and time consuming, not to mention dangerous and sometimes destabilizing

Grid Cyber characteristics:

• Probes and attacks are happening all the time by multiple parties and damage of various degrees is being absorbed by all involved
• All cyber war is, by definition, limited
• Our best defenses are multi-layered, resilient and constantly evolving
• Damage is infinitely variable in severity and often hard to detect
• Often cannot identify attack origin or attacker
• Any country, organization or individual with access to the Internet can be an attacker

So the Cyber wars are already well underway and yet you are still able to read this post on your computer or smart phone. This is because given the degree of inter-dependency of the global economy, most industrialized nations have little desire to wreak massive cyber havoc on their neighbors, who, while they compete in many domains, are also full time partners. Though you'll sometimes hear speculation to this effect, especially as it concerns the Smart Grid as a "hackers' paradise", it's unlikely (though possible) that catastrophic harm can befall the diverse US national grid from cyber attack alone. But that doesn't mean major localized or regional damage couldn't be wrought.

Take aways:

• Unlike with nukes, where deterrence between nuclear nations has worked so far, no one is fully deterred from experimenting with and sometimes wielding cyber weapons against our grid or other critical US infrastructure systems. Most nations do, however, seem deterred from launching massive cyber attacks on us and others ... and life and commerce go on
• International crime gangs and other non-state bad actors abide by completely different rule sets from those described above. Deterrence means much less to them, so we've got to continue to bring our cyber security "A game" to the Smart Grid build out as well as to the rest of our critical national infrastructure
• Understanding and accepting that all sides "own" other systems conjures up the alternative title to the Cold War classic "Dr. Strangelove," which was "How I Learned to Stop Worrying and Love the Bomb." I'm not suggesting you begin loving cyber risks to the grid or Smart Grid; just want you to worry a little less if the 60 Minutes piece has rendered you sleepless or immobile. Clearly we’ve got work to do, but as NASA and the NY Times said today, we’re not going to die tomorrow or the day after tomorrow
• For a somewhat more detailed, balanced examination of cyber risks to the grid, see University of Minnesota's Dr. Massoud Amin's short paper "Electricity Infrastructure Security", PDF downloadable here.

So, if you've made it this far, I've got a question for you: did the kitten help?

Photo Credit: Catherine Caf on Flickr

Smart Meter Increases "Suit" Pacific Gas and Electric

On November 16, 2006, at a lucky customer's home in Bakersfield, CA, PG&E launched its SmartMeter program, designed to alleviate costs for customers, costs for supporting the power grid, and the cost of generating so much energy in the area. Even the commissioners were optimistic, as reported in a PG&E press release:

"I am pleased to witness today the installation of the first smart meter for a PG&E customer," said Michael R. Peevey, president of the California Public Utilities Commission. "This technology will link the prices energy consumers pay to the costs of that energy in the wholesale market, empowering consumers with the information necessary to make sound energy choices. Research suggests that even modest levels of price sensitivity in the retail market can yield substantial benefits as customers decrease or shift their energy usage. These types of demand response programs are one of the best ways to meet the energy needs of California's growing population, as outlined in our Energy Action Plan."

It is hard to know exactly when the honeymoon ended, whether it was when Bakersfield.com reported on a customer who found his power usage had tripled during a six-hour blackout, or at the town meeting in Fresno on October 20th which quickly became a unanimous indictment of Smart Meter-ing, or now in November, as a class-action suit has been filed against PG&E, asserting a variety of mistakes and misrepresentations. For those of us who have spent a fair amount of time researching the potential for advances derived from Smart Metering, these developments are disconcerting.

From a security perspective, there are two very important areas of guidance to take from these developments, and from the likely continuing negative perception of Smart Metering in some areas.

Integrity and Availability of Data

As we wrote here, and as others opined elsewhere, there is likely an abundance of information about to flood utilities. Some have rejected, or at least resisted, the idea that anything like high volume sampling would happen, and that aggregated data would be the more probable artifacts that utilities would store for billing and management. This suit and the ongoing outcry for justification of higher bills are exactly the reason why more detailed and regular metering information will need to be gathered and stored.

See, it is likely that these bills are actually accurate. As the commissioner stated at the outset, "modest levels of price sensitivity in the retail market can yield substantial benefits". Ok, so maybe the hot tar and chicken feathers are not necessarily a benefit, but they highlight a new awareness on the part of the consumers. It is surprising that this message of usage and contention for power has not been better absorbed by the public. Take an average citizen. They use power, like everybody else, from 8-6. Enter the Smart Grid, and the smart meters. In an attempt to incent off-peak usage, and to compensate for the increased cost of peak generation, power is more expensive from 8-6, and so the average consumer's bill, if they do not change their behaviors, is going to be higher. The smart meter only becomes an engine of positive financial impact for consumers when they figure out ways in which to really alter their power use to advantage the off-hour charges.

Until that happens, expect that there will be continuing challenges to the veracity of the smart meter data, and continuing scrutiny of the systems that collect and store it. This equals what we described in earlier posts, a need for lots of data, lots of governance of that data, and good security from authenticating the user to authorizing the billing.

Actual Smart Meter Opponents

Any publicly-perceived inequitable grab for cash by a business or utility can spawn a grass-roots movement in opposition. Ignoring the more fringe folks who bring you the youtube videos of jack-booted thugs monitoring your hot-tub to charge you with profligate energy spending, there are others who are more credibly mobilizing around this issue. An example is San Francisco-based TURN (Toward Utility Rate Normalization). With a 35 year history in utility consumer advocacy and activism, the have a new focus on the perceived inequity of a smart metering infrastructure that saves costs for utilities (better management, less truck-rolls, easier disconnects) while increasing the actual bills for consumers.

With group action, and organized effort, there comes increasing visibility and controversy around the issues, and there are likely to be more critical assessments made of Smart Metering infrastructures. This will naturally splash as well onto the overall Smart Grid approach of which smart meters are such an important part. With any such increase in visibility and controversy, individuals outside the credible groups may well begin to conspire to take more aggressive action, potentially creating a new wave of "hacktivism", with the focus in this cycle being the Grid. This will change the nature of the threat to the Smart Grid enormously, making it much more likely to experience the types of attacks that more typically plague governmental and military infrastructures.

Some of the Solution is in the Data

Many of the same constituencies who are actively opposing the Smart Meter evolution are also very much interested and involved in the promotion of more efficient energy usage and more integration of alternative sources. It is now the responsibility of the utilities to educate their customers about the actual dynamics of power and power pricing, to help them to better understand the choices that they will need to make.

For those utilities who have not yet begun to alter the finances of their customers through higher peak pricing, there is a cautionary tale here. It seems that it might well be worth 3-6 months of reporting on usage, with simulated billing and recommendations for changes, prior to actually instituting those changes. It would better showcase the insight provided by Smart Metering, would provide a sense of empowerment for the users, and would certainly eliminate some of what seems to be a sense of blindsiding on the part of the consumer.

Image thanks to the whimsical stylings of Roger Wood

Smart Meters as Rough Yardsticks

In reading through the successful Grant recipients from the Smart Grid Investment Grant Program, it was interesting to make a couple of notes:

1. Smart Meter Roll-out
   

   In the FERC's Demand-Response Paper from September of 2009, the number of Smart Meters currently implemented is roughly 8 million. Looking at the total of the specifically identified smart meters implemented as a result of successful SGIG requests, that number is now funded to get to a total of 18 million with the SGIG funding. That means that the SGIG will carry smart meter deployment to more than 20% of the FERC demand response projection of 80 million meters by 2019. Let's hope that the meters are chosen correctly.

   Per-Meter Costs
   There is enormous variability in the costs of the smart-meter roll-outs as described by the various grants. This is understandable in that the number of meters is only one criteria of many of these proposals. For some, these are an initial effort, for others they are scaling existing investment up. The meters, though, do loosely equate to the public involvement (connected by meters) that the SGIG is attempting to accelerate. As such the range and variety are worth noting.

   • 79% of grants expect associated costs of < $500/meter
   • 18% of grants expect associated costs of $500-$1000/meter
     
   • 2% of grants expect associated costs of $1000-$2000/meter
     
   • 1% of grants expect associated costs of >$2000/meter

So what does this tell us?
The information is pretty scant in the released SGIG award documents, but there are some insights, if not actual conclusions, that can be drawn from it.

1. Its about Usage
   According to the rudimentary data that is provided, Smart Meter-related projects are consuming by far the largest section of SGIG funding, and at least 85% of the total investment (SGIG and Utility/Vendor) expected for these projects. There are mentions of accommodating other energy sources, but the projects seem pretty focused on how power is consumed, and how that consumption is measured, as opposed to how it will be generated and distributed.


2. There is No Clear Standardization of Direction
   While these grants are providing the impetus for some organizations to begin work on Smart Grid infrastructure, the sheer size of them make the investment much more about rapidly scaling that adoption. Given that, and given the need to maintain stability in power, the projects themselves seem to be surprisingly one-off's, each intending to validate or optimize one organization's view of the new generation of Grid. As an example of this, take a look at the wording provided for two projects in North Carolina, from Duke Energy and Progress Energy, respectively
   

   [Duke Energy] Comprehensive grid modernization for Duke Energy’s Midwest electric system encompassing Ohio, Indiana, and Kentucky. Includes installing open, interoperable, two-way communications networks, deploying smart meters for 1.4 million customers, automating advanced distribution applications, developing dynamic pricing programs, and supporting the deployment of plug-in electric vehicles. Will also benefit customers in IN and OH. ($200,000,000 SGIG/$851,700,000 Total)

   and
   

   [Progress Energy]Build a green Smart Grid virtual power plant through conservation, efficiency and advanced load shaping technologies, including installation of over 160,000 meters across its multi-state service area. Will also benefit customers in SC. ($200,000,000 SGIG/$520,000,000 Total)

   It is hard to think of projects of this magnitude as test beds


3. Ready or Not, Here We Come
   From a security perspective, this is a massive investment in expanding the exposed surface of the grid, and it will impact a new generation of underlying communications infrastructure. Most of the synopsis data includes things like two-way communications, interactivity, new networking infrastructure, etc. That is a wholesale shift for millions of customers, and we continue to hope that people are putting hard thought into it, because those dollars will be spent, and we will need to reconcile the security one way or another.
   

I guess that last conclusion that I draw is that this program also tells us that even in these small-ish numbers, the costs are huge. Through either market forces or another wave of government investment, getting to the FERC's "partial adoption" could easily cost another $15B of government funding on this route, and another $20-30B in private investment. The numbers to get to a fuller adoption are far higher. From a security perspective, all of this continues to point back to understanding what is necessary within the new infrastructure, and what acquisition guidelines should drive these enormous purchases, because it will be impossible to unwind this once it gets moving.

The SGIG has put fuel into a very powerful and creative technical engine within the energy industry, and like an automobile, that power is generating speed. As that speed builds up, we need to see similar emphasis on keeping the headlights on so we don't crash on these unfamiliar roads.

60 Minutes Sounds Grid Security Alarm

Hat tip to my classmate and former Discovery Channel Powrtalk colleague Chris Davis for alerting me to the show that aired tonight. The popular news journal interviews former Director of National Intelligence (DNI) Mike McConnell, FBI Cyber Division Assistant Director Shawn Henry and others. It begins with cyber crime in the DOD world, goes through some real-world financial services industry examples, and concludes with conviction that the computers that run the Grid have been seriously compromised and that there's little the US government has been able to do to make private operators close out their vulnerabilities.

Remember, the subject here is the current Grid, the pre-cursor to the future Smart Grid, which will bring with it new types of additional abilities but also better ways of isolating some of them when necessary. The segment is called "Sabotaging the System" and you can watch it in its entirety right here, right now ... after a brief commercial, that is.


Watch CBS News Videos Online

Smart Grid Intro for CSO's

Having come to the Smart Grid Security discussion from the Security side of the equation, I have for years spoken at the highlight events, whether RSA, Gartner ITExpo, etc. This spring, when asked to present at CSI, I thought it would be a good opportunity that we could use to begin to bridge that IT and Utility security gap that Andy has written a fair amount on.

As such, last week I presented the following deck at the CSI IT show at the Gaylord National conference center, and it was meant to give just a taste of the Smart Grid to traditional IT security professionals, and to give some security information and guideposts to any utility folks that were there.

It turned out that we had representatives of both groups in the audience, and I have had several requests for the materials, mainly because these people wanted to begin the process of informing their own colleagues and managers. Be aware that it is intentionally light, it touches a few of the areas that are important, but it is by no means supposed to be an education on Smart Grid Security. It is more like the free chapter you would get if a book existed on the topic. Hopefully it was enough to energize some of these people who self-selected into the room and who are at least aware that there is a grid that is Smart, and there are security issues that may plague it.

Here is the deck. Please feel free to share it, and to generate a more aware population wherever you are. Andy and I expect to launch a version with voice-over in the next few weeks, so stay tuned for a truly simple way to get people to understand more about the nature of some of the challenges of securing the Smart Grid.

NERC Grid Security Update: On the Lookout for a New Order

This article is a bit jumbled, but it does communicate the gravity with which NERC CSO Michael Assante approaches cyber threats to the national grid. Quite simply, he views the threats to the grid and emerging Smart Grid to be something beyond what we've ever faced before. This from a recent panel appearance:

There was a known security rule set in the Cold War. We knew and expected behaviors. We could calculate escalation. We took this into any account when we planned any action. When cyber defenses and communications entered the military, it was a force multiplier. We appreciated what it gave us. What we didn't realize was that cyber would be the thing that destroyed the rules of order.

That last line really got my attention. We are just beginning to learn the new rules. But you have to be careful and alert. So many experts from other domains giving advice about how to secure the Smart Grid these days, pretending they understand what it's ultimately going to look like. When in fact, these are still the early days and, given the pace of technological change we've witnessed in recent years and decades, the Smart Grid of 2020 will look quite different than we imagine it today. Like Assante and NERC, all of us "good guys" need to make ourselves ready for what's coming.

Photo Credit: US Army on Flickr

Seriously - A Surge

A couple of weeks ago, I took a look at the data provided by the teams at PGE and Austin Energy, combined it with data provided by DOE, and I arrived at the conclusion that the Smart Grid will create a glut of information that the utilities had best begin planning for, because it could easily swamp both the utility and the networks that are expected to carry it.

Unsurprisingly, there was a fair amount of interest in both the conclusions I had reached and in the substantiation of the data I had used. Some of the inquiries were pretty straightforward. My thanks to Editor Katie Fehrenbacher from Earth2Tech for her thoughtful questioning and for introducing me to some equally reasonable experts from the IEEE.

Others were less open to the concept, and there were two main objections to the data. The first was based in existing utility practices. This line of questioning had within it the expectation that a meter read would only contain basic information about the identity of the power meter, the timestamp, and the meter reading itself. Were that the case, it would be possible that the data would be in a paltry range, around 14 bytes per read, resulting in a belief that such a small amount of data would never amount to anything like the avalanche I had described in the piece. The second objection was that there was little likelihood that such data was going to be stored for long, meaning, I guess, that we could design the system as though it had never arrived at all. Many of the questions came from individuals with strong/long histories in utilities, so I felt it my responsibility to validate, again, my data.

While I consider myself to be relatively well-versed on the core of these topics, it is the nature of this blog to focus on my expectations of the future based on information provided elsewhere, by others more directly in the path of the Smart Grid. That said, credibility is a big deal for us, and I decided to go back to Austin Energy, and understand better the reality of the situation from the folks who are actually doing the job, and who are considering these concerns as fundamental parts of their planning for successfully serving their clients on the new grid in the years to come. Andy and I called Andres Carvallo and Karl R. Rábago at Austin Energy, and they generously agreed to help us understand the world and the Smart Grid that they are planning for.

Smarter Grid versus Simpler Meter-Reading

One of the first things I learned was the richness of information gathering and interactivity that these gentlemen expect to coax from the new grid infrastructure. While time, location, and power used are at the heart of a meter read, there is much more to be learned. Investment in the Smart Grid would have a maximum return when the savings were more than a human reader's footwear and gasoline. Some examples are:

Device Health Information

By watching for varying temperature, periods since outage, battery power, heartbeat, and other meter variables, it is possible to better predict and recover from any failures that may happen.

Real Time Monotoring

As has happened historically with most new technologies, it can be expected that people yearning for more data will only be satisfied by that which is most current. It is unlikely to happen in the general population immediately, but history shows us that it is likely that such a real time monitoring feed may be in demand almost immediately, as customers recognize that there is now more information through which they can better manage their energy.

Energy Services Provision trumps Energy Provision Services

There are doubtless going to be additional requirements from the newly informed and empowered customer base for functionality that is logically delivered by the provider. This was a real eye opener for me, that Power Providers are now actively thinking about services that they can offer over the new and smarter infrastructure. Things like profiled energy use: "I am going away, manage my power." or "There is a spike in prices, manage me down by 10%", or "I only want to use power that is generated from renewable resources." These all require data, new interfaces, and a channel overwhich all of the control and monitoring information can be passed. Winners in the new market will be finding ways to capitalize on the need for energy-related services, and will not limit their investment to further driving down the costs of simply providing energy.

Networking Overhead

Given the complexity, regularity, and importance of this data, it is clear that a protocol (Like IP) will probably be adopted to package up and send all of this information in a payload to central systems for analysis, aggregation, storage, and action. Protocols carry their own overhead in terms of describing their content, sources, destinations, etc. None of this is free from the perspective of the systems carrying or storing the data.

Other Factors

We are only just beginning to see the potential for Smart Grid and Soft Grid enablers, leading me to believe that even my estimates are very likely to be low, particularly as we clamor for realtime monitoring and data analysis.

Based on all of this, it looks like the numbers are far from a simple 14 Byte read, and are more likely in the range given by Andres of 4K to 16K per reading. If we estimate the maximum case, the numbers are even higher than I had referenced in the earlier article. Let's not think about real-time (the numbers are mind-numbing), but instead look at a simple check every 5 minutes. 12 (reads/hr) X 24 (hrs/day) X (365 days/yr) X 16K (Bytes/read) yields roughly 1.7GB/meter/year. Multiply that by the number of meters (pick your own scope), and I think the challenge is clear. For more reality, take that number and multiply by 5 for readings every minute, or by 300 for readings every second. That's big.

So, is this a problem because the data going to cause the Smart Grid to explode like a flawed radiator hose in July? I don't think so. I think that time has proven that technical advancement has always helped us stay ahead of crushing data or processing burdens by decreasing computing and memory costs. This has allowed us to paper over our excesses with iron and silicon.

No, this is a problem because rushed, tactical, and incremental hardware adds will not make that data secure. It has to be expected that as organizations run out of room for data, they will simply rush to add more. Caught in a flood of data, the pressures for survival and successful operation will naturally trump any meaningful consideration of rearchitecting data storage for adequate and appropriate security.

This planning (and budgeting) needs to happen now. As Andres said on our call, "You cannot simply build an airplane for passengers who are 5'6" tall and weigh 140, because you can guess that your average passenger, much less your larger passengers, will simply not fit, because they are not that small." In other words, you need to plan for what you can reasonably expect, not for what will make your life, your business, or your CFO, ecstatic.

I think that this is the final insight. For firms that are seeing the Smart Grid as an enabler for cost-savings by transferring operations onto an IP infrastructure, or a wireless metering system, there is little reason to be concerned with a data glut.

For those who recognize that the Smart Grid and the coming Soft Grid will need data, and will need security, and will likely grow to fill whatever space is available, the call is clear. Plan for an avalanche, for a flood. Create systems and segregations that will allow for managing these flows reliably. Characterize what must come through, and what can be dropped, along the way to the back end. Do all of those things and the current systems will be fine, the next systems will not choke, and the ultimate end state will be similar enough to what has been planned to ensure stability, quality, and cost-effective services to all who connect to the grid.

The data surge is coming, and you can either surf it, or be pounded by it. You certainly will not be able to ignore it.

Image Thanks to:

http://www.flickr.com/photos/ndave/ / CC BY 2.0

Notes from 2009 Control Systems Cyber Security Conference

We first posted on Joe Weiss's work back in July following a presentation he gave to the Air Force. Now here's a great review of a significant annual conference, one that focuses not on IT or internet security in a Smart Grid context, but rather on the security issues related to the millions of control systems that automate the Grid. This is Joe's summary:

The Ninth Control Systems Cyber Security Conference was hosted by Applied Control Solutions (ACS) the week of October 19 in Bethesda, MD. The festivities started Monday morning with parallel activities. A tour was arranged of Washington Suburban Sanitary Commission’s Rock Creek water treatment facility. In parallel, the initial meeting of the ISA Nuclear Plant Cyber Security Joint Working Group was held.

The ACS Conference started Monday afternoon with two introductory sessions: Control Systems for the non-Control System Engineer and IT for the Control Systems Engineer.The Conference began in earnest Tuesday with approximately 110 attendees from US and international electric and water utilities, chemical and oil/gas companies, IT and control system suppliers and consultants, universities, and US and international government agencies. The Conference is called Control Systems Cyber Security is that industrial control systems are common across multiple industries. The agenda can be found at www.realtimeacs.com.

There were two hacking demonstrations of control systems and several discussions on control system cyber vulnerabilities. There was also a discussion on the need for technical control system cyber security curriculum (policy programs exist). There were two keynotes: the Honorable Yvette Clarke (D-NY), Chairwoman of the Subcommittee on Emerging Threats, Cybersecurity, Science and Technology and member of the Intelligence, Information Sharing and Terrorism Risk Assessment Subcommittee provided the lunch keynote. Whitfield Diffie gave the evening keynote and discussed control system cyber security issues from the Tuesday’s session.

There were four different sessions on actual control system cyber incidents – none of which was public! In one session, two control system engineers from two different utilities that have control systems from every major supplier discussed their recent control system cyber incidents – one had his plant shutdown. A couple interesting side notes were that existing control system logging are adequate to identify control system incidents and their control system suppliers weren’t of much help when it came to providing control system cyber security support. Both engineers felt it was so important to share information they attended the Conference on their own nickel. This is in marked contrast to the utility and industry leadership who didn’t think this conference was important enough to attend even though many were based in Washington. Wednesday evening, the Honorable James Langevin gave the evening keynote. Congressman Langevin felt this was so important he spent 30-45 minutes after his presentation answering questions and talking to the attendees.

We received a summary of government activities including legislative efforts on cyber security, cyber security activities by the Nuclear Regulatory Commission, efforts on-going at the Bonneville Power Administration using the NIST Framework, and non-governmental activities in certification and cyber incident collection. Also got a very interesting presentation on cyber security legal issues and a discussion of the Russian cyber attack on Estonia.

On the last day, NIST held training sessions on two very relevant NIST standards:

-- SP 800-53 - Recommended Security Controls for Federal Information Systems - including those for the Bulk Power System
-- SP 800-82 - Guide to Industrial Control Systems (ICS) Security provides guidance on securing Industrial Control Systems (ICS), including Supervisory Control and Data Acquisition (SCADA) systems, Distributed Control Systems (DCS), and other control system configurations

Electric Car Conundrum: V2G a Smart Grid Blessing or Curse?

Initially arriving in the US in low volume in late 2010, the addition of thousands and later millions of cars with 5-10 KW battery packs drawing power from (and sometimes giving back to) the grid is cast as both a positive and a negative, depending on your point of view.

On the positive side, as this article says, high performance, deep cycle lithium ion and lithium air batteries en mass may be the energy storage solution the industry has been searching for. Here's an example starring Duke:

Duke Energy committed to an electric vehicle future when it committed with the FPL Group to buy 10,000 electric vehicles and plug-in hybrids in the coming decade, as they upgrade their fleets. The energy storage in these vehicles could eliminate the need for peaking plants and enable the expanded use of renewable energy. Duke Energy’s electric vehicle future may save billions in future power plant investments.

Sounds good, but others worry, here, that local electrical infrastructure can barely handle the additional iPods and iPhones it's had to deal with lately. Adding clusters of electric cars charging at approximately the same time each evening might break the camel's back in many neighborhoods. According to Peter Darbee, the CEO of Pacific Gas & Electric:

A high concentration of plug-in electric vehicles poses a serious challenge to utilities. Plug-in electric cars could draw electricity equivalent the amount needed to run one home, or up to three homes in certain places. You can see if you have three or five electric cars arrive in a neighborhood, you're going to overload the local circuits, and that will lead to blackouts. So we see it as an opportunity but we also see it as a challenge of significant proportions.

We all know how neighbors like to mimic and compete with each other (have you seen the Halloween decorations next door !?). One electric car will beget two will beget ten or twenty. Scheduling software will help, but much depends on fast this goes, and how close to edge local circuit gear is at the outset.

Nissan Leaf photo credit: Wikimedia Commons

Why Smart Grid Security is about so much more than Smart Grid Security

Frankly, after having worked in the security industry for ten years now, there are days when I feel like I've had my fill. At a recent Smart Grid conference I sometimes wished I could focus solely on cool new functionality like Vehicles to Grid (V2G) for instance.

But then I remember that what got me into energy was a passion for renewable technologies. A passion which was refreshed last week when by futurist Ray Kurzweil, speaking at MIT's Enterprise Forum, reminded us that solar energy technology is now on an exponential growth curve, just seven evolutionary steps away from reaching price/performance parity with the cheapest fossil fuels: coal.

Well guess what? If solar was ready for prime time today the grid couldn't handle it. Wouldn't that be depressing? We've got a few more years to get the grid ready by making it smarter, more flexible and able to handle the intermittent aspect of solar and wind.

So we need this Smart Grid to be up, running and well along its nation-wide implementation in the next 5-10 years. During this period, security consultancies like IOActive and Wurldtech will continue to tell us that the Smart Grid is a house of cards, ready to be blown over by casual hackers, let alone organized criminal gangs, non-state combatants, and nation states determined to harm the USA. There will be times when we'll second guess what we're doing, when we'll question whether NERC's vigilance and NIST's Smart Grid security standards are up to the task, whether key industry players are putting enough thought and effort into the security elements of their solutions, or are simply trying to sell us what was "secure enough" in the past.

Ultimately, the Smart Grid must both appear to be secure (so we continue to invest in and deploy it) and actually be secure, so it doesn't suffer a knock-out blow in its formative days. All this security stuff, while potentially tedious to some, is an acknowledgement that a secure Smart Grid is a mandatory prerequisite to our nation's energy future, nothing less.

Photo: Wikimedia Commons

Military Planning For Prolongued Outages via Smart/Micro Grid Technologies

While the US Department of Defense has many unique tasks and requirements, many of its concerns and challenges re: the current grid, Smart Grid and Smart Grid security are common to all enterprises. Much of what motivates DOD motivates others, including:

• Desire for continuous operation and continuous service to customers by keeping core systems running during (possibly prolonged) power outages impacting local communities
• Energy efficiency savings via reduction in electricity and fossil fuel usage
• Demonstrating proactive/compliance measures vis-a-vis climate change and the increased use of renewable energy sources
• Maintaining confidentiality/privacy of data and doing all of the above is a safe and secure manner

So along those lines, here's an excerpt from a recent post on the DOD Energy Blog on the so-called "brittle grid" problem I believe you'll find interesting:

Eighteen months have now passed since the public release the "Defense Science Board Task Force Report on Energy" This is from the section called "Managing Risks to Installations":

For various reasons, the grid has far less margin today than in earlier years between capacity and demand. The level of spare parts kept in inventory has declined, and spare parts are often co-located with their operational counterparts putting both at risk from a single act. In some cases, industrial capacity to produce critical spares is extremely limited, available only from overseas sources and very slow and difficult to transport due to physical size.

In many cases, installations have not distinguished between critical and non-critical loads when configuring backup power systems, leaving critical missions competing with non-essential loads for power. The Task Force finds that separating critical from noncritical loads is an important first step toward improving the resilience of critical missions using existing backup sources in the event of commercial power outage. The confluence of these trends, namely increased critical load demand, decreased resilience of commercial power, inadequacy of backup generators, and lack of transformer spares in sufficient numbers to enable quick repair, create an unacceptably high risk to our national security from a long-term interruption of commercial power.

Granted, DOD's not the only organization with these concerns ... and the obligation to plan accordingly. Hospitals, police & fire, essential services, etc. all have to think this way. DOD is exploring campus microgrid strategies (including on-site power generation, energy management and energy storage systems, and more) to allow bases to "island" themselves away from commercial grid infrastructure.

The technology is getting to the point where this approach is becoming just as feasible for industry. We'll be investigating further and will post the results right here.

Photo Credit: Kristen Holden on Flickr

Smart Grid Security: Answers in Questions

Over the past year, Andy and I have written about the risks and opportunities in the growing software sector of the Smart Grid Marketplace. We have described the space, some of the firms, the investment, and what we are seeing for security in those organizations we speak with. In response, and I think with genuine interest, we've been asked what we are worried about, and in turn, what recommendations would we specifically make to individuals who are either investing in these solutions, or who are actually building them.

In the recent NIST strategy and requirement recommendations release, there was a substantial body of information to be reviewed, and this post is not meant to summarize or to supplant those results (obviously). This is a relatively lightweight view of heavy duty and high-level considerations in software as a critical element in the development of the Smart Grid. It is a practical list of questions that organizations should be able to answer before they commit to software that will either replace or broker their interactions with the Smart Grid.

What is the software's provenance?

Provenance is a term that gets thrown around a lot, but I use it to express the idea of origin. Where did the software come from? Who made it? What was it made from? While absolute provenance is difficult or impossible to ascertain, these answers can help to guide risk awareness and management. Is it new software built for me? Is it existing software that has run similar systems elsewhere? Is it a new solution from an existing partner, or revision 0.9 from a start-up? Is it built from the ground up, or does it contain elements of legacy applications, particularly those that my have been written with a different security mindset? By understanding more about the roots of software, the strategy to secure its use will be better informed.

Why ask the question? Unless you know about the origins of software, it is very hard to put together a plan to ascertain its security. Knowing who built it provides a resource to ask about the way in which it was built. Knowing about its components provides information to use in testing it or researching testing done by others.

What is the plan for ongoing governance?

Governance, similarly, has a variety of depths of detail and application, particularly in IT. For our purposes, the questions can be limited. How will the software be updated? Who will make those decisions? What is the process to initiate or approve a change? New software in any environment, and even established software in a dynamic environment, will face frequent opportunities and requirements to change. Understanding the models through which those changes are considered, approved, and delivered enables organizations to measure and manage their own risk from flux in the software, and in any collateral instability introduced to dependent systems.

Why ask the question? Instability = Insecurity. Haphazard or non-existent governance leads to more frequent changes, less testing time for the solution in place, and to inevitable discontinuities if the software is a component of a larger system. Weak governance also increases the opportunities and likelihood of malicious coding behavior by simply increasing the chaos during the software delivery process.

What does the software do with data?

Data is at the root of almost every application's function and purpose. Whether it exists to generate data, to gather it, or to analyze it, data is not only central to the application, it is often the prime target for an attacker. For that reason, there are multiple facets to consider. What kinds of data does the application gather, where does it come from, and how does it enter the system? Once the data has entered the system, does it get stored, and is it stored with appropriate protection of privacy and integrity? If the data ever moves between components of the system or between multiple systems, is it appropriately protected by the software for privacy and integrity? Does the system restrict access to the data, and is access control sufficiently granular to permit only authorized individuals to enter into the system? Each of these questions naturally results in a series of more technical and specific questions about the behavior of the application, but requiring answers to these high-level queries will mean that these will not be ignored.

Why ask the question? Data is central to the smartness of the Smart Grid, and its protection is expected by subscribers, is in many cases mandated by regulation, and is certainly necessary to ensure reliable operation of the Smart Grid.

How has the software been tested?

The testing of software, particularly for security issues, is still a developing field. There are a variety of approaches and mechanisms, each with their own strengths and deficiencies. What testing has been done, and on what components? What approaches were used, and with what results? Have all components been considered for security issues prior to their inclusion, and how were they vetted prior to selection?

Why ask the question? Understanding the testing process for the software can uncover blindspots to some sets of security issues, and can also identify weaknesses in methodology that can indicate systemic problems from the provider. If the testing ignores a specific area, like data storage or access control, then that lack of attention raises the likelihood that there could have been a similar lack of focus during its construction. Testing has many facets, and security must be among them.

These questions are intended to be a very brief introduction to some of the underlying and quite concrete issues that must be considered during the Grid's evolution to a Smart Grid. In time, each of these areas must be expanded into multiple levels of detail, but for now, this is a start. It is the start of generating more informed awareness, and of describing the types and amount of data that is required to feel secure during the adoption of new Smart Grid technologies.

In return, though, having those answers will certainly bring more confidence, more security, and more opportunity for success in the new Smart Grid.

Image kudos to flickr ( http://www.flickr.com/photos/marcobellucci/ / CC BY 2.0 )

Islands No More

In a bracing report from Australia, we learn from the Sydney Morning Herald that Integral Energy was inundated with a virus on non-critical systems, but at such a penetration level that they chose to rebuild 1000 desktop machines to eliminate the problem before it "spreads to the machines controlling the power grid."

The security consultant interviewed in the piece, Chris Gatford from HackLabs mentions that in his experience there is ample evidence that the networks may well have been connected despite the efforts of the utility to separate them. This is particularly problematic, I am sure, because there are not only power control systems to worry about, but also online payment, user account management, and other relatively advanced functions at Integral Energy.

His comments seemed familiar to me, so I went back through my notes, all the way to a report from the team at Riptech in 2001 ( Bought by Symantec) called " Understanding SCADA System Security Vulnerabilities ", where the authors describe a very similar disconnect between assumptions and reality in these internal networks:

MISCONCEPTION #1 – “The SCADA system resides on a physically separate, standalone network.”

Most SCADA systems were originally built before and often separate from other corporate networks. As a result, IT managers typically operate on the assumption that these systems cannot be accessed through corporate networks or from remote access points. Unfortunately, this belief is usually fallacious.

In reality, SCADA networks and corporate IT systems are often bridged as a result of two key changes in information management practices. First, the demand for remote access computing has encouraged many utilities to establish connections to the SCADA system that enable SCADA engineers to monitor and control the system from points on the corporate network. Second, many utilities have added connections between corporate networks and SCADA networks in order to allow corporate decision makers to obtain instant access to critical data about the status of their operational systems. Often, these connections are implemented without a full understanding of the corresponding security risks. In fact, the security strategy for utility corporate network infrastructures rarely accounts for the fact that access to these systems might allow unauthorized access and control of SCADA systems.

MISCONCEPTION #2 – “Connections between SCADA systems and other corporate networks are protected by strong access controls.”

Many of the interconnections between corporate networks and SCADA systems require the integration of systems with different communications standards. The result is often an infrastructure that is engineered to move data successfully between two unique systems. Due to the complexity of integrating disparate systems, network engineers often fail to address the added burden of accounting for security risks.

As a result, access controls designed to protect SCADA systems from unauthorized access through corporate networks are usually minimal, which is largely attributable to the fact that network managers often overlook key access points connecting these networks. Although the strategic use of internal firewalls and intrusion detection systems (IDS), coupled with strong password policies, is highly recommended, few utilities protect all entry points to the SCADA system in this manner.

I think that the team at Integral Energy knows this as well. Their actions show that they felt it necessary to take serious and disruptive measures to eradicate a virus outbreak before it jeopardized the entire infrastructure. Their willingness to speak of it publicly also provides a real service to those of us who are considering the impacts of the introduction of multitudes of new systems and new access points into those same networks.

One sees allusions to the concept of separate networks, with various properties, in existing regulation, CIP descriptions, etc. If we can agree that there are likely to be unintended cross-overs between these systems and their populations, then we must also agree to stop considering the artifice of disjoint networks as being anything but an anachronism, and treat the security of each network with the same rigor and protective approaches, regardless of our faith in its isolation from sources of corruption.

( Image Thanks to Flickr: http://www.flickr.com/photos/cloudsoup/ / CC BY 2.0 )

CSOs and the Smart Grid

Setting the Stage
So you're an executive in charge of security at a medium, large or very large organization. You might be called Chief Security Officer (CSO) or Chief Information Security Officer (CISO) or maybe VP or Director of Security. You most likely report to the Corporate CIO, or you're in a business division and you and your boss plug into a General Manager. You decide, with blessing from above for the big stuff, the following:

• Where you'll get the biggest risk reduction (or compliance) bang for your limited budget buck
• Which technologies get purchased and implemented
• Which vendors will augment your in-house security team, and,
• Corporate security policies, and how to best promulgate them to other parts of the co. for whom security is at best an annoyance, and at worst, something to be openly resisted

Yours is a world of risk management as you oversee the wellness (e.g., integrity, reliability, performance, compliance) of your IT, networking and communications systems (and true CSOs own physical security as well). In addition to managing for threats coming from those directions, in recent years, new threat vectors from service oriented architectures (SOA), Web 2.0 and cloud computing have kept you busy.

Hey, Have you Heard of Smart Grid?
So how much time do you spend on future threats? If you have heard of the Smart Grid, and if you've been reading up on it, then you probably don't need to read further here. You're in the top 10% of your class and get a star on your forehead. If however, you're like some CSOs I've talked with who claim to have never heard the term, then this is your wake up call. There has been little written to guide CSOs through the early stages of preparing to protect their organizations in a world where the power systems they rely on look increasingly like the Internet (and in some cases are the Internet!).

How is it different from today's electrical grid? For starters, it's a 2 x 2-way system. Thanks to advanced metering infrastructure (AMI) and net metering, electricity and usage information will flow from generators to consumers and back again. The total amount of information, which in the beginning will be substantial, will quickly become enormous. Data protection will be crucial, and demand management strategies which could save your organization significant money, could also get you in trouble fast. Water and other services will also be impacted for better and worse. In short, for each benefit a Smarter Grid will bring an organization, there is a commensurate risk to mitigate. And it's your job to know (and plan for) this.

Only CSOs at utilities see this world first hand, and even in the energy and utilities vertical, many of those CSOs work in a balkanized world where their policies touch only IT, and the "rubber meets the road" part of their company, field operations, doesn't want to anything to do with them.

So most CSOs are left to infer what they need to know from a mountain of Smart Grid articles and a multiplicity of Smart Grid conferences. My guess is once they've poked a toe into these confusing waters one time, they soon find their time better spent working on present challenges. The appropriate information has not yet been boiled down for this most important enterprise leadership function ... one that could and would do the right things, proactively, if it had the right knowledge to work with.

CSO Info Resources Not Too Helpful Yet
Where do CSOs turn for expert guidance and to learn from what their successful peers are doing? Why, the journals and other news sources that serve them. Yet from the looks of these two articles from CSO Online and the CSO Roundtable, all they're getting is high level introductory material that in no way considers how Smart Grid trends intersect with CSOs' particular responsibilities. I would advise these orgs to get on the ball: it's their job to see over the horizon and around corners to give their readers the info they need to protect their companies ... and their jobs.

No Answers Yet, But Here are a Few Starter Questions
NIST and other standards bodies are working around the clock to bring appropriate and helpful security standards to this new domain and you don't have to know them yet (however, for a sneak peak, here's the most recent draft edition of Smart Grid Cyber Security Strategy and Requirements from NIST). So much is still in flux that doing too much at present might be as bad as doing too little. But that doesn't mean you shouldn't start getting your head around this challenge and thinking through some of the scenarios. Here's a handful:

1. Supply Chain - Similar to Y2K preparation in some respects, even if you get your house in order for the arrival of the Smart Grid, if the companies yours depends on are not prepared it may affect you. It's time to talk about this with them.
2. Vehicle Fleet - More choices are coming, including hybrid electric, full electric, natural gas, etc. Are you thinking about the challenges and opportunities that present themselves in beginning to move away from gasoline and diesel? What are the security implications of your enterprise depending on these new transportation technologies?
3. Local utilities - All utilities are under guidance to prepare for Smart Grid standards and technologies. What are your providers doing in your different locations and how soon will their actions begin to affect you? What do you need to do to not get blind sided?
4. Smart Grid pilots - With stimulus help from the Fed Gov, pilots are springing up everywhere. Related to number 3 above, are there any pilots going on you could participate in? While this might take resources away from more proximate concerns, the education might more than pay for the time invested.
5. Centralized policy and control - If yours is a geographically distributed operation, to what extent will you attempt to define and enforce Smart Grid-related security policy in a uniform way, versus allowing disparate facilities and offices to determine their own best approaches?

That's all for now, but on each of these and many more there's a ton of thinking and planning to be done. While in most cases it's too early to implement, it's certainly not too early to imagine.

And there there was none...

News from the Smart Grid Investment Grant program

Early Birds win again! Looks like the interest and enthusiasm for Smart Grid Programs has rapidly outstripped even the Government's own $3.4B largess. In an amendment dated September 21, the DOE announced that:

The Department of Energy has received a significant number of high quality applications and our review continues. The dollar value of applications far exceeds the funding available under this Funding Opportunity Announcement. As a result, Phase III is canceled.

and

Given the facts cited above, the Department may decide to cancel Phase II following final selection decisions made on applications currently under review.

So, what was intended to be a three phase investment program in new approaches to energy and grid management has become at best a two-phase program, and likely a single shot of stimulus into the Grid. Taking the amendment on its face, that the dollar value of applications already received far exceeds the funding available, we can conclude:

In the planned Phase I application period, running from the initial solicitation date of June 25th, 2009, to August 6th, 2009, there were requests for grants FAR EXCEEDING $3.4B. This means that, on average, the DOE received grant requests FAR EXCEEDING $113M every business day of the Phase I application period.

Each of these applications was expected to include many things, not least among them a well-articulated security plan. You will remember, from the cyber security requirements description:

Submitted Project Plans are also required to include a section on the technical approach to cyber security. Cyber security should be addressed in every phase of the engineering lifecycle of the project, including design and procurement, installation and commissioning, and the ability to provide ongoing maintenance and support. Cyber security solutions should be comprehensive and capable of being extended or upgraded in response to changes to the threat or technological environment.

Yikes. And more specifically must include:

• A summary of the cyber security risks and how they will be mitigated at each stage of the lifecycle (focusing on vulnerabilities and impact).


• A summary of the cyber security criteria utilized for vendor and device selection.


• A summary of the relevant cyber security standards and/or best practices that will be followed.


• A summary of how the project will support emerging smart grid cyber security standards.

In 20ish years of working in security, I have seldom found an organization that could create this level of cyber security detail within six months for an existing system, much less create it in 30 business days for a brand new project.

The infusion of SGIG capital has definitely gotten things moving, but we should all hang on. This looks to be a bumpy ride.

( Photo thanks to http://www.flickr.com/photos/pleasantpointinn/ / CC BY 2.0 )