PRODUCT SECURITY OVERVIEW

Kip Boyle: 

Hi everyone. This is Your Cyber Path. We are the podcast that helps you get your dream cybersecurity job. We want you to get your dream cybersecurity job. That’s why we’re doing this. I’m Kip Boyle, Wes Shriner is my co-host, and we are, guess what? Experienced hiring managers of cybersecurity professionals. And that’s what we want to share with you is, if you want to get the attention of a hiring manager, because you want a cybersecurity job or you’re ready for the next level, because you already have your cybersecurity job, then that’s what we’re going to help you figure out is, how can you become attractive, dare I say, irresistible to cybersecurity hiring managers? That’s what we want for you.

So this episode is available in two ways, we have an audio only recording that you can find just by searching for Your Cyber Path podcast in your favorite podcast app. We’re also on YouTube. Just again, do the same search on YouTube, you’ll find our channel. And so you can listen to us or you can watch us and listen to us, whatever you like. We are going to show you some visuals. So you might want to, either watch us on YouTube or download the slide deck, which you can do by checking the show notes.

So right now what we’re doing is we’re going through a series of episodes, and the goal is to tell you all about the way that typical cybersecurity organizations are put together, because there’s opportunity in there. But if you’re just getting into cybersecurity, some of the jobs, some of the opportunity is going to be easier for you to access, and so we want to point that out to you. And for those of you who are already in cybersecurity or if you’re just a big time, long range thinker, then there’s going to be other opportunities that you’re going to have a chance to move up to, and we want to point those out to you as well.

So, today, what we’re going to focus on is product security for those organizations that have products that need to be secure, because they work over the internet in one way or another. This is the part of the cybersecurity organization that’s going to do that. And I’m happy to tell you, and you can probably see if you’re watching us on the video. We have a guest who’s going to help us explore this organizational unit, and try to make some sense out of it. So Wes, tell us all about Matt.

Wes Shriner: 

I’m glad to have Matt Clapham here today with us. He is the Director of Cybersecurity for Software and Cloud at GE Healthcare. He spent the last 20 years at several major software organizations, securing the internal and external facing properties. And so really glad to have you with us today, Matt.

Matt Clapham: 

Happy to be here.

Kip Boyle: 

Hey, Matt.

Wes Shriner: 

And it looks like your Twitter handle is actually @ProdSec, so we can find-

Matt Clapham: 

I managed to snag that one.

Kip Boyle: 

Wow.

Matt Clapham: 

We are talking to the Twitter handle today @ProdSec, love it.

Kip Boyle:

If you get more work than you can handle, just let me know, Matt.

Wes Shriner: 

All right. So Matt, tell us a little bit about yourself and your career. Where are you at? What are you doing?

Matt Clapham: 

Sure. So the way I like to describe what I do is, I make products more secure. And how I got there, is I started out in development, I was a software tester for a number of years, and I really learned a lot about how apps are put together. And I was right there on the cutting edge of the trustworthy computing initiative. When it really became apparent, they had to really focus on product security, and everything was focused on making the products more secure.

And then, from there, I moved into things like IT and IT security, and learning a lot about how video games are put together. And then I put that all together and I’ve been working on product security teams pretty much ever since. In that, as we’re looking at all that stuff about how do we make a product, how do we make a product securely? I put all of that together, take that knowledge I have about how software is built, and the knowledge I have about how IT security works, and merge them together to that end goal.

Wes Shriner:

That’s tremendous.

Kip Boyle: 

Excellent.

Wes Shriner: 

I’m glad to have you with us today. It sounds like you are truly an expert in the space, and it’s going to be a lot of fun to have this conversation.

Kip Boyle: 

Yeah. And Matt, can I just validate something that I think I heard you say, which is, the way you got into cybersecurity was you started out as a software tester, right?

Matt Clapham:

Mm-hmm (affirmative).

Kip Boyle:

And then you moved into cybersecurity. So I always like to point out the path that our guests took to get into cybersecurity. Because I think it’s inspirational for some people, who are watching us here, are going, “Oh, my God, I’m a software tester. I could do that too.”

Matt Clapham: 

Yeah.

Kip Boyle:

Great.

Matt Clapham:

I started, I was, I think, about six years as a software tester.

Kip Boyle: 

Oh, I’m sorry.

Wes Shriner:

I was just looking at the previous slide telling me all about you, and now-

Matt Clapham: 

That explains a lot Wes.

Wes Shriner:

I can share that with the world. Okay, there we go. Glad to be sharing that with the world, all right. We’re going to jump ahead now, since I can see it, now, I’m going to share it with you too. All right, this is a reminder that we are taking a look at the common security team structure of a large security organization.

We’ve got security engineering architecture and test, we’ve got governance, risk, and compliance, security operations, and what we’re focusing on today is our product security space, that bottom left. I’m going to bring that into focus a little more here, that bottom left corner of this diagram. We are going to spend today’s episode on that, and then we’ll go from there. Next episode, we’ll get a chance to look at security operations.

Kip Boyle: 

Yeah, it’s going to be great.

Wes Shriner: 

Inside product security, we’ve got two disciplines. We’ve called out five specific teams and two security service catalog line items. Now, there’s a lot inside of there, so I’m looking forward to breaking it down with you. But I think this is a great place to get started, to understand what is product security.

When I think about products, I think about products as anything in front of us, that could be a technical product like Xbox or a mobile device or a handset. It could be an IoT. It could be a point of sale kiosk. It could be an ATM machine. A product could be in a customer’s hands or something the customer approaches. It’s anything physical that can be held. Am I saying that right?

Matt Clapham:

Yeah. And don’t forget the virtual world as well. Imagine a software as a service solution, is itself a product.

Wes Shriner: 

That’s true, when we’ve made a differentiation here, we talk about products versus product services. And so I think we’ll get a chance to talk about services separately from products, because securing a physical asset is very different than securing an online service of some sort.

Matt Clapham:

Fair point. I kind of merged the two, because they all kind of blend in a DevOps world. But, yeah, let’s do that.

Wes Shriner: 

Outstanding, thanks for that. Folks, we’re going to talk a lot about customers today. We’re going to talk about internal customers and external customers. So I want to clarify an internal customer or lowercase C. We’re going to call a client, and an external customer is somebody who pays us money, that’s a capital C for capital C customer. We’re going to call that person a customer. So client versus customer will be our vernacular for today’s episode.

When I think a little bit about the hardware pieces that go into device security, I think about that whole assembly line. I’ve got actual hardware coming from somewhere, or maybe I’m building it in my basement. I’ve got a firmware that loads on top of that, and an operating system that loads on top of that. I’ve got software that goes on top of that. And then some certificates and some sort of pipeline that delivers that software in a version controlled manner. And each one of those layers is very much a part of device security, when we think about product security.

Matt Clapham: 

Yeah. We’ve learned that the supply chain of how the bits get to the box is critically important. And then also making sure that the quality of those bits that go to the box is crucial.

Kip Boyle: 

Why is that? I mean, just a little bit more about why that’s so important.

Matt Clapham: 

How the bits get to the box?

Kip Boyle: 

Yes. Thank you.

Matt Clapham: 

Okay. So imagine you’re-

Wes Shriner:

Let’s break it down, what’s bits? And what’s box?

Matt Clapham: 

Wes, I might jump ahead a little bit here, but imagine you’re building that whole structure to take care of the product, the thing. From the, well, I like to say from the silicon on up, you’ve got to make sure that you’ve got a good foundation. So you thought about it at the device layer, that you thought about it at the tooling on top of that, that you thought about the base operating system you might be building yourself or consuming from somewhere else. All the way up into the level that becomes the device as a platform, security pervades that whole thing.

Wes Shriner: 

I can’t tell you how many times marketing has brought me an IoT biscuit and said, “We love this thing. It’s going to be great.” And we find out it’s actually built in a garage in Australia, right?

Matt Clapham: 

Yep.

Wes Shriner: 

And we found that the garage in Australia is actually ordering its hardware from China. And when we find that the devices they gave us for security testing have changed, because the firmware was updated, and they didn’t even know the firmware version had changed and their software wasn’t running on it successfully any longer. We really need to understand our supply chain in order to understand what we’re putting on top of that. You keeping those lights on back there?

Matt Clapham: 

Yeah. Yeah. You might see me wave my hand like this, occasionally, it’s my lights are turning off.

Wes Shriner: 

These are not the lights you’re looking for.

Matt Clapham: 

These are not the lights you’re looking for. Actually, these are the lights I’m trying to keep on, so you can see my shiny bald head. But, anyway, keep going.

Wes Shriner: 

Another tricky part about this is, are we dealing with this through patents? Are we securing our software through open aware patent process? Or are we securing our software through trade secrets that we can’t tell anybody about? And depending on how we approach that is going to depend on how we handle our supply chain as well.

Matt Clapham: 

Very much so. If you look at it from an intellectual property protection perspective, you really got to think about, how super special is it? And have a real honest, hard conversation about that. Sometimes people think their stuff is so cool, but it’s really not. It’s just yet another piece of software that does something. There’s a bunch of other ones on the same market, if you think you’re going to be protected by the fact that you have a trademark or a copyright or a patent on something, no, that’s just a legal protection.

Wes Shriner: 

It is. And I think legal is really the one that make those calls in a lot of ways. That’s your product team working with your legal team, and security is informed on that decision, but not necessarily even consulted.

Matt Clapham: 

Yeah. And at the end of the day, you got to put your software on something to get it out the door. So you’re basically giving away the software to an extent.

Wes Shriner: Indeed, indeed. So when you’re designing your device, you’ve got to decide, is it going to have wired or wireless communication? Is this going to be a one-way or two-way communication? Is that going to be credentialed? Or is it going to be open, unauthenticated traffic?

One of the big complaints of an IoT device, if we looked at the IoT manifesto, might be, can it be updated? Can it be improved over time? Once the customer owns it, is it dead forever? Or can I pass an update to it along the way? And if I do, can anyone else update that model also? Can the customer modify configurations for that, in some way? Can they apply an if this, then that type logic to it? And if so, is that a dedicated interface? Or are they sharing the administrator interface when they do? A lot to think about when we start thinking about what is that device and how can it be secured and then maintained over time.

Matt Clapham: 

All those factors that do deal with the user experience of managing the device securely or the user experience of securing the device itself, all play in there.

Wes Shriner: 

Indeed. We talked a little bit about third-party risk last episode, when we talked about third-party risk inside our enterprise. Third-party risk to our customers is very much a real, additional threat here in the product security space. And so understanding both our product and device security and our third-party risk management strategy here is very, very important.

Matt Clapham: 

Yeah, there’s some well-known cases, where it seems that the attacks came through some sort of third-party, who had a very poorly secured IoT solution or something like that. And it was the foothold that the attackers use to then keep attacking and pivoting throughout the organization.

Wes Shriner: 

Is there anything else you guys would like to add to device security specific, before we jumped to services?

Kip Boyle: 

I think-

Wes Shriner: 

And I’m sorry, I’m only talking about technology devices. It could be anything, product security is product security.

Kip Boyle:

Well, the example we’ve given before, is like an Xbox, or some kind of a handheld device. That’s still what we’re saying here, right?

Wes Shriner:

Should be. Should be the same thing.

Kip Boyle: 

Okay. Yeah. And I think that the difficulty that we’ve talked about before with these, is, and not to get too crazy here, but these are devices where we’ve always assumed that if you didn’t control the hardware, if it wasn’t in your possession, then it couldn’t be secured. And so that’s one of the things that makes it so tough, is you got people doing reverse engineers on these devices, and it makes it very difficult to affect product security on something that you just don’t have physical possession of. That’s how come DVD encryption got broken, I mean, I think that’s a good example.

Matt Clapham: 

Well, you’ve got to think about the trade-offs there. I mean, in some cases you want to try to make it really hard for somebody with physical possession of the device to do something bad with it. But then on the flip side, even if you’re not in that world, you still want to care about the quality and the security of the software that you’re putting on top of it. Yeah, sure, you have to write off, somebody with a debugger and a JTAG can attack it, but that doesn’t mean that somebody who can just view the packets on the network should be able to attack it.

Kip Boyle: 

Right. Right. I guess, the only point that I’m making here is that this is actually tough. This is actually a very difficult thing to accomplish.

Matt Clapham: 

Yeah. You’re building a security from the silicon on up. So everything from like how the CPU boots and talks to the TPM matters.

Kip Boyle: 

Yeah. Yep. Yeah. And that’s a maximum is insecurity in general, is how you do things matters, it’s not just about the outcomes that you get. Because if you get a good outcome, but you do it in a shoddy way, somebody is going to figure that out and is going to eat your birthday cake.

Matt Clapham: 

My team, generally, and my approach to product security is to look at the behaviors really. Because you think about it from product security, you want to have that culture. You want to make sure the teams are doing all the right things to design, build, deploy, and operate those secure products, secure solutions. So you’ve really got to walk them through that process. And at the end of the day, there’s still going to be problems that come out the other end. But at least they should be known, understood, and tried really hard to minimize them, and then we can come up with a plan to deal with them.

Kip Boyle: 

Yeah. Yep. Well, it’s going to keep us in business. That’s for sure.

Wes Shriner: 

So when I secure the back office, when I secure the enterprise itself, I am protecting the organization’s reputation. But when I’m securing the product, the device that actually goes in a family, in their home, I’m actually in a way, a much bigger impact to the organizational brand, if that device is popped, right?

Kip Boyle: 

Oh, yeah.

Wes Shriner: 

So I would associate product security even more with customer trust than I would even the enterprise security piece.

Matt Clapham: 

I actually have a prop for that.

Wes Shriner: 

Tell me about that.

Matt Clapham: 

This is the Cheerios box from a number of years ago, they were bold enough to put trusted on the box. This is a Cheerios box, and then they said, “Our product is trusted. Think of product security is the same way, is trying to help make sure that they can call the product trusted and say, “We did all this due diligence, we think it’s a really secure product.” And be able to say that with a straight face, and not have a problem later on.

Kip Boyle: 

I need a prop like that, man, that’s good.

Wes Shriner: 

That is powerful.

Matt Clapham: 

They don’t sell them anymore. I seriously went on, bought like a half a dozen of them, when they had them at the time.

Kip Boyle: 

It’s toasted oats-

Wes Shriner: 

That’s one of the first-

Kip Boyle:

but it’s trusted.

Wes Shriner: 

That’s one of the first foods you feed a baby after they moved to solid foods. And so-

Matt Clapham:

In fact, that’s what this is about, is about finger foods.

Wes Shriner: 

And-

Matt Clapham: 

I think it’s around the time I got it, too.

Wes Shriner: 

And if I’m going to trust you with my child, then that’s trust, so that’s really powerful

Matt Clapham:

As a product security engineer, my goal is to make sure we don’t violate that trust. Because once we do, it’s 10 years to fix that brand damage.

Kip Boyle:

Wow.

Wes Shriner: 

If it’s fixable.

Matt Clapham: 

If it’s fixable even, yeah.

Kip Boyle: 

Wow. 10 years. I’ve never heard anybody put a number on it. Where’d you get that number? That’s cool.

Matt Clapham: 

Oh, look at Microsoft. Look at how long it took them to actually shake the oh, were unsecure software-

Kip Boyle: 

Oh, yeah.

Matt Clapham:

10 years. And even still today, you still find people like, “Ah, Microsoft, totally untrusted.”

Kip Boyle: 

Yeah. Yeah. You know what? You’re right. That’s a great example. Thank you.

Wes Shriner: 

So half a generation is what it takes. That’s-

Matt Clapham: 

For one screw up, or a couple of really big ones, you know?

Kip Boyle:

Yeah. Mm-hmm (affirmative). Yep, that’s [crosstalk].

Wes Shriner:

On the product security side. Yeah. It’s got to big impact.

Matt Clapham: 

A lot of pressure.

Wes Shriner: 

So those products are the devices that go in the customer’s hands, that’s the Xbox. And then we’ve got a second piece, the Xbox online, the services that come in behind those products, and those are equally as valuable to our customers and their experience. So if our customer brings home an IoT device, and that’s a GPS registration device that allows you to track their dog as the dog runs around the neighborhood, that customer is going to go online, register that device to their home, to their name, to their pet, and then their pet can be tracked. But if that online service is vulnerable, then that gives a lot of personal information away about that family and that home as well.

Matt Clapham: 

Yeah, definitely. And imagine an even worst case scenario, so the service is talking to the device, and the service gets to tell the device what to do. The service gets compromised, it could potentially all those devices.

Wes Shriner: 

That’s true, we’re not just talking about an individual compromise here. We could be talking about the entire product farm being comprised.

Matt Clapham: 

Break once, break everywhere.

Wes Shriner: 

Right. And that gets even more interesting when we get into the healthcare industry, where we have medical devices that are life safety devices.

Kip Boyle: 

Yeah. Yeah. Like an IV infusion pump, holy moly. Those things are on a wireless networks all throughout the hospital, and pacemakers have wireless capabilities. Oh, yeah, it just goes on and on.

Wes Shriner: 

So Matt, I should probably call out that you’re not here on behalf of your company. Your company does know you’re here, but you’re speaking-

Matt Clapham:

Appreciate that Wes, yeah. I’m going to decline healthcare topics, because of where I work. And, yes, I’m here on my own personal accord, and not directly as a-

Wes Shriner:

Outstanding.

Matt Clapham: 

Job of my employer.

Wes Shriner: 

But you’ve got, a cleverly placed Michigan hat just behind you there. I’m sure that’s totally-

Matt Clapham: 

It’s on the desk. I just-

Kip Boyle: 

It’s not on your head, so I mean-

Matt Clapham: 

Well, it won’t fit over my-

Kip Boyle:

that’s plausible deniability.

Matt Clapham: 

It doesn’t fit over my headset.

Wes Shriner: 

That’s why. Outstanding. So product services security is as much a part of protecting our customer experience, because that’s web application protection. That is backend database, and usually it’s SaaS protection, it’s software as a service. It is in AWS or Azure. It’s understanding how am I going to protect the cloud, the SaaS, and the services that are publicly wild available, right?

Matt Clapham: 

Mm-hmm (affirmative).

Wes Shriner: 

Which also means, wild attackable.

Matt Clapham: 

Let me take it to a next level. You also have security within the service itself. And not just about the IT security around the edges. Here’s a great example, fraud monitoring. And I’m not talking about fraud monitoring against the company. I’m talking about fraud misuse within the app itself.

Kip Boyle: 

Do you have an example?

Matt Clapham:

Your credit card. When you-

Wes Shriner: 

[crosstalk]-

Kip Boyle: 

My credit card? I didn’t show you my credit card.

Matt Clapham:

Yeah, your credit card Kip. So most banks today, will look at the purchase orders, the purchase sequence, whatever the regular use, and they’ll say, “Wait, that doesn’t look right. How could you go from Los Angeles to New York in three seconds and make a purchase?” Right?

Kip Boyle: 

Okay, yep.

Matt Clapham: 

And so they’ll put some analysis, some monitoring on there, and then say, “Hey, that looks like somebody’s doing something with the credit card number they’re not supposed to be able to,” and they’ll flag that. That’s security monitoring of a form within the application itself.

Kip Boyle: 

Got it.

Wes Shriner: 

Indeed. Cool. Thank you. When we look at product services security, we also understand there’s a lot of duplication. Not inappropriate, but actually appropriate duplication, where we’ve got to have an event monitoring and incident response framework available to us on the customer side, not just the client side of our organization. We’ve got to build out our own blue team protection strategies. We’ve got to have our MITRE ATT&CK framework in place, in order to protect, first, the enterprise, and then the services.

Matt Clapham: 

Yeah. A lot of the same things you would normally have in a typical enterprise, you have to have focused on that particular service, because we have to have that whole soup to nuts, beginning to end life cycle covered. So all of the same things you would have in the enterprise could be, and usually are replicated there. Or there’s some sort of shared team that can do both, depending on the nature of the particular product that’s in question there.

Wes Shriner: 

And your product network is going to need a part of that. You’re going to have compliance accountabilities as well, right?

Matt Clapham:

Yep.

Wes Shriner: 

Your product may have compliance accountability separate from your enterprise.

Matt Clapham:

Yeah. So you might have to deal with regulation in different regions and different product types, so that, you have to not just worry about the certifications that the enterprise cares about, but you have to worry about the regulations that apply across the industry or the certifications that the customers care about. That may not be the same as the enterprise.

Wes Shriner:

And if your service is operating in different countries, then you may have different laws that you’re accountable to depending on every individual country, state, or in the case of some places, county.

Matt Clapham: 

Yep. You might have things like in country for country or even worse in country for country by country.

Wes Shriner: 

Tell us what that means.

Kip Boyle: 

Yeah.

Matt Clapham:

I’m not a lawyer, so I have to do the disclaimer. But there are some countries that obliged, not only the running of it in country, but the operating of it in country, too. So that means that it would have to be operated by people who are in that region, in that location, in that country, operating on IT assets or cloud assets that are operating in a region in that country, and at some data center there.

Wes Shriner: 

So when we look at the European Union, we looked specifically at Ireland and Germany, who have some very specific privacy laws about what data is available about their citizens, where that data can be stored, and where it can be viewed. Is that what you mean?

Matt Clapham: 

Yeah. That plus, imagine all of the IT stuff behind it, all of the virtual machines that are running in the cloud, would have to be in that country as well, and run by people who live and work and operate in are probably citizens of that country. So you can’t just put it in Ireland, if you’re serving wherever-istan. You’ve got to have it actually run by people in that country.

Wes Shriner: Outstanding. Well, thank you. That’s a really good understanding of what the product security and product services security look like. Let’s take product security and see how it fits into our organization. And then as we go through the organization, maybe we can understand and find, where do we fit in that role? Because that’s really what we’re doing here is, what is my dream cybersecurity job? And where do I see myself fitting into that? So let’s jump ahead and see what we’ve got.

The product security space has several common functions and tools, and you’ll find that they’re actually a repeat of almost every function tool of the enterprise security space. You’re going to find architecture, you’re going to find threat modeling, and testing, incident response, you’re going to find your SEIM and SOC, you’re going to find your risk and third party risk, and you’re going to find compliance expectations.

We’re also going to see processes and standards of monitoring and response architecture, engineering, test, and even the risk and compliance functions in the product space as well. I’ll call out some internal security partnerships. And this is, I think, it was an interesting exercise for me as we went through and looked at product security, and what partnerships it has to internal security orgs? And really it’s every one of them. We called out governance, risk, incident response, pen testing, and bug bounty. But really that product security is a partner across each one of the internal security organizations, because it’s mirroring every one of those processes.

Matt Clapham: 

It needs to have it covered. So we have to cover all the bases with product security. And that means that we need to either have a directly ourselves. Or we have to partner with the enterprise doing it for itself, and make sure that it’s covering not just the enterprise concerns, but also the greater customer concerns that would be part of product security.

Wes Shriner: 

Jumping ahead, I look at the enterprise dependencies and enterprise partnerships that the product security team will have. Sometimes these product security teams are actually not part or inside the corporate security team. They’re actually embedded in the product teams themselves. And I find that’s a really effective way to handle product security. As long as you have strong dotted-line relationships back to our core security functions.

Kip Boyle: 

Wes, why do you think that’s the best way to do it?

Wes Shriner: 

I don’t know that I’d say it’s the best, but I think it aligns you well with your product teams themselves, and allows the product teams to be in consistent alignment, in lockstep with what security is guiding them to. You have less of a conflict and more of a collaboration happening when they’re co-aligned like that.

Kip Boyle: 

Okay.

Matt Clapham:

Wherever we can find synergy-

Wes Shriner:

Do you have some thoughts on that?

Matt Clapham:

I’m sorry, go ahead.

Kip Boyle: 

Oh, I’d love to hear what, Matt, has to think about that.

Matt Clapham: 

Okay. I didn’t mean to cut you off. Wherever we have synergies-

Kip Boyle: 

No, no, no. That was where I was going next.

Matt Clapham: 

Okay. Wherever we have the synergies, it makes perfect sense. We try really hard to not reinvent wheels if we don’t have to, but we have to make sure that we have, as I mentioned before, that, hey, the product has some slightly different concerns there. But there’s some things that product security, we can recognize that, yeah, we need to have, like intellectual property protection, like making sure we’ve got a good, solid source code management solution, but we don’t want to sit there and vex on that and worry about that too much. We’d rather make sure that our intellectual property protection team or the team that’s making sure that we’re securing all of assets is engaged and working with us so that we can focus on, let’s go back to the design of the product, the thing that we’re making.

Kip Boyle: 

And what are some of the things that can go wrong, if that alignment isn’t very good? So you’ve got a whole bunch of security people, not in the security org, but they’re making super important security decisions, but the friendship there just isn’t working.

Matt Clapham: 

We try really hard to keep a good friendship. But some problems that I’ve seen go wrong, and are usually telling indicators to me, that say, okay, I got to go and talk to some different people and work some stuff out. You see it show up as things like a duplication of effort, or a classic example that I’ve seen over the years in both jobs is, requirements that come in late in the game, because they weren’t fielded upfront. And it was one of those kinds of situations where differing concerns, differing focus areas means that you have one set that comes in, and says, “Hey, yeah, this design makes sense. It’s really secure for the design of focusing on the product.” But when they actually get to implement it and the enterprise team tries to actually go and deploy it, they’re like, “Wait, what about this one security setting that we really care about?”

And to them, it makes perfect sense. They really worry about unencrypted S3 buckets, or just pulling some example out of the air. And that makes good sense. But from product, we say like, “Well, they already encrypting the whole data stream in the database, so we’ve got it covered.”

But it shows that you’ve got two teams that are not talking and coordinating very well. And especially when you have a team that’s already internal and doing that, already in parallel to the product security team, it really helps if they can be roughly on the same page.

Wes Shriner: 

One of my favorite challenges is when the product security team says, “We need a different security policy than the rest of the core security team, because our policies are different than your policies for what we need to enforce.”

Matt Clapham: 

Sometimes they’re right.

Wes Shriner: 

Well, sometimes they’re right. But-

Matt Clapham: 

Sometimes.

Wes Shriner:

for the most part, once you start cornering them and saying, “Hey folks, I need you to write down what you would propose or draft as an alternative security policy for your product space.” Once it’s written down, it actually really looks similar. There’s really not a lot of difference.

Matt Clapham: 

Yeah. There’s often a lot of synergy there. The biggest challenge I’ve seen is in the flexibility of some of that stuff. So the classic example I can think of is, the enterprise might say, “You shall only use this one particular algorithm for this one particular encryption in the backend of the database,” or something like that. And the product security team might have to say, “Look, we can’t specify that, because we need it to be flexible and the customer gets to pick.” So that’d be one of those kind of conflicts, where if you read the letter of the law, it’s like, “Well, no, we’ll never be able to resolve this.” But really what you need to get to is, we both want encrypted databases, it’s just, we need to have a setting that can be set. And if you’re an enterprise, it has to go over to this setting, and if you’re just a generic customer of the product, then it could be whatever.

Wes Shriner: 

And I would argue that if your policies and standards are describing a specific vendor for your solution or prescribing a specific way to meet the requirements, you haven’t written standards, you’ve written guidance.

Matt Clapham: 

Policies, standards, procedures, guidance, yet there’s a lot of variants in that taxonomy.

Wes Shriner: 

Yeah. There is. There is.

Kip Boyle:

Yeah. It’s defined a lot of different ways.

Wes Shriner: 

Good stuff. Well, some of the common enterprise partnerships we see are marketing, sales, legal, and privacy, are good friends of our product security group. I’m going to-

Matt Clapham:

Yeah, marketing-

Wes Shriner:

jump ahead… Go ahead.

Matt Clapham: 

One comment on marketing and sales, is more and more, it’s important to have that partnership, because they’re starting to get requests from customers that say, “Hey, tell me about the security of this product. How do I make sure you’re not going to cause my environment to get owned?” Like that earlier example we talked about.

And so the sales team needs to have at least a basic set of talking points, a basic understanding of the process that the company uses to make those secure products, so that they can go back and talk to them, and say, “Yeah, we try really hard, and here’s some of the kinds of things that we do.” So that they can, actually, preemptively get in front of that security concern of the customer or potential customer.

Wes Shriner: 

Calling out security as a competitive advantage is a risk, it looks like Cheerios did that. But even when you don’t call it out as a risk, it’s very much something that we need to enable our marketing and sales teams to be able to do, if they need to.

Matt Clapham: 

Yes.

Wes Shriner: 

So here’s a reminder of the swagger of how governance might work for product security. This is of a large organization, and could be one of many Fortune 100 companies, where the suggestion is that maybe 11% of your overall security budget might be a swagger approached. I’m sorry, 11% of your overall security staff might apply to your product security space. And then 17% of your overall security budget might be applied to the product security space. And that’s primarily because, in a lot of cases you’re extending or purchasing additional security contracts and third-parties to help you with your product security objectives.

Matt Clapham: 

Yeah. And some extra thoughts on the scale out there. From experience, I often see the scale out, it’s easier to gauge in terms of like number of products. So a typical scale out for like a product security leader or somebody who’s in the product security team, engaging with those product teams, is like about one product security engineer to about one to two dozen or 12 to 24 different products themselves. Or if you look at it from a developer number, maybe there’s one product security engineer to roughly an organization with 500 to 2000 developers.

Wes Shriner: 

That’s actually a pretty broad… One to 500 to 2000, that’s a pretty high ratio.

Matt Clapham: 

Well, that’s the whole development organization. That doesn’t include the number of products. There might be, depending on the size of the product, it could be two, or it could be 200.

Wes Shriner: 

But one to 2000 sounds like a pretty high ratio. That one person-

Matt Clapham: 

It’s just my swag.

Wes Shriner:

is going to be pretty busy. Well, and-

Kip Boyle:

Everybody gets in their own swag.

Wes Shriner: 

And understand, when you get in startups or mid market, this percentage may actually be turned on its head. If you’re the first security person in a startup organization, you might be 90% product security, and only 10% enterprise security in the beginning. And so understand, this is very much a rough swagger for a mature large organization, and your organization may be very different.

Matt Clapham: 

Yeah. And another thing to consider in the distribution of where the budget funds go, especially if you’re a largely geo-distributed organization, you’re your product security team is going to spend more in travel. Now I know we’re all stuck with the pandemic, and we can’t move around a whole lot right now. But in that world where we have product teams all over, engineering groups all over, we’re going to need to make sure that the product security engineers can get, at least, maybe, once a year, sitting down with the engineers on that particular product team, to make sure that they are touching base, building relationship, double checking things, just kind of building that rapport, that makes the product security experience go a lot smoother.

Wes Shriner:

Thank you. Good thoughts. As we step ahead into, what kind of jobs might we find in a product security organization, there are both some senior roles and some start-up roles, some intern and, and first-year type roles. So some of those senior opportunities might be transition opportunities from existing technologies. Glad your lights are working again. That might be an architect or an engineer. It might be a support engineer or a software developer. We might find some technical product and project management staff, and we might find some lead analyst type roles.

In the more introductory roles, we might find support engineers and software developers as well. We might find all sorts of analysts. And I mean this, compliance analysts and business analysts and functional analysts and security analysts and analyst PM hybrids, and SOC analysts. And I don’t know, every kind of analyst we can think of, am I-

Matt Clapham: 

Even data analysts.

Wes Shriner: 

doing that right? Say that?

Matt Clapham: 

Data analysts

Wes Shriner: 

Data analysts, you’re right.

Matt Clapham: 

Take all that detail you get from the product and how it’s being used and how it might be abused, and use that to create interesting ways to spot that fraud problem like I was talking about.

Wes Shriner: 

And so we have a very much excluded fraud and loss prevention from the overall security organization picture. That is a-

Matt Clapham: 

Understood.

Wes Shriner:

separate part of the organization. But you’re right, product security itself very much takes in the fraud and loss prevention accountabilities, in a lot of ways, in product security. Would you agree with that?

Matt Clapham: 

Yep. We need to make sure our engineering teams have to cover that as one thought. Because, you think about that trusted brand, we might ask them, how are you making money on this thing? And we need to make sure that we understand that and help them protect those assets, especially that lead them to money. And if it’s all about the dollars and transferring dollars around, then fraud might become an issue. And so for that particular product or that particular service, we need to make sure they’ve got that covered.

Wes Shriner: 

One of my favorite parts about product security is, in a corporate environment, understanding what my role is in relationship to the customer is critical to understanding the value I’m adding to my company. And sometimes in a corporate security environment, you’re two or three clicks away from customer experience. But in the product security space, you are right there. You’re on the front lines.

Matt Clapham: 

Sometimes I’m even one click away. Here’s an example, as we’re helping those engineering teams build that product securely, we may have to give them, not just making sure that they have secure features, we may have to help them define and develop security features.

Wes Shriner: 

Nice. Thank you. And with that, I’m going to jump ahead to give you the last word here on today’s podcast. I want to know, what have been the keys to your success? If you had a little brother or little sister that was coming through right now, what would you tell them to focus on? And what do you wish you knew then that you do know now?

Matt Clapham: 

Okay. Great questions. With respect to the keys to success, I did some thinking about it, and I think my ability to break down the complex problem into smaller chunks and kind of separate out the concerns, has helped a lot. Because that means that we’re not, focusing in on too much of the broad picture, and missing the key detail that actually is going to make that security difference. Especially as we’re engaging with the team, we’ve got to make sure they’re securing the right things. Not necessarily looking at the, oh my God, that’s really scary.

Wes Shriner: 

So you’re not going to eat the whole elephant all at once.

Matt Clapham: 

Exactly. So that kind of differing scope there. And then also being able to threat model, being able to understand, take a system, take that breaking it down into its parts, being able to look at that, and use that to say, “Okay, well, because I know how it’s put together and I know where the weak spots are, now I can plan ahead to where I need to add the mitigations to try to prevent some of those things from happening.”

So with respect to the, what would I tell a mentee? I would have look at two key things that I think are more covered now in the university level than when I was coming through it. But those two things are threat modeling-

Wes Shriner:

Did they have universities when you came through?

Matt Clapham: 

Thanks. Yeah, back when I had hair, let’s just say that. Threat-

Kip Boyle: 

That’s right, taunt the guests, just as we’re getting into the finale.

Matt Clapham: 

Thanks Wes.

Kip Boyle: 

It was a good episode-

Matt Clapham: 

I deserve it.

Kip Boyle:

but now.

Matt Clapham: 

All right. Okay. Just one second. All right.

Kip Boyle:

It’s water.

Matt Clapham: 

Yeah, okay. Mine’s not. Threat modeling, and I know that threat-

Wes Shriner: 

The episodes get better the longer we go.

Matt Clapham: 

Yeah. Threat modeling, I think we might’ve touched on it, I don’t remember it even talking about it much at the university level, but there was a thought about overall design at the university level. But being able to do the opposite, analyze the design, break it down, similar to that thing I’ve been talking about with bringing stuff down. But then be able to take that plus the threat model and say, “How do I design solutions? How do I design those mitigation?” So if you’re a mentee looking at threat modeling, and then at design analysis, I think would really help a lot.

Wes Shriner: 

I’d like to take this moment to call out that Adam Shostack, who is the author of a threat modeling book, recently published, along with 25 other authors, the threat modeling manifesto. And so it’s a two-page manifesto. It’s found at threatmodelingmanifesto.org. They didn’t pay me for this. And I do recommend you take a look at it, if you can.

Matt Clapham: 

Oh, they didn’t pay me either, and I would second that motion.

Kip Boyle: 

Yeah. It’s good stuff, and we definitely need it.

Matt Clapham: 

Yeah. Okay. So the last point about what do I know now that I wish I knew then, starting out in security and even starting out in some of the more product security kind of role, even at the enterprise level, I was a little too compliant centric. Here’s the procedure, here’s the thing, did you do it?

In order to achieve the outcomes, we have to fight the war, not the battle. What I know now is that, focus on the war and that will eventually solve all the other battle problems. So being that ability to be that business partner, be flexible, to not just say, you have this one thing you have to hit. No, it’s saying like, “Look, we’re trying to make a good, successful outcome, a good successful product, where you can put trusted on the box and not worry about it. So let’s focus on, what do we need to do to get there? And maybe we have a few misses along the way, but we’re still going to eventually succeed in that goal and that outcome.”

Wes Shriner:

Matt, those are wise words. And I’m glad to have you on this episode today. I think you’ve spoken wisdom to Kip and I, as well as to the audience. And I think this is going to be a fun one to put in the books.

Matt Clapham:

Thank you. Happy to be here.

Kip Boyle:

Yeah. I’m really glad you were here, Matt. Okay. Everybody, that’s kind of the episode. I just want to close with an idea that I’d like you to consider. If you like the series, if you like what we’re doing here with the podcast with Wes and our guests, such as Matt, then you might want to get this free guide that we recently put together to help you to be irresistible, really, to cybersecurity hiring managers, and it’s called, Play to Win: Getting your Dream Cybersecurity Job. And you can actually see a couple of pages of that guide right here on the slide on the screen, pages six and seven.

And so what we’ve done is, we’ve created a way for you to take the skills that you’ve gained from playing capture the flag, and put them to work, to help hunt down and capture a dream cybersecurity job for yourself.

So I think you should go check it out. It’s free. Go to yourcyberpath.com/pdf. The URL’s right on the screen, for those of you who are watching. And we’d love to know what you think about it. If you think it’s on point, let us know. If you think it’s lacking in any way, I would love to know that as well. So, all right, so until next time, just remember, you’re one path away from your dream cybersecurity job. Or as Wes likes to tell me, you’re just one click away, in the COVID economy.

Wes Shriner: 

There you go.

Kip Boyle: 

Thanks everybody.

Wes Shriner: 

Thanks all.

Kip Boyle: 

See you next time.