Mastering Organizational Security Culture with Trupti Shiralkar

TLDR;

• To build a security centric culture, first educate other teams about security roadmap & improve security awareness. It can be as simple as Brown Bags or Lunch & Learns focused on specific Security Areas like XSS or Cryptojacking.
• In order to prioritize & address security debt, have a combined Backlog Review discussion between Engineering & Security teams. Instead of separate discussions.
• To work with today’s Supply Chain Security challenges, define & follow standards for Open Source components & have tooling in place to evaluate these components.
• For Risk Management, focus on Thread Modeling & Threat Agents for quantification & prioritization of Security Risks.

Transcript

Host: Welcome everyone, to a new episode of Scale to Zero. I’m your host, Purusottam, co founder and CTO of Cloudanix. Scale to Zero is a forum where we collect questions from curious security professionals and invite security experts to learn about their journey and also to get these questions answered. Our goal is to build a community where we learn about security together and leave no security questions unanswered. With that, let’s get started with today’s episode.

For today’s episode, we have Trupti Shiralkar. Trupti is the Senior Engineering Manager for Product Security at Datadog.

She has over 17 years of experience in the industry focusing on security and privacy. And she has had various roles in security prior to Datadog, like working as a head of Apps at Illumina or Product Security leader at Amazon. Trupti, it’s wonderful to have you here. Thanks for joining me in the show.

Guest: The pleasure is all mine. Thank you for having me.

Host: Absolutely. So the way we generally do the recording is we have two sections. First section focuses on security questions, and the second part, which is the fun part, is the rapid-fire section. So let’s start with the security questions.

And in that, I want to start with your experience in incorporating security into product life cycles. So for someone who is starting into product security,

How should they approach the security in the product development process and what are some recommended approaches to it?

And one of the key things that we see with product security is getting consensus from team members and also from the EXX. So how should they go about it as well?

Guest: That’s a great question, Purusottam. Anybody who’s new in the world of product security, it’s extremely important to understand that product security is not at all about rolling security tools into CI CD pipeline and blocking developers for high critical beliefs. That’s the first big no no if you are new in the organization. If you’re new person doing product security, I highly recommend spend first three to four months of your time at the new job understanding the organizational culture, especially the engineering culture. Understand the structure of engineering. Is it top down heavy, it is bottoms up heavy or it is relatively flat structure. Spend some time creating application inventory. If it already exists, great. If not, forget about security. Understand what are the applications a typical organization has that you want to prioritize before applying security there next thing, as you said, how to build consensus.

So product management offers very differently than engineering culture. They operate very differently than the security engineering.org. Then there is Tpm.org, right? TPMS are like nervous system of the organization. So it is extremely important for a product security engineer to understand all these different personas. And they need to identify who are the change catalyst that can bring in the desired security change.

So in my earlier days Purusottam, I used to have fun lunch and learn conversations with developers where I would order Pizza soft drinks for everybody and walk them through, let’s say, Capital One breach or how server side request forgery works. We need to make security interesting, intriguing. We need to make engineering audience who want security. We need to break the knowledge silos. So having lunch and lunch does the trick. Second thing, in order to deal with executives, whether engineering executives or product management executives, we need to think strategically. So having a strategic roadmap or some sort of strategic doc that can well align with organizations mission vision, that really helps a long way.

And once you have this initial understanding and basic foundational relationship with all the key players then start talking about how to bring in security. I hope that answers your question.

Host: Yeah, it does. And you touched on many areas and I want to talk about that a little bit. Like the first thing, you started with was culture and that is very key when it comes to not just engineering but also security. Right. Because generally, engineers don’t pay a lot of attention to security unless they have to. So your idea of lunch and Learns or brown bags where you can say take one topic and go do a deep dive that makes a lot of sense and like having the inventory or having the roadmap of your security so that you can talk to your EXXs will certainly help for somebody who’s starting in product security. Right? Yeah, absolutely makes a lot of sense.

So I want to dive little deeper into this. Right? So when it comes to looking at like when working on product security, when looking at risk which needs to be prioritized most organizations follow like a risk matrix with two dimensions probability of a risk and its impact. And sometimes it feels incomplete because some of the risks, there could be multiple risk groups fall into the same category, either critical or high and prioritization can sometimes become a challenge. So in your analysis or in your like, how do you approach quantifying risks and how should that be used for prioritization?

Guest: So the reason why we see risk is broadly calculated in terms of likelihood versus impact. That’s because NIST 800 Framework series advocates that approach since last 20-30 years and we have not taken efforts to make changes to that fundamental approach and it’s very easy to think in terms of okay, this is the security issue I have, how often this can get exploited and what’s the worst case impact can happen. Right. But if you look at today’s modern architecture, especially based in cloud, gone are the days of monolith applications or few risks.

Now we have thousands of risk in extremely complicated environment where there are a lot of dependencies. So just by removing certain dependencies the risk rating can change. Right? We need a newer approach for calculating risk and I highly recommend all product security engineers to start thinking about spread engine are you trying to save your application from malicious insider or are you trying to safeguard your application from an external state actor who has all sorts of resources in place? Right. So thread agents is extremely important to add to that equation. Next, it’s not. When it comes to prioritization, it cannot be just about likelihood and impact. What we need to understand is no matter how much good job we do in identifying the risk, the real achievement is in mitigating the risk faster.

What is the cost associated with remediation effort? Is this an easy quote or does this require architectural changes? Now, in order to make this equation even more interesting and realistic, we need to understand what are the existing security detective controls? Component setting controls are in place. So I’m going to take a hypothetical example of cross site scripting. A typical rule for cross site scripting is input validation or output encoding. Let’s say it’s going to take engineering team two, three weeks to fix that. Do we have a waf in place that we can tune to avoid this? Or do we need to look into creating a content security policy that may take three months? So we also need to understand what is the cost associated with remediation control and then collectively determine the risk. So every time I have discussions related to a particular product or application, I make sure the product manager, the architect, software developer, quality assurance person, everybody is in the room and collectively making that decision. Does that help?

Host: Yeah, it does. And I liked the thing that you highlighted about threat agent and threat modeling. Like earlier, maybe it was easier to just say it is high priority or low priority, but nowadays you need to look at different vectors and see that whether it’s a threat or not, how severe the threat is, it cannot be just categorized in high video blow. And you are sort of done with your analysis, right? So, yeah, totally makes a lot of sense. And with the workloads moving to Cloud and let’s say Kubernetes, it has become even more challenging. Right.

Guest: I would like to suggest all security vendors who are out there, if along with the deception, if they can somehow summarize the cost to fix the issue, that will take security engineers and developers a long way. It will actually motivate them more to fix the issues right away or come up with some sort of strategic roadmap to if the remediation is going to take a lot more time.

Host: Yeah, I love that because that in a way shows how it will impact your business overall. Right. Like adding business value to your security items as well. Yeah, that’s lovely.

So I want to dig a little deeper into it. Like once, let’s say a company finds out that, hey, there are so many risks that we need to address, then often what happens is because you have business to run, you don’t focus on addressing all the security issues, we received one of the questions from a fastgrowing fintech startup. What they are trying to understand is some of the things which are not getting addressed, they go into security debt similar to like technical debt or product debt or engineering debt.

So how should they address that?

Like how should they define and measure the security debt and how should they again prioritize them so that they don’t stay in the debt category? Right? Rather they get addressed.

Guest: This is another great question for Purusottam, 15 20 years back when compliance used to be the main driving force behind security efforts, security used to be completely security team’s responsibility. Fast forward 10 12 years. With the rise of cloud, security became a shared security model. Right? As for the cloud security model, it’s a shared responsibility between security teams and engineering teams. But if you look at further evolution of native cloud security application, enterprise application, it is so complex and the velocity at which engineering organization design Buildership these new features, there is no way for security team to do the intervention, even talk about security depth. So I would highly recommend try to merge your security processes as part of engineering agile processes. They have a weekly bug grooming meeting and if you look closely, security issues are actually software issues, right? Or they are It issues.

If they are related to infrastructure, these are actually bugs just like any other software bug. Why have a separate routine or process to prune them? What I started doing is I started sitting down with engineering teams when they do their backlog pruning and that’s where I got like five minutes to talk about security issues and which one we can mitigate right away. In this way we kind of develop that discipline into engineering culture of treating security equally and not separately.

Host: Yeah, I love that idea of like working with the team when they’re doing the backlog review. So that way security issues get also prioritized and get addressed so that you can clear off your security that one at a time, right?

Yeah. Makes a lot of sense.

Guest: You can be the voice right there and there itself. If some issues have potential of creating a breach or big security incident, your voice will be heard by all the stakeholders in that meeting itself.

Host: Yeah, agreed. And that also sets the culture that you are not treating security as a second class, but rather part of your product development lifecycle.

One of the things that you mentioned multiple times is like workloads are moving to cloud and compliance was maybe like the right solution, maybe 5 10 years ago, maybe not today. And with the workloads move to cloud, with the workloads moving to let’s say, Containers or Kubernetes, there is adoption of open source software has gone like wild, right. Every application that runs nowadays uses some sort of open source library or some tool. So with that comes the supply chain security related issues, right? Like we saw the attack that happened on solar vents or Twilio or even PyPy recently. So we got this question from a first time security leader. They’re trying to understand like

how should they tackle these challenges, like the supply chain security related challenges and where should they start?

Guest: Supply chain security issue is a big problem. And if you notice pushup, President Biden recently signed an executive order to tackle supply chain security issue more strategically.

So it’s a big problem. Around 80% to 90% of the components library constitute to any major enterprise application architecture, right? So it’s a big issue. Where to start? I would recommend start using a sleeprung approach. First of all, have an organizational open source security policy or standards that will become a forcing function and it will help engineering to think about how to improve the overall stage of open source security as part of their product. The second prong is around schooling. There are so many security vendors out there who promise earlier they used to advertise as software composition analysis tools. And after precedent Biden’s executive order, they started marketing SBA tools as sbomb software.

So having tools that will gel very well with the engineering processes and engineering culture and integrate well with the CI CD pipeline is the second most important thing. And third one is education and awareness. Education and awareness takes time to build, right? That’s why that’s the first one. The open source security policy will make everybody seem to take a look at this very strategically. The tooling will help create a footprint. How does my open source landscape look like? Where to prioritize and where we can afford to wait? Usually I tell engineering teams to categorize your open source software into two types of inventory, something that is low impact. When you upgrade these low impact components regularly, it will not cause any regression.

And components which are capable of causing regression, the high impact one, they can be updated very less frequently. So whether there is a security issue or not, just get into the habit of staying on top of your updates.

And third thing, unfortunately in today’s universities, there are not enough cybersecurity education available. Those who have keen interest in security, push with them. They go attend one security class, right, and get familiar with the topic. But that does not help them to educate about all sorts of security issues. So I see that our universities have to do a big lifting in every software, engineering or technology class. They must discuss about security, whether it’s a database class or algorithms class or programming languages. They must discuss what are the secure ways of doing coding and what are the insecure ways that can introduce vulnerabilities.

And only when that level of awareness will come, the developers will start introducing these reoccurring security clause into open source code. But that’s a long way to go and industry cannot wait for universities to change, right? So I highly recommend all these software companies, they should have security championship program to elevate their developers security knowledge.

Host: So I like how you put them in three buckets, starting with your culture, like mapping them to culture tooling and having a standard. I just have one follow up question on the first step, like setting up the standard. At what stage of the company would you suggest that we define the standard? Let’s say a startup, should they be thinking about a standard? Or maybe once they are at a growth phase, they should think about it.

Guest: That’s a great and very practical question. So till CDs we have approximately 20 developers or total 20 strength, right? And then it can go further. I would recommend do not wait for your first security engineer to come join you. This should be part of the CTO’s responsibility to have these standards in place.

The thing is, people come to startup world from all sort of backgrounds, right? They may be security aware, they may not be security aware. So having a standard right from the beginning really helps them to understand, hey, this is important, that’s why the standard exists and then they follow. But if there is no standard and you wait your startup to grow to 100 to 100 people, then it becomes difficult, right? Then you have to convince all the people to follow. So have really high bar for security right from the beginning. Because I genuinely believe security is no different than quality engineering. It’s part of quality, right?

Host: So I think it goes back again to setting the culture right from day one, right? Whether it’s your engineering culture or your security culture. So if you have the standards and everything in place, somebody who joins you, they know that it’s not optional, whether it’s a must that you have to follow security best practices that laid out by your team. So I love that.

And the next question is somewhat very similar to it because most of the times culture is what holds the teams together, right? And every company has a different culture. Like some have engineering, some sales, some security development. So,

As a security leader, what methods would you recommend to bring awareness and sort of create a security-centric culture in an organization?

Guest: It’s a very important question that you ask because as I said in the beginning of this podcast, security professionals, when they hyper, focus just on pooling. It takes a long way to go. The right way to focus to solve this problem is by taking a look at existing culture. I recommend starting at the hiring process as an organization.

When you hire new people, whether sales folks or HR people or developers, QA Assurance Engineer, take a look at their security DNA. Do they value security? It’s a different story whether they have security knowledge or not. But it’s extremely important to make them understand security is important for this organization. And we are going to check your security knowledge. So start from hiring then have programs like Security Championship. Security Championship really creates great team building experiences. Then they introduce engineering or to the most exciting and nicer side of security else security is taken.

I mean security is so reactive. There is an incident, there is a breach and everybody starts running. That’s not the way how we should introduce security to organisations. Let’s have some fun events, do a lock-picking village, teach them about fun cryptography side or privacy. And at the same time we need to make them aware that hey, if you don’t comply with this privacy GDPR stuff then the fine will be millions and billions. So we need to bring that realistic picture. So I would recommend don’t just talk about vulnerabilities and finding. Show the other side of security.

Host: Yeah, I love that. So it should not be like a one time activity but rather you constantly engage with other teams and show them the value and teach them as well. Right? Like what is the impact of cross-side scripting? Do a brown bag and let everybody listen and see the impact and that will help them even in their prioritizations that they think about like engineering teams think about security as one of the areas as well. So yeah, makes a lot of sense.

Guest: I’m going to add a little bit twist to this through Security engagement model. So let’s say out of 100 applications you have top five critical applications have embedded security engineer working with those teams to elevate the overall posture of the security. Let’s say you have 80 medium low risk application.

Go train security champions. There are lot of developers who are very curious about security. Go train them, teach them how to use basic tools. Go actually build security tools for them so that they can easily weave in those tooling in their existing SPLC. And then bottom 15 20% you can provide a self service so that anybody can use it. And coming back to the original question poroshowktam when somebody’s designing a new service, do they always go to UX researcher to get UX help? Do they always go to performance Engineering to get cost performance Optimization best practices? Why keep security silos? They should be thinking about security along with cost performance and other aspects of software engineering and education. Does that trick to them?

Host: Yeah, I like that suggestion.

That the way you do UX research and you get feedback and you include that as part of your engineering scrum or agile plan. You should have security research as well and you should add those as stories for your sprints. Yeah, makes a lot of sense. And thank you so much for sharing all of these. There are many things that I learned which I can apply at Cloud Ethics as well. So yeah, thank you so much.

Rapid Fire:

So now we’ll move to the rapid fire section.

Host: So the first question is which animal from the animal kingdom do you relate the most?

Guest:I like panda the most and I relate to panda. I like the way pandas communicate and their coloring as camouflage and they are exotic and rare.

Host: Nice one liner quote that keeps you going.

Guest: Be fearless. Sky is the limit.

Host: The last question is what advice would you give to your 25-year-old self starting in security and why?

Guest: This is really tough one. Looking back, I would say if you’re starting your security career around 25, be open to all sorts of experimentation. Do not say no right away. Go try those new security tools. If you’re in cryptography, go try privacy engineering. Go try application security. Do not restrict yourself.

Just experiment. Experiment till you find your way.

Host: I love that like have the learners mindset so that you are not like siloed to one specific area but experiment so that you figure out what’s the right path for you to move forward.

Thank you so much. Trupti, it was a very good discussion. Like there are many things which I learned. Looking forward to learn more from you in future.

Guest: You’re very welcome. And once again, thank you for having me Purusottam. It’s an honor and privilege.

Host: Thank you so much for your kind words and to our viewers, thank you so much for watching. Hope you learned something new. If you have any questions around security, share those@scaletozero.com. We’ll get those answered by an expert in the security space. See you in the next episode. Thank you so much.