From Detection to Recovery: Understanding Incident Response Lifecycle with Giorgio Perticone

• Listen to the episode on Spotify

TLDR;

• One of the key skills necessary during Incident Response is to stay calm. Analyze the incident before acting on it. Find the right balance between Speed and Accuracy of Analysis
• For Preparedness, Define Processes, Playbooks, Document Landscape and Stakeholder Communication plans to avoid figuring things out during an incident. Reducing the stress on the Incident Response Team.
• Containment is key to limit the impact and also avoid repeat breaches / attacks of the same kind. Quick decision making and proper incident analysis will help in containment plan in future.

Transcript

Host: Hi, everyone. This is Purusottam, and thanks for tuning into ScaleToZero podcast. Today's episode is with Giorgio.

Giorgio is a senior detection and response consultant at Vectra AI, and he has a deep passion for incident response and digital forensics. He also has over a decade of experience in field of system administration. He's a big believer in communities and have been contributing to multiple ones in recent years. He also hosts a podcast with the name Security Break and simultaneously runs a startup community called Security Cert. Thank you so much, Giorgio, for joining me in the podcast.

Giorgio Perticone: Oh! thanks for having me.

Host: Absolutely. So before we kick off, like I did a quick intro, but anything you want to add on top of that?

Giorgio Perticone: Yeah, well, I'll try to be quick with that. mainly... So I dropped from university, which is the thing that most of people ask me about. And I had the urge to start, you know, basically working. And my... I can say that my only skill was about computers. You know, I was just the… I would say probably the geek guy in the surroundings in my hometown. And that was just the best idea to pursue. So I tried just to find out what kind of jobs are out there involving computers. And I ended up being a system administrator, like you mentioned before.

That actually became quite a passion for me just to keep… an organization running from the technology point of view, what's very, very challenging and entertaining for me. And that was until I, myself and my organization suffered some breaches. I took it quite personal because I was the one-man band in there, right? I had to do anything from security to anything really.

And I, after that, I just realized that I enjoyed it a lot. The process of, you know, trying to understand what happened, how they got in and I can, you know, kick them off from my network. And, you know, there was that realization in there and just started looking for, okay, is this actually a kind of job that I can do full time? And yeah, that's how it started basically.

Host: That's a funny way of getting into incident response detection and response, like from your real life experience rather than academics. So where you learn more, I strongly believe in that. So hopefully we'll be able to touch on some of those ideas today.

Before we get into that, one question that we ask all of our guests is what does a day in your life look like?

Giorgio Perticone: Okay, well, I don't know how interesting that could be from my side, but... Well, I work from home currently, and that means that I barely wake up just maybe a quarter before I actually have to start working. Hopefully that doesn't get mad at my boss or something, but no one has complained so far.

Yeah, so I just wake up, just… Not so long before starting working, I grabbed just a quick breakfast already in front of my PC actually. So that's already, know, having a little breakfast and having a little work on the other side just to see what happened while I wasn't there.

Because I'm part of a 24x7 team. So someone was working before me, so I have to understand what was going on before me. So, I maybe read some emails, read some stuff while I have my breakfast. And yeah, I mean, we are in this industry. So that means that the rest of my day is in front of a PC. I just try to keep any chance to do something outside of the computer industry. that means that, well, I have a dog, so I take that as an excuse to go somewhere with him. I enjoy playing drums and I just started again recently.

So whenever I can, I just move from my desk to the other chair that's in front of something different, but still a chair. But yeah, mean, very quickly that's part of my daily routine and it's not so fancy or nothing.

Host: No, but I'm glad that you have a forcing function, right? You have a dog, so you have to take it out. So that way you are not glued to the computer 24x7 or like all the hours that you work. That's good. And you also play drums, which like music often helps you relax, right? And one of the challenges of being in the security industry is you can burn out. So I'm glad that you have two ways to relax from your day-to-day work.

So today we are going to talk about incident detection and response, right? Like how you got into this field itself.

So the first question that comes to my mind is effective security response requires quick thinking and experience. And you started your career with one of the live incidents and you have been working in cybersecurity for a few years and you must have seen a lot of breaches. Can you share an experience and security incident that you have dealt with?

Giorgio Perticone: Okay, so, I cannot say the details for NDA reasons mostly. it's hard nowadays, I have a problem, right? Nowadays, most of the incidents here involves pretty much ransomware and very standardized kind of attacks and patterns from attackers, which means there's hardly something that stands out between you various instances you face most of the time.

And that also means that unfortunately, the ones that actually stand out when it comes to, you know, incident response engagements are the ones that actually go very, very bad, or at least go very bad, you know, in the best way possible. I have something interesting in my mind right now that is fairly recent.

And it was interesting because it was a fake ransomware infection, meaning that we had to engage with this customer that was concerned because they found a lot of, how you call them, ransom notes. So basically a file where you can see the attackers asking for money in exchange for getting back your data or maybe prevent some exfiltration, but… know, upper investigation, that was it. Like there was no evidence of any infection or any exfiltration other than the files themselves that were written all over the device that was infected.

So basically we concluded that the attacker just tried, you know, to scare our customer, you know, just saving these files, but we're not really able to do anything, anything malicious other than getting access to the host.

That was quite different, if you ask me. Not very, maybe, or something. But that's part of the job, actually understanding how bad the situation is. And maybe sometimes it's not that bad.

Host: So that sort of tells me that you do not have to react right away, right? You should rather analyze it properly. I have never heard about fake ransomware attacks. I always thought if there is a ransomware attack, then there are ransomware nodes. Somebody is asking for some sort of like either money, crypto, whatever it is, right? But I never realized that there are these types of attacks as well. Interesting.

So let's say you did the analysis, you found out that there is not a breach like bigger breach. It's just the notes were left. What lessons did you learn from these incidents?

Giorgio Perticone: That actually was a pretty important lesson, I think. Most of the time we have the biggest challenge, is kind of, and of course I'm talking from the service provider point of view, right? We have to deal with our customers and try to understand what's their current emotional status and what's their reaction about the incident they are facing, right?

Sometimes they could be very, very scared and very, very eager to reach to the point where they feel safe, they recover from all of the damages they are facing, and they are just so eager to see results. They are just asking, okay, can we say it's okay now? Can we say we're safe? We can go back to business? That's usually difficult to say because you want, from an analyst point of view, you want to take your time, understand that everything is really safe, that you addressed all of the possible lateral movements from the attacker or any backdoors possibly, but that you found out really what was the initial access, so how they got into the network and make sure they fix it before declaring that it's all right.

In this case, it was actually the opposite. We don't really want to stress too much just when you see the first indication that something is odd or something is just not right in your environment. You just need to be careful, know your network, know your environment, know what are your skills and knowledge and how to address that kind of things. If you cannot, of course, reach out to someone that can help you.

Hust wait for the first results of analysis. Maybe it's not a common, as you said. I also spoke with some friends and people from the industry and most of them say to me that they've never seen something like this before.

But yeah, still, it could be that they still did not cause too much damage and you can feel a little more relaxed that you can fix the problem as soon as possible.

Host: Yeah, I like one key thing that you said, right? Like, wait for the analysis to finish. Because there is often that challenge of balancing between speed and accuracy. If you take a lot of time, then that doesn't help. And if it is not accurate, it doesn't help either.

So you have to balance that out. One thing that you clearly highlighted so that once the analysis is done, you at least know where to fix, what was the initial access point, so that even if you remediate it, it doesn't happen right away, or it doesn't happen again. You have sort of remediated it rather than just killing one of the nodes which got impacted.

So next question that comes to my mind is, let's say with every project, we do a lot of planning. I would believe the same thing would apply to incident detection and response. Like when you went to this customer, when you saw that they are breached, prior to that, you must have done a lot of planning in terms of what is their infrastructure, they land, you understand that, what does their network posture look like? You might have done a lot of planning.

So what are some of the key components that you have seen that should be thought about when you are building an incident response plan?

Giorgio Perticone: Well, there are different parts of it. There are processes and there are, let's say, more technical elements of an instant response plan. The preparation, so creating a response plan is really the first part of the instant response process itself. once again, I think the most important things are also the ones that are most commonly the biggest challenges, once again, from my point of view, from my experience.

First of all, is having a process at all? Do you know what to do, who to call, who to involve when it comes to something like that? Because the point is that many organizations out there just face any new incident like it's the first. Painting, Trying to involve as many people as possible, which is not really a good idea, or just struggling to reach out to the right people because they are not available, because they are on vacation, because they just don't know what you're talking about. on the other point of view, if we call about more technical aspects, think awareness of your environment and visibility are two very big challenges.

Like it's the trickiest and the more difficult cases are the ones where, you know, for me, the customer itself doesn't really know what we're looking at. Right. Okay. Maybe we have some, you know, breached or infected or, you know, compromised host and credentials, but we don't really know what those hosts are supposed to do. If it's, you know, if it's too bad, if we shut them down.

Or whether we are allowed to shut them down at all or who we have to involve to do that. So the thing is that, you know, the bigger is the organization, the more difficult it is to know everything about your environment. And of course, having, I don't even start to talk about CMDBs or getting a list of all of your devices and stuff, because it's kind of too far.

It's more an ideological thing. But at least having that knowledge shared between people, just not having a single person that if he's sick or anything, no one knows anything, that's very bad. And even when you know something, how to access those devices, who have access, how to get the data out of it because most of time you need to analyze something that is kind of pulled out from the host itself.

You don't want to work on the host while the attackers are there as well. That's not something that you want to do. So yeah, mean, those two big pillars, I think, are maybe something that most of the people need to focus on when it comes to preparing for an incident.

Host: So yeah, I like how you structured it, having a defined process, having awareness of your environment, having to know whom to reach out when something like this happens. Some of these are very key rather than figuring it out while there is a breach going on. Next question that comes to mind is, let's say I have the plan in place. How do I test? Because I cannot simulate what an attacker thinks or maybe I can.

What do you recommend? Like how do I test? How do I validate that my plan is effective?

Giorgio Perticone: Very good point, maybe one of the most important things. unfortunately, once again, it's very rare that I find about an organization that is mature enough to have a proper incident response program. But when they have it, it's even more rare that they ever tried it before. They just created it, maybe with some external consultants or something and it's written somewhere. I really hope that someone actually read it at some point, other than the consultant who wrote it. And the other thing is actually testing it. And with testing it means that you actually will need to simulate an incident. And I'm not just talking about hire a pen tester and make him try to exploit some remedies.

I mean, what the… We had an incident now, okay? Data were exploited. These hosts were infected, compromised. And now what do we do? So you actually should be there. You should start the process. You should look for the people, involve the people. And I'm talking about, I don't know, legal department, public relations, C-level that needs to take the business decisions, finance people that needs to verify that last money movement was actually approved or not, and see if they get involved, if they know what you're talking about, if they are available and they reply as soon as possible.

And have someone who, and this is very important for me, have someone who managed the, let's say the, it's not a project, but if you want, be the project manager of the incident response. So kind of handle both the, you know, all of those departments, non-technical ones. And at the same time, all of the analysts were actually looking at the data and taking conclusions from them. Aand kind of translating the technical part to the people that have to take the decisions. Okay, we need to, I don't know, isolate the domain controller. And if we do that, we shut down the company for at least 4, 8, 12 hours.

Okay, who can take the decision? Okay, let's call the CEO. Does he know what I'm talking about? Does he know who I am? And then, you know, that I can actually call him for this reason. Can he take this decision in a small amount of time, knowing what are the consequences? Okay. If he says, yes, who is the person who can go in there and click the button to shut down the domain controller or just isolate from the network or whatever? Can he do that in a small amount of time as well? Or do we need to wait for someone else from another part of the world to maybe wake up because he's not working on the same time zone? And do that for us. And maybe at that time it's already too late.

So that's the kind of thing. Do we test all of the process to involve all of the people every time, even though it seems like a very big thing? And we know that everyone is busy with their own daily duties. But still, all of this, if you don't test it, the point is that you're going to waste a lot of more time later on.

Where, you cannot work even if you want to do it because maybe they encrypted everything in your environment. And that still happens even more than I would like to admit.

Host: No, I think you highlighted some of the key points. One thing that stood out was having a dedicated sort of project manager for the incident, which I had not thought about, but they sort of take out the pain of communication, whether it's internal or external, or even between security folks, non security folks.

The project manager can do that activity for you, right? The folks who are doing the analysis, maybe they don't need to be disturbed every 30 minutes or an hour to give an update to everyone and things like that.

And yeah, on top of that, having the understanding of key roles in the incident response and for what you can reach out to them, how you can reach out to them would certainly play a major role.

And I think I have also seen some customers where they have the plan in place. They had the plan, like they created it six months ago. No one has ever touched it after that. And when there's an incident, there is panic because that document is out of date and you're trying to figure things out on the fly.

Giorgio Perticone: If I may, I need to be, let's say, honest with myself. Even though you really work hard on your plan, and you actually tested it and everything, there's always going to be something that does not really fit your plan, and you still have to improvise at some point. That's always something.

But as the wiser people than me say, it's better to improvise when you have a plan that you have to improvise all of it and have to have to do something different every time you actually want to do. I would say 50 % of a standard process and then improvise the rest of it. Then actually to improvise 100 % every time.

Essentially you want to save as much time as possible because you know already what steps you need to do. And, one last thing, just because once again, it's a topic which I care a lot about. That project manager role actually has some kind of, it has a name, right? It should be called an incident commander, but unfortunately it's nothing that I really see out there anywhere, right? So, you know, I just hope that at some point it becomes like a standard or something.

Host: No, I agree. 10 % or 20 % improvement versus 100 % improvement is a huge thing, huge difference, especially when you are in the middle of a breach. One of the things that you highlighted in the earlier responses, working with other teams. And often with incident response, you have to collaborate with other parts of the organization as well. It could be engineering. could be product management, and it could be leadership, it could be external parties.

What are the challenges that you have seen while coordinating with different teams?

And let me break it into two phases. One phase is when you are sort of working on an incident, and the other one is before and after the incident.

Giorgio Perticone: Well, I think when facing an incident, that's something I mentioned before, just know who to call. The problem is that, okay, you know that maybe you should involve your legal department because maybe the breach you're addressing right now, it involves sensitive data or personal data, right?

Do you have the number? Do you know the email address? Do you know who to involve? Or do you have to understand if you have a legal department, if that's internal, if that's external, so you have to actually call some kind of external entity at that point. And that's a big topic, right? Because there are entities that you have to involve from your internal organization and maybe third parties, being that, your consultants from the technical part or non-technical part, maybe law enforcement in a lot of cases. Maybe you have to reach out to your customers at some point or some other kind of stakeholders. Do you know how to do that? Is there a PR team in your organization? Do they have an emergency communication thing? that they can push in a small amount of time, or do we have to create it on the spot and that's pretty bad.

So awareness of all of the people that must be involved is the biggest thing for me. that's when you go back to testing it, right? Because you have different scenarios, for different scenarios you will have different people involved for different kind of escalations of the incident. you will involve even more important people in the hierarchy of the company. So maybe most of the time you just talk to your CISO, the Chief Information Security Office, that makes the most of the decision when it comes to security in organization.

But at some point, when it becomes pretty critical for the business, you want to involve your CEO and so on and so forth.

So that's the thing. Those are people that are not so easy to be contacted when it's not like you work with them every day. And that's not, unfortunately, the security team scenario. So yeah. I mean, what was the question again?

Host: So one is like, what challenges have you seen during the incident and before the other part is before and after the incident is over, let's say. What are the challenges you have seen? Yeah. I think one key thing that you are highlighting is preparedness. The more you have things written down, the more you have practiced, like you have done fire drill exercises, the more aware you are. Similar to how you need to be aware of your environment, you also need to be aware of the incident response plan, be it communication plan, PR plan, be it analysis plan, whatever it is. You need to be aware of that instead of figuring things out on the day of the breach or during the breach.

And so that is more around when it is happening, like when you are in the middle of it. Before the incident or after the incident, how do you collaborate with others?

Giorgio Perticone: Well, before, I think we said it already, right? If you are testing your plan before facing an actual incident or in the middle of incidents, you kind of create fake incidents so you can test it. That's the way you address it before.

After it, that's all of the part of what we call the lesson learned, right? We want to… Let's say that we recovered from our incident, right? We are technically fine. We can go back to normal day to day. At the same time, we want to bring everything we learned from the incident itself, how they got in, what was the problem, what kind of software code was involved, how we can improve that, how can we make sure that doesn't happen anymore, at least not in the same exact way, because we cannot prevent all of the future incidents, but we can make sure that we are prepared for this type of incident the next time. Either we prevent it or if it happens, we know what to do. That's basically it. If something happened that was not already on our plan, that we were not really prepared already, that's the thing. We need to update our plan. What was wrong?

We needed to… Let me think about the scenario here. We needed to rebuild an entire network from scratch because the incident was pretty bad. Who is going to be involved with that? Do you have system administrators or engineers that have access to your cloud environment or your virtualization environment and can recover some old backups of your virtual machine or something? Do you have contact for those people. Do you have already spoken with those people that in this specific case, you have to stop doing what you're doing and listen to me because you need to help the organization?

That's basically it. Include new people in the plan whenever it makes sense to include new people, include new steps. Maybe identify that you had a lack in visibility so you cannot really detect that kind of incident the next time. So I don't know, either you improve what you have already in your environment. you need to, yeah, sometimes buy some new technologies you don't have in your environment. Either you have the capability to build it, right? That's something fine if you can do it, or you have to buy it. or you have to start looking for something out there.

Maybe there's another department that really looks into technologies to buy. So you need to contact that as well, and you need to train, of course, your people in order to be able to analyze that kind of data or be able to use that kind of tool because it's going to be something new. And of course, everything needs to make sense with everything you had already before. So if it's a new tool, you don't have the knowledge, no one knows how to use it. You're spending the money, but it's not really… helping, you need to build new, let's say, pieces of the planet you have already so that that plan still makes sense, know, in a high-level view.

Host: Yeah, no, that's a great way of explaining what challenges you face and also how you can resolve them as well. One particular thing that I really liked is bringing new people because they often bring in a different perspective. So yeah, that certainly helps. Like if you have experts in the team who were maybe working on a different breach, they might have seen something similar in the past.

I want to slightly pivot to containment. So you recently shared a post about your experience at B-Side Switzerland and you shared your thoughts on the shift from detection to containment. So can you maybe give a little bit of idea of how do you see them different? Like what do you see containment as and how do you see it as different from detection?

Giorgio Perticone: So yeah, I recently had this talk when I was talking about this, and we're basically talking about two different phases of an incident response process. I just realized from my point of view that for a very long time as an industry, we have focused a lot on the detection part of it, because of course, if you don't know that something weird is happening, you cannot even start to respond to the incident. And that's very important. That's great.

And I think we did a lot on that part in the recent years. We have access to a lot of technologies, open source ones, paid ones, a lot of vendors. You can really choose whatever you want. For each kind of technology, you have a lot of alternatives and so on and so forth. So there are plenty of possibilities in there.

If you have time, resources, and money, of course, you can reach the level of visibility that you really prefer. But the point is that I see a lot of organization, finally, of course, focusing on that, spending money on that, and having a lot of visibility. But the point is that when it comes to, OK, I see an incident. What I do now? no one really has or takes the responsibility to say, okay, I'm going to go in there and stop the entire device, the entire user or the entire department, the entire network, depending on how bad is the scenario. And just because we want to prevent the things to get worse, right? Because we saw

Host: Yeah, but even from spreading more and more. Yeah, makes sense.

Giorgio Perticone: Exactly. And that's the thing. I had actually a lot of different scenarios, once again, as a consultant, from the service provider point of view, where we have customers who found out about the incident, maybe call us to help with them. And then we say, okay, it's actually pretty clear. We see this. see this. This is also infected. This user was compromised and their credentials are maybe out there on the internet, anyone can use it right now. We need to stop this part of the network. We need to stop this thing and this thing.

And then everything freezes because no one wants to do it. No one knows how to do it. No one has the authority to actually take the decision. And then, you know, either they say, no, we cannot do this for whatever reason, which is, I mean, I understand the reasons, right.

Companies don't want to lose money to stop working. So it's pretty hard to take that decision like that. But at the same time, if you don't take the decisions right now, it's just going to be worse. And we had various scenarios where it just took too much time to decide what to do. And eventually a ransomware infection spread across your network. Or another thing is, okay, you add your total compromising, right?

You do all of the work that is needed to recover from that, but still you don't focus on, but how I prevent this to happen again. Okay, I need to, I don't know, just another example there. I need to patch a system that is internet facing and was compromised. And that's how the attackers got in. But okay, but if I apply this patch right now, I need to shut down the system for, I don't know, few hours. And this is customer facing, so I cannot do that right now. I'll wait for, I don't know, the next month.

Cool. Just a week later, the same attacker, the same compromise, the entire network again. This really happened. And it was a fairly important and big company. That's the thing, We need to, think we at a point as an industry to focus our shift to the second phase, right? Because if we, and we have the technology already, right? So most of the providers are there to do not only give you the possibility to identify breaches but also help you with the containment and the response, right?

But in my experience, very few companies decides to actually enable those features to automatically contain stuff because once again, it's hard to give the responsibility to tools instead of humans. But that's a huge, something that is slowing us down a lot and it's giving attackers a lot of leverage versus us.

Host: So that they can repeat sort of the attacks. I love that example that you gave, right? The organization decided to maybe work on the containment in a month and there was a repeat attack.

So apart from that, what other challenges have you seen when organizations start shifting their focus from detection to containment? Is it more of budgetary constraints or is it more of the focus that maybe the team or the leadership is more detection-oriented, maybe that's why they are spending more time on that, not on the containment. What challenges have you seen organizations face?

Giorgio Perticone: I think, and this is my personal point of view, of course, we have two big problems. One thing is the authority of the thing. So we have very few people inside an organization that can take so very difficult decisions and can give you the okay to once again contain part of the network or something.

Once again, I really believe that you need to grant that kind of authority to someone else that in case of emergency, they have the knowledge, they know how to interpret the data and they can understand when is the time to take the decision. And maybe it's better to try to justify your decision later on while you did it instead of just taking your time before you do it because it's just going to be worse.

And that's the first thing. And I still believe that, you know, that it could be, it's not the only answer to that, but that incident commander role in there could be one of the options that you could have if you have someone who is, you know, entrusted to take those kinds of decisions in the moment of the emergency.

The other thing is that still it's really hard to understand, you know, when the incident is so bad that requires that kind, that level of containment, right? We were talking, sometimes you just need to contain an individual laptop. And most of the time that's fine, that's okay. When it comes to servers, when it comes to part of the networks, it's an entire different story.

So the other thing I believe is, and we go back to something we mentioned before, of your environment and not only knowing what's out there but also how important that is for your organization, from a business perspective. What happens if I shut down this server? Do we lose money? Can we still provide some kind of products or services depending on our kind of organization? And if this goes down for 24 hours, how bad it is for the company?

On a scale from one to five, is it one or two, or is it actually five? And depending on the kind of knowledge, you know, what kind of data you have to see first to take that kind of decision.

So my point is if you have that kind of information and you see that you have, the business criticality for this device is one or two. Okay. No matter what, no matter what, I will stop the device, and do my investigation and then say, okay, it was actually fine. I didn't have to do it, but nothing happened. It was not that bad. didn't lose, I don't know, 50 % of our revenue because of it.

So I actually still had a chance to prevent a bigger incident, so I will do it. But if I don't know the kind of information before, if I don't have really, at least for the most important parts of their network, servers, DMZ, whatever, it's really providing services that makes the revenue of organization.

The boring thing about this is that it's not technical things about security, right? We are talking business. And of course, many people in the industry are not really interested in that part. And of course, at the same time, the business people are not entirely interested in the security part because they have another focus. The intersection between the two is really important here.

Host: So, speaking of challenges, when it comes to an incident response, so we reached out to Simon Onofre and he has asked, what is the most challenging phase of an incident response?

Giorgio Perticone: Huh, I think we mentioned a few. One thing is, as I said before, get that decision. OK, we need to isolate the network. That's one of the most or the biggest challenges. The other thing is, and I think we mentioned this as well, you know to rush to get back to business state.

Because you said it before, the comparison between having to be very quick and fast in addressing the issue, and at the same time, taking your time to be sure that you really considered every scenario, that you really analyzed all of the data, that you collected all the data, and so on and so forth.

When it comes to the analysis part, you really want to take your time. want to be sure that the moment that you hit that button to go back to business, you're not in the same state you were before the incident because otherwise it's going to just be worse. You already slowed down the business for a certain amount of time to investigate and that time was not spent well because maybe you will have to slow down again later on.

So when you want to go back to to business, need to be, I mean, you can never be 100 % sure most of the time, but you have to be fairly sure that's not going to happen the next day, once again.

And especially if you're, I mean, probably in any case, but especially if you're a consultant and you're speaking with someone else and it's their business, not yours, and having to convince them that, you know, it's maybe better if we wait another four or eight hours to really make sure that we understand everything, that we know if there are any backdoors, if there's any persistence mechanisms, or that we know what was the initial access, that we patched that already before we re-enable the services and everything. That's very difficult. And once again, it's not a technical aspect. It's more on making people understand what are the consequences and what is at stake if you take those kinds of decisions.

Host: So you mentioned about like, let's say clients getting compromised and things like that. You're working with multiple customers, multiple clients, and you see that they are getting compromised.

What's the right way or what's the most humane way of approaching a client who has been compromised? So this is again, question from Simone.

Giorgio Perticone: So that's pretty tricky. It really depends on the initial emotional status of the client, right? It depends on the role of the person you're speaking with, right? Because I don't know, it happens to me that for very small companies and clients, you can end up very quickly talking with the C-level people, let's say. Or you can just talk with the IT people that maybe do not have a lot of security knowledge, but they are still in charge of the infrastructure. Or you could end up with someone who is not technical, maybe someone more involved in the legal or policy sides of things, that still is in charge in case there's an incident. you need to understand what's there for their status. Okay, are they panicking? Are they fairly, let's say, calm and just want to get it right? You want to kind of, yeah, make feel them secure in a scenario where you don't really feel secure,

Make them understand that you're going to do whatever is needed in order to contain the damages because that's most of the time the thing. And that if we do everything right or in the best way possible, we will go back to business very soon. that maybe, that's when you're still involved in the detection phase.

You need also them to understand that you still need to maybe verify and make sure that something is really happening or not. So you need to confirm that there is actually an incident. So you need them to understand this so that they don't panic. They can just wait, provide all of the information needed. And then when you will hear them again, you will decide what's the next steps if you still need to do something or maybe it's actually fine. And I mean,

The only way you can do it is just trying to establish a proper communication process, taking the time to speak with the people. If that's not the right person to speak with, trying to understand whether they can give you the contact of someone else maybe, and then making sure that you will have constant updates regarding what you found out.

And yeah, whenever they have a new question or they want to know something, actually provide that information because even though you could not have any news or any updates, and that's actually pretty common, The thing is that it's only going to be worse if you don't provide any response at all.

So it's better to have, I don't know, it depends on the scenarios, but how early or you know, twice a day or once a day, know, calls or meetings or updates when you just, you actually say, okay, we're in the process of investigating. We don't have any more news at this point, but we're, we're planning to do that. These are the next steps. This is what we worked on so far and what we are going to do next. And we'll have another meeting in, I don't know, one hour or the next day or something. Then actually say, okay, no, don't,

Host: Right. No meetings.

Giorgio Perticone: Haha! No meetings or just do not reply to the emails. That's not a good thing in any way.

Host: Yeah, yeah, especially, especially during an incident. It's it's tricky, right? Like everyone is looking to find out. Have we kind of like found out the root cause? Have we remediated it? Have we contained it? And no news is not good news in this scenario, right? Like because you are already in a panicking state. So it's better to have updates and show what progress you have made. Maybe you have not found the root cause, but at least you have tried out different approaches, a of them have not worked out, what are you planning to, what a playbook you are going to use next, things like that. That sort of calms the stakeholders down a little bit, but no communication is not good in this case for sure.

Giorgio Perticone: Absolutely.

Host: So we spoke about detection and response quite a bit. I want to understand what's the role of automation in this? Like can automation and orchestration be used to improve the efficiency of incident detection and response.

Giorgio Perticone: It definitely does and that's for me when it comes down to the containment phase we were talking about before. If you spent some of your resources on detection technologies, there's a good chance you already have some capabilities to automate part of the containment phase.

If you did it, if you have those capabilities, if you have those the technologies in place, this really, I mean, there are reasons why you are concerned about its usage, but still, I think that most of the organizations out there are not really leveraging that kind of support and help that technologies can provide and automation can provide, right?

You just need to understand, let's say when it's safe, what are those scenarios where it's safer to actually go with automation, right… without thinking too much about it. And maybe there are always going to be some more sensitive scenarios where you just cannot apply automation.

But that's the thing. Everyone has just limited resources, even when you just speak about people in your team. If your team is busy working on stuff that could be automated, they will be busy when something more important will happen.

But if you just can save them part of their time automating the most easy things and the less critical stuff, then you will have their full attention when something very bad happens and something where you cannot apply automation, they can give you 100 % of their focus. So that's my thing. Automation is needed in order to save time. It's probably not going to be there to prevent the most important incidents of your organization ever at. But if you can slow down the burden on your teammates a little bit, there's still a lot of improvement when it comes to the very important things.

Host: So totally, absolutely. Anything that can be automated should be automated so that you are not spending time on things which might take eight hours manually. With automation, maybe it's taking an hour. So you are saving seven hours for analysis, communication. There are so many activities which need to be done during a breach.

But how do you ensure that the automation that you apply doesn't compromise security? Like doesn't open up new attack vectors.

Giorgio Perticone: Well, that's a tricky question, right? Because there's definitely some scenarios where, I mean, you need to enable the technology to be able to contain stuff, right? And most of the time that involves giving accounts and some elements of the technology some high privileges. And of course, you don't secure them, that's another a possible attack vector or you're definitely enlarging your attack surface, right?

But I would say that most of time that's not really a concern or not one of your first concerns when it comes to that because you already possibly have so many solutions and technologies with very high privileges in your environment that you're not really bothered with even if maybe you should, right?

So many admin accounts in there, so many service accounts that you maybe forgot about it. And it's true, you're introducing a new piece of tech, you're introducing even more devices that you have to maintain and update and be sure that they only have the privileges they really need and everything. But the thing is that that's just a drop in an ocean when it comes to the most, the biggest corporate networks out there.

And the value you get out of it, compared to other tools that are not security-related, just removes the security concern in my point of view. Once again, if the automation tools are actually the same detection tools you have, Those tools are there anyway. So your security is compromised already, let's say, the technology is already in place. There are already service accounts with very high privileges just to observe what's happening in your network. Even if you just enable the contain if this happens, that's not really going to be worse from a security perspective. Actually, much better.

Host: Mm-hmm. Mm-hmm. So we spoke about tools. We spoke about automation. But there is always that human element, right? Like there are folks who do the analysis, do the communication, incident commander, and there are stakeholders. So you cannot take out the human element from the system. And this can be applied to any field of business, right? So

What is the role of human analysts? Let's say even like today we have AI and ML. It's helping with a lot of analysis. Do you see in future that we'll need human analysts or automation and AI can sort of completely take care of the incident response?

Giorgio Perticone: I mean, I don't think that's a thing at all. I mean is, work myself, you know, I work for a company which uses AI to help analysis and everything. But, you know, what my company itself says, we're trying to analysts, right? We're trying to get the analyst job easier because otherwise there's just too much data to analyze, you don't have time to go through everything. It's difficult to standardize everything you have in your environment in a single dashboard and everything.

But, technology has always been a tool from the human history. AI is just a new tool, a better tool possibly, right? It is getting us new possibilities. But at the same time, once again, the worst incidents we are engaged for or we are facing are actually manually perpetrated most of the time by humans, right? From the other side.

And the same time from our side, we need humans to look at the data and understand what's happening. We are applying automation. We should apply automation, but that's when a human decides that it's, you know, in that specific scenario, it's safe to apply automation, right? And then it's going to be a human who decides, okay, let's go back to the previous status because it was okay or it was not okay.

The decision that was there. And maybe I need to change my automation rules after I investigated this specific case. Right? So the thing is that technology is there, no matter what kind of technology today is AI, tomorrow we will have something else. It's just No, it's supposed to get our job easier, but it's not really replacing us at any point in my point of view.

Host: So one last question that comes to my mind is like we started with you, like you have dogs, you have a dog and you play drums. So that's how you manage your stress. When you are in incident detection and response, you are always in the zone of out maneuvering the attackers and things like that. So there is a lot of stress. So how do you… I mean, I understand how do you manage your burnout and stress.

What advice would you give others to manage their own burnout and stress?

Giorgio Perticone: So I know it seems like I always answer with the same stuff, right? I took all the questions. My point is that if you, if you just not wait for the next incident and panic when it happens and you actually test your plan, you simulate things when it's safe to test your plan. And it's means that your people in your team actually know what's going to happen, know what they are going to do or need to do, they will feel safer.

The level of stress will be lower. There will be some, of course, but they will face the incident in a better way. So that's the thing. The key component here is to be prepared. The more prepared you are, the more incidents you've faced before, either real incidents or simulated incidents the better you will face the level of stress during an incident.

And the other thing, and this is possibly, I guess, a suggestion for managers out there, you cannot pretend that someone in your team can work, I don't know, 48 hours straight to an incident, because that's what's gonna happen. I don't think everyone understands what an incidental responder is gonna do. It's gonna work on the incident, until it's going to be fixed.

That is not going to take most of the time, just a hours. A real incident at a level where you're really being compromised, it's going to take weeks.

That means a lot of non-stop hours to work on understanding what's happening and at the same time trying to improve your plan and trying to get some data you didn't have your hands on. So that's a lot of work in a very small amount of time. So the thing is, you must be prepared to that as well. You must have some kind of 24x7 coverage. So some people should get some rest and some other people should take from that point.

Either you are able to create the capability internally, or you figure it out with some external partners. There's no right way to do it. It depends only on what you can do it at that moment and what you cannot.

And you just need to understand as a manager that some very stressed and some very tired analyst is not going to do a great job. After eight hours of work, you don't really understand what you see in front of you. So it's not in your interest to really squeeze everything out of one single analyst.

So if you're working on that budget for the next year, maybe it's not only regarding technologies to buy, maybe it's a lot on hiring more people, train your people, and make sure that in the worst case scenario, you can split the burden of the incident between multiple people at the same time, someone can take some rest while the other one is working.

Host: Yeah, so finding that balance between overwork and stress so that you are you can think properly, right? If you have been working for 48 hours straight, as you said, like after eight hours, you are just looking at some maybe you're looking at logs, but even though you are looking at the core reason of a breach, you might miss it, right? Because you your brain has been looking at logs throughout the day or looking at a lot of things throughout the day. So makes sense.

So another question that Simone has asked is, what advice would you give to someone who wants to pursue a career in student response? Since you have been in this field for some time, do you have any advice?

Giorgio Perticone: My usual advice to this kind of question is not very common or usually appreciated. And the thing is that don't look at what the other people did before you.

Because the thing is, what worked for me, it may not work for you. So I get a lot of questions regarding the university thing. Should you go to university? Should you not? I mean, I didn't, but that doesn't mean you shouldn't.

I had a specific scenario. It was maybe a better idea to actually go to university if I think about it right now. But the point is that what can you do right now? Are you able to take university? Can you spend that amount of time without working and of course getting any money? Is there anyone that is paying that for you? Do you live in a country where that costs a lot or is actually pretty cheap? Can you maybe afford to work while you study? Because I think that's a very great point.

The best professional is going to start in the best way possible if they did some work already. So I always encourage someone that is studying to also find some kind of part-time job or something, any time of work really. It's better if it's technology-related, of course, because you start to make your hands dirty with some stuff. But at the same time, any kind of work that we said before, there's a lot of communication involved in this kind of job. So if you never had to professionally communicate, with clients, colleagues, superiors and stuff, that's going to be a big challenge for you. So if you had the kind of experience already, even if you didn't work really in a computer-related environment, there's still a lot of improvement.

And well, then, of course, I mean, it's my opinion again, but I always feel like security is a specialization, right? And we are talking about incident response and detection right now. So that's even a specialization of this specialization, if you think about it. And I always encourage everyone to at least consider working a general IT job before moving to security. So either you do...

Host: Yes, yes. So getting your hands dirty before you get into security, you understand how things work in a way. Sorry.

Giorgio Perticone: Yeah. mean, either coding, either you start as a developer or a sysadmin or now we have DevOps and stuff. But the thing is that you're going to work with a lot of different technologies. You're going to need to understand a lot of different technologies and you probably will never know all of them. But if you start in a position where you already know a lot about some technologies because you developed those technologies, or you had to maintain the update and design networks regarding those technologies. That's already a big and important thing.

And one last thing is that this is not only my opinion honestly. When you spend 10 years in the industry and you had different jobs and you did quite some interviews with different companies, I get the feedback.

I get the feedback that hiring managers appreciate when you did a non-security related job before and you had that kind of different perspective before. So I always leverage the fact that I was a sys admin before being a security analyst. I mean, for me, it worked. I'm not sure it's going to work for you as well, but if you have the chance and if that works for you, you Give it a chance.

Host: Yeah, that's a very good response. There is no one-size-fits-all solution, right? Whether it's security, engineering, whatever it is, it always depends on your scenarios, your environment. There are many factors which come into play.

One last question that we got from Mike Privette is, how not to build a detection engineering capability at a company? So you have seen detection engineering capabilities built at companies.

Giorgio Perticone: Okay, have another uncommon answer to this. I hope it's fine for everyone. I don't think every company should have it at all. I mean, we're talking about detection engineering. We didn't really speak about it during this session. But the thing is that we're talking about having a team of people, right? that only works about engineering your detection capability.

So defining rules, creating technology internally that is able to identify incidents and stuff. And I think that's something that only a very, very, very mature company from the security perspective will do. So you start doing that, in my point of view, when you think that the detection capability that you can get out there with just money is not enough because you can do better. If you can do better than what you can buy out there, then maybe, okay,

Maybe you should start creating that kind of capability inside, but in all of the other scenarios, in my point of view, the time and resources you spend in trying to create your own detection rules versus the amount of time that is needed to just deploy something ready from a third party. I mean, it's, you know, it's, it's different to compare, honestly. It needs a lot of work and there are companies just doing that, you know, with the hundreds of people, 24x7. How, I mean, how do you think you can be better? I mean, if you did that,

Maybe if you were from a provider perspective and then you move to another company and you have already that experience and knowledge, you can do it yourself better because you already studied and did it for 10 years before. Definitely go for it. You know better.

But otherwise, as a CISO or security manager, you decide that you need to hire five people at least to work on… Yeah, decision engineering, I don't think that that's really worth it. You should manage your security budget in another way and think about that later on if you really need it.

Host: Okay, that makes sense. With that, we come to the end of the podcast.

But before we end, one last question that I have is, any recommendations that you have, like reading recommendation, could be blog, it could be a book, or it could be a podcast, or any video or something like that for our audience.

Giorgio Perticone: So there's a thing, I'm more into books, but I have a very bad memory, so I always forget the names of the books themselves, but I have my to-read list in front of me. So, well, if that's okay, I will quickly suggest three books that I really liked. The first of all is The Cuckoo's Egg from Cliff Stoll. It's a very… I guess famous one and old one as well, but it's still relevant nowadays. And if you think that it was written by someone who was not really into IT or security, that's mind blowing for me. And by the way, all of these books are basically real world stories about cybersecurity. So this is the first. The Cookies from Cliff's Toll.

Host: Wow. Okay.

Giorgio Perticone: The second will be Spam Nation from Brian Krebs. That's a very great writer, by the way. He also has a blog. You should go and look at it. Spam Nation is also talking about cybercrime in I think, 90s when it was mostly focused on the spam business. And yeah, of course, Sandworm by Andy Griebmerk that is also more into APTs more than cyber crimes. know, a state sponsored groups. Yeah, you know, I actually have a big list, but if I have to think about three of them, those will be the ones.

Host: OK, awesome. Yeah, thank you for sharing your recommendation. So when we publish the episode, we'll add the book details so that our audience can get benefit out of it. With that, thank you so much, Giorgio, for joining, sharing your insights. It was a lovely learning experience for me as well, like the incident commander position that should be there, or the people aspect, or the tools aspect, and things like that. So yeah, thank you so much for joining.

Giorgio Perticone: Thanks again for having me. It doesn't take so much to have this kind of conversation. And I do think they make some difference in the community for the people who are watching. keep up the good work. And thank you for that.

Host: Thank you so much. And to our audience, thank you so much for watching. See you in the next episode.