Apr 3, 2024

Beyond the Basics: Understanding Threat Hunting and Security Research with Josh Pyorre

TLDR;

Threat researchers use threat hunting to learn about trends, and correlations, to narrow focus of the research. And they use this information to watch for other threats and also to help bring awareness in organizations.
Threat research needs creative and out-of-the-box thinking. By following a checklist, threat researchers often do not get, often do not find out novel or unique attacks. So it doesn't help. It doesn't aid in threat hunting process.
GenAI platforms are a boon for researchers. They can quickly get started, learn about new attack vectors. At the same time, guardrails should be put in place to ensure that data is not leaked or employees should be trained to learn about new GenAI-based or inspired attacks.

Transcript

Host: Hi, everyone. This is Purusottam, and thanks for tuning in to ScaletoZero podcast. Today's episode is with Josh Pyorre. Josh is a security research engineering technical leader with CISCO Talos. Been in security since 2000, working as a researcher and an analyst at organizations such as CISCO, NASA, Mandiant, and various nonprofits as well, and was also a principal product manager for advanced threat protection at Cscaler. He has presented at many conferences like DEF CON, RSA, B-Sites, and many more. He's also the host and producer of the security podcast, Fruit Access. And you can find more information about him at about his personal, from his personal website, which is pyosec.com. Josh, welcome to the show. Thank you so much for coming. For our audience, do you want to briefly share about your journey?

Josh Pyorre: Thank you for having me. Sure. Yeah. Like you mentioned, I've been in security for a while. I mean, I started out the very beginning doing tech support, field service engineer, help desk things, and then sys admin. And then I came over with a.com bust happened, but that happened. I was out of work for a little while, and then I got a job as a technical director for a nonprofit.

So I did that for quite a while for a homeless shelter in San Francisco. And then kind of that moved me towards security as it was like, I should learn Python. I should learn all these things. And then I answered an ad on Craigslist of all places for a job. Then ended up being at NASA, which was pretty cool. So I got to be one of the, one of the first two analysts in their new sock in Ames research center, where I learned a bunch of stuff, looking at a firehose of IDS data and then from then on, I just. kept moving forward with working in incident response at Mandiant and working in product management briefly, working with Cisco and working for myself too as well in some cases there.

And then I love to teach and speak at conferences, so I'm always researching something that I can speak on.

Host: Lovely. That's quite a unique journey. And I've never seen or heard anyone getting a security job from Craigslist. That's very unique.

Josh Pyorre: I'm going to go ahead and close the video. It didn't actually say NASA. It just looked, I was like, oh, this looks really interesting. So I applied and they contacted me and I was like, oh NASA. It wasn't a civil servant job. It was a contractor job as most government jobs are, but it was still pretty cool.

Host: Mm-hmm. Yeah. So you mentioned like you have done many things throughout your career and you have been doing. So what does a day in your life look like today?

Josh Pyorre: Oh wow, okay. Hectic. I usually like, I try to have a good work-life balance. I mean, I think I do actually. I have family, so I spend time with them in the morning. And then it's changed with the nature of my jobs. But currently I'm doing a lot of more focus on data science, security, data science to support security things in my day job. And then, of course, if I'm researching something.

I'm often doing that in the middle of work because it's tied to work or I'm doing it outside of work, or like after hours. So maybe when it works over at five or 6 p.m., I have family time and then they all go to bed and I stay up till two in the morning working on something, depending on what I'm doing.

And I try to throw in some other things too, like exercise and I have a sauna, I go in the sauna and things like that. It seems that the days have a regular cadence to them, but to me, they also seem a little bit chaotic. Yeah.

Host: Sounds like everyone's day nowadays, like with multiple focus areas.

So, and one of the things you mentioned multiple times, which is around research, right? And that's the topic for today, like threat hunting, security research, and security at startups.

So let's dive in. So we hear about cyber attacks or cyber threats on a daily basis. And these attacks span from simpler ones like phishing attacks all the way to sophisticated like ransomware or crypto mining attacks.

So in order to prevent these organizations generally spend time on building threat research team or security research team. So to set the stage for our audience, can you provide a brief overview

What threat research or security research or threat hunting is?

Josh Pyorre: Yeah, sure. So I feel like one part of that question that organizations set up threat hunting or security research teams, hopefully they do, sometimes actually there's more of an industry now of you can hire a company. So like most companies have a, they are your threat research team.

But either way, if they're thinking ahead, they're setting up their threat hunting and security research. Usually threat hunting is the first thing they would set up, I would think, because they wanna… look for future threats to their organization that may be coming down.

When I was at NASA, for example, I would look at all the phishing emails that came in for the entire agency and try to find trends based on that. So that could be counted as threat hunting. I've in many cases just been analyzing malware, looking for correlations between different malware variants and trying to find just relationships. And then when you combine that with looking at what's happening in the world… So you can kind of, he's trying to make predictions, like predict the future basically. That's what threat hunting generally entails.

And ultimately a product that you are releasing as a threat hunter is IOCs or things you can block or things you can watch or sinkhole or whatever. And then security research, I feel is a broader term. It's kind of, maybe it's harder to define, but I mean, it's research. You know, it's, I didn't go to… I did eight years in community college, but I still have a PhD in community college, but I don't have any degree of any kind.

So I didn't do full on research, but I do it now with my work and my own passions and interests and doing a security research, you're exploring things that are hypothetical or outside the bounds of normal, whatever, and threat hunting can fuel that, I think. That was a kind of long answer, but.

Host: Okay. No, no, it makes sense. Like it makes sense that you narrowed it down to threat hunting and that's where organizations will start and then slowly grow. And one of the things which I did not know is that even you can hire or work with external agencies for your own threat hunting or threat research as well. A follow up question to

Josh Pyorre: That's the easiest way to get the pen up. Oh yeah, go ahead.

Host: Hmm, interesting. A follow-up question to it, why is it important for organizations in the context of cyber security or cloud security?

Josh Pyorre: I think it's important because you need to know what you're at risk for and what to watch out for. If you are an organization and you aren't doing some kind of threat hunting focused on what your business is focused on, you might be just reacting to everything that you see around you. You might be assuming that the newest whatever variant of malware is going to affect you, or maybe it's not even close to your industry or you have to worry about it. So threat hunting narrows your focus on what you have to worry about.

And It also keeps you aware of just, you know, what's out there against you basically And also predicts the future. Thank you. You're not um You're not chasing after things. You're you're looking forward and looking toward things that might be coming down

Host: Okay, so it's understanding the current challenges and also looking at sort of predicting what might happen to the organization in future. So in cybersecurity, there are several approaches, right, when it comes to threat research, like proactive or reactive. How does these approaches differ from each other?

Josh Pyorre: Oh yeah, so I guess I started to touch on that a little bit. But a lot of organizations I've worked for in the past, not when they weren't security organizations, were primarily reactive. And I would say a lot of the industry is reactive unless they're going out and starting their own threat-hunting teams and researcher teams or hiring someone.

But reactive, of course, you're putting out fires all the time and it's very stressful. And you're just… you're acting like nothing's gonna happen to you and then all of a sudden you get ransomware in your system and then you're fighting that and then you move forward, you remediate that and then the next thing comes down and of course proactive.

We all wanna be proactive with everything. Just like you're driving down the road, you wanna pay attention to what's gonna come towards you. You don't wanna just react to people hitting you in the car, you know.

I feel like it's common sense that you would be proactive with anything you do, especially threat research, threat hunting to keep your business and organization and individual self safe, basically.

Host: Yeah, that's a very good point. So now let's say as an organization, there are benefits to both, right? And as you said, often organizations spend more time on the reactive method because you see the fire and you start fixing it. Now,

What challenges have you seen with organizations adopting either strategies?

And what benefits have you seen from that?

Josh Pyorre: Well, I've typically seen smaller organizations, they do they reactive one, unfortunately, with a little bit of proactive, like usually in terms of the most common things of like securing their firewall and securing their systems a little bit.

Yeah, but the larger organizations will, of course, be a little more proactive, but we all, I think in the security industry, it's commonly known that security and many times comes last in the budget, unfortunately! and it doesn't appear.

I mean, you look at the news and places are being ransomed, places are having the data stolen, and there are things that they could have done way in advance. And so, yeah.

Host: So that's why there is clear differentiation. Investing in proactive would also help so that you don't get into the fire in a way. Of course, you need to have reactive approaches as well. But yeah, proactive does help.

Josh Pyorre: Yeah. You do. Yeah, it's an investment. It's like investing in insurance. But it's more important, I'd say, because it's gonna happen. It's gonna happen if, yeah, you leave your systems exposed, they're going to be found.

Host: Mm-hmm. Yeah. So one of the things you mentioned earlier was around that it's a vast area, right? Like security research or threat research. And compared to traditional security research, it needs a mindset shift. So how do you use creativity in the cybersecurity threat research area?

Josh Pyorre: Oh yeah, I think about this all the time because I come from a creative background. I'm an artist, I make music. I mean, I'm more than just a security researcher and practitioner. But I feel like the job itself and the work is creative.

And I feel like any work in the world is creative. You could do anything and find creativity in it. There are rules, there are certain rules like they have to follow, but inside those rules, there's… ways you can move around and you can, it's kind of hard to explain, is that you can be creative because it's a mindset that I have.

But when I'm faced with a problem, I have so many options to explore and they're outside the usual boxes that I work with. The boxes are great and they're things that other people created once at one point that were creative and outside the box, but there's always room to explore even more.

So I just find creativity, like a creative approach. artistic approach, a humanistic approach is to doing security is important. There's still the technical things, you know, you set your whatever correctly and your rules correctly and all this stuff and set up your IDS's correctly and all those things, but there are, when you're researching something, there's room for exploration.

Host: Right. That makes a lot of sense that you are sort of combining your creative knowledge and then using that for such a technical topic as well. Right. So can you share maybe a specific instance where you have thought differently or you have thought in a creative way and that has helped you identify a threat or a fix a threat.

Josh Pyorre: That's tough, because I've always approached things this way. So I would say almost everything I do is approached via creative processes. I mean, it's very rare that I'm doing basic data entry kind of things. And even then, I'll write a Python script to do it faster for me.

But I can think in the industry, there are some methods where creative play or creative practice has… uh, been used in security, I think probably in the, in terms of like attribution identification, when you're looking at, um, something like an attack chain of, of a phishing email comes in with some attachment or whatever, it leads to a compromise, does a whole attack chain and they're putting it pretty standard across the board. Like they have to do a certain couple of things.

And this is shown in the MITRE ATT&CK framework. Um, and the MITRE ATT&CK framework has an attempt to put, um, like definitions on each of those parts of the attack chain.

But there's also behaviors, and it also follows behaviors. But trying to find when you're researching different malware variants and using the matter attack rule set and other things related to that to find relationships between malware variants and their attack chain activity. And then attributing it to some threat actor or group as is a creative process.

I mean, when you look at the code and you're finding the variables that match with the variables of the other code and some other malware variant, or just looking at the hosting infrastructure or whatever, you can kind of explore basically. It's not so follow the rules. And you end up with a result of like, hey, this group in whatever country did this, which is interesting.

Host: Yeah, so you mentioned quite a few areas, right? Like, let's say, you just gave an example. Let's say if we get a phishing email,

what is it exploiting?

What is it attaching?

Look at the MITRE framework to understand what areas is it impacting?

You mentioned about writing Python scripts and stuff like that. So you touched on many tools which threat researchers use.

So now one question that comes to mind is, often new technology brings new tools and ideas for threat research. And along with that comes challenges. And recently there is an introduction of Gen.AI. And a lot of folks are adopting it. And it's helping secure, like there are two angles to it, right? One is the researchers can use generative AI platforms to learn from it. And the other area is that generative AI integration can get attacked as well. Right. So

How do you envision generative AI playing a role in security research as there is a rise in the adoption of gene AI?

Josh Pyorre: Oh yeah, I, well, I love it. I'm hoping that it doesn't go, um, doesn't, I don't know where it's going to go. I just, but right now, like it's, it's helped me quite a bit. I deal with a lot of different technologies and things are changing and, um, quite often in the different technologies I work with, like I use some data science things and I use Python quite a bit. In many cases, I mean, I'm not, I'm not, I don't know all of Python through my memory, so it's nice to be able to say. Write me a script that will take care of this little thing. And I'm always trying to find ways to increase the work I can do so I can focus on what's more important to me.

So I personally use it. In my last recent talk, I just gave a presentation. I did a lot of using like a chat GPT and some other language models to help me write some of the code, some of the more complex code. So I was able to do it in two months, whereas it would have taken me six months instead.

So that's really helpful, basically. And I think it's gonna improve things more. I don't think it's gonna take our jobs. I think it's gonna make it so that we can have more opportunities to create more detection mechanisms and be better security practitioners.

Host: Yeah. I'm glad you said that, hey, we are not going to lose our jobs because quite a few folks think that, hey, after the charge GPT or the GenAI systems, are we going to lose our jobs? So that is one area, right? You highlighted that, hey, you are getting benefit out of it. Now,

How do you see this becoming a challenge for researchers?

Josh Pyorre: Yeah, that's another thing. I mean, we've already seen instances where phishing, the text of phishing emails are better written, better well written.

Previously you can always be like, oh, this is all quickly phishing or spam or whatever. Now you can, if you don't speak English, you can paste whatever you want in there and it'll rewrite it for you. And then it'll rewrite it again. You can say, rewrite it as this kind of person and it'll rewrite it like that person. Or you can make your own AI agent that is a certain kind of personality. And it's just gonna get harder for us.

Like I've always prided myself whenever I've gotten the phishing tests or actual phishing emails. I'm like, oh, I can always find it. And I look at the headers, I look at all sorts of things. And I have an intuition for it, but there are recently in the last year or so, when really the stuff has come the last year, there's been a couple that are almost got me. And that's just one example, but you could...

They have guardrails in place. You know, you can say to chat GPT, I'm using them as examples, tons of them, but you could say, I want, write me some thing to exploit this vulnerability. And it'll say, I can't do that, but you can get around that really easily with different kinds of prompts.

And you can also, I even run one on my machine. I run a large language model on my machine and I can ask anything I want. So there's no real guardrail. So I feel like the sky's the limit. The threat actors are doing the same thing that we're doing and they're as creative, they're more creative, I think, than we are, because we're bound by rules in many cases. They'll do whatever they can to get past the whatever. Little bit scary.

Host: And you were spot on, like I was also, I think I asked a simple question, like what are all the attack vectors in AWS? And I got a response that hey, I can as a labs language model, I cannot give you this information and all of that.

But if you slightly tweak the question, it starts giving you different information, and different attack vectors also. So yeah, even though there are guardrails, there are ways to work around it.

And another thing, the key thing that you highlighted is if you're running the large language model on your own machine, then there are no guardrails. You can ask any question. You can get any answer. Yeah, so that sounds.

Josh Pyorre: Oh, yeah. And I can additionally tie in Python or, yeah, I can tie in my programming to whatever it is on the machine and I can have it automatically rewriting my code as I'm working on it or rewriting whatever it is I'm doing, you know, or recreating whatever I'm doing. And then you can go even further. I don't wanna take this out of tangent, but you can go further with like, that's just coding. I mean, there's of course, deep fake stuff and changing voices and impersonating people. And impersonating, it's already been used with phone calls to people to try to get money from people saying, I'm in trouble, you know, and it's there, they hear their son or daughter on the phone and it's actually just a, you know, AI kind of thing. So.

Host: Yeah, no, that's the sort of with this new technology, there is a lot of benefit we can gain. And at the same time, there are these risk vectors, right, which are coming into us as well. And folks like you are spending a lot of time doing research, how we can get around it. So there is that balance has to be kept in check, right? Otherwise, we'll be in trouble later.

Josh Pyorre: Mm-hmm. Yeah. Mm-hmm. Yeah. I don't know, we'll see where it goes. Yeah. I feel like it's good. It's hard to keep a balance. Yeah.

Host: So, yeah, true. I know that you present at many conferences and your conference talks are often you show unconventional approaches to security. How do you balance presenting complex security topics and make them accessible to diverse audience?

Josh Pyorre: Well, I have, I'm fortunate because I did all that work in my early years as help desk and tech support. And so I had to guide any kind of user or a person through complex topics. And I had to be nice about it and not talk down to them.

You know, so I, and then you, as you, as you get further along in your career, you learn to speak professionally to people and, and to, to work in a way that, that is like you can convey your information.

So it's a learned thing, but it's easy. It's so easy in my line of work, especially currently what I'm doing with the data science stuff. I'm in a silo. I'm like in my computer, like head down in my computer, trying to work through some problem. And then, but I like to get out and talk to people so I can distill that information and explain it to a broad audience. So anyway, I'll like...

I'll take a complex topic and I will break it down and like, what's the problem I'm trying to solve and then give some answers of what I'm working towards in a simple way, but I don't wanna talk in a way that makes the audience feel like I'm belittling them, of course. I mean, I'm always open for questions, you know?

So sometimes I do have to show in my presentations code, videos of code running or actually the screens of code because there's no other way to explain it. But I'll walk through it and I try to make it as friendly and easy as possible while also making it easier for the people who is super technical who get it right away to not be bored. So it's a weird balancing act, very tricky.

Host: Yeah, yeah, true. One last question that I wanted to ask, and often this happens with enterprises, they have the funding, the resources to invest in security, threat research, all of these areas.

When it comes to small businesses or even nonprofits, you have worked at nonprofits as well. Why do you think it's crucial that the entire ecosystem, security ecosystem should also prioritize security of the smaller businesses or nonprofits.

Josh Pyorre: Oh yeah, so I think about this a lot, having worked at small businesses and then also working in environments where companies hire the company I work for to remediate something or fix something. As you look at big corporations, big companies can afford the security services that are provided.

And I've worked at a couple of security places or a few of them that have consistently raised their prices and removed lower tier offerings that they used to have when they first started out and but they were them because they wanted better quality clients, you know clients that will just pay a huge amount and leave them alone for the most part, but the So it kind of out prices these smaller businesses and when you look at like most of the Most of the compromises that have happened like I can't I think it was Merrick that this happened that

They had an accounting firm, MeEdoc was compromised and an update was Trojan. And that's what caused their whole systems to shut down and shutting down all of global shipping for a while costing billions of dollars. And there's a whole story about it. But it was a small company that was compromised.

And in many cases, I've seen law firms compromised. These are just maybe just a law firm of like four lawyers, you know, because they don't have time or the money to the security or they don't even think about it. So I just, yeah, there's a disconnect, like removing those services for those that are affordable, making it so that you can't get in if you don't have a certain amount of money. And then of course, those smaller companies are the targets because they have access to the bigger companies.

Host: Right. So in that case, what role can larger organizations or security vendors play in supporting? Like, how do you see or what would you like to see larger organizations or security vendors to do for these this thing?

Josh Pyorre: Yeah. I think that what they could do is provide cheaper offerings. Just, I mean, I know it's, it costs them money. It costs everyone money to create something like that, but to create like either a donation program or that would be the best for nonprofits.

I'm sure there's some way to make that work in an organization or to keep lower tier offerings, like really lower tier.

When I worked at a nonprofit I was paid very, I thought I was paid well, I was in my 20s, so I was paid 50,000 a year. There's not much, that's the most they can afford. I mean, I actually had to get that out of that and that was really hard.

That was a technical director. So they can't afford anything. All the computers we got were donated. Everything was donated pretty much or like it really subsidized. So security wasn't on the table, it wasn't an option. And whatever I knew was what I could give them basically. And at the time I didn't know as much as I wanted to.

So, yeah, if large companies could provide an easy pathway for nonprofit small businesses to have this, the same enterprise security, until maybe they reach a certain level and they can start paying more or something. That's my ideal. And if I had a company, that's what I would do.

Host: Makes sense. And that is one of the things I really like about Cloudflare. I'm not trying to promote any company or anything like that. We use Cloudflare and they have very like basic pricing which even covers many security areas like firewalls and DDoS stuff like that, which are equally important for smaller companies, right? And they offer it at a very cheap price.

So yeah, it makes a lot of sense when you say maybe offer that lower price tier for smaller companies or startups as long as you can so that it helps the overall ecosystem.

Josh Pyorre: Yeah, and we need more like Cloudflare. Like Cloudflare, that's great that it's so cheap. I've seen it used in smaller organizations. But then you're missing on the really fancy things like incident response. How do you hire someone to come in when you've had an incident? Or like, import protection. You can use antivirus, but antivirus, as we all know, it's not the same as having proper import protection from a company with threat researchers and stuff. So, yeah.

Host: Yeah. True, true, absolutely. And yeah, that's a great way to end the security question section.

Now let's go to the next section, which is the rating security practices.

Rating Security Practices

So here I'll share one security practice. And you need to rate them between 1 to 5, and 1 being the worst and 5 being the best. And you can add context, like why you are giving a particular rating.

So let's start with the first one. Provide training and awareness programs to employees to help them identify and respond to potential security threats.

Josh Pyorre: I think this is a five. Like I think it's extremely important. Your employees and users of your systems are gonna be the easiest way in, besides going for a small company that you contract out to. And their users too.

Host: Right, yeah. Yeah, let's go to the next one. Development regularly tests an incident response plan to help quickly detect, respond to and recover from security incidents.

Josh Pyorre: I also think this is a five. It's the other side of that, or Threat hunting, but it's a five because if you've made backups and whatever of things and you've got ransoms, you need to be able to know that it's gonna work when you're trying to restore it. That's the most basic part. And then an instant response and other items in this category, they're important. You need to be able to quickly recover basically.

Host: Yeah, absolutely. The last security rating question is, DevOps practices are needed to move fast and deploy code to production. Security practices are not the most important right now.

Josh Pyorre: I'm going to say one. I know it doesn't change a lot because I've even written things very fast. I need to get it done. And I'm like, I'll deal with security later, but I've learned my mistakes. And most companies though, they will do this though, right? Build their thing and you want to build fast and you build the worst person, worst version you can. And then increase, improve upon it. And other things come later. The security should be always thought of as, as important because later when you add security, you bolt it on your creating spaghetti code, you're creating issues, and when it's in production, you may not want to go and mess with it. So it's gotta be thought throughout the whole process of a product.

Host: That makes a lot of sense. So last question before we end the podcast is any reading recommendation that you have for our audience. It can be a blog or a book or a podcast or anything that you would want to recommend.

Josh Pyorre: Wow. Oh my gosh. Interesting. You know, actually, this is a book that I'm reading currently and it's not security related.

I actually find I get a lot of value from non-security books. I have a ton of books and I listen to a lot of podcasts. I'm going to give you two recommendations. One's a security thing and I'm sure most people have heard of Darknet Diaries podcast. That's a great podcast with stories behind things that have happened or interviews with people.

And then the book that I'm reading, it's called The Changing New World Order by Ray Dalio. It's more on economics and the economy of the world and like how things, the global currency of the world, you know, it was some British and then, oh, it was Dutch, then British and then the United States is the leader and then now it's moving towards China. This gives you an indication of reading things like this. This book specifically is giving me a feel of like where society is going. And that also changes where security is going, so what to watch out for.

Host: Yeah. Thank you so much for sharing these two. What we'll do is when we publish the podcast, we'll also tag these. And I have read principles from Ray Dalio. That's also a great book. Yeah. So when you said changing the new world, I was like, yeah, I have heard about it, but I have not read. I have bought and it's in my bookshelf. I need to read. But yeah, that's a good recommendation for sure. And yeah, that's a great way to end the episode Great.

Thank you so much, Josh, for joining us and sharing your learning with us.

Josh Pyorre: Thank you for having me. It was a pleasure.

Host: Absolutely. And to our audience, thank you for watching and see you in the next episode. Thank you.