Understanding Jupyter Notebooks with Ashwin Patil
TLDR;
- When it comes to threat hunting exercises, instead of focusing on the top fives of the world, focus on areas which are applicable and can be exploited in your architecture.
- As part of your incident response plan, conduct frequent fire drill or tabletop exercises to evaluate preparedness.
- Learning from a security incident is one of the best ways to improve threat hunting process.
Transcript
Host: Hi everyone, thanks for tuning into another episode of Scale to Zero. I’m Purusottam Cofounder and CTO of Cloudanix. For today’s episode we have Ashwin Patil with us. Ashwin currently works as a Senior Security Researcher for Microsoft Security Research team and has been working in security monitoring and incident response for over a decade. In his current role, he primarily works on threat hunting detection research using KQL, which is the custodial query language for Microsoft Sentinel, and develops Jupyter notebooks to do threat hunting investigation across both cloud and on-premise data sources. He has previously spoken at conferences such as Jupyter Thon Sans, Purple Team Summit, Blue Team Village, on threat hunting on cloud data sets and using Jupyter notebooks. So Ashwin, it’s wonderful to have you in the show for our audience. Do you want to briefly share about your journey?
Ashwin: Yeah, sure. First of all, thank you. Thank you for giving the opportunity and inviting me for this podcast. So yeah, I think you’ve covered most of the introduction. I think I started my career as MSSP, as a Sock analyst in India and then worked for Cisco primarily for three, four years. And then since now over nine years I’ve been working with Microsoft. Most of the roles have been pretty much on the same parts like Sock SIM and Security Monitoring, incident Response or those.
In my current role I’m primarily focused on cloud security due to the product that I’m currently working on.
Host: Yeah, it’s a pleasure to have you in the show because there are certain areas which we have not covered in our podcast, like usage of Jupyter notebook and how you are using it for threat hunting. I’m curious, I want to learn more about it. So the way we do the show is we have two sections.
The first section focuses on the security question and the second section is mostly around rapid fire. So let’s start with the security questions. Right?
So when it comes to threat hunting, threat detection, there are many areas like bug bounty, Pen, test, vulnerability research, threat research. So when it comes to cloud, what area do you cover? Like does your work cover?
Ashwin: Yeah, right now I’m part of the Microsoft Sector Research, which is like is a broader umbrella of various research teams working in relation to security under that I’m part of XDR and Same Team which basically deals with the product like N 365 and Microsoft Sentinel.
So I’m primarily focused on the threat research part rather than the other components like Pentest, bug bounty, and other things. The other things does come into the picture like whenever there is an incident related to let’s say, bug bounty or reported vulnerability by the researcher or that sort of thing.
But the primary focus is like threat research, doing research related to various cloud attack paths, writing detections on those cloud data sources, and also like working with the product groups if they are developing certain features which require the analyst's or like researcher's perspective. So, yeah.
Host: Okay, so I have a follow-up question on this. How does it differ from the way traditional red teaming or blue teaming activities are performed?
Ashwin: Yeah, so I in I would say like a red team, the blue team is kind of like one avenue of what we do as part of the threat research. Again, the threat research can be fueled by various sources. So the Pen test threat team or can be like one source of information that can basically spin up the ideas for the telemetry gaps detections that we need to develop. But apart from that, I think there are various other sources where we get kind of a requirement or ideas about developing new detections such as let’s say a new telemetry is available by a cloud vendor or new attack path is published by a researcher. How do we detect those things and then basically doing research, back end research, understanding what telemetry is available to detect those attacks and then how do we write detections from our platform which is Microsoft Sentinel and then query language that we use is basically Kosto query language. So most of our work have been published as open source GitHub that’s available to everyone that they can check.
So that’s good in a way that people can refer those same logic and then probably also write similar kind of detections on other platforms.
Host: Yeah, so the GitHub repo that you highlighted, I have used that in the past as well for our audience. What we will do is we’ll tag that GitHub repo when we publish the video so that they can also get benefit out of the research that you guys are doing, the new attack paths that you have studied, and your findings and stuff like that.
Ashwin: Yeah, one thing that I would probably just highlight is that GitHub kind of allows you to get inputs from the community members as well. So even though probably most of the detections were authored by Microsoft, there are also large community contributions that we have where various researchers or various people have also submitted detections that have now become a part of the product as well. So yeah, definitely good to know about what is out there.
Host: Yeah, absolutely makes sense. One of the things that you highlighted is you look at the attack paths as well, right. And you try to classify the threads and stuff like that and there are many frameworks to do it right. And one of the most popular is the MITRE ATT&CK framework.
How should organizations leverage both the framework, let’s say the MITRE ATT&CK framework and also the research, the GitHub repo that you have highlighted. Right. How should organizations leverage that to improve their own security?
Ashwin: Yeah, if you are not aware of the Mitra Tech framework, I think just like probably a simple introduction about is it’s basically a framework or like collection of real world adversaries and then techniques that they use. Right. So it is actually a need for the security industry to have such kind of framework that’s been adopted by vendor as well as the researchers as well, so that we know where we are in terms of detection coverage and other things.
Again, there are various ways that you can go after, like how do we operationalize that MITRE tech framework? And again, there are various guidelines that I would probably recommend reading from Mitratack their own blogs. They have basically outlined how does our organization operationalize the MITRE data framework. And then also there are various industry experts or researchers or the members who have basically operationalized that framework in their organization. Right. So there’s a lot of reading material out there already, but what I would say shortly is basically the framework is a kind of pointer if you don’t have anything to get started. So it’s a good pointer to get started and then looking at where we should go next and where we should fill the gaps and other things.
Again, there’s like a holistic approach that you need to do basically understanding also like make an assumption that framework is not a complete perspective of all the techniques that are available out there. Right. You really need to assess as an organization like which actors are likely targeted, which actors are likely to target your organizations or your business sector, and then which techniques are basically used most often by those actors. Right. And then understanding again what data sources that you have available and then maybe create kind of a high level plan on how do we go after this program as a whole. Right, okay. And then again, this is again just a pointer to kind of understand where we stand in terms of detection coverage. Again, there can be various ways that a certain technique can be detected. So it’s not like if you have a detection for that specific techniques, then you are covered.
Right. So there can be multiple ways that you can detect a certain technique. Right. So again, those are like some of the points that you should be think through before you start that. And then there’s more comprehensive reading that you should go after and then read microdack papers and blogs and the threat articles that people have published on how they have operationalized successfully in their organization.
Host: Right. It makes a lot of sense. Right. One thing that stood out from what you said is it’s not that it’s final. Like whatever is documented, that’s final. It’s evolving as well. Right. There are new threat techniques are coming up or new attack vectors are coming up, so it’s always good to stay up to date and apply those, whichever is applicable to your organization. And another thing that you highlighted is there are many techniques and many areas.
Right.
So in your mind, do you have top five areas that you look for from a threat hunting or blue teaming perspective.
Ashwin: Yeah, again, this question is very debatable. I believe the one way that I’ll probably approach this is understanding the broader trend reports. So I think one thing that we have recently seen is the CrowdStrike publishes the Threat Trend Report where they have basically outlined, let’s say, what are the top techniques that they have seen. Like most of the attacks being used. Right. So again, you can pick up like depending upon in which area that you are working, there can be specific techniques that may be available. Again, most of the I believe those reports is also focused most on the windows or on premises side. There are other reports that may be available for cloud security or like cloud attack paths.
Yeah, again, I think it really depends upon where or like which attacks that you prominently are seeing. Real world attacks such as if you take an example of Capital One breach, like there’s certain techniques that is available in order to get initial access. So valid accounts or exploiting the public facing websites are kind of like password spraying. The organizations are kind of like initial access techniques that you can get started and then there can be other techniques that are available in other microtech techniques.
Host: That makes sense. It’s more around what applies to your organization and what attack paths or what is the exploitability for your setup. Maybe follow those rather than just going through top five somebody has documented. Right, that makes a lot of sense.
So now, coming from reading about the attack frameworks, threat hunting, now doing the actual work. Right. So when it comes to threat hunting, many team members work right on this and there are many tools also are used. And this is one of the areas we have not covered yet in our podcast is around how do you use, let’s say Jupyter network notebook or any Threat Hunter playbooks that you use for your threat hunting work.
So, since you use Jupyter Notebook a lot, can you just briefly tell us what does Jupyter Notebook do? How does it work?
Ashwin: Yeah, certainly. I think Jupyter Notebook is not like a new technology. It has been used in other areas like finance also as well as the education space, like to teach certain tutorials and other things. But very recently or not very, I would say in the last few years, we have seen the adoption of that in the security industry as well. And specifically I’ve been involved, also involved in the specific conference, Jupyter Thorn, which was all about how does various organizations are using the Jupyter notebooks. Right. So I believe before we go into the start, I think Jupyter Notebook is basically just a live document where you can combine your kind of text notes along with the live code that you can run. Right.
So think of it like a live document which can have your incident response playbook, but it’s just not instructions, but you can also include the queries and then get inline outputs in the same document. Right, so there can be various use cases where you can go and use that thing. Again, the automation is not like the area that probably jupyter notebook is solving. There are also like other specific tools available, like logic apps and other things that allows you to do this kind of like step by step instruction and then make it as a workflow and then run repeatedly. Right. What jupyter notebook is capable is you can really write complex language core such as like Python visualization libraries and also like various other, let’s say, ML detection approaches and then incorporate those into the jupyter notebooks. So you can really take the capability of that live code thing in whatever way that you want.
Right, so think of it to use from a visualization perspective or think of it from a detection perspective or maybe implementing any complex workflows which requires like, getting data from various data sources and then doing some munging and then maybe making some changes along the way and then publishing outputs finally to your SIM platform. So, yeah, there are various use cases, and again, there are various interesting stuff that if you want to see how organizations are using, you can definitely check the jupyter on the couple of conferences that we have done recently that have various talks that are available, like how various companies are using jupyter notebooks in various other use cases.
Host: Yeah, so I like how it combines both the notebook, as in you are just taking notes, and also the playground aspect, right, where you can experiment with your data. And I’ve seen a lot of adoption around ML use cases, right. You pass the data, you run some model, and you have the output as well, all part of the same notebook. So it makes a lot of sense. I’m curious, how do you use it for your threat research?
Ashwin: Yeah, in our scenarios, we are basically publishing various customer scenarios that customers can adopt in their environment. So few of the things that we have done in the past is like kind of templatizing the common investigation workflows. Right. So let’s say when an incident happens, there are some series of steps that customer wants to do, such as like, devoting across various data sources, gathering the context, and then making it available right away in that notebook, right, so that the analyst or the investigator do not have to time spend on executing those queries, right.
Once they know that there are some series of steps that needs to be done on a regular basis when an incident occurs, such as gathering those contexts and then making that available. So that is one use case. Again, there are specific notebooks that are available where we can showcase various anomaly detection approaches by such as ML libraries or like statistics approaches. So, very recently, we published a blog post on the anomaly detection approaches using isolation forest. So that part is also incorporated as a part of Jupyter notebook that people can basically use it in relation to any other data sources.
Host: Okay, makes sense. And we’ll also gather the blog post details from you and we’ll share that with our audience so that they can also benefit from it. So one of the things that you highlighted is you have playbooks, right, when there is an incident occurs, what should organizations do? And what we have read, what we have noticed is when there is a security incident, most organizations are not prepared, they’re not sure how to handle it, right? How should they be dealing the attack and particularly if it is happening for the first time, where they are not sure at all. Right,
so how should the organization determine the level of the event in that case? Like what severity that event is and how can they once they understand what is the level, how do they mitigate that and ensure that there is minimal damage which is caused?
Ashwin: Yeah, so I think this question has like a lot of avenues. What I would start with again is having the right incident response plan and then preparedness is very essential if you have never handled a security incident, right? So first of all, like documenting what one should do when an incident occurs. Like how do we engage various teams, various stakeholders? If you are a part of broader or where there are various teams, how do we engage those teams on a quick basis and then engage and then basically operationalize any mitigation plan if you want to is first of all very essential that can come under the preparation or the planning approaches.
Again, if you never handle an incident, maybe it’s a good idea to start conducting some tabletop exercises and then see where do you stand if an incident occurs. So that will give you the capability to understand how do we assess the incident scope? Primarily, if you want to understand the scope of the incident is pretty much comes down to how do we assess the impact, like what are systems are affected, which area of your organization are affected. Right? And then based on that impact, maybe there can be like variety of mitigations that may be available, such as, let’s say an incident related to a specific vulnerability. Maybe first identify the impacted systems and then maybe going through your inventory and understand like these systems are impacted. And then maybe engaging the right stakeholders to basically implement that mitigation plan.
Deploying patches or doing various other if there are no patches, then maybe find out some workarounds to minimize the impact such as probably if it’s a public facing vulnerability, maybe how do we deploy network-level rules to minimize the impact. So yeah, I think it’s kind of subjective and requires different context to basically answer that. But yeah, I would say like preparedness, conducting tabletop exercises again, there’s various material available from Cert and other things that you can refer which kind of provides like a good framework on how should we define those processes and documents.
Host: Okay, that makes sense. So having a plan and also doing fire trail exercises so that you know your preparedness. So you do threat hunting and you might have some findings from there and you have an incident plan. Do you think combining these two would help organizations in understanding any exploitability of their environment and prevent any incident to reoccur in future?
Ashwin: Yeah, definitely. Lessons learned is probably the last step of the incident response plan. Right? So whenever an incident is kind of close, what lessons that you have learned from that incident and then how do we make sure that to incorporate that plan into as part of your regular workflow? So let’s say patch is kind of required as part of the incident, like maybe going back and then seeing how does that patch kind of aligns with your patch guidelines and other workflows, right? Yeah, it’s kind of connecting the dots back to your processes and then improving those processes so that to make sure that how do we make sure that as part of the root cause analysis of that incident, how do we basically prevent that not happening, that incident again in future?
Host: Okay, that makes sense. So in a way it’s a loop, right? You got an incident, you do a review of exactly what you learned, use that for your threat hunting again, and it becomes a cycle so that you improve that process for future.
The last question that I have around this is let’s say somebody who wants to start their career in threat hunting or cloud security in general, where should they start from?
Ashwin: Yeah, I would say like this is very, I guess, rich age to live in. If you’re starting with security, there are various resources available. If I remember like 1215 years back when I started my career, there were very limited resources to understand the technical aspects. There were like less resources available, maybe reading through books and understanding what was happening. But in this age, I think there are various resources available. So I would say if you’re starting up with security, with your career and then you want to gain into the threat hunting or cloud security in general, maybe understanding which role you want to go after, there are again various kinds of career path that exists in those areas which are very broad, like threatening security. Right, so understanding what you want to do and then probably maybe creating a kind of study plan or kind of plan that you wanted to do in order to achieve that.
Right. For example, in my case, if you really wanted to be a threat hunting or a detection engineer or a threat researcher, maybe starting with again, I started with an understanding of operating system and system administration or those concepts that’s kind of like additional level knowledge that’s required. And then that has been helpful for me in many parts of my career. So after having that foundation maybe again in relation to cloud, maybe understanding how are the various cloud platform works, what is the various core components that exist when we say that we have moved to the cloud right? And then in relation to security, it’s basically understanding again. You can start with attack framework detection engineers. So there are again similar to microdac framework. There are other frameworks exist in relation to detection such as Sigma and then there are also similar kind of rule sets that are available from other vendor.
Like Elastic has their own rules repository and then various research organizations can have similar way, right? So that out there maybe following certain people who are like industry leaders on social media, maybe you will not understand half of the things that they say in your business. But then over the I think you will get understanding of and then probably able to use some of the things that they have said in your part of the world. So yeah, I’m not sure gives a structured plan but I think those are like few things that I would mention on a high level.
Host: Yeah,I think the thing that stood out is the basics, right? Understanding if you are going for cloud security maybe understand how the cloud operates in detail so that you have better knowledge of how it works. If you are going for more traditional then maybe look at how the operating system works, networks work and stuff like that. So yeah those make sense.
So a follow up question on that is any reference material that you use or any blogs, podcasts, videos and on top of that any tools that you use? I know that you use jupyter notebook quite a bit. Other than that, any other tools that you use which somebody who is starting can benefit from?
Ashwin: Yeah, I think there are again I primarily use social media. Again, I have certain researchers that I follow and then I’ve created kind of Twitter list out of those cloud security researchers so there are various blogs, articles and newsletters that get published on a weekly basis or like maybe more frequent that I follow. So having that curated list is kind of I start with those things and then probably maybe pick up, maybe cherry pick up some of the things that I do as part of the work such as which are related to mostly related to threat detections and other things. So yeah, I think there are various research firms like we again there are other forms that are continuously published like threat research blogs and then there are also various researchers like Dr. Nestori who publishes lot of articles around like aid internals and various ADAC techniques. So understanding what is published out there and then probably following those things and how do we basically deploy detection on those things.
Host: Okay, makes sense. So what we’ll do is we’ll tag some of these when we publish the video so that our audience can also go there and learn from the same resources. So, yeah, that’s a great way to end the security question.
Host: So let’s go to the rapid fire section.
Rapid Fire
Host: Now, the first question is a one liner code that keeps you going.
AshwinYeah. So I think this is from the book that I’ve read a few months back, atomic Habits. I think it has a basically becoming a lifelong learner can really leverage on the compounding ability of knowledge. Right? So basically similar to the compound interest is the Eight Wonder of the World. Right. But using that analogy in relation to the knowledge. So keep learning small, small things and then it will eventually lead you to success in the long term.
Host: I agree. I have read that book as well. So it’s all about like if you improve 1% every day at the end of the year, you have improved a lot compared to folks who are maybe not even doing that. Right. So, yeah, spot on. The second question is if you were a superhero of cybersecurity, which power would you choose to happen to?
Ashwin: Yeah, so I would say from a threat hunting and maybe on the similar terms, if I had to choose, I think Hawkeye is probably a superhero. And then maybe having that instead of like a bow and arrow, I would probably use keyboard and some of the queries I can use to get probably catch the bad guys.
Right. That kind of have similar analogy. And then having that power of maybe hunting for bad guys using the keyboard.
Host: No, makes sense. For technology people, it makes a lot of sense. Right. So the last question is three blogs or books or websites which you go to to stay up to date.
Ashwin: Yeah, so I think recently I’ve been the fan of the newsletters which kind of gives like a curated list of blogs and other things. So there are a couple of newsletters that I’ve subscribed to is a cloud sake list. It’s very good one, which is really resonates to what I do on a day to day basis about the cloud security. TLDR Sake is also another one which has a good curated list of articles that you want to publish. Apart from that, again, the social media is kind of like a good kind of dopamine to get new ideas and then adding those new articles and new blogs. So, yeah, I think those are my go to list. Again, with the social media, it always gets, like, distracting and then move across on going on various other things. But, yeah, if you want to be focused on maybe starting with those curated lists is a good way.
Host: Yeah, it’s funny that you mentioned those two blogs, right? Because I also have subscribed. I also get updates from there. And then again, the third point which you highlighted is have some curated research or threat hunters that you follow so that you also learn from their learnings, right. In a way. So, yeah, makes a lot of sense. So, yeah. Thank you so much, Ashwin, for coming to the show, sharing your knowledge. I learned around Jupyter notebook, which I had no idea. Threat hunting, I learned some more details around that as well.
So, yeah, thank you for coming to the show.
Ashwin: Yeah, thank you. Thank you for giving this opportunity. And it’s good to always connect with various people and industries to give the insight. So thank you for the opportunity, for bringing me on the show and then sharing those insights.
Host: Yeah, absolutely. Thank you for coming. And to our viewers, thank you for watching. Hope you have learned something new. If you have any questions around security, share those at scale to zero and we’ll get those answered by an expert in the security space.
With this, we end our Season 1. Thanks for sticking with us and be part of our journey.
We will be starting with our Season 2 soon. Hope to see you there.
See you in our next episode. Thank You.
