https://retiari.us/categories/security/ Recent content in Security on Alex Maestretti Hugo -- gohugo.io en-us Thu, 13 Sep 2018 22:31:21 +0800 https://retiari.us/posts/ir_in_la/ Thu, 07 Mar 2019 19:27:40 +0000 https://retiari.us/posts/ir_in_la/ I am excited to announce the latest opening on our Security Incident Response Team (SIRT), which is also our first LA-based role. Netflix is working hard to provide a wide range of content choices for our customers and with all that business activity comes some amount of work for the SIRT. In this role you will be exposed to a high operational tempo with a lot of growth and ambiguity. It is a very independent role where you will set the agenda for how SIRT can best add value in content related spaces. That is not to say you will be without support! You will be a core member of SIRT with peers in Los Gatos to help with training, tooling, and crisis management, as well as a growing Studio Information Security team next to you in LA. This role will be part of the SIRT on-call rotation so you will respond to issues across all lines of business while on call. We work hard to keep our on-call burden reasonable and SIRT members are expected to make progress against longer term objectives to ensure on-call stays engaging but not overwhelming. SIRT members tackle a range of projects from programmatic improvements to our response, training our team and others on how to respond better, developing tooling to automate crisis management and forensic activities, and other cross team contributions. https://retiari.us/posts/ti_for_vuln_mgmt/ Sat, 22 Sep 2018 16:59:48 +0000 https://retiari.us/posts/ti_for_vuln_mgmt/ Effective patch management remains as much a challenge today as it was a decade ago. The missing foundational piece is asset inventory - I would argue the biggest security gap most organizations have. Even assuming good inventory challenges remain for Threat Intelligence (TI) to identify new vulnerabilities in the supply chain - to include vendor products, open source libraries, software tool-chains, etc. There are frequent race conditions between what is known and the emerging unknown. Rapidly identifying those emerging unknowns - eg. zero days - is a key part of a threat intelligence program. Currently the one-stop-shop for vulnerability information is MITRE’s Common Vulnerabilities and Exposures (CVE) program. There are a number of smart dedicated people working on this tough problem space, but still the program has significant shortcomings as called out recently by the US House Energy and Commerce committee. Among the issues are that they provide at once too much information on inconsequential vulnerabilities causing fatigue, and too little information on those you do care about. This is compounded with a lack of structure or consistency to the reported information that is available. Perhaps most concerning is the significant delays between discovery and CVE release. For the most impactful vulnerabilities, those that provide remote execution and are ‘worm-able’ (allow for self propagation or mass scanning), the announcement of the vulnerability kicks off a race between defenders seeking to identify and remediate their systems, and attackers trying to weaponize the vulnerability then discover and exploit vulnerable systems. https://retiari.us/posts/attribution_and_threat_models/ Thu, 20 Sep 2018 16:58:39 +0000 https://retiari.us/posts/attribution_and_threat_models/ Thank you for joining us on article two of our series about the Future of Threat Intelligence at Netflix. Early in my career I liked to imagine myself locked in a battle of wits with a Karla of somesort. The reality is less literary archrival and more diverse set of ne’er-do-wells, but equally challenging. Absent a specific villian, what is the role of attribution in a corporate threat intelligence program? As with most things it turns on business value delivered - if knowing the name of the person targeting your systems helps reduce risk for your organization then discovering that information should be a goal of your intelligence program; however, this is seldom relevant to business risk reduction outside the rare cases where one might cooperate with law enforcement to target a particularly problematic fraud actor, or, as we will discuss later, in the world of physical and information security convergence. While law enforcement needs individual attribution, most often corporate security programs can work at a coarser granularity without losing impact. The goal is to better understand the intent and capabilities that comprise a threat. Attribution is valuable during an incident inasmuch as it provides leads on how to scope the intrusion or detect the attacker. https://retiari.us/posts/future_of_iocs/ Thu, 13 Sep 2018 16:56:00 +0000 https://retiari.us/posts/future_of_iocs/ Thank you for joining us for our series on Threat Intelligence. This is the first of our content posts, linked from our introductory post. The one constant in the information threat landscape is that it is always changing. Yet too often, when the term ‘threat intelligence’ is used, it is in reference to static Indicators Of Compromise (IOCs). Are these traditional indicators still relevant or useful today? How about in the near future? Most importantly, how adept is your Threat Intelligence (TI) program at dealing with these constant shifts in landscape? Can your current TI detect a black swan event, or a black swan hatching? IOCs or “Oh, I see” While there is still a great deal of commodity malware that can be found using file hashes, DNS names and IP addresses, many modern attacker techniques render these indicators unusable. For example, malware can be repacked thus changing the resulting hash, IP addresses are largely communal and interchangeable in today’s virtual hosting environments, and otherwise legitimate domains can be hijacked (or in the case of sites with user generated content, simply repurposed) to serve as malicious command and control infrastructure. A common shortcoming in TI programs is an outsized focus on malware. https://retiari.us/posts/future_of_intelligence/ Wed, 12 Sep 2018 16:54:43 +0000 https://retiari.us/posts/future_of_intelligence/ UPDATE: I am pleased to announce that Chris Cochran has joined the team to lead our intelligence program. He will be posting the rest of the articles in this series and I will link over to his work from here! I have been chatting with Mary Landesman, our intelligence lead, about how best to derrive business value from threat intelligence in Netflix’s unique culture and modern technology stack. Threat intelligence can be a slippery concept. While we have seen it work well, we have also seen it struggle to keep up with changing technologies and observed uneven return on investment across industries and organizations. We came up with a bunch of questions and a few promising theories. I will share those in a series of posts in the hope of drawing out debate and new ideas. Mary is looking towards retirement! So we are looking for the next generation intelligence leader to continue our momentum for Netflix, and the broader community. If that sounds interesting please get in touch: amaestretti@netflix.com Thanks to Mary for the insights and rewrites. Some of the topics (I will add links as they post): The evolution of IOCs Attribution Intelligence for Vulnerability Management https://retiari.us/posts/geekwire/ Fri, 15 Jun 2018 12:56:35 -0700 https://retiari.us/posts/geekwire/ Inside the private event where Microsoft, Google, Salesforce and other rivals share security secrets. https://retiari.us/talks/first_2018_podcast/ Thu, 07 Jun 2018 12:56:35 -0700 https://retiari.us/talks/first_2018_podcast/ A podcast for FIRST 2018 on how the Netflix SIRT operates. https://retiari.us/posts/a-learning-security-organization/ Wed, 30 May 2018 22:51:59 +0000 https://retiari.us/posts/a-learning-security-organization/ A Learning Security Organization When I was six my family spent a year in France as part of a Fulbright exchange. I dragged my parents to every castle I could. I loved to climb around the fortifications and explore the armory museums that were often attached. I recall my father pointing out a particularly imposing suite of full plate armor, with a tiny hole in the left breast plate. This was likely from target practice as I don’t think that armor was ever intended for the field, but the lesson was clear - technology can quickly swing the balance between defense and offense. Complexity and change are dominant factors today - If anything can be relied upon in the modern world it is that nothing can be relied upon for long. In The Fifth Discipline Peter Senge outlines an organization whose product is learning. He posits that such a ‘learning organization’ is the best model to handle change and complexity, particularly compared with more rigid industrial approaches to knowledge work. Complexity and change are certainly hallmarks of the technology sector, and security, so how could we build a security organization around the principles of learning? Senge’s five disciplines are: Personal Mastery, Mental Models, Shared Vision, Team Learning and Systems Thinking. https://retiari.us/posts/socless/ Thu, 26 Apr 2018 16:25:49 +0000 https://retiari.us/posts/socless/ A SOCless Detection Program at Netflix I am excited to share that we are investing in greater detection capabilities as part of the SIRT mission. There are a number of existing detection efforts across Netflix security teams. This is an opportunity to enable those efforts, while creating stronger alignment between detection and response. Of course this being Netflix our culture and our tech stack loom large in our consideration of how to expand our detection program. We want to avoid the traditional pitfalls and optimizes for our overall security approach. The last thing we want is a bunch of lame alerts creating busy work for a large standing SOC. Required reading for anyone interested in this area are Ryan McGeehan’s Lessons Learned in Detection Engineerin*g and theAlerting and Detection Strategy work from Dane Stuckey. I have borrowed liberally from their efforts. There are many ways to break this down, but I have settled on the following: Within these categories, this is what a mature program looks like: Data Sources One needs to find out about existing security relevant data as well as create it. This includes data discovery, deployment of instrumentation like eBPF or OSQuery, and working with application teams around logging best practices. https://retiari.us/posts/rsa2017_recap/ Fri, 17 Nov 2017 12:07:25 +0000 https://retiari.us/posts/rsa2017_recap/ This year I attended Bsides SF, AGC’s tech summit, a threat sharing group meeting, Google’s security talks and a bit of RSA; along with various coffee’s, dinners and drinks. I presented on eBPF at Bsides as well as at a threat sharing group (APT SIG) and received some great feedback. I also hosted a P2P session on IR in the Public Cloud that was well attended and I thought the discussion went well. The ecosystem of conferences that have sprung up in RSA’s orbit is impressive. While I could not bring myself to get any closer to the expo floor than the overwatch position from the entrance level, I did get to chat with investors, founders, and folks like me on the buy side in more civilized venues around the city. I wanted to take a moment to organize my thoughts and share some of the trends. Always a good roadmap to the security space from Momentum Partners here. Artificial Intelligence AI is the new, ML which is the new Big Data. While there are some very reasonable applications of ML and Big Data to security, calling any of them AI is a big stretch. We asked a couple different VCs about their reaction to companies claiming to solve security problem X with AI. https://retiari.us/posts/memory_forensics_cloud_containers/ Wed, 11 Oct 2017 12:59:25 +0000 https://retiari.us/posts/memory_forensics_cloud_containers/ At Netflix my targets are primarily Linux VMs running in a microservices environment. I rarely have to pursue criminal prosecution but often want to figure out what is happening. Much of the work in memory forensics has targeted Windows systems and forensically sound capture methodologies that pull the entire system memory. This focus makes sense given the attack surface that user systems represent; however, given my goals I believe that userland memory acquisition and targeted analysis of memory at process granularity could yield more efficient results. Microservices, CI/CD deployments of immutable and ephemeral instances, the trend to containers, and various hardening efforts in the kernel, confer advantages on the defender that we must leverage to keep up with the scale and velocity of our environment. Another area that is not well developed is when target processes on the edge manage their own memory, as many of today’s popular languages do. I am curious what folks are doing in this space, so I wanted to mention some trends I see and solicit feedback: Desktops vs Datacenters vs Microservices User systems are the most difficult to secure. Particularly general compute devices operated by a technical workforce that needs full access to their systems, and will install all manner of tooling from around the web to accomplish their jobs. https://retiari.us/posts/uservice_for_disk_forensics/ Wed, 04 Oct 2017 12:55:22 +0000 https://retiari.us/posts/uservice_for_disk_forensics/ This is a follow-on to my previous post here. In the last few months I have had the chance to talk to a number of other companies and gotten a lot of great inbound response to the post about disk forensics in the modern era. To summarize: “Our goal is to move towards a streaming model for disk forensics by updating various existing tools that are built around the concept of imaging a single machine and creating a file. In a cloud environment we need to image multiple (even hundreds) of machines and process all those images into meaningful event data. Acquisition is fairly painless via API, but then what to do with the resulting snapshots?” While we were hoping for a cloud agnostic solution it seems we will need to solve this locally first and then seek areas to reuse code and integrate our systems in the future. We continue to approach this problem by decoupling acquisition, processing and analysis. We also hope to implement this in a scalable manner, ideally leaning into serverless techniques to minimize infrastructure maintenance. Acquisition Google is investing in DFTimewolf as part of the log2timeline/Plaso, along with GRR and Rekall. I think the Rekall agent architecture is the way to go. https://retiari.us/talks/aws_forensics_reinvent/ Mon, 07 Aug 2017 12:56:35 -0700 https://retiari.us/talks/aws_forensics_reinvent/ A talk from Re:Invent 2017 on how to look for evidence on disk in AWS. https://retiari.us/posts/disk-forensics-pipeline/ Mon, 22 May 2017 12:05:48 +0000 https://retiari.us/posts/disk-forensics-pipeline/ Last week we hosted colleagues from Google, AirBnB and Adobe to work on a joint effort to improve our disk forensics capability. Our goal is to move towards a streaming model for disk forensics by updating various existing tools that are built around the concept of imaging a single machine and creating a file. In a cloud environment we need to image multiple (even hundreds) of machines and process all those images into meaningful event data. Acquisition is fairly painless via API, but then what to do with the resulting snapshots? We are working towards a cloud agnostic streaming service for disk forensics. To get there we are currently focusing on acquisition in AWS and then sending those snapshots through Plaso for processing, pushing results into ElasticSearch, which is then fronted by Timesketch for analysis. I filed a number of issues with Timesketch to move us towards this, and we are working with the Plaso maintainers as they do some refactoring as well, but even now, with a few hacks, we can compose a system that achieves our goals. Acquisition Currently we have a set of boto scripts that enables us (Netflix SIRT) to capture snapshots of a group of instances within AWS. https://retiari.us/talks/aws_hunting_sans/ Fri, 07 Apr 2017 12:56:35 -0700 https://retiari.us/talks/aws_hunting_sans/ A talk from SANS Threat Hunting Summit 2017 on how to look for adversary activity in AWS. https://retiari.us/talks/ebpf_bsidessf/ Tue, 07 Feb 2017 12:56:35 -0700 https://retiari.us/talks/ebpf_bsidessf/ A talk from BsidesSF 2017 on how to use extended Berkley Packet Filter (eBPF) to monitor linux workloads. https://retiari.us/posts/fixing-ir/ Fri, 16 Sep 2016 12:11:02 +0000 https://retiari.us/posts/fixing-ir/ What makes detecting and responding to malicious activity on information systems so difficult? Incident Response and investigation can be incredibly intellectually rewarding. You get to puzzle together what transpired and take action to defend your systems, while working against a wily and adaptive adversary. On a bad day, however, you will find that most of the puzzle pieces are missing and there is no wily adversary present, only a deluge of false positive security events being happily emitted by an expensive PRNG called a SIEM. The core issues are volume of activity on, coupled with the complexity and change of, the systems you are defending. Traditional IT management compounds these issues rendering effective response impossible. DevOps, microservices and cloud infrastructure, in solving for scalability and development velocity, also offer solutions to these core security problems, and have the potential to Make IR Great Again. IR allows you to exercise a wide range of security skills. The diversity of events that one encounters ensures a varied and engaging workflow that continues to challenge. It takes not only a keen intellect and technical skill, but a body of experience from which to draw. Often there is no runbook, so you cobble together inferences from similar past cases and drive on. https://retiari.us/posts/apparmor-as-auditd/ Thu, 25 Aug 2016 00:24:23 +0000 https://retiari.us/posts/apparmor-as-auditd/ When we in security first approached our performance team with the bright idea to monitor all the syscalls, we were met with distinctly different views of the project. Whereas security was imagining hackers weeping outside the gates of our impenetrable systems; performance saw a specific dollar cost for monitoring that had better add value. At Netflix’s scale even a small performance improvement translates directly into large savings. As we dug into the call flow with Brendan Gregg and his set of really useful performance tools, we found that AppArmor is already inspecting system calls by default in Ubuntu. We already use AppArmor to secure various executables and are expanding its use. Also it turns out to be hard to remove from Ubuntu, so at this point so we had a sunk cost from performance’s perspective working in our favor. The challenge was could we replicate the functionality we got from the k/auditd framework with AppArmor? There are a few design choices working against us here: 1) AppArmor works on a whitelist principal, only allowing certain actions, 2) AppArmor is focused on enforcement of a policy based on a file path to a specific executable. With audit we want to ‘allow all’ (non-blocking) then audit certain things, and we want to do this globally not per executable file. https://retiari.us/posts/auditd-for-network-monitoring/ Mon, 22 Aug 2016 00:24:23 +0000 https://retiari.us/posts/auditd-for-network-monitoring/ Short of full PCAP it would be nice to capture metadata on network connections to and from an instance. In AWS EC2 it is difficult to perform network monitoring using traditional appliances as inter-instance traffic is not exposed to customers. You can get VPC flow logs which are useful; however, it is still desirable to have both a network and a host perspective on this activity. Both are valuable. VPC logs are difficult to corrupt from the host, and the host has more context on the executables that initiated the communication. So our goal is to collect metadata on network connections, including the remote address and the local process, then correlate that with VPC logs. Let’s see if we can pull this off with auditd. By monitoring at the system call level this technique should be difficult to subvert. Other methods might include alternate system call monitoring technologies, or various network utilities (tcpdump, iptraf, etc). Using nftables (iptables) logging could work… but onward with auditd. Curling an Address If you want to try this, set up auditd (apt-get install auditd on Ubuntu) to monitor successful socket() system calls using auditctl (reference this tutorial from RedHat). The first step for process communication is to allocate via socket(int domain, int type, int protocol) which results in an entry like this: https://retiari.us/posts/what-to-monitor-on-instance/ Wed, 17 Aug 2016 00:24:23 +0000 https://retiari.us/posts/what-to-monitor-on-instance/ I think of security relevant endpoint data within three domains: disk, memory and network. There are plenty of other taxonomies one could use. Disk, is generally addressed with File Integrity Monitoring (FIM). Memory, I am really talking about process execution, so anything that is exec’d and what it subsequently does (ie syscalls). Network, data transiting through external interfaces. Of course all three domains overlap - network configurations are stored on disk, sockets are opened with system calls, etc. There are also other useful optics like user identity and access that span these primitives. To cover these areas on our production systems we look to the Linux audit framework. Audit provides two types of rule; file and system call. With these two primitives any other type of desired monitoring can be built. In fact ‘file’ is simply a wrapper on the open() syscall. In addition to system call monitoring, Linux system logging functions provide pre-built enrichments that correlate and present low level data in a more relevant format. For example auth.log should be captured, rather than trying to rebuild those events from syscalls yourself. The logging landscape in Linux is rather convoluted and evolving (ex. systemd-journald), so your configurations are likely to be distribution specific. https://retiari.us/posts/dfir_aws/ Thu, 11 Aug 2016 00:24:23 +0000 https://retiari.us/posts/dfir_aws/ I previously discussed some of the challenges and opportunities of working in a public cloud when it comes to forensic response. Since you don’t own the datacenter, traditional tactics of grabbing boxes and imaging drives with write-blockers don’t apply. The goal is still to capture an unaltered image of both the disk and live memory, preferably without any interaction with the target system. Network traffic is another useful optic, and in traditional models can be captured out-of-band without disturbing the target through taps. The reality is it is often inefficient to answer your questions, especially fleet-wide, by capturing full live memory and performing analysis on it, in which case you might consider live response capabilities as well to query the running system. The danger is that by interacting with the system you might alter the evidence or tip off the adversary. Ideally you will have already logged relevant activity and stored it off target. All of these optics are valuable, and their value compounds when combined. In deception it is difficult to control multiple channels; so for example, while the OS might lie, it is hard for that lie to stand up to correlation across network data. In AWS the snapshot API for EBS-backed instances is an excellent tool. https://retiari.us/posts/security_monitoring_cloud/ Mon, 08 Aug 2016 00:24:23 +0000 https://retiari.us/posts/security_monitoring_cloud/ What should security monitoring look like in an AWS EC2 environment? Consider an EC2 environment running standard Linux AMIs with two use cases: detection and response. Detection is best supported by streams of real time events emitted by the endpoints. You can run signatures, heuristics and ML on these streams to detect suspicious conditions and trigger automated response: block, bounce, increase level of monitoring, etc. Response generally involves a human who wants to ask questions about the entire fleet, a subset or a single instance. With public cloud there is a new layer of relevant information, in our case from CloudTrail, CloudWatch, VPC flow logs, and other AWS sources. More on those later; we also need traditional system-level monitoring of instances from our side of the shared responsibility model. For detection ‘pull’ is not ideal. Many solutions rely on polling on some interval which results in a window between pulls. An attacker that is aware of your logging then has an opportunity to cover their tracks. For detection I prefer push or event driven solutions. A system event occurs, a log entry is created and immediately sent off-instance to remote analysis and storage. The drawback is limited context available at the moment of generation with which to determine relevance. https://retiari.us/posts/aws_ops_rig/ Mon, 01 Aug 2016 12:46:56 +0000 https://retiari.us/posts/aws_ops_rig/ Every responder needs a solid platform from which to operate. Something fast, secure and pre-loaded with relevant tools. Conducting incident response in a public cloud is the same as traditional IR but different. The public cloud is after all just other people’s servers; however, the ‘other people’s’ part changes things up a bit. You can’t roll into Amazon’s data center and grab a disk out of the rack, but Amazon provides an API for that. The concept they have put forward is ‘shared responsibility’ in which they handle the hypervisor and below, and above that is on you. We are moving towards a serverless approach to forensic response leveraging lambda functions and Amazon APIs; however, there is always a need for a solid ops box stocked with tools and enough permissions to get the job done. Design Goals: AWS EC2 native - leverage AWS APIs; minimize external dependencies - can’t rely on much in a crisis. Ephemeral instances - Spin up fresh, clean and quick; and kill them off once needed data is persisted to S3. Tools for response on Debian/Ubuntu AMIs with headless server (no GUI). Use corporate services day-to-day, but provide fallback options if they are down.