https://retiari.us/categories/detection/ Recent content in Detection on Alex Maestretti Hugo -- gohugo.io en-us Fri, 15 Jun 2018 12:56:35 -0700 https://retiari.us/posts/geekwire/ Fri, 15 Jun 2018 12:56:35 -0700 https://retiari.us/posts/geekwire/ Inside the private event where Microsoft, Google, Salesforce and other rivals share security secrets. https://retiari.us/posts/socless/ Thu, 26 Apr 2018 16:25:49 +0000 https://retiari.us/posts/socless/ A SOCless Detection Program at Netflix I am excited to share that we are investing in greater detection capabilities as part of the SIRT mission. There are a number of existing detection efforts across Netflix security teams. This is an opportunity to enable those efforts, while creating stronger alignment between detection and response. Of course this being Netflix our culture and our tech stack loom large in our consideration of how to expand our detection program. We want to avoid the traditional pitfalls and optimizes for our overall security approach. The last thing we want is a bunch of lame alerts creating busy work for a large standing SOC. Required reading for anyone interested in this area are Ryan McGeehan’s Lessons Learned in Detection Engineerin*g and theAlerting and Detection Strategy work from Dane Stuckey. I have borrowed liberally from their efforts. There are many ways to break this down, but I have settled on the following: Within these categories, this is what a mature program looks like: Data Sources One needs to find out about existing security relevant data as well as create it. This includes data discovery, deployment of instrumentation like eBPF or OSQuery, and working with application teams around logging best practices. https://retiari.us/talks/aws_hunting_sans/ Fri, 07 Apr 2017 12:56:35 -0700 https://retiari.us/talks/aws_hunting_sans/ A talk from SANS Threat Hunting Summit 2017 on how to look for adversary activity in AWS. https://retiari.us/talks/ebpf_bsidessf/ Tue, 07 Feb 2017 12:56:35 -0700 https://retiari.us/talks/ebpf_bsidessf/ A talk from BsidesSF 2017 on how to use extended Berkley Packet Filter (eBPF) to monitor linux workloads. https://retiari.us/posts/apparmor-as-auditd/ Thu, 25 Aug 2016 00:24:23 +0000 https://retiari.us/posts/apparmor-as-auditd/ When we in security first approached our performance team with the bright idea to monitor all the syscalls, we were met with distinctly different views of the project. Whereas security was imagining hackers weeping outside the gates of our impenetrable systems; performance saw a specific dollar cost for monitoring that had better add value. At Netflix’s scale even a small performance improvement translates directly into large savings. As we dug into the call flow with Brendan Gregg and his set of really useful performance tools, we found that AppArmor is already inspecting system calls by default in Ubuntu. We already use AppArmor to secure various executables and are expanding its use. Also it turns out to be hard to remove from Ubuntu, so at this point so we had a sunk cost from performance’s perspective working in our favor. The challenge was could we replicate the functionality we got from the k/auditd framework with AppArmor? There are a few design choices working against us here: 1) AppArmor works on a whitelist principal, only allowing certain actions, 2) AppArmor is focused on enforcement of a policy based on a file path to a specific executable. With audit we want to ‘allow all’ (non-blocking) then audit certain things, and we want to do this globally not per executable file. https://retiari.us/posts/auditd-for-network-monitoring/ Mon, 22 Aug 2016 00:24:23 +0000 https://retiari.us/posts/auditd-for-network-monitoring/ Short of full PCAP it would be nice to capture metadata on network connections to and from an instance. In AWS EC2 it is difficult to perform network monitoring using traditional appliances as inter-instance traffic is not exposed to customers. You can get VPC flow logs which are useful; however, it is still desirable to have both a network and a host perspective on this activity. Both are valuable. VPC logs are difficult to corrupt from the host, and the host has more context on the executables that initiated the communication. So our goal is to collect metadata on network connections, including the remote address and the local process, then correlate that with VPC logs. Let’s see if we can pull this off with auditd. By monitoring at the system call level this technique should be difficult to subvert. Other methods might include alternate system call monitoring technologies, or various network utilities (tcpdump, iptraf, etc). Using nftables (iptables) logging could work… but onward with auditd. Curling an Address If you want to try this, set up auditd (apt-get install auditd on Ubuntu) to monitor successful socket() system calls using auditctl (reference this tutorial from RedHat). The first step for process communication is to allocate via socket(int domain, int type, int protocol) which results in an entry like this: https://retiari.us/posts/what-to-monitor-on-instance/ Wed, 17 Aug 2016 00:24:23 +0000 https://retiari.us/posts/what-to-monitor-on-instance/ I think of security relevant endpoint data within three domains: disk, memory and network. There are plenty of other taxonomies one could use. Disk, is generally addressed with File Integrity Monitoring (FIM). Memory, I am really talking about process execution, so anything that is exec’d and what it subsequently does (ie syscalls). Network, data transiting through external interfaces. Of course all three domains overlap - network configurations are stored on disk, sockets are opened with system calls, etc. There are also other useful optics like user identity and access that span these primitives. To cover these areas on our production systems we look to the Linux audit framework. Audit provides two types of rule; file and system call. With these two primitives any other type of desired monitoring can be built. In fact ‘file’ is simply a wrapper on the open() syscall. In addition to system call monitoring, Linux system logging functions provide pre-built enrichments that correlate and present low level data in a more relevant format. For example auth.log should be captured, rather than trying to rebuild those events from syscalls yourself. The logging landscape in Linux is rather convoluted and evolving (ex. systemd-journald), so your configurations are likely to be distribution specific. https://retiari.us/posts/dfir_aws/ Thu, 11 Aug 2016 00:24:23 +0000 https://retiari.us/posts/dfir_aws/ I previously discussed some of the challenges and opportunities of working in a public cloud when it comes to forensic response. Since you don’t own the datacenter, traditional tactics of grabbing boxes and imaging drives with write-blockers don’t apply. The goal is still to capture an unaltered image of both the disk and live memory, preferably without any interaction with the target system. Network traffic is another useful optic, and in traditional models can be captured out-of-band without disturbing the target through taps. The reality is it is often inefficient to answer your questions, especially fleet-wide, by capturing full live memory and performing analysis on it, in which case you might consider live response capabilities as well to query the running system. The danger is that by interacting with the system you might alter the evidence or tip off the adversary. Ideally you will have already logged relevant activity and stored it off target. All of these optics are valuable, and their value compounds when combined. In deception it is difficult to control multiple channels; so for example, while the OS might lie, it is hard for that lie to stand up to correlation across network data. In AWS the snapshot API for EBS-backed instances is an excellent tool. https://retiari.us/posts/security_monitoring_cloud/ Mon, 08 Aug 2016 00:24:23 +0000 https://retiari.us/posts/security_monitoring_cloud/ What should security monitoring look like in an AWS EC2 environment? Consider an EC2 environment running standard Linux AMIs with two use cases: detection and response. Detection is best supported by streams of real time events emitted by the endpoints. You can run signatures, heuristics and ML on these streams to detect suspicious conditions and trigger automated response: block, bounce, increase level of monitoring, etc. Response generally involves a human who wants to ask questions about the entire fleet, a subset or a single instance. With public cloud there is a new layer of relevant information, in our case from CloudTrail, CloudWatch, VPC flow logs, and other AWS sources. More on those later; we also need traditional system-level monitoring of instances from our side of the shared responsibility model. For detection ‘pull’ is not ideal. Many solutions rely on polling on some interval which results in a window between pulls. An attacker that is aware of your logging then has an opportunity to cover their tracks. For detection I prefer push or event driven solutions. A system event occurs, a log entry is created and immediately sent off-instance to remote analysis and storage. The drawback is limited context available at the moment of generation with which to determine relevance.