https://retiari.us/tags/monitoring/ Recent content in Monitoring on Alex Maestretti Hugo -- gohugo.io en-us Fri, 07 Apr 2017 12:56:35 -0700 https://retiari.us/talks/aws_hunting_sans/ Fri, 07 Apr 2017 12:56:35 -0700 https://retiari.us/talks/aws_hunting_sans/ A talk from SANS Threat Hunting Summit 2017 on how to look for adversary activity in AWS. https://retiari.us/talks/ebpf_bsidessf/ Tue, 07 Feb 2017 12:56:35 -0700 https://retiari.us/talks/ebpf_bsidessf/ A talk from BsidesSF 2017 on how to use extended Berkley Packet Filter (eBPF) to monitor linux workloads. https://retiari.us/posts/apparmor-as-auditd/ Thu, 25 Aug 2016 00:24:23 +0000 https://retiari.us/posts/apparmor-as-auditd/ When we in security first approached our performance team with the bright idea to monitor all the syscalls, we were met with distinctly different views of the project. Whereas security was imagining hackers weeping outside the gates of our impenetrable systems; performance saw a specific dollar cost for monitoring that had better add value. At Netflix’s scale even a small performance improvement translates directly into large savings. As we dug into the call flow with Brendan Gregg and his set of really useful performance tools, we found that AppArmor is already inspecting system calls by default in Ubuntu. We already use AppArmor to secure various executables and are expanding its use. Also it turns out to be hard to remove from Ubuntu, so at this point so we had a sunk cost from performance’s perspective working in our favor. The challenge was could we replicate the functionality we got from the k/auditd framework with AppArmor? There are a few design choices working against us here: 1) AppArmor works on a whitelist principal, only allowing certain actions, 2) AppArmor is focused on enforcement of a policy based on a file path to a specific executable. With audit we want to ‘allow all’ (non-blocking) then audit certain things, and we want to do this globally not per executable file. https://retiari.us/posts/auditd-for-network-monitoring/ Mon, 22 Aug 2016 00:24:23 +0000 https://retiari.us/posts/auditd-for-network-monitoring/ Short of full PCAP it would be nice to capture metadata on network connections to and from an instance. In AWS EC2 it is difficult to perform network monitoring using traditional appliances as inter-instance traffic is not exposed to customers. You can get VPC flow logs which are useful; however, it is still desirable to have both a network and a host perspective on this activity. Both are valuable. VPC logs are difficult to corrupt from the host, and the host has more context on the executables that initiated the communication. So our goal is to collect metadata on network connections, including the remote address and the local process, then correlate that with VPC logs. Let’s see if we can pull this off with auditd. By monitoring at the system call level this technique should be difficult to subvert. Other methods might include alternate system call monitoring technologies, or various network utilities (tcpdump, iptraf, etc). Using nftables (iptables) logging could work… but onward with auditd. Curling an Address If you want to try this, set up auditd (apt-get install auditd on Ubuntu) to monitor successful socket() system calls using auditctl (reference this tutorial from RedHat). The first step for process communication is to allocate via socket(int domain, int type, int protocol) which results in an entry like this: https://retiari.us/posts/what-to-monitor-on-instance/ Wed, 17 Aug 2016 00:24:23 +0000 https://retiari.us/posts/what-to-monitor-on-instance/ I think of security relevant endpoint data within three domains: disk, memory and network. There are plenty of other taxonomies one could use. Disk, is generally addressed with File Integrity Monitoring (FIM). Memory, I am really talking about process execution, so anything that is exec’d and what it subsequently does (ie syscalls). Network, data transiting through external interfaces. Of course all three domains overlap - network configurations are stored on disk, sockets are opened with system calls, etc. There are also other useful optics like user identity and access that span these primitives. To cover these areas on our production systems we look to the Linux audit framework. Audit provides two types of rule; file and system call. With these two primitives any other type of desired monitoring can be built. In fact ‘file’ is simply a wrapper on the open() syscall. In addition to system call monitoring, Linux system logging functions provide pre-built enrichments that correlate and present low level data in a more relevant format. For example auth.log should be captured, rather than trying to rebuild those events from syscalls yourself. The logging landscape in Linux is rather convoluted and evolving (ex. systemd-journald), so your configurations are likely to be distribution specific. https://retiari.us/posts/security_monitoring_cloud/ Mon, 08 Aug 2016 00:24:23 +0000 https://retiari.us/posts/security_monitoring_cloud/ What should security monitoring look like in an AWS EC2 environment? Consider an EC2 environment running standard Linux AMIs with two use cases: detection and response. Detection is best supported by streams of real time events emitted by the endpoints. You can run signatures, heuristics and ML on these streams to detect suspicious conditions and trigger automated response: block, bounce, increase level of monitoring, etc. Response generally involves a human who wants to ask questions about the entire fleet, a subset or a single instance. With public cloud there is a new layer of relevant information, in our case from CloudTrail, CloudWatch, VPC flow logs, and other AWS sources. More on those later; we also need traditional system-level monitoring of instances from our side of the shared responsibility model. For detection ‘pull’ is not ideal. Many solutions rely on polling on some interval which results in a window between pulls. An attacker that is aware of your logging then has an opportunity to cover their tracks. For detection I prefer push or event driven solutions. A system event occurs, a log entry is created and immediately sent off-instance to remote analysis and storage. The drawback is limited context available at the moment of generation with which to determine relevance.