https://retiari.us/ Recent content on Alex Maestretti Hugo -- gohugo.io en-us Tue, 25 Jun 2019 14:37:03 -0700 https://retiari.us/books/who/ Tue, 25 Jun 2019 14:37:03 -0700 https://retiari.us/books/who/ Who was given to me by a friend of mine who is a partner at a venture firm. They have been using this to grow their practice, and giving it to all their CEOs. The text lays out a solid strategy for hiring ‘A players’ into your organization. It provides a set of steps that address many of the shortcomings in hiring that I have witnessed in organizations, in terms of lack of investment and rigor in the hiring process. Too often hiring managers just dust off the last job description they used, wait for HR to send some resumes, and then make a hiring decision based on gut. The authors lay out a much better approach, and break it into Sourcing, Selecting, and Selling, which are the same basic phases that I have written about for my hiring approach. It is a quick read with some good details, but there are of course summaries available that capture the broad outline well. Sourcing starts with a rigorous evaluation of your needs for that particular role, all too often folks just start hiring without understanding the specific needs of the role. Hire specialists, not generalists. Sourcing includes all your personnel’s resources for networking. https://retiari.us/books/fooled_by_random/ Sat, 22 Jun 2019 14:37:03 -0700 https://retiari.us/books/fooled_by_random/ While focused on investment banking there are many generally applicable concepts from Fooled by Random. The basic idea is that far more of our lives, our successes and failures, are dominated by chance than we care to admit, or are even able to perceive. Survivorship bias plays an outsized role in how society celebrates success. The hero worship around the rich and famous incorrectly ascribes the majority of their success to skill or intelligence or hard work or other traits that we claim to hold worthy; while the reality is they likely have some of those traits but no more so than the many others who did not get that pivotal first role, or whose company did not time their product market fit as fortuitously. While the societal implications of this survivor bias are interesting, especially in the age of social media and widespread feelings of inadequacy, more relevant to security is the trading strategy he proposes, wherein one seeks to make small gains during normal events while protecting yourself from going bust on unexpected events, or better yet capitalize on ‘impossible’ events to make large returns. He is highly critical of the value of historical data in predicting future performance. https://retiari.us/books/the_advantage/ Sat, 22 Jun 2019 14:37:03 -0700 https://retiari.us/books/the_advantage/ This title is extremely popular with Netflix leadership. The first hit on Google for a good summary delivers. The overall argument of the book is that the best advantage you can give your business is organizational health, a concept that, to me, resonated as consistency. Consistency between the values your organization espouses and those that it lives, as well as consistency in the understanding of what your organization’s purpose and priorities are - particularly across your leaders. Inconsistency in any of these leads to politics, dysfunction, confusion, and bureaucracy. The meat of the text provides a framework for achieving consistency (or clarity) with six questions, and explores each to some depth, providing pointers to the author’s other work to flesh out those areas. There is also a lead in with some thoughts on the importance and suggested structure of a leadership team, and text wraps up with suggested cadence for meetings to maintain momentum and clarity once established. The six questions are:  Why do we exist? What motivated the creation of the organization? How do we behave? This establishes the values of the organization, and includes a good discussion of core vs aspirational and minimum values.  What do we do? https://retiari.us/books/essentialism/ Mon, 17 Jun 2019 04:37:03 +0000 https://retiari.us/books/essentialism/ I already agreed strongly with the thesis going in. There is also a Netflix original on Minimalists worth watching. Everything you say yes to is a de facto no to something else, possibly something you are already doing. Pareto Principle from 1896. 20% of our efforts create 80% of our results. Most things we do are superfluous and should be deprioritized in favor of high impact activities. The trick is in identifying which are the 20%, which takes time and space to consider. It helps to have a concrete and inspirational goal to refer to. Anything not advancing that goal you gracefully say no to. If it is not advancing the goal you halt activities - zero based budgeting, don’t be trapped by sunk costs or ownership biases (we value things we own more than we would pay to acquire them). If you are not 90% sure you should take on a task, or hire someone, then that is a no. It is important to protect your health and not over extend. It is important to prioritize your family and relationships - set boundaries. Build buffer and slack into your schedule, but keep a queue of things to work when you find extra time. https://retiari.us/books/high_output_management/ Mon, 10 Jun 2019 04:37:03 +0000 https://retiari.us/books/high_output_management/ The book opens with an introduction to manufacturing (managing a breakfast factory) and suggests a framework for managing knowledge work based on the same principals. The correlations are imperfect, but it is a useful optic. Part II focuses on how to manage by looking for the best leverage in improving your team and adjacent team’s output. Some tips on meetings and a good taxonomy of types of meetings and how to best utilize them. An interesting overview of hybrid (matrixed) organizations, some of the pitfalls but ultimately the necessity of them and how to maneuver in them for best impact. He uses a sports analogy which resonated with me, given Netflix similar approach for high performance teams. A great section on modes of control and when to employ different management styles based on a concept of Task-Relevant Maturity. “Put them on a racetrack” the power of competitive spirit. References Parkinson’s Law which is one of my favorites. Has elements of Essentialism, finding the highest leverage activities and deprioritizing the rest. A bit dated in the performance and promotion sections, defends the Peter Principle as inevitable. In the technology sector at least we are beginning to unwind the societal pressures to move high performing technical people into management as the only outlet for career progression. https://retiari.us/books/benedictine_rule_of_leadership/ Thu, 30 May 2019 04:37:03 +0000 https://retiari.us/books/benedictine_rule_of_leadership/ Written in AD 530 the Benedictine order survives to this day in no small part thanks to the this management guide. While much has changed in the world around us, human nature has remained stubbornly consistent, and thus this book relevant. There are three parts and 15 total rules that make up the framework. Many familiar themes make an appearance: provide alignment on a common goal, hire the best, recognize merit over seniority, devolve power to individuals, encourage debate (questioning status quo), insist on ethical behavior. Leadership is done by example and with humility. A couple of items don’t resonate as strongly in a modern environment: stability is one of the tenets with members making a lifetime commitment. While a degree of stability is necessary to encourage taking risks, lifelong commitment would tend to limit the vital cross pollination of disciplines required for success today, there is a great deal more specialization than in the sixth century. The section on discipline reads like a PIP. While again, smart risk taking is to be encouraged through a sense of security and fairness (a-political analysis of failure) , changing underlying personality or performance through a formal process is generally less effective than making smart cuts. https://retiari.us/books/first_break_all_the_rules/ Sat, 25 May 2019 04:37:03 +0000 https://retiari.us/books/first_break_all_the_rules/ My main take away was: play to your employees’ strengths rather than trying to correct weaknesses. You cannot change the essential talents a person possesses, so enhance them and manage around the shortcomings, or find them a better fit. Provides an ordered list of 12 questions to gauge the health or your workplace. Discusses four key activities for a manager: select for talent (over experience, intellect or work ethic), define the right outcomes (context not control), focus on strengths, and find the right fit (person to role based on talents, traditional career ‘progression’ can work against this). https://retiari.us/books/army_leadership_manual/ Mon, 20 May 2019 04:37:03 +0000 https://retiari.us/books/army_leadership_manual/ A number of familiar themes and some that don’t apply as readily (physical fitness as a criteria for leadership). It is interesting that even in a very top down organization, where a commander’s orders have the force of law, there is a strong emphasis on empowering individuals with the resources and authority to accomplish their assigned mission, rather than control their every action. It is interesting to see many similar traits and approaches suggested at Netflix and in a military manual - candid communication for instance. The section on developing others is good and an area we beginning to explore more extensively at Netflix. https://retiari.us/posts/joy_of_feedback/ Wed, 15 May 2019 04:37:03 +0000 https://retiari.us/posts/joy_of_feedback/ Feedback is both a personal skill to master and a team sport. This post touches on feedback’s place at Netflix, how a team can leverage it to their advantage, and individually what we can do to become more effective at giving and receiving it. https://retiari.us/posts/my-first-post/ Tue, 07 May 2019 12:56:35 -0700 https://retiari.us/posts/my-first-post/ This is the first post! So naturally it needs to explain the build and deploy pipeline for this website. From Google Cloud Platform (GCP): Source Repositories, Builder, and Firebase. The site itself is a static site generated by Hugo. Google Cloud Source Repositories Are git repositories integrated into the GCP environment. You can use gcloud form the Goolge Cloud SDK (builtin to Cloud Shell) to authenticate rather than setting up SSH keys. git config --global credential.'https://source.developers.google.com'.helper gcloud.sh Then you can use git as normal, as well as control google specific aspects of the repositories gcloud source repos commands. Google Cloud Builder Uses a series of docker containers to build your project. We use two community docker containers for our steps. First to generate the site from the code repository with Hugo, and then to deploy it to Firebase static hosting. In the future I plan to develop a stage that pulls new content from Google Drive so I can maintain this site from gDocs. Firebase Hosts the static site. This requires granting the Cloud Builder role access to Firebase in the IAM console. Firebase also takes care of TLS and DNS once you link your custom domain, in this case from GoDaddy. https://retiari.us/posts/ir_in_la/ Thu, 07 Mar 2019 19:27:40 +0000 https://retiari.us/posts/ir_in_la/ I am excited to announce the latest opening on our Security Incident Response Team (SIRT), which is also our first LA-based role. Netflix is working hard to provide a wide range of content choices for our customers and with all that business activity comes some amount of work for the SIRT. In this role you will be exposed to a high operational tempo with a lot of growth and ambiguity. It is a very independent role where you will set the agenda for how SIRT can best add value in content related spaces. That is not to say you will be without support! You will be a core member of SIRT with peers in Los Gatos to help with training, tooling, and crisis management, as well as a growing Studio Information Security team next to you in LA. This role will be part of the SIRT on-call rotation so you will respond to issues across all lines of business while on call. We work hard to keep our on-call burden reasonable and SIRT members are expected to make progress against longer term objectives to ensure on-call stays engaging but not overwhelming. SIRT members tackle a range of projects from programmatic improvements to our response, training our team and others on how to respond better, developing tooling to automate crisis management and forensic activities, and other cross team contributions. https://retiari.us/posts/ti_for_vuln_mgmt/ Sat, 22 Sep 2018 16:59:48 +0000 https://retiari.us/posts/ti_for_vuln_mgmt/ Effective patch management remains as much a challenge today as it was a decade ago. The missing foundational piece is asset inventory - I would argue the biggest security gap most organizations have. Even assuming good inventory challenges remain for Threat Intelligence (TI) to identify new vulnerabilities in the supply chain - to include vendor products, open source libraries, software tool-chains, etc. There are frequent race conditions between what is known and the emerging unknown. Rapidly identifying those emerging unknowns - eg. zero days - is a key part of a threat intelligence program. Currently the one-stop-shop for vulnerability information is MITRE’s Common Vulnerabilities and Exposures (CVE) program. There are a number of smart dedicated people working on this tough problem space, but still the program has significant shortcomings as called out recently by the US House Energy and Commerce committee. Among the issues are that they provide at once too much information on inconsequential vulnerabilities causing fatigue, and too little information on those you do care about. This is compounded with a lack of structure or consistency to the reported information that is available. Perhaps most concerning is the significant delays between discovery and CVE release. For the most impactful vulnerabilities, those that provide remote execution and are ‘worm-able’ (allow for self propagation or mass scanning), the announcement of the vulnerability kicks off a race between defenders seeking to identify and remediate their systems, and attackers trying to weaponize the vulnerability then discover and exploit vulnerable systems. https://retiari.us/posts/attribution_and_threat_models/ Thu, 20 Sep 2018 16:58:39 +0000 https://retiari.us/posts/attribution_and_threat_models/ Thank you for joining us on article two of our series about the Future of Threat Intelligence at Netflix. Early in my career I liked to imagine myself locked in a battle of wits with a Karla of somesort. The reality is less literary archrival and more diverse set of ne’er-do-wells, but equally challenging. Absent a specific villian, what is the role of attribution in a corporate threat intelligence program? As with most things it turns on business value delivered - if knowing the name of the person targeting your systems helps reduce risk for your organization then discovering that information should be a goal of your intelligence program; however, this is seldom relevant to business risk reduction outside the rare cases where one might cooperate with law enforcement to target a particularly problematic fraud actor, or, as we will discuss later, in the world of physical and information security convergence. While law enforcement needs individual attribution, most often corporate security programs can work at a coarser granularity without losing impact. The goal is to better understand the intent and capabilities that comprise a threat. Attribution is valuable during an incident inasmuch as it provides leads on how to scope the intrusion or detect the attacker. https://retiari.us/posts/future_of_iocs/ Thu, 13 Sep 2018 16:56:00 +0000 https://retiari.us/posts/future_of_iocs/ Thank you for joining us for our series on Threat Intelligence. This is the first of our content posts, linked from our introductory post. The one constant in the information threat landscape is that it is always changing. Yet too often, when the term ‘threat intelligence’ is used, it is in reference to static Indicators Of Compromise (IOCs). Are these traditional indicators still relevant or useful today? How about in the near future? Most importantly, how adept is your Threat Intelligence (TI) program at dealing with these constant shifts in landscape? Can your current TI detect a black swan event, or a black swan hatching? IOCs or “Oh, I see” While there is still a great deal of commodity malware that can be found using file hashes, DNS names and IP addresses, many modern attacker techniques render these indicators unusable. For example, malware can be repacked thus changing the resulting hash, IP addresses are largely communal and interchangeable in today’s virtual hosting environments, and otherwise legitimate domains can be hijacked (or in the case of sites with user generated content, simply repurposed) to serve as malicious command and control infrastructure. A common shortcoming in TI programs is an outsized focus on malware. https://retiari.us/posts/future_of_intelligence/ Wed, 12 Sep 2018 16:54:43 +0000 https://retiari.us/posts/future_of_intelligence/ UPDATE: I am pleased to announce that Chris Cochran has joined the team to lead our intelligence program. He will be posting the rest of the articles in this series and I will link over to his work from here! I have been chatting with Mary Landesman, our intelligence lead, about how best to derrive business value from threat intelligence in Netflix’s unique culture and modern technology stack. Threat intelligence can be a slippery concept. While we have seen it work well, we have also seen it struggle to keep up with changing technologies and observed uneven return on investment across industries and organizations. We came up with a bunch of questions and a few promising theories. I will share those in a series of posts in the hope of drawing out debate and new ideas. Mary is looking towards retirement! So we are looking for the next generation intelligence leader to continue our momentum for Netflix, and the broader community. If that sounds interesting please get in touch: amaestretti@netflix.com Thanks to Mary for the insights and rewrites. Some of the topics (I will add links as they post): The evolution of IOCs Attribution Intelligence for Vulnerability Management https://retiari.us/posts/geekwire/ Fri, 15 Jun 2018 12:56:35 -0700 https://retiari.us/posts/geekwire/ Inside the private event where Microsoft, Google, Salesforce and other rivals share security secrets. https://retiari.us/talks/first_2018_podcast/ Thu, 07 Jun 2018 12:56:35 -0700 https://retiari.us/talks/first_2018_podcast/ A podcast for FIRST 2018 on how the Netflix SIRT operates. https://retiari.us/posts/a-learning-security-organization/ Wed, 30 May 2018 22:51:59 +0000 https://retiari.us/posts/a-learning-security-organization/ A Learning Security Organization When I was six my family spent a year in France as part of a Fulbright exchange. I dragged my parents to every castle I could. I loved to climb around the fortifications and explore the armory museums that were often attached. I recall my father pointing out a particularly imposing suite of full plate armor, with a tiny hole in the left breast plate. This was likely from target practice as I don’t think that armor was ever intended for the field, but the lesson was clear - technology can quickly swing the balance between defense and offense. Complexity and change are dominant factors today - If anything can be relied upon in the modern world it is that nothing can be relied upon for long. In The Fifth Discipline Peter Senge outlines an organization whose product is learning. He posits that such a ‘learning organization’ is the best model to handle change and complexity, particularly compared with more rigid industrial approaches to knowledge work. Complexity and change are certainly hallmarks of the technology sector, and security, so how could we build a security organization around the principles of learning? Senge’s five disciplines are: Personal Mastery, Mental Models, Shared Vision, Team Learning and Systems Thinking. https://retiari.us/posts/socless/ Thu, 26 Apr 2018 16:25:49 +0000 https://retiari.us/posts/socless/ A SOCless Detection Program at Netflix I am excited to share that we are investing in greater detection capabilities as part of the SIRT mission. There are a number of existing detection efforts across Netflix security teams. This is an opportunity to enable those efforts, while creating stronger alignment between detection and response. Of course this being Netflix our culture and our tech stack loom large in our consideration of how to expand our detection program. We want to avoid the traditional pitfalls and optimizes for our overall security approach. The last thing we want is a bunch of lame alerts creating busy work for a large standing SOC. Required reading for anyone interested in this area are Ryan McGeehan’s Lessons Learned in Detection Engineerin*g and theAlerting and Detection Strategy work from Dane Stuckey. I have borrowed liberally from their efforts. There are many ways to break this down, but I have settled on the following: Within these categories, this is what a mature program looks like: Data Sources One needs to find out about existing security relevant data as well as create it. This includes data discovery, deployment of instrumentation like eBPF or OSQuery, and working with application teams around logging best practices. https://retiari.us/posts/oncall_devs/ Mon, 04 Dec 2017 12:02:31 +0000 https://retiari.us/posts/oncall_devs/ In hiring for my team I sometimes get questions about on call. I wanted to share some context on how I view on call for our team vs a dedicated engineering team. To the question of whether a role is a responder role or a developer role: It is both. I feel strongly that successful engineers solve their own problems so I am looking for responder-developers that can play both positions. In the larger more traditional SOC/SIRT teams they will often carve off engineering as a specialization mostly to ease hiring as they scale. The downside is the dedicated tools team can loose touch with the operational aspects of the role and start building things the ops team doesn’t need, even as they get more and more mired in requirements documentation and program oversight. The Netflix model, whether in security or building the UI, is that every engineer owns their program from start to finish; so you develop the requirements, you decide what to work, you build it and you operate it. Everyone is ‘on call’ and this aligns the incentives to write good clean functional code; when it breaks you get paged. I know many folks coming out of a SOC/SIRT role at a more traditional company are looking to leave on call behind, but that is mostly due to the type of on call they had to endure. https://retiari.us/posts/rsa2017_recap/ Fri, 17 Nov 2017 12:07:25 +0000 https://retiari.us/posts/rsa2017_recap/ This year I attended Bsides SF, AGC’s tech summit, a threat sharing group meeting, Google’s security talks and a bit of RSA; along with various coffee’s, dinners and drinks. I presented on eBPF at Bsides as well as at a threat sharing group (APT SIG) and received some great feedback. I also hosted a P2P session on IR in the Public Cloud that was well attended and I thought the discussion went well. The ecosystem of conferences that have sprung up in RSA’s orbit is impressive. While I could not bring myself to get any closer to the expo floor than the overwatch position from the entrance level, I did get to chat with investors, founders, and folks like me on the buy side in more civilized venues around the city. I wanted to take a moment to organize my thoughts and share some of the trends. Always a good roadmap to the security space from Momentum Partners here. Artificial Intelligence AI is the new, ML which is the new Big Data. While there are some very reasonable applications of ML and Big Data to security, calling any of them AI is a big stretch. We asked a couple different VCs about their reaction to companies claiming to solve security problem X with AI. https://retiari.us/posts/memory_forensics_cloud_containers/ Wed, 11 Oct 2017 12:59:25 +0000 https://retiari.us/posts/memory_forensics_cloud_containers/ At Netflix my targets are primarily Linux VMs running in a microservices environment. I rarely have to pursue criminal prosecution but often want to figure out what is happening. Much of the work in memory forensics has targeted Windows systems and forensically sound capture methodologies that pull the entire system memory. This focus makes sense given the attack surface that user systems represent; however, given my goals I believe that userland memory acquisition and targeted analysis of memory at process granularity could yield more efficient results. Microservices, CI/CD deployments of immutable and ephemeral instances, the trend to containers, and various hardening efforts in the kernel, confer advantages on the defender that we must leverage to keep up with the scale and velocity of our environment. Another area that is not well developed is when target processes on the edge manage their own memory, as many of today’s popular languages do. I am curious what folks are doing in this space, so I wanted to mention some trends I see and solicit feedback: Desktops vs Datacenters vs Microservices User systems are the most difficult to secure. Particularly general compute devices operated by a technical workforce that needs full access to their systems, and will install all manner of tooling from around the web to accomplish their jobs. https://retiari.us/posts/uservice_for_disk_forensics/ Wed, 04 Oct 2017 12:55:22 +0000 https://retiari.us/posts/uservice_for_disk_forensics/ This is a follow-on to my previous post here. In the last few months I have had the chance to talk to a number of other companies and gotten a lot of great inbound response to the post about disk forensics in the modern era. To summarize: “Our goal is to move towards a streaming model for disk forensics by updating various existing tools that are built around the concept of imaging a single machine and creating a file. In a cloud environment we need to image multiple (even hundreds) of machines and process all those images into meaningful event data. Acquisition is fairly painless via API, but then what to do with the resulting snapshots?” While we were hoping for a cloud agnostic solution it seems we will need to solve this locally first and then seek areas to reuse code and integrate our systems in the future. We continue to approach this problem by decoupling acquisition, processing and analysis. We also hope to implement this in a scalable manner, ideally leaning into serverless techniques to minimize infrastructure maintenance. Acquisition Google is investing in DFTimewolf as part of the log2timeline/Plaso, along with GRR and Rekall. I think the Rekall agent architecture is the way to go. https://retiari.us/talks/aws_forensics_reinvent/ Mon, 07 Aug 2017 12:56:35 -0700 https://retiari.us/talks/aws_forensics_reinvent/ A talk from Re:Invent 2017 on how to look for evidence on disk in AWS. https://retiari.us/posts/disk-forensics-pipeline/ Mon, 22 May 2017 12:05:48 +0000 https://retiari.us/posts/disk-forensics-pipeline/ Last week we hosted colleagues from Google, AirBnB and Adobe to work on a joint effort to improve our disk forensics capability. Our goal is to move towards a streaming model for disk forensics by updating various existing tools that are built around the concept of imaging a single machine and creating a file. In a cloud environment we need to image multiple (even hundreds) of machines and process all those images into meaningful event data. Acquisition is fairly painless via API, but then what to do with the resulting snapshots? We are working towards a cloud agnostic streaming service for disk forensics. To get there we are currently focusing on acquisition in AWS and then sending those snapshots through Plaso for processing, pushing results into ElasticSearch, which is then fronted by Timesketch for analysis. I filed a number of issues with Timesketch to move us towards this, and we are working with the Plaso maintainers as they do some refactoring as well, but even now, with a few hacks, we can compose a system that achieves our goals. Acquisition Currently we have a set of boto scripts that enables us (Netflix SIRT) to capture snapshots of a group of instances within AWS. https://retiari.us/talks/aws_hunting_sans/ Fri, 07 Apr 2017 12:56:35 -0700 https://retiari.us/talks/aws_hunting_sans/ A talk from SANS Threat Hunting Summit 2017 on how to look for adversary activity in AWS. https://retiari.us/talks/ebpf_bsidessf/ Tue, 07 Feb 2017 12:56:35 -0700 https://retiari.us/talks/ebpf_bsidessf/ A talk from BsidesSF 2017 on how to use extended Berkley Packet Filter (eBPF) to monitor linux workloads. https://retiari.us/posts/fixing-ir/ Fri, 16 Sep 2016 12:11:02 +0000 https://retiari.us/posts/fixing-ir/ What makes detecting and responding to malicious activity on information systems so difficult? Incident Response and investigation can be incredibly intellectually rewarding. You get to puzzle together what transpired and take action to defend your systems, while working against a wily and adaptive adversary. On a bad day, however, you will find that most of the puzzle pieces are missing and there is no wily adversary present, only a deluge of false positive security events being happily emitted by an expensive PRNG called a SIEM. The core issues are volume of activity on, coupled with the complexity and change of, the systems you are defending. Traditional IT management compounds these issues rendering effective response impossible. DevOps, microservices and cloud infrastructure, in solving for scalability and development velocity, also offer solutions to these core security problems, and have the potential to Make IR Great Again. IR allows you to exercise a wide range of security skills. The diversity of events that one encounters ensures a varied and engaging workflow that continues to challenge. It takes not only a keen intellect and technical skill, but a body of experience from which to draw. Often there is no runbook, so you cobble together inferences from similar past cases and drive on. https://retiari.us/posts/apparmor-as-auditd/ Thu, 25 Aug 2016 00:24:23 +0000 https://retiari.us/posts/apparmor-as-auditd/ When we in security first approached our performance team with the bright idea to monitor all the syscalls, we were met with distinctly different views of the project. Whereas security was imagining hackers weeping outside the gates of our impenetrable systems; performance saw a specific dollar cost for monitoring that had better add value. At Netflix’s scale even a small performance improvement translates directly into large savings. As we dug into the call flow with Brendan Gregg and his set of really useful performance tools, we found that AppArmor is already inspecting system calls by default in Ubuntu. We already use AppArmor to secure various executables and are expanding its use. Also it turns out to be hard to remove from Ubuntu, so at this point so we had a sunk cost from performance’s perspective working in our favor. The challenge was could we replicate the functionality we got from the k/auditd framework with AppArmor? There are a few design choices working against us here: 1) AppArmor works on a whitelist principal, only allowing certain actions, 2) AppArmor is focused on enforcement of a policy based on a file path to a specific executable. With audit we want to ‘allow all’ (non-blocking) then audit certain things, and we want to do this globally not per executable file. https://retiari.us/posts/auditd-for-network-monitoring/ Mon, 22 Aug 2016 00:24:23 +0000 https://retiari.us/posts/auditd-for-network-monitoring/ Short of full PCAP it would be nice to capture metadata on network connections to and from an instance. In AWS EC2 it is difficult to perform network monitoring using traditional appliances as inter-instance traffic is not exposed to customers. You can get VPC flow logs which are useful; however, it is still desirable to have both a network and a host perspective on this activity. Both are valuable. VPC logs are difficult to corrupt from the host, and the host has more context on the executables that initiated the communication. So our goal is to collect metadata on network connections, including the remote address and the local process, then correlate that with VPC logs. Let’s see if we can pull this off with auditd. By monitoring at the system call level this technique should be difficult to subvert. Other methods might include alternate system call monitoring technologies, or various network utilities (tcpdump, iptraf, etc). Using nftables (iptables) logging could work… but onward with auditd. Curling an Address If you want to try this, set up auditd (apt-get install auditd on Ubuntu) to monitor successful socket() system calls using auditctl (reference this tutorial from RedHat). The first step for process communication is to allocate via socket(int domain, int type, int protocol) which results in an entry like this: https://retiari.us/posts/what-to-monitor-on-instance/ Wed, 17 Aug 2016 00:24:23 +0000 https://retiari.us/posts/what-to-monitor-on-instance/ I think of security relevant endpoint data within three domains: disk, memory and network. There are plenty of other taxonomies one could use. Disk, is generally addressed with File Integrity Monitoring (FIM). Memory, I am really talking about process execution, so anything that is exec’d and what it subsequently does (ie syscalls). Network, data transiting through external interfaces. Of course all three domains overlap - network configurations are stored on disk, sockets are opened with system calls, etc. There are also other useful optics like user identity and access that span these primitives. To cover these areas on our production systems we look to the Linux audit framework. Audit provides two types of rule; file and system call. With these two primitives any other type of desired monitoring can be built. In fact ‘file’ is simply a wrapper on the open() syscall. In addition to system call monitoring, Linux system logging functions provide pre-built enrichments that correlate and present low level data in a more relevant format. For example auth.log should be captured, rather than trying to rebuild those events from syscalls yourself. The logging landscape in Linux is rather convoluted and evolving (ex. systemd-journald), so your configurations are likely to be distribution specific. https://retiari.us/posts/dfir_aws/ Thu, 11 Aug 2016 00:24:23 +0000 https://retiari.us/posts/dfir_aws/ I previously discussed some of the challenges and opportunities of working in a public cloud when it comes to forensic response. Since you don’t own the datacenter, traditional tactics of grabbing boxes and imaging drives with write-blockers don’t apply. The goal is still to capture an unaltered image of both the disk and live memory, preferably without any interaction with the target system. Network traffic is another useful optic, and in traditional models can be captured out-of-band without disturbing the target through taps. The reality is it is often inefficient to answer your questions, especially fleet-wide, by capturing full live memory and performing analysis on it, in which case you might consider live response capabilities as well to query the running system. The danger is that by interacting with the system you might alter the evidence or tip off the adversary. Ideally you will have already logged relevant activity and stored it off target. All of these optics are valuable, and their value compounds when combined. In deception it is difficult to control multiple channels; so for example, while the OS might lie, it is hard for that lie to stand up to correlation across network data. In AWS the snapshot API for EBS-backed instances is an excellent tool. https://retiari.us/posts/security_monitoring_cloud/ Mon, 08 Aug 2016 00:24:23 +0000 https://retiari.us/posts/security_monitoring_cloud/ What should security monitoring look like in an AWS EC2 environment? Consider an EC2 environment running standard Linux AMIs with two use cases: detection and response. Detection is best supported by streams of real time events emitted by the endpoints. You can run signatures, heuristics and ML on these streams to detect suspicious conditions and trigger automated response: block, bounce, increase level of monitoring, etc. Response generally involves a human who wants to ask questions about the entire fleet, a subset or a single instance. With public cloud there is a new layer of relevant information, in our case from CloudTrail, CloudWatch, VPC flow logs, and other AWS sources. More on those later; we also need traditional system-level monitoring of instances from our side of the shared responsibility model. For detection ‘pull’ is not ideal. Many solutions rely on polling on some interval which results in a window between pulls. An attacker that is aware of your logging then has an opportunity to cover their tracks. For detection I prefer push or event driven solutions. A system event occurs, a log entry is created and immediately sent off-instance to remote analysis and storage. The drawback is limited context available at the moment of generation with which to determine relevance. https://retiari.us/posts/aws_ops_rig/ Mon, 01 Aug 2016 12:46:56 +0000 https://retiari.us/posts/aws_ops_rig/ Every responder needs a solid platform from which to operate. Something fast, secure and pre-loaded with relevant tools. Conducting incident response in a public cloud is the same as traditional IR but different. The public cloud is after all just other people’s servers; however, the ‘other people’s’ part changes things up a bit. You can’t roll into Amazon’s data center and grab a disk out of the rack, but Amazon provides an API for that. The concept they have put forward is ‘shared responsibility’ in which they handle the hypervisor and below, and above that is on you. We are moving towards a serverless approach to forensic response leveraging lambda functions and Amazon APIs; however, there is always a need for a solid ops box stocked with tools and enough permissions to get the job done. Design Goals: AWS EC2 native - leverage AWS APIs; minimize external dependencies - can’t rely on much in a crisis. Ephemeral instances - Spin up fresh, clean and quick; and kill them off once needed data is persisted to S3. Tools for response on Debian/Ubuntu AMIs with headless server (no GUI). Use corporate services day-to-day, but provide fallback options if they are down.