1. Cyber Pattern Discovery using Linked Data Analysis November 12, 2014 Page 1 © Hortonworks Inc. 2014 Webinar with Hortonworks and Sqrrl 2. Webinar with Hortonworks and Sqrrl Cyber Pattern Discovery using Linked Data Analysis November 12, 2014 Page 2 © Hortonworks Inc. 2014 Joe Travaglini Director of Products Sqrrl John Kreisa VP Strategic Marketing Hortonworks 3. The Modern Data Architecture Page 3 © Hortonworks Inc. 2014 Hortonworks. We do Hadoop. John Kreisa, VP Strategic Marketing Hortonworks 4. Agenda • Apache Hadoop and a Modern Data Architecture • Security in a comprehensive data management platform • Security Analytics using (Big) Cybersecurity Data • Case study: Internal network breach Page 4 © Hortonworks Inc. 2014 5. Our Mission: Power your Modern Data Architecture with HDP and Enterprise Apache Hadoop Who we are June 2011: Original 24 architects, developers, operators of Hadoop from Yahoo! June 2014: An enterprise software company with 420+ Employees Our model Innovate and deliver Apache Hadoop as a complete enterprise data platform completely in the open, backed by a world class support organization Key Partners Page 5 © Hortonworks Inc. 2014 6. Why a Modern Data Architecture? Business Analytics LIMITATIONS Silos & Expensive Single Purpose DATA SYSTEM APPLICATIONS Custom Applications Page 6 © Hortonworks Inc. 2014 Packaged Applications RDBMS EDW MPP MDA: Key Drivers 1. Leverage new types of data 2. IT optimization 3. Enable a data lake GOALS • Extend new data sets across existing data platforms • Common data platform, multiple processing engines • Batch, interactive and real time on a single data platform EXISTING Systems Clickstream Web &Social Geoloca9on Sensor & Machine Server Logs Unstructured SOURCES 7. A Modern Data Architecture Includes Hadoop Page 7 © Hortonworks Inc. 2014 Hadoop compliments and enhances existing technologies Common data set, multiple applications • Optionally land all data in a single cluster • Batch, interactive & real-time use cases • Support multi-tenant access, processing & segmentation of data YARN: Architectural center of Hadoop • Consistent security, governance & operations • Ecosystem applications certified " by Hortonworks to run natively in Hadoop SOURCES EXISTING Systems Clickstream Web &Social Batch Interactive Real-Time HDFS (Hadoop Distributed File System) Geoloca9on Sensor & Machine Server Logs Unstructured DATA SYSTEM APPLICATIONS Business Analytics Custom Applications Packaged Applications RDBMS EDW MPP YARN: Data Operating System 1 ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° N 8. Unlock New Applications from New Types of Data INDUSTRY USE CASE Sentiment Page 8 © Hortonworks Inc. 2014 & Web Clickstream & Behavior Machine & Sensor Geographic Server Logs Structured & Unstructured Financial Services New Account Risk Screens ✔ ✔ Trading Risk ✔ Insurance Underwriting ✔ ✔ ✔ Telecom Call Detail Records (CDR) ✔ ✔ Infrastructure Investment ✔ ✔ Real-time Bandwidth Allocation ✔ ✔ ✔ Retail 360° View of the Customer ✔ ✔ ✔ Localized, Personalized Promotions ✔ Website Optimization ✔ Manufacturing Supply Chain and Logistics ✔ Assembly Line Quality Assurance ✔ Crowd-sourced Quality Assurance ✔ Healthcare Use Genomic Data in Medial Trials ✔ ✔ ✔ Monitor Patient Vitals in Real-Time ✔ ✔ Pharmaceuticals Recruit and Retain Patients for Drug Trials ✔ ✔ Improve Prescription Adherence ✔ ✔ ✔ ✔ Oil & Gas Unify Exploration & Production Data ✔ ✔ ✔ ✔ Monitor Rig Safety in Real-Time ✔ ✔ ✔ Government ETL Offload/Federal Budgetary Pressures ✔ ✔ Sentiment Analysis for Government Programs ✔ 9. Break Down Silos with a Security Data Lake SCALE Page 9 © Hortonworks Inc. 2014 SCOPE Unlocking the Data Lake RDBMS MPP EDW • Data Lake Enabled by YARN • Single data repository, shared infrastructure • Multiple security apps accessing all the data • Enable a shift from reactive to proactive interactions • Gain new insight across the entire enterprise New Analytic Apps or IT Optimization HDP 2.1 Governance & Integration Security Operations Data Access YARN Data Management 10. Big Data is Changing Cyber Security “By 2016, more than 25 percent of global firms will adopt big data analytics for at least one security and fraud detection use case, up from current eight percent.” – Gartner Cyber Security report Feb 2014 Gartner recommendations • Align security capabilities in a holistic security strategy tailored to the threats and risks • Target a single architecture to collect, index, normalize, analyze and share all information • Organizations should look for profile accounts, users or other entities, and look for anomalous transactions against those profiles Page 10 © Hortonworks Inc. 2014 11. How Can Big Data Analytics Help Cyber Security? • To prioritize threats, vulnerabilities, and attacks • To control endpoints and mobile connections /devices • To prevent insecure devices from accessing secure systems • To provide intelligence about the threat land- scape • To reduce false positives Page 11 © Hortonworks Inc. 2014 12. Page 12 © Hortonworks Inc. 2014 Securely explore your data CYBER PATTERN DISCOVERY USING LINKED DATA ANALYSIS A Big Data Solution with Hortonworks and Sqrrl Joe Travaglini, Director of Products, Sqrrl © 2014 Sqrrl Data, Inc. | All Rights Reserved 13. Who We Are Page 13 © Hortonworks Inc. 2014 © 2014 Sqrrl Data, Inc. | All Rights Reserved 14. Agenda • Security Analytics using (Big) Cybersecurity Data • Dealing with the new security dilemma • Why Hadoop and HDP are the perfect fit • The ‘Linked Data’ Approach • Case study: internal network breach • Overview of scenario • Data modeling with Sqrrl • Visual, contextual research and analysis Page 14 © Hortonworks Inc. 2014 © 2014 Sqrrl Data, Inc. | All Rights Reserved 15. The Numbers Don’t Lie 229 87% 90% $12.7M Page 15 © Hortonworks Inc. 2014 Source: Mandiant Source: Verizon Source: Verizon Source: Ponemon © 2014 Sqrrl Data, Inc. | All Rights Reserved 16. Targeted Attacks Have Changed the Game Page 16 © Hortonworks Inc. 2014 © 2014 Sqrrl Data, Inc. | All Rights Reserved Source: Battery Ventures 17. What Does This Mean For Us? Dissolution of the Secure Perimeter • You’ve been breached. Deal with it. • Empower the investigator • Research and respond: better, faster, smarter • It’s all about speed to understanding Page 17 © Hortonworks Inc. 2014 © 2014 Sqrrl Data, Inc. | All Rights Reserved 18. The Security Data Dilemma Detecting attacks requires more (i.e. BIG) data Page 18 © Hortonworks Inc. 2014 © 2014 Sqrrl Data, Inc. | All Rights Reserved But your tools can’t handle the big data wave So attackers are spilling in 19. A Modern Data Architecture Hortonworks and Sqrrl Solution • Hortonworks Data Platform at the core • Sqrrl Enterprise stack at the app layer Hadoop enables us to look at data differently Page 19 © Hortonworks Inc. 2014 © 2014 Sqrrl Data, Inc. | All Rights Reserved 20. Sqrrl Enterprise Architecture Interface Processing Data Model Data Storage Page 20 © Hortonworks Inc. 2014 Visualization / API ML + Anomaly Detection Query Engine Bulk/Graph Processing Raw Events Linked Data Model HDFS + Accumulo Commodity Hardware © 2014 Sqrrl Data, Inc. | All Rights Reserved Physical Security Audit Cryptography Labeling + Policy 21. Big Data Transformed Data Sources Linked Contextual Knowledge Analysis Email Page 21 © Hortonworks Inc. 2014 © 2014 Sqrrl Data, Inc. | All Rights Reserved Security Data VPN FW Network Data Proxy NetFlow Application Data HR USB 22. Linked Data Analysis Adding structure to the noise Page 22 © Hortonworks Inc. 2014 © 2014 Sqrrl Data, Inc. | All Rights Reserved 23. Page 23 © Hortonworks Inc. 2014 Case Study: Compromised Network © 2014 Sqrrl Data, Inc. | All Rights Reserved 24. Breach Detection Scenario Page 24 © Hortonworks Inc. 2014 © 2014 Sqrrl Data, Inc. | All Rights Reserved © 2014 Sqrrl Data, Inc. | All Rights Reserved 25. Case Study Model Data Sources Page 25 © Hortonworks Inc. 2014 © 2014 Sqrrl Data, Inc. | All Rights Reserved Linked Meta Model Users Hosts login flow login DNS records Netflow Host logs Database logs External Alerts 26. Case Study Example Mapping Page 26 © Hortonworks Inc. 2014 © 2014 Sqrrl Data, Inc. | All Rights Reserved Netflow Records startTime endTime sourceIP destIP sourcePort destPort protocol tcpFlags bytesIn bytesOut 10/22/14 8:58 10/22/14 8:58 10.0.2.15 192.168.0.123 37051 139 TCP ...RS. 100 3355 10/22/14 8:45 10/22/14 8:45 10.0.2.15 192.168.0.6 0 3328 ICMP ...... 40 100 10/22/14 8:59 10/22/14 8:59 192.168.0.119 10.0.2.15 139 60071 TCP .A..S. 46 351 10.0.2.15 192.168. 0.123 Class=Flow, totalBytes = 3455 192.168. Class=Flow, 0.6 totalBytes = 140 27. Case Study Example Data Page 27 © Hortonworks Inc. 2014 © 2014 Sqrrl Data, Inc. | All Rights Reserved 28. Investigation Process 1. Set the Stage 2. Enable Search Page 28 © Hortonworks Inc. 2014 and Discovery © 2014 Sqrrl Data, Inc. | All Rights Reserved 3. Automate Analysis • Define the security-centric entity/ relationship model • Extract and maintain the model • Visually navigate assets and actors in the network • Drill down to the raw data seeding the model • Use behavioral analytics to build expectations of ‘normal’ • Flag entities as potentially ‘abnormal’ and sniff them out 29. Page 29 © Hortonworks Inc. 2014 Visualizing the Threat © 2014 Sqrrl Data, Inc. | All Rights Reserved 30. Page 30 © Hortonworks Inc. 2014 © 2014 Sqrrl Data, Inc. | All Rights Reserved 31. Page 31 © Hortonworks Inc. 2014 Thanks! © 2014 Sqrrl Data, Inc. | All Rights Reserved Joe Travaglini Director of Products, Sqrrl Data, Inc. @joe_travaglini
[email protected] http://www.sqrrl.com