Big Data Analytics for Security


An Evolution Beyond Security Information & Event Management

Limitations of SIEM

Depending on which company or startup we speak with, Security Information & Event Management (SIEM) is either dead or will live on forever. Quite different answers. In our minds, Big Data Analytics represents an evolution – not revolution – beyond the aggregation, alerts, and response facilitated by a classic SIEM solution. Big Data approaches differ from SIEM in two key ways: 1) unstructured data is acceptable, and 2) huge datasets are no longer a challenge. Of course, #1 and #2 resulted from new technologies we've spoken about before, which were created for purposes other than security.

Enterprises now realize that complete prevention of security incidents is impossible. Instead, there must be an increased focus on timely detection and response. Breaches WILL HAPPEN – so find them and contain them quickly. Both classic SIEM and Big Data approaches are compatible with this mindset and seek to unlock value through the aggregation and analysis of events generated by disparate systems. **The problem is that SIEM promised the world but under delivered. **Verizon's 2013 Data Breach Investigations Report provides an indication of this, noting that only about 1% of data breaches were discovered through log review. This is due to tools being configured only for certain use cases, monitoring teams being understaffed and undertrained, and the sheer volume of activity, among other reasons.

More directly, Gartner found that many organizations have "genuinely outgrown their SQL-based SIEM and moved to Hadoop-based systems." This has been publicly confirmed by CISOs at companies such as IDT and Jeffries & Company at the 2014 RSA Conference (see TechTarget write-up), who found that existing SIEM solutions simply failed from a performance standpoint under Big Data volume.

What must enterprises do now?

To start, existing SIEM solutions still form part of a security monitoring and analytics capability. The events collected and normalized by an existing SIEM must feed into a security-focused environment for Big Data analytics. This environment will also aggregate security events from solutions such as Data Loss Prevention (DLP) and Identity & Access Management (IAM), as well as data sources related to vulnerabilities, threats, and asset inventories (e.g., business processes, applications, databases, and hosts). Enterprises must start their Big Data Security Analytics focus now. A few points to remember:

  • Start Small: build environments along focused use cases
  • Educate: select/hire initial team members and encourage training of both Big Data technologies and data science concepts
  • Remember Quality: the principle "garbage in, garbage out" still applies to the data being aggregated
  • Integrate and Centralize: communicate with technology teams and business areas in order to gain and automate access to data
  • Secure: remember that access controls and other security concepts still apply
  • Report: determine how insights can be easily reported to management

Where do we go from here?

Stay tuned for Part II, where we will talk about machine learning, actively hunting for attackers, and relevant vendor products.

comments powered by Disqus