IANS Log Management Webcast

I was able to sit in on the IANS “Navagating the Data Stream without Boiling the Ocean” webcast presented by fellow IANS faculty member Dr. Anton Chuvakin, and included very knowledgeable panelists like Paul Melson, Michael Buley, Paul Borchardt. 

Anton started things off with a level set review of his Log Maturity Curve to help viewers understand where in the maturity model they might be so that they could frame questions most appropriately.

From my perspective webcasts with such a broad focus (Log Management and SIEM) and so many participants it becomes hard to focus or capture all the relevant points (a point I need to remember for my own use).  Some of the key information I was able to retain was as follows:

1. Log Management is used initially to create an enterprise repository for search facilitating both Compliance, IT Operations and Incident Response Searches, but has really become a staple in SIEM deployment architecture in order to help intelligently process information and scale SIEM deployments. 

2. Certain organizations might have 6+ years of log retention requirements the most common log retention period seems to be about 1 year for most logs.  To me that seems reasonable from both a storage costing perspective and an investigative perspective.

My favorite point was made by Paul Melson (Priority Health) as he hinted at how his usage of these systems has evolved over the years and has rotated through use-cases in both security and compliance.  Listening closely to Paul you could start to here his story, but he really deserves some dedicated time to fully articulate how he has help Priority Health evolve with Log Management and SIEM.  His first “external” users basically built search query so really only needed log management, but over time they have evolved to look at very advanced compliance and security use-cases.  Paul made a special point to highlight that he only sends information to his SIEM from the Log Management engine that has meaning for him.  It has to be information that meets a defined Use-Case, either something for correlation, statistical monitoring, workflow, etc.  All other data is retained for search purposes in the Log Management system.  Perfect!

The event was underwritten by ArcSight and while I personally think they would have been better served by   expansion on their customer testimonial than the marketing representatives slides, the marketing team did provide some basic thoughts on what they think a Log Management system must do for you as part of their closing statements.

According to ArcSight Marketing your Log Management System Must:

1. Help you with Security, IT Operations and Compliance Use-Cases.

2. It must be able to collect data from everything (including custom applications)

3. Be able to analyze structured/unstructured data.

4. Be fast collection, provide detailed and fast analysis and still provide efficient storage

5. It must normalize data (not just parse data) into a event types (authenticate versus log-in login, log in, etc)

6. It must provide audit quality log collection (data integrity, availability)

7. Provide Pre-packaged content (reports, etc)

8. Flexible options for storage ranging from economic to robust long term storage.

9. Real time alerting (Not necessarily workflow, but at least notification)

10. Bi-directional integration with SIEM (send/receive events/searches/etc).

There is an obvious spin on these above mentioned requirements, but yes I’d agree that your Log Management better be able to:

1. Eat anything (Omnivore)
2. Regurgitate specific meals on demand (Report)
3. Alert you to tummy aches (Notify)
4. Turn calories into energy (Basic analysis capabilities and Intelligent Forwarding Capabilities)
5. Intelligently communicate with your SIEM (Bi-directional feeds, searches).

I’d like to thank IANS for letting me listen in, Anton for facilitating the discussion and each of the panelists for providing their input!

Comments

[Steve]

Two thoughts on this post.

First, I am always troubled by the term normalize and it''s use as a base requirement for SIEM tools. Normalization makes it easier for vendors (and indireclty end users) to access and make use of the received data. But what does it reallly mean? If you transform it from it's orginal state (say into CEF) it fallls afoul of the compliance regs so if you do translate, you end up keeping multiple copies of the log. Having a normalization scheme also can lead to a least common denominator problem, and you get these Rule GUIs that have wonderful demonstrations but once you want to write something real you end up having to code. to be able to manipulate the data at the level required.

Yet not having any normalization means then when you author a rule, it is very specific to log as there is no data mapping. Rather like Assembler was to a 4GL. Powerful, fast but not real flexible and a bear to reuse. So it always seems In these discussions with vendors about normalization it always seems to end up being exactly what the vendor does -- no more no less. As a common requirement however not all that useful.

The second thought was your requirement for the log management layer to communicate to the SIEM layer. I assume you are using the Gartner definition of SIEM which is really the convergence of SIM and SEM. I would argue that ESM is not really a SIEM -- it is event management/correlation engine -- a SEM not a SIEM. Darn good one too if you can afford it. Logger on the other hand is firmly in the SIM category so to me logger + ESM == SIEM

[Rocky DeStefano]

@steve thank you for taking the time for such a fantastic response. I do want to state that the assertions made were by the vendor and not myself. I've learned a lot about how many vendors solve this problem and have respect for most of their methods. I usually trivialize log management into eating/regurgitating logs, and therefore don't always respect the "normalization" mandate, I think it is more important to normalize in a SIEM.

BTW you have a fantastic grasp of the differences in these systems. Personally, I've given up on the differences between SIM, SEM and SIEM. To me at this point in their evolution they are either Log Management or SIEM. I would say the Logger + ArcSight ESM does more than equal SIEM (becuase the platform actually extends way beyond traditional security on both technologies). SIM and SEM as platforms by themselves are mostly outdated or at best they are lost within with market they want to compete in.

Like you say, normalization varies greatly by the vendor. I will say that with very few exceptions the vendors who fail to normalize also have harder time scaling and making the technology easy to use (at least on SIEM). Yes you must retain multiple "copies" of the data, but if you are compressing it and leveraging the information more appropriately does it really matter? Most of these systems store their "cooked" data in a format that actually is smaller than the original log file which is amazing given some of the context that can be added to the original data set.

With regards to your statements about "correlation" rules being difficult to write I'm not going to disagree, they can be nasty at times but if you're writing rules about specific events, then you're probably not leveraging the system in the most effective manner.

I'd love the opportunity to talk with you further on your thoughts, please keep them coming!