SIEM Evaluation Criteria - Functionality Matrix

UPDATE: 10 June 2011 - Since DabbleDB is no more :( - I’ve moved the Matrix to Google Docs and made it more accessible (including download for your personal use).  Enjoy

I’ve been lucky enough that over the years I’ve been involved with hundreds of SIEM evaluations for customers, service providers, investors, acquiring organizations and of course the vendors themselves.  Now I’m not going to share the results of each of these reviews because 1) NDA’s and such and 2) they instantly become outdated as vendors merge/evolve but I did want to share some of the more basic questions I look for when I do an evaluation.  Listed are over 140 criteria and yes they do further break down into hundreds of other discreet questions and measurements, but this will give you a glimpse into what I look at when I break down SIEM Players.

 The list below is part of the database I keep on these vendors, obviously without the details.  I’ll keep updating it over the next few months and leave the link “live”.  Feel free to use this information or some subset of it in your own evaluations.  Remember - Define your Use-Cases first - then worry about scoring the criteria.  This is just an information collection mechanism and covers most use-cases, but the prioritizing of these entries will greatly depend on your usage.

Note:  If the embedded table does not function with your RSS reader I apologize, please refer to the source visiblerisk website for the entry:  http://www.visiblerisk.com/blog/2011/2/7/siem-evaluation-criteria-functionality-matrix.html 

About Rocky

Update on Jun 10, 2011 by Rocky DeStefano

10 June 2011:  I Finally updated and shared SIEM Evaluation Matrix to Google Docs so it’s more usable.  http://bit.ly/ffGxcu or directly via http://www.visiblerisk.com/blog/2011/2/7/siem-evaluation-criteria-functionality-matrix.html

Pushing Conversations

Last week there was talk about the lack of recent in-depth conversation occuring in blogs.  This was attributed to at least partially to twitter dumbing down the conversation into bite size increments and that overall the larger stories may not be getting explored in depth.  I certianly found myself in that cycle.  For me sometimes it is easier to have a quick conversation and reset my thermastat by releasing steam on twitter than it is to create a post for the blog.  The downside to that has been lack of in-depth public conversation because twitter more a “live” communications mechanism.  So this last week I focused on blogging and engaging in conversation and “wow” - what a series of conversations I was able to be involved with.  Here is a sample of the public conversations that occured.

Topic 1:  LogLogic

LogLogic announed last week about slashing price on SIEM (well actually SEM) I posted my thoughts about their move here.  Who knew the response I’d get.  Here’s a sample of how the conversation exploded via blogs over the last week. 

The Response (so far):

LogLogic was nice enough to email and call me to help me understand their thinking, they also crafted 3 blog posts on the subject (Part 1, Part 2 and Part 3). 

Paul Melson enhanced the conversation (“Twitter Killed the Blog Star” and “SIEM Market Conversation Continues” and now Q1Labs jumps into the fray with their perspective.  There were plenty of private conversations from other vendors, analysts, end-users that occured off-line - I’m hoping more of those will see the light of day soon!

 Count: 1 post (6 Blog Responses + comments)

Topic 2: Tetragon

Then the real fun started.  Gartner released their Magic Quadrant for SIEM and I had a little fun with it and then I had an absolute blast with my Tetragon post.

The Response (so far):

Chris Blask on the new AlienVault Blog responded to my First MQ post with his insightful perspective.

Rich Mogull and Mike Rothman of Securosis had some fun with my Tetragon post and did a great job adding even more perspective to how and why these “tetragon’s” get so much attention. 

Anton Chuvakin made reference to the posts here.

Q1Labs talks about the MQ any my post here.

Count: 2 Posts (4 Blog Responses + Comments and more coming)

I’d like to thank everyone that participated in the conversations (public and private), everyone who publicized the conversation via blog, re-tweeting or email.  Analysts, Vendors and End-Users everyone was involved over the last week!

As numerous and insightful as the public blog and twitter conversations were, the private DM, Skype, Email and Phone call were also insightful and at times absolutely hilarious.  I wish I could share some of them, but I’m hoping some those conversation will take the form of a blog post or 12 soon.  There are a lot of great ideas waiting to push these conversations even further.  All in all it was an amazing experience to jump in, start a conversation or two and see where passionate people take it. 

Now that I know more people are paying attention and willing to join the conversation I think I’ll do this more often :) 

-Rocky

Tetragon of Prestidigitation

I present to you the latest in analytical innovation.  The industry’s first completely transparent vendor evaluation system. The Visible Risk “Tetragon of Prestidigitation” (tm). Bask in it’s glory and simplistic brilliance.

Why worry about defining requirements? Dont stress about how to consider the impact of this technology when in specific use within your environment. The Tetragon knows the answer, it IS the answer. 

Obviously no analyst group wants their work misread or misinterpreted.  They put a ton of work into these offerings. We’re human and as such no matter what someone may want us to comprehend, we’re all going to read things from our own perspectives.  Sometimes our understanding and follow-up discussion enhances the conversation, sometimes we can be completely off the mark and are provided ourselves with an opportunity to learn. In the end as long as the conversation continues everyone wins.

Since I wrote the blog post about this years SIEM review(s) and the marketplace in general there have been a few dozen conversations that I’ve have the pleasure of participating in.  One in particular, rekindled a thought process in my head about maintaining perspective and context when reading reviewing publications of this sort and I thought I’d share the humor and story with you.

I’d like to present to you three hypothetical perspectives each reviewing the new “Tetragon of Prestidigitation” report.  The views are from The Veteran Information Security Professional, A Vendor and The Executive.  Certainly these do not represent all of the potential views but it is a broad enough sample to get the point across.  The point being that we need to seek to understand all of the perspective’s better and then push our thoughts on how to better collect and analysis the data.

Case 1: The Veteran Information Security Professional:

This realist wants to believe that products are getting better and that analysts get their “hands dirty” when providing information that will define which products they’re going to get stuck with over the next few years.   From years of experience they know that if the product isn’t in the upper right or leaders area of the review they’ll face a nearly impossible battle for project justification and funding.  These professionals are passionate, battle hardened and look at any publication with a deeply technical perview about what will work in the environments they support.  Many in this group don’t necessarily find enough “meat” in the report to make any specific determination about technologies presented so they have a tendancy to discount the report in general.  It becoms a necessary evil.  For those that fall into that camp (I did for a long time and still revert to that tendancy at times) I suggest a reading of this post to put into perspective the effort that goes into these type of reviews.  It isn’t trivial and there is a method to the madness. 

That said, the security professional’s interpretation of these types of reviews isn’t likely to change, perhaps it is time to add a few more dimensions than a 2D table can represent? My question to anyone that falls into this camp (or similar one) -  What are the questions and context that should be included in these reports moving forward?  How would you contribute to the conversation?  Provide input, Press buttons, Express yourself - I think you’ll be amazed at the response.

 How a Jaded InfoSec Professional might see the Tetragon of Prestidigitation:

Case 2: The Vendor

The Vendor’s perspective is perhaps the strangest of all involved.  Based on how certain elements of the community interprete these reports they have the most to lose and unless they are put into the “right” place very little to gain.  Of course as mush as the vendors complain about their relative ranking (assuming #2 or lower) they will all use these reviews as justification for market positioning in competitive situations.  Funny how you’ll see 20 vendors representing their positions in press releases following these reports each claiming a victory of sorts.  Either “we’re the best”, “we’re the future”, “we scale down”.  Propaganda is powerful marketing.  

Now what control do the vendors have in this situation?  In truth, very little.  Vendors get beat up by analysts, customers and of course other vendors.  Do I feel sorry for them? Nope, not even a little bit.  Do I respect them for putting everything on the line to try and provide a solution? Yes.  Personally, I think if some of these teams spent more of that mindshare that is dedicated to “spinning” the results into influencing a better process to begin with, we’d all be be better for the effort. 

Let’s look at how our example Vendor might see the Tetragon of Prestidigitation:

Case 3: The Executive:

The overwhelmed corporate executive is the target consumer of these reports.  Well, at least, the info graphic portion of the report.  A very select few will understand every detail, spend time looking at collateral documents, comparing them against known information and dissecting the body looking for nuggets the help differentiate the products as it might apply to the environment.  Instead they will rightfully leave that effort to their techncial staff.  On the other hand the executive team needs to make decisions that are going to allow the project to be successful and if they can find a way to reduce the time and pain associated with the product selecation and  purchase process then they’ve done a good thing for the business. 

From the executive’s perspective having hundreds of peer experiences aggregated into a single report or table seems like a gift from above.  It allows for focus and being the standard across the industry it also allows some level of protection. 

Every rational person knows that it isn’t wise to rely on any single piece of evidence in the decision making process, but yet it happens hundreds of times per year because the business has limited resources and a mandate to become compliant.  Management works with the best set of information available and makes the best possible decision based on what they have in front of them.  They have a sometimes defined mission, numerous competing priorities and a very limited set of resources.  The “Tetragon” provides them a shortcut in the conversation with the procurement team.  

How an executive (or procurement team) might view the “Tetragon of Prestidigitation”:

Summary:

Industry analysts put significant effort into the report and just as many disclaimers and qualifiers about how to use the reports.  We still are inclined to extract only what is relevant to our situation and that makes our lives easier (Technical, Vendor or Management).  In order to use the information presented in the “tetragon” report it needs to be analysed by your team in the context of your overall requirements, processes, your team and numerous other criteria. 

We all have our perspective, and some balance of time / motivation guiding us when we read these reports but it’s time to step back and put their value into perspective.  The “Tetragon” is a macro statement and your situation requires micro level details to be successful.  Certainly there are trends to consider, but in no way should these reports be the determining or even scope defining factor(s). 

Call for Action:

Throughout this post I started to highlight some simple ways to help push for better quality and more meaningful analysis across industry.  In short we need to speak up from each and every level (Tech, Management, Vendor, Executive and Analyst) and articulate clearly the information that would best enhance our perspective about the technologies, the use-cases and the market in general.  What else would you like to see in the Tetragon or similar reports?

The “Tetragon of Prestidigitation” is just my little way of reminding you to not settle for what there is today - always seek more!

Following-up:

I don’t beleive in defining a problem and not at least attempting to present the beginnings of a solution.  Soon, I’ll be posting Chapter 2 the follow-up to SIEM Evolution Chapter 1 which will help me articulate what I see as major trends or innovations in the SIEM industry.   Additionally, as soon as I can find the right mechanism for sharing the information I intend to post my SIEM evaluation checklist so that at a minimum my thoughts on the subject are provided to the world as a starting point and then you can help by re-defining the “right” questions to be asking (and expected responses) moving forward.

Thank you,

-Rocky

LogLogic discounts SIEM

Today LogLogic announced it is attempting to push the market towards a more cost effective solution for SEM/SIEM.  On one hand I get “it” and applaud the attempt to push price downward, but on the other hand the overall strategy seems to me, albeit without the benefit of the larger picture, ill-conceived, ill-timed and in my opinion it may devalues the company’s effort in that space especially when you consider the quote at the end of the press release (more on that later).

Positive Reflections on this announcement:

LogLogic is a almost exclusively a channel driven sales entity and the “channel play” with a move like this makes perfect sense.  It allows even more flexibility by the channel to sell downmarket, but I would have expected an announcement like this to coincide with a major channel or partner strategy announcement.  A refreshed service offering being made available through partners or a partnership with several managed services to deliver more value, etc.  Something that would represent a larger strategy is in play.  Maybe I missed it?

Help me maybe I’m missing something:

If the focus of the pricing strategy was log management, storage or just about any product outside of SIEM that might be considered a commodity within the IT world I’d understand the pricing movement.  To me SIEM is much more complex and as such requires continuous effort and is not ready for this pressure.  Right strategy, wrong product.  In my opinion they should have lowered price on the commodity (Log Management / IT Search).

Very often I’ve heard of Symantec, RSA (via EMC), Q1 (through Juniper/Enterasys), SenSage (through HP) and of course Mars (via Cisco) basically giving SIEM away for free as a bundle with other products or services, yet companies like Q1 and ArcSight continue to knock the ball out of the park at a “premium” rate (via direct or indirect sales). 

If the leaders in the space can compete at that level with the huge machines in this environment how does LogLogic affect the market at all with this announcement?  Without the context of a larger product strategy or channel/partner offering this new pricing effort seems like a missed opportunity at best and a last ditch effort at worst.  (I hope it is not the latter).

As a company LogLogic seems to be doing well (43% year over year growth) but I think this SIEM pricing strategy confirms that the growth is relative to the strengths of their Log Management product.

Another Possibility:

Ok so there is another thread running through my head about this subject.  Which is that this pricing move simply reflects the reality of their effective price book.  Depending on the spin doctor this might be taken a few different ways:

1. SIEM has a ton of features that most users can’t quite implement because the vendors haven’t made it easy enough - so why charge a premium?

- or -

2. We don’t have the ability to do “x” or “y” so we can’t charge a premium and compete in that space.

Given what I know about the LogLogic team, it actually makes sense that this was the thought process.  They are sharp and motivated to do the right thing.  Even so, it doesn’t detract from some of the things I presented above as potential missed opportunities.  If this is true then does it also mean when (if) those advanced features are developed the pricing model gets complicated?

Misguided Quote:

The quote at the end of the press release is even more haphazard than the announcement itself.  (Paraphasing the quote by Mike Davis: “We see 80+% of the needs of the SIEM space being satisfied by log management. SEM fulfills a further 10+% of those needs” to be that quote is dangerously misguided on two fronts.

1. This quote signifies the lack of market understanding, a lack of appreciation of SIEM or maybe just an overestimation of Log Management. Perhaps the quote is relative to only the qualities of a SEM (versus SIEM), but that doesn’t reflect the reality of the market today (Log Management and SIEM) and serves no real purpose other than to discount the value of SIEM/SEM.
2. The quote also serves to invalidate a product that LogLogic invested in with the purchase of ExaProtect and continues to attempt to sell as a competitive offering in the space.  It seems odd to invest in a technology that only meets 10% of a market need (still leaving 10% unfulfilled).  It is my opinion that the quote does more damage than service to Log Logic.

Summary:

I don’t disagree that there should be pricing rationalization that occurs across the market, but I’m not convinced LogLogic took the best approach (or perhaps I simply don’t understand the strategy well enough at this point).  I’m hoping they’ll take the time to enlighten me!

Reference:  http://loglogic.com/news/news-releases/2010/05/loglogic-slashes-sem-prices/

Note:  I apologize I wasn’t able to get input from LogLogic in time for this initial post but I will update the comments of this post with any lessons learned from LogLogic.   I really do want to understand and not just seem like I’m beating them (or anybody) down.

2010 Gartner MQ for SIEM

The 2010 Gartner Magic Quandrant for Security Information and Event Management. 

I took some liberties with the MQ in order to segment it out the information in the report for my own analysis.  Simple geographical/mathmatical groupings were applied and a few notes added.

The cluser of providers in the middle was not unexpected as I have a similar group in my head as I think about this.  Having an outlyers like RSA to me means that RSA’s vision resonated very well with Gartner.  As it should, however I’d expect that to be meaningful once they get to that execution.  I can’t find the rationale for them escaping the “Clump”.  I did use RSA’s relative position as a measuring stick to divide the clump and segment out a couple of “Maybe’s”

The “Clump”: Symantec, IBM, CA, LogLogic, Novel, SenSage, LogRhythm, Trustwave, NetIQ, TriGeo, NetForensics, eIQnetworks

The “Maybes”: Nitro and Q1 Labs

The “Others” Quest, Prism, Tenable, LogMatrix

Next Year:  AlienVault and several others might add more to the middle of the page.  Splunk was left off of SIEM this year for a reason and I applaud Gartner for doing so!

Note:  I do beleive that the The Strengths and Cautions tell a little better story this year than previous years. 

Trends and Subplots:

1. The re-emergance of NetIQ & SenSage as a SIEM (and SenSage’s continued relationship with HP) is interesting.

2. The “Clump” is a great way to illustrate that there are way to many products that are virtually indistinguishable to the market.

3. I’d had expected more from Q1 via the Enterasys and Juniper OEM deals.  I also expected Tenable to fair a little better, but Security Center 4 was just released so I understand it’s standing based on a review of the older technology.

4. Enterprise Players (RSA, IBM, CA, Symantec) seperating out functionality into various products instead of having it all within the SIEM.  I think these guys are ranked a bit too high and right.  RSA with Archer will be a great improvement, someday.  Even more importantly RSA has a chance with EMC to really do well in virtualization if they focus on it.  We’ll see what happends.

5. The exists a Vast Ocean between ArcSight and the rest of the field.  Out of the “Clump or Maybe’s” I’d expect 1 or 2 to rise over the next year and begin to challenge more directly for #2 in the space.  This will require innovation on either User Anomaly/Behavior (beyond IAM), System Config (FIM?), Network Content (beyond netflow), and of course ease of use for Security Use-Cases such as Threat Monitoring and Dynamic Updates!

6. The inability to execute in a viable manner with applications, hosts attributes, vulnerability information or even get data out of the system still plagues many vendors in this space - whether or not they admit to it.  Read between the lines of the report to figure out who this affects.

You should read the report.  Each of the vendors will have copies linked off their webpages and with minimal registration you should have it in a matter of seconds.  Enjoy!

About Rocky