Today Ellen Nakashima of The Washington Post published an article about DHS USCERT, NSA and Telecommunications providers collaborating to monitor Civilian Agency Internet traffic using DHS’s planned Einstein 3 tool to help defend these civilian government entities.  The article correctly illustrates that NSA has the expertise and tools like Tutelage to know more about the context of the attacks.  It also states that DHS has the authorization to monitor using Einstein (enforced by the TIC program).  If you’ll remember a while back I talked about Trusted Internet Connection (TIC) and its role in consolidating Internet points of presence and providing chokepoints to monitor and defend for the government.
For reference see:  http://blog.decurity.com/index.php/dec_template/more/dhs_einstein_tic_overview/    and   http://blog.decurity.com/index.php/dec_template/more/dhs_blog_round_table/

In short, TIC mandated government agencies to meet very stringent requirements in order to become a TICAP (provider) or use pre-approved TICAP’s (Telecom or other Agency) for all Internet traffic.  The monitoring capabilities of these TIC’s is referenced in my earlier posts, but let’s just say its EVERYTHING.  Not that I’m complaining, from a capabilities perspective I think NSA and Cyber Command should be making the most out of this information to help protect the government and as Richard Bejtlich speculates eventually “.com” .  NSA has the expertise and intelligence data while DHS has the authorization to monitor, the framework to force everyone to play (TIC) and a toolset that is evolving (Einstein v2 is still being rolled out, v3 is in development) On a side note, I do have to wonder why the government isn’t using more capable tools like NetWitness or Solera in conjunction with NSA tools and building a META SIEM to incorporate Intelligence feeds, but that’s a topic for a later post. 

My biggest question is this…. I wonder how US-CERT and NSA are going to collaborate more effectively -  Is Einstein raw data going to be handled by NSA, if so what’s the point of US-CERT in the future?

Should be interesting to see what happens once the cyber czar is appointed, from what I can tell his/her kingdom has already layed a very clear path forward, the czar may simply be along for the ride while NSA drives over everyone else.

Update 1: (3 July 2009; 0930 EDT) SIOBHAN GORMAN of The Wall Street Journal also has an article on this topic “Troubles Plague Cyberspy Defense” .  In this article takes more conservative approach in describing what is happening across government with regards to consolidated monitoring.  According to the article Einstein v3 will be updated/rebuilt to more closely align with NSA Tutelage and is at least 18 months out.  The idea is that it would start to develop full packet inspection capabilities (Like NetWitness, Solera and a few others).

My Notes:  If this perspective is more accurate it seems US-CERT would monitor using technology enabled by NSA, instead of NSA accomplishing the monitoring.  IMHO - From what I’ve seen certain executive layers at DHS have not enabled the US-CERT to be effective enough to actually function as a true analytical center, even though USCERT has some very good people capable of executing on that misson.  In fact, I’d go as far as to say DHS is at risk of losing key staff if they don’t figure out a better way to enable their team.  The place is known as a revolving door for a reason, the people they hire are very capable and motivated, the organization itself may not be best suited for that expertise and vision. 

Earlier this month The 451 Group released a Market Insight Service Impact Report about Decurity.  Wow.  We are flattered that The 451 Group (Nick Selby and his team) took the time to to understand what our team is doing at Decurity.  From the phone calls, tweets and emails I’ve received the readership of this report is pretty amazing.  It’s an honor to have a respected entity like the 451Group take notice of our baby.  Paul and I (and the entire Decurity team) are sincere in our appreciation of all of those that have taken the time to learn about our team and our vision.  I can’t even begin to express my appreciation for the trust and partnership that our clients have extended to Decurity over the last two years.  We have amazing customers, partners and friends and we hope that we can continue to earn the trust and respect for the next several years! 

I will say this - while we are very proud of what we’ve accomplished so far, there is a lot more coming from Decurity in the very near future - keep an eye (or RSS reader) our for more about us! 

UPDATE:  We’ve secured reprint rights and will post the document on our soon to be updated webpage in the coming days.  Link to the 451 MIS Impact Report on Decurity:  451_Group_Decurity_IMPACT_REPORT_-_10_June_2009.pdf

Gartner 2009 SIEM Magic Quadrant - It’s pretty easy to digest - Everyone is in the leader quadrant or is a “leader” of some sort.  No vendor has any “real” negatives, they will all work in nearly any situation, everything is peachy.  You can register with many of the vendors to receive your copy of the Gartner Magic Quadrant for SIEM 2009 report.  A few of those links are provided here:  Q1, ArcSight, LogLogic , RSA

My obviously frustrated opinion:
If you are an executive, director, manager, technician or otherwise breath air and you use this report to make any sort of determination about SIEM vendor qualifications you should probably be seeking professional career advice.  This report is great at providing the SIEM vendors with additional fodder to confuse the sales prospects, but it does absolutely nothing for the reader seeking information about which SIEM might fit their environment.  This is the one time that I don’t blame the vendors, they have to play the game.  The game is to mitigate the effect of competitive “spin” based on the results of the report by the “leaders” listed within the report.  Along those lines I won’t use this post to congratulate or bash vendors as to their relative position in the quadrant because it’s just insane to justify anything about this report.

Every year, I review these SIEM reports and find myself hoping that the next issue will reveal something insightful or even slightly meaningful.  I scrub the “cautions” looking for anything that points to a material technical weakness in the technology and usually the most meaningful thing I find is a veiled “feature request” for some trivial item.  A review of the “strengths” of each organization shows the points to be at best highly subjective and usually just completely irrelevant.  I always leave frustrated, but hopeful that the next version will set things right. 

Gartner isn’t alone in it’s pursuit of mediocrity here.  Most (not all) of the “analyst” firms and industry magazines offer a strikingly similar lack of useful information in their reports.  Please note: this is in no way a personal attack on any author or company, it’s a rant against crappy information as a whole.  Over the years I’ve met most of the reviewers and they “seem to get it” in person, it’s just the nature of these “ranking without context” reports that simply kills the value of any insight the authors might have tried to present.  The 451Group and a few others have tried to buck that trend over the years and are making some progress, but despite their efforts the overall industry standard is still too watered down to be of any real value. 

I know I’m not endearing myself to the analyst community right now, and I expect certain vendors won’t appreciate what I’m saying but bear with me here.  I think we can make this better and everyone can benefit.  As an industry we must start expecting better from our information providers.  We need to provide specific feedback about what information these reports should provide in order to be meaningful.  I have tried to influence better context and more meaningful technical criteria through several older blog posts and through conversations with anyone that will listen.  I’ll step up my game and offer even more direct advice in the coming months - I’m just asking that everyone do the same.  Let’s encourage our information providers to pursue a higher standard. 

Maybe, next year… Hey I’m a Cubs fan - There is always next year! 

Recently, I initiated a test survey using Linkedin’s Polling feature.  The survey was a quick and dirty way to help me gain a bit more perspective as to the number of resources organizations are putting towards their SIEM Projects.  There are a ton of limitations to this type of survey, including the fact you only have 75 characters to present your question to the audience.  That said, I’m sure I could have worded my survey question and responses even more clearly.  Those limitations aside, the results were very consistent with my observations over the last several years.  Put simply it requires a lot of effort, even halfway through 2009, to run a successful SIEM Project. 

Over the last month 13 organizations (75%+ Large or Enterprise Organizations) have taken the time to respond directly to the LinkedIn poll.  Quite a few more people went further and emailed me with details, questions and observations.  Just using the survey results, nearly 70% indicated that they had 2 or more FTE’s dedicated to SIEM.  With an additional 15% responding simply “Not Enough”.  100% responded with at least 1 FTE. 

My take on this initial poll - Even with all the improvements SIEM vendors are making in their products, SIEM’s are still not “plug ‘n play” systems, you need dedicated resources (internal, consultant, partner, etc) to extract the maximum value from your SIEM.  Duh!, I’ve been saying that for years, but it’s nice to see others nodding their head every once in a while.

I’m working on a better series of questions and responses with a more clear focus on for the next set of survey’s and I’ll use a more robust mechanism to accomplish that goal.  I’ll be using SurveyMonkey to get a better perspective and more clear insight into all things SIEM (and Log Management).  Look for those survey’s to begin in early July with results and analysis to follow shortly thereafter.  If you have questions or observations you’d like me to ask to the world about Log Management or SIEM - leave me a comment or email/DM me!  Thanks.

So if you follow me on Facebook or Twitter you may have heard our family had a bit of excitement over the weekend.  My wife and two youngest children (2 and 7)  got stuck at the top of a ride at Busch Gardens due to a “technical malfunction”.  I know that mechanical and/or technical failures happen all the time at theme parks, but when it’s your family up there and you’re on the ground, it sucks.

Busch Gardens did everything right, they quickly informed everyone on the ride of the malfunction, asked them to stay calm and at the same time sent emergency responders up to the top of the ride to help get everyone off safely.  No running around crazy, no unnecessary escalations, no waiting on approvals, no idle hands… Everyone played their role.  It got me thinking about the obvious parallels in incident response (well parts of it at least)

The ride was designed with safety mechanisms including emergency exit and communication mechanisms.  The “owners” had procedures that were extremely well tested, communicated and executed by the “administrators”.  Everyone had their role, understood it and was authorized to just “do it” and it worked out.  Once completed, they accomplished the repair, tested the ride, re-tested it from another perspective and then once approved by management they put the ride back into production for the park visitors (“users”).  Sure the visitors had to wait a few minutes, but everyone was understanding once they had the right information made available to them.  Certainly, I’d prefer this sort of thing to never happen, but that’s unrealistic given all variables in place at a Theme park in Florida with millions of visitors.  I’m just happy everyone was safe and we were able to enjoy the rest of the day.    and then just when you think it’s over…

Not more than 20 minutes later we saw another ride fail.  The sky-ride (gondola) got stuck mid-ride for over 10 Minutes.  Luckily, we were not on that ride.  I’d have gotten a bit suspicious at that point . 

Actually, at that time we were on a train ride enjoying a peaceful ride through the park, pointing out animals to my two year old, when a grumpy Rhino tried to prove to the train that he was in control and decided to give it a little shove to encourage the train to keep moving along.  I’m not sure if it was a full moon, an everyday occurrence for the park or Murphy’s Law that caused all the excitement. 

It just goes to show you that you can’t predict what’s going to go wrong, just that something will go wrong - it always does.  We must prepare for as many types of Incidents as we can and enable our teams to react effectively, and expect that they will.  Obviously, a lot of pre-planning, risk assessment, exercise activities, documentation and training goes into the equation.  Everyone has to become involved, if a barely over minimum wage them park worker can be trained to play a role during an emergency, certainly we can figure out how to more effectively involve our “owners”, “administrators”, and management in our incident response activities. 

Ok, enough excitement for one evening I’m off to bed,  I can’t wait for next week’s cruise and the lessons that will bring..

Sara Peters at Information Week recently posted an article titled “SIEM Case Study: Israeli e-government ISP” In this article, Assaf Keren, information security manager at the Israeli e-government ISP Project (called “Tehila”) calls our attention to some very important details to consider when Implementing a SIEM.  Keren’s advice is that a successful SIEM implementation requires:

1. Detailed planning,
2. Fastidious attention to detail,
3. Superb communication between concerned parties
4. Attentive oversight of vendor activity.
Another Key Point from Mr. Keren - don’t outsource this “theory phase.”

Note:  I agree with Mr. Keren that the SIEM requirements have to be driven from within your organization.  However, I believe that expert external entities can and should help drive discussions and help extract and refine requirements from your team. Obviously, the expert external entity MUST NOT be from a Vendor or reseller of any SIEM Products.

Looking back over hundreds of SIEM deployments and seeing so many consistent decisions (or indecisions) that adversely affected the success of the SIEM I felt compelled to add a bit more context to augment the lessons Mr. Keren shared.

Overview:
1. It takes a village, building planners, city inspectors, etc:  Probably, the most important takeaway from this post is that you should take the necessary time to fully comprehend and vet your requirements, as well as decide on your service delivery model, gain consensus on that approach and have realistic expectations along the way.  SIEM failures are more often the fault of poor planning, moving tactically while ignoring the strategic nature of the project, or simply misaligned expectations rather than a pure technology failure. 

2. Know what you are going to do with the Output before you make it Input:  It is tough to make sense (and therefore derive any value) out of billons of events by adding even more events to be evaluated into the mix.  Intelligent Collection, Analysis, Escalation and Remediation and workflow efforts defined before you start (and refined along the way) means that you’ll have a better idea what to do with the information your presented and a much higher chance for success in both end-user usage of the system and aligning that usage of the SIEM with the needs of your organization’s security or compliance program.

3. Purchase the “right” technology, but do it incrementally:  Quite candidly some SIEM products should be avoided at all costs, however it should be noted that most of them can at least be used to help you meet some very basic requirements.  Consider your business and technical requirements over a 24-month period, but only purchase what is necessary to deliver based on the next 6 months of work you expect to get accomplished.  The system needs to be flexible to support all of those upcoming needs, but there is no need to spend money today to support tasks you won’t even consider touching for over 12 months. 

A successful SIEM tool supporting your organization’s Security and/or Compliance needs really boils down to some very simple concepts:

Define Success
Have a strategic vision about how you want your Security Operation and/or Compliance Program to run and use that to help define requirements for how the SIEM (and Log Management) tools will provide input or drive workflow related to that Program.  Involve all the stakeholders early and keep them engaged along the way!

• If your rationale for buying a SIEM is PCI Compliance, STOP.

• If your rationale for investing in SIEM is to provide “x”,”y” and “z” data sets to business unit “a” and “b” and initiating workflow for your SOC; and you understand the event sources necessary/business logic to compile the data sets for each customer; and you fully understand how they intend to use that information the you are much closer to being ready to work with a SIEM.

Related Resources:
• SIEM: Basic Implementation Success Criteria
• SIEM: Before you Buy

Plan Accordingly
SIEM is not an overnight project, and yes even an Appliance-based SIEM’s require significant attention to work to their maximum potential for your organization.

• Gather requirements from all “stakeholders” Compliance, Legal, IT, Business Units, Security, Executive, everyone that will help you get information into the SIEM or receive information from the SIEM (or your service offering that leverages SIEM). 

• Define Event Sources based on end-user needs:  Security, IT Operations and Compliance teams all have distinct needs and therefore may require different event source information.  At a minimum they may require different “views” of similar information set available in the SIEM or Log Management Tool.  Ensure you have the proper information sets, logging at the right levels and the information is available in a logical and meaningful manner.

• End-User Requirements are the most valuable.  The more your team understands how your “customers” value the data and service offering the more you can benefit from the functionality of the SIEM. 

• Analytical and Workflow Requirements.  Security Analysts need to be able to quickly identify, analyze, prioritize and escalate the data with context in order for the SIEM to meet its most basic functions.  This functionality is not as common as you would think across different SIEM’s.  Be sure that the SIEM integrates with your workflow systems in an acceptable fashion.

Related Resources:
• SIEM: Best Practices in Collection

Vendor Selection
Now that you have your requirements documented and prioritized compare them against SIEM: Evaluation Criteria and refine them even further…

• Either partner with an expert that can tell you exactly why certain Vendors can not meet your needs (today/tomorrow) and compare those answers a n honest discussion with the vendor or invest in a Pilot in an effort to prove out ALL of your requirements (not just the top three.)
• Make sure you have data either directly from production event sources or a reasonably similar source.  If you use combined Log Management and SIEM architecture, make sure you can configure outbound events in a format the SIEM can comprehend for more than just Syslog events.  If the SIEM can natively handle ODBC but your architecture requires Log Management to be the Collection Tier and forward events to the SIEM – How does the LM reformat those events and how does the SIEM handle that data? 

• Customer Referrals are nice, but be careful.  I’ve seen this scenario too many times.  Victim asks a SIEM Reference Client about a key area of concern, say scalability and the reference client dutifully answers the questions with a resounding “Yes, the $VENDOR scales to meet my global organization’s amazing needs” in all the excitement it was overlooked that it takes 100+ systems to get there and oh yeah, by the way none of these SIEM systems can cross correlate information.  As your requirements are defined, build out testing plans if the requirement is that critical and test it prior to purchasing.

• Maximize your dollar.  Ensure the vendor is prepared to partner with you for the long haul, you both have a vested interest in the success of the program – make sure they are going to be there for you
• Find out the vendor’s fiscal period and plan your purchase accordingly.  Fiscal Quarter end and Fiscal Year end are great times to make deals (especially enterprise deals) with vendors.
• Purchase what you need not what you want.  If you don’t have a documented requirement that you can reasonably achieve in the next 6 months don’t buy it yet.  Conversely, don’t skimp on things you absolutely do need.  If you have a requirement to store 8 Billon events a day over a 10-year period and you expect to do that with local storage or even DAS, NAS.  Stop and rethink things a bit.

Focused Effort:
Ensure that you have dedicated enough time and energy to the success of your SIEM Effort.  If you are a large enterprise this is at least 2 FTE’s or an Expert Partner

Seriously, Requirements Gathering, Vendor Selection, Pilot, Implementation, Initial Operating Capability, Operational Refinements, Final Operating Capability (Formal Service Delivery), On-going Enhancements, Patches, Upgrades, Lab Testing, Additional Content Tuning, Expansion and the related Coordination, Planning, Execution, Oversight and Measurements is enough to keep an entire team busy.  Doing all of that within the framework of your overall Strategic Security Program and not just tactically solving issues as the “pop-up” on a daily basis is the key to success with SIEM and ultimately your entire security and/or compliance program.

Having the wrong team or not listening to the right team is about the same as not having resources at all.  Spend the time to ensure your SIEM team is baked into your Security/Compliance Program(s) so they can help you plan for today and tomorrow and save a lot of headaches in new hardware, storage or even total SIEM replacement.  If your not ready to dedicate the right Resources/Partner’s then you may be better off waiting and then introducing SIEM into your organization when the requirements, proposed solution and funding are more in line. 

Lifecycle Planning
This goes way beyond simple O&M tasks.  SIEM is part of your overall Security Program and as such need to stay in step with that Program.  Your SIEM Team (Partner) needs to be involved along the way to help ensure compatibility and/or flexibility as you evolve.  Service Delivery, Technology, Business and Compliance requirement changes and/or reprioritizations can all have a significant impact on the success or failure of the overall program.  The tighter the team is with the thought process around those upcoming changes the more likely your SIEM Program will meet your needs.

I just wanted to let everyone know that Verizon Business has published the 2009 Data Breach Report.  The breadth and depth of these reports are invaluable.  Since there are very few solid sources of this type of information the release of this report dominates the availability of the few brain cells I have remaining.

Press Release Here: http://www.verizonbusiness.com/products/security/risk/databreach/
Actual Report Here: http://www.verizonbusiness.com/resources/security/reports/2009_databreach_rp.pdf
Look for updates/comments from the authors/team at http://securityblog.verizonbusiness.com

From my first 5 minute glance at the report here are some of my favorite things:

Figure 31. Time Span of breach event by percent of breaches. This may be the best metric we as security professionals can look to improve.  Seeking to reduce the time to Incident Identification and Mitigation

Figure 32. Breach Discovery methods by percent of breaches.  Interesting observations about how things are detected, nearly 70% by third parties, only 7% by “active” internal teams.

Figure 34. Detective Controls by percent of breach victims.  System and Application Logs are KEY (don’t just rely on security devices).

Many of the recommendations seem brain dead simple so I won’t cover them here, nor will I go into the pseudo risk calculations or PCI “Compliance” at this time.  All in all a ton of food for thought in this report.  I’m going to wait to post more comprehensive notes on this report to allow it all to sink in a bit more.  Verizon obviously puts a lot of thought and effort into this report and I find myself spending hours dissecting it every time.  To my friends over at Verizon Business - Thanks again for the information!  Everyone else - I encourage you to take the time to review it thoroughly.

Hackers for Charity is Johnny Long’s new website and mission in life.  Saying that I applaud him on this effort is the biggest understatement I can make.  On a personal level I am very moved by his passion and commitment to server others, here and everywhere.  Johnny has taken his talents and applied them in ways that help so many people across the world.  Just thinking about what he is accomplishing motivates me to seek better out of myself.  Please do pop over to his site and find a way to help Johnny and his family on their upcoming year-long efforts in Uganda.  Equipment, Advise, Money - anything you can provide will help Johnny, his family and so many others in Uganda and across the world!

Rocky

Network World’s recent article provides additional evidence that Log Management and SIEM Vendors are still trying to evolve.

Recently, Log Management and SIEM vendors have spent a lot of time updating/fixing their products.  Over the past few months some vendors have quietly passed over other solutions in terms of market relevance and certainly the door has been opened to a whole bunch of upstarts trying to make a name for themselves.  While the majority of Log Management and SIEM business (and therefore product direction) is driven by compliance activities, I appreciate the forward movement towards enterprise security that many in the field are trying to make.  The initial execution on that product vision I’m seeing from many of the vendors this year is very welcome.  IMHO the entire space had gotten very stale with the big guys mainly focusing on compliance appliances or playing feature catch-up with one another.  Here’s my summary of what’s going on in SIEM and Log Management so far in 2009.

SOURCE Boston was an amazing experience for me.  A great deal of the thought leaders in our industry shared thoughtful insights and experiences.  I had the opportunity to really engage in a great many conversations with people like Raffy Marty, Ron Gula, Marcus Ranum, Peter Kuper, Amit Yoran,  Dov Yoran, and Jamie Fullerton the list could go on for days.  It renewed my energy to be around so many intelligent and security focused people.  Stacey Thayer and the entire SOURCE team should be very proud of this CON.  One presentation really stood out in my mind.  Hoff’s “The Frogs who desired a King” was easily one of the best presentations I’ve seen in years.  My quick summary of SOURCE and links to other reviews follow:

The following notional diagram provides some basic recommendations to consider when deploying and managing Log Management and SIEM systems together.  A well-maintained Log Management and SIEM deployment can significantly reduce the time to Incident Identification and really enhance your overall information security capability.  The diagram attempts to illustrate that all information from the Event Sources are processed through the appropriate Log Collection Mechanism and then forwarded to the Log Management System. The Log Management system eats, stores and can regurgitate everything put into it.  The Log Management Solution also can further refine the data set and forward only applicable events for analysis to the correlation engine (SIEM) through the use of intelligent “tagging” of events.  Overall data reduction is only part of the end goal, more importantly we want to ensure the right data is forwarded and evaluated so that we can gain from the overall efficiencies offered by the SIEM.  In short we’re ensuring the system has the correct information available to it so that it can respond to the questions you want to ask of it and reduce the garbage as much as possible.

Decurity’s new subscription offering works with your organization to understand your requirements and then supplies the necessary configurations and Log Management and SIEM data elements (Intelligent forwarding, correlation, reports, etc) to make this model work for you.

This week I was able to participate in the IANS Mid-Atlantic Information Security Forum in Washington DC.  It was a whirlwind of activity - from stepping off the plane and arriving just in time as the “Security Operations” session track with Marcus Ranum (which I’m honored to be a co-facilitator) was being introduced to dashing off to the airport yesterday afternoon every moment was consumed with interesting and important conversations about security operations and incident response.  In two days I was able to have solid conversations with folks like Chris Hoff, Peter Kuper, Nick Selby, Glen Sharlun, Aaron Turner, Ron Gula, Raffy Marty and Richard Bejtlich and so many more who really don’t like their names publicized

Decurity often has the opportunity to our customers find the right Log Management and/or SIEM solution.  We are honored that our customers trust us with that very important question so we wanted to take a moment and explain our requirements gathering/documentation process for vendor selection and hope that our explanation helps a few of more folks out there!  We also get asked by Vendors on how they can improve their products, but that’s a entirely different blog post.