What I learned from Malcolm Gladwell

Do you ever get the feeling that you should be doing something else? Do you wonder what you would do if you had a year to do research without having to worry about getting paid? I’ve always thought that academic sabbaticals would be great for everyone, even people who don’t work in academia. Over the last year I have had a rare opportunity to try different jobs, research some of my interests, and basically take time to figure out what I want to do next.

After leaving my last log management startup, I spent several months in a fog trying to figure out what to do next without doing much at all. I spent some time working with a couple of startups that had nothing to do with log management, but I felt like I was wasting both my time and the knowledge that I had developed over the previous ten years. Also, I felt like I knew too little about the other technology at the time; it felt like I was starting all over again. It just wasn’t fun or interesting.

During this time I started reading the book Outliers by Malcolm Gladwell. In his book Gladwell examines the factors that contribute to high levels of success for people. He examines how some people become very successful – from professional hockey players, to musicians, to technology tycoons (like Steve Jobs and Bill Gates). The book shows that these people weren’t overnight sensations, but rather that it took many years of work, a passion for what they were doing, good timing, and lucky breaks along the way to become successful and become an ‘Outlier’.

All this got me to thinking about my own situation. To be most successful at what I was doing, I needed to focus on my knowledge base and the things about which I was passionate. This took me down the path of researching the opportunities in log management and the direction technology was heading. Finally, I hoped luck and good timing would be on my side.

My conclusion was that log management is a very hot field, and I had spent over ten years helping to develop it. It was where I needed to be.

I have spent the past nine months doing research and talking to friends and former colleagues who work with logs. I have found that there are many opportunities for innovative software companies working with log data. It’s exciting to see that in the large enterprise there has been a tremendous shift in the IT operations staff and an increased focus on log data. Governance, Risk, and Compliance have really pushed security and IT operations staff to focus on collecting and analyzing corporate log data. It started with the SIEM vendors collecting security logs. It has moved to log management vendors expanding to network and operating system log data.

My self-induced sabbatical is almost over. Thank you Malcolm Gladwell!

It has been a few months since my last post. I have been busy working on my next startup idea. I plan to announce what I am doing over the next couple of weeks.

Log Analysis a top 10 strategic technology for 2010, says Gartner

According to Gartner 2010 should be a good year for vendors in the Log Management market.  Gartner lists “Security – Activity Monitoring” as one of their top ten strategic technologies for 2010.

Gartner’s top ten states, “Information security professionals face the challenge of detecting malicious activity in a constant stream of discrete events that are usually associated with an authorized user and are generated from multiple network, system and application sources. At the same time, security departments are facing increasing demands for ever-greater log analysis and reporting to support audit requirements.”

It’s nice to see that more attention is going toward Log Management and SEIM, finally.  I remember about five years ago, while I was at LogLogic, we had a meeting with a Gartner analyst who told us there was ‘no market for log management’.  I disagreed, and I’m glad we listened to our customers more than the analyst.

Indexing vs. Normalization of logs

Recently a friend asked me whether he should normalize or index logs for faster reporting.  My response was that it depends on who is interpreting the reports.  I suggested he use indexing if the logs are coming from a single application and his users understand the log message format.  However, if the logs come from many different applications or have complicated message formats, then normalization is the quickest and easiest method.  In most cases the best method is to use both indexed and normalized log data if it is an option.
Continue reading

Quality Log Data is the Foundation of Log Management

LM Hierarchy of Needs - step1

The foundation of the Log Management Hierarchy of Needs is for the applications and devices to produce quality log data. Continue reading

Log Management Hierarchy of Needs

Log Management has a hierarchy of needs similar to Maslow’s famous hierarchy.  An organization needs to strategize and automate their basic logging tasks before they can focus on the complicated, solutions such as correlation and analysis. Continue reading

6 Reasons I Hate Logs

hatelogs

Ok, Hate might be too strong of a word.  But there are a lot of times I can be very frustrated in working with Logs. Continue reading

6 Reasons I Love Logs

The “I Love Logs” was one of the marketing slogans that we used when I was at LogLogic and they still use it today.  At an early age I was taught not to love anything that can’t love you back.  Besides, I’m a man and men don’t throw around the ‘L’ word very often.  So I just like logs.  Here are some of the reasons that I really like logs:

Continue reading