Comments for LogAdvisor blog http://logadvisor.com Mon, 23 Nov 2009 22:45:10 +0000 hourly 1 http://wordpress.org/?v=3.2.1 Comment on Indexing vs. Normalization of logs by Tweets that mention Indexing vs. Normalization of logs - LogAdvisor -- Topsy.com http://logadvisor.com/indexing-vs-normalization-of-logs/#comment-12 Tweets that mention Indexing vs. Normalization of logs - LogAdvisor -- Topsy.com Mon, 23 Nov 2009 22:45:10 +0000 http://www.logadvisor.com/?p=178#comment-12 [...] This post was mentioned on Twitter by File Replication Pro and Dimitri McKay, Thomas Grabowski. Thomas Grabowski said: Should I index or normalize log data? Both - http://bit.ly/5U54KX with Splunk , Loglogic, or Arcsight [...] [...] This post was mentioned on Twitter by File Replication Pro and Dimitri McKay, Thomas Grabowski. Thomas Grabowski said: Should I index or normalize log data? Both – http://bit.ly/5U54KX with Splunk , Loglogic, or Arcsight [...]

]]> Comment on Indexing vs. Normalization of logs by Michael Wilde http://logadvisor.com/indexing-vs-normalization-of-logs/#comment-11 Michael Wilde Mon, 23 Nov 2009 18:31:09 +0000 http://www.logadvisor.com/?p=178#comment-11 Thanks for mentioning Splunk. It should be noted (in case you weren't aware), Splunk indexes the full text of any message, organizes it by the time at which it occurred--making its indexing far more useful than Lucene--and then uniquely does the "normalization" at search time. Splunk calls this "search-time field extraction". The best answer to your question of normalize vs. index is both. Indexing provides a far faster and more broad search than traditional schema/database solutions--but to answer the full question, you really have to have the fields and structure at some point in time. Splunk just saves the user massive amounts of time and offers insane flexibility by "first eat, then organize" Michael Wilde Splunk Ninja http://splunkninja.com Thanks for mentioning Splunk. It should be noted (in case you weren’t aware), Splunk indexes the full text of any message, organizes it by the time at which it occurred–making its indexing far more useful than Lucene–and then uniquely does the “normalization” at search time. Splunk calls this “search-time field extraction”. The best answer to your question of normalize vs. index is both. Indexing provides a far faster and more broad search than traditional schema/database solutions–but to answer the full question, you really have to have the fields and structure at some point in time. Splunk just saves the user massive amounts of time and offers insane flexibility by “first eat, then organize”

Michael Wilde
Splunk Ninja
http://splunkninja.com

]]> Comment on 6 Reasons I Hate Logs by Polprav http://logadvisor.com/6-reasons-i-hate-logs/#comment-9 Polprav Thu, 22 Oct 2009 03:21:29 +0000 http://www.logadvisor.com/?p=136#comment-9 Hello from Russia! Can I quote a post in your blog with the link to you? Hello from Russia!
Can I quote a post in your blog with the link to you?

]]> Comment on Log Management Hierarchy of Needs by Mike Epplin http://logadvisor.com/log-management-hierarchy-of-needs/#comment-10 Mike Epplin Tue, 20 Oct 2009 13:02:00 +0000 http://www.logadvisor.com/?p=163#comment-10 I like the approach you have developed here. I think its actually a common approach that people try to take, but nothing is documented around it. Devices and applications producing quality log data is a goal, but to often we have very little control over what log data the devices produce, and it takes you down the path of tuning the device or application wherever possible, so it becomes a recursive call, as does much of the pyramid. Once you get to the top 3 levels, they become very enterprise centric, depending on what regulations and compliance initiatives are native to your organization. It's the alphabet soup problem we all know and love. As I stated previously, I think its a faily accurate representation of the log management process, but each step becomes a recursive process, and once you get to the top of the mountain, you need to revisit base camp and climb up again. Reliable and secure collection is another goal, but to often we deal with protocols like syslog which aren't always secure in transmission, or, being UDP based, reliable. The I like the approach you have developed here. I think its actually a common approach that people try to take, but nothing is documented around it. Devices and applications producing quality log data is a goal, but to often we have very little control over what log data the devices produce, and it takes you down the path of tuning the device or application wherever possible, so it becomes a recursive call, as does much of the pyramid.

Once you get to the top 3 levels, they become very enterprise centric, depending on what regulations and compliance initiatives are native to your organization. It’s the alphabet soup problem we all know and love.

As I stated previously, I think its a faily accurate representation of the log management process, but each step becomes a recursive process, and once you get to the top of the mountain, you need to revisit base camp and climb up again.

Reliable and secure collection is another goal, but to often we deal with protocols like syslog which aren’t always secure in transmission, or, being UDP based, reliable.

The

]]> Comment on 6 Reasons I Love Logs by Martin LaFica http://logadvisor.com/6-reasons-i-love-logs/#comment-3 Martin LaFica Tue, 13 Oct 2009 21:06:40 +0000 http://www.logadvisor.com/?p=116#comment-3 Nothing is perfect. Log formats change, standards are weak at best and collection methods are sometimes complicated but the fact remains that their is value for every IT shop in the tsunami of log data. I remember one occasion with a prospective client a few years ago when he started running around the office rounding up his peers to show them the value in simple PIX logs. It is this type of reaction that drives me to help my clients evidence compliance and solve their problems using log data. Today logs come from the edge of the network all the way to the center from revenue impacting applications and databases. The value and risk is to high to ignore logs. I LOVE LOGS! Nothing is perfect. Log formats change, standards are weak at best and collection methods are sometimes complicated but the fact remains that their is value for every IT shop in the tsunami of log data. I remember one occasion with a prospective client a few years ago when he started running around the office rounding up his peers to show them the value in simple PIX logs. It is this type of reaction that drives me to help my clients evidence compliance and solve their problems using log data. Today logs come from the edge of the network all the way to the center from revenue impacting applications and databases. The value and risk is to high to ignore logs. I LOVE LOGS!

]]> Comment on 6 Reasons I Hate Logs by grabowskit http://logadvisor.com/6-reasons-i-hate-logs/#comment-8 grabowskit Tue, 13 Oct 2009 18:18:06 +0000 http://www.logadvisor.com/?p=136#comment-8 Tomas, Yes, I believe there could be one, but I don't see that happening for a long time (at least 5-10+ years). All it would take is some large clients (like the telecom industry) demanding it or some large vendors like Cisco and Microsoft making it a priority. I also agree with your point on the complexity of managing large deployments of logging systems in today's products. Depth and Breadth seem to be the first priorities in most logging solutions, hopefully ease-of-use and ease-of-administration will become a higher priority for some of the larger vendors now. Tomas,

Yes, I believe there could be one, but I don’t see that happening for a long time (at least 5-10+ years). All it would take is some large clients (like the telecom industry) demanding it or some large vendors like Cisco and Microsoft making it a priority.

I also agree with your point on the complexity of managing large deployments of logging systems in today’s products. Depth and Breadth seem to be the first priorities in most logging solutions, hopefully ease-of-use and ease-of-administration will become a higher priority for some of the larger vendors now.

]]> Comment on 6 Reasons I Hate Logs by grabowskit http://logadvisor.com/6-reasons-i-hate-logs/#comment-7 grabowskit Tue, 13 Oct 2009 18:03:46 +0000 http://www.logadvisor.com/?p=136#comment-7 Marty, As you know, I <a href="http://www.logadvisor.com/logging/6-reasons-i-love-logs/" rel="nofollow">Love Logs</a> too and when used right the benefits far outweigh the costs. But, that said, log data could be <em>soooo</em> much more valuable if there were more consistency, standardization, documentation, and understanding from the vendors that create it. Marty,
As you know, I Love Logs too and when used right the benefits far outweigh the costs. But, that said, log data could be soooo much more valuable if there were more consistency, standardization, documentation, and understanding from the vendors that create it.

]]> Comment on 6 Reasons I Hate Logs by Tomas Carlsson http://logadvisor.com/6-reasons-i-hate-logs/#comment-6 Tomas Carlsson Tue, 13 Oct 2009 11:16:26 +0000 http://www.logadvisor.com/?p=136#comment-6 Hi, some comments, 1. What is your long term prediction for this ? Will there ever be ONE log protocol ? Will there ever be a product or system that actually can normalize "all" logs ? 6. Management and maintenance of log systems and log infrastructure are interesting topics. I find it strange that the management capabilities of log management systems often are like firewall management was 10-15 years ago i.e. useless. It is impossible to have control of your logs if the systems you use to manage them can not be managed efficiently. So this is where I hope the most interesting development will be over the next years. /TC Hi,
some comments,

1.
What is your long term prediction for this ? Will there ever be ONE log protocol ? Will there ever be a product or system that actually can normalize “all” logs ?

6.
Management and maintenance of log systems and
log infrastructure are interesting topics. I find it
strange that the management capabilities of log management systems often are like firewall management was 10-15 years ago i.e. useless. It is impossible to have control of your logs if the systems you use to manage them can not be managed efficiently. So this is where I hope the most interesting development will be over the next years.

/TC

]]> Comment on 6 Reasons I Hate Logs by Martin LaFica http://logadvisor.com/6-reasons-i-hate-logs/#comment-5 Martin LaFica Mon, 12 Oct 2009 23:59:01 +0000 http://www.logadvisor.com/?p=136#comment-5 Nothing is perfect. Log formats change and collection methods are sometimes complicated but the fact remains that their is value for every IT shop in the tsunami of log data. I remember one occaision with a prosective client a few years ago when he started running around the office rounding up his peers to show them the value in simple PIX logs. It I this type of reaction that drives me to help my clients evidence their problems using log data. Today logs come from the edge of the network all the way to the center from revenue impacting applications and databases. The value and risk is to high to ignore logs. I LOVE LOGS! Nothing is perfect. Log formats change and collection methods are sometimes complicated but the fact remains that their is value for every IT shop in the tsunami of log data. I remember one occaision with a prosective client a few years ago when he started running around the office rounding up his peers to show them the value in simple PIX logs. It I this type of reaction that drives me to help my clients evidence their problems using log data. Today logs come from the edge of the network all the way to the center from revenue impacting applications and databases. The value and risk is to high to ignore logs. I LOVE LOGS!

]]> Comment on 6 Reasons I Hate Logs by 6 Reasons I Love Logs - LogAdvisor http://logadvisor.com/6-reasons-i-hate-logs/#comment-4 6 Reasons I Love Logs - LogAdvisor Mon, 12 Oct 2009 18:34:25 +0000 http://www.logadvisor.com/?p=136#comment-4 [...] me why you Love Logs  …or tell me why you hate logs here Share and [...] [...] me why you Love Logs  …or tell me why you hate logs here Share and [...]

]]>