Posts by Rapid7

2 min IT Ops

Revisiting the Past with Logentries and Syslog

When I was younger I was always amazed by my grandfather. He would revel me in stories about how, when growing up, not only did he not have a car or television, but he had to walk up hill “both ways” to get to school – without shoes! And here I am today more or less saying the same thing about the late 70’s and early 80’s when client-server changeover that occurred during that time and we all had servers sitting in our closets. Setting up syslog [https://logentries.com/doc/about-syslog/] back in

2 min IT Ops

AWS CloudWatch Logs - Making Innovation Easier and Cheaper

Last week AWS made an exciting announcement at their NYC Summit, which I believe is a big step in helping to reduce the cost of log management in the cloud [https://logentries.com/doc/log-management/] for end users and vendors alike – good news all around! What was announced? Amazon announced ‘CloudWatch Logs’ which essentially allows you to send log data from your EC2 instances into CloudWatch for storage and some rudimentary analysis. How does it work? CloudWatch logs works by collecting y

2 min Logentries

Logs to Metrics to JSON to Geckoboard

The world of log management [https://logentries.com/doc/log-management/] and application performance monitoring [/2014/04/apm-is-not-enough/]are evolving and there is no longer a big distinction between the two. Up to even a year ago, many people in IT viewed logs as simply a means of identifying and tracking issues once they have occurred within their system. It was rare that people were using their logs for more than basic application event tracking and watching for exceptions. But over the

3 min IT Ops

TCP or UDP for Logging – Which One to Use?

When sending data over the Internet, we want to be sure that data has arrived safely and is delivered in a reliable manner. Assuming that our hardware and operating systems do the job well, we can use transport protocol on the application level to achieve that goal. In theOSI [http://en.wikipedia.org/wiki/OSI_model] model the two most populartransport protocols [http://en.wikipedia.org/wiki/Transport_layer] areTCP [http://en.wikipedia.org/wiki/Transmission_Control_Protocol] andUDP [http://en.wik

2 min IT Ops

Using Log Analysis to Find the .18%

Even the most non-technical audiences realize the business value in analyzing their log data. DevOps professionals are constantly being asked to monitor their application performance, and often rely on log data for troubleshooting, diagnostics and application systems monitoring. The challenge is that the amount of log data generated can be completely overwhelming. Why log analysis is so difficult? In one word — volume. Even relatively small applications, can generate millions, or billions of lo

4 min Logentries

How to Get Full Visibility of CloudTrail, CloudWatch & System Logs

At re:Invent last year [/2013/11/reinvent-keynote-overview-amazon-cloudtrail-appstream-and-workspaces-announced/] AWS launched CloudTrail [http://aws.amazon.com/cloudtrail/], which is essentially a “trail” of all activity in your AWS Cloud environment… the clue is in the name ;). CloudTrail has been widely accepted as a great addition to the AWS toolset as it is useful for numerous use cases including security analysis, troubleshooting, tracking of AWS resource usage, and more. While CloudTrai

4 min IT Ops

Google Cloud DataFlow – A Game Changer?

So the biggest revolution in database and analytics technology – namely the distributed batch processing technique known as MapReduce (and the associated Hadoop-centric ecosystem that has built up around it) is a legacy technology [http://venturebeat.com/2014/06/25/google-cloud-dataflow/] for one Silicon Valley player. Last week Google announced the arrival of Google Cloud Dataflow – a new service for cloud-based big data analytics that, Google says, supersedes MapReduce.[ While various VCs and

3 min IT Ops

Smart Integration Testing with Dropwizard, Flyway and Retrofit

It’s widely understood that increasing the scope and complexity of a piece of software almost always dramatically increases the effort required to verify it. Verification typically entails testing the behavior of the new feature, plus ensuring no existing functionality has been adversely affected. Because this kind of testing can quickly become painful for even the simplest of components, and because at Logentries [https://logentries.com/centralize-log-data-automatically/?le_trial=dropwizard_f

3 min IT Ops

Why You Should Analyze Your Logs When Load Testing

We recently hosted awebinar about running load tests [/2014/06/how-to-run-jmeter-tests-for-500k-users-analyze-log-data-results/] (with BlazeMeter [http://www.blazemeter.com]) and then analyzing the log data generated on the application under test (with Logentries [http://logentries.com] ). We had a great response and I wanted to share a few more thoughts about why load and stress testing your applications is an important step in understanding how your systems will behave in production when incr

1 min IT Ops

How to Run JMeter Tests for 500k Users & Analyze Log Data Results

Logentries recently co-hosted a webinar with BlazeMeter [http://blazemeter.com] on advanced JMeter testing for 500,000+ users. The webinar had almost 1,000 registrants, and focused on helping  users understand what’s happening on their systems during load testing; identify any warnings or exceptions being thrown; and how to use log data to analyze performance under this load level. In this webinar, Logentries co-founder Trevor Parsons and BlazeMeter DevOps Team Leader Refael Botbol covered how

3 min IT Ops

Correlating Logs and Server Metrics...Finally!

Server resource usage information is a key data point that is critical for any Operations/DevOps team. I like to think of server resource usage information as analogous to the dashboard of your car — it can tell you if you are running out of gas or if something is overheating your system. Similarly, if you keep a record of your server metrics you can also use them to investigate and debug issues in your systems, understand performance trends as well as use in capacity planning over time. [/con

4 min IT Ops

Debugging Rails with Logs

This article was written by Jack Kinsella, a Logentries customer based in Berlin. Jack owns Oxbridge Notes and co-founded Bolivian Express. Below Jack has shared an excerpt from his writings on A Comprehensive Guide to Debugging Rails. Seasoned Rails pros will have interpreting logs down second nature, but I’ll outline the important realizations I had as a beginner for the sake of the version of me out there who is new to Rails development. Whenever you visit a page/fill a form/do an Ajax reque

2 min IT Ops

Collecting and Analyzing Logs from Your Java Application

le_java is one of several libraries that allow you to send log data directly to your Logentries account from your application.  le_java supports both the log4j and logback logging libraries in java, and can be configured to work with either – in just a few steps!  (Check out: github.com/logentries/le_java/blob/master/README.md [https://github.com/logentries/le_java/blob/master/README.md] for our complete tutorial.) [/content/images/le-img/2014/06/java.jpg] Once you have your application send

2 min IT Ops

Our Favorite Linux Performance Monitoring Tools

As a part of monitoring and troubleshooting our system and applications, we often need to get a quick snapshot of information about the status of our server. This usually means SSHing into a server and running certain commands to get to the information we need. I wanted to share a quick overview of my top 5 commands to get a snapshot of this server information. htop [http://hisham.hm/htop/] htop is an ncurses-based interactive process viewer for Linux. It gives information about the CPU utiliz

2 min IT Ops

Using D3.js to Graph Your Log Data

At Logentries, we use the open-source D3.js visualization library for a number of our graphs, including our recently released Insights feature [/2014/05/create-dashboards-from-your-logs-data/]. In a nutshell, D3 allows you to efficiently manipulate documents based on data with minimal overhead. While it could in fact be used for all types of DOM manipulation that you might do with jQuery instead for example, we have used it purely for its graphing functionality. One of the reasons we chose D3 i