Last updated at Wed, 01 Nov 2017 20:02:30 GMT

When sending data over the Internet, we want to be sure that data has arrived safely and is delivered in a reliable manner. Assuming that our hardware and operating systems do the job well, we can use transport protocol on the application level to achieve that goal. In the OSI model the two most popular transport protocols are TCP and UDP.

For most people today, reliability is critical and TCP is a natural choice for most applications where data is important; for example, everything from web page, to mail,  to SSH are TCP -based protocols.

TCP can perform well but there are still overheads such as creating and keeping a connection (the state); making sure that data integrity is saved; making sure that data is sent in a reliable way by resending packages if needed;  making sure that packets are delivered in correct order and also not duplicated; the list goes on…

Why would you ever need UDP at all?

UDP often means choosing performance over reliability. As you can see in various sources, UDP is used in certain situations where TCP does not perform well. Some examples of this include;

  • Live media streaming (radio, TV, internet calls like Skype and Hangouts) – people perception can forgive small transmission errors like scratches in image or crackles in the sound and we can still understand it quite well.
  • Broadcasting/multicasting (auto-discovery mechanisms like avahi, streaming to multiple clients).
  • VPN/tunneling – tunneled protocols use TCP and they will handle reliability, so there is no reason to double it.
  • Creating a quick response with low overhead – like DNS.
  • Creating your own reliability protocols.
  • Stock quotations – when only the most recent possible data is important, it does not matter if you miss a quotation or two in some cases.

Why do you need UDP for logs?!

In the previous list of UDP usage examples there is no place for logs. But the logs are important, and must be part of process for reliably delivering applications. You need the logs…

Some reasons for using UDP for transporting logs are:

  • Legacy (some devices/software do not support TCP for logs, especially old devices).
  • Application performance – UDP is send and forget; don’t wait for server confirmation! This is extremely important when you want to send huge amounts of non-critical logs.
  • Transferring state information.
  • Performance overhead – A lot of monitoring applications use UDP to minimize performance overhead to the system that is being monitored. Does it mean that we are losing monitoring data? Probably. Are we losing all of it? Probably not. Can we live with incomplete monitoring data? In most cases probably yes.
  • Monitoring data – especially metrics data, is sent as absolute counters instead of relative measures.

Quick example: we are sending packages like:

Packet 1: bytes_sent=1233213
Packet 2: bytes_sent=1342123
Packet 3: bytes_sent=1381887

In that case even if we lose some packages we still have information about the system state. Also, even if some information is missing we can calculate some variability approximation.

Sending relative monitoring data can be dangerous. For example:

Packet 1: bytes_sent=+22829
Packet 2: bytes_sent=+108910
Packet 3: bytes_sent=+39764

If one of the packets is lost, we are losing information not only about dynamics but also about absolute amount of data sent.

How does Logentries handle TCP and UDP?

Logentries allows you to set up alerts and notifications that should be triggered on specific patterns when detected in logs. Also, Logentries does not distinguish between inputs when processing logs. All data is important to us and we trigger alerts and notifications on all incoming data you specify. If any packet comes into our system, we handle it carefully.

You can use UDP for:

  • Monitoring data with absolute values, such as:

LOG 1: bytes_transmitted=3331232
LOG 2: bytes_transmitted=3342038
LOG 3: bytes_transmitted=3352345

  • Events that are non-critical (you have to choose) and occur in a huge number; losing some events will not affect results that much, such as:

LOG 1: User clicked item
LOG 2: User clicked item
LOG 3: User clicked item

LOG 1: Page reload
LOG 2: Page reload
LOG 3: Page reload

  • A certain level of error tolerance can result in a big performance gain.

You can use TCP for:

  • Handling application errors.
  • Logging important events, such as: - User registered
  • User logged in

With Logentries you can centralize, search and analyze your logs very easily;  but you can also use Logentries as a performance monitoring tool. By using aggregated searches, counting events with specific patterns, and extracting metrics from logs, you can easily create real-time dashboards  showing you all of your important  performance information in one place.

Ideally, you can bring together the best of TCP and UDP to create integrated and dynamic graphs and dashboards to assure ongoing performance and application delivery.