Archive for the ‘latency’ Category

ZFS fundamentals: the write throttle

It’s no small feat to build a stable, modern filesystem. The more I work with ZFS, the more impressed I am with how much it got right, and how malleable it’s proved. It has evolved to fix shortcomings and accommodate underlying technological shifts. It’s not surprising though that even while its underpinnings have withstood the [...]

Posted on December 26, 2013 at 5:40 pm by ahl · Permalink · Comments Closed
In: latency, MattAhrens, OpenZFS, performance, Throughput, ZFS

Detecting Outliers

In computer performance, we’re especially concerned about latency outliers: very slow database queries, application requests, disk I/O, etc. The term “outlier” is subjective: there is no rigid mathematical definition. From [Grubbs 69]: An outlying observation, or “outlier,” is one that appears to deviate markedly from other members of the sample in which it occurs. Outliers [...]

Posted on July 1, 2013 at 10:43 am by Brendan Gregg · Permalink · Comments Closed
In: frequencytrail, latency, outliers, performance, statistics, visualizations

Revealing Hidden Latency Patterns

Latency Heat Map Response time – or latency – is crucial to understand in detail, but many of the common presentations of this data hide important details and patterns. Latency heat maps are an effective way to reveal these. I often use tools that provide heat maps directly, but sometimes I have separate trace output [...]

Posted on May 19, 2013 at 2:56 pm by Brendan Gregg · Permalink · Comments Closed
In: heatmaps, latency, performance, visualizations

Off-CPU Performance Analysis

I’ve been exercising a new approach for examining application performance, which involves measuring high resolution off-CPU time that is synchronous to the workload. This reveals which code-paths are blocked and waiting while off-CPU, and for how long exactly. This differs from traditional profiling which often samples the activity of threads at a given interval, and [...]

Posted on July 8, 2011 at 9:36 am by Brendan Gregg · Permalink · Comments Closed
In: DTrace, latency, mysql, off-cpu, performance

Breaking Down MySQL/Percona Query Latency With DTrace

In May I spoke at PerconaLive 2011 in New York, giving a talk titled “Breaking Down MySQL/Percona Query Latency With DTrace”. It was a great conference, and I’d recommend attending the next one if you get a chance. I also attended a number of the talks, and have also caught up on some via video [...]

Posted on July 6, 2011 at 1:48 pm by Brendan Gregg · Permalink · Comments Closed
In: DTrace, latency, mysql, performance, video

File System Latency: part 5

This is part 5 of File System Latency, a series on storage I/O performance from the application perspective (see parts 1, 2, 3 and 4). Previously I explained why disk I/O metrics may not reflect application performance, and how some file system issues may be invisible at the disk I/O level. I then showed how [...]

Posted on June 3, 2011 at 12:02 pm by Brendan Gregg · Permalink · Comments Closed
In: cloud analytics, DTrace, filesystem, latency, mysql, performance, vfsstat

File System Latency: part 4

This is part 4 on file system latency, a series on storage I/O performance from the application perspective (see part 1, part 2 and part 3). In the previous post I showed how to trace file system latency from within MySQL using the pid provider. In this post I’ll show how similar data can be [...]

Posted on May 24, 2011 at 5:57 pm by Brendan Gregg · Permalink · Comments Closed
In: DTrace, filesystem, latency, mysql, performance

File System Latency: part 3

This is part 3 on file system latency, a series on storage I/O performance from the application perspective (see part 1 and part 2). Here I’ll show how to actually measure file system I/O latency – the time spent waiting on the file system to complete I/O. Examining this can save a lot of time [...]

Posted on May 18, 2011 at 9:28 am by Brendan Gregg · Permalink · Comments Closed
In: DTrace, filesystem, latency, mysql, performance

File System Latency: part 2

This is part 2 of File System Latency, a series on storage I/O performance from the application perspective. In the first part I explained why disk I/O is difficult to associate with an application, and why it can be altered from what the application requested. In this part I’ll focus more on the file system, [...]

Posted on May 13, 2011 at 12:26 pm by Brendan Gregg · Permalink · Comments Closed
In: DTrace, filesystem, latency, mysql, performance

File System Latency: part 1

When considering I/O performance from the perspective of applications, file systems can prove to be a better target for analysis than disks. With modern file systems using more DRAM-based cache and performing more asynchronous disk I/O, what the application experiences can vastly differ to what the disks are doing. I’ll demonstrate this by examining I/O [...]

Posted on May 11, 2011 at 6:47 pm by Brendan Gregg · Permalink · Comments Closed
In: DTrace, filesystem, latency, mysql, performance