Discussion:
[GTALUG] System Monitor / sysmon
D. Hugh Redelmeier via talk
2018-08-19 15:53:40 UTC
Permalink
I have sysmon on my Linux systems. It is a nice graphical tool to show
CPU, memory, and network loads. It produces strip charts.

(It also does other things, but that's not my current concern.)

When I want to figure out what's taking so long, sysmon is a handy tool.

But I think that my system's bottleneck is often disk I/O. sysmon has no
strip chart for that. I don't even know how it should be done:

- each device should probably have a different trace, just like each
processor core does

- percentage would be ideal but I don't know what that would mean. Do you
count I/O ops? Queue length? Processes waiting?

+ sequential I/O is much faster than random I/O (at least on a real
disk).

+ Large operations are somewhat slower than small ones (generally
speaking).

+ output is generally non-blocking (roughly: the write goes into the
kernel's memory, queued for writing to the device, and the process
proceeds without waiting to the write to actually complete)

+ input usually blocks: the process issues a read and waits until
the read finishes before it proceeds

+ Perhaps the simplest metric might be the number of processes
waiting for disk I/O completion. But that leaves writes
unaccounted for.

Do you have any suggestions for a simple (possibly simplistic) tool to
show disk bottlenecks in real time?

PS:

On the system I'm looking at, in the sysmon pane for "Processes",
there is a column for "Disk read total" and another for "Disk write
total". After a day of running the Libreswan test suite,
gnome-terminal server wins with 3.4GiB of read and 3.9GiB of write.

I wonder what that's all about. I would not expect Gnome Terminal to
do disk I/O. As it happens, essentially all the output to the
terminal is being captured by a script(1) command (run within the gnome
terminal session). It shows 11.4MiB of disk output for one complete run,
which should be half or a third of all that has gone to Gnome Terminal
because I've run the suite two or three times since logging in.
But it is more like a 11.4/3400 or roughly 1/300.

Guess: gnome terminal reads and writes to X count as disk I/O. That's
pretty messed up.
---
Talk Mailing List
***@gta
Kevin Cozens via talk
2018-08-20 04:18:08 UTC
Permalink
Post by D. Hugh Redelmeier via talk
But I think that my system's bottleneck is often disk I/O. sysmon has no
strip chart for that.
[snip]
Post by D. Hugh Redelmeier via talk
Do you have any suggestions for a simple (possibly simplistic) tool to
show disk bottlenecks in real time?
A quick internet search suggests you should look at the sar command that is
part of sysstat.
--
Cheers!

Kevin.

http://www.ve3syb.ca/ | "Nerds make the shiny things that
https://www.patreon.com/KevinCozens | distract the mouth-breathers, and
| that's why we're powerful"
Owner of Elecraft K2 #2172 |
#include <disclaimer/favourite> | --Chris Hardwick
---
Talk Mailing List
***@gtalu
David Collier-Brown via talk
2018-08-20 15:32:29 UTC
Permalink
On the sar front, you watch out for avgqu-sz (and/or await and svctm)
suddenly increasing as tps increases.
The queue length is like the "load factor", where <= 1 is good, 4 is
bad, and more is evil.

--dave
Post by Kevin Cozens via talk
But I think that my system's bottleneck is often disk I/O.  sysmon
has no
strip chart for that.
[snip]
Do you have any suggestions for a simple (possibly simplistic) tool to
show disk bottlenecks in real time?
A quick internet search suggests you should look at the sar command
that is part of sysstat.
--
David Collier-Brown, | Always do right. This will gratify
System Programmer and Author | some people and astonish the rest
***@spamcop.net | -- Mark Twain

---
Talk Mailing List
***@gtalug.org
https://gtalu
D. Hugh Redelmeier via talk
2018-08-20 22:10:12 UTC
Permalink
| From: Kevin Cozens via talk <***@gtalug.org>

| A quick internet search suggests you should look at the sar command that is
| part of sysstat.

Thanks for the research!

I want a stupid GUI program. sar is a command-line tool, I think.

sar's manual makes it look very complex to use. Not really inviting.

If I just act dumb and type "sar", I find it isn't installed (but the
shell offers to install it). When I say yes, it is installed, but the
shell forgets that I want to run it. When I again ask to run it, I get:

Cannot open /var/log/sa/sa20: No such file or directory
Please check if data collecting is enabled

All-in-all, not as brainless as running sysmon.
---
Talk Mailing List
***@gtalug.org
https://gtalug.org/mailman
Christopher Browne via talk
2018-08-20 22:36:39 UTC
Permalink
Post by D. Hugh Redelmeier via talk
Thanks for the research!
I want a stupid GUI program. sar is a command-line tool, I think.
I do recall sar... Definitely one of the classic tools for analyzing disk
activity. It's probably worth some learning curve to attempt it.

I have never gotten terribly deep with any of these tools.

If there's something modern and either
A) brainless easy, or
B) deeply better
That would be awesome to know of.

sar and iostat are the ones I always remember.

I seem to recall sar being a bit risky to use as it might induce a fair bit
of I/O load itself.
Anthony de Boer via talk
2018-08-22 14:29:15 UTC
Permalink
Post by D. Hugh Redelmeier via talk
Do you have any suggestions for a simple (possibly simplistic) tool to
show disk bottlenecks in real time?
I generally fire up "iotop -ko" when I'm tracking anything I/O-bound.

Other friends are htop and iftop.
--
Anthony de Boer
---
Talk Mailing List
***@gtalug.org
https://gtalug.org/mailman/list
Loading...