[GTALUG] Booting Fedora from M.2 Optane Nvme

[Russell via talk]

With all the CPU cache fencing going on I thought it might be fun to fiddle with NVM express on Fedora 27. My primary SSD on this z370-A, was/is in build limbo while I updated bios and waited for outcomes.

I installed a 32gib module from Newegg ($90 delivered) and I booted from F27-workstation live on usb. I let the anaconda installer do the automatic partitioning. 5gib of the 32gib reported was left unallocated for over-provisioning.

First boot appeared to be twice as fast as on the SATA SSD. Three or four seconds to get to the gui login as opposed to 7 or 8 on the original drive.

Had to install hdparm from the repo.

# hdparm -tT --direct /dev/sda1

/dev/sda1:
Timing O_DIRECT cached reads: 930 MB in 2.00 seconds = 464.92 MB/sec
Timing O_DIRECT disk reads: 1370 MB in 3.00 seconds = 456.30 MB/sec

# hdparm -tT --direct /dev/nvme0n1

/dev/nvme0n1:
Timing O_DIRECT cached reads: 2302 MB in 2.00 seconds = 1151.84 MB/sec
Timing O_DIRECT disk reads: 3474 MB in 3.00 seconds = 1157.65 MB/sec

More info on Nvme with a nice diagram of the Linux storage stack here.

https://en.m.wikipedia.org/wiki/NVM_Express

I switched back to the original SSD. An os-probe mapped the device to /dev/dm-1 and I was able to remake grub.conf and add this install to the bootloader on that drive.

A couple of switches back and forth between drives and it looks like the 10% performance hit I had experienced after the microcode updates, is now somewhat reduced, +-5%.

Guilding the lilly a bit, as I didn't have any day to day problems with performance. Although it's hard to name Intel's product actions around this issue a field of Lilly's.

[Lennart Sorensen via talk]

Post by Russell via talk
With all the CPU cache fencing going on I thought it might be fun to fiddle with NVM express on Fedora 27. My primary SSD on this z370-A, was/is in build limbo while I updated bios and waited for outcomes.
I installed a 32gib module from Newegg ($90 delivered) and I booted from F27-workstation live on usb. I let the anaconda installer do the automatic partitioning. 5gib of the 32gib reported was left unallocated for over-provisioning.
First boot appeared to be twice as fast as on the SATA SSD. Three or four seconds to get to the gui login as opposed to 7 or 8 on the original drive.
Had to install hdparm from the repo.
# hdparm -tT --direct /dev/sda1
Timing O_DIRECT cached reads: 930 MB in 2.00 seconds = 464.92 MB/sec
Timing O_DIRECT disk reads: 1370 MB in 3.00 seconds = 456.30 MB/sec
# hdparm -tT --direct /dev/nvme0n1
Timing O_DIRECT cached reads: 2302 MB in 2.00 seconds = 1151.84 MB/sec
Timing O_DIRECT disk reads: 3474 MB in 3.00 seconds = 1157.65 MB/sec
More info on Nvme with a nice diagram of the Linux storage stack here.
https://en.m.wikipedia.org/wiki/NVM_Express
I switched back to the original SSD. An os-probe mapped the device to /dev/dm-1 and I was able to remake grub.conf and add this install to the bootloader on that drive.
A couple of switches back and forth between drives and it looks like the 10% performance hit I had experienced after the microcode updates, is now somewhat reduced, +-5%.
Guilding the lilly a bit, as I didn't have any day to day problems with performance. Although it's hard to name Intel's product actions around this issue a field of Lilly's.

Not bad, but I think for my money I would get an EVO 960 instead.
Twice the speed and 8 times the space for just under twice the money.
Or 16 times the space for three times the money. I just don't see any
advantage to the Optane at this point.

[Russell via talk]

Post by Russell via talk

Post by Russell via talk
With all the CPU cache fencing going on I thought it might be fun to

fiddle with NVM express on Fedora 27. My primary SSD on this z370-A,
was/is in build limbo while I updated bios and waited for outcomes.

Post by Russell via talk
I installed a 32gib module from Newegg ($90 delivered) and I booted

from F27-workstation live on usb. I let the anaconda installer do the
automatic partitioning. 5gib of the 32gib reported was left unallocated
for over-provisioning.

Post by Russell via talk
First boot appeared to be twice as fast as on the SATA SSD. Three or

four seconds to get to the gui login as opposed to 7 or 8 on the
original drive.

Post by Russell via talk
Had to install hdparm from the repo.
# hdparm -tT --direct /dev/sda1
Timing O_DIRECT cached reads: 930 MB in 2.00 seconds = 464.92

MB/sec

Post by Russell via talk
Timing O_DIRECT disk reads: 1370 MB in 3.00 seconds = 456.30 MB/sec
# hdparm -tT --direct /dev/nvme0n1
Timing O_DIRECT cached reads: 2302 MB in 2.00 seconds = 1151.84

MB/sec

Post by Russell via talk
Timing O_DIRECT disk reads: 3474 MB in 3.00 seconds = 1157.65

MB/sec

Post by Russell via talk
More info on Nvme with a nice diagram of the Linux storage stack

here.

Post by Russell via talk
https://en.m.wikipedia.org/wiki/NVM_Express
I switched back to the original SSD. An os-probe mapped the device to

/dev/dm-1 and I was able to remake grub.conf and add this install to
the bootloader on that drive.

Post by Russell via talk
A couple of switches back and forth between drives and it looks like

the 10% performance hit I had experienced after the microcode updates,
is now somewhat reduced, +-5%.

Post by Russell via talk
Guilding the lilly a bit, as I didn't have any day to day problems

with performance. Although it's hard to name Intel's product actions
around this issue a field of Lilly's.
Not bad, but I think for my money I would get an EVO 960 instead.
Twice the speed and 8 times the space for just under twice the money.
Or 16 times the space for three times the money. I just don't see any
advantage to the Optane at this point.

Yea, this is just a tinker-toy. I had figured to put a small backup OS on this, remove its allocation from the chainloader and then leave it alone in the box in case I need it, instead of using a live usb for troubleshooting etc.

It doesn't look like I can mask it's physical presence tho, except by populating a pcie slot and co-opting it's assigned pci lanes, so no security through obscurity.

Instead I went with full Fedora w/gnome. I have 7gib on the drive after a full workstation install so I may rethink stuff and use it as my primary. Optane was meant to be a cache accelerator for booting Windows 10 on a standard drive and thats about the sum of it.

Although, with all the elbow room and thumbscrews in this box, you could use it as a key drive for sneakernet backup to a safety deposit box pretty handily. You'd just need the right finger screw for the 1_M.2 slot standoff post, which is exposed. The other slot is buried under a heat sink.

Post by Russell via talk
--
Len Sorensen

[Lennart Sorensen via talk]

Post by Russell via talk
Yea, this is just a tinker-toy. I had figured to put a small backup OS on this, remove its allocation from the chainloader and then leave it alone in the box in case I need it, instead of using a live usb for troubleshooting etc.
It doesn't look like I can mask it's physical presence tho, except by populating a pcie slot and co-opting it's assigned pci lanes, so no security through obscurity.
Instead I went with full Fedora w/gnome. I have 7gib on the drive after a full workstation install so I may rethink stuff and use it as my primary. Optane was meant to be a cache accelerator for booting Windows 10 on a standard drive and thats about the sum of it.
Although, with all the elbow room and thumbscrews in this box, you could use it as a key drive for sneakernet backup to a safety deposit box pretty handily. You'd just need the right finger screw for the 1_M.2 slot standoff post, which is exposed. The other slot is buried under a heat sink.

Well as a toy, $90 is easier to justify than $170.

Not sure how many insertions that M.2 slot is rated for. I don't think
I would do anything that involves frequent swapping.

An external thunderbolt connected NVMe drive would survive that much
better.

[Russell via talk]

Post by Russell via talk

Post by Russell via talk
Yea, this is just a tinker-toy. I had figured to put a small backup

OS on this, remove its allocation from the chainloader and then leave
it alone in the box in case I need it, instead of using a live usb for
troubleshooting etc.

Post by Russell via talk
It doesn't look like I can mask it's physical presence tho, except by

populating a pcie slot and co-opting it's assigned pci lanes, so no
security through obscurity.

Post by Russell via talk
Instead I went with full Fedora w/gnome. I have 7gib on the drive

after a full workstation install so I may rethink stuff and use it as
my primary. Optane was meant to be a cache accelerator for booting
Windows 10 on a standard drive and thats about the sum of it.

Post by Russell via talk
Although, with all the elbow room and thumbscrews in this box, you

could use it as a key drive for sneakernet backup to a safety deposit
box pretty handily. You'd just need the right finger screw for the
1_M.2 slot standoff post, which is exposed. The other slot is buried
under a heat sink.
Well as a toy, $90 is easier to justify than $170.
Not sure how many insertions that M.2 slot is rated for. I don't think
I would do anything that involves frequent swapping.

I wouldn't recommend it commercially or for production, but personal privacy may take a lot of different forms and methods. In a sense nonstandard can be a good thing.

There is a pcie adapter for M.2 so I imagine that some amount of removal and insertion is built into the lifespan of the product, albeit without warranty of purpose.

Post by Russell via talk
An external thunderbolt connected NVMe drive would survive that much
better.

In fact I just wanted to look at a measure of cache performance mitigations for now.

Onboard recovery OS or, deposit box storage of the card w/os & data, are just a way of self justifing the cost of playing around with Linux and perhaps extending the scope of my tinkering.

Externals are possibly subject to theft, loss and impact damage from falling off the desk. Stuff attached on the inside of the box, is just a bit more inconvienent to access and that makes it a little more physically secure.

Three second delay for bios and three more seconds to login prompt, pretty much changed my mind about useage of this card.

I'll probably use it as my primary drive for the time being. There is an apparent improvement in browsing the web. Seems like pages load faster than on the SSD, but that may just be wishfull thinking.

All in good fun.

Post by Russell via talk
--
Len Sorensen

[Lennart Sorensen via talk]

Post by Russell via talk
I wouldn't recommend it commercially or for production, but personal privacy may take a lot of different forms and methods. In a sense nonstandard can be a good thing.
There is a pcie adapter for M.2 so I imagine that some amount of removal and insertion is built into the lifespan of the product, albeit without warranty of purpose.

I would not be surprised if the M.2 connector has a rating of 50
insertions. It is obviously designed for internal use and not frequent
changes.

Post by Russell via talk
In fact I just wanted to look at a measure of cache performance mitigations for now.
Onboard recovery OS or, deposit box storage of the card w/os & data, are just a way of self justifing the cost of playing around with Linux and perhaps extending the scope of my tinkering.
Externals are possibly subject to theft, loss and impact damage from falling off the desk. Stuff attached on the inside of the box, is just a bit more inconvienent to access and that makes it a little more physically secure.
Three second delay for bios and three more seconds to login prompt, pretty much changed my mind about useage of this card.

If only the BIOS could be faster.

Post by Russell via talk
I'll probably use it as my primary drive for the time being. There is an apparent improvement in browsing the web. Seems like pages load faster than on the SSD, but that may just be wishfull thinking.
All in good fun.

Well some browsers do like to maintain their state in databases in your
home directory so a fast disk sure could help.

[Russell via talk]

Post by Russell via talk

Post by Russell via talk
I wouldn't recommend it commercially or for production, but personal

privacy may take a lot of different forms and methods. In a sense
nonstandard can be a good thing.

Post by Russell via talk
There is a pcie adapter for M.2 so I imagine that some amount of

removal and insertion is built into the lifespan of the product, albeit
without warranty of purpose.
I would not be surprised if the M.2 connector has a rating of 50
insertions. It is obviously designed for internal use and not frequent
changes.

Probably the same with the pcie cards, but if the data backups for offsite (personal data) are once or twice a year, with carefull handling the card could probably last a generation.

Post by Russell via talk

Post by Russell via talk
In fact I just wanted to look at a measure of cache performance

mitigations for now.

Post by Russell via talk
Onboard recovery OS or, deposit box storage of the card w/os & data,

are just a way of self justifing the cost of playing around with Linux
and perhaps extending the scope of my tinkering.

Post by Russell via talk
Externals are possibly subject to theft, loss and impact damage from

falling off the desk. Stuff attached on the inside of the box, is just
a bit more inconvienent to access and that makes it a little more
physically secure.

Post by Russell via talk
Three second delay for bios and three more seconds to login prompt,

pretty much changed my mind about useage of this card.
If only the BIOS could be faster.

Actually the three seconds is the delay to allow the user to hit F2 to enter bios. Could probably reduce that to one, I'll have to check that.

Post by Russell via talk

Post by Russell via talk
I'll probably use it as my primary drive for the time being. There is

an apparent improvement in browsing the web. Seems like pages load
faster than on the SSD, but that may just be wishfull thinking.

Post by Russell via talk
All in good fun.

Well some browsers do like to maintain their state in databases in your
home directory so a fast disk sure could help.

Sure seems to. I had intended to put in a larger SATA platter and move my old data onto it. I think now I'll wait a bit longer and see where the CAD vs. tech price point ends up over the next little while and go directly to nvme.

That is hoping the dollar doesn't tank any further (unlikely), the loonie hasn't been this low in quite a while and also that the bitcoiners stop buying out the stock (also unlikely) but we'll see.

Post by Russell via talk
--
Len Sorensen