How to find CPU microcode version ?

[Kurt Jaeger]

Hi!

How do I find the microcode version a Intel CPU is currently using ?

Background:

When Spectre/Meltdown hit, Intel released new microcode around 20180108.

I've used the devcpu-data port at that time to update the
microcode at that time, and the box in question developed
strange instability issues approx. since 14th of Februar.

It had two updates since 8th of January, to r328013 on the 16th of Jan.
and to r328899 on the 6th of February.

Around the 14th, processes started to be killed for out of swapspace,
even if the box had almost no load at all.

pid 1056 (mutt), uid 104, was killed: out of swap space
pid 536 (devd), uid 0, was killed: out of swap space
pid 15093 (exim), uid 26, was killed: out of swap space
pid 91868 (mutt), uid 104, was killed: out of swap space
pid 1086 (sshd), uid 104, was killed: out of swap space

Reboot of the box and waiting a few hours and the problem re-occured.

The box has 32 GB RAM and 34 GB swap, which never was utilized
beyond a few hundred MBs.

Moving the two disks involved to a very similar hardware and
the 'dying' stopped.

The 'strange box' runs a debian version now for 24 hours, no problems.

[Rainer Duffner]

Post by Kurt Jaeger
Hi!
How do I find the microcode version a Intel CPU is currently using ?

AFAIK:
All Linux-vendors have retracted their microcode-updates, for the time being.

If your BIOS-vendor didn’t provide you an updated BIOS, just forget about „fixing" the various Spectre-variants.

[Kurt Jaeger]

Hi!

Post by Kurt Jaeger
How do I find the microcode version a Intel CPU is currently using ?

I was pointed to

https://forums.freebsd.org/threads/introducing-cpupdate-a-microcode-tool-for-freebsd.64588/

which points to:

https://github.com/kernschmelze/cpupdate

Works!

[Kurt Jaeger]

Hi!

Post by Kurt Jaeger

Post by Kurt Jaeger
How do I find the microcode version a Intel CPU is currently using ?

I was pointed to
https://forums.freebsd.org/threads/introducing-cpupdate-a-microcode-tool-for-freebsd.64588/
https://github.com/kernschmelze/cpupdate
Works!

So:

On the box which showed problems, I had:

-> CPUID: 306a9 Family: 06 Model: 3A Stepping: 09 uCodeRev: 00000012

Update done with

pkg install devcpu-data

service microcode_update start

Now I have:

-> CPUID: 306a9 Family: 06 Model: 3A Stepping: 09 uCodeRev: 0000001C

On the spare box that shows no sign of trouble:

-> CPUID: 306a9 Family: 06 Model: 3A Stepping: 09 uCodeRev: 00000019

Hmm.

[Andrea Venturoli]

Post by Kurt Jaeger
Reboot of the box and waiting a few hours and the problem re-occured.

Doesn't the work of microcode_update vanish after a reboot?
Unless of course you set up rc.conf to repeat it at every start...

bye
av.

[Kurt Jaeger]

Hi!

Post by Andrea Venturoli

Post by Kurt Jaeger
Reboot of the box and waiting a few hours and the problem re-occured.

Doesn't the work of microcode_update vanish after a reboot?

Thanks. I was not sure and tested this. Indeed, yes, the microcode update is
reverted by rebooting.

Then I have some other issue, because my problems were ongoing after
reboots.

Post by Andrea Venturoli
Unless of course you set up rc.conf to repeat it at every start...

[Cy Schubert]

Yes. Intel and AMD docs describe this. CPU reset clears any microcode update.

---
Sent using a tiny phone keyboard.
Apologies for any typos and autocorrect.
Also, this old phone only supports top post. Apologies.

Cy Schubert
<***@cschubert.com> or <***@freebsd.org>
The need of the many outweighs the greed of the few.
---

-----Original Message-----
From: Kurt Jaeger
Sent: 18/02/2018 10:50
To: Andrea Venturoli
Cc: freebsd-***@freebsd.org
Subject: Re: How to find CPU microcode version ?

Hi!

Post by Andrea Venturoli

Post by Kurt Jaeger
Reboot of the box and waiting a few hours and the problem re-occured.

Doesn't the work of microcode_update vanish after a reboot?

Thanks. I was not sure and tested this. Indeed, yes, the microcode update is
reverted by rebooting.

Then I have some other issue, because my problems were ongoing after
reboots.

Post by Andrea Venturoli
Unless of course you set up rc.conf to repeat it at every start...

[Shane Ambler]

Post by Kurt Jaeger
Hi!
Around the 14th, processes started to be killed for out of swapspace,
even if the box had almost no load at all.
pid 1056 (mutt), uid 104, was killed: out of swap space
pid 536 (devd), uid 0, was killed: out of swap space
pid 15093 (exim), uid 26, was killed: out of swap space
pid 91868 (mutt), uid 104, was killed: out of swap space
pid 1086 (sshd), uid 104, was killed: out of swap space
Reboot of the box and waiting a few hours and the problem re-occured.
The box has 32 GB RAM and 34 GB swap, which never was utilized
beyond a few hundred MBs.

One situation that can cause out of swap errors is large amounts of
wired ram. Wired ram is memory that can't be swapped out. When you get
around 80 or 90% of physical ram wired you start seeing those errors.

You can see the amount of wired displayed in top or
sysctl vm.stats.vm.v_wire_count then times that by hw.pagesize

There is a vm.max_wired but it doesn't appear to be an enforced limit.

[Kurt Jaeger]

Hi!

Post by Shane Ambler
One situation that can cause out of swap errors is large amounts of
wired ram.

Indeed: Very large amounts of wired RAM. What happened ? This
did not happen before the last upgrade ? I'm at r328899.

CPU: 0.0% user, 0.0% nice, 0.1% system, 0.0% interrupt, 99.9% idle
Mem: 2472K Active, 8K Inact, 31G Wired, 515M Free
ARC: 25G Total, 19G MFU, 3792M MRU, 840K Anon, 492M Header, 2040M Other
22G Compressed, 36G Uncompressed, 1.67:1 Ratio
Swap: 34G Total, 76M Used, 34G Free, 8K In, 92K Out

[Kurt Jaeger]

Hi!

Post by Kurt Jaeger

Post by Shane Ambler
One situation that can cause out of swap errors is large amounts of
wired ram.

Indeed: Very large amounts of wired RAM. What happened ? This
did not happen before the last upgrade ? I'm at r328899.

vfs.zfs.arc_max was at 32345493504

changed that to

vfs.zfs.arc_max=16172746752

Looks better now.