CURRENT r331284: crashing with USB

[Hartmann, O.]

Hello.

Incident: CURRENT r331284 can be brought down reliably with an USB
flash drive plugged in and out without mounting or doing anything with
it.

I first recognized the incident with a ZFS on a SanDisk 32GB USB 3.0
flash drive. Plugging the USB flash and typing "zpool import" revealed
the very first time I issue this command the existence of the ZFS
fielsystem. Usually, I import then this USB drive for maintenance
purposes. Now, typing "zpool import" a second time, nothing is shown at
all. I see that umass0 has been destroyed - although the USB drive is
still plugged in.

Pulling the USB flash drive without having actually imported the
ZFS makes CURRENT crash and reboot.

I tried different USB flash drives, 3.0, 2.0, different boxes running
CURRENT, different hardware (Notebooks, Fujitsu workstations, HP
servers). It seems that the USB subsystem does have a serious problem -
not the ZFS. I can plugin the USB and then unplug it and after two or
three times doing this, the box goes down.


Does anyone else observe this bug?

By the way: all ZFS USB drives I use or all other USB flash drives
cause no problem on FreeBSD 11.1-RELENG-p7!


Kind regards,

Oliver

[Hans Petter Selasky]

Post by Hartmann, O.
Hello.
Incident: CURRENT r331284 can be brought down reliably with an USB
flash drive plugged in and out without mounting or doing anything with
it.
I first recognized the incident with a ZFS on a SanDisk 32GB USB 3.0
flash drive. Plugging the USB flash and typing "zpool import" revealed
the very first time I issue this command the existence of the ZFS
fielsystem. Usually, I import then this USB drive for maintenance
purposes. Now, typing "zpool import" a second time, nothing is shown at
all. I see that umass0 has been destroyed - although the USB drive is
still plugged in.
Pulling the USB flash drive without having actually imported the
ZFS makes CURRENT crash and reboot.
I tried different USB flash drives, 3.0, 2.0, different boxes running
CURRENT, different hardware (Notebooks, Fujitsu workstations, HP
servers). It seems that the USB subsystem does have a serious problem -
not the ZFS. I can plugin the USB and then unplug it and after two or
three times doing this, the box goes down.
Does anyone else observe this bug?
By the way: all ZFS USB drives I use or all other USB flash drives
cause no problem on FreeBSD 11.1-RELENG-p7!

I've seen something similar, but I thought the issue was fixed by:

https://reviews.freebsd.org/D14456

--HPS

[Hyun Hwang]

Post by Hartmann, O.
Hello.
Incident: CURRENT r331284 can be brought down reliably with an USB
flash drive plugged in and out without mounting or doing anything with
it.
[...]
Does anyone else observe this bug?

Can confirm: whenever I plug my Transcend USB microSD reader into my builder (amd64, r331284), the kernel does attach da0 then immediately panics and falls down to `db>` prompt.

Post by Hartmann, O.
I can plugin the USB and then unplug it and after two or three times doing this, the box goes down.

I did not even have to plug-unplug the reader three times; plug the reader in and bam! immediate panic.

AFAIK, r331115 did not have this issue because I was able to update my RPi 2 with the very reader from the very builder.
I managed to salvage kernel binary dump; in case the dump is needed, please let me know.

[Warner Losh]

On Wednesday, March 21, 2018, 12:07 PM (UTC+0100), "Hartmann, O." <

Post by Hartmann, O.
Hello.
Incident: CURRENT r331284 can be brought down reliably with an USB
flash drive plugged in and out without mounting or doing anything with
it.
[...]
Does anyone else observe this bug?

Can confirm: whenever I plug my Transcend USB microSD reader into my
builder (amd64, r331284), the kernel does attach da0 then immediately
panics and falls down to `db>` prompt.

Do you have a traceback?

Warner

Post by Hartmann, O.
I can plugin the USB and then unplug it and after two or three times

doing this, the box goes down.
I did not even have to plug-unplug the reader three times; plug the reader
in and bam! immediate panic.
AFAIK, r331115 did not have this issue because I was able to update my RPi
2 with the very reader from the very builder.
I managed to salvage kernel binary dump; in case the dump is needed, please let me know.
--
Hyun Hwang
_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-current

[Warner Losh]

Post by Warner Losh

On Wednesday, March 21, 2018, 12:07 PM (UTC+0100), "Hartmann, O." <

Post by Hartmann, O.
Hello.
Incident: CURRENT r331284 can be brought down reliably with an USB
flash drive plugged in and out without mounting or doing anything with
it.
[...]
Does anyone else observe this bug?

Can confirm: whenever I plug my Transcend USB microSD reader into my
builder (amd64, r331284), the kernel does attach da0 then immediately
panics and falls down to `db>` prompt.

Do you have a traceback?

actually, can you test https://reviews.freebsd.org/D14792 for me please?
The hardware I bought to provoke this wound up in my wife's bags for a trip
she's still on and I won't be able to test until Friday (which is why I've
been slow to fix this). I hesitate to commit another change I'm sure will
fix it on the off chance I'll be wrong again...

Occasionally, we'll send a TUR to the device. To make sure that the periph
doesn't go away while that's going on, we acquire a reference to the
device. When the command completes we release it. The problem is that
there's a race that the new asserts I put in uncovered. If we've sent a TUR
to the device, but it hasn't completed when damediapoll timeout fires, it
will think that we can send a TUR since we cleared the TUR work flag. This
bumps the count, and bang! we have two TURs in flight. The Transend USB
reader, at least the one I got takes a long time for TUR to return, so this
can trigger the race. The above fix simply says that if a TUR is in
flight, don't schedule another one. We'll poll again later anyway, and we
have the TUR in flight already, so we'll accomplish the goal of TUR even
though we chose to omit one we might otherwise do.

Warner

Post by Warner Losh
Warner

Post by Hartmann, O.
I can plugin the USB and then unplug it and after two or three times

doing this, the box goes down.
I did not even have to plug-unplug the reader three times; plug the
reader in and bam! immediate panic.
AFAIK, r331115 did not have this issue because I was able to update my
RPi 2 with the very reader from the very builder.
I managed to salvage kernel binary dump; in case the dump is needed, please let me know.
--
Hyun Hwang
_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-current
"

[Hyun Hwang]

Post by Warner Losh
Do you have a traceback?

This I got from core.txt:
```
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe000046e3c0
vpanic() at vpanic+0x18d/frame 0xfffffe000046e420
panic() at panic+0x43/frame 0xfffffe000046e480
dadone() at dadone+0x1cc9/frame 0xfffffe000046e9e0
xpt_done_process() at xpt_done_process+0x390/frame 0xfffffe000046ea20
xpt_done_td() at xpt_done_td+0xf6/frame 0xfffffe000046ea70
fork_exit() at fork_exit+0x84/frame 0xfffffe000046eab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe000046eab0
```
Is this suffice? Full dump is available [here](https://www.amazon.com/clouddrive/share/48KzBn0452wRk6bS56oDu1MGXG8p8ug7TGsj8yhGkKN) if you need. (I'll unshare this next month.)
Also, judging by the stack trace, this problem looks very similar to [this](https://lists.freebsd.org/pipermail/freebsd-current/2018-March/068900.html), I guess?

Post by Warner Losh
actually, can you test https://reviews.freebsd.org/D14792 for me please?

I would love to and I can try, but...

Post by Warner Losh
I won't be able to test until Friday

Same here, I cannot physically access the machine of interest until Friday. :(

[bob prohaska]

Post by Hartmann, O.
not the ZFS. I can plugin the USB and then unplug it and after two or
three times doing this, the box goes down.
Does anyone else observe this bug?

An RPI2 running r331179 didn't crash, but it did complain about setting
addresses. The same error crops up from time to time when pl2303 serial
adapters have been in use for some time (hours):

login: ugen0.5: <FTDI USB - Serial> at usbus0 (disconnected)
uftdi0: at uhub1, port 4, addr 5 (disconnected)
uftdi0: detached
ugen0.5: <SanDisk Extreme> at usbus0
umass1 on uhub1
umass1: <SanDisk Extreme, class 0/0, rev 2.10/0.10, addr 5> on usbus0
umass1: SCSI over Bulk-Only; quirks = 0x0100
umass1:1:1: Attached to scbus1
da1 at umass-sim1 bus 1 scbus1 target 0 lun 0
da1: <SanDisk Extreme 0001> Removable Direct Access SPC-4 SCSI device
da1: Serial Number AA010428162242131598
da1: 40.000MB/s transfers
da1: 59836MB (122544516 512 byte sectors)
da1: quirks=0x2<NO_6_BYTE>


FreeBSD/arm (www.zefox.com) (ttyu0)

login: ugen0.5: <SanDisk Extreme> at usbus0 (disconnected)
umass1: at uhub1, port 5, addr 5 (disconnected)
da1 at umass-sim1 bus 1 scbus1 target 0 lun 0
da1: <SanDisk Extreme 0001> s/n AA010428162242131598 detached
(da1:umass-sim1:1:0:0): Periph destroyed
umass1: detached
ugen0.5: <SanDisk Extreme> at usbus0
umass1 on uhub1
umass1: <SanDisk Extreme, class 0/0, rev 2.10/0.10, addr 5> on usbus0
umass1: SCSI over Bulk-Only; quirks = 0x0100
umass1:1:1: Attached to scbus1
da1 at umass-sim1 bus 1 scbus1 target 0 lun 0
da1: <SanDisk Extreme 0001> Removable Direct Access SPC-4 SCSI device
da1: Serial Number AA010428162242131598
da1: 40.000MB/s transfers
da1: 59836MB (122544516 512 byte sectors)
da1: quirks=0x2<NO_6_BYTE>
ugen0.5: <SanDisk Extreme> at usbus0 (disconnected)
umass1: at uhub1, port 5, addr 5 (disconnected)
da1 at umass-sim1 bus 1 scbus1 target 0 lun 0
da1: <SanDisk Extreme 0001> s/n AA010428162242131598 detached
(da1:umass-sim1:1:0:0): Periph destroyed
umass1: detached
usbd_req_re_enumerate: addr=5, set address failed! (USB_ERR_IOERROR, ignored)
usbd_setup_device_desc: getting device descriptor at addr 5 failed, USB_ERR_IOERROR
usbd_req_re_enumerate: addr=5, set address failed! (USB_ERR_IOERROR, ignored)
usbd_setup_device_desc: getting device descriptor at addr 5 failed, USB_ERR_IOERROR
usb_alloc_device: Failure selecting configuration index 0:USB_ERR_IOERROR, port 5, addr 5 (ignored)
ugen0.5: <SanDisk Extreme> at usbus0
ugen0.5: <SanDisk Extreme> at usbus0 (disconnected)
ugen0.5: <SanDisk Extreme> at usbus0
umass1 on uhub1
umass1: <SanDisk Extreme, class 0/0, rev 2.10/0.10, addr 5> on usbus0
umass1: SCSI over Bulk-Only; quirks = 0x0100
umass1:1:1: Attached to scbus1
da1 at umass-sim1 bus 1 scbus1 target 0 lun 0
da1: <SanDisk Extreme 0001> Removable Direct Access SPC-4 SCSI device
da1: Serial Number AA010428162242131598
da1: 40.000MB/s transfers
da1: 59836MB (122544516 512 byte sectors)
da1: quirks=0x2<NO_6_BYTE>

Thanks for reading, I hope it's useful.

bob prohaska

[bob prohaska]

Post by bob prohaska

Post by Hartmann, O.
not the ZFS. I can plugin the USB and then unplug it and after two or
three times doing this, the box goes down.
Does anyone else observe this bug?

An RPI2 running r331179 didn't crash, but it did complain about setting
addresses. The same error crops up from time to time when pl2303 serial

An RPI3 running r331146 reacted more drastically to the insertion of a
USB 3.0 flash drive additional to the one holding /usr and /var; the
console began to emit a flood of gibberish:

ï¿œ`!ￜￜￜQsb"Jï¿œ ￜￜBￜￜￜￜdevices... ￜￜmï¿œoï¿œTￜￜￜￜHￜￜￜￜￜￜￜￜￜￜHUï¿œmￜￜ** First descriptorￜￜￜOo]ￜￜￜￜￜￜￜￜￜscￜￜyￜￜￜRRjï¿œ1 Storage Device(s) found
sￜ0ￜHￜￜ *
"ￜR ￜￜￜXￜt Devￜￜֹ.. 1 Ethernet Device(s) found
Hit any keyￜ}/ￜￜￜￜ/_Kￜ}?ￜￜￜ

which persisted across shutdown -r and momentary power cutoff. Removing power
for a minute or two seems to have restored normal operation. Note the snippets
of clear text; the serial link isn't the culprit, it's working fine.

The flash drive had been used as swap on a Raspbian system successfully,
and had a vestigal partitioning scheme from a previous FreeBSD-current
installation. However, it wasn't mounted in any way, just plugged in, to
cause the upset seen above. I wanted to try using the flash drive as extra
swap, per advice given elsewhere. Clearly that isn't going to work.

Thanks for reading, and any ideas.

bob prohaska

[Ian Lepore]

Post by bob prohaska

Post by bob prohaska

Post by Hartmann, O.
not the ZFS. I can plugin the USB and then unplug it and after two or
three times doing this, the box goes down.
Does anyone else observe this bug?

An RPI2 running r331179 didn't crash, but it did complain about setting
addresses. The same error crops up from time to time when pl2303 serial

An RPI3 running r331146 reacted more drastically to the insertion of a
USB 3.0 flash drive additional to the one holding /usr and /var; the
�`!���Qsb"J� ��B����devices... ��m�o�T����H����������HU�m��**
First descriptor���Oo]���������sc��y���RRj�1 Storage Device(s) found
       s�0�H�� *
                 "�R ���X�t Dev��ֹ.. 1 Ethernet Device(s) found
Hit any key�}/����/_K�}?���
which persisted across shutdown -r and momentary power cutoff.
Removing power 
for a minute or two seems to have restored normal operation. Note the snippets
of clear text; the serial link isn't the culprit, it's working fine.
The flash drive had been used as swap on a Raspbian system
successfully,
and had a vestigal partitioning scheme from a previous FreeBSD-
current
installation. However, it wasn't mounted in any way, just plugged in, to
cause the upset seen above. I wanted to try using the flash drive as extra
swap, per advice given elsewhere. Clearly that isn't going to work.
Thanks for reading, and any ideas.
bob prohaska

Those snippets are some of the first things uboot says when it starts.

-- Ian

[bob prohaska]

Post by Ian Lepore
Those snippets are some of the first things uboot says when it starts.
-- Ian

Yes, understood. The initial console text flooded off the terminal scrollback,
so it couldn't be captured. The portion selected was meant mostly to
relieve suspicions it was a serial malfunction of some kind, copied after
several attempts to reboot. It was quite surprising to see the flood of
nonsense persist after a shutdown -r now had been issued, and apparently
acted upon (the controlling shell exited, but the console flood continued).

The console flood came back after a quick power disconnect/reconnect, but
finally stopped when the power was left off for a minute or so. Then all
returned to normal.

Thanks for reading!

bob prohaska