cannot access pass device from within jail

Discussion:

(too old to reply)

Konstantin Belousov

2017-12-17 20:37:34 UTC

Hello,
What suggestions do you have for where I should look next? I'm happy to start installing various builds of FreeBSD in order to track down which commit caused this.
I'm trying to access a tape library from within a jail running on a FreeBSD 11.1 host. sa(4) devices are working (e.g. I can rewind nsa0).
The device appears in the jail and has appropriate permissions. This access was granted
via /etc/devfs.rules using the same approach I used for FreeBSD 10.3
crw------- 1 root operator 0x74 Dec 16 21:52 /dev/pass7
cannot open SCSI device '/dev/pass7' - Operation not permitted
Here is the truss output of the command in question: https://gist.github.com/dlangille/b80ee804b8080e1cbf5b5ab67f0bdabe

Does it work to access the pass device from host using host' /dev ?
Same question for the host access using the nodes of the jailed devfs mount.

Bjoern A. Zeeb

2017-12-17 20:37:59 UTC

Permalink

Hello,
What suggestions do you have for where I should look next? I'm happy
to start installing various builds of FreeBSD in order to track down
which commit caused this.
I'm trying to access a tape library from within a jail running on a
FreeBSD 11.1 host. sa(4) devices are working (e.g. I can rewind
nsa0).
pass(4) devices (i.e. the tape changer ch0) are not working. This
https://lists.freebsd.org/pipermail/freebsd-scsi/2017-December/007608.html
The device appears in the jail and has appropriate permissions. This
access was granted
via /etc/devfs.rules using the same approach I used for FreeBSD 10.3
crw------- 1 root operator 0x74 Dec 16 21:52 /dev/pass7
cannot open SCSI device '/dev/pass7' - Operation not permitted
https://gist.github.com/dlangille/b80ee804b8080e1cbf5b5ab67f0bdabe

You don’t by any chance have a securelevel > 1 set for that jail?

/bz

Dan Langille

2017-12-17 20:45:45 UTC

Permalink

Post by Bjoern A. Zeeb

You don’t by any chance have a securelevel > 1 set for that jail?

On the host:

$ sysctl kern.securelevel
kern.securelevel: -1

On the jail:

$ sysctl kern.securelevel
kern.securelevel: -1

Thank you

--
Dan Langille - BSDCan / PGCon
***@langille.org

Dan Langille

2017-12-17 20:49:15 UTC

Permalink

Post by Konstantin Belousov

Does it work to access the pass device from host using host' /dev ?

Yes, it does. see "This command on the host" at https://lists.freebsd.org/pipermail/freebsd-scsi/2017-December/007610.html

Post by Konstantin Belousov
Same question for the host access using the nodes of the jailed devfs mount.

I didn't try that, but I will soon. To be clear, does this command on the host look like what you have in mind?

mtx -f /usr/jails/bacula-sd-02/dev/pass7 status

--
Dan Langille - BSDCan / PGCon
***@langille.org

Konstantin Belousov

2017-12-17 21:01:20 UTC

Permalink

Post by Dan Langille

Post by Konstantin Belousov
Does it work to access the pass device from host using host' /dev ?

Yes, it does. see "This command on the host" at https://lists.freebsd.org/pipermail/freebsd-scsi/2017-December/007610.html

Ok.

Post by Dan Langille

Post by Konstantin Belousov
Same question for the host access using the nodes of the jailed devfs mount.

I didn't try that, but I will soon. To be clear, does this command on the host look like what you have in mind?
mtx -f /usr/jails/bacula-sd-02/dev/pass7 status

I do not know. Check with truss which node gets accessed.

Warner Losh

2017-12-17 21:04:19 UTC

Permalink

What's the permissions of /dev/xpt0 in the jail? If it's not there I know
at least camcontrol won't work. I've not used mtx, so I can't say if it's
affected too or not.

However, looking at the truss output:

openat(AT_FDCWD,"/dev/pass7",O_RDWR|O_EXCL,00) ERR#1 'Operation not
permitted'
suggests something other than the canonical xpt0 issue else is going on. If
we look at passopen in cam, I can see two exit paths:

error = securelevel_gt(td->td_ucred, 1); if (error != 0) {...
return error; }
securelevel_gt is just "return (cr->cr_prison->pr_securelevel > level ?
EPERM : 0);" which might be possible. What's the securelevel of the jail?
Maybe this is going on somehow?

The second is basically
if (((flags & FWRITE) == 0) || ((flags & FREAD) == 0)) {... return
EPERM; }
which isn't happening because of the O_RDWR in the truss output.

The other possibility is that something above the pass driver is doing the
check. I've not looked at that code path yet, buy you can see if it's
making it to passopen() with dtrace and checking its return value. I don't
see anything in how we register the device, though, that would suggest
filtering it in jails.

Warner

Hello,
What suggestions do you have for where I should look next? I'm happy to
start installing various builds of FreeBSD in order to track down which
commit caused this.
I'm trying to access a tape library from within a jail running on a
FreeBSD 11.1 host. sa(4) devices are working (e.g. I can rewind nsa0).
pass(4) devices (i.e. the tape changer ch0) are not working. This morning
December/007608.html
The device appears in the jail and has appropriate permissions. This
access was granted
via /etc/devfs.rules using the same approach I used for FreeBSD 10.3
crw------- 1 root operator 0x74 Dec 16 21:52 /dev/pass7
cannot open SCSI device '/dev/pass7' - Operation not permitted
https://gist.github.com/dlangille/b80ee804b8080e1cbf5b5ab67f0bdabe
Thank you.
--
Dan Langille - BSDCan / PGCon
_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-current

Dan Langille

2017-12-17 21:08:08 UTC

Permalink

Post by Warner Losh
What's the permissions of /dev/xpt0 in the jail? If it's not there I know
at least camcontrol won't work. I've not used mtx, so I can't say if it's
affected too or not.

I have tried both with and without xpt0. When I tried, it was:

# ls -l /dev/xpt0
crw------- 1 root operator 0x4c Dec 16 21:52 /dev/xpt0

Post by Warner Losh
openat(AT_FDCWD,"/dev/pass7",O_RDWR|O_EXCL,00) ERR#1 'Operation not
permitted'
suggests something other than the canonical xpt0 issue else is going on. If
error = securelevel_gt(td->td_ucred, 1); if (error != 0) {...
return error; }
securelevel_gt is just "return (cr->cr_prison->pr_securelevel > level ?
EPERM : 0);" which might be possible. What's the securelevel of the jail?
Maybe this is going on somehow?

On the host:

$ sysctl kern.securelevel
kern.securelevel: -1

On the jail:

$ sysctl kern.securelevel
kern.securelevel: -1

Post by Warner Losh
The second is basically
if (((flags & FWRITE) == 0) || ((flags & FREAD) == 0)) {... return
EPERM; }
which isn't happening because of the O_RDWR in the truss output.
The other possibility is that something above the pass driver is doing the
check. I've not looked at that code path yet, buy you can see if it's
making it to passopen() with dtrace and checking its return value. I don't
see anything in how we register the device, though, that would suggest
filtering it in jails.
Warner

Hello,
What suggestions do you have for where I should look next? I'm happy to
start installing various builds of FreeBSD in order to track down which
commit caused this.
I'm trying to access a tape library from within a jail running on a
FreeBSD 11.1 host. sa(4) devices are working (e.g. I can rewind nsa0).
pass(4) devices (i.e. the tape changer ch0) are not working. This morning
December/007608.html
The device appears in the jail and has appropriate permissions. This
access was granted
via /etc/devfs.rules using the same approach I used for FreeBSD 10.3
crw------- 1 root operator 0x74 Dec 16 21:52 /dev/pass7
cannot open SCSI device '/dev/pass7' - Operation not permitted
https://gist.github.com/dlangille/b80ee804b8080e1cbf5b5ab67f0bdabe
Thank you.
--
Dan Langille - BSDCan / PGCon
_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-current

_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-current

Warner Losh

2017-12-17 21:48:39 UTC

Permalink

Sorry to top post. The problem did turn out to be securelevel. We found
this by doing

dtrace -n 'fbt::securelevel_gt:return {print(args[1];)}'

Though you could also replace securelevel_gt with passopen to see it was in
passopen too...

Warner

Post by Dan Langille

Post by Warner Losh
What's the permissions of /dev/xpt0 in the jail? If it's not there I know
at least camcontrol won't work. I've not used mtx, so I can't say if it's
affected too or not.

# ls -l /dev/xpt0
crw------- 1 root operator 0x4c Dec 16 21:52 /dev/xpt0

Post by Warner Losh
openat(AT_FDCWD,"/dev/pass7",O_RDWR|O_EXCL,00) ERR#1 'Operation not
permitted'
suggests something other than the canonical xpt0 issue else is going on.

Post by Warner Losh
error = securelevel_gt(td->td_ucred, 1); if (error != 0) {...
return error; }
securelevel_gt is just "return (cr->cr_prison->pr_securelevel > level ?
EPERM : 0);" which might be possible. What's the securelevel of the jail?
Maybe this is going on somehow?

$ sysctl kern.securelevel
kern.securelevel: -1
$ sysctl kern.securelevel
kern.securelevel: -1

the

Post by Warner Losh
check. I've not looked at that code path yet, buy you can see if it's
making it to passopen() with dtrace and checking its return value. I

don't

Post by Warner Losh
see anything in how we register the device, though, that would suggest
filtering it in jails.
Warner

morning
pipermail/freebsd-scsi/2017-

Post by Warner Losh

December/007608.html
The device appears in the jail and has appropriate permissions. This
access was granted
via /etc/devfs.rules using the same approach I used for FreeBSD 10.3
crw------- 1 root operator 0x74 Dec 16 21:52 /dev/pass7
cannot open SCSI device '/dev/pass7' - Operation not permitted
https://gist.github.com/dlangille/b80ee804b8080e1cbf5b5ab67f0bdabe
Thank you.
--
Dan Langille - BSDCan / PGCon
_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-current

freebsd.org"

Post by Warner Losh
_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-current

freebsd.org"

Dan Langille

2017-12-17 21:53:46 UTC

Permalink

Post by Warner Losh
Sorry to top post. The problem did turn out to be securelevel. We found
this by doing
dtrace -n 'fbt::securelevel_gt:return {print(args[1];)}'
Though you could also replace securelevel_gt with passopen to see it was in
passopen too...

On the host we ran:

[***@r710-01:~] # sudo dtrace -n 'fbt::securelevel_gt:return {print(args[1]);}'
dtrace: description 'fbt::securelevel_gt:return ' matched 1 probe
CPU ID FUNCTION:NAME
21 50269 securelevel_gt:return int 0x1

In the jail: mtx -f /dev/pass7 status

Based on the dtrace output, I again checked securelevel in the jail:

[***@bacula-sd-02] $ sysctl kern.securelevel
kern.securelevel: 2

WTF? I'd already checked that as seen at https://lists.freebsd.org/pipermail/freebsd-scsi/2017-December/007614.html

Oh wait. The above URL refers to bacula-sd-01, not bacula-sd-02. Ouch. User error... me.

After going back into the host, I set:

$ sudo iocage set securelevel=1 bacula-sd-02
Property: securelevel has been updated to 1

After restarting the jail and getting back into it:

[***@bacula-sd-02 ~]# sysctl kern.securelevel
kern.securelevel: 1

[***@bacula-sd-02 ~]# mtx -f /dev/pass7 status
Storage Changer /dev/pass7:2 Drives, 47 Slots ( 0 Import/Export )
Data Transfer Element 0:Full (Storage Element 1 Loaded):VolumeTag = 000001L4
....

Success.

Sorry for the noise which would have been reduced if I'd gotten my sysctl on the right host.

Thank you all.

Post by Warner Losh
Warner

Post by Dan Langille

Post by Warner Losh
What's the permissions of /dev/xpt0 in the jail? If it's not there I know
at least camcontrol won't work. I've not used mtx, so I can't say if it's
affected too or not.

# ls -l /dev/xpt0
crw------- 1 root operator 0x4c Dec 16 21:52 /dev/xpt0

Post by Warner Losh
openat(AT_FDCWD,"/dev/pass7",O_RDWR|O_EXCL,00) ERR#1 'Operation not
permitted'
suggests something other than the canonical xpt0 issue else is going on.

$ sysctl kern.securelevel
kern.securelevel: -1
$ sysctl kern.securelevel
kern.securelevel: -1

the

Post by Warner Losh
check. I've not looked at that code path yet, buy you can see if it's
making it to passopen() with dtrace and checking its return value. I

don't

Post by Warner Losh
see anything in how we register the device, though, that would suggest
filtering it in jails.
Warner

morning
pipermail/freebsd-scsi/2017-

Post by Warner Losh

freebsd.org"

Post by Warner Losh
_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-current

freebsd.org"

_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-current