Jens Axboe <axboe@kernel.dk> <axboe@fb.com>
Jens Axboe <axboe@kernel.dk> <axboe@meta.com>
Jens Osterkamp <Jens.Osterkamp@de.ibm.com>
+Jens Wiklander <jenswi@kernel.org> <jens.wiklander@linaro.org>
Jernej Skrabec <jernej.skrabec@gmail.com> <jernej.skrabec@siol.net>
Jesper Dangaard Brouer <hawk@kernel.org> <brouer@redhat.com>
Jesper Dangaard Brouer <hawk@kernel.org> <hawk@comx.dk>
Nicholas Piggin <npiggin@gmail.com> <npiggin@suse.de>
Nicholas Piggin <npiggin@gmail.com> <nickpiggin@yahoo.com.au>
Nicholas Piggin <npiggin@gmail.com> <piggin@cyberone.com.au>
-Nick Desaulniers <nick.desaulniers+lkml@gmail.com> <ndesaulniers@google.com>
Nicolas Ferre <nicolas.ferre@microchip.com> <nicolas.ferre@atmel.com>
Nicolas Pitre <nico@fluxnic.net> <nicolas.pitre@linaro.org>
Nicolas Pitre <nico@fluxnic.net> <nico@linaro.org>
Quentin Monnet <qmo@kernel.org> <quentin.monnet@netronome.com>
Quentin Monnet <qmo@kernel.org> <quentin@isovalent.com>
Quentin Perret <qperret@qperret.net> <quentin.perret@arm.com>
+Radu Rendec <radu@rendec.net> <radu.rendec@ines.ro>
+Radu Rendec <radu@rendec.net> <rrendec@arista.com>
+Radu Rendec <radu@rendec.net> <radu.rendec@gmail.com>
+Radu Rendec <radu@rendec.net> <rrendec@redhat.com>
Rae Moar <raemoar63@gmail.com> <rmoar@google.com>
Rafael J. Wysocki <rjw@rjwysocki.net> <rjw@sisk.pl>
Rajeev Nandan <quic_rajeevny@quicinc.com> <rajeevny@codeaurora.org>
Simon Wunderlich <sw@simonwunderlich.de> <simon.wunderlich@saxnet.de>
Simon Wunderlich <sw@simonwunderlich.de> <simon@open-mesh.com>
Simon Wunderlich <sw@simonwunderlich.de> <siwu@hrz.tu-chemnitz.de>
+SJ Park <sj@kernel.org>
Sricharan Ramabadhran <quic_srichara@quicinc.com> <sricharan@codeaurora.org>
Srinivas Kandagatla <srini@kernel.org> <srinivas.kandagatla@st.com>
Srinivas Kandagatla <srini@kernel.org> <srinivas.kandagatla@linaro.org>
Stanislav Fomichev <sdf@fomichev.me> <sdf@google.com>
Stanislav Fomichev <sdf@fomichev.me> <stfomichev@gmail.com>
Stefan Wahren <wahrenst@gmx.net> <stefan.wahren@i2se.com>
-Stéphane Grosjean <stephane.grosjean@hms-networks.com> <s.grosjean@peak-system.com>
-Stéphane Grosjean <stephane.grosjean@hms-networks.com> <stephane.grosjean@free.fr>
+Stéphane Grosjean <s.grosjean@peak-system.fr> <s.grosjean@peak-system.com>
+Stéphane Grosjean <s.grosjean@peak-system.fr> <stephane.grosjean@free.fr>
Stéphane Witzmann <stephane.witzmann@ubpmes.univ-bpclermont.fr>
Stephen Hemminger <stephen@networkplumber.org> <shemminger@linux-foundation.org>
Stephen Hemminger <stephen@networkplumber.org> <shemminger@osdl.org>
S: 75013 Paris
S: France
+N: Wolfram Sang
+E: wsa@kernel.org
+W: sang-engineering.com
+P: rsa4096/140DE4CC14A029B6 3991 B1EA B9E2 6751 A4F7 645D 140D E4CC 14A0 29B6
+D: I2C Maintainer 2012 - 2026
+S: Berlin, Germany
+
N: Aleksa Sarai
E: cyphar@cyphar.com
W: https://www.cyphar.com/
Reading this attribute gives the state of the DbC. It
can be one of the following states: disabled, enabled,
- initialized, connected or configured.
+ initialized, connected, configured or suspended.
What: /sys/bus/pci/drivers/xhci_hcd/.../dbc_idVendor
Date: March 2023
what: /sys/kernel/mm/damon/
Date: Mar 2022
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Interface for Data Access MONitoring (DAMON). Contains files
for controlling DAMON. For more details on DAMON itself,
please refer to Documentation/admin-guide/mm/damon/index.rst.
What: /sys/kernel/mm/damon/admin/
Date: Mar 2022
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Interface for privileged users of DAMON. Contains files for
controlling DAMON that aimed to be used by privileged users.
What: /sys/kernel/mm/damon/admin/kdamonds/nr_kdamonds
Date: Mar 2022
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Writing a number 'N' to this file creates the number of
directories for controlling each DAMON worker thread (kdamond)
named '0' to 'N-1' under the kdamonds/ directory.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/state
Date: Mar 2022
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Writing 'on' or 'off' to this file makes the kdamond starts or
stops, respectively. Reading the file returns the keywords
based on the current status. Writing 'commit' to this file
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/pid
Date: Mar 2022
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Reading this file returns the pid of the kdamond if it is
running.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/refresh_ms
Date: Jul 2025
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Writing a value to this file sets the time interval for
automatic DAMON status file contents update. Writing '0'
disables the update. Reading this file returns the value.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/nr_contexts
Date: Mar 2022
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Writing a number 'N' to this file creates the number of
directories for controlling each DAMON context named '0' to
'N-1' under the contexts/ directory.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/avail_operations
Date: Apr 2022
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Reading this file returns the available monitoring operations
sets on the currently running kernel.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/operations
Date: Mar 2022
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Writing a keyword for a monitoring operations set ('vaddr' for
virtual address spaces monitoring, 'fvaddr' for fixed virtual
address ranges monitoring, and 'paddr' for the physical address
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/addr_unit
Date: Aug 2025
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Writing an integer to this file sets the 'address unit'
parameter of the given operations set of the context. Reading
the file returns the last-written 'address unit' value.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/pause
Date: Mar 2026
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Writing a boolean keyword to this file sets the 'pause' request
parameter for the context. Reading the file returns the
last-written 'pause' value.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/monitoring_attrs/intervals/sample_us
Date: Mar 2022
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Writing a value to this file sets the sampling interval of the
DAMON context in microseconds as the value. Reading this file
returns the value.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/monitoring_attrs/intervals/aggr_us
Date: Mar 2022
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Writing a value to this file sets the aggregation interval of
the DAMON context in microseconds as the value. Reading this
file returns the value.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/monitoring_attrs/intervals/update_us
Date: Mar 2022
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Writing a value to this file sets the update interval of the
DAMON context in microseconds as the value. Reading this file
returns the value.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/monitoring_attrs/intervals/intrvals_goal/access_bp
Date: Feb 2025
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Writing a value to this file sets the monitoring intervals
auto-tuning target DAMON-observed access events ratio within
the given time interval (aggrs in same directory), in bp
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/monitoring_attrs/intervals/intrvals_goal/aggrs
Date: Feb 2025
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Writing a value to this file sets the time interval to achieve
the monitoring intervals auto-tuning target DAMON-observed
access events ratio (access_bp in same directory) within.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/monitoring_attrs/intervals/intrvals_goal/min_sample_us
Date: Feb 2025
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Writing a value to this file sets the minimum value of
auto-tuned sampling interval in microseconds. Reading this
file returns the value.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/monitoring_attrs/intervals/intrvals_goal/max_sample_us
Date: Feb 2025
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Writing a value to this file sets the maximum value of
auto-tuned sampling interval in microseconds. Reading this
file returns the value.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/monitoring_attrs/nr_regions/min
WDate: Mar 2022
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Writing a value to this file sets the minimum number of
monitoring regions of the DAMON context as the value. Reading
this file returns the value.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/monitoring_attrs/nr_regions/max
Date: Mar 2022
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Writing a value to this file sets the maximum number of
monitoring regions of the DAMON context as the value. Reading
this file returns the value.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/targets/nr_targets
Date: Mar 2022
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Writing a number 'N' to this file creates the number of
directories for controlling each DAMON target of the context
named '0' to 'N-1' under the contexts/ directory.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/targets/<T>/pid_target
Date: Mar 2022
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Writing to and reading from this file sets and gets the pid of
the target process if the context is for virtual address spaces
monitoring, respectively.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/targets/<T>/obsolete_target
Date: Oct 2025
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Writing to and reading from this file sets and gets the
obsoleteness of the matching parameters commit destination
target.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/targets/<T>/regions/nr_regions
Date: Mar 2022
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Writing a number 'N' to this file creates the number of
directories for setting each DAMON target memory region of the
context named '0' to 'N-1' under the regions/ directory. In
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/targets/<T>/regions/<R>/start
Date: Mar 2022
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Writing to and reading from this file sets and gets the start
address of the monitoring region.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/targets/<T>/regions/<R>/end
Date: Mar 2022
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Writing to and reading from this file sets and gets the end
address of the monitoring region.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/nr_schemes
Date: Mar 2022
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Writing a number 'N' to this file creates the number of
directories for controlling each DAMON-based operation scheme
of the context named '0' to 'N-1' under the schemes/ directory.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/action
Date: Mar 2022
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Writing to and reading from this file sets and gets the action
of the scheme.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/target_nid
Date: Jun 2024
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Action's target NUMA node id. Supported by only relevant
actions.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/apply_interval_us
Date: Sep 2023
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Writing a value to this file sets the action apply interval of
the scheme in microseconds. Reading this file returns the
value.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/access_pattern/sz/min
Date: Mar 2022
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Writing to and reading from this file sets and gets the minimum
size of the scheme's target regions in bytes.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/access_pattern/sz/max
Date: Mar 2022
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Writing to and reading from this file sets and gets the maximum
size of the scheme's target regions in bytes.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/access_pattern/nr_accesses/min
Date: Mar 2022
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Writing to and reading from this file sets and gets the manimum
'nr_accesses' of the scheme's target regions.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/access_pattern/nr_accesses/max
Date: Mar 2022
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Writing to and reading from this file sets and gets the maximum
'nr_accesses' of the scheme's target regions.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/access_pattern/age/min
Date: Mar 2022
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Writing to and reading from this file sets and gets the minimum
'age' of the scheme's target regions.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/access_pattern/age/max
Date: Mar 2022
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Writing to and reading from this file sets and gets the maximum
'age' of the scheme's target regions.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/quotas/ms
Date: Mar 2022
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Writing to and reading from this file sets and gets the time
quota of the scheme in milliseconds.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/quotas/bytes
Date: Mar 2022
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Writing to and reading from this file sets and gets the size
quota of the scheme in bytes.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/quotas/effective_bytes
Date: Feb 2024
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Reading from this file gets the effective size quota of the
scheme in bytes, which adjusted for the time quota and goals.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/quotas/reset_interval_ms
Date: Mar 2022
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Writing to and reading from this file sets and gets the quotas
charge reset interval of the scheme in milliseconds.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/quotas/goals/nr_goals
Date: Nov 2023
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Writing a number 'N' to this file creates the number of
directories for setting automatic tuning of the scheme's
aggressiveness named '0' to 'N-1' under the goals/ directory.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/quotas/goals/<G>/target_metric
Date: Feb 2024
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Writing to and reading from this file sets and gets the quota
auto-tuning goal metric.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/quotas/goals/<G>/target_value
Date: Nov 2023
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Writing to and reading from this file sets and gets the target
value of the goal metric.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/quotas/goals/<G>/current_value
Date: Nov 2023
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Writing to and reading from this file sets and gets the current
value of the goal metric.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/quotas/goals/<G>/nid
Date: Apr 2025
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Writing to and reading from this file sets and gets the nid
parameter of the goal.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/quotas/goals/<G>/path
Date: Oct 2025
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Writing to and reading from this file sets and gets the path
parameter of the goal.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/quotas/goal_tuner
Date: Mar 2026
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Writing to and reading from this file sets and gets the
goal-based effective quota auto-tuning algorithm to use.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/quotas/fail_charge_num
Date: Mar 2026
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Writing to and reading from this file sets and gets the
action-failed memory quota charging ratio numerator.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/quotas/fail_charge_denom
Date: Mar 2026
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Writing to and reading from this file sets and gets the
action-failed memory quota charging ratio denominator.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/quotas/weights/sz_permil
Date: Mar 2022
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Writing to and reading from this file sets and gets the
under-quota limit regions prioritization weight for 'size' in
permil.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/quotas/weights/nr_accesses_permil
Date: Mar 2022
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Writing to and reading from this file sets and gets the
under-quota limit regions prioritization weight for
'nr_accesses' in permil.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/quotas/weights/age_permil
Date: Mar 2022
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Writing to and reading from this file sets and gets the
under-quota limit regions prioritization weight for 'age' in
permil.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/watermarks/metric
Date: Mar 2022
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Writing to and reading from this file sets and gets the metric
of the watermarks for the scheme. The writable/readable
keywords for this file are 'none' for disabling the watermarks
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/watermarks/interval_us
Date: Mar 2022
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Writing to and reading from this file sets and gets the metric
check interval of the watermarks for the scheme in
microseconds.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/watermarks/high
Date: Mar 2022
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Writing to and reading from this file sets and gets the high
watermark of the scheme in permil.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/watermarks/mid
Date: Mar 2022
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Writing to and reading from this file sets and gets the mid
watermark of the scheme in permil.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/watermarks/low
Date: Mar 2022
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Writing to and reading from this file sets and gets the low
watermark of the scheme in permil.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/core_filters
Date: Feb 2025
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Directory for DAMON core layer-handled DAMOS filters.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/core_filters/nr_filters
Date: Feb 2025
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Writing a number 'N' to this file creates the number of
directories for setting filters of the scheme named '0' to
'N-1' under the core_filters/ directory.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/core_filters/<F>/type
Date: Feb 2025
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Writing to and reading from this file sets and gets the type of
the memory of the interest. 'anon' for anonymous pages,
'memcg' for specific memory cgroup, 'young' for young pages,
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/core_filters/<F>/memcg_path
Date: Feb 2025
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: If 'memcg' is written to the 'type' file, writing to and
reading from this file sets and gets the path to the memory
cgroup of the interest.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/core_filters/<F>/addr_start
Date: Feb 2025
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: If 'addr' is written to the 'type' file, writing to or reading
from this file sets or gets the start address of the address
range for the filter.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/core_filters/<F>/addr_end
Date: Feb 2025
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: If 'addr' is written to the 'type' file, writing to or reading
from this file sets or gets the end address of the address
range for the filter.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/core_filters/<F>/min
Date: Feb 2025
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: If 'hugepage_size' is written to the 'type' file, writing to
or reading from this file sets or gets the minimum size of the
hugepage for the filter.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/core_filters/<F>/max
Date: Feb 2025
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: If 'hugepage_size' is written to the 'type' file, writing to
or reading from this file sets or gets the maximum size of the
hugepage for the filter.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/core_filters/<F>/damon_target_idx
Date: Feb 2025
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: If 'target' is written to the 'type' file, writing to or
reading from this file sets or gets the index of the DAMON
monitoring target of the interest.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/core_filters/<F>/matching
Date: Feb 2025
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Writing 'Y' or 'N' to this file sets whether the filter is for
the memory of the 'type', or all except the 'type'.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/core_filters/<F>/allow
Date: Feb 2025
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Writing 'Y' or 'N' to this file sets whether to allow or reject
applying the scheme's action to the memory that satisfies the
'type' and the 'matching' of the directory.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/ops_filters
Date: Feb 2025
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Directory for DAMON operations set layer-handled DAMOS filters.
Files under this directory works same to those of
/sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/core_filters
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/filters
Date: Dec 2022
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Directory for DAMOS filters. Files under this directory works
same to those of
/sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/{core,ops}_filters
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/dests/nr_dests
Date: Jul 2025
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Writing a number 'N' to this file creates the number of
directories for setting action destinations of the scheme named
'0' to 'N-1' under the dests/ directory.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/dests/<D>/id
Date: Jul 2025
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Writing to and reading from this file sets and gets the id of
the DAMOS action destination. For DAMOS_MIGRATE_{HOT,COLD}
actions, the destination node's node id can be written and
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/dests/<D>/weight
Date: Jul 2025
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Writing to and reading from this file sets and gets the weight
of the DAMOS action destination to select as the destination of
each action among the destinations.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/stats/nr_tried
Date: Mar 2022
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Reading this file returns the number of regions that the action
of the scheme has tried to be applied.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/stats/sz_tried
Date: Mar 2022
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Reading this file returns the total size of regions that the
action of the scheme has tried to be applied in bytes.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/stats/nr_applied
Date: Mar 2022
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Reading this file returns the number of regions that the action
of the scheme has successfully applied.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/stats/sz_applied
Date: Mar 2022
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Reading this file returns the total size of regions that the
action of the scheme has successfully applied in bytes.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/stats/sz_ops_filter_passed
Date: Dec 2024
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Reading this file returns the total size of memory that passed
DAMON operations layer-handled filters of the scheme in bytes.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/stats/qt_exceeds
Date: Mar 2022
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Reading this file returns the number of the exceed events of
the scheme's quotas.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/stats/nr_snapshots
Date: Dec 2025
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Reading this file returns the total number of DAMON snapshots
that the scheme has tried to be applied.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/stats/max_nr_snapshots
Date: Dec 2025
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Writing a number to this file sets the upper limit of
nr_snapshots that deactivates the scheme when the limit is
reached or exceeded.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/tried_regions/total_bytes
Date: Jul 2023
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Reading this file returns the total amount of memory that
corresponding DAMON-based Operation Scheme's action has tried
to be applied.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/tried_regions/<R>/start
Date: Oct 2022
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Reading this file returns the start address of a memory region
that corresponding DAMON-based Operation Scheme's action has
tried to be applied.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/tried_regions/<R>/end
Date: Oct 2022
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Reading this file returns the end address of a memory region
that corresponding DAMON-based Operation Scheme's action has
tried to be applied.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/tried_regions/<R>/nr_accesses
Date: Oct 2022
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Reading this file returns the 'nr_accesses' of a memory region
that corresponding DAMON-based Operation Scheme's action has
tried to be applied.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/tried_regions/<R>/age
Date: Oct 2022
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Reading this file returns the 'age' of a memory region that
corresponding DAMON-based Operation Scheme's action has tried
to be applied.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/tried_regions/<R>/sz_filter_passed
Date: Dec 2024
-Contact: SeongJae Park <sj@kernel.org>
+Contact: SJ Park <sj@kernel.org>
Description: Reading this file returns the size of the memory in the region
that passed DAMON operations layer-handled filters of the
scheme in bytes.
1-2. Why RDMA controller needed?
1-3. How is RDMA controller implemented?
2. Usage Examples
+ 3. RDMA Interface Files
1. Overview
===========
(d) Delete resource limit::
echo mlx4_0 hca_handle=max hca_object=max > /sys/fs/cgroup/rdma/1/rdma.max
+
+3. RDMA Interface Files
+========================
+
+The following interface files are available in each non-root RDMA cgroup.
+
+ rdma.max
+ A read-write file which describes the configured resource limit
+ for an RDMA/IB device. See the Usage Examples above.
+
+ rdma.current
+ A read-only file which describes the current resource usage.
+
+ rdma.peak
+ A read-only nested-keyed file which shows the historical high
+ watermark of resource usage per device since the cgroup was created.
+
+ An example for mlx4 and ocrdma device follows::
+
+ mlx4_0 hca_handle=1 hca_object=20
+ ocrdma1 hca_handle=0 hca_object=23
+
+ rdma.events
+ A read-only nested-keyed file which exists on non-root cgroups
+ and contains the following keys:
+
+ max
+ The number of times a process in this cgroup or its
+ descendants attempted an RDMA resource allocation that
+ was rejected because a rdma.max limit in the subtree
+ was reached. This is a hierarchical counter propagated
+ upward to all ancestor cgroups. A value change in this
+ file generates a file modified event.
+
+ alloc_fail
+ The number of RDMA resource allocation attempts that
+ originated in this cgroup or its descendants and failed
+ due to a rdma.max limit being reached. This is a
+ hierarchical counter propagated upward.
+
+ An example for mlx4 device follows::
+
+ mlx4_0 hca_handle.max=5 hca_handle.alloc_fail=3 hca_object.max=0 hca_object.alloc_fail=0
+
+ rdma.events.local
+ Similar to rdma.events but the fields are local to the cgroup,
+ i.e. not hierarchical. The file modified event generated on this
+ file reflects only the local events.
+
+ The following nested keys are defined.
+
+ max
+ The number of times a process in this cgroup or its
+ descendants attempted an RDMA resource allocation that
+ was rejected because this cgroup's own rdma.max limit
+ was reached.
+
+ alloc_fail
+ The number of RDMA resource allocation attempts
+ originating from this cgroup that failed due to this
+ cgroup's or an ancestor's rdma.max limit.
+
+ An example for mlx4 device follows::
+
+ mlx4_0 hca_handle.max=5 hca_handle.alloc_fail=0 hca_object.max=0 hca_object.alloc_fail=0
sock (npn)
Amount of memory used in network transmission buffers
- vmalloc (npn)
+ vmalloc
Amount of memory used for vmap backed memory.
shmem
Number of pages written from zswap to swap.
zswap_incomp
- Number of incompressible pages currently stored in zswap
+ Amount of memory used by incompressible pages currently stored in zswap
without compression. These pages could not be compressed to
a size smaller than PAGE_SIZE, so they are stored as-is.
So the ideal way to configure this is to set io.latency in groups A, B, and C.
Generally you do not want to set a value lower than the latency your device
supports. Experiment to find the value that works best for your workload.
-Start at higher than the expected latency for your device and watch the
-avg_lat value in io.stat for your workload group to get an idea of the
-latency you see during normal operation. Use the avg_lat value as a basis for
-your real setting, setting at 10-15% higher than the value in io.stat.
+Start at higher than the expected latency for your device and, with
+blkcg_debug_stats enabled, watch the avg_lat value in io.stat for your
+workload group to get an idea of the latency you see during normal operation.
+Use the avg_lat value as a basis for your real setting, setting at 10-15%
+higher than the value in io.stat.
How IO Latency Throttling Works
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
io.stat
If the controller is enabled you will see extra stats in io.stat in
- addition to the normal ones.
+ addition to the normal ones. These debug stats are only emitted when
+ the blkcg_debug_stats module parameter is enabled (it is disabled by
+ default).
depth
This is the current queue depth for the group.
Misc Interface Files
~~~~~~~~~~~~~~~~~~~~
-Miscellaneous controller provides 3 interface files. If two misc resources (res_a and res_b) are registered then:
+Miscellaneous controller provides the following interface files. If two misc
+resources (res_a and res_b) are registered then:
misc.capacity
A read-only flat-keyed file shown only in the root cgroup. It shows
as such devices are emulated.
CPU Hotplug on virtual systems is supported. It is distinct from physical
-CPU Hotplug as all resources are described as ``present``, but CPUs may be
-marked as disabled by firmware. Only the CPU's online/offline behaviour is
-influenced by firmware. An example is where a virtual machine boots with a
-single CPU, and additional CPUs are added once a cloud orchestrator deploys
-the workload.
+CPU Hotplug as all vCPU resources are statically described in the firmware
+configuration tables (e.g. MADT), meaning their maximum possible count is
+known at boot. However, vCPUs that are not enabled at boot are not marked
+as ``present`` by the kernel until they are hotplugged. An example is where
+a virtual machine boots with a single CPU, and additional CPUs are added
+once a cloud orchestrator deploys the workload.
For a virtual machine, the VMM (e.g. Qemu) plays the part of firmware.
``DENIED``.
The ACPI tables must describe all the resources of the virtual machine. CPUs
-that firmware wishes to disable either from boot (or later) should not be
-``enabled`` in the MADT GICC structures, but should have the ``online capable``
-bit set, to indicate they can be enabled later. The boot CPU must be marked as
-``enabled``. The 'always on' GICR structure must be used to describe the
-redistributors.
+that are hot-pluggable must have the ``online capable`` bit set and the
+``enabled`` bit cleared in the MADT GICC structures to indicate they can be
+enabled later. The boot CPU must be marked as ``enabled`` with its
+``online capable`` bit cleared. The 'always on' GICR structure must be used
+to describe the redistributors.
CPUs described as ``online capable`` but not ``enabled`` can be set to enabled
by the DSDT's Processor object's _STA method. On virtual systems the _STA method
-must always report the CPU as ``present``. Changes to the firmware policy can
-be notified to the OS via device-check or eject-request.
+must always set the ``ACPI_STA_DEVICE_PRESENT`` bit, while toggling the
+``ACPI_STA_DEVICE_ENABLED`` bit to reflect its plug status. The kernel will
+then dynamically mark the vCPU as ``present`` within the OS when the
+``ACPI_STA_DEVICE_ENABLED`` bit becomes set during hot-add. Changes to the
+firmware policy can be notified to the OS via device-check or eject-request.
CPUs described as ``enabled`` in the static table, should not have their _STA
modified dynamically by firmware. Soft-restart features such as kexec will
version 1.0 of the RISC-V Vector extension manual.
* :c:macro:`RISCV_HWPROBE_EXT_ZBA`: The Zba address generation extension is
- supported, as defined in version 1.0 of the Bit-Manipulation ISA
- extensions.
+ supported, as defined in version 1.0 of the Bit-Manipulation ISA
+ extensions.
* :c:macro:`RISCV_HWPROBE_EXT_ZBB`: The Zbb extension is supported, as defined
- in version 1.0 of the Bit-Manipulation ISA extensions.
+ in version 1.0 of the Bit-Manipulation ISA extensions.
* :c:macro:`RISCV_HWPROBE_EXT_ZBS`: The Zbs extension is supported, as defined
- in version 1.0 of the Bit-Manipulation ISA extensions.
+ in version 1.0 of the Bit-Manipulation ISA extensions.
* :c:macro:`RISCV_HWPROBE_EXT_ZICBOZ`: The Zicboz extension is supported, as
- ratified in commit 3dd606f ("Create cmobase-v1.0.pdf") of riscv-CMOs.
+ ratified in commit 3dd606f ("Create cmobase-v1.0.pdf") of riscv-CMOs.
* :c:macro:`RISCV_HWPROBE_EXT_ZBC` The Zbc extension is supported, as defined
- in version 1.0 of the Bit-Manipulation ISA extensions.
+ in version 1.0 of the Bit-Manipulation ISA extensions.
* :c:macro:`RISCV_HWPROBE_EXT_ZBKB` The Zbkb extension is supported, as
- defined in version 1.0 of the Scalar Crypto ISA extensions.
+ defined in version 1.0 of the Scalar Crypto ISA extensions.
* :c:macro:`RISCV_HWPROBE_EXT_ZBKC` The Zbkc extension is supported, as
- defined in version 1.0 of the Scalar Crypto ISA extensions.
+ defined in version 1.0 of the Scalar Crypto ISA extensions.
* :c:macro:`RISCV_HWPROBE_EXT_ZBKX` The Zbkx extension is supported, as
- defined in version 1.0 of the Scalar Crypto ISA extensions.
+ defined in version 1.0 of the Scalar Crypto ISA extensions.
* :c:macro:`RISCV_HWPROBE_EXT_ZKND` The Zknd extension is supported, as
- defined in version 1.0 of the Scalar Crypto ISA extensions.
+ defined in version 1.0 of the Scalar Crypto ISA extensions.
* :c:macro:`RISCV_HWPROBE_EXT_ZKNE` The Zkne extension is supported, as
- defined in version 1.0 of the Scalar Crypto ISA extensions.
+ defined in version 1.0 of the Scalar Crypto ISA extensions.
* :c:macro:`RISCV_HWPROBE_EXT_ZKNH` The Zknh extension is supported, as
- defined in version 1.0 of the Scalar Crypto ISA extensions.
+ defined in version 1.0 of the Scalar Crypto ISA extensions.
* :c:macro:`RISCV_HWPROBE_EXT_ZKSED` The Zksed extension is supported, as
- defined in version 1.0 of the Scalar Crypto ISA extensions.
+ defined in version 1.0 of the Scalar Crypto ISA extensions.
* :c:macro:`RISCV_HWPROBE_EXT_ZKSH` The Zksh extension is supported, as
- defined in version 1.0 of the Scalar Crypto ISA extensions.
+ defined in version 1.0 of the Scalar Crypto ISA extensions.
* :c:macro:`RISCV_HWPROBE_EXT_ZKT` The Zkt extension is supported, as defined
- in version 1.0 of the Scalar Crypto ISA extensions.
+ in version 1.0 of the Scalar Crypto ISA extensions.
* :c:macro:`RISCV_HWPROBE_EXT_ZVBB`: The Zvbb extension is supported as
- defined in version 1.0 of the RISC-V Cryptography Extensions Volume II.
+ defined in version 1.0 of the RISC-V Cryptography Extensions Volume II.
* :c:macro:`RISCV_HWPROBE_EXT_ZVBC`: The Zvbc extension is supported as
- defined in version 1.0 of the RISC-V Cryptography Extensions Volume II.
+ defined in version 1.0 of the RISC-V Cryptography Extensions Volume II.
* :c:macro:`RISCV_HWPROBE_EXT_ZVKB`: The Zvkb extension is supported as
- defined in version 1.0 of the RISC-V Cryptography Extensions Volume II.
+ defined in version 1.0 of the RISC-V Cryptography Extensions Volume II.
* :c:macro:`RISCV_HWPROBE_EXT_ZVKG`: The Zvkg extension is supported as
- defined in version 1.0 of the RISC-V Cryptography Extensions Volume II.
+ defined in version 1.0 of the RISC-V Cryptography Extensions Volume II.
* :c:macro:`RISCV_HWPROBE_EXT_ZVKNED`: The Zvkned extension is supported as
- defined in version 1.0 of the RISC-V Cryptography Extensions Volume II.
+ defined in version 1.0 of the RISC-V Cryptography Extensions Volume II.
* :c:macro:`RISCV_HWPROBE_EXT_ZVKNHA`: The Zvknha extension is supported as
- defined in version 1.0 of the RISC-V Cryptography Extensions Volume II.
+ defined in version 1.0 of the RISC-V Cryptography Extensions Volume II.
* :c:macro:`RISCV_HWPROBE_EXT_ZVKNHB`: The Zvknhb extension is supported as
- defined in version 1.0 of the RISC-V Cryptography Extensions Volume II.
+ defined in version 1.0 of the RISC-V Cryptography Extensions Volume II.
* :c:macro:`RISCV_HWPROBE_EXT_ZVKSED`: The Zvksed extension is supported as
- defined in version 1.0 of the RISC-V Cryptography Extensions Volume II.
+ defined in version 1.0 of the RISC-V Cryptography Extensions Volume II.
* :c:macro:`RISCV_HWPROBE_EXT_ZVKSH`: The Zvksh extension is supported as
- defined in version 1.0 of the RISC-V Cryptography Extensions Volume II.
+ defined in version 1.0 of the RISC-V Cryptography Extensions Volume II.
* :c:macro:`RISCV_HWPROBE_EXT_ZVKT`: The Zvkt extension is supported as
- defined in version 1.0 of the RISC-V Cryptography Extensions Volume II.
+ defined in version 1.0 of the RISC-V Cryptography Extensions Volume II.
* :c:macro:`RISCV_HWPROBE_EXT_ZFH`: The Zfh extension version 1.0 is supported
- as defined in the RISC-V ISA manual.
+ as defined in the RISC-V ISA manual.
* :c:macro:`RISCV_HWPROBE_EXT_ZFHMIN`: The Zfhmin extension version 1.0 is
- supported as defined in the RISC-V ISA manual.
+ supported as defined in the RISC-V ISA manual.
* :c:macro:`RISCV_HWPROBE_EXT_ZIHINTNTL`: The Zihintntl extension version 1.0
- is supported as defined in the RISC-V ISA manual.
+ is supported as defined in the RISC-V ISA manual.
* :c:macro:`RISCV_HWPROBE_EXT_ZVFH`: The Zvfh extension is supported as
- defined in the RISC-V Vector manual starting from commit e2ccd0548d6c
- ("Remove draft warnings from Zvfh[min]").
+ defined in the RISC-V Vector manual starting from commit e2ccd0548d6c
+ ("Remove draft warnings from Zvfh[min]").
* :c:macro:`RISCV_HWPROBE_EXT_ZVFHMIN`: The Zvfhmin extension is supported as
- defined in the RISC-V Vector manual starting from commit e2ccd0548d6c
- ("Remove draft warnings from Zvfh[min]").
+ defined in the RISC-V Vector manual starting from commit e2ccd0548d6c
+ ("Remove draft warnings from Zvfh[min]").
* :c:macro:`RISCV_HWPROBE_EXT_ZFA`: The Zfa extension is supported as
- defined in the RISC-V ISA manual starting from commit 056b6ff467c7
- ("Zfa is ratified").
+ defined in the RISC-V ISA manual starting from commit 056b6ff467c7
+ ("Zfa is ratified").
* :c:macro:`RISCV_HWPROBE_EXT_ZTSO`: The Ztso extension is supported as
- defined in the RISC-V ISA manual starting from commit 5618fb5a216b
- ("Ztso is now ratified.")
+ defined in the RISC-V ISA manual starting from commit 5618fb5a216b
+ ("Ztso is now ratified.")
* :c:macro:`RISCV_HWPROBE_EXT_ZACAS`: The Zacas extension is supported as
- defined in the Atomic Compare-and-Swap (CAS) instructions manual starting
- from commit 5059e0ca641c ("update to ratified").
+ defined in the Atomic Compare-and-Swap (CAS) instructions manual starting
+ from commit 5059e0ca641c ("update to ratified").
* :c:macro:`RISCV_HWPROBE_EXT_ZICNTR`: The Zicntr extension version 2.0
- is supported as defined in the RISC-V ISA manual.
+ is supported as defined in the RISC-V ISA manual.
* :c:macro:`RISCV_HWPROBE_EXT_ZICOND`: The Zicond extension is supported as
- defined in the RISC-V Integer Conditional (Zicond) operations extension
- manual starting from commit 95cf1f9 ("Add changes requested by Ved
- during signoff")
+ defined in the RISC-V Integer Conditional (Zicond) operations extension
+ manual starting from commit 95cf1f9 ("Add changes requested by Ved
+ during signoff")
* :c:macro:`RISCV_HWPROBE_EXT_ZIHINTPAUSE`: The Zihintpause extension is
- supported as defined in the RISC-V ISA manual starting from commit
- d8ab5c78c207 ("Zihintpause is ratified").
+ supported as defined in the RISC-V ISA manual starting from commit
+ d8ab5c78c207 ("Zihintpause is ratified").
* :c:macro:`RISCV_HWPROBE_EXT_ZIHPM`: The Zihpm extension version 2.0
- is supported as defined in the RISC-V ISA manual.
+ is supported as defined in the RISC-V ISA manual.
* :c:macro:`RISCV_HWPROBE_EXT_ZVE32X`: The Vector sub-extension Zve32x is
supported, as defined by version 1.0 of the RISC-V Vector extension manual.
supported, as defined by version 1.0 of the RISC-V Vector extension manual.
* :c:macro:`RISCV_HWPROBE_EXT_ZIMOP`: The Zimop May-Be-Operations extension is
- supported as defined in the RISC-V ISA manual starting from commit
- 58220614a5f ("Zimop is ratified/1.0").
+ supported as defined in the RISC-V ISA manual starting from commit
+ 58220614a5f ("Zimop is ratified/1.0").
* :c:macro:`RISCV_HWPROBE_EXT_ZCA`: The Zca extension part of Zc* standard
- extensions for code size reduction, as ratified in commit 8be3419c1c0
- ("Zcf doesn't exist on RV64 as it contains no instructions") of
- riscv-code-size-reduction.
+ extensions for code size reduction, as ratified in commit 8be3419c1c0
+ ("Zcf doesn't exist on RV64 as it contains no instructions") of
+ riscv-code-size-reduction.
* :c:macro:`RISCV_HWPROBE_EXT_ZCB`: The Zcb extension part of Zc* standard
- extensions for code size reduction, as ratified in commit 8be3419c1c0
- ("Zcf doesn't exist on RV64 as it contains no instructions") of
- riscv-code-size-reduction.
+ extensions for code size reduction, as ratified in commit 8be3419c1c0
+ ("Zcf doesn't exist on RV64 as it contains no instructions") of
+ riscv-code-size-reduction.
* :c:macro:`RISCV_HWPROBE_EXT_ZCD`: The Zcd extension part of Zc* standard
- extensions for code size reduction, as ratified in commit 8be3419c1c0
- ("Zcf doesn't exist on RV64 as it contains no instructions") of
- riscv-code-size-reduction.
+ extensions for code size reduction, as ratified in commit 8be3419c1c0
+ ("Zcf doesn't exist on RV64 as it contains no instructions") of
+ riscv-code-size-reduction.
* :c:macro:`RISCV_HWPROBE_EXT_ZCF`: The Zcf extension part of Zc* standard
- extensions for code size reduction, as ratified in commit 8be3419c1c0
- ("Zcf doesn't exist on RV64 as it contains no instructions") of
- riscv-code-size-reduction.
+ extensions for code size reduction, as ratified in commit 8be3419c1c0
+ ("Zcf doesn't exist on RV64 as it contains no instructions") of
+ riscv-code-size-reduction.
* :c:macro:`RISCV_HWPROBE_EXT_ZCMOP`: The Zcmop May-Be-Operations extension is
- supported as defined in the RISC-V ISA manual starting from commit
- c732a4f39a4 ("Zcmop is ratified/1.0").
+ supported as defined in the RISC-V ISA manual starting from commit
+ c732a4f39a4 ("Zcmop is ratified/1.0").
* :c:macro:`RISCV_HWPROBE_EXT_ZAWRS`: The Zawrs extension is supported as
- ratified in commit 98918c844281 ("Merge pull request #1217 from
- riscv/zawrs") of riscv-isa-manual.
+ ratified in commit 98918c844281 ("Merge pull request #1217 from
+ riscv/zawrs") of riscv-isa-manual.
* :c:macro:`RISCV_HWPROBE_EXT_ZAAMO`: The Zaamo extension is supported as
- defined in the in the RISC-V ISA manual starting from commit e87412e621f1
- ("integrate Zaamo and Zalrsc text (#1304)").
+ defined in the in the RISC-V ISA manual starting from commit e87412e621f1
+ ("integrate Zaamo and Zalrsc text (#1304)").
* :c:macro:`RISCV_HWPROBE_EXT_ZALASR`: The Zalasr extension is supported as
- frozen at commit 194f0094 ("Version 0.9 for freeze") of riscv-zalasr.
+ frozen at commit 194f0094 ("Version 0.9 for freeze") of riscv-zalasr.
* :c:macro:`RISCV_HWPROBE_EXT_ZALRSC`: The Zalrsc extension is supported as
- defined in the in the RISC-V ISA manual starting from commit e87412e621f1
- ("integrate Zaamo and Zalrsc text (#1304)").
+ defined in the in the RISC-V ISA manual starting from commit e87412e621f1
+ ("integrate Zaamo and Zalrsc text (#1304)").
* :c:macro:`RISCV_HWPROBE_EXT_SUPM`: The Supm extension is supported as
- defined in version 1.0 of the RISC-V Pointer Masking extensions.
+ defined in version 1.0 of the RISC-V Pointer Masking extensions.
* :c:macro:`RISCV_HWPROBE_EXT_ZFBFMIN`: The Zfbfmin extension is supported as
- defined in the RISC-V ISA manual starting from commit 4dc23d6229de
- ("Added Chapter title to BF16").
+ defined in the RISC-V ISA manual starting from commit 4dc23d6229de
+ ("Added Chapter title to BF16").
* :c:macro:`RISCV_HWPROBE_EXT_ZVFBFMIN`: The Zvfbfmin extension is supported as
- defined in the RISC-V ISA manual starting from commit 4dc23d6229de
- ("Added Chapter title to BF16").
+ defined in the RISC-V ISA manual starting from commit 4dc23d6229de
+ ("Added Chapter title to BF16").
* :c:macro:`RISCV_HWPROBE_EXT_ZVFBFWMA`: The Zvfbfwma extension is supported as
- defined in the RISC-V ISA manual starting from commit 4dc23d6229de
- ("Added Chapter title to BF16").
+ defined in the RISC-V ISA manual starting from commit 4dc23d6229de
+ ("Added Chapter title to BF16").
* :c:macro:`RISCV_HWPROBE_EXT_ZICBOM`: The Zicbom extension is supported, as
- ratified in commit 3dd606f ("Create cmobase-v1.0.pdf") of riscv-CMOs.
+ ratified in commit 3dd606f ("Create cmobase-v1.0.pdf") of riscv-CMOs.
* :c:macro:`RISCV_HWPROBE_EXT_ZABHA`: The Zabha extension is supported as
- ratified in commit 49f49c842ff9 ("Update to Rafified state") of
- riscv-zabha.
+ ratified in commit 49f49c842ff9 ("Update to Rafified state") of
+ riscv-zabha.
* :c:macro:`RISCV_HWPROBE_EXT_ZICBOP`: The Zicbop extension is supported, as
- ratified in commit 3dd606f ("Create cmobase-v1.0.pdf") of riscv-CMOs.
+ ratified in commit 3dd606f ("Create cmobase-v1.0.pdf") of riscv-CMOs.
* :c:macro:`RISCV_HWPROBE_EXT_ZILSD`: The Zilsd extension is supported as
- defined in the RISC-V ISA manual starting from commit f88abf1 ("Integrating
- load/store pair for RV32 with the main manual") of the riscv-isa-manual.
+ defined in the RISC-V ISA manual starting from commit f88abf1 ("Integrating
+ load/store pair for RV32 with the main manual") of the riscv-isa-manual.
* :c:macro:`RISCV_HWPROBE_EXT_ZCLSD`: The Zclsd extension is supported as
- defined in the RISC-V ISA manual starting from commit f88abf1 ("Integrating
- load/store pair for RV32 with the main manual") of the riscv-isa-manual.
+ defined in the RISC-V ISA manual starting from commit f88abf1 ("Integrating
+ load/store pair for RV32 with the main manual") of the riscv-isa-manual.
+
+ * :c:macro:`RISCV_HWPROBE_EXT_ZICFILP`: The Zicfilp extension is supported,
+ as defined in version 1.0 of the RISC-V Control-flow Integrity (CFI)
+ extensions specification, ratified in commit 302a2d45c243
+ ("Update build-pdf.yml") of riscv-cfi.
* :c:macro:`RISCV_HWPROBE_KEY_CPUPERF_0`: Deprecated. Returns similar values to
- :c:macro:`RISCV_HWPROBE_KEY_MISALIGNED_SCALAR_PERF`, but the key was
- mistakenly classified as a bitmask rather than a value.
+ :c:macro:`RISCV_HWPROBE_KEY_MISALIGNED_SCALAR_PERF`, but the key was
+ mistakenly classified as a bitmask rather than a value.
* :c:macro:`RISCV_HWPROBE_KEY_MISALIGNED_SCALAR_PERF`: An enum value describing
the performance of misaligned scalar native word accesses on the selected set
* :c:macro:`RISCV_HWPROBE_KEY_TIME_CSR_FREQ`: Frequency (in Hz) of `time CSR`.
* :c:macro:`RISCV_HWPROBE_KEY_MISALIGNED_VECTOR_PERF`: An enum value describing the
- performance of misaligned vector accesses on the selected set of processors.
+ performance of misaligned vector accesses on the selected set of processors.
* :c:macro:`RISCV_HWPROBE_MISALIGNED_VECTOR_UNKNOWN`: The performance of misaligned
vector accesses is unknown.
* MIPS
* :c:macro:`RISCV_HWPROBE_VENDOR_EXT_XMIPSEXECTL`: The xmipsexectl vendor
- extension is supported in the MIPS ISA extensions spec.
+ extension is supported in the MIPS ISA extensions spec.
* :c:macro:`RISCV_HWPROBE_KEY_VENDOR_EXT_THEAD_0`: A bitmask containing the
thead vendor extensions that are compatible with the
* T-HEAD
* :c:macro:`RISCV_HWPROBE_VENDOR_EXT_XTHEADVECTOR`: The xtheadvector vendor
- extension is supported in the T-Head ISA extensions spec starting from
- commit a18c801634 ("Add T-Head VECTOR vendor extension. ").
+ extension is supported in the T-Head ISA extensions spec starting from
+ commit a18c801634 ("Add T-Head VECTOR vendor extension. ").
* :c:macro:`RISCV_HWPROBE_KEY_ZICBOM_BLOCK_SIZE`: An unsigned int which
represents the size of the Zicbom block in bytes.
* SIFIVE
* :c:macro:`RISCV_HWPROBE_VENDOR_EXT_XSFVQMACCDOD`: The Xsfqmaccdod vendor
- extension is supported in version 1.1 of SiFive Int8 Matrix Multiplication
- Extensions Specification.
+ extension is supported in version 1.1 of SiFive Int8 Matrix Multiplication
+ Extensions Specification.
* :c:macro:`RISCV_HWPROBE_VENDOR_EXT_XSFVQMACCQOQ`: The Xsfqmaccqoq vendor
- extension is supported in version 1.1 of SiFive Int8 Matrix Multiplication
- Instruction Extensions Specification.
+ extension is supported in version 1.1 of SiFive Int8 Matrix Multiplication
+ Instruction Extensions Specification.
* :c:macro:`RISCV_HWPROBE_VENDOR_EXT_XSFVFNRCLIPXFQF`: The Xsfvfnrclipxfqf
- vendor extension is supported in version 1.0 of SiFive FP32-to-int8 Ranged
- Clip Instructions Extensions Specification.
+ vendor extension is supported in version 1.0 of SiFive FP32-to-int8 Ranged
+ Clip Instructions Extensions Specification.
* :c:macro:`RISCV_HWPROBE_VENDOR_EXT_XSFVFWMACCQQQ`: The Xsfvfwmaccqqq
- vendor extension is supported in version 1.0 of Matrix Multiply Accumulate
- Instruction Extensions Specification.
+ vendor extension is supported in version 1.0 of Matrix Multiply Accumulate
+ Instruction Extensions Specification.
* :c:macro:`RISCV_HWPROBE_KEY_ZICBOP_BLOCK_SIZE`: An unsigned int which
represents the size of the Zicbop block in bytes.
* :c:macro:`RISCV_HWPROBE_KEY_IMA_EXT_1`: A bitmask containing additional
extensions that are compatible with the
:c:macro:`RISCV_HWPROBE_BASE_BEHAVIOR_IMA`: base system behavior.
+
+ * :c:macro:`RISCV_HWPROBE_EXT_ZICFISS`: The Zicfiss extension is supported,
+ as defined in version 1.0 of the RISC-V Control-flow Integrity (CFI)
+ extensions specification, ratified in commit 302a2d45c243
+ ("Update build-pdf.yml") of riscv-cfi.
AES
---
-Support for the AES block cipher.
+This API provides support for the AES block cipher.
.. kernel-doc:: include/crypto/aes.h
DES
---
-Support for the DES block cipher. This algorithm is obsolete and is supported
-only for backwards compatibility.
+This API provides support for the DES block cipher. This algorithm is obsolete
+and is supported only for backwards compatibility.
.. kernel-doc:: include/crypto/des.h
AES-CMAC and AES-XCBC-MAC
-------------------------
-Support for the AES-CMAC and AES-XCBC-MAC message authentication codes.
+This API provides support for the AES-CMAC and AES-XCBC-MAC message
+authentication codes.
.. kernel-doc:: include/crypto/aes-cbc-macs.h
BLAKE2b
-------
-Support for the BLAKE2b cryptographic hash function.
+This API provides support for the BLAKE2b cryptographic hash function.
.. kernel-doc:: include/crypto/blake2b.h
BLAKE2s
-------
-Support for the BLAKE2s cryptographic hash function.
+This API provides support for the BLAKE2s cryptographic hash function.
.. kernel-doc:: include/crypto/blake2s.h
GHASH and POLYVAL
-----------------
-Support for the GHASH and POLYVAL universal hash functions. These algorithms
-are used only as internal components of other algorithms.
+This API provides support for the GHASH and POLYVAL universal hash functions.
+These algorithms are used only as internal components of other algorithms.
.. kernel-doc:: include/crypto/gf128hash.h
MD5
---
-Support for the MD5 cryptographic hash function and HMAC-MD5. This algorithm is
-obsolete and is supported only for backwards compatibility.
+This API provides support for the MD5 cryptographic hash function and HMAC-MD5.
+This algorithm is obsolete and is supported only for backwards compatibility.
.. kernel-doc:: include/crypto/md5.h
NH
--
-Support for the NH universal hash function. This algorithm is used only as an
-internal component of other algorithms.
+This API provides support for the NH universal hash function. This algorithm is
+used only as an internal component of other algorithms.
.. kernel-doc:: include/crypto/nh.h
Poly1305
--------
-Support for the Poly1305 universal hash function. This algorithm is used only
-as an internal component of other algorithms.
+This API provides support for the Poly1305 universal hash function. This
+algorithm is used only as an internal component of other algorithms.
.. kernel-doc:: include/crypto/poly1305.h
SHA-1
-----
-Support for the SHA-1 cryptographic hash function and HMAC-SHA1. This algorithm
-is obsolete and is supported only for backwards compatibility.
+This API provides support for the SHA-1 cryptographic hash function and
+HMAC-SHA1. This algorithm is obsolete and is supported only for backwards
+compatibility.
.. kernel-doc:: include/crypto/sha1.h
SHA-2
-----
-Support for the SHA-2 family of cryptographic hash functions, including SHA-224,
-SHA-256, SHA-384, and SHA-512. This also includes their corresponding HMACs:
-HMAC-SHA224, HMAC-SHA256, HMAC-SHA384, and HMAC-SHA512.
+This API provides support for the SHA-2 family of cryptographic hash functions,
+including SHA-224, SHA-256, SHA-384, and SHA-512. This also includes their
+corresponding HMACs: HMAC-SHA224, HMAC-SHA256, HMAC-SHA384, and HMAC-SHA512.
.. kernel-doc:: include/crypto/sha2.h
SHA-3
-----
-The SHA-3 functions are documented in :ref:`sha3`.
+The SHA-3 API is documented in :ref:`sha3`.
SM3
---
-Support for the SM3 cryptographic hash function.
+This API provides support for the SM3 cryptographic hash function.
.. kernel-doc:: include/crypto/sm3.h
ML-DSA
------
-Support for the ML-DSA digital signature algorithm.
+This API provides support for the ML-DSA digital signature algorithm.
.. kernel-doc:: include/crypto/mldsa.h
Crypto library
==============
-``lib/crypto/`` provides faster and easier access to cryptographic algorithms
-than the traditional crypto API.
+The Linux kernel's crypto library (``lib/crypto/``) provides kernel-internal
+users of cryptographic algorithms with faster and easier access to those
+algorithms than the traditional kernel crypto API.
Each cryptographic algorithm is supported via a set of dedicated functions.
"Crypto agility", where needed, is left to calling code.
- mediatek,mt8183-mmc
- mediatek,mt8186-mmc
- mediatek,mt8188-mmc
- - mediatek,mt8189-mmc
- mediatek,mt8195-mmc
- mediatek,mt8196-mmc
- mediatek,mt8516-mmc
- const: axi_cg
- const: ahb_cg
+ - if:
+ properties:
+ compatible:
+ contains:
+ const: mediatek,mt8189-mmc
+ then:
+ properties:
+ clocks:
+ minItems: 6
+ items:
+ - description: source clock
+ - description: HCLK which used for host
+ - description: independent source clock gate
+ - description: bus clock used for internal register access
+ - description: peripheral bus clock gate
+ - description: AXI bus clock gate
+ - description: crypto clock used for data encrypt/decrypt (optional)
+ clock-names:
+ minItems: 6
+ items:
+ - const: source
+ - const: hclk
+ - const: source_cg
+ - const: bus_clk
+ - const: pclk_cg
+ - const: axi_cg
+ - const: crypto
+
unevaluatedProperties: false
examples:
merged in the riscv-isa-manual by commit dbc79cf28a2 ("Initial seed
of zc.adoc to src tree.").
+ - const: zclsd
+ description:
+ The Zclsd extension implements the compressed (16-bit) version of the
+ Load/Store Pair for RV32. As with Zilsd, this extension was ratified
+ in commit f88abf1 ("Integrating load/store pair for RV32 with the
+ main manual") of riscv-isa-manual.
+
- const: zcmop
description:
The standard Zcmop extension version 1.0, as ratified in commit
in commit 64074bc ("Update version numbers for Zfh/Zfinx") of
riscv-isa-manual.
+ - const: zicbom
+ description:
+ The standard Zicbom extension for base cache management operations as
+ ratified in commit 3dd606f ("Create cmobase-v1.0.pdf") of riscv-CMOs.
+
+ - const: zicbop
+ description:
+ The standard Zicbop extension for cache-block prefetch instructions
+ as ratified in commit 3dd606f ("Create cmobase-v1.0.pdf") of
+ riscv-CMOs.
+
+ - const: zicboz
+ description:
+ The standard Zicboz extension for cache-block zeroing as ratified
+ in commit 3dd606f ("Create cmobase-v1.0.pdf") of riscv-CMOs.
+
- const: ziccamoa
description:
The standard Ziccamoa extension for main memory (cacheability and
guarantee on LR/SC sequences, as ratified in commit b1d806605f87
("Updated to ratified state.") of the riscv profiles specification.
+ - const: zicfilp
+ description: |
+ The standard Zicfilp extension for enforcing forward edge
+ control-flow integrity as ratified in commit 3f8e450 ("merge
+ pull request #227 from ved-rivos/0709") of riscv-cfi
+ github repo.
+
+ - const: zicfiss
+ description: |
+ The standard Zicfiss extension for enforcing backward edge
+ control-flow integrity as ratified in commit 3f8e450 ("merge
+ pull request #227 from ved-rivos/0709") of riscv-cfi
+ github repo.
+
+ - const: zicntr
+ description:
+ The standard Zicntr extension for base counters and timers, as
+ ratified in the 20191213 version of the unprivileged ISA
+ specification.
+
+ - const: zicond
+ description:
+ The standard Zicond extension for conditional arithmetic and
+ conditional-select/move operations as ratified in commit 95cf1f9
+ ("Add changes requested by Ved during signoff") of riscv-zicond.
+
+ - const: zicsr
+ description: |
+ The standard Zicsr extension for control and status register
+ instructions, as ratified in the 20191213 version of the
+ unprivileged ISA specification.
+
+ This does not include Chapter 10, "Counters", which documents
+ special case read-only CSRs, that were moved into the Zicntr and
+ Zihpm extensions after the ratification of the 20191213 version of
+ the unprivileged specification.
+
+ - const: zifencei
+ description:
+ The standard Zifencei extension for instruction-fetch fence, as
+ ratified in the 20191213 version of the unprivileged ISA
+ specification.
+
+ - const: zihintntl
+ description:
+ The standard Zihintntl extension for non-temporal locality hints, as
+ ratified in commit 0dc91f5 ("Zihintntl is ratified") of the
+ riscv-isa-manual.
+
+ - const: zihintpause
+ description:
+ The standard Zihintpause extension for pause hints, as ratified in
+ commit d8ab5c7 ("Zihintpause is ratified") of the riscv-isa-manual.
+
+ - const: zihpm
+ description:
+ The standard Zihpm extension for hardware performance counters, as
+ ratified in the 20191213 version of the unprivileged ISA
+ specification.
+
- const: zilsd
description:
The standard Zilsd extension which provides support for aligned
encodings, as ratified in commit f88abf1 ("Integrating
load/store pair for RV32 with the main manual") of riscv-isa-manual.
- - const: zclsd
+ - const: zimop
description:
- The Zclsd extension implements the compressed (16-bit) version of the
- Load/Store Pair for RV32. As with Zilsd, this extension was ratified
- in commit f88abf1 ("Integrating load/store pair for RV32 with the
- main manual") of riscv-isa-manual.
+ The standard Zimop extension version 1.0, as ratified in commit
+ 58220614a5f ("Zimop is ratified/1.0") of the riscv-isa-manual.
- const: zk
description:
in version 1.0 of RISC-V Cryptography Extensions Volume I
specification.
- - const: zicbom
- description:
- The standard Zicbom extension for base cache management operations as
- ratified in commit 3dd606f ("Create cmobase-v1.0.pdf") of riscv-CMOs.
-
- - const: zicbop
- description:
- The standard Zicbop extension for cache-block prefetch instructions
- as ratified in commit 3dd606f ("Create cmobase-v1.0.pdf") of
- riscv-CMOs.
-
- - const: zicboz
- description:
- The standard Zicboz extension for cache-block zeroing as ratified
- in commit 3dd606f ("Create cmobase-v1.0.pdf") of riscv-CMOs.
-
- - const: zicfilp
- description: |
- The standard Zicfilp extension for enforcing forward edge
- control-flow integrity as ratified in commit 3f8e450 ("merge
- pull request #227 from ved-rivos/0709") of riscv-cfi
- github repo.
-
- - const: zicfiss
- description: |
- The standard Zicfiss extension for enforcing backward edge
- control-flow integrity as ratified in commit 3f8e450 ("merge
- pull request #227 from ved-rivos/0709") of riscv-cfi
- github repo.
-
- - const: zicntr
- description:
- The standard Zicntr extension for base counters and timers, as
- ratified in the 20191213 version of the unprivileged ISA
- specification.
-
- - const: zicond
- description:
- The standard Zicond extension for conditional arithmetic and
- conditional-select/move operations as ratified in commit 95cf1f9
- ("Add changes requested by Ved during signoff") of riscv-zicond.
-
- - const: zicsr
- description: |
- The standard Zicsr extension for control and status register
- instructions, as ratified in the 20191213 version of the
- unprivileged ISA specification.
-
- This does not include Chapter 10, "Counters", which documents
- special case read-only CSRs, that were moved into the Zicntr and
- Zihpm extensions after the ratification of the 20191213 version of
- the unprivileged specification.
-
- - const: zifencei
- description:
- The standard Zifencei extension for instruction-fetch fence, as
- ratified in the 20191213 version of the unprivileged ISA
- specification.
-
- - const: zihintpause
- description:
- The standard Zihintpause extension for pause hints, as ratified in
- commit d8ab5c7 ("Zihintpause is ratified") of the riscv-isa-manual.
-
- - const: zihintntl
- description:
- The standard Zihintntl extension for non-temporal locality hints, as
- ratified in commit 0dc91f5 ("Zihintntl is ratified") of the
- riscv-isa-manual.
-
- - const: zihpm
- description:
- The standard Zihpm extension for hardware performance counters, as
- ratified in the 20191213 version of the unprivileged ISA
- specification.
-
- - const: zimop
- description:
- The standard Zimop extension version 1.0, as ratified in commit
- 58220614a5f ("Zimop is ratified/1.0") of the riscv-isa-manual.
-
- const: ztso
description:
The standard Ztso extension for total store ordering, as ratified
instructions, as ratified in commit 56ed795 ("Update
riscv-crypto-spec-vector.adoc") of riscv-crypto.
- - const: zvksh
- description: |
- The standard Zvksh extension for ShangMi suite: SM3 secure hash
- instructions, as ratified in commit 56ed795 ("Update
- riscv-crypto-spec-vector.adoc") of riscv-crypto.
-
- const: zvksg
description:
The standard Zvksg extension for ShangMi algorithm suite with GCM
instructions, as ratified in commit 56ed795 ("Update
riscv-crypto-spec-vector.adoc") of riscv-crypto.
+ - const: zvksh
+ description: |
+ The standard Zvksh extension for ShangMi suite: SM3 secure hash
+ instructions, as ratified in commit 56ed795 ("Update
+ riscv-crypto-spec-vector.adoc") of riscv-crypto.
+
- const: zvkt
description:
The standard Zvkt extension for vector data-independent execution
- qcom,sc8280xp-lpass-rx-macro
- items:
- enum:
+ - qcom,eliza-lpass-rx-macro
- qcom,kaanapali-lpass-rx-macro
- qcom,sm8650-lpass-rx-macro
- qcom,sm8750-lpass-rx-macro
- qcom,sc8280xp-lpass-tx-macro
- items:
- enum:
+ - qcom,eliza-lpass-tx-macro
- qcom,kaanapali-lpass-tx-macro
- qcom,sm8650-lpass-tx-macro
- qcom,sm8750-lpass-tx-macro
- qcom,sc8280xp-lpass-va-macro
- items:
- enum:
+ - qcom,eliza-lpass-va-macro
- qcom,glymur-lpass-va-macro
- qcom,kaanapali-lpass-va-macro
- qcom,sm8650-lpass-va-macro
- qcom,sc8280xp-lpass-wsa-macro
- items:
- enum:
+ - qcom,eliza-lpass-wsa-macro
- qcom,glymur-lpass-wsa-macro
- qcom,kaanapali-lpass-wsa-macro
- qcom,sm8650-lpass-wsa-macro
- const: qcom,sdm845-sndcard
- items:
- enum:
+ - qcom,eliza-sndcard
- qcom,kaanapali-sndcard
- qcom,sm8550-sndcard
- qcom,sm8650-sndcard
Google Kees Cook <keescook@chromium.org>
- LLVM Nick Desaulniers <nick.desaulniers+lkml@gmail.com>
+ LLVM Nick Desaulniers <ndesaulniers@google.com>
============= ========================================================
If you want your organization to be added to the ambassadors list, please
Where to Look
=============
-* ``include/linux/sched/ext.h`` defines the core data structures, ops table
- and constants.
+* ``include/linux/sched/ext.h`` defines the core data structures and
+ constants, while the ops table (``struct sched_ext_ops``) is defined in
+ ``kernel/sched/ext/internal.h``.
* ``kernel/sched/ext/ext.c`` contains sched_ext core implementation and helpers.
The functions prefixed with ``scx_bpf_`` can be called from the BPF
===============
The APIs provided by sched_ext to BPF schedulers programs have no stability
-guarantees. This includes the ops table callbacks and constants defined in
+guarantees. This includes the ops table callbacks defined in
+``kernel/sched/ext/internal.h`` and the constants defined in
``include/linux/sched/ext.h``, as well as the ``scx_bpf_`` kfuncs defined in
``kernel/sched/ext/ext.c`` and ``kernel/sched/ext/idle.c``.
Google Kees Cook <keescook@chromium.org>
- LLVM Nick Desaulniers <nick.desaulniers+lkml@gmail.com>
+ LLVM Nick Desaulniers <ndesaulniers@google.com>
============= ========================================================
Si quiere que su organización se añada a la lista de embajadores, por
F: drivers/iio/adc/ep93xx_adc.c
ARM/CIX SOC SUPPORT
-M: Peter Chen <peter.chen@cixtech.com>
+M: Gary Yang <gary.yang@cixtech.com>
M: Fugang Duan <fugang.duan@cixtech.com>
R: CIX Linux Kernel Upstream Group <cix-kernel-upstream@cixtech.com>
L: linux-arm-kernel@lists.infradead.org (moderated for non-subscribers)
S: Maintained
-T: git git://git.kernel.org/pub/scm/linux/kernel/git/peter.chen/cix.git
+T: git https://github.com/cixtech/linux-mainline.git
F: Documentation/devicetree/bindings/arm/cix.yaml
F: Documentation/devicetree/bindings/mailbox/cix,sky1-mbox.yaml
F: arch/arm64/boot/dts/cix/
F: include/linux/mfd/cs42l43*
F: include/sound/cs*
F: sound/hda/codecs/cirrus*
+F: sound/hda/codecs/side-codecs/cirrus*
F: sound/hda/codecs/side-codecs/cs*
F: sound/hda/codecs/side-codecs/hda_component*
F: sound/soc/codecs/cs*
CLANG/LLVM BUILD SUPPORT
M: Nathan Chancellor <nathan@kernel.org>
-R: Nick Desaulniers <nick.desaulniers+lkml@gmail.com>
+R: Nick Desaulniers <ndesaulniers@google.com>
R: Bill Wendling <morbo@google.com>
R: Justin Stitt <justinstitt@google.com>
L: llvm@lists.linux.dev
F: drivers/platform/x86/dasharo-acpi.c
DAMON
-M: SeongJae Park <sj@kernel.org>
+M: SJ Park <sj@kernel.org>
L: damon@lists.linux.dev
L: linux-mm@kvack.org
S: Maintained
KERNEL VIRTUAL MACHINE FOR ARM64 (KVM/arm64)
M: Marc Zyngier <maz@kernel.org>
M: Oliver Upton <oupton@kernel.org>
+R: Fuad Tabba <tabba@google.com>
R: Joey Gouly <joey.gouly@arm.com>
R: Steffen Eiden <seiden@linux.ibm.com>
R: Suzuki K Poulose <suzuki.poulose@arm.com>
R: Vlastimil Babka <vbabka@kernel.org>
R: Harry Yoo <harry@kernel.org>
R: Jann Horn <jannh@google.com>
+R: Lance Yang <lance.yang@linux.dev>
L: linux-mm@kvack.org
S: Maintained
F: include/linux/rmap.h
R: Dev Jain <dev.jain@arm.com>
R: Barry Song <baohua@kernel.org>
R: Lance Yang <lance.yang@linux.dev>
+R: Usama Arif <usama.arif@linux.dev>
L: linux-mm@kvack.org
S: Maintained
W: http://www.linux-mm.org
F: drivers/net/phy/ncn*
OP-TEE DRIVER
-M: Jens Wiklander <jens.wiklander@linaro.org>
+M: Jens Wiklander <jenswi@kernel.org>
L: op-tee@lists.trustedfirmware.org (moderated for non-subscribers)
S: Maintained
F: Documentation/ABI/testing/sysfs-bus-optee-devices
PAGE CACHE
M: Matthew Wilcox (Oracle) <willy@infradead.org>
-R: Jan Kara <jack@suse.cz>
+M: Jan Kara <jack@suse.cz>
L: linux-fsdevel@vger.kernel.org
L: linux-mm@kvack.org
S: Supported
S: Maintained
W: https://github.com/spacemit-com/linux/wiki
C: irc://irc.libera.chat/spacemit
-T: git https://github.com/spacemit-com/linux
+T: https://git.kernel.org/pub/scm/linux/kernel/git/spacemit/linux.git
F: arch/riscv/boot/dts/spacemit/
N: spacemit
K: spacemit
F: drivers/media/platform/sunxi/sun8i-rotate/
RPMB SUBSYSTEM
-M: Jens Wiklander <jens.wiklander@linaro.org>
+M: Jens Wiklander <jenswi@kernel.org>
L: linux-kernel@vger.kernel.org
S: Supported
F: drivers/misc/rpmb-core.c
F: include/media/i2c/tw9910.h
TEE SUBSYSTEM
-M: Jens Wiklander <jens.wiklander@linaro.org>
+M: Jens Wiklander <jenswi@kernel.org>
R: Sumit Garg <sumit.garg@kernel.org>
L: op-tee@lists.trustedfirmware.org (moderated for non-subscribers)
S: Maintained
F: include/linux/usb.h
F: include/linux/usb/
F: include/uapi/linux/usb/
+F: rust/kernel/usb.rs
USB TYPEC BUS FOR ALTERNATE MODES
M: Heikki Krogerus <heikki.krogerus@linux.intel.com>
VERSION = 7
PATCHLEVEL = 2
SUBLEVEL = 0
-EXTRAVERSION = -rc2
+EXTRAVERSION = -rc4
NAME = Baby Opossum Posse
# *DOCUMENTATION*
CONFIG_I2C=y
CONFIG_I2C_CHARDEV=y
CONFIG_I2C_DESIGNWARE_CORE=y
-CONFIG_I2C_DESIGNWARE_PLATFORM=y
# CONFIG_HWMON is not set
CONFIG_DRM=m
CONFIG_DRM_I2C_ADV7511=m
CONFIG_I2C=y
CONFIG_I2C_CHARDEV=y
CONFIG_I2C_DESIGNWARE_CORE=y
-CONFIG_I2C_DESIGNWARE_PLATFORM=y
# CONFIG_HWMON is not set
CONFIG_FB=y
CONFIG_FRAMEBUFFER_CONSOLE=y
CONFIG_I2C=y
CONFIG_I2C_CHARDEV=y
CONFIG_I2C_DESIGNWARE_CORE=y
-CONFIG_I2C_DESIGNWARE_PLATFORM=y
# CONFIG_HWMON is not set
CONFIG_DRM=m
CONFIG_DRM_I2C_ADV7511=m
CONFIG_I2C=y
# CONFIG_I2C_COMPAT is not set
CONFIG_I2C_DESIGNWARE_CORE=y
-CONFIG_I2C_DESIGNWARE_PLATFORM=y
CONFIG_GPIO_SYSFS=y
# CONFIG_HWMON is not set
# CONFIG_USB_SUPPORT is not set
#include <linux/irqdomain.h>
#include <linux/export.h>
#include <linux/of_fdt.h>
+#include <linux/string.h>
#include <asm/mach_desc.h>
#include <asm/setup.h>
{
unsigned long dt_root = of_get_flat_dt_root();
const char *buf;
+ int len;
- buf = of_get_flat_dt_prop(dt_root, name, NULL);
- if (!buf)
+ buf = of_get_flat_dt_prop(dt_root, name, &len);
+ if (!buf || !memchr(buf, '\0', len))
return -EINVAL;
if (cpulist_parse(buf, cpumask))
select IXP4XX_TIMER
select USB_EHCI_BIG_ENDIAN_DESC
select USB_EHCI_BIG_ENDIAN_MMIO
- select USE_OF
help
Support for Intel's IXP4XX (XScale) family of processors.
select PINCTRL
select POWER_SUPPLY
select PXA25x
- select USE_OF
help
Include support for Marvell PXA25x based platforms using
the device tree. Needn't select any other machine while
select PINCTRL
select POWER_SUPPLY
select PXA27x
- select USE_OF
help
Include support for Marvell PXA27x based platforms using
the device tree. Needn't select any other machine while
select PINCTRL
select POWER_SUPPLY
select PXA3xx
- select USE_OF
help
Include support for Marvell PXA3xx based platforms using
the device tree. Needn't select any other machine while
#size-cells = <0>;
cpu0_0: cpu@0 {
- compatible = "arm,cortex-a78";
+ compatible = "arm,cortex-a78ae";
device_type = "cpu";
reg = <0x00000>;
};
cpu0_1: cpu@100 {
- compatible = "arm,cortex-a78";
+ compatible = "arm,cortex-a78ae";
device_type = "cpu";
reg = <0x00100>;
};
cpu0_2: cpu@200 {
- compatible = "arm,cortex-a78";
+ compatible = "arm,cortex-a78ae";
device_type = "cpu";
reg = <0x00200>;
};
cpu0_3: cpu@300 {
- compatible = "arm,cortex-a78";
+ compatible = "arm,cortex-a78ae";
device_type = "cpu";
reg = <0x00300>;
};
cpu1_0: cpu@10000 {
- compatible = "arm,cortex-a78";
+ compatible = "arm,cortex-a78ae";
device_type = "cpu";
reg = <0x10000>;
};
cpu1_1: cpu@10100 {
- compatible = "arm,cortex-a78";
+ compatible = "arm,cortex-a78ae";
device_type = "cpu";
reg = <0x10100>;
};
cpu1_2: cpu@10200 {
- compatible = "arm,cortex-a78";
+ compatible = "arm,cortex-a78ae";
device_type = "cpu";
reg = <0x10200>;
};
cpu1_3: cpu@10300 {
- compatible = "arm,cortex-a78";
+ compatible = "arm,cortex-a78ae";
device_type = "cpu";
reg = <0x10300>;
};
cpu2_0: cpu@20000 {
- compatible = "arm,cortex-a78";
+ compatible = "arm,cortex-a78ae";
device_type = "cpu";
reg = <0x20000>;
};
cpu2_1: cpu@20100 {
- compatible = "arm,cortex-a78";
+ compatible = "arm,cortex-a78ae";
device_type = "cpu";
reg = <0x20100>;
};
cpu2_2: cpu@20200 {
- compatible = "arm,cortex-a78";
+ compatible = "arm,cortex-a78ae";
device_type = "cpu";
reg = <0x20200>;
};
cpu2_3: cpu@20300 {
- compatible = "arm,cortex-a78";
+ compatible = "arm,cortex-a78ae";
device_type = "cpu";
reg = <0x20300>;
};
gpcdma: dma-controller@8400000 {
- compatible = "nvidia,tegra264-gpcdma", "nvidia,tegra186-gpcdma";
+ compatible = "nvidia,tegra264-gpcdma";
reg = <0x0 0x08400000 0x0 0x210000>;
interrupts = <GIC_SPI 584 IRQ_TYPE_LEVEL_HIGH>,
<GIC_SPI 585 IRQ_TYPE_LEVEL_HIGH>,
d-cache-sets = <256>;
};
- cpu1: cpu@1 {
+ cpu1: cpu@10000 {
compatible = "arm,neoverse-v3ae";
device_type = "cpu";
reg = <0x10000>;
reg = <0x0 0x8c400000 0x0 0x02000000>;
no-map;
};
+
+ /* DRAM controller inline ECC areas */
+ ecc@10cccc0000 {
+ reg = <0x10 0xcccc0000 0x0 0x33340000>;
+ no-map;
+ };
+
+ ecc@12cccc0000 {
+ reg = <0x12 0xcccc0000 0x0 0x33340000>;
+ no-map;
+ };
+
+ ecc@14cccc0000 {
+ reg = <0x14 0xcccc0000 0x0 0x33340000>;
+ no-map;
+ };
+
+ ecc@16cccc0000 {
+ reg = <0x16 0xcccc0000 0x0 0x33340000>;
+ no-map;
+ };
+
+ ecc@18cccc0000 {
+ reg = <0x18 0xcccc0000 0x0 0x33340000>;
+ no-map;
+ };
+
+ ecc@1a66660000 {
+ reg = <0x1a 0x66660000 0x0 0x999a0000>;
+ no-map;
+ };
+
+ ecc@1c66660000 {
+ reg = <0x1c 0x66660000 0x0 0x999a0000>;
+ no-map;
+ };
+
+ ecc@1e66660000 {
+ reg = <0x1e 0x66660000 0x0 0x999a0000>;
+ no-map;
+ };
};
};
bool failed;
};
+static inline void fail_s1_walk(struct s1_walk_result *wr, u8 fst, bool s1ptw)
+{
+ wr->fst = fst;
+ wr->ptw = s1ptw;
+ wr->s2 = s1ptw;
+ wr->failed = true;
+}
+
int __kvm_translate_va(struct kvm_vcpu *vcpu, struct s1_walk_info *wi,
struct s1_walk_result *wr, u64 va);
int __kvm_find_s1_desc_level(struct kvm_vcpu *vcpu, u64 va, u64 ipa,
#ifndef _ARCH_ARM64_TLBBATCH_H
#define _ARCH_ARM64_TLBBATCH_H
-#include <linux/cpumask.h>
-
struct arch_tlbflush_unmap_batch {
-#ifdef CONFIG_ARM64_ERRATUM_4193714
/*
- * Track CPUs that need SME DVMSync on completion of this batch.
- * Otherwise, the arm64 HW can do tlb shootdown, so we don't need to
- * record cpumask for sending IPI
+ * For arm64, HW can do TLB shootdown, so we don't need to record a
+ * cpumask for sending IPIs.
*/
- cpumask_var_t cpumask;
-#endif
};
#endif /* _ARCH_ARM64_TLBBATCH_H */
#ifdef CONFIG_ARM64_ERRATUM_4193714
+extern cpumask_t sme_active_cpus;
+
void sme_do_dvmsync(const struct cpumask *mask);
static inline void sme_dvmsync(struct mm_struct *mm)
sme_do_dvmsync(mm_cpumask(mm));
}
-static inline void sme_dvmsync_add_pending(struct arch_tlbflush_unmap_batch *batch,
- struct mm_struct *mm)
+static inline void sme_dvmsync_batch(void)
{
if (!alternative_has_cap_unlikely(ARM64_WORKAROUND_4193714))
return;
- /*
- * Order the mm_cpumask() read after the hardware DVMSync.
- */
- dsb(ish);
- if (cpumask_empty(mm_cpumask(mm)))
- return;
-
- /*
- * Allocate the batch cpumask on first use. Fall back to an immediate
- * IPI for this mm in case of failure.
- */
- if (!cpumask_available(batch->cpumask) &&
- !zalloc_cpumask_var(&batch->cpumask, GFP_ATOMIC)) {
- sme_do_dvmsync(mm_cpumask(mm));
- return;
- }
-
- cpumask_or(batch->cpumask, batch->cpumask, mm_cpumask(mm));
-}
-
-static inline void sme_dvmsync_batch(struct arch_tlbflush_unmap_batch *batch)
-{
- if (!alternative_has_cap_unlikely(ARM64_WORKAROUND_4193714))
- return;
-
- if (!cpumask_available(batch->cpumask))
- return;
-
- sme_do_dvmsync(batch->cpumask);
- cpumask_clear(batch->cpumask);
+ sme_do_dvmsync(&sme_active_cpus);
}
#else
static inline void sme_dvmsync(struct mm_struct *mm)
{
}
-static inline void sme_dvmsync_add_pending(struct arch_tlbflush_unmap_batch *batch,
- struct mm_struct *mm)
-{
-}
-static inline void sme_dvmsync_batch(struct arch_tlbflush_unmap_batch *batch)
+static inline void sme_dvmsync_batch(void)
{
}
sme_dvmsync(mm);
}
-static inline void __tlbi_sync_s1ish_batch(struct arch_tlbflush_unmap_batch *batch)
+static inline void __tlbi_sync_s1ish_batch(void)
{
dsb(ish);
__repeat_tlbi_sync(vale1is, 0);
- sme_dvmsync_batch(batch);
+ sme_dvmsync_batch();
}
static inline void __tlbi_sync_s1ish_kernel(void)
*/
static inline void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch)
{
- __tlbi_sync_s1ish_batch(batch);
+ __tlbi_sync_s1ish_batch();
}
/*
__flush_tlb_range(&vma, start, end, PAGE_SIZE, 3,
TLBF_NOWALKCACHE | TLBF_NOSYNC);
- sme_dvmsync_add_pending(batch, mm);
}
static inline bool __pte_flags_need_flush(ptval_t oldval, ptval_t newval)
return *pcpu;
}
+ set_cpu_present(*pcpu, true);
return 0;
}
EXPORT_SYMBOL(acpi_map_cpu);
int acpi_unmap_cpu(int cpu)
{
+ set_cpu_present(cpu, false);
return 0;
}
EXPORT_SYMBOL(acpi_unmap_cpu);
* SME/CME erratum handling.
*/
static cpumask_t sme_dvmsync_cpus;
+cpumask_t sme_active_cpus;
/*
* These helpers are only called from non-preemptible contexts, so
return;
cpumask_set_cpu(cpu, mm_cpumask(current->mm));
+ cpumask_set_cpu(cpu, &sme_active_cpus);
/*
* A subsequent (post ERET) SME access may use a stale address
* translation. On C1-Pro, a TLBI+DSB on a different CPU will wait for
- * the completion of cpumask_set_cpu() above as it appears in program
- * order before the SME access. The post-TLBI+DSB read of mm_cpumask()
- * will lead to the IPI being issued.
+ * the completion of the cpumask_set_cpu() operations above as they
+ * appear in program order before the SME access. The post-TLBI+DSB
+ * read of mm_cpumask() or sme_active_cpus will lead to the IPI being
+ * issued.
*
* https://lore.kernel.org/r/ablEXwhfKyJW1i7l@J2N7QTR9R3
*/
* completed on entering EL1.
*/
cpumask_clear_cpu(cpu, mm_cpumask(current->mm));
+ cpumask_clear_cpu(cpu, &sme_active_cpus);
}
static void sme_dvmsync_ipi(void *unused)
flush_gcs();
}
-#ifdef CONFIG_ARM64_ERRATUM_4193714
-
-static void arch_dup_tlbbatch_mask(struct task_struct *dst)
-{
- /*
- * Clear the inherited cpumask with memset() to cover both cases where
- * cpumask_var_t is a pointer or an array. It will be allocated lazily
- * in sme_dvmsync_add_pending() if CPUMASK_OFFSTACK=y.
- */
- if (alternative_has_cap_unlikely(ARM64_WORKAROUND_4193714))
- memset(&dst->tlb_ubc.arch.cpumask, 0,
- sizeof(dst->tlb_ubc.arch.cpumask));
-}
-
-static void arch_release_tlbbatch_mask(struct task_struct *tsk)
-{
- if (alternative_has_cap_unlikely(ARM64_WORKAROUND_4193714))
- free_cpumask_var(tsk->tlb_ubc.arch.cpumask);
-}
-
-#else
-
-static void arch_dup_tlbbatch_mask(struct task_struct *dst)
-{
-}
-
-static void arch_release_tlbbatch_mask(struct task_struct *tsk)
-{
-}
-
-#endif /* CONFIG_ARM64_ERRATUM_4193714 */
-
void arch_release_task_struct(struct task_struct *tsk)
{
- arch_release_tlbbatch_mask(tsk);
fpsimd_release_task(tsk);
}
*dst = *src;
- arch_dup_tlbbatch_mask(dst);
-
/*
* Drop stale reference to src's sve_state and convert dst to
* non-streaming FPSIMD mode.
{
acpi_handle acpi_handle = acpi_get_processor_handle(cpu);
struct cpu *c = &per_cpu(cpu_devices, cpu);
- acpi_status status;
unsigned long long sta;
-
- if (!acpi_handle) {
- pr_err_once("Removing a CPU without associated ACPI handle\n");
- return;
- }
+ acpi_status status;
status = acpi_evaluate_integer(acpi_handle, "_STA", NULL, &sta);
- if (ACPI_FAILURE(status))
- return;
-
- /* For now do not allow anything that looks like physical CPU HP */
- if (cpu_present(cpu) && !(sta & ACPI_STA_DEVICE_PRESENT)) {
+ if (!ACPI_FAILURE(status) &&
+ cpu_present(cpu) && !(sta & ACPI_STA_DEVICE_PRESENT))
pr_err_once("Changing CPU present bit is not supported\n");
- return;
- }
unregister_cpu(c);
}
}
EXPORT_SYMBOL_GPL(acpi_cpu_get_madt_gicc);
+static bool acpi_cpu_is_present(int cpu)
+{
+ return acpi_cpu_get_madt_gicc(cpu)->flags & ACPI_MADT_ENABLED;
+}
+
/*
* acpi_map_gic_cpu_interface - parse processor MADT entry
*
early_map_cpu_to_node(i, acpi_numa_get_nid(i));
}
#else
+static bool acpi_cpu_is_present(int cpu)
+{
+ return false;
+}
#define acpi_parse_and_init_cpus(...) do { } while (0)
#endif
if (err)
continue;
- set_cpu_present(cpu, true);
+ if (acpi_disabled || acpi_cpu_is_present(cpu))
+ set_cpu_present(cpu, true);
numa_store_cpu_info(cpu);
}
}
#include <asm/kvm_mmu.h>
#include <asm/lsui.h>
-static void fail_s1_walk(struct s1_walk_result *wr, u8 fst, bool s1ptw)
-{
- wr->fst = fst;
- wr->ptw = s1ptw;
- wr->s2 = s1ptw;
- wr->failed = true;
-}
-
#define S1_MMU_DISABLED (-127)
static int get_ia_size(struct s1_walk_info *wi)
(spsr & PSR_MODE32_BIT) ||
(vcpu_el2_tge_is_set(vcpu) && (mode == PSR_MODE_EL1t ||
mode == PSR_MODE_EL1h))) {
+ u64 mask;
+
/*
- * The guest is playing with our nerves. Preserve EL, SP,
- * masks, flags from the existing PSTATE, and set IL.
- * The HW will then generate an Illegal State Exception
- * immediately after ERET.
+ * On an illegal exception return, the flags and masks are
+ * taken from the SPSR while PSTATE.{EL,SP,nRW} and, if
+ * FEAT_GCS, PSTATE.EXLOCK are unchanged (R_VWJHB). Set IL
+ * so the HW generates an Illegal State Exception right
+ * after ERET.
*/
- spsr = *vcpu_cpsr(vcpu);
+ mask = PSR_D_BIT | PSR_A_BIT | PSR_I_BIT | PSR_F_BIT |
+ PSR_N_BIT | PSR_Z_BIT | PSR_C_BIT | PSR_V_BIT;
+
+ if (kvm_has_feat(vcpu->kvm, ID_AA64MMFR1_EL1, PAN, IMP))
+ mask |= PSR_PAN_BIT;
+ if (kvm_has_feat(vcpu->kvm, ID_AA64PFR1_EL1, NMI, IMP))
+ mask |= ALLINT_ALLINT;
+ /* FEAT_SPE_EXC and FEAT_TRBE_EXC also gate PSTATE.PM one day... */
+ if (kvm_has_feat(vcpu->kvm, ID_AA64DFR1_EL1, EBEP, IMP))
+ mask |= BIT_ULL(32); /* SPSR_ELx.PM */
+
+ spsr &= mask;
- spsr &= (PSR_D_BIT | PSR_A_BIT | PSR_I_BIT | PSR_F_BIT |
- PSR_N_BIT | PSR_Z_BIT | PSR_C_BIT | PSR_V_BIT |
- PSR_MODE_MASK | PSR_MODE32_BIT);
+ mask = PSR_MODE_MASK | PSR_MODE32_BIT;
+ if (kvm_has_feat(vcpu->kvm, ID_AA64PFR1_EL1, GCS, IMP))
+ mask |= BIT_ULL(34); /* PSTATE.EXLOCK */
+
+ spsr |= *vcpu_cpsr(vcpu) & mask;
spsr |= PSR_IL_BIT;
}
* ERET handling, and the guest will have a little surprise.
*/
if (kvm_has_pauth(vcpu->kvm, FPACCOMBINE) && !(spsr & PSR_IL_BIT)) {
- esr &= ESR_ELx_ERET_ISS_ERETA;
+ esr &= (ESR_ELx_ERET_ISS_ERETA | ESR_ELx_IL);
esr |= FIELD_PREP(ESR_ELx_EC_MASK, ESR_ELx_EC_FPAC);
kvm_inject_nested_sync(vcpu, esr);
return;
break;
case except_type_serror:
kvm_pend_exception(vcpu, EXCEPT_AA64_EL2_SERR);
+ vcpu_write_sys_reg(vcpu, esr_el2, ESR_EL2);
break;
default:
WARN_ONCE(1, "Unsupported EL2 exception injection %d\n", type);
* vSError injection. Manually populate EC for an emulated SError
* exception.
*/
- esr |= FIELD_PREP(ESR_ELx_EC_MASK, ESR_ELx_EC_SERROR);
+ esr |= FIELD_PREP(ESR_ELx_EC_MASK, ESR_ELx_EC_SERROR) | ESR_ELx_IL;
return kvm_inject_nested(vcpu, esr, except_type_serror);
}
static inline bool kvm_hyp_handle_mops(struct kvm_vcpu *vcpu, u64 *exit_code)
{
+ u64 spsr;
+
*vcpu_pc(vcpu) = read_sysreg_el2(SYS_ELR);
arm64_mops_reset_regs(vcpu_gp_regs(vcpu), vcpu->arch.fault.esr_el2);
write_sysreg_el2(*vcpu_pc(vcpu), SYS_ELR);
/*
* Finish potential single step before executing the prologue
- * instruction.
+ * instruction. Modify the hardware SPSR_EL2 directly, as vcpu_cpsr()
+ * may hold a synthetic (vEL2) value for a guest hypervisor.
*/
- *vcpu_cpsr(vcpu) &= ~DBG_SPSR_SS;
- write_sysreg_el2(*vcpu_cpsr(vcpu), SYS_SPSR);
+ spsr = read_sysreg_el2(SYS_SPSR);
+ write_sysreg_el2(spsr & ~DBG_SPSR_SS, SYS_SPSR);
return true;
}
return false;
break;
case ESR_ELx_EC_SYS64:
- if (WARN_ON_ONCE(!is_hyp_ctxt(vcpu)))
- return false;
fallthrough;
case ESR_ELx_EC_SVE:
if (!sve_guest)
u64 sz = (u64)range->pg_cnt * FFA_PAGE_SIZE;
u64 pfn = hyp_phys_to_pfn(range->address);
- if (!PAGE_ALIGNED(sz))
+ if (!PAGE_ALIGNED(sz | range->address))
break;
if (__pkvm_host_share_ffa(pfn, sz / PAGE_SIZE))
u64 sz = (u64)range->pg_cnt * FFA_PAGE_SIZE;
u64 pfn = hyp_phys_to_pfn(range->address);
- if (!PAGE_ALIGNED(sz))
+ if (!PAGE_ALIGNED(sz | range->address))
break;
if (__pkvm_host_unshare_ffa(pfn, sz / PAGE_SIZE))
DECLARE_REG(u32, fraglen, ctxt, 2);
DECLARE_REG(u64, addr_mbz, ctxt, 3);
DECLARE_REG(u32, npages_mbz, ctxt, 4);
+ u32 offset, nr_ranges, checked_offset, em_mem_access_off;
struct ffa_mem_region_attributes *ep_mem_access;
struct ffa_composite_mem_region *reg;
struct ffa_mem_region *buf;
- u32 offset, nr_ranges, checked_offset;
int ret = 0;
+ size_t mem_region_len = FFA_MEM_REGION_SZ(hyp_ffa_version);
if (addr_mbz || npages_mbz || fraglen > len ||
fraglen > KVM_FFA_MBOX_NR_PAGES * PAGE_SIZE) {
goto out;
}
- if (fraglen < sizeof(struct ffa_mem_region) +
- sizeof(struct ffa_mem_region_attributes)) {
+ if (fraglen < mem_region_len + ffa_emad_size_get(hyp_ffa_version)) {
ret = FFA_RET_INVALID_PARAMETERS;
goto out;
}
buf = hyp_buffers.tx;
memcpy(buf, host_buffers.tx, fraglen);
- ep_mem_access = (void *)buf +
- ffa_mem_desc_offset(buf, 0, hyp_ffa_version);
+ em_mem_access_off = ffa_mem_desc_offset(buf, 0, hyp_ffa_version);
+ if ((u64)em_mem_access_off + ffa_emad_size_get(hyp_ffa_version) > fraglen) {
+ ret = FFA_RET_INVALID_PARAMETERS;
+ goto out_unlock;
+ }
+
+ ep_mem_access = (void *)buf + em_mem_access_off;
offset = ep_mem_access->composite_off;
if (!offset || buf->ep_count != 1 || buf->sender_id != HOST_FFA_ID) {
ret = FFA_RET_INVALID_PARAMETERS;
DECLARE_REG(u32, handle_lo, ctxt, 1);
DECLARE_REG(u32, handle_hi, ctxt, 2);
DECLARE_REG(u32, flags, ctxt, 3);
+ u32 offset, len, fraglen, fragoff, em_mem_access_off;
struct ffa_mem_region_attributes *ep_mem_access;
struct ffa_composite_mem_region *reg;
- u32 offset, len, fraglen, fragoff;
struct ffa_mem_region *buf;
int ret = 0;
u64 handle;
len = res->a1;
fraglen = res->a2;
- ep_mem_access = (void *)buf +
- ffa_mem_desc_offset(buf, 0, hyp_ffa_version);
+ em_mem_access_off = ffa_mem_desc_offset(buf, 0, hyp_ffa_version);
+ if ((u64)em_mem_access_off + ffa_emad_size_get(hyp_ffa_version) > fraglen) {
+ ret = FFA_RET_INVALID_PARAMETERS;
+ ffa_rx_release(res);
+ goto out_unlock;
+ }
+
+ ep_mem_access = (void *)buf + em_mem_access_off;
offset = ep_mem_access->composite_off;
/*
* We can trust the SPMD to get this right, but let's at least
* check that we end up with something that doesn't look _completely_
* bogus.
*/
- if (WARN_ON(offset > len ||
- fraglen > KVM_FFA_MBOX_NR_PAGES * PAGE_SIZE)) {
+ if (offset + CONSTITUENTS_OFFSET(0) > len ||
+ fraglen > KVM_FFA_MBOX_NR_PAGES * PAGE_SIZE) {
ret = FFA_RET_ABORTED;
ffa_rx_release(res);
goto out_unlock;
ffa_rx_release(res);
}
+ reg = (void *)buf + offset;
+ if (offset + CONSTITUENTS_OFFSET(reg->addr_range_cnt) > len) {
+ ret = FFA_RET_ABORTED;
+ goto out_unlock;
+ }
+
ffa_mem_reclaim(res, handle_lo, handle_hi, flags);
if (res->a0 != FFA_SUCCESS)
goto out_unlock;
- reg = (void *)buf + offset;
/* If the SPMD was happy, then we should be too. */
WARN_ON(ffa_host_unshare_ranges(reg->constituents,
reg->addr_range_cnt));
bool kvm_host_ffa_handler(struct kvm_cpu_context *host_ctxt, u32 func_id)
{
- struct arm_smccc_1_2_regs res;
+ struct arm_smccc_1_2_regs res = {0};
/*
* There's no way we can tell what a non-standard SMC call might
/* Fake up a data abort (level 3 translation fault on write) */
vcpu->arch.fault.esr_el2 = (ESR_ELx_EC_DABT_LOW << ESR_ELx_EC_SHIFT) |
- ESR_ELx_WNR | ESR_ELx_FSC_FAULT |
+ ESR_ELx_IL | ESR_ELx_WNR |
+ ESR_ELx_FSC_FAULT |
FIELD_PREP(ESR_ELx_FSC_LEVEL, 3);
/* Shuffle the IPA around into the HPFAR */
write_sysreg_el1(esr, SYS_ESR);
write_sysreg_el1(read_sysreg_el2(SYS_ELR), SYS_ELR);
+ write_sysreg_el1(read_sysreg_el2(SYS_SPSR), SYS_SPSR);
write_sysreg_el2(*vcpu_pc(vcpu), SYS_ELR);
write_sysreg_el2(*vcpu_cpsr(vcpu), SYS_SPSR);
}
*/
static void inject_undef64(struct kvm_vcpu *vcpu)
{
- inject_sync64(vcpu, (ESR_ELx_EC_UNKNOWN << ESR_ELx_EC_SHIFT));
+ inject_sync64(vcpu, (ESR_ELx_EC_UNKNOWN << ESR_ELx_EC_SHIFT) | ESR_ELx_IL);
}
static u64 read_id_reg(const struct kvm_vcpu *vcpu,
if (prot & KVM_PGTABLE_PROT_W)
set |= KVM_PTE_LEAF_ATTR_LO_S2_S2AP_W;
- ret = stage2_set_xn_attr(prot, &xn);
- if (ret)
- return ret;
+ if (prot & KVM_PGTABLE_PROT_X) {
+ ret = stage2_set_xn_attr(prot, &xn);
+ if (ret)
+ return ret;
- set |= xn & KVM_PTE_LEAF_ATTR_HI_S2_XN;
- clr |= ~xn & KVM_PTE_LEAF_ATTR_HI_S2_XN;
+ set |= xn & KVM_PTE_LEAF_ATTR_HI_S2_XN;
+ clr |= ~xn & KVM_PTE_LEAF_ATTR_HI_S2_XN;
+ }
ret = stage2_update_leaf_attrs(pgt, addr, 1, set, clr, NULL, &level, flags);
if (!ret || ret == -EAGAIN)
- kvm_call_hyp(__kvm_tlb_flush_vmid_ipa_nsh, pgt->mmu, addr, level);
+ kvm_call_hyp(__kvm_tlb_flush_vmid_ipa_nsh, pgt->mmu, addr,
+ (ret == -EAGAIN) ? TLBI_TTL_UNKNOWN : level);
return ret;
}
pend_sync_exception(vcpu);
/*
- * Build an {i,d}abort, depending on the level and the
- * instruction set. Report an external synchronous abort.
+ * Build an {i,d}abort, depending on the level.
+ * Report an external synchronous abort.
*/
- if (kvm_vcpu_trap_il_is32bit(vcpu))
- esr |= ESR_ELx_IL;
+ esr |= ESR_ELx_IL;
/*
* Here, the guest runs in AArch64 mode when in EL1. If we get
static void inject_undef64(struct kvm_vcpu *vcpu)
{
- u64 esr = (ESR_ELx_EC_UNKNOWN << ESR_ELx_EC_SHIFT);
-
- /*
- * Build an unknown exception, depending on the instruction
- * set.
- */
- if (kvm_vcpu_trap_il_is32bit(vcpu))
- esr |= ESR_ELx_IL;
+ u64 esr = (ESR_ELx_EC_UNKNOWN << ESR_ELx_EC_SHIFT) | ESR_ELx_IL;
kvm_inject_sync(vcpu, esr);
}
*/
if (!serror_is_masked(vcpu)) {
pend_serror_exception(vcpu);
- esr |= FIELD_PREP(ESR_ELx_EC_MASK, ESR_ELx_EC_SERROR);
+ esr |= FIELD_PREP(ESR_ELx_EC_MASK, ESR_ELx_EC_SERROR) | ESR_ELx_IL;
vcpu_write_sys_reg(vcpu, esr, exception_esr_elx(vcpu));
return 1;
}
len = kvm_vcpu_dabt_get_as(vcpu);
data = kvm_mmio_read_buf(run->mmio.data, len);
+ trace_kvm_mmio(KVM_TRACE_MMIO_READ, len, run->mmio.phys_addr,
+ &data);
+ data = vcpu_data_host_to_guest(vcpu, data, len);
+
if (kvm_vcpu_dabt_issext(vcpu) &&
len < sizeof(unsigned long)) {
mask = 1U << ((len * 8) - 1);
if (!kvm_vcpu_dabt_issf(vcpu))
data = data & 0xffffffff;
- trace_kvm_mmio(KVM_TRACE_MMIO_READ, len, run->mmio.phys_addr,
- &data);
- data = vcpu_data_host_to_guest(vcpu, data, len);
vcpu_set_reg(vcpu, kvm_vcpu_dabt_get_rd(vcpu), data);
}
struct s1_walk_result wr;
u64 hpa;
+ bool hpa_writable;
/* -1 when not mapped on a CPU */
int cpu;
gfn = vt->wr.pa >> PAGE_SHIFT;
memslot = gfn_to_memslot(vcpu->kvm, gfn);
- if (!memslot)
+ if (!memslot) {
+ fail_s1_walk(&vt->wr, ESR_ELx_FSC_EXTABT, false);
return -EFAULT;
+ }
*is_gmem = kvm_slot_has_gmem(memslot);
if (!*is_gmem) {
pfn = __kvm_faultin_pfn(memslot, gfn, write_fault ? FOLL_WRITE : 0,
&writable, &page);
- if (is_error_noslot_pfn(pfn) || (write_fault && !writable))
+ if (is_error_noslot_pfn(pfn)) {
+ fail_s1_walk(&vt->wr, ESR_ELx_FSC_EXTABT, false);
return -EFAULT;
+ }
} else {
ret = kvm_gmem_get_pfn(vcpu->kvm, memslot, gfn, &pfn, &page, NULL);
if (ret) {
write_fault, false, false);
return ret;
}
+
+ writable = !(memslot->flags & KVM_MEM_READONLY);
+ }
+
+ /*
+ * FIXME: This check is too restrictive as KVM allows cacheable memory
+ * attributes for PFNMAP VMAs that have cacheable attributes in host
+ * stage-1.
+ */
+ if (!pfn_is_map_memory(pfn)) {
+ kvm_release_faultin_page(vcpu->kvm, page, true, false);
+ fail_s1_walk(&vt->wr, ESR_ELx_FSC_EXTABT, false);
+ return -EINVAL;
}
scoped_guard(write_lock, &vcpu->kvm->mmu_lock) {
vt->gva = va;
vt->hpa = pfn << PAGE_SHIFT;
+ vt->hpa_writable = writable;
vt->valid = true;
vt->cpu = -1;
kvm_make_request(KVM_REQ_MAP_L1_VNCR_EL2, vcpu);
- kvm_release_faultin_page(vcpu->kvm, page, false, vt->wr.pw);
+ kvm_release_faultin_page(vcpu->kvm, page, false, vt->wr.pw && vt->hpa_writable);
}
- if (vt->wr.pw)
+ if (vt->wr.pw && vt->hpa_writable)
mark_page_dirty(vcpu->kvm, gfn);
return 0;
}
-static void inject_vncr_perm(struct kvm_vcpu *vcpu)
+static void handle_vncr_perm(struct kvm_vcpu *vcpu)
{
struct vncr_tlb *vt = vcpu->arch.vncr_tlb;
u64 esr = kvm_vcpu_get_esr(vcpu);
+ u64 fsc;
+
+ /*
+ * Promote to an external abort if the stage-1 permits writes but the
+ * HPA is read-only (e.g. RO memslot).
+ */
+ if (kvm_is_write_fault(vcpu) && vt->wr.pw && !vt->hpa_writable)
+ fsc = ESR_ELx_FSC_EXTABT;
+ /*
+ * Otherwise, inject a permission fault using the guest's translation
+ * level rather than the host's.
+ */
+ else
+ fsc = ESR_ELx_FSC_PERM_L(vt->wr.level);
- /* Adjust the fault level to reflect that of the guest's */
esr &= ~ESR_ELx_FSC;
- esr |= FIELD_PREP(ESR_ELx_FSC,
- ESR_ELx_FSC_PERM_L(vt->wr.level));
+ esr |= FIELD_PREP(ESR_ELx_FSC, fsc);
kvm_inject_nested_sync(vcpu, esr);
}
-static bool kvm_vncr_tlb_lookup(struct kvm_vcpu *vcpu)
-{
- struct vncr_tlb *vt = vcpu->arch.vncr_tlb;
-
- lockdep_assert_held_read(&vcpu->kvm->mmu_lock);
-
- if (!vt->valid)
- return false;
-
- if (read_vncr_el2(vcpu) != vt->gva)
- return false;
-
- if (vt->wr.nG)
- return get_asid_by_regime(vcpu, TR_EL20) == vt->wr.asid;
-
- return true;
-}
-
int kvm_handle_vncr_abort(struct kvm_vcpu *vcpu)
{
struct vncr_tlb *vt = vcpu->arch.vncr_tlb;
u64 esr = kvm_vcpu_get_esr(vcpu);
+ bool is_gmem = false;
+ bool perm;
+ int ret;
WARN_ON_ONCE(!(esr & ESR_ELx_VNCR));
if (kvm_vcpu_abt_issea(vcpu))
return kvm_handle_guest_sea(vcpu);
- if (esr_fsc_is_permission_fault(esr)) {
- inject_vncr_perm(vcpu);
- } else if (esr_fsc_is_translation_fault(esr)) {
- bool valid, is_gmem = false;
- int ret;
-
- scoped_guard(read_lock, &vcpu->kvm->mmu_lock)
- valid = kvm_vncr_tlb_lookup(vcpu);
-
- if (!valid)
- ret = kvm_translate_vncr(vcpu, &is_gmem);
- else
- ret = -EPERM;
-
- switch (ret) {
- case -EAGAIN:
- /* Let's try again... */
- break;
- case -ENOMEM:
- /*
- * For guest_memfd, this indicates that it failed to
- * create a folio to back the memory. Inform userspace.
- */
- if (is_gmem)
- return 0;
- /* Otherwise, let's try again... */
- break;
- case -EFAULT:
- case -EIO:
- case -EHWPOISON:
- if (is_gmem)
- return 0;
- fallthrough;
- case -EINVAL:
- case -ENOENT:
- case -EACCES:
- /*
- * Translation failed, inject the corresponding
- * exception back to EL2.
- */
- BUG_ON(!vt->wr.failed);
+ if (!esr_fsc_is_translation_fault(esr) && !esr_fsc_is_permission_fault(esr)) {
+ KVM_BUG(1, vcpu->kvm, "Unhandled VNCR abort, ESR=%llx\n", esr);
+ return -EIO;
+ }
- esr &= ~ESR_ELx_FSC;
- esr |= FIELD_PREP(ESR_ELx_FSC, vt->wr.fst);
+ ret = kvm_translate_vncr(vcpu, &is_gmem);
+ switch (ret) {
+ case -EAGAIN:
+ /* Let's try again... */
+ return 1;
+ case -ENOMEM:
+ /*
+ * For guest_memfd, this indicates that it failed to
+ * create a folio to back the memory. Inform userspace.
+ */
+ if (is_gmem)
+ return 0;
+ /* Otherwise, let's try again... */
+ break;
+ case -EFAULT:
+ case -EIO:
+ case -EHWPOISON:
+ if (is_gmem)
+ return 0;
+ fallthrough;
+ case -EINVAL:
+ case -ENOENT:
+ case -EACCES:
+ /*
+ * Translation failed, inject the corresponding
+ * exception back to EL2.
+ */
+ esr &= ~ESR_ELx_FSC;
+ esr |= FIELD_PREP(ESR_ELx_FSC, vt->wr.fst);
- kvm_inject_nested_sync(vcpu, esr);
- break;
- case -EPERM:
- /* Hack to deal with POE until we get kernel support */
- inject_vncr_perm(vcpu);
- break;
- case 0:
- break;
- }
- } else {
- WARN_ONCE(1, "Unhandled VNCR abort, ESR=%llx\n", esr);
+ kvm_inject_nested_sync(vcpu, esr);
+ break;
+ case 0:
+ perm = kvm_is_write_fault(vcpu) ? vt->wr.pw && vt->hpa_writable : vt->wr.pr;
+ if (!perm)
+ handle_vncr_perm(vcpu);
+ break;
}
return 1;
vt->cpu = smp_processor_id();
- if (vt->wr.pw && vt->wr.pr)
+ if (vt->hpa_writable && vt->wr.pw && vt->wr.pr)
prot = PAGE_KERNEL;
else if (vt->wr.pr)
prot = PAGE_KERNEL_RO;
page = pfn_to_page(mapping->pfn);
WARN_ON_ONCE(mapping->nr_pages != 1);
unpin_user_pages_dirty_lock(&page, 1, true);
- account_locked_vm(current->mm, 1, false);
+ account_locked_vm(kvm->mm, 1, false);
pkvm_mapping_remove(mapping, &pgt->pkvm_mappings);
kfree(mapping);
}
struct kvm_io_device *kvm_io_dev;
struct vgic_io_device *iodev;
+ guard(srcu)(&kvm->srcu);
+
kvm_io_dev = kvm_io_bus_get_dev(kvm, KVM_MMIO_BUS, db);
if (!kvm_io_dev)
return ERR_PTR(-EINVAL);
list_for_each_entry_safe(irq, tmp, &vgic_cpu->ap_list_head, ap_list) {
if (irq_is_lpi(vcpu->kvm, irq->intid)) {
raw_spin_lock(&irq->irq_lock);
+ irq->pending_latch = false;
list_del(&irq->ap_list);
irq->vcpu = NULL;
raw_spin_unlock(&irq->irq_lock);
continue;
}
- /* This interrupt looks like it has to be migrated. */
+ /*
+ * This interrupt looks like it has to be migrated,
+ * make sure it is kept alive while locks are dropped.
+ */
+ vgic_get_irq_ref(irq);
raw_spin_unlock(&irq->irq_lock);
raw_spin_unlock(&vgic_cpu->ap_list_lock);
raw_spin_lock(&irq->irq_lock);
/*
- * If the affinity has been preserved, move the
- * interrupt around. Otherwise, it means things have
- * changed while the interrupt was unlocked, and we
- * need to replay this.
+ * If the interrupt is still ours and its affinity has
+ * been preserved, move it around. Otherwise, it means
+ * things have changed while the interrupt was unlocked
+ * (it may even have been taken off the list with its
+ * affinity left untouched), and we need to replay this.
*
* In all cases, we cannot trust the list not to have
* changed, so we restart from the beginning.
*/
- if (target_vcpu == vgic_target_oracle(irq)) {
+ if (irq->vcpu == vcpu && target_vcpu == vgic_target_oracle(irq)) {
struct vgic_cpu *new_cpu = &target_vcpu->arch.vgic_cpu;
list_del(&irq->ap_list);
raw_spin_unlock(&vcpuB->arch.vgic_cpu.ap_list_lock);
raw_spin_unlock(&vcpuA->arch.vgic_cpu.ap_list_lock);
+ deleted_lpis |= vgic_put_irq_norelease(vcpu->kvm, irq);
+
if (target_vcpu_needs_kick) {
kvm_make_request(KVM_REQ_IRQ_PENDING, target_vcpu);
kvm_vcpu_kick(target_vcpu);
if (free_mapped) {
/* CONT blocks are not supported in the vmemmap */
WARN_ON(pmd_cont(pmd));
- flush_tlb_kernel_range(addr, addr + PMD_SIZE);
+ /*
+ * Invalidating a block entry requires just
+ * a single overlapping TLB invalidation,
+ * so limit the range of the flush to a single
+ * page.
+ */
+ flush_tlb_kernel_range(addr, addr + PAGE_SIZE);
free_hotplug_page_range(pmd_page(pmd),
PMD_SIZE, altmap);
}
if (pud_leaf(pud)) {
pud_clear(pudp);
if (free_mapped) {
- flush_tlb_kernel_range(addr, addr + PUD_SIZE);
+ /* See comment in unmap_hotplug_pmd_range(). */
+ flush_tlb_kernel_range(addr, addr + PAGE_SIZE);
free_hotplug_page_range(pud_page(pud),
PUD_SIZE, altmap);
}
UnsignedEnum 7:4 BWE
0b0000 NI
0b0001 FEAT_BWE
- 0b0002 FEAT_BWE2
+ 0b0010 FEAT_BWE2
EndEnum
UnsignedEnum 3:0 STEP
0b0000 NI
static void __init m523x_fec_init(void)
{
/* Set multi-function pins to ethernet use */
- mcf_write8(read8(MCFGPIO_PAR_FECI2C) | 0xf0, MCFGPIO_PAR_FECI2C);
+ mcf_write8(mcf_read8(MCFGPIO_PAR_FECI2C) | 0xf0, MCFGPIO_PAR_FECI2C);
}
/***************************************************************************/
mcf_write16(read16(MCFGPIO_PEPAR) & ~(1 << (5 * 2)), MCFGPIO_PEPAR);
/* Make portE.5 an output */
- mcf_write8(read8(MCFGPIO_PDDR_E) | (1 << 5), MCFGPIO_PDDR_E);
+ mcf_write8(mcf_read8(MCFGPIO_PDDR_E) | (1 << 5), MCFGPIO_PDDR_E);
/* Now toggle portE.5 from low to high */
- mcf_write8(read8(MCFGPIO_PODR_E) & ~(1 << 5), MCFGPIO_PODR_E);
- mcf_write8(read8(MCFGPIO_PODR_E) | (1 << 5), MCFGPIO_PODR_E);
+ mcf_write8(mcf_read8(MCFGPIO_PODR_E) & ~(1 << 5), MCFGPIO_PODR_E);
+ mcf_write8(mcf_read8(MCFGPIO_PODR_E) | (1 << 5), MCFGPIO_PODR_E);
printk(KERN_EMERG "Failed to hibernate. Halting!\n");
}
#define clear_user_page(addr, vaddr, page) \
do { clear_page(addr); \
flush_dcache_page(page); \
+ (void)(vaddr); \
} while (0)
#define copy_user_page(to, from, vaddr, page) \
do { copy_page(to, from); \
flush_dcache_page(page); \
+ (void)(vaddr); \
} while (0)
extern unsigned long m68k_memoffset;
+++ /dev/null
-/* SPDX-License-Identifier: GPL-2.0 */
-#ifndef __ASM_POWERPC_PREEMPT_H
-#define __ASM_POWERPC_PREEMPT_H
-
-#include <asm-generic/preempt.h>
-
-#if defined(CONFIG_PREEMPT_DYNAMIC)
-#include <linux/jump_label.h>
-DECLARE_STATIC_KEY_TRUE(sk_dynamic_irqentry_exit_cond_resched);
-#define need_irq_preemption() \
- (static_branch_unlikely(&sk_dynamic_irqentry_exit_cond_resched))
-#else
-#define need_irq_preemption() (IS_ENABLED(CONFIG_PREEMPTION))
-#endif
-
-#endif /* __ASM_POWERPC_PREEMPT_H */
if (IS_ENABLED(CONFIG_PPC64))
return mask_user_address_simple(ptr);
- if (IS_ENABLED(CONFIG_E500))
+ if (IS_ENABLED(CONFIG_PPC_E500))
return mask_user_address_isel(ptr);
if (TASK_SIZE <= UL(SZ_2G) && border >= UL(SZ_2G))
return mask_user_address_simple(ptr);
if (isa >= ISA_V3_1) {
cur_cpu_spec->cpu_features |= CPU_FTR_ARCH_31;
cur_cpu_spec->cpu_user_features2 |= PPC_FEATURE2_ARCH_3_1;
+
+ /*
+ * CPU_FTR_P11_PVR is a kernel-internal flag to identify
+ * Power11 and later processors. While ISA v3.1 is supported
+ * by Power10+, this flag specifically indicates Power11+
+ * for code that needs to distinguish between P10 and P11.
+ */
+ if (PVR_VER(mfspr(SPRN_PVR)) >= PVR_POWER11)
+ cur_cpu_spec->cpu_features |= CPU_FTR_P11_PVR;
}
}
}
}
-#ifdef CONFIG_NO_HZ_COMMON
/**
* vtime_reset - Fast forward vtime entry clocks
*
#endif
}
+#ifdef CONFIG_NO_HZ_COMMON
/**
* vtime_dyntick_start - Inform vtime about entry to idle-dynticks
*
static void __init init_decrementer_clockevent(void)
{
register_decrementer_clockevent(smp_processor_id());
+ vtime_reset();
}
void secondary_cpu_time_init(void)
/* FIME: Should make unrelated change to move snapshot_timebase
* call here ! */
register_decrementer_clockevent(smp_processor_id());
+ vtime_reset();
}
/*
* set and we are preemptible. The hack here is to schedule a
* decrementer to fire here and reschedule for us if necessary.
*/
- if (need_irq_preemption() && need_resched())
+ if (need_resched())
set_dec(1);
return 0;
}
{ .compatible = "fsl,qoriq-pcie-v2.3", },
{ .compatible = "fsl,qoriq-pcie-v2.2", },
{ .compatible = "fsl,fman", },
+ /* IFC NAND and NOR controllers */
+ { .compatible = "fsl,ifc", },
{},
};
if (write && !(vma->vm_flags & VM_WRITE))
return -EACCES;
+ if (offset >= LS_SIZE)
+ return -EFAULT;
if (spu_acquire(ctx))
return -EINTR;
- if ((offset + len) > vma->vm_end)
- len = vma->vm_end - offset;
+ if ((offset + len) > LS_SIZE)
+ len = LS_SIZE - offset;
local_store = ctx->ops->get_ls(ctx);
if (write)
memcpy_toio(local_store + offset, buf, len);
config VPA_PMU
tristate "VPA PMU events"
depends on KVM_BOOK3S_64_HV && HV_PERF_CTRS
+ default m
help
Enable access to the VPA PMU counters via perf. This enables
code that support measurement for KVM on PowerVM(KoP) feature.
esi_buf_size = ESI_HDR_SIZE + (CURR_MAX_ESI_ATTRS * max_esi_attrs);
temp_esi_buf = krealloc(esi_buf, esi_buf_size, GFP_KERNEL);
- if (temp_esi_buf)
- esi_buf = temp_esi_buf;
- else
- return -ENOMEM;
-
+ if (!temp_esi_buf)
+ goto out_free_esi_buf;
+ esi_buf = temp_esi_buf;
goto retry;
}
select HAVE_DEBUG_KMEMLEAK
select HAVE_DMA_CONTIGUOUS if MMU
select HAVE_DYNAMIC_FTRACE if MMU && (CLANG_SUPPORTS_DYNAMIC_FTRACE || GCC_SUPPORTS_DYNAMIC_FTRACE)
- select FUNCTION_ALIGNMENT_4B if HAVE_DYNAMIC_FTRACE && RISCV_ISA_C
+ select FUNCTION_ALIGNMENT_4B if DYNAMIC_FTRACE && RISCV_ISA_C
select HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS if HAVE_DYNAMIC_FTRACE_WITH_CALL_OPS
select HAVE_DYNAMIC_FTRACE_WITH_CALL_OPS if (DYNAMIC_FTRACE_WITH_ARGS && !CFI)
select HAVE_DYNAMIC_FTRACE_WITH_ARGS if HAVE_DYNAMIC_FTRACE
help
This enables support for the RISC-V based T-HEAD SoCs.
+config ARCH_ULTRARISC
+ bool "UltraRISC RISC-V SoCs"
+ help
+ This enables support for UltraRISC SoC platform hardware,
+ including boards based on the UR-DP1000.
+
config ARCH_VIRT
bool "QEMU Virt Machine"
select POWER_RESET
CONFIG_ARCH_SUNXI=y
CONFIG_ARCH_TENSTORRENT=y
CONFIG_ARCH_THEAD=y
+CONFIG_ARCH_ULTRARISC=y
CONFIG_ARCH_VIRT=y
CONFIG_ARCH_CANAAN=y
CONFIG_SMP=y
flush_icache_mm(vma->vm_mm, 0); \
} while (0)
-#ifdef CONFIG_64BIT
+#if defined(CONFIG_64BIT) && defined(CONFIG_MMU)
/* This is accessed in assembly code. cpumask_var_t would be too complex. */
extern DECLARE_BITMAP(new_valid_map_cpus, NR_CPUS);
extern char _end[];
#define flush_cache_vmap flush_cache_vmap
static inline void flush_cache_vmap(unsigned long start, unsigned long end)
{
- if (is_vmalloc_or_module_addr((void *)start))
+ if (is_vmalloc_or_module_addr((void *)start) ||
+ (start >= VMEMMAP_START && end <= VMEMMAP_END))
mark_new_valid_map();
}
#define flush_cache_vmap_early(start, end) local_flush_tlb_kernel_range(start, end)
#define readsw(addr, buffer, count) __readsw(addr, buffer, count)
#define readsl(addr, buffer, count) __readsl(addr, buffer, count)
+#ifdef CONFIG_HAS_IOPORT
__io_reads_ins(ins, u8, b, __io_pbr(), __io_par(addr))
__io_reads_ins(ins, u16, w, __io_pbr(), __io_par(addr))
__io_reads_ins(ins, u32, l, __io_pbr(), __io_par(addr))
#define insb(addr, buffer, count) __insb(PCI_IOBASE + (addr), buffer, count)
#define insw(addr, buffer, count) __insw(PCI_IOBASE + (addr), buffer, count)
#define insl(addr, buffer, count) __insl(PCI_IOBASE + (addr), buffer, count)
+#endif
__io_writes_outs(writes, u8, b, __io_bw(), __io_aw())
__io_writes_outs(writes, u16, w, __io_bw(), __io_aw())
#define writesw(addr, buffer, count) __writesw(addr, buffer, count)
#define writesl(addr, buffer, count) __writesl(addr, buffer, count)
+#ifdef CONFIG_HAS_IOPORT
__io_writes_outs(outs, u8, b, __io_pbw(), __io_paw())
__io_writes_outs(outs, u16, w, __io_pbw(), __io_paw())
__io_writes_outs(outs, u32, l, __io_pbw(), __io_paw())
#define outsb(addr, buffer, count) __outsb(PCI_IOBASE + (addr), buffer, count)
#define outsw(addr, buffer, count) __outsw(PCI_IOBASE + (addr), buffer, count)
#define outsl(addr, buffer, count) __outsl(PCI_IOBASE + (addr), buffer, count)
+#endif
#ifdef CONFIG_64BIT
__io_reads_ins(reads, u64, q, __io_br(), __io_ar(addr))
#define readsq(addr, buffer, count) __readsq(addr, buffer, count)
+#ifdef CONFIG_HAS_IOPORT
__io_reads_ins(ins, u64, q, __io_pbr(), __io_par(addr))
#define insq(addr, buffer, count) __insq(PCI_IOBASE + (addr), buffer, count)
+#endif
__io_writes_outs(writes, u64, q, __io_bw(), __io_aw())
#define writesq(addr, buffer, count) __writesq(addr, buffer, count)
+#ifdef CONFIG_HAS_IOPORT
__io_writes_outs(outs, u64, q, __io_pbr(), __io_paw())
#define outsq(addr, buffer, count) __outsq(PCI_IOBASE + (addr), buffer, count)
#endif
+#endif
#include <asm-generic/io.h>
/*
* VCPU interrupts
*
- * We have a lockless approach for tracking pending VCPU interrupts
- * implemented using atomic bitops. The irqs_pending bitmap represent
- * pending interrupts whereas irqs_pending_mask represent bits changed
- * in irqs_pending. Our approach is modeled around multiple producer
- * and single consumer problem where the consumer is the VCPU itself.
+ * The irqs_pending bitmap represents pending interrupts whereas
+ * irqs_pending_mask represents bits changed in irqs_pending. Updates
+ * to these bitmaps are serialized so vcpu interrupt sync/flush cannot
+ * drop a newly injected interrupt while syncing guest-visible HVIP.
*/
#define KVM_RISCV_VCPU_NR_IRQS 64
+ raw_spinlock_t irqs_pending_lock;
DECLARE_BITMAP(irqs_pending, KVM_RISCV_VCPU_NR_IRQS);
DECLARE_BITMAP(irqs_pending_mask, KVM_RISCV_VCPU_NR_IRQS);
.Lrestore_kernel_tpsp:
csrr tp, CSR_SCRATCH
-#ifdef CONFIG_64BIT
+#if defined(CONFIG_64BIT) && defined(CONFIG_MMU)
/*
* The RISC-V kernel does not flush TLBs on all CPUS after each new
* vmalloc mapping or kfence_unprotect(), which may result in
if (image->segment[i].memsz <= sizeof(fdt))
continue;
+ if (!image->segment[i].buf)
+ continue;
+
if (image->file_mode)
memcpy(&fdt, image->segment[i].buf, sizeof(fdt));
else if (copy_from_user(&fdt, image->segment[i].buf, sizeof(fdt)))
if (cpusetsize > cpumask_size())
cpusetsize = cpumask_size();
+ cpumask_clear(&cpus);
ret = copy_from_user(&cpus, cpus_user, cpusetsize);
if (ret)
return -EFAULT;
endif
# Disable -pg to prevent insert call site
-CFLAGS_REMOVE_vgettimeofday.o = $(CC_FLAGS_FTRACE) $(CC_FLAGS_SCS)
-CFLAGS_REMOVE_getrandom.o = $(CC_FLAGS_FTRACE) $(CC_FLAGS_SCS)
-CFLAGS_REMOVE_hwprobe.o = $(CC_FLAGS_FTRACE) $(CC_FLAGS_SCS)
+CFLAGS_REMOVE_vgettimeofday.o = $(CC_FLAGS_FTRACE) $(CC_FLAGS_SCS) $(CC_FLAGS_LTO)
+CFLAGS_REMOVE_getrandom.o = $(CC_FLAGS_FTRACE) $(CC_FLAGS_SCS) $(CC_FLAGS_LTO)
+CFLAGS_REMOVE_hwprobe.o = $(CC_FLAGS_FTRACE) $(CC_FLAGS_SCS) $(CC_FLAGS_LTO)
# Force dependency
$(obj)/$(vdso_o): $(obj)/$(vdso_so)
#include <asm/unistd.h>
#include <asm/assembler.h>
+/*
+ * WARNING: Do NOT add a CFI landing pad at the start of this function.
+ * Unwinders such as libgcc identify the sigreturn trampoline by matching the
+ * instruction sequence. Adding a landing pad here would break unwinding from
+ * signal handlers.
+ *
+ * This trampoline is used only for signal return and not via an indirect
+ * call/jump from userspace, so adding CFI landing pad is unnecessary.
+ */
.text
SYM_FUNC_START(__vdso_rt_sigreturn)
.cfi_startproc
.cfi_signal_frame
- vdso_lpad
li a7, __NR_rt_sigreturn
ecall
.cfi_endproc
struct kvm_vcpu_aia_csr *csr = &vcpu->arch.aia_context.guest_csr;
unsigned long mask, val;
+ lockdep_assert_held(&vcpu->arch.irqs_pending_lock);
+
if (!kvm_riscv_aia_available())
return;
- if (READ_ONCE(vcpu->arch.irqs_pending_mask[1])) {
- mask = xchg_acquire(&vcpu->arch.irqs_pending_mask[1], 0);
- val = READ_ONCE(vcpu->arch.irqs_pending[1]) & mask;
+ mask = vcpu->arch.irqs_pending_mask[1];
+ if (mask) {
+ vcpu->arch.irqs_pending_mask[1] = 0;
+ val = vcpu->arch.irqs_pending[1] & mask;
csr->hviph &= ~mask;
csr->hviph |= val;
{
struct kvm_vcpu_aia_csr *csr = &vcpu->arch.aia_context.guest_csr;
+ lockdep_assert_held(&vcpu->arch.irqs_pending_lock);
+
if (kvm_riscv_aia_available())
csr->vsieh = ncsr_read(CSR_VSIEH);
}
bool kvm_riscv_vcpu_aia_has_interrupts(struct kvm_vcpu *vcpu, u64 mask)
{
unsigned long seip;
+#ifdef CONFIG_32BIT
+ unsigned long flags;
+ bool pending;
+#endif
if (!kvm_riscv_aia_available())
return false;
#ifdef CONFIG_32BIT
- if (READ_ONCE(vcpu->arch.irqs_pending[1]) &
- (vcpu->arch.aia_context.guest_csr.vsieh & upper_32_bits(mask)))
+ raw_spin_lock_irqsave(&vcpu->arch.irqs_pending_lock, flags);
+ pending = vcpu->arch.irqs_pending[1] &
+ (vcpu->arch.aia_context.guest_csr.vsieh &
+ upper_32_bits(mask));
+ raw_spin_unlock_irqrestore(&vcpu->arch.irqs_pending_lock, flags);
+
+ if (pending)
return true;
#endif
{
struct kvm_vcpu_aia_csr *csr = &vcpu->arch.aia_context.guest_csr;
unsigned long regs_max = sizeof(struct kvm_riscv_aia_csr) / sizeof(unsigned long);
+#ifdef CONFIG_32BIT
+ unsigned long flags;
+#endif
if (!riscv_isa_extension_available(vcpu->arch.isa, SSAIA))
return -ENOENT;
((unsigned long *)csr)[reg_num] = val;
#ifdef CONFIG_32BIT
- if (reg_num == KVM_REG_RISCV_CSR_AIA_REG(siph))
- WRITE_ONCE(vcpu->arch.irqs_pending_mask[1], 0);
+ if (reg_num == KVM_REG_RISCV_CSR_AIA_REG(siph)) {
+ raw_spin_lock_irqsave(&vcpu->arch.irqs_pending_lock, flags);
+ vcpu->arch.irqs_pending_mask[1] = 0;
+ raw_spin_unlock_irqrestore(&vcpu->arch.irqs_pending_lock,
+ flags);
+ }
#endif
}
*/
#include <linux/bitops.h>
+#include <linux/cpufeature.h>
#include <linux/errno.h>
#include <linux/kvm_host.h>
#include <linux/module.h>
#include <linux/pgtable.h>
#include <asm/kvm_gstage.h>
+#include <asm/hwcap.h>
#ifdef CONFIG_64BIT
unsigned long kvm_riscv_gstage_max_pgd_levels __ro_after_init = 3;
}
if (pte_val(*ptep) != pte_val(map->pte)) {
+ bool was_invalid = !pte_val(*ptep);
set_pte(ptep, map->pte);
- if (gstage_pte_leaf(ptep))
+ if (gstage_pte_leaf(ptep) &&
+ !(was_invalid && riscv_has_extension_unlikely(RISCV_ISA_EXT_SVVPTC)))
gstage_tlb_flush(gstage, current_level, map->addr);
}
pgprot_t prot;
unsigned long pfn;
phys_addr_t addr, end;
+ unsigned long pgd_levels = kvm->arch.pgd_levels;
struct kvm_mmu_memory_cache pcache = {
.gfp_custom = (in_atomic) ? GFP_ATOMIC | __GFP_ACCOUNT : 0,
.gfp_zero = __GFP_ZERO,
if (!writable)
map.pte = pte_wrprotect(map.pte);
- ret = kvm_mmu_topup_memory_cache(&pcache, kvm->arch.pgd_levels);
+ ret = __kvm_mmu_topup_memory_cache(&pcache, pgd_levels, pgd_levels);
if (ret)
goto out;
static void kvm_riscv_reset_vcpu(struct kvm_vcpu *vcpu, bool kvm_sbi_reset)
{
+ unsigned long flags;
bool loaded;
/**
kvm_riscv_vcpu_aia_reset(vcpu);
+ raw_spin_lock_irqsave(&vcpu->arch.irqs_pending_lock, flags);
bitmap_zero(vcpu->arch.irqs_pending, KVM_RISCV_VCPU_NR_IRQS);
bitmap_zero(vcpu->arch.irqs_pending_mask, KVM_RISCV_VCPU_NR_IRQS);
+ raw_spin_unlock_irqrestore(&vcpu->arch.irqs_pending_lock, flags);
kvm_riscv_vcpu_pmu_reset(vcpu);
/* Setup VCPU hfence queue */
spin_lock_init(&vcpu->arch.hfence_lock);
+ raw_spin_lock_init(&vcpu->arch.irqs_pending_lock);
spin_lock_init(&vcpu->arch.reset_state.lock);
{
struct kvm_vcpu_csr *csr = &vcpu->arch.guest_csr;
unsigned long mask, val;
+ unsigned long flags;
- if (READ_ONCE(vcpu->arch.irqs_pending_mask[0])) {
- mask = xchg_acquire(&vcpu->arch.irqs_pending_mask[0], 0);
- val = READ_ONCE(vcpu->arch.irqs_pending[0]) & mask;
+ raw_spin_lock_irqsave(&vcpu->arch.irqs_pending_lock, flags);
+
+ mask = vcpu->arch.irqs_pending_mask[0];
+ if (mask) {
+ vcpu->arch.irqs_pending_mask[0] = 0;
+ val = vcpu->arch.irqs_pending[0] & mask;
csr->hvip &= ~mask;
csr->hvip |= val;
/* Flush AIA high interrupts */
kvm_riscv_vcpu_aia_flush_interrupts(vcpu);
+
+ raw_spin_unlock_irqrestore(&vcpu->arch.irqs_pending_lock, flags);
}
void kvm_riscv_vcpu_sync_interrupts(struct kvm_vcpu *vcpu)
{
unsigned long hvip;
+ unsigned long flags;
struct kvm_vcpu_arch *v = &vcpu->arch;
struct kvm_vcpu_csr *csr = &vcpu->arch.guest_csr;
/* Sync-up HVIP.VSSIP bit changes does by Guest */
hvip = ncsr_read(CSR_HVIP);
+
+ raw_spin_lock_irqsave(&v->irqs_pending_lock, flags);
+
if ((csr->hvip ^ hvip) & (1UL << IRQ_VS_SOFT)) {
if (hvip & (1UL << IRQ_VS_SOFT)) {
- if (!test_and_set_bit(IRQ_VS_SOFT,
- v->irqs_pending_mask))
- set_bit(IRQ_VS_SOFT, v->irqs_pending);
+ if (!__test_and_set_bit(IRQ_VS_SOFT,
+ v->irqs_pending_mask))
+ __set_bit(IRQ_VS_SOFT, v->irqs_pending);
} else {
- if (!test_and_set_bit(IRQ_VS_SOFT,
- v->irqs_pending_mask))
- clear_bit(IRQ_VS_SOFT, v->irqs_pending);
+ if (!__test_and_set_bit(IRQ_VS_SOFT,
+ v->irqs_pending_mask))
+ __clear_bit(IRQ_VS_SOFT, v->irqs_pending);
}
}
/* Sync up the HVIP.LCOFIP bit changes (only clear) by the guest */
if ((csr->hvip ^ hvip) & (1UL << IRQ_PMU_OVF)) {
if (!(hvip & (1UL << IRQ_PMU_OVF)) &&
- !test_and_set_bit(IRQ_PMU_OVF, v->irqs_pending_mask))
- clear_bit(IRQ_PMU_OVF, v->irqs_pending);
+ !__test_and_set_bit(IRQ_PMU_OVF, v->irqs_pending_mask))
+ __clear_bit(IRQ_PMU_OVF, v->irqs_pending);
}
/* Sync-up AIA high interrupts */
kvm_riscv_vcpu_aia_sync_interrupts(vcpu);
+ raw_spin_unlock_irqrestore(&v->irqs_pending_lock, flags);
+
/* Sync-up timer CSRs */
kvm_riscv_vcpu_timer_sync(vcpu);
}
int kvm_riscv_vcpu_set_interrupt(struct kvm_vcpu *vcpu, unsigned int irq)
{
+ unsigned long flags;
+
/*
* We only allow VS-mode software, timer, and external
* interrupts when irq is one of the local interrupts
irq != IRQ_PMU_OVF)
return -EINVAL;
- set_bit(irq, vcpu->arch.irqs_pending);
- smp_mb__before_atomic();
- set_bit(irq, vcpu->arch.irqs_pending_mask);
+ raw_spin_lock_irqsave(&vcpu->arch.irqs_pending_lock, flags);
+ __set_bit(irq, vcpu->arch.irqs_pending);
+ __set_bit(irq, vcpu->arch.irqs_pending_mask);
+ raw_spin_unlock_irqrestore(&vcpu->arch.irqs_pending_lock, flags);
kvm_vcpu_kick(vcpu);
int kvm_riscv_vcpu_unset_interrupt(struct kvm_vcpu *vcpu, unsigned int irq)
{
+ unsigned long flags;
+
/*
* We only allow VS-mode software, timer, counter overflow and external
* interrupts when irq is one of the local interrupts
irq != IRQ_PMU_OVF)
return -EINVAL;
- clear_bit(irq, vcpu->arch.irqs_pending);
- smp_mb__before_atomic();
- set_bit(irq, vcpu->arch.irqs_pending_mask);
+ raw_spin_lock_irqsave(&vcpu->arch.irqs_pending_lock, flags);
+ __clear_bit(irq, vcpu->arch.irqs_pending);
+ __set_bit(irq, vcpu->arch.irqs_pending_mask);
+ raw_spin_unlock_irqrestore(&vcpu->arch.irqs_pending_lock, flags);
return 0;
}
bool kvm_riscv_vcpu_has_interrupts(struct kvm_vcpu *vcpu, u64 mask)
{
+ unsigned long flags;
unsigned long ie;
+ bool ret;
+ raw_spin_lock_irqsave(&vcpu->arch.irqs_pending_lock, flags);
ie = ((vcpu->arch.guest_csr.vsie & VSIP_VALID_MASK)
<< VSIP_TO_HVIP_SHIFT) & (unsigned long)mask;
ie |= vcpu->arch.guest_csr.vsie & ~IRQ_LOCAL_MASK &
(unsigned long)mask;
- if (READ_ONCE(vcpu->arch.irqs_pending[0]) & ie)
- return true;
+ ret = vcpu->arch.irqs_pending[0] & ie;
+ raw_spin_unlock_irqrestore(&vcpu->arch.irqs_pending_lock, flags);
/* Check AIA high interrupts */
- return kvm_riscv_vcpu_aia_has_interrupts(vcpu, mask);
+ if (!ret)
+ ret = kvm_riscv_vcpu_aia_has_interrupts(vcpu, mask);
+
+ return ret;
}
void __kvm_riscv_vcpu_power_off(struct kvm_vcpu *vcpu)
return kvm_riscv_vcpu_mmio_store(vcpu, run,
fault_addr,
trap->htinst);
+ case EXC_INST_GUEST_PAGE_FAULT: {
+ /*
+ * No memslot backs this GPA and an instruction fetch
+ * cannot be emulated as MMIO. On bare metal a fetch
+ * from an unbacked physical address raises an
+ * instruction access fault, so reflect that back to
+ * the guest.
+ */
+ struct kvm_cpu_trap inst_trap = {
+ .sepc = trap->sepc,
+ .scause = EXC_INST_ACCESS,
+ .stval = trap->stval,
+ .htval = 0,
+ .htinst = 0,
+ };
+
+ kvm_riscv_vcpu_trap_redirect(vcpu, &inst_trap);
+ return 1;
+ }
default:
return -EOPNOTSUPP;
};
reg_val = vcpu->arch.isa[0] & KVM_RISCV_BASE_ISA_MASK;
break;
case KVM_REG_RISCV_CONFIG_REG(zicbom_block_size):
- if (kvm_riscv_isa_check_host(ZICBOM))
- return -ENOENT;
- reg_val = riscv_cbom_block_size;
+ reg_val = (kvm_riscv_isa_check_host(ZICBOM)) ? 0 : riscv_cbom_block_size;
break;
case KVM_REG_RISCV_CONFIG_REG(zicboz_block_size):
- if (kvm_riscv_isa_check_host(ZICBOZ))
- return -ENOENT;
- reg_val = riscv_cboz_block_size;
+ reg_val = (kvm_riscv_isa_check_host(ZICBOZ)) ? 0 : riscv_cboz_block_size;
break;
case KVM_REG_RISCV_CONFIG_REG(zicbop_block_size):
- if (kvm_riscv_isa_check_host(ZICBOP))
- return -ENOENT;
- reg_val = riscv_cbop_block_size;
+ reg_val = (kvm_riscv_isa_check_host(ZICBOP)) ? 0 : riscv_cbop_block_size;
break;
case KVM_REG_RISCV_CONFIG_REG(mvendorid):
reg_val = vcpu->arch.mvendorid;
}
break;
case KVM_REG_RISCV_CONFIG_REG(zicbom_block_size):
- if (kvm_riscv_isa_check_host(ZICBOM))
- return -ENOENT;
- if (reg_val != riscv_cbom_block_size)
+ if (reg_val && reg_val != riscv_cbom_block_size)
return -EINVAL;
break;
case KVM_REG_RISCV_CONFIG_REG(zicboz_block_size):
- if (kvm_riscv_isa_check_host(ZICBOZ))
- return -ENOENT;
- if (reg_val != riscv_cboz_block_size)
+ if (reg_val && reg_val != riscv_cboz_block_size)
return -EINVAL;
break;
case KVM_REG_RISCV_CONFIG_REG(zicbop_block_size):
- if (kvm_riscv_isa_check_host(ZICBOP))
- return -ENOENT;
- if (reg_val != riscv_cbop_block_size)
+ if (reg_val && reg_val != riscv_cbop_block_size)
return -EINVAL;
break;
case KVM_REG_RISCV_CONFIG_REG(mvendorid):
{
struct kvm_vcpu_csr *csr = &vcpu->arch.guest_csr;
unsigned long regs_max = sizeof(struct kvm_riscv_csr) / sizeof(unsigned long);
+ unsigned long flags;
if (reg_num >= regs_max)
return -ENOENT;
((unsigned long *)csr)[reg_num] = reg_val;
- if (reg_num == KVM_REG_RISCV_CSR_REG(sip))
- WRITE_ONCE(vcpu->arch.irqs_pending_mask[0], 0);
+ if (reg_num == KVM_REG_RISCV_CSR_REG(sip)) {
+ raw_spin_lock_irqsave(&vcpu->arch.irqs_pending_lock, flags);
+ vcpu->arch.irqs_pending_mask[0] = 0;
+ raw_spin_unlock_irqrestore(&vcpu->arch.irqs_pending_lock, flags);
+ }
return 0;
}
u64 size;
u64 reg;
- /*
- * Avoid reporting config reg if the corresponding extension
- * was not available.
- */
- if (i == KVM_REG_RISCV_CONFIG_REG(zicbom_block_size) &&
- kvm_riscv_isa_check_host(ZICBOM))
- continue;
- else if (i == KVM_REG_RISCV_CONFIG_REG(zicboz_block_size) &&
- kvm_riscv_isa_check_host(ZICBOZ))
- continue;
- else if (i == KVM_REG_RISCV_CONFIG_REG(zicbop_block_size) &&
- kvm_riscv_isa_check_host(ZICBOP))
- continue;
-
size = IS_ENABLED(CONFIG_32BIT) ? KVM_REG_SIZE_U32 : KVM_REG_SIZE_U64;
reg = KVM_REG_RISCV | size | KVM_REG_RISCV_CONFIG | i;
}
}
/* Start the counters that have been configured and requested by the guest */
- for_each_set_bit(i, &ctr_mask, RISCV_MAX_COUNTERS) {
+ for_each_set_bit(i, &ctr_mask, BITS_PER_LONG) {
pmc_index = array_index_nospec(i + ctr_base,
RISCV_KVM_MAX_COUNTERS);
if (!test_bit(pmc_index, kvpmu->pmc_in_use))
}
/* Stop the counters that have been configured and requested by the guest */
- for_each_set_bit(i, &ctr_mask, RISCV_MAX_COUNTERS) {
+ for_each_set_bit(i, &ctr_mask, BITS_PER_LONG) {
pmc_index = array_index_nospec(i + ctr_base,
RISCV_KVM_MAX_COUNTERS);
if (!test_bit(pmc_index, kvpmu->pmc_in_use))
if (conf->flags & SBI_FWFT_SET_FLAG_LOCK)
return SBI_ERR_DENIED_LOCKED;
- conf->flags = flags;
+ ret = conf->feature->set(vcpu, conf, false, value);
+ if (ret == SBI_SUCCESS)
+ conf->flags = flags;
- return conf->feature->set(vcpu, conf, false, value);
+ return ret;
}
static int kvm_sbi_fwft_get(struct kvm_vcpu *vcpu, unsigned long feature,
#include <linux/errno.h>
#include <linux/err.h>
#include <linux/kvm_host.h>
+#include <linux/nospec.h>
#include <linux/uaccess.h>
#include <asm/cpufeature.h>
#include <asm/kvm_isa.h>
return -ENOENT;
}
} else if (reg_num <= KVM_REG_RISCV_VECTOR_REG(31)) {
+ unsigned long reg_offset;
+
if (reg_size != vlenb)
return -EINVAL;
WARN_ON(!cntx->vector.datap);
- *reg_addr = cntx->vector.datap +
- (reg_num - KVM_REG_RISCV_VECTOR_REG(0)) * vlenb;
+ /*
+ * The reg_num is derived from the userspace-provided ONE_REG
+ * id. Sanitize it with array_index_nospec() to prevent
+ * speculative out-of-bounds access to the vector register
+ * buffer (32 vector registers: v0..v31).
+ */
+ reg_offset = array_index_nospec(
+ reg_num - KVM_REG_RISCV_VECTOR_REG(0), 32);
+ *reg_addr = cntx->vector.datap + reg_offset * vlenb;
} else {
return -ENOENT;
}
#include "../kernel/head.h"
+#if defined(CONFIG_64BIT) && defined(CONFIG_MMU)
DECLARE_BITMAP(new_valid_map_cpus, NR_CPUS);
+#endif
struct kernel_mapping kernel_map __ro_after_init;
EXPORT_SYMBOL(kernel_map);
static int memtop_get_page_count(unsigned long *res, unsigned long level)
{
static unsigned long memtop_pages[DIAG310_LEVELMAX];
- unsigned long pages;
+ unsigned long pages, idx;
int rc;
if (level > DIAG310_LEVELMAX || level < DIAG310_LEVELMIN)
return -EINVAL;
- pages = READ_ONCE(memtop_pages[level - 1]);
+ idx = array_index_nospec(level - 1, ARRAY_SIZE(memtop_pages));
+ pages = READ_ONCE(memtop_pages[idx]);
if (!pages) {
rc = diag310_get_memtop_size(&pages, level);
if (rc)
return rc;
- WRITE_ONCE(memtop_pages[level - 1], pages);
+ WRITE_ONCE(memtop_pages[idx], pages);
}
*res = pages;
return 0;
#include <linux/init.h>
#include <linux/miscdevice.h>
#include <linux/perf_event.h>
+#include <linux/nospec.h>
#include <asm/cpu_mf.h>
#include <asm/hwctrset.h>
if (!is_userspace_event(ev)) {
if (ev >= ARRAY_SIZE(cpumf_generic_events_user))
return -EOPNOTSUPP;
+ ev = array_index_nospec(ev, ARRAY_SIZE(cpumf_generic_events_user));
ev = cpumf_generic_events_user[ev];
}
} else if (!attr->exclude_kernel && attr->exclude_user) {
if (!is_userspace_event(ev)) {
if (ev >= ARRAY_SIZE(cpumf_generic_events_basic))
return -EOPNOTSUPP;
+ ev = array_index_nospec(ev, ARRAY_SIZE(cpumf_generic_events_basic));
ev = cpumf_generic_events_basic[ev];
}
}
* Here we can supply some information useful to userland.
*/
+#include <linux/build-salt.h>
#include <linux/uts.h>
#include <linux/version.h>
#include <linux/elfnote.h>
ELFNOTE_START(Linux, 0, "a")
.long LINUX_VERSION_CODE
ELFNOTE_END
+
+BUILD_SALT
else if (walk->ops->pte_entry)
rc = dat_pte_walk_range(max(start, cur), min(end, next),
dereference_pmd(crste.pmd), walk);
+ if (rc)
+ break;
}
}
return rc;
/* Only allow one real-space gmap shadow. */
list_for_each_entry(sg, &parent->children, list) {
if (sg->guest_asce.r) {
- scoped_guard(write_lock, &parent->kvm->mmu_lock)
+ if (write_trylock(&parent->kvm->mmu_lock)) {
gmap_unshadow(sg);
+ write_unlock(&parent->kvm->mmu_lock);
+ } else {
+ gmap_put(new);
+ return ERR_PTR(-EAGAIN);
+ }
break;
}
}
* PGSTEs might have cmma_d set.
*/
WRITE_ONCE(kvm->arch.migration_mode, 0);
- if (kvm->arch.use_cmma)
- kvm_s390_sync_request_broadcast(kvm, KVM_REQ_STOP_MIGRATION);
+ if (!kvm->arch.use_cmma)
+ return 0;
+
+ kvm_s390_sync_request_broadcast(kvm, KVM_REQ_STOP_MIGRATION);
/* Clear cmma_d on all existing PGSTEs and set cmma_dirty_pages to 0. */
gmap_set_cmma_all_clean(kvm->arch.gmap);
atomic64_set(&kvm->arch.cmma_dirty_pages, 0);
gaite->gisc = fib->fmt0.isc;
gaite->count++;
- gaite->aisbo = fib->fmt0.aisbo;
- gaite->aisb = virt_to_phys(page_address(aisb_page) + (fib->fmt0.aisb &
- ~PAGE_MASK));
+ if (fib->fmt0.sum == 1) {
+ gaite->aisbo = fib->fmt0.aisbo;
+ gaite->aisb = virt_to_phys(page_address(aisb_page) +
+ (fib->fmt0.aisb & ~PAGE_MASK));
+ } else {
+ gaite->aisbo = 0;
+ gaite->aisb = 0;
+ }
aift->kzdev[zdev->aisb] = zdev->kzdev;
spin_unlock_irq(&aift->gait_lock);
unpin1:
unpin_user_page(aibv_page);
out:
+ kvm_s390_gisc_unregister(kvm, fib->fmt0.isc);
return rc;
}
if (!cpu_has_vx()) {
if (copy)
memcpy(dst, src, len);
- return cksm(dst, len, sum);
+ return cksm(src, len, sum);
}
kernel_fpu_begin(&vxstate, KERNEL_VXR_V16V23);
fpu_vlvgf(16, (__force u32)sum, 1);
return PAGE_ALIGN(STACK_TOP - gap - rnd);
}
-static int get_align_mask(struct file *filp, unsigned long flags)
+static unsigned long get_align_mask(struct file *filp, unsigned long flags)
{
if (filp && is_file_hugepages(filp))
return huge_page_mask_align(filp);
char val[MAX_ADDR_LEN] = { };
int ret;
- ret = cmdline_find_option("acpi_rsdp", val, MAX_ADDR_LEN);
+ ret = cmdline_find_option("acpi_rsdp", val, sizeof(val));
if (ret < 0)
return 0;
+ if (ret >= sizeof(val)) {
+ warn("acpi_rsdp= value too long; ignoring");
+ return 0;
+ }
+
if (boot_kstrtoul(val, 16, &addr))
return 0;
#endif
static void parse_console_uart8250(void)
{
char optstr[64], *options;
- int baud = DEFAULT_BAUD;
+ int baud;
int port = 0;
/*
else
return;
- if (options && (options[0] == ','))
- baud = simple_strtoull(options + 1, &options, 0);
- else
+ if (options && (options[0] == ',')) {
+ baud = simple_strtoull(options + 1, NULL, 0);
+ if (!baud)
+ baud = DEFAULT_BAUD;
+ } else {
baud = probe_baud(port);
+ }
if (port)
early_serial_init(port, baud);
amd_brs_disable();
}
-static bool amd_brs_match_plm(struct perf_event *event, u64 to)
+static bool amd_brs_match_plm(struct perf_event *event, u64 from, u64 to)
{
int type = event->attr.branch_sample_type;
int plm_k = PERF_SAMPLE_BRANCH_KERNEL | PERF_SAMPLE_BRANCH_HV;
int plm_u = PERF_SAMPLE_BRANCH_USER;
- if (!(type & plm_k) && kernel_ip(to))
+ if (!(type & plm_k) && (kernel_ip(to) || kernel_ip(from)))
return 0;
if (!(type & plm_u) && !kernel_ip(to))
*/
to = (u64)(((s64)to << shift) >> shift);
- if (!amd_brs_match_plm(event, to))
- continue;
-
rdmsrq(brs_from(brs_idx), from);
+ if (!amd_brs_match_plm(event, from, to))
+ continue;
+
perf_clear_branch_entry_bitfields(br+nr);
br[nr].from = from;
x86_pmu_enable_event(event);
}
-static void amd_pmu_enable_all(int added)
+static void __amd_pmu_enable_all(void)
{
struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
int idx;
- amd_brs_enable_all();
-
for_each_set_bit(idx, x86_pmu.cntr_mask, X86_PMC_IDX_MAX) {
/* only activate events which are marked as active */
if (!test_bit(idx, cpuc->active_mask))
}
}
+static void amd_pmu_enable_all(int added)
+{
+ amd_brs_enable_all();
+ __amd_pmu_enable_all();
+}
+
static void amd_pmu_v2_enable_event(struct perf_event *event)
{
struct hw_perf_event *hwc = &event->hw;
* set global enable bits once again
*/
amd_pmu_v2_disable_all();
- amd_pmu_enable_all(0);
+ __amd_pmu_enable_all();
amd_pmu_v2_enable_all(0);
return;
}
}
/* If type does not correspond, then discard */
- if (type == X86_BR_NONE || (br_sel & type) != type) {
+ if (type == X86_BR_NONE || (br_sel & type) != type ||
+ (!(br_sel & X86_BR_KERNEL) && kernel_ip(cpuc->lbr_entries[i].from))) {
cpuc->lbr_entries[i].from = 0; /* mark invalid */
compress = true;
}
obj-$(CONFIG_CPU_SUP_CYRIX_32) += cyrix.o
obj-$(CONFIG_CPU_SUP_CENTAUR) += centaur.o
obj-$(CONFIG_CPU_SUP_TRANSMETA_32) += transmeta.o
-obj-$(CONFIG_CPU_SUP_UMC_32) += umc.o
obj-$(CONFIG_CPU_SUP_ZHAOXIN) += zhaoxin.o
obj-$(CONFIG_CPU_SUP_VORTEX_32) += vortex.o
if (irqfd->irq_entry.type == KVM_IRQ_ROUTING_MSI) {
ret = kvm_pi_update_irte(irqfd, &irqfd->irq_entry);
- if (ret)
+ if (ret) {
kvm->arch.nr_possible_bypass_irqs--;
+ irqfd->producer = NULL;
+ }
}
spin_unlock_irq(&kvm->irqfds.lock);
if (apic_lvtt_tscdeadline(apic) || ktimer->hv_timer_in_use)
ktimer->expired_tscdeadline = ktimer->tscdeadline;
- if (!from_timer_fn && apic->apicv_active) {
+ if (!from_timer_fn && apic->apicv_active && vcpu->wants_to_run) {
WARN_ON(kvm_get_running_vcpu() != vcpu);
kvm_apic_inject_pending_timer_irqs(apic);
return;
struct kvm_lapic *apic)
{
int vector;
+
+ if (unlikely(!pv_eoi_enabled(vcpu))) {
+ __clear_bit(KVM_APIC_PV_EOI_PENDING, &vcpu->arch.apic_attention);
+ return;
+ }
+
/*
* PV EOI state is derived from KVM_APIC_PV_EOI_PENDING in host
* and KVM_PV_EOI_ENABLED in guest memory as follows:
* KVM_APIC_PV_EOI_PENDING is set, KVM_PV_EOI_ENABLED is unset:
* -> host enabled PV EOI, guest executed EOI.
*/
- BUG_ON(!pv_eoi_enabled(vcpu));
-
if (pv_eoi_test_and_clr_pending(vcpu))
return;
vector = apic_set_eoi(apic);
*/
if (tdp_enabled && invalid_list &&
child->role.guest_mode &&
+ !child->root_count &&
!atomic_long_read(&child->parent_ptes.val))
return kvm_mmu_prepare_zap_page(kvm, child,
invalid_list);
if (r != RET_PF_CONTINUE)
return r;
- r = RET_PF_RETRY;
write_lock(&vcpu->kvm->mmu_lock);
- if (is_page_fault_stale(vcpu, fault))
- goto out_unlock;
-
r = make_mmu_pages_available(vcpu);
if (r)
goto out_unlock;
+ if (is_page_fault_stale(vcpu, fault)) {
+ r = RET_PF_RETRY;
+ goto out_unlock;
+ }
+
r = direct_map(vcpu, fault);
out_unlock:
static void mmu_destroy_caches(void)
{
kmem_cache_destroy(pte_list_desc_cache);
+ pte_list_desc_cache = NULL;
kmem_cache_destroy(mmu_page_header_cache);
+ mmu_page_header_cache = NULL;
}
static void kvm_wake_nx_recovery_thread(struct kvm *kvm)
}
#endif
- r = RET_PF_RETRY;
write_lock(&vcpu->kvm->mmu_lock);
- if (is_page_fault_stale(vcpu, fault))
- goto out_unlock;
-
r = make_mmu_pages_available(vcpu);
if (r)
goto out_unlock;
+
+ if (is_page_fault_stale(vcpu, fault)) {
+ r = RET_PF_RETRY;
+ goto out_unlock;
+ }
+
r = FNAME(fetch)(vcpu, fault, &walker);
out_unlock:
if (ret)
return ret;
+ /* Do not allow SNP VM migration until additional state transfer is implemented */
if (kvm->arch.vm_type != source_kvm->arch.vm_type ||
- sev_guest(kvm) || !sev_guest(source_kvm)) {
+ sev_guest(kvm) || !sev_guest(source_kvm) || sev_snp_guest(source_kvm)) {
ret = -EINVAL;
goto out_unlock;
}
* disallow out-of-band SEV/SEV-ES init if the target is already an
* SEV guest, or if vCPUs have been created. KVM relies on vCPUs being
* created after SEV/SEV-ES initialization, e.g. to init intercepts.
+ * Also do not allow SNP VM mirroring until additional state transfer is implemented.
*/
- if (sev_guest(kvm) || !sev_guest(source_kvm) ||
+ if (sev_guest(kvm) || !sev_guest(source_kvm) || sev_snp_guest(source_kvm) ||
is_mirroring_enc_context(source_kvm) || kvm->created_vcpus) {
ret = -EINVAL;
goto e_unlock;
return r;
sd = per_cpu_ptr(&svm_data, me);
- sd->asid_generation = 1;
+ /*
+ * Bump the current asid_generation value to ensure any vCPU that
+ * previously ran on this CPU sees a stale generation and is forced
+ * to acquire a new ASID, preventing a latent ASID collision.
+ */
+ sd->asid_generation++;
sd->max_asid = cpuid_ebx(SVM_CPUID_FUNC) - 1;
sd->next_asid = sd->max_asid + 1;
sd->min_asid = max_sev_asid + 1;
TP_printk("%s%s%s%s%s",
__print_symbolic(__entry->exception, kvm_trace_sym_exc),
!__entry->has_error ? "" : " (",
- !__entry->has_error ? "" : __print_symbolic(__entry->error_code, { }),
+ !__entry->has_error ? "" : __print_symbolic(__entry->error_code),
!__entry->has_error ? "" : ")",
__entry->reinjected ? " [reinjected]" : "")
);
static void free_nested(struct kvm_vcpu *vcpu)
{
struct vcpu_vmx *vmx = to_vmx(vcpu);
+ struct vmcs *shadow_vmcs;
if (WARN_ON_ONCE(vmx->loaded_vmcs != &vmx->vmcs01))
vmx_switch_vmcs(vcpu, &vmx->vmcs01);
vmx->nested.current_vmptr = INVALID_GPA;
if (enable_shadow_vmcs) {
vmx_disable_shadow_vmcs(vmx);
- vmcs_clear(vmx->vmcs01.shadow_vmcs);
- free_vmcs(vmx->vmcs01.shadow_vmcs);
+
+ /*
+ * Keep the pointer visible until after VMCLEAR, so migration
+ * can clear an active shadow VMCS on the old CPU.
+ */
+ shadow_vmcs = vmx->vmcs01.shadow_vmcs;
+ vmcs_clear(shadow_vmcs);
vmx->vmcs01.shadow_vmcs = NULL;
+ free_vmcs(shadow_vmcs);
}
kfree(vmx->nested.cached_vmcs12);
vmx->nested.cached_vmcs12 = NULL;
static int nested_vmx_check_tpr_shadow_controls(struct kvm_vcpu *vcpu,
struct vmcs12 *vmcs12)
{
+ gpa_t vtpr_gpa = vmcs12->virtual_apic_page_addr + APIC_TASKPRI;
+ u32 vtpr;
+
if (!nested_cpu_has(vmcs12, CPU_BASED_TPR_SHADOW))
return 0;
if (CC(!nested_cpu_has_vid(vmcs12) && vmcs12->tpr_threshold >> 4))
return -EINVAL;
+ /*
+ * Do the illegal vTPR vs. TPR Threshold consistency check if and only
+ * if KVM is configured to WARN on missed consistency checks, otherwise
+ * it's a waste of time. KVM needs to rely on hardware to fully detect
+ * an illegal combination due to the vTPR being writable by L1 at all
+ * times (it's an in-memory value, not a VMCS field). I.e. even if the
+ * check passes now, it might fail at the actual VM-Enter.
+ *
+ * If reading guest memory fails, skip the check as KVM's de facto ABI
+ * for VMX instruction accesses to non-existent memory is to provide
+ * PCI Bus Error semantics (reads return 0xFFs), in which case the vTPR
+ * is guaranteed to greater than or equal to the threshold.
+ *
+ * Note! Deliberately use the VM-scoped API when reading guest memory,
+ * to ensure the read doesn't hit SMRAM when restoring L2 state on RSM,
+ * and only perform the check when in KVM_RUN, to avoid a false failure
+ * if userspace hasn't yet configured memslots during state restore.
+ */
+ if (warn_on_missed_cc && vcpu->wants_to_run &&
+ nested_cpu_has(vmcs12, CPU_BASED_TPR_SHADOW) &&
+ !nested_cpu_has_vid(vmcs12) &&
+ !nested_cpu_has2(vmcs12, SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES) &&
+ !kvm_read_guest(vcpu->kvm, vtpr_gpa, &vtpr, sizeof(vtpr)) &&
+ CC((vmcs12->tpr_threshold & GENMASK(3, 0)) > ((vtpr >> 4) & GENMASK(3, 0))))
+ return -EINVAL;
+
return 0;
}
return 0;
}
-static int nested_vmx_check_controls_late(struct kvm_vcpu *vcpu,
- struct vmcs12 *vmcs12)
-{
- void *vapic = to_vmx(vcpu)->nested.virtual_apic_map.hva;
- u32 vtpr = vapic ? (*(u32 *)(vapic + APIC_TASKPRI)) >> 4 : 0;
-
- /*
- * Don't bother with the consistency checks if KVM isn't configured to
- * WARN on missed consistency checks, as KVM needs to rely on hardware
- * to fully detect an illegal vTPR vs. TRP Threshold combination due to
- * the vTPR being writable by L1 at all times (it's an in-memory value,
- * not a VMCS field). I.e. even if the check passes now, it might fail
- * at the actual VM-Enter.
- *
- * Keying off the module param also allows treating an invalid vAPIC
- * mapping as a consistency check failure without increasing the risk
- * of breaking a "real" VM.
- */
- if (!warn_on_missed_cc)
- return 0;
-
- if ((exec_controls_get(to_vmx(vcpu)) & CPU_BASED_TPR_SHADOW) &&
- nested_cpu_has(vmcs12, CPU_BASED_TPR_SHADOW) &&
- !nested_cpu_has_vid(vmcs12) &&
- !nested_cpu_has2(vmcs12, SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES) &&
- (CC(!vapic) ||
- CC((vmcs12->tpr_threshold & GENMASK(3, 0)) > (vtpr & GENMASK(3, 0)))))
- return -EINVAL;
-
- return 0;
-}
-
static int nested_vmx_check_address_space_size(struct kvm_vcpu *vcpu,
struct vmcs12 *vmcs12)
{
&vmx->nested.pre_vmenter_ssp_tbl);
/*
- * Overwrite vmcs01.GUEST_CR3 with L1's CR3 if EPT is disabled. In the
- * event of a "late" VM-Fail, i.e. a VM-Fail detected by hardware but
- * not KVM, KVM must unwind its software model to the pre-VM-Entry host
- * state. When EPT is disabled, GUEST_CR3 holds KVM's shadow CR3, not
- * L1's "real" CR3, which causes nested_vmx_restore_host_state() to
- * corrupt vcpu->arch.cr3. Stuffing vmcs01.GUEST_CR3 results in the
- * unwind naturally setting arch.cr3 to the correct value. Smashing
- * vmcs01.GUEST_CR3 is safe because nested VM-Exits, and the unwind,
- * reset KVM's MMU, i.e. vmcs01.GUEST_CR3 is guaranteed to be
- * overwritten with a shadow CR3 prior to re-entering L1.
+ * Stash L1's CR3, so that in the event of a "late" VM-Fail, i.e. a
+ * VM-Fail detected by hardware but not KVM, KVM can unwind its
+ * software model to the pre-VM-Entry host state. When EPT is
+ * disabled, GUEST_CR3 holds KVM's shadow CR3, not L1's "real" CR3,
+ * and so simply restoring from vmcs01.GUEST_CR3 would corrupt
+ * vcpu->arch.cr3.
*/
- if (!enable_ept)
- vmcs_writel(GUEST_CR3, vcpu->arch.cr3);
+ vmx->nested.pre_vmenter_cr3 = kvm_read_cr3(vcpu);
vmx_switch_vmcs(vcpu, &vmx->nested.vmcs02);
return NVMX_VMENTRY_KVM_INTERNAL_ERROR;
}
- if (nested_vmx_check_controls_late(vcpu, vmcs12)) {
- vmx_switch_vmcs(vcpu, &vmx->vmcs01);
- return NVMX_VMENTRY_VMFAIL;
- }
-
if (nested_vmx_check_guest_state(vcpu, vmcs12,
&entry_failure_code)) {
exit_reason.basic = EXIT_REASON_INVALID_STATE;
if (!from_vmentry)
return NVMX_VMENTRY_VMEXIT;
+ nested_put_vmcs12_pages(vcpu);
+
load_vmcs12_host_state(vcpu, vmcs12);
vmcs12->vm_exit_reason = exit_reason.full;
if (enable_shadow_vmcs || nested_vmx_is_evmptr12_valid(vmx))
vmx_set_cr4(vcpu, vmcs_readl(CR4_READ_SHADOW));
nested_ept_uninit_mmu_context(vcpu);
- vcpu->arch.cr3 = vmcs_readl(GUEST_CR3);
+ vcpu->arch.cr3 = vmx->nested.pre_vmenter_cr3;
kvm_register_mark_available(vcpu, VCPU_REG_CR3);
/*
goto out;
}
- if (init_vm->cpuid.padding) {
+ /*
+ * Reject the request if userspace changes cpuid.nent between the
+ * initial read and the subsequent copy.
+ */
+ if (init_vm->cpuid.padding || init_vm->cpuid.nent != nr_user_entries) {
ret = -EINVAL;
goto out;
}
bool has_preemption_timer_deadline;
bool preemption_timer_expired;
+ /*
+ * Used to restore L1's CR3 if hardware detects a VM-Fail Consistency
+ * Check that KVM does not, in which case KVM needs to unwind CR3 back
+ * to its pre-VM-Enter state, NOT to vmcs01.HOST_CR3.
+ */
+ unsigned long pre_vmenter_cr3;
+
/*
* Used to snapshot MSRs that are conditionally loaded on VM-Enter in
* order to propagate the guest's pre-VM-Enter value into vmcs02. For
if (!pci_is_display(pdev))
return false;
- if (pdev == vga_default_device())
- return true;
-
#ifdef CONFIG_SCREEN_INFO
numres = screen_info_resources(si, res, ARRAY_SIZE(res));
- for (i = 0; i < numres; ++i) {
- if (!(res[i].flags & IORESOURCE_MEM))
- continue;
+ if (numres > 0) {
+ for (i = 0; i < numres; ++i) {
+ if (!(res[i].flags & IORESOURCE_MEM))
+ continue;
+
+ if (pci_find_resource(pdev, &res[i]))
+ return true;
+ }
- if (pci_find_resource(pdev, &res[i]))
- return true;
+ return false;
}
#endif
- return false;
+ /*
+ * No framebuffer was set up by the firmware/bootloader, so fall back
+ * to the default VGA device.
+ */
+ return pdev == vga_default_device();
}
EXPORT_SYMBOL(video_is_primary_device);
goto unlock;
}
+ wbinvd_on_all_cpus();
+
/*
* MtrrFixDramModEn is not shared between threads on a core,
* therefore it must be set on all CPUs prior to enabling SNP.
bio = bio_alloc_percpu_cache(bs);
} else {
opf &= ~REQ_ALLOC_CACHE;
+ }
+
+ /*
+ * For a bioset without a percpu cache, or when the percpu cache was
+ * empty, try a slab allocation with optimistic GFP_ flags before
+ * falling back to the mempool.
+ */
+ if (!bio) {
p = kmem_cache_alloc(bs->bio_slab, gfp);
if (p)
bio = p + bs->front_pad;
* for the next iteration.
*/
static int bio_iov_iter_align_down(struct bio *bio, struct iov_iter *iter,
- unsigned len_align_mask)
+ struct bio_vec *bv, unsigned len_align_mask)
{
size_t nbytes = bio->bi_iter.bi_size & len_align_mask;
iov_iter_revert(iter, nbytes);
bio->bi_iter.bi_size -= nbytes;
- do {
- struct bio_vec *bv = &bio->bi_io_vec[bio->bi_vcnt - 1];
-
- if (nbytes < bv->bv_len) {
- bv->bv_len -= nbytes;
- break;
- }
-
+ while (nbytes >= bv->bv_len) {
if (bio_flagged(bio, BIO_PAGE_PINNED))
unpin_user_page(bv->bv_page);
- bio->bi_vcnt--;
+ if (!--bio->bi_vcnt)
+ return -EFAULT;
nbytes -= bv->bv_len;
- } while (nbytes);
-
- if (!bio->bi_vcnt)
- return -EFAULT;
+ bv--;
+ }
+ bv->bv_len -= nbytes;
return 0;
}
if (is_pci_p2pdma_page(bio->bi_io_vec->bv_page))
bio->bi_opf |= REQ_NOMERGE;
- return bio_iov_iter_align_down(bio, iter, len_align_mask);
+ return bio_iov_iter_align_down(bio, iter,
+ &bio->bi_io_vec[bio->bi_vcnt - 1], len_align_mask);
}
static struct folio *folio_alloc_greedy(gfp_t gfp, size_t *size,
struct folio *folio;
while (*size > minsize) {
- folio = folio_alloc(gfp | __GFP_NORETRY, get_order(*size));
+ folio = folio_alloc(gfp | __GFP_NORETRY | __GFP_NOWARN,
+ get_order(*size));
if (folio)
return folio;
*size = rounddown_pow_of_two(*size - 1);
bio_for_each_bvec_all(bv, bio, i) {
struct folio *folio = bvec_folio(bv);
- if (!is_zero_folio(folio))
+ if (!is_zero_folio(folio) && !is_huge_zero_folio(folio))
folio_put(folio);
}
}
if (!bio->bi_iter.bi_size)
return -ENOMEM;
- return bio_iov_iter_align_down(bio, iter, minsize - 1);
+ return bio_iov_iter_align_down(bio, iter,
+ &bio->bi_io_vec[bio->bi_vcnt - 1], minsize - 1);
}
static int bio_iov_iter_bounce_read(struct bio *bio, struct iov_iter *iter,
{
size_t len = min3(iov_iter_count(iter), maxlen, SZ_1M);
struct folio *folio;
+ ssize_t ret;
folio = folio_alloc_greedy(GFP_KERNEL, &len, minsize);
if (!folio)
return -ENOMEM;
do {
- ssize_t ret;
-
ret = iov_iter_extract_bvecs(iter, bio->bi_io_vec + 1, len,
&bio->bi_vcnt, bio->bi_max_vecs - 1, 0);
if (ret <= 0) {
- if (!bio->bi_vcnt) {
- folio_put(folio);
- return ret;
- }
+ if (!bio->bi_vcnt)
+ goto out_folio_put;
break;
}
len -= ret;
bvec_set_folio(&bio->bi_io_vec[0], folio, bio->bi_iter.bi_size, 0);
if (iov_iter_extract_will_pin(iter))
bio_set_flag(bio, BIO_PAGE_PINNED);
- return bio_iov_iter_align_down(bio, iter, minsize - 1);
+
+ /* The first vec stores the bounce buffer, so do not subtract 1 here. */
+ ret = bio_iov_iter_align_down(bio, iter,
+ &bio->bi_io_vec[bio->bi_vcnt], minsize - 1);
+ if (ret)
+ goto out_folio_put;
+
+ /* Update the bounc buffer bv_len to the aligned down size. */
+ bio->bi_io_vec[0].bv_len = bio->bi_iter.bi_size;
+ return 0;
+
+out_folio_put:
+ folio_put(folio);
+ return ret;
}
/**
blkg->pd[i]->online = true;
}
}
+ blkg->online = true;
}
- blkg->online = true;
spin_unlock(&blkcg->lock);
if (!ret)
return blkg;
/* @blkg failed fully initialized, use the usual release path */
- blkg_put(blkg);
+ percpu_ref_kill(&blkg->refcnt);
return ERR_PTR(ret);
err_free_blkg:
gfp_t gfp_mask)
{
unsigned long addr = (unsigned long) kbuf;
+ bool do_copy;
struct bio *bio;
int ret;
if (!len || !kbuf)
return -EINVAL;
- if (!blk_rq_aligned(rq->q, addr, len) || object_is_on_stack(kbuf))
+ do_copy = !blk_rq_aligned(rq->q, addr, len) || object_is_on_stack(kbuf);
+ if (do_copy)
bio = bio_copy_kern(rq, kbuf, len, gfp_mask);
else
bio = bio_map_kern(rq, kbuf, len, gfp_mask);
return PTR_ERR(bio);
ret = blk_rq_append_bio(rq, bio);
- if (unlikely(ret))
+ if (unlikely(ret)) {
+ if (do_copy)
+ bio_free_pages(bio);
blk_mq_map_bio_put(bio);
+ }
return ret;
}
EXPORT_SYMBOL(blk_rq_map_kern);
struct io_comp_batch *iob, unsigned int flags)
{
int ret;
+ unsigned long timeout = jiffies + 2;
do {
ret = q->mq_ops->poll(hctx, iob);
if (ret < 0 || (flags & BLK_POLL_ONESHOT))
break;
cpu_relax();
- } while (!need_resched());
+ } while (!need_resched() && time_before(jiffies, timeout));
return 0;
}
wbt_update_limits(RQWB(rqos));
}
+static bool wbt_set_lat_changed(struct request_queue *q, u64 val)
+{
+ struct rq_qos *rqos = wbt_rq_qos(q);
+ struct rq_wb *rwb;
+
+ if (!rqos)
+ return true;
+
+ rwb = RQWB(rqos);
+ if (rwb->min_lat_nsec != val)
+ return true;
+
+ return rwb_enabled(rwb) != !!val;
+}
+
static void wbt_exit(struct rq_qos *rqos)
{
struct rq_wb *rwb = RQWB(rqos);
else if (val >= 0)
val *= 1000ULL;
- if (wbt_get_min_lat(q) == val)
+ mutex_lock(&disk->rqos_state_mutex);
+ if (!wbt_set_lat_changed(q, val)) {
+ mutex_unlock(&disk->rqos_state_mutex);
goto out;
+ }
+ mutex_unlock(&disk->rqos_state_mutex);
blk_mq_quiesce_queue(q);
if (!disk->zone_wplugs_pool)
goto free_hash;
- disk->zone_wplugs_wq =
- alloc_workqueue("%s_zwplugs", WQ_MEM_RECLAIM | WQ_HIGHPRI | WQ_PERCPU,
- pool_size, disk->disk_name);
- if (!disk->zone_wplugs_wq)
- goto destroy_pool;
+ /*
+ * We may already have a zone write plug workqueue as this function may
+ * be called after disk_free_zone_resources(), which does not destroy
+ * the workqueue (the zone write plugs workqueue is destroyed at
+ * disk_release() time).
+ */
+ if (!disk->zone_wplugs_wq) {
+ disk->zone_wplugs_wq =
+ alloc_workqueue("%s_zwplugs",
+ WQ_MEM_RECLAIM | WQ_HIGHPRI | WQ_PERCPU,
+ pool_size, disk->disk_name);
+ if (!disk->zone_wplugs_wq)
+ goto destroy_pool;
+ }
disk->zone_wplugs_worker =
kthread_create(disk_zone_wplugs_worker, disk,
if (IS_ERR(disk->zone_wplugs_worker)) {
ret = PTR_ERR(disk->zone_wplugs_worker);
disk->zone_wplugs_worker = NULL;
- goto destroy_wq;
+ goto destroy_pool;
}
wake_up_process(disk->zone_wplugs_worker);
return 0;
-destroy_wq:
- destroy_workqueue(disk->zone_wplugs_wq);
- disk->zone_wplugs_wq = NULL;
destroy_pool:
mempool_destroy(disk->zone_wplugs_pool);
disk->zone_wplugs_pool = NULL;
kfree_rcu_mightsleep(zones_cond);
}
-void disk_free_zone_resources(struct gendisk *disk)
+static void disk_free_zone_resources(struct gendisk *disk)
{
if (disk->zone_wplugs_worker) {
kthread_stop(disk->zone_wplugs_worker);
}
WARN_ON_ONCE(!list_empty(&disk->zone_wplugs_list));
- if (disk->zone_wplugs_wq) {
- destroy_workqueue(disk->zone_wplugs_wq);
- disk->zone_wplugs_wq = NULL;
- }
+ if (disk->zone_wplugs_wq)
+ drain_workqueue(disk->zone_wplugs_wq);
disk_destroy_zone_wplugs_hash_table(disk);
disk->nr_zones = 0;
}
+void disk_release_zone_resources(struct gendisk *disk)
+{
+ if (disk->zone_wplugs_wq) {
+ destroy_workqueue(disk->zone_wplugs_wq);
+ disk->zone_wplugs_wq = NULL;
+ }
+
+ disk_free_zone_resources(disk);
+}
+
struct blk_revalidate_zone_args {
struct gendisk *disk;
u8 *zones_cond;
#ifdef CONFIG_BLK_DEV_ZONED
void disk_init_zone_resources(struct gendisk *disk);
-void disk_free_zone_resources(struct gendisk *disk);
+void disk_release_zone_resources(struct gendisk *disk);
static inline bool bio_zone_write_plugging(struct bio *bio)
{
return bio_flagged(bio, BIO_ZONE_WRITE_PLUGGING);
static inline void disk_init_zone_resources(struct gendisk *disk)
{
}
-static inline void disk_free_zone_resources(struct gendisk *disk)
+static inline void disk_release_zone_resources(struct gendisk *disk)
{
}
static inline bool bio_zone_write_plugging(struct bio *bio)
static inline u64 blk_time_get_ns(void)
{
struct blk_plug *plug = current->plug;
+ u64 now;
if (!plug || !in_task())
return ktime_get_ns();
* 0 could very well be a valid time, but rather than flag "this is
* a valid timestamp" separately, just accept that we'll do an extra
* ktime_get_ns() if we just happen to get 0 as the current time.
+ *
+ * cur_ktime can be zeroed by pre-emption the moment PF_BLOCK_TS is set.
*/
- if (!plug->cur_ktime) {
- plug->cur_ktime = ktime_get_ns();
+ now = READ_ONCE(plug->cur_ktime);
+ if (!now) {
+ now = ktime_get_ns();
+ WRITE_ONCE(plug->cur_ktime, now);
+ /* Ensure PF_BLOCK_TS is set after cur_ktime. */
+ barrier();
current->flags |= PF_BLOCK_TS;
}
- return plug->cur_ktime;
+ return now;
}
static inline ktime_t blk_time_get(void)
* reference during concurrent disk deletion:
* update_nr_hwq_lock -> kn->active (via del_gendisk -> kobject_del)
* kn->active -> update_nr_hwq_lock (via this sysfs write path)
+ *
+ * Use the writer lock instead of the reader lock of update_nr_hwq_lock
+ * to serialize the two-stage elevator switch steps in
+ * elevator_change(): the core switch step under the elevator lock and
+ * the elevator_change_done() step outside the elevator lock.
*/
- if (!down_read_trylock(&set->update_nr_hwq_lock)) {
+ if (!down_write_trylock(&set->update_nr_hwq_lock)) {
ret = -EBUSY;
goto out;
}
} else {
ret = -ENOENT;
}
- up_read(&set->update_nr_hwq_lock);
+ up_write(&set->update_nr_hwq_lock);
out:
if (ctx.type)
rcu_read_lock();
list_for_each_entry_rcu(inj, &disk->error_injection_list, entry) {
- seq_printf(s, "%llu:%llu status=%s,chance=%u",
- inj->start, inj->end,
- blk_status_to_tag(inj->status), inj->chance);
+ seq_printf(s, "%llu:%llu op=%s,status=%s,chance=%u",
+ inj->start, inj->end,
+ blk_op_str(inj->op),
+ blk_status_to_tag(inj->status), inj->chance);
seq_putc(s, '\n');
}
rcu_read_unlock();
struct device *ddev = disk_to_dev(disk);
if (!(disk->flags & GENHD_FL_HIDDEN)) {
- /* Make sure the first partition scan will be proceed */
- if (get_capacity(disk) && disk_has_partscan(disk))
- set_bit(GD_NEED_PART_SCAN, &disk->state);
-
bdev_add(disk->part0, ddev->devt);
if (get_capacity(disk))
disk_scan_partitions(disk, BLK_OPEN_READ);
disk_release_events(disk);
kfree(disk->random);
- disk_free_zone_resources(disk);
+ disk_release_zone_resources(disk);
xa_destroy(&disk->part_tbl);
kobject_put(&disk->queue_kobj);
if (n) {
int foundlvs = 0;
- for (i = 0; foundlvs < numlvs && i < state->limit; i += 1) {
+ /*
+ * The lvd array was read as a single sector; only the
+ * struct lvd entries that fit in it are valid. Bound the
+ * scan so an on-disk numlvs larger than that cannot walk
+ * the read buffer out of bounds.
+ */
+ for (i = 0; foundlvs < numlvs && i < state->limit &&
+ i < SECTOR_SIZE / (int)sizeof(struct lvd); i++) {
lvip[i].pps_per_lv = be16_to_cpu(p[i].num_lps);
if (lvip[i].pps_per_lv)
foundlvs += 1;
tristate "AES (Advanced Encryption Standard)"
select CRYPTO_ALGAPI
select CRYPTO_LIB_AES
- select CRYPTO_LIB_AES_CBC_MACS if CRYPTO_CMAC || CRYPTO_XCBC || CRYPTO_CCM
- select CRYPTO_HASH if CRYPTO_CMAC || CRYPTO_XCBC || CRYPTO_CCM
+ select CRYPTO_LIB_AES_CBC_MACS if CRYPTO_CMAC != n || CRYPTO_XCBC != n || CRYPTO_CCM != n
+ select CRYPTO_HASH if CRYPTO_CMAC != n || CRYPTO_XCBC != n || CRYPTO_CCM != n
help
AES cipher algorithms (Rijndael)(FIPS-197, ISO/IEC 18033-3)
if (!hwctx->cus)
return -ENOMEM;
- ret = amdxdna_pm_resume_get_locked(xdna);
+ ret = amdxdna_pm_resume_get(xdna);
if (ret)
goto free_cus;
static void aie2_cmd_wait(struct amdxdna_hwctx *hwctx, u64 seq)
{
struct dma_fence *out_fence = aie2_cmd_get_out_fence(hwctx, seq);
+ struct amdxdna_dev *xdna = hwctx->client->xdna;
if (!out_fence) {
- XDNA_ERR(hwctx->client->xdna, "Failed to get fence");
+ XDNA_ERR(xdna, "Failed to get fence");
return;
}
+ mutex_unlock(&xdna->dev_lock);
dma_fence_wait_timeout(out_fence, false, MAX_SCHEDULE_TIMEOUT);
+ mutex_lock(&xdna->dev_lock);
dma_fence_put(out_fence);
}
found = false;
down_write(&xdna->notifier_lock);
list_for_each_entry(mapp, &abo->mem.umap_list, node) {
- if (mapp->invalid) {
+ if (mapp->invalid && kref_get_unless_zero(&mapp->refcnt)) {
found = true;
break;
}
up_write(&xdna->notifier_lock);
return 0;
}
- kref_get(&mapp->refcnt);
+
up_write(&xdna->notifier_lock);
- XDNA_DBG(xdna, "populate memory range %lx %lx",
- mapp->vma->vm_start, mapp->vma->vm_end);
mm = mapp->notifier.mm;
if (!mmget_not_zero(mm)) {
amdxdna_umap_put(mapp);
u64 addr;
int ret;
- ret = amdxdna_pm_resume_get_locked(xdna);
- if (ret)
- return ret;
-
addr = amdxdna_obj_dma_addr(heap);
ret = aie2_add_host_buf(xdna->dev_handle, hwctx->fw_ctx_id,
addr, heap->mem.size);
hwctx->name, heap->mem.size, ret);
}
- amdxdna_pm_suspend_put(xdna);
-
return ret;
}
static int aie2_init_exec_req(void *req, struct amdxdna_gem_obj *cmd_abo,
size_t *size, u32 *msg_op)
{
- struct amdxdna_dev *xdna = cmd_abo->client->xdna;
+ struct amdxdna_dev *xdna = to_xdna_dev(to_gobj(cmd_abo)->dev);
int ret;
u32 op;
aie2_cmdlist_fill_slot(void *slot, struct amdxdna_gem_obj *cmd_abo,
size_t *size, u32 *cmd_op)
{
- struct amdxdna_dev *xdna = cmd_abo->client->xdna;
+ struct amdxdna_dev *xdna = to_xdna_dev(to_gobj(cmd_abo)->dev);
int ret;
u32 op;
if (!drm_dev_enter(dev, &idx))
return -ENODEV;
+ mutex_lock(&xdna->client_lock);
mutex_lock(&xdna->dev_lock);
hwctx = xa_erase(&client->hwctx_xa, args->handle);
if (!hwctx) {
XDNA_DBG(xdna, "PID %d destroyed HW context %d", client->pid, args->handle);
out:
mutex_unlock(&xdna->dev_lock);
+ mutex_unlock(&xdna->client_lock);
drm_dev_exit(idx);
return ret;
}
return -EINVAL;
}
- guard(mutex)(&xdna->dev_lock);
+ ret = amdxdna_pm_resume_get(xdna);
+ if (ret) {
+ XDNA_ERR(xdna, "Resume failed, ret %d", ret);
+ goto free_buf;
+ }
+
+ mutex_lock(&xdna->client_lock);
+ mutex_lock(&xdna->dev_lock);
hwctx = xa_load(&client->hwctx_xa, args->handle);
if (!hwctx) {
XDNA_DBG(xdna, "PID %d failed to get hwctx %d", client->pid, args->handle);
ret = -EINVAL;
- goto free_buf;
+ goto unlock;
}
ret = xdna->dev_info->ops->hwctx_config(hwctx, args->param_type, val, buf, buf_size);
+unlock:
+ mutex_unlock(&xdna->dev_lock);
+ mutex_unlock(&xdna->client_lock);
+ amdxdna_pm_suspend_put(xdna);
free_buf:
kfree(buf);
return ret;
if (!gobj)
return -EINVAL;
+ ret = amdxdna_pm_resume_get(xdna);
+ if (ret) {
+ XDNA_ERR(xdna, "Resume failed, ret %d", ret);
+ goto put_obj;
+ }
+
abo = to_xdna_obj(gobj);
- guard(mutex)(&xdna->dev_lock);
+ mutex_lock(&xdna->client_lock);
+ mutex_lock(&xdna->dev_lock);
hwctx = xa_load(&client->hwctx_xa, abo->assigned_hwctx);
if (!hwctx) {
ret = -EINVAL;
- goto put_obj;
+ goto unlock;
}
ret = xdna->dev_info->ops->hwctx_sync_debug_bo(hwctx, debug_bo_hdl);
+unlock:
+ mutex_unlock(&xdna->dev_lock);
+ mutex_unlock(&xdna->client_lock);
+ amdxdna_pm_suspend_put(xdna);
put_obj:
drm_gem_object_put(gobj);
return ret;
break;
}
- mutex_unlock(&client->mm_lock);
ret = xdna->dev_info->ops->hwctx_heap_expand(hwctx, heap);
- mutex_lock(&client->mm_lock);
if (ret) {
amdxdna_gem_unpin(heap);
drm_gem_object_put(to_gobj(heap));
unsigned long hwctx_id;
int ret;
- guard(mutex)(&client->mm_lock);
+ ret = amdxdna_pm_resume_get_locked(client->xdna);
+ if (ret)
+ return ret;
- if (hwctx)
- return amdxdna_hwctx_expand_heap(hwctx);
+ mutex_lock(&client->mm_lock);
- amdxdna_for_each_hwctx(client, hwctx_id, hwctx) {
+ if (hwctx) {
ret = amdxdna_hwctx_expand_heap(hwctx);
- if (ret)
- return ret;
+ } else {
+ amdxdna_for_each_hwctx(client, hwctx_id, hwctx) {
+ ret = amdxdna_hwctx_expand_heap(hwctx);
+ if (ret)
+ break;
+ }
}
+ mutex_unlock(&client->mm_lock);
- return 0;
+ amdxdna_pm_suspend_put(client->xdna);
+
+ return ret;
}
static void
amdxdna_arg_bos_put(job);
amdxdna_gem_put_obj(job->cmd_bo);
dma_fence_put(job->fence);
+ mmdrop(job->mm);
}
int amdxdna_cmd_submit(struct amdxdna_client *client,
int ret, idx;
XDNA_DBG(xdna, "Command BO hdl %d, Arg BO count %d", cmd_bo_hdl, arg_bo_cnt);
+
+ if (!xdna->dev_info->ops->cmd_submit)
+ return -EOPNOTSUPP;
+
job = kzalloc_flex(*job, bos, arg_bo_cnt);
if (!job)
return -ENOMEM;
ret = -EINVAL;
goto free_job;
}
+ } else if (!drv_cmd) {
+ /*
+ * Only internal driver commands (drv_cmd != NULL) may omit a
+ * command BO. A user command submission with the invalid handle
+ * would leave job->cmd_bo NULL and later fault when the scheduler
+ * dereferences it in amdxdna_cmd_set_state().
+ */
+ XDNA_DBG(xdna, "Command BO handle required for user submission");
+ ret = -EINVAL;
+ goto free_job;
}
ret = amdxdna_arg_bos_lookup(client, job, arg_bo_hdls, arg_bo_cnt);
job->hwctx = hwctx;
job->mm = current->mm;
+ mmgrab(job->mm);
job->fence = amdxdna_fence_create(hwctx);
if (!job->fence) {
cmd_put:
amdxdna_gem_put_obj(job->cmd_bo);
free_job:
+ if (job->mm)
+ mmdrop(job->mm);
kfree(job);
return ret;
}
*/
void *amdxdna_gem_vmap(struct amdxdna_gem_obj *abo)
{
+ struct amdxdna_dev *xdna = to_xdna_dev(to_gobj(abo)->dev);
struct iosys_map map = IOSYS_MAP_INIT_VADDR(NULL);
int ret;
if (!abo->mem.kva) {
ret = drm_gem_vmap(to_gobj(abo), &map);
if (ret)
- XDNA_ERR(abo->client->xdna, "Vmap bo failed, ret %d", ret);
+ XDNA_ERR(xdna, "Vmap bo failed, ret %d", ret);
else
abo->mem.kva = map.vaddr;
}
xdna = to_xdna_dev(to_gobj(abo)->dev);
XDNA_DBG(xdna, "Invalidating range 0x%lx, 0x%lx, type %d",
- mapp->vma->vm_start, mapp->vma->vm_end, abo->type);
+ mapp->range.start, mapp->range.end, abo->type);
if (!mmu_notifier_range_blockable(range))
return false;
.invalidate = amdxdna_hmm_invalidate,
};
+static inline bool compare_range(struct amdxdna_umap *mapp,
+ struct mm_struct *mm,
+ unsigned long start, unsigned long end)
+{
+ return (!mapp->unmapped && mapp->notifier.mm == mm &&
+ mapp->range.start == start && mapp->range.end == end);
+}
+
static void amdxdna_hmm_unregister(struct amdxdna_gem_obj *abo,
struct vm_area_struct *vma)
{
struct amdxdna_dev *xdna = to_xdna_dev(to_gobj(abo)->dev);
struct amdxdna_umap *mapp;
- down_read(&xdna->notifier_lock);
+ down_write(&xdna->notifier_lock);
list_for_each_entry(mapp, &abo->mem.umap_list, node) {
- if (!vma || mapp->vma == vma) {
+ if (!vma || compare_range(mapp, vma->vm_mm, vma->vm_start, vma->vm_end)) {
if (!mapp->unmapped) {
queue_work(xdna->notifier_wq, &mapp->hmm_unreg_work);
mapp->unmapped = true;
break;
}
}
- up_read(&xdna->notifier_lock);
+ up_write(&xdna->notifier_lock);
}
static void amdxdna_umap_release(struct kref *ref)
{
struct amdxdna_umap *mapp = container_of(ref, struct amdxdna_umap, refcnt);
struct amdxdna_gem_obj *abo = mapp->abo;
- struct vm_area_struct *vma = mapp->vma;
struct amdxdna_dev *xdna;
mmu_interval_notifier_remove(&mapp->notifier);
- if (is_import_bo(abo) && vma->vm_file && vma->vm_file->f_mapping)
- mapping_clear_unevictable(vma->vm_file->f_mapping);
xdna = to_xdna_dev(to_gobj(mapp->abo)->dev);
down_write(&xdna->notifier_lock);
unsigned long len = vma->vm_end - vma->vm_start;
unsigned long addr = vma->vm_start;
struct amdxdna_umap *mapp;
- u32 nr_pages;
+ unsigned long nr_pages;
int ret;
- if (!amdxdna_pasid_on(abo->client)) {
+ /*
+ * When PASID is off, amdxdna_gem_obj_open() called amdxdna_dma_map_bo()
+ * and mem.dma_addr is valid; use the DMA address directly and skip HMM.
+ * Avoid dereferencing abo->client which may be NULL (cleared in close())
+ * while internal kernel references are still held.
+ */
+ if (abo->mem.dma_addr != AMDXDNA_INVALID_ADDR) {
/* Need to set uva for heap uva validation */
abo->mem.uva = addr;
return 0;
}
+ down_read(&xdna->notifier_lock);
+ list_for_each_entry(mapp, &abo->mem.umap_list, node) {
+ if (compare_range(mapp, current->mm, addr, addr + len)) {
+ up_read(&xdna->notifier_lock);
+ return 0;
+ }
+ }
+ up_read(&xdna->notifier_lock);
+
mapp = kzalloc_obj(*mapp);
if (!mapp)
return -ENOMEM;
mapp->range.start = vma->vm_start;
mapp->range.end = vma->vm_end;
mapp->range.default_flags = HMM_PFN_REQ_FAULT;
- mapp->vma = vma;
mapp->abo = abo;
kref_init(&mapp->refcnt);
INIT_WORK(&mapp->hmm_unreg_work, amdxdna_hmm_unreg_work);
- if (is_import_bo(abo) && vma->vm_file && vma->vm_file->f_mapping)
- mapping_set_unevictable(vma->vm_file->f_mapping);
down_write(&xdna->notifier_lock);
if (list_empty(&abo->mem.umap_list))
close_vma:
vma->vm_ops->close(vma);
+ return ret;
put_obj:
drm_gem_object_put(gobj);
return ret;
/* No need to set up dma addr mapping in PASID mode. */
if (!amdxdna_pasid_on(abo->client)) {
ret = amdxdna_dma_map_bo(xdna, abo);
- if (ret)
+ if (ret) {
+ abo->open_ref--;
+ abo->client = NULL;
return ret;
+ }
}
amdxdna_gem_add_bo_usage(abo);
#include "amdxdna_pci_drv.h"
struct amdxdna_umap {
- struct vm_area_struct *vma;
struct mmu_interval_notifier notifier;
struct hmm_range range;
struct work_struct hmm_unreg_work;
static inline u64 amdxdna_dev_bo_offset(struct amdxdna_gem_obj *abo)
{
- return amdxdna_gem_dev_addr(abo) - abo->client->xdna->dev_info->dev_mem_base;
+ return amdxdna_gem_dev_addr(abo) - to_xdna_dev(to_gobj(abo)->dev)->dev_info->dev_mem_base;
}
static inline u64 amdxdna_obj_dma_addr(struct amdxdna_gem_obj *abo)
{
- return amdxdna_pasid_on(abo->client) ? amdxdna_gem_uva(abo) : abo->mem.dma_addr;
+ /*
+ * amdxdna_gem_obj_open() calls amdxdna_dma_map_bo() only when PASID is
+ * off, leaving mem.dma_addr at AMDXDNA_INVALID_ADDR when PASID is on.
+ * Avoid dereferencing abo->client, which is cleared to NULL by
+ * amdxdna_gem_obj_close() while internal kernel references remain.
+ */
+ return (abo->mem.dma_addr != AMDXDNA_INVALID_ADDR) ?
+ abo->mem.dma_addr : amdxdna_gem_uva(abo);
}
void amdxdna_umap_put(struct amdxdna_umap *mapp);
{
struct amdxdna_dev *xdna = to_xdna_dev(ddev);
struct amdxdna_client *client;
+ int ret;
client = kzalloc_obj(*client);
if (!client)
return -ENOMEM;
+ ret = init_srcu_struct(&client->hwctx_srcu);
+ if (ret)
+ goto free_client;
+
client->pid = pid_nr(rcu_access_pointer(filp->pid));
client->xdna = xdna;
client->pasid = IOMMU_PASID_INVALID;
XDNA_WARN(xdna, "PASID not available for pid %d", client->pid);
if (!amdxdna_use_carveout(xdna)) {
XDNA_ERR(xdna, "PASID unavailable and carveout not configured");
- kfree(client);
- return -EINVAL;
+ ret = -EINVAL;
+ goto cleanup_srcu;
}
}
}
mmgrab(client->mm);
- init_srcu_struct(&client->hwctx_srcu);
xa_init_flags(&client->hwctx_xa, XA_FLAGS_ALLOC);
xa_init_flags(&client->dev_heap_xa, XA_FLAGS_ALLOC);
drm_mm_init(&client->dev_heap_mm, xdna->dev_info->dev_mem_base,
XDNA_DBG(xdna, "pid %d opened", client->pid);
return 0;
+
+cleanup_srcu:
+ cleanup_srcu_struct(&client->hwctx_srcu);
+free_client:
+ kfree(client);
+ return ret;
}
static void amdxdna_client_cleanup(struct amdxdna_client *client)
if (ret)
return ret;
- drmm_mutex_init(ddev, &xdna->dev_lock);
+ ret = drmm_mutex_init(ddev, &xdna->dev_lock);
+ if (ret)
+ return ret;
+
init_rwsem(&xdna->notifier_lock);
INIT_LIST_HEAD(&xdna->client_list);
pci_set_drvdata(pdev, xdna);
ivpu_dbg(vdev, FW_BOOT, "Invalid header size 0x%x\n", log->header_size);
return -EINVAL;
}
+ if (log->size < log->header_size) {
+ ivpu_dbg(vdev, FW_BOOT, "Invalid log size 0x%x\n", log->size);
+ return -EINVAL;
+ }
if ((char *)log + log->size > (char *)ivpu_bo_vaddr(bo) + ivpu_bo_size(bo)) {
ivpu_dbg(vdev, FW_BOOT, "Invalid log size 0x%x\n", log->size);
return -EINVAL;
static void diagnose_failure_lnl(struct ivpu_device *vdev)
{
- u32 reg = REGB_RD32(VPU_HW_BTRS_MTL_INTERRUPT_STAT) & BTRS_LNL_IRQ_MASK;
+ u32 reg = REGB_RD32(VPU_HW_BTRS_LNL_INTERRUPT_STAT) & BTRS_LNL_IRQ_MASK;
if (REG_TEST_FLD(VPU_HW_BTRS_LNL_INTERRUPT_STAT, ATS_ERR, reg)) {
ivpu_err(vdev, "ATS_ERR_LOG1 0x%08x ATS_ERR_LOG2 0x%08x\n",
static void binder_free_transaction(struct binder_transaction *t)
{
- struct binder_proc *target_proc = t->to_proc;
+ struct binder_thread *target_thread;
+ struct binder_proc *target_proc;
+
+ spin_lock(&t->lock);
+ target_proc = t->to_proc;
+ target_thread = t->to_thread;
+ /*
+ * Pin target_thread to keep target_proc alive. Undelivered
+ * transactions with !target_thread are safe, as target_proc
+ * can only be the current context there.
+ */
+ if (target_thread)
+ atomic_inc(&target_thread->tmp_ref);
+ spin_unlock(&t->lock);
if (target_proc) {
binder_inner_proc_lock(target_proc);
t->buffer->transaction = NULL;
binder_inner_proc_unlock(target_proc);
}
+
+ if (target_thread)
+ binder_thread_dec_tmpref(target_thread);
+
if (trace_binder_txn_latency_free_enabled())
binder_txn_latency_free(t);
/*
int t_debug_id = atomic_inc_return(&binder_last_id);
ktime_t t_start_time = ktime_get();
struct lsm_context lsmctx = { };
+ size_t lsmctx_aligned_size = 0;
LIST_HEAD(sgc_head);
LIST_HEAD(pf_head);
const void __user *user_buffer = (const void __user *)
if (target_node && target_node->txn_security_ctx) {
u32 secid;
- size_t added_size;
security_cred_getsecid(proc->cred, &secid);
ret = security_secid_to_secctx(secid, &lsmctx);
return_error_line = __LINE__;
goto err_get_secctx_failed;
}
- added_size = ALIGN(lsmctx.len, sizeof(u64));
- extra_buffers_size += added_size;
- if (extra_buffers_size < added_size) {
+ lsmctx_aligned_size = ALIGN(lsmctx.len, sizeof(u64));
+ extra_buffers_size += lsmctx_aligned_size;
+ if (extra_buffers_size < lsmctx_aligned_size) {
binder_txn_error("%d:%d integer overflow of extra_buffers_size\n",
thread->pid, proc->pid);
return_error = BR_FAILED_REPLY;
size_t buf_offset = ALIGN(tr->data_size, sizeof(void *)) +
ALIGN(tr->offsets_size, sizeof(void *)) +
ALIGN(extra_buffers_size, sizeof(void *)) -
- ALIGN(lsmctx.len, sizeof(u64));
+ lsmctx_aligned_size;
t->security_ctx = t->buffer->user_data + buf_offset;
err = binder_alloc_copy_to_buffer(&target_proc->alloc,
off_end_offset = off_start_offset + tr->offsets_size;
sg_buf_offset = ALIGN(off_end_offset, sizeof(void *));
sg_buf_end_offset = sg_buf_offset + extra_buffers_size -
- ALIGN(lsmctx.len, sizeof(u64));
+ lsmctx_aligned_size;
off_min = 0;
for (buffer_offset = off_start_offset; buffer_offset < off_end_offset;
buffer_offset += sizeof(binder_size_t)) {
if let Some(offsets) = info.offsets.clone() {
let view = AllocationView::new(self, offsets.start);
- for i in offsets.step_by(size_of::<usize>()) {
+ for i in offsets.step_by(size_of::<u64>()) {
if view.cleanup_object(i).is_err() {
pr_warn!("Error cleaning up object at offset {}\n", i)
}
}
fn cleanup_object(&self, index_offset: usize) -> Result {
- let offset = self.alloc.read(index_offset)?;
+ let offset = self.alloc.read::<u64>(index_offset)?;
+ let offset: usize = offset.try_into().map_err(|_| EINVAL)?;
let header = self.read::<BinderObjectHeader>(offset)?;
match header.type_ {
BINDER_TYPE_WEAK_BINDER | BINDER_TYPE_BINDER => {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
match self.reply {
BR_FAILED_REPLY => match self.source.as_ref() {
- Some(source) => f
- .debug_struct("BR_FAILED_REPLY")
- .field("source", source)
- .finish(),
+ Some(source) => source.fmt(f),
None => f.pad("BR_FAILED_REPLY"),
},
BR_DEAD_REPLY => f.pad("BR_DEAD_REPLY"),
BR_FROZEN_REPLY => f.pad("BR_FROZEN_REPLY"),
BR_TRANSACTION_PENDING_FROZEN => f.pad("BR_TRANSACTION_PENDING_FROZEN"),
BR_TRANSACTION_COMPLETE => f.pad("BR_TRANSACTION_COMPLETE"),
- _ => f
- .debug_struct("BinderError")
- .field("reply", &self.reply)
- .finish(),
+ _ => match self.source.as_ref() {
+ Some(source) => source.fmt(f),
+ None => self.reply.fmt(f),
+ },
}
}
}
}
impl FreezeListener {
- pub(crate) fn on_process_exit(&self, proc: &Arc<Process>) {
+ /// Called when this freeze listener is cleared abnormally.
+ ///
+ /// This occurs either because the process exited or because the process dropped its last
+ /// refcount on the node ref without explicitly removing the freeze listener first.
+ ///
+ /// The returned `KVVec` is just a value that should be dropped outside of the lock.
+ pub(crate) fn on_process_cleanup(&self, proc: &Process) -> KVVec<Arc<Process>> {
if !self.is_clearing {
- self.node.remove_freeze_listener(proc);
+ return self.node.remove_freeze_listener(proc);
}
+ KVVec::new()
}
}
}
}
- pub(crate) fn remove_freeze_listener(&self, p: &Arc<Process>) {
- let _unused_capacity;
+ pub(crate) fn remove_freeze_listener(&self, p: &Process) -> KVVec<Arc<Process>> {
let mut guard = self.owner.inner.lock();
let inner = self.inner.access_mut(&mut guard);
let len = inner.freeze_list.len();
- inner.freeze_list.retain(|proc| !Arc::ptr_eq(proc, p));
+ inner
+ .freeze_list
+ .retain(|proc| !core::ptr::eq::<Process>(&**proc, p));
if len == inner.freeze_list.len() {
pr_warn!(
"Could not remove freeze listener for {}\n",
);
}
if inner.freeze_list.is_empty() {
- _unused_capacity = mem::take(&mut inner.freeze_list);
+ return mem::take(&mut inner.freeze_list);
}
+ KVVec::new()
}
pub(crate) fn freeze_list<'a>(&'a self, guard: &'a ProcessInner) -> &'a [Arc<Process>] {
pub(crate) fn get_transaction_node(&self, handle: u32) -> BinderResult<NodeRef> {
// When handle is zero, try to get the context manager.
if handle == 0 {
- Ok(self.ctx.get_manager_node(true)?)
+ let node_ref = self.ctx.get_manager_node(true)?;
+ if core::ptr::eq(self, &*node_ref.node.owner) {
+ return Err(EINVAL.into());
+ }
+ Ok(node_ref)
} else {
Ok(self.get_node_from_handle(handle, true)?)
}
// To preserve original binder behaviour, we only fail requests where the manager tries to
// increment references on itself.
+ let _to_free_freeze_listener;
+ let _to_free_freeze_listener_cleanup;
let mut refs = self.node_refs.lock();
if let Some(info) = refs.by_handle.get_mut(&handle) {
if info.node_ref().update(inc, strong) {
unsafe { info.node_ref2().node.remove_node_info(info) };
let id = info.node_ref().node.global_id();
+
+ if let Some(freeze) = *info.freeze() {
+ if let Some(fl) = refs.freeze_listeners.remove(&freeze) {
+ _to_free_freeze_listener_cleanup = fl.on_process_cleanup(&self);
+ _to_free_freeze_listener = fl;
+ }
+ }
+
refs.by_handle.remove(&handle);
refs.by_node.remove(&id);
refs.handle_is_present.release_id(handle as usize);
// Clean up freeze listeners.
let freeze_listeners = take(&mut self.node_refs.lock().freeze_listeners);
for listener in freeze_listeners.values() {
- listener.on_process_exit(&self);
+ listener.on_process_cleanup(&self);
}
drop(freeze_listeners);
"BC_DEAD_BINDER_DONE",
"BC_TRANSACTION_SG",
"BC_REPLY_SG",
+ "BC_REQUEST_FREEZE_NOTIFICATION",
+ "BC_CLEAR_FREEZE_NOTIFICATION",
+ "BC_FREEZE_NOTIFICATION_DONE",
};
const char * const binder_return_strings[] = {
"BR_FAILED_REPLY",
"BR_FROZEN_REPLY",
"BR_ONEWAY_SPAM_SUSPECT",
- "BR_TRANSACTION_PENDING_FROZEN"
+ "BR_TRANSACTION_PENDING_FROZEN",
+ "BR_FROZEN_BINDER",
+ "BR_CLEAR_FREEZE_NOTIFICATION_DONE",
};
#define CREATE_TRACE_POINTS
use kernel::sync::atomic::{ordering::Relaxed, Atomic};
use kernel::{ioctl::_IOC_NR, seq_file::SeqFile, seq_print};
-const BC_COUNT: usize = _IOC_NR(BC_REPLY_SG) as usize + 1;
-const BR_COUNT: usize = _IOC_NR(BR_TRANSACTION_PENDING_FROZEN) as usize + 1;
+const BC_COUNT: usize = _IOC_NR(BC_FREEZE_NOTIFICATION_DONE) as usize + 1;
+const BR_COUNT: usize = _IOC_NR(BR_CLEAR_FREEZE_NOTIFICATION_DONE) as usize + 1;
pub(crate) static GLOBAL_STATS: BinderStats = BinderStats::new();
Ok(())
}
+ pub(crate) fn clear_extended_error(&self, debug_id: usize) {
+ self.inner.lock().extended_error = ExtendedError::new(debug_id as u32, BR_OK, 0);
+ }
+
pub(crate) fn get_extended_error(&self, data: UserSlice) -> Result {
let mut writer = data.writer();
- let ee = self.inner.lock().extended_error;
+ let mut inner = self.inner.lock();
+ let ee = inner.extended_error;
+ inner.extended_error = ExtendedError::new(0, BR_OK, 0);
+ drop(inner);
writer.write(&ee)?;
Ok(())
}
inner.pop_transaction_to_reply(thread.as_ref())
} {
let reply = Err(BR_DEAD_REPLY);
- if !transaction.from.deliver_single_reply(reply, &transaction) {
+ if !transaction
+ .from
+ .deliver_single_reply(reply, &transaction, None)
+ {
break;
}
&self,
reply: Result<DLArc<Transaction>, u32>,
transaction: &DArc<Transaction>,
+ extended_error: Option<ExtendedError>,
) {
- if self.deliver_single_reply(reply, transaction) {
+ if self.deliver_single_reply(reply, transaction, extended_error) {
transaction.from.unwind_transaction_stack();
}
}
&self,
reply: Result<DLArc<Transaction>, u32>,
transaction: &DArc<Transaction>,
+ extended_error: Option<ExtendedError>,
) -> bool {
if let Ok(transaction) = &reply {
crate::trace::trace_transaction(true, transaction, Some(&self.task));
return true;
}
+ if let Some(ee) = extended_error {
+ if inner.extended_error.command == BR_OK {
+ inner.extended_error = ee;
+ }
+ }
+
match reply {
Ok(work) => {
inner.push_work(work);
info.buffers_size = td.buffers_size as usize;
// SAFETY: Above `read` call initializes all bytes, so this union read is ok.
info.target_handle = unsafe { td.transaction_data.target.handle };
+
+ info.debug_id = super::next_debug_id();
+
Ok(())
}
let mut info = TransactionInfo::zeroed();
self.read_transaction_info(cmd, reader, &mut info)?;
+ self.clear_extended_error(info.debug_id);
+
let ret = if info.is_reply {
self.reply_inner(&mut info)
} else if info.is_oneway() {
};
if let Err(err) = ret {
- if err.reply != BR_TRANSACTION_COMPLETE {
- info.reply = err.reply;
- }
-
self.push_return_work(err.reply);
- if let Some(source) = &err.source {
- info.errno = source.to_errno();
+ if err.reply != BR_TRANSACTION_COMPLETE {
info.reply = err.reply;
+ if let Some(source) = &err.source {
+ info.errno = source.to_errno();
- {
- let mut ee = self.inner.lock().extended_error;
- ee.command = err.reply;
- ee.param = source.to_errno();
+ {
+ let mut inner = self.inner.lock();
+ inner.extended_error =
+ ExtendedError::new(info.debug_id as u32, err.reply, source.to_errno());
+ }
}
pr_warn!(
- "{}:{} transaction to {} failed: {source:?}",
+ "{}:{} transaction to {} failed: {err:?}",
info.from_pid,
info.from_tid,
info.to_pid
let allow_fds = orig.flags & TF_ACCEPT_FDS != 0;
let reply = Transaction::new_reply(self, process, info, allow_fds)?;
self.inner.lock().push_work(completion);
- orig.from.deliver_reply(Ok(reply), &orig);
+ orig.from.deliver_reply(Ok(reply), &orig, None);
Ok(())
})()
.map_err(|mut err| {
// At this point we only return `BR_TRANSACTION_COMPLETE` to the caller, and we must let
// the sender know that the transaction has completed (with an error in this case).
+
pr_warn!(
- "Failure {:?} during reply - delivering BR_FAILED_REPLY to sender.",
- err
+ "{}:{} reply to {} failed: {err:?}",
+ info.from_pid,
+ info.from_tid,
+ info.to_pid
);
- let reply = Err(BR_FAILED_REPLY);
- orig.from.deliver_reply(reply, &orig);
+
+ let param = err.source.as_ref().map_or(0, |e| e.to_errno());
+ let ee = ExtendedError::new(info.debug_id as u32, err.reply, param);
+ orig.from
+ .deliver_reply(Err(BR_FAILED_REPLY), &orig, Some(ee));
err.reply = BR_TRANSACTION_COMPLETE;
err
});
pub(crate) reply: u32,
pub(crate) oneway_spam_suspect: bool,
pub(crate) is_reply: bool,
+ pub(crate) debug_id: usize,
}
impl TransactionInfo {
from: &Arc<Thread>,
info: &mut TransactionInfo,
) -> BinderResult<DLArc<Self>> {
- let debug_id = super::next_debug_id();
let allow_fds = node_ref.node.flags & FLAT_BINDER_FLAG_ACCEPTS_FDS != 0;
let txn_security_ctx = node_ref.node.flags & FLAT_BINDER_FLAG_TXN_SECURITY_CTX != 0;
let mut txn_security_ctx_off = if txn_security_ctx { Some(0) } else { None };
let mut alloc = match from.copy_transaction_data(
to.clone(),
info,
- debug_id,
+ info.debug_id,
allow_fds,
txn_security_ctx_off.as_mut(),
) {
let data_address = alloc.ptr;
Ok(DTRWrap::arc_pin_init(pin_init!(Transaction {
- debug_id,
+ debug_id: info.debug_id,
target_node: Some(target_node),
from_parent,
sender_euid: Kuid::current_euid(),
info: &mut TransactionInfo,
allow_fds: bool,
) -> BinderResult<DLArc<Self>> {
- let debug_id = super::next_debug_id();
let mut alloc =
- match from.copy_transaction_data(to.clone(), info, debug_id, allow_fds, None) {
+ match from.copy_transaction_data(to.clone(), info, info.debug_id, allow_fds, None) {
Ok(alloc) => alloc,
Err(err) => {
pr_warn!("Failure in copy_transaction_data: {:?}", err);
alloc.set_info_clear_on_drop();
}
Ok(DTRWrap::arc_pin_init(pin_init!(Transaction {
- debug_id,
+ debug_id: info.debug_id,
target_node: None,
from_parent: None,
sender_euid: Kuid::current_euid(),
let send_failed_reply = ScopeGuard::new(|| {
if self.target_node.is_some() && self.flags & TF_ONE_WAY == 0 {
let reply = Err(BR_FAILED_REPLY);
- self.from.deliver_reply(reply, &self);
+ self.from.deliver_reply(reply, &self, None);
}
self.drop_outstanding_txn();
});
// If this is not a reply or oneway transaction, then send a dead reply.
if self.target_node.is_some() && self.flags & TF_ONE_WAY == 0 {
let reply = Err(BR_DEAD_REPLY);
- self.from.deliver_reply(reply, &self);
+ self.from.deliver_reply(reply, &self, None);
}
self.drop_outstanding_txn();
/* do we need to do it? */
if ((dev->class != ATA_DEV_ATA && dev->class != ATA_DEV_ZAC) ||
!ata_id_has_lba(dev->id) || !ata_id_hpa_enabled(dev->id) ||
- (dev->quirks & ATA_QUIRK_BROKEN_HPA))
+ (dev->quirks & ATA_QUIRK_BROKEN_HPA) || ata_id_is_locked(dev->id))
return 0;
/* read native max address */
/* verify n_sectors hasn't changed */
if (dev->class != ATA_DEV_ATA || !n_sectors ||
- dev->n_sectors == n_sectors)
+ dev->n_sectors == n_sectors || ata_id_is_locked(dev->id))
return 0;
/* n_sectors has changed */
#include <linux/device.h>
#include <linux/dmaengine.h>
#include <linux/of.h>
-#include <linux/of_irq.h>
#include <linux/platform_device.h>
#include <linux/phy/phy.h>
#include <linux/libata.h>
struct sata_dwc_device *hsdev)
{
struct device *dev = &pdev->dev;
- struct device_node *np = dev->of_node;
hsdev->dma = devm_kzalloc(dev, sizeof(*hsdev->dma), GFP_KERNEL);
if (!hsdev->dma)
hsdev->dma->id = pdev->id;
/* Get SATA DMA interrupt number */
- hsdev->dma->irq = irq_of_parse_and_map(np, 1);
- if (!hsdev->dma->irq) {
- dev_err(dev, "no SATA DMA irq\n");
- return -ENODEV;
- }
+ hsdev->dma->irq = platform_get_irq(pdev, 1);
+ if (hsdev->dma->irq < 0)
+ return hsdev->dma->irq;
/* Get physical SATA DMA register base address */
hsdev->dma->regs = devm_platform_ioremap_resource(pdev, 1);
static void clear_interrupt_bit(struct sata_dwc_device *hsdev, u32 bit)
{
- sata_dwc_writel(&hsdev->sata_dwc_regs->intpr,
- sata_dwc_readl(&hsdev->sata_dwc_regs->intpr));
+ sata_dwc_writel(&hsdev->sata_dwc_regs->intpr, bit);
}
static u32 qcmd_tag_to_mask(u8 tag)
status = ap->ops->sff_check_status(ap);
dev_dbg(ap->dev, "%s ATA status register=0x%x\n", __func__, status);
- tag = 0;
while (tag_mask) {
- while (!(tag_mask & 0x00000001)) {
- tag++;
- tag_mask <<= 1;
- }
-
- tag_mask &= (~0x00000001);
+ tag = __ffs(tag_mask);
+ tag_mask &= ~(1U << tag);
qc = ata_qc_from_tag(ap, tag);
if (unlikely(!qc)) {
dev_err(ap->dev, "failed to get qc");
static int sata_dwc_probe(struct platform_device *ofdev)
{
struct device *dev = &ofdev->dev;
- struct device_node *np = dev->of_node;
struct sata_dwc_device *hsdev;
u32 idr, versionr;
char *ver = (char *)&versionr;
/* Save dev for later use in dev_xxx() routines */
hsdev->dev = dev;
- /* Enable SATA Interrupts */
- sata_dwc_enable_interrupts(hsdev);
-
/* Get SATA interrupt number */
- irq = irq_of_parse_and_map(np, 0);
- if (!irq) {
- dev_err(dev, "no SATA DMA irq\n");
- return -ENODEV;
- }
+ irq = platform_get_irq(ofdev, 0);
+ if (irq < 0)
+ return irq;
#ifdef CONFIG_SATA_DWC_OLD_DMA
- if (!of_property_present(np, "dmas")) {
+ if (!of_property_present(dev->of_node, "dmas")) {
err = sata_dwc_dma_init_old(ofdev, hsdev);
if (err)
return err;
if (err)
dev_err(dev, "failed to activate host");
+ /* Enable SATA Interrupts */
+ sata_dwc_enable_interrupts(hsdev);
return 0;
error_out:
data_size -= digest_size;
}
+ if (data_size < 0) {
+ drbd_err(peer_device, "Invalid data reply size\n");
+ return -EIO;
+ }
+
/* optimistically update recv_cnt. if receiving fails below,
* we disconnect anyways, and counters will be reset. */
peer_device->device->recv_cnt += data_size>>9;
struct queue_limits lim;
struct file *filp;
gfp_t gfp = lo->old_gfp_mask;
+ int err;
spin_lock_irq(&lo->lo_lock);
filp = lo->lo_backing_file;
disk_force_media_change(lo->lo_disk);
- if (lo->lo_flags & LO_FLAGS_PARTSCAN) {
- int err;
-
- /*
- * open_mutex has been held already in release path, so don't
- * acquire it if this function is called in such case.
- *
- * If the reread partition isn't from release path, lo_refcnt
- * must be at least one and it can only become zero when the
- * current holder is released.
- */
- err = bdev_disk_changed(lo->lo_disk, false);
- if (err)
- pr_warn("%s: partition scan of loop%d failed (rc=%d)\n",
- __func__, lo->lo_number, err);
- /* Device is gone, no point in returning error */
- }
+ /*
+ * Remove all partitions, including partitions added manually with
+ * BLKPG, which may exist even if LO_FLAGS_PARTSCAN is not set.
+ *
+ * open_mutex has been held already in release path, so don't acquire
+ * it here.
+ */
+ err = bdev_disk_changed(lo->lo_disk, false);
+ if (err)
+ pr_warn("%s: partition scan of loop%d failed (rc=%d)\n",
+ __func__, lo->lo_number, err);
+ /* Device is gone, no point in returning error */
/*
- * lo->lo_state is set to Lo_unbound here after above partscan has
+ * lo->lo_state is set to Lo_unbound here after removing partitions has
* finished. There cannot be anybody else entering __loop_clr_fd() as
* Lo_rundown state protects us from all the other places trying to
* change the 'lo' device.
#define UBLK_CMD_BATCH_TMP_BUF_SZ (48 * 10)
struct ublk_batch_io_iter {
void __user *uaddr;
+ const u8 *kaddr;
unsigned done, total;
unsigned char elem_bytes;
/* copy to this buffer from user space */
while (iter->done < iter->total) {
unsigned int len = min(sizeof(iter->buf), iter->total - iter->done);
- if (copy_from_user(iter->buf, iter->uaddr + iter->done, len)) {
+ if (iter->kaddr) {
+ memcpy(iter->buf, iter->kaddr + iter->done, len);
+ } else if (copy_from_user(iter->buf, iter->uaddr + iter->done,
+ len)) {
pr_warn("ublk%d: read batch cmd buffer failed\n",
data->ub->dev_info.dev_id);
return -EFAULT;
.total = uc->nr_elem * uc->elem_bytes,
.elem_bytes = uc->elem_bytes,
};
+ void *cmd_buf;
int ret;
+ cmd_buf = vmemdup_user(iter.uaddr, iter.total);
+ if (IS_ERR(cmd_buf))
+ return PTR_ERR(cmd_buf);
+ iter.kaddr = cmd_buf;
+
mutex_lock(&data->ub->mutex);
ret = ublk_walk_cmd_buf(&iter, data, ublk_batch_prep_io);
if (ret && iter.done)
ublk_batch_revert_prep_cmd(&iter, data);
mutex_unlock(&data->ub->mutex);
+ kvfree(cmd_buf);
return ret;
}
if (!shadow[j].request)
continue;
+ /*
+ * For requests split across multiple slots, process the
+ * underlying request only once: skip the linked, sg-less
+ * secondary slot.
+ */
+ if (shadow[j].associated_id != NO_ASSOCIATED_ID &&
+ shadow[j].num_sg == 0)
+ continue;
+
/*
* Get the bios in the request so we can re-queue them.
*/
if (IS_ERR(skb))
return PTR_ERR(skb);
- bt_dev_info(hdev, "%s", (char *)(skb->data + 1));
+ /* Bounded print: the device controls skb->len. */
+ if (skb->len > 1) {
+ int len = skb->len - 1;
- hci_set_fw_info(hdev, "%s", skb->data + 1);
+ bt_dev_info(hdev, "%.*s", len, (char *)(skb->data + 1));
+ hci_set_fw_info(hdev, "%.*s", len, skb->data + 1);
+ }
kfree_skb(skb);
return 0;
if (test_bit(BTINTEL_PCIE_CORE_HALTED, &data->flags))
return -ENODEV;
+ if (test_bit(BTINTEL_PCIE_RECOVERY_IN_PROGRESS, &data->flags))
+ return -ENODEV;
+
/* Due to the fw limitation, the type header of the packet should be
* 4 bytes unlike 1 byte for UART. In UART, the firmware can read
* the first byte to get the packet type and redirect the rest of data
}
}
-static int btintel_pcie_setup_hdev(struct btintel_pcie_data *data);
static void btintel_pcie_reset(struct hci_dev *hdev);
static int btintel_pcie_acpi_reset_method(struct btintel_pcie_data *data)
}
}
+/*
+ * Issue a Function Level Reset and hand teardown/re-init off to the PCI
+ * core via device_reprobe(), mirroring the PLDR path's contract.
+ *
+ * Caller must hold pci_lock_rescan_remove() and must have already
+ * disabled interrupts and drained both rx_work and coredump_work.
+ */
+static int btintel_pcie_perform_flr(struct btintel_pcie_data *data)
+{
+ struct pci_dev *pdev = data->pdev;
+ int err;
+
+ /* pci_try_reset_function() avoids the device_lock ABBA against
+ * btintel_pcie_remove(): .remove() runs with device_lock held and
+ * then waits for this work via disable_work_sync(); the blocking
+ * pci_reset_function() would deadlock by trying to re-acquire
+ * device_lock here.
+ */
+ err = pci_try_reset_function(pdev);
+ if (err) {
+ BT_ERR("Failed resetting the pcie device (%d)", err);
+ return err;
+ }
+
+ /* device_reprobe() always detaches the driver first (running
+ * .remove(), which frees 'data'); any re-probe failure leaves the
+ * device unbound but 'data' is already gone, so just log it.
+ */
+ if (device_reprobe(&pdev->dev))
+ BT_ERR("BT reprobe failed for BDF:%s", pci_name(pdev));
+
+ return 0;
+}
+
static void btintel_pcie_reset_work(struct work_struct *wk)
{
struct btintel_pcie_data *data =
container_of(wk, struct btintel_pcie_data, reset_work);
struct pci_dev *pdev = data->pdev;
- int err;
pci_lock_rescan_remove();
disable_work_sync(&data->coredump_work);
bt_dev_dbg(data->hdev, "Release bluetooth interface");
+
+ /* Both reset paths follow the same contract: on success they
+ * destroy 'data' via device_reprobe() (a fresh probe re-INIT_WORKs
+ * the coredump_work with disable count 0), so enable_work() must
+ * NOT be called on the success path. Only the FLR path can fail
+ * with 'data' still alive, in which case we balance the
+ * disable_work_sync() above so a later successful reset is not
+ * permanently blocked.
+ *
+ * pci_lock_rescan_remove() (held above) serializes against PCI
+ * device addition/removal (hotplug), so no device can be added to
+ * or removed from the bus list while this code runs.
+ */
if (data->reset_type == BTINTEL_PCIE_IOSF_PRR_PLDR) {
- /* This function holds pci_lock_rescan_remove(), which acquires
- * pci_rescan_remove_lock. This mutex serializes against PCI device
- * addition/removal (hotplug), so no device can be added to or
- * removed from the bus list while this code runs.
- *
- * device_reprobe() inside btintel_pcie_perform_pldr() destroys
- * 'data' via .remove(); a fresh probe re-INIT_WORKs the
- * coredump_work with disable count 0, so we must not call
- * enable_work() on this path.
- */
btintel_pcie_perform_pldr(data);
goto out;
}
- btintel_pcie_release_hdev(data);
-
- /* Use pci_try_reset_function() rather than pci_reset_function() to
- * avoid an ABBA deadlock against btintel_pcie_remove(): the PCI core
- * calls .remove() with device_lock held, and remove() then waits for
- * this work via cancel_work_sync(); pci_reset_function() would in
- * turn try to acquire the same device_lock, deadlocking both paths.
- */
- err = pci_try_reset_function(pdev);
- if (err) {
- BT_ERR("Failed resetting the pcie device (%d)", err);
- goto out_enable;
- }
- btintel_pcie_enable_interrupts(data);
- btintel_pcie_config_msix(data);
-
- err = btintel_pcie_enable_bt(data);
- if (err) {
- BT_ERR("Failed to enable bluetooth hardware after reset (%d)",
- err);
- goto out_enable;
- }
-
- btintel_pcie_reset_ia(data);
- btintel_pcie_start_rx(data);
- data->flags = 0;
+ if (btintel_pcie_perform_flr(data))
+ enable_work(&data->coredump_work);
- err = btintel_pcie_setup_hdev(data);
- if (err) {
- BT_ERR("Failed registering hdev (%d)", err);
- goto out_enable;
- }
-
-out_enable:
- /* Balance disable_work_sync() above on every exit. Leaving the
- * counter incremented on a failed reset would permanently disable
- * coredump_work even after a later successful reset.
- */
- enable_work(&data->coredump_work);
out:
pci_dev_put(pdev);
pci_unlock_rescan_remove();
}
nxpdev->fw_dnld_v3_offset = offset - nxpdev->fw_v3_offset_correction;
+ if (nxpdev->fw_dnld_v3_offset >= nxpdev->fw->size ||
+ len > nxpdev->fw->size - nxpdev->fw_dnld_v3_offset) {
+ bt_dev_err(hdev, "FW download out of bounds, ignoring request");
+ len = 0;
+ goto free_skb;
+ }
serdev_device_write_buf(nxpdev->serdev, nxpdev->fw->data +
nxpdev->fw_dnld_v3_offset, len);
idx = 0;
data = tlv->data;
- while (idx < length - sizeof(struct tlv_type_nvm)) {
+ while (idx + sizeof(struct tlv_type_nvm) <= length) {
tlv_nvm = (struct tlv_type_nvm *)(data + idx);
tag_id = le16_to_cpu(tlv_nvm->tag_id);
}
BT_DBG("length=%x offset=%x index %d", patch_length, patch_offset, i);
- min_size = patch_offset + patch_length;
- if (btrtl_dev->fw_len < min_size)
+ if (patch_length < sizeof(epatch_info->fw_version) ||
+ patch_offset > btrtl_dev->fw_len ||
+ patch_length > btrtl_dev->fw_len - patch_offset)
return -EINVAL;
/* Copy the firmware into a new buffer and write the version at
BT_DBG("hdev %p tty %p", hdev, tty);
+ disable_work_sync(&hu->write_work);
+
if (hu->tx_skb) {
kfree_skb(hu->tx_skb); hu->tx_skb = NULL;
}
percpu_up_read(&hu->proto_lock);
+ /* Resume TX. Also reschedule in case work was queued concurrently;
+ * this may schedule write_work although there's nothing to do.
+ */
+ enable_work(&hu->write_work);
+ clear_bit(HCI_UART_SENDING, &hu->tx_state);
+ if (test_bit(HCI_UART_TX_WAKEUP, &hu->tx_state))
+ hci_uart_tx_wakeup(hu);
+
return 0;
}
/* Close device */
static int hci_uart_close(struct hci_dev *hdev)
{
- struct hci_uart *hu = hci_get_drvdata(hdev);
-
BT_DBG("hdev %p", hdev);
- cancel_work_sync(&hu->write_work);
-
hci_uart_flush(hdev);
hdev->flush = NULL;
return 0;
if (!(qca_memdump->ram_dump_size)) {
bt_dev_err(hu->hdev, "Rx invalid memdump size");
kfree(qca_memdump);
+ qca->qca_memdump = NULL;
+ qca->memdump_state = QCA_MEMDUMP_COLLECTED;
+ clear_and_wake_up_bit(QCA_MEMDUMP_COLLECTION, &qca->flags);
+ clear_bit(QCA_IBS_DISABLED, &qca->flags);
kfree_skb(skb);
mutex_unlock(&qca->hci_memdump_lock);
return;
tpm_common_open(file, chip, priv, NULL);
- return 0;
+ return nonseekable_open(inode, file);
out:
clear_bit(0, &chip->is_open);
tpm_common_open(file, chip, &priv->priv, &priv->space);
- return 0;
+ return nonseekable_open(inode, file);
}
static int tpmrm_release(struct inode *inode, struct file *file)
cpu = cpumask_first(policy->related_cpus);
max_freq = arch_scale_freq_ref(cpu);
+ if (!max_freq)
+ max_freq = policy->cpuinfo.max_freq;
+
capped_freq = policy->max;
/*
static void hybrid_get_capacity_perf(struct cpudata *cpu)
{
+ u64 hwp_cap = READ_ONCE(cpu->hwp_cap_cached);
+
if (READ_ONCE(global.no_turbo)) {
- cpu->capacity_perf = cpu->pstate.max_pstate_physical;
+ cpu->capacity_perf = HWP_GUARANTEED_PERF(hwp_cap);
return;
}
- cpu->capacity_perf = HWP_HIGHEST_PERF(READ_ONCE(cpu->hwp_cap_cached));
+ cpu->capacity_perf = HWP_HIGHEST_PERF(hwp_cap);
}
static void hybrid_set_capacity_of_cpus(void)
read_unlock_bh(&ldev->dmb_ht_lock);
return -EINVAL;
}
+ if ((u64)offset + size > rmb_node->len) {
+ read_unlock_bh(&ldev->dmb_ht_lock);
+ return -EINVAL;
+ }
+
memcpy((char *)rmb_node->cpu_addr + offset, data, size);
sba_idx = rmb_node->sba_idx;
read_unlock_bh(&ldev->dmb_ht_lock);
{
struct udmabuf *ubuf = buf->priv;
struct device *dev = ubuf->device->this_device;
- int ret = 0;
if (!ubuf->sg) {
ubuf->sg = get_sg_table(dev, buf, direction);
if (IS_ERR(ubuf->sg)) {
+ int ret;
+
ret = PTR_ERR(ubuf->sg);
ubuf->sg = NULL;
+ return ret;
} else {
ubuf->sg_dir = direction;
}
- } else {
- dma_sync_sgtable_for_cpu(dev, ubuf->sg, direction);
}
- return ret;
+ dma_sync_sgtable_for_cpu(dev, ubuf->sg, direction);
+ return 0;
}
static int end_cpu_udmabuf(struct dma_buf *buf,
if (!dpll_pin_available(ref_sync_pin))
continue;
ref_sync_pin_priv = dpll_pin_on_dpll_priv(dpll, ref_sync_pin);
+ /* Pin may have been unregistered from this dpll already */
+ if (!ref_sync_pin_priv)
+ continue;
if (WARN_ON(!ops->ref_sync_get))
return -EOPNOTSUPP;
ret = ops->ref_sync_get(pin, pin_priv, ref_sync_pin,
#include <linux/interrupt.h>
#include <linux/io.h>
#include <linux/kernel.h>
+#include <linux/minmax.h>
#include <linux/module.h>
#include <linux/mm.h>
#include <linux/mutex.h>
(FIELD_PREP(SENDER_ID_MASK, (s)) | FIELD_PREP(RECEIVER_ID_MASK, (r)))
#define RXTX_MAP_MIN_BUFSZ_MASK GENMASK(1, 0)
-#define RXTX_MAP_MIN_BUFSZ(x) ((x) & RXTX_MAP_MIN_BUFSZ_MASK)
+#define RXTX_MAP_MAX_BUFSZ_MASK GENMASK(31, 16)
+#define RXTX_MAP_MIN_BUFSZ(x) (FIELD_GET(RXTX_MAP_MIN_BUFSZ_MASK, (x)))
+#define RXTX_MAP_MAX_BUFSZ(x) (FIELD_GET(RXTX_MAP_MAX_BUFSZ_MASK, (x)))
#define FFA_MAX_NOTIFICATIONS 64
struct ffa_composite_mem_region *composite;
struct ffa_mem_region_addr_range *constituents;
struct ffa_mem_region_attributes *ep_mem_access;
- u32 idx, frag_len, length, buf_sz = 0, num_entries = sg_nents(args->sg);
+ u32 idx, frag_len, length, buf_sz = 0, num_entries = sg_nents(args->sg), ep_offset;
+ u32 emad_end, emad_size = ffa_emad_size_get(drv_info->version);
mem_region->tag = args->tag;
mem_region->flags = args->flags;
mem_region->sender_id = drv_info->vm_id;
mem_region->attributes = ffa_memory_attributes_get(func_id);
+
+ ffa_mem_region_additional_setup(drv_info->version, mem_region);
composite_offset = ffa_mem_desc_offset(buffer, args->nattrs,
drv_info->version);
+ if (composite_offset + sizeof(*composite) > max_fragsize)
+ return -ENXIO;
for (idx = 0; idx < args->nattrs; idx++) {
- ep_mem_access = buffer +
- ffa_mem_desc_offset(buffer, idx, drv_info->version);
+ ep_offset = ffa_mem_desc_offset(buffer, idx, drv_info->version);
+ if (check_add_overflow(ep_offset, emad_size, &emad_end))
+ return -ENXIO;
+
+ if (emad_end > max_fragsize)
+ return -ENXIO;
+
+ ep_mem_access = buffer + ep_offset;
+ memset(ep_mem_access, 0, emad_size);
ep_mem_access->receiver = args->attrs[idx].receiver;
ep_mem_access->attrs = args->attrs[idx].attrs;
ep_mem_access->composite_off = composite_offset;
- ep_mem_access->flag = 0;
- ep_mem_access->reserved = 0;
ffa_emad_impdef_value_init(drv_info->version,
ep_mem_access->impdef_val,
args->attrs[idx].impdef_val);
}
mem_region->handle = 0;
mem_region->ep_count = args->nattrs;
- ffa_mem_region_additional_setup(drv_info->version, mem_region);
composite = buffer + composite_offset;
composite->total_pg_cnt = ffa_get_num_pages_sg(args->sg);
constituents = buffer;
}
- if ((void *)constituents - buffer > max_fragsize) {
+ if ((void *)constituents + sizeof(*constituents) - buffer > max_fragsize) {
pr_err("Memory Region Fragment > Tx Buffer size\n");
return -EFAULT;
}
constituents->pg_cnt = args->sg->length / FFA_PAGE_SIZE;
constituents->reserved = 0;
constituents++;
- frag_len += sizeof(struct ffa_mem_region_addr_range);
+ frag_len += sizeof(*constituents);
} while ((args->sg = sg_next(args->sg)));
return ffa_transmit_fragment(func_id, addr, buf_sz, frag_len,
uuid_t uuid;
struct ffa_partition_info *pbuf;
- if (uuid_parse(uuid_str, &uuid)) {
+ if (!uuid_str || uuid_parse(uuid_str, &uuid)) {
pr_err("invalid uuid (%s)\n", uuid_str);
return -ENODEV;
}
{
int ret;
u32 buf_sz;
- size_t rxtx_bufsz = SZ_4K;
+ size_t rxtx_min_bufsz = SZ_4K, rxtx_max_bufsz = 0, rxtx_bufsz;
if (IS_BUILTIN(CONFIG_ARM_FFA_TRANSPORT) &&
is_protected_kvm_enabled() && !is_pkvm_initialized())
ret = ffa_features(FFA_FN_NATIVE(RXTX_MAP), 0, &buf_sz, NULL);
if (!ret) {
if (RXTX_MAP_MIN_BUFSZ(buf_sz) == 1)
- rxtx_bufsz = SZ_64K;
+ rxtx_min_bufsz = SZ_64K;
else if (RXTX_MAP_MIN_BUFSZ(buf_sz) == 2)
- rxtx_bufsz = SZ_16K;
+ rxtx_min_bufsz = SZ_16K;
else
- rxtx_bufsz = SZ_4K;
+ rxtx_min_bufsz = SZ_4K;
+
+ rxtx_max_bufsz = RXTX_MAP_MAX_BUFSZ(buf_sz) * SZ_4K;
+ if (rxtx_max_bufsz != 0 && rxtx_max_bufsz < rxtx_min_bufsz)
+ rxtx_max_bufsz = rxtx_min_bufsz;
}
- rxtx_bufsz = PAGE_ALIGN(rxtx_bufsz);
- drv_info->rxtx_bufsz = rxtx_bufsz;
+ rxtx_bufsz = min_not_zero(PAGE_ALIGN(rxtx_min_bufsz), rxtx_max_bufsz);
drv_info->rx_buffer = alloc_pages_exact(rxtx_bufsz, GFP_KERNEL);
if (!drv_info->rx_buffer) {
ret = -ENOMEM;
ret = ffa_rxtx_map(virt_to_phys(drv_info->tx_buffer),
virt_to_phys(drv_info->rx_buffer),
rxtx_bufsz / FFA_PAGE_SIZE);
+ if (ret == -EINVAL && !rxtx_max_bufsz && rxtx_min_bufsz < rxtx_bufsz) {
+ rxtx_bufsz = rxtx_min_bufsz;
+ ret = ffa_rxtx_map(virt_to_phys(drv_info->tx_buffer),
+ virt_to_phys(drv_info->rx_buffer),
+ rxtx_bufsz / FFA_PAGE_SIZE);
+ }
if (ret) {
pr_err("failed to register FFA RxTx buffers\n");
goto free_pages;
}
+ drv_info->rxtx_bufsz = rxtx_bufsz;
mutex_init(&drv_info->rx_lock);
mutex_init(&drv_info->tx_lock);
firmware.
This driver can also be built as a module. If so, the module will be
- called scmi_power_control. Note this may needed early in boot to catch
- early shutdown/reboot SCMI requests.
+ called scmi_power_control. Note this may be needed early in boot to
+ catch early shutdown/reboot SCMI requests.
endmenu
static int scmi_clock_determine_rate(const struct scmi_protocol_handle *ph,
u32 clk_id, unsigned long *rate)
{
- u64 fmin, fmax, ftmp;
+ u64 fmin, fmax, ftmp, step;
struct scmi_clock_info *clk;
struct scmi_clock_desc *clkd;
struct clock_info *ci = ph->get_priv(ph);
return 0;
}
+ step = clkd->r.rates[RATE_STEP];
+ if (!step)
+ return -EINVAL;
+
ftmp = *rate - fmin;
- ftmp += clkd->r.rates[RATE_STEP] - 1; /* to round up */
- ftmp = div64_ul(ftmp, clkd->r.rates[RATE_STEP]);
+ ftmp = DIV64_U64_ROUND_UP(ftmp, step);
- *rate = ftmp * clkd->r.rates[RATE_STEP] + fmin;
+ *rate = ftmp * step + fmin;
return 0;
}
return -EINVAL;
}
if (kfifo_avail(&r_evt->proto->equeue.kfifo) < sizeof(eh) + len) {
- dev_warn(handle->dev,
- "queue full, dropping proto_id:%d evt_id:%d ts:%lld\n",
- proto_id, evt_id, ktime_to_ns(ts));
+ dev_warn_ratelimited(handle->dev,
+ "queue full, dropping proto_id:%d evt_id:%d ts:%lld\n",
+ proto_id, evt_id, ktime_to_ns(ts));
return -ENOMEM;
}
unsigned int flags;
struct reset_control *rst;
struct clk_bulk_data clks[DWAPB_NR_CLOCKS];
+ bool clocks_on_for_wake;
struct dwapb_gpio_port ports[] __counted_by(nr_ports);
};
dwapb_write(gpio, GPIO_INT_POLARITY, pol);
}
+static int dwapb_irq_init_hw(struct gpio_chip *gc)
+{
+ struct dwapb_gpio *gpio = to_dwapb_gpio(gc);
+
+ /*
+ * GPIO interrupts may retain stale state across warm reboots when
+ * peripherals stay powered. Force a known-safe state before the GPIO
+ * irqchip and irq domain are set up.
+ */
+ dwapb_write(gpio, GPIO_INTEN, 0);
+ dwapb_write(gpio, GPIO_INTMASK, 0xffffffff);
+ dwapb_write(gpio, GPIO_PORTA_EOI, 0xffffffff);
+
+ return 0;
+}
+
static u32 dwapb_do_irq(struct dwapb_gpio *gpio)
{
struct gpio_generic_chip *gen_gc = &gpio->ports[0].chip;
struct dwapb_gpio *gpio = to_dwapb_gpio(gc);
struct dwapb_context *ctx = gpio->ports[0].ctx;
irq_hw_number_t bit = irqd_to_hwirq(d);
+ u32 wake_en = ctx->wake_en;
if (enable)
- ctx->wake_en |= BIT(bit);
+ wake_en |= BIT(bit);
else
- ctx->wake_en &= ~BIT(bit);
+ wake_en &= ~BIT(bit);
+
+#ifdef CONFIG_IRQ_DOMAIN_HIERARCHY
+ if (d->parent_data && !!ctx->wake_en != !!wake_en) {
+ int err;
+
+ err = irq_chip_set_wake_parent(d, enable);
+ if (err)
+ return err;
+ }
+#endif
+
+ ctx->wake_en = wake_en;
return 0;
}
girq = &gc->irq;
girq->handler = handle_bad_irq;
girq->default_type = IRQ_TYPE_NONE;
+ girq->init_hw = dwapb_irq_init_hw;
port->pirq = pirq;
int i;
scoped_guard(gpio_generic_lock_irqsave, gen_gc) {
+ gpio->clocks_on_for_wake = false;
+
for (i = 0; i < gpio->nr_ports; i++) {
unsigned int offset;
unsigned int idx = gpio->ports[i].idx;
ctx->int_pol = dwapb_read(gpio, GPIO_INT_POLARITY);
ctx->int_type = dwapb_read(gpio, GPIO_INTTYPE_LEVEL);
ctx->int_deb = dwapb_read(gpio, GPIO_PORTA_DEBOUNCE);
+ }
+ }
+ }
+
+ return 0;
+}
+
+static int dwapb_gpio_suspend_noirq(struct device *dev)
+{
+ struct dwapb_gpio *gpio = dev_get_drvdata(dev);
+ struct gpio_generic_chip *gen_gc = &gpio->ports[0].chip;
+ bool wake_enabled = false;
+ int i;
- /* Mask out interrupts */
+ scoped_guard(gpio_generic_lock_irqsave, gen_gc) {
+ for (i = 0; i < gpio->nr_ports; i++) {
+ unsigned int idx = gpio->ports[i].idx;
+ struct dwapb_context *ctx = gpio->ports[i].ctx;
+
+ if (idx == 0) {
+ wake_enabled = ctx->wake_en;
dwapb_write(gpio, GPIO_INTMASK, ~ctx->wake_en);
+ break;
}
}
+
+ gpio->clocks_on_for_wake = wake_enabled;
+ }
+
+ if (wake_enabled) {
+ device_set_wakeup_path(dev);
+ return 0;
}
clk_bulk_disable_unprepare(DWAPB_NR_CLOCKS, gpio->clks);
return 0;
}
-static int dwapb_gpio_resume(struct device *dev)
+static int dwapb_gpio_resume_noirq(struct device *dev)
{
struct dwapb_gpio *gpio = dev_get_drvdata(dev);
- struct gpio_chip *gc = &gpio->ports[0].chip.gc;
- struct gpio_generic_chip *gen_gc = to_gpio_generic_chip(gc);
- int i, err;
+ int err;
+
+ if (gpio->clocks_on_for_wake)
+ return 0;
err = clk_bulk_prepare_enable(DWAPB_NR_CLOCKS, gpio->clks);
- if (err) {
+ if (err)
dev_err(gpio->dev, "Cannot reenable APB/Debounce clocks\n");
- return err;
- }
+
+ return err;
+}
+
+static int dwapb_gpio_resume(struct device *dev)
+{
+ struct dwapb_gpio *gpio = dev_get_drvdata(dev);
+ struct gpio_chip *gc = &gpio->ports[0].chip.gc;
+ struct gpio_generic_chip *gen_gc = to_gpio_generic_chip(gc);
+ int i;
guard(gpio_generic_lock_irqsave)(gen_gc);
return 0;
}
-static DEFINE_SIMPLE_DEV_PM_OPS(dwapb_gpio_pm_ops,
- dwapb_gpio_suspend, dwapb_gpio_resume);
+static const struct dev_pm_ops dwapb_gpio_pm_ops = {
+ SYSTEM_SLEEP_PM_OPS(dwapb_gpio_suspend, dwapb_gpio_resume)
+ NOIRQ_SYSTEM_SLEEP_PM_OPS(dwapb_gpio_suspend_noirq,
+ dwapb_gpio_resume_noirq)
+};
static struct platform_driver dwapb_gpio_driver = {
.driver = {
{
struct irq_domain *domain = data;
+ irq_domain_remove_generic_chips(domain);
irq_domain_remove(domain);
}
return ret;
}
+static int palmas_gpio_get_direction(struct gpio_chip *gc, unsigned int offset)
+{
+ struct palmas_gpio *pg = gpiochip_get_data(gc);
+ struct palmas *palmas = pg->palmas;
+ unsigned int val;
+ unsigned int reg;
+ int ret;
+ int gpio16 = (offset/8);
+
+ offset %= 8;
+ reg = (gpio16) ? PALMAS_GPIO_DATA_DIR2 : PALMAS_GPIO_DATA_DIR;
+ ret = palmas_read(palmas, PALMAS_GPIO_BASE, reg, &val);
+ if (ret)
+ return ret;
+
+ return (val & BIT(offset)) ? GPIO_LINE_DIRECTION_OUT : GPIO_LINE_DIRECTION_IN;
+}
+
static int palmas_gpio_to_irq(struct gpio_chip *gc, unsigned offset)
{
struct palmas_gpio *pg = gpiochip_get_data(gc);
palmas_gpio->gpio_chip.can_sleep = true;
palmas_gpio->gpio_chip.direction_input = palmas_gpio_input;
palmas_gpio->gpio_chip.direction_output = palmas_gpio_output;
+ palmas_gpio->gpio_chip.get_direction = palmas_gpio_get_direction;
palmas_gpio->gpio_chip.to_irq = palmas_gpio_to_irq;
palmas_gpio->gpio_chip.set = palmas_gpio_set;
palmas_gpio->gpio_chip.get = palmas_gpio_get;
struct gpio_chip gc;
struct gpio_shared_desc *shared_desc;
struct device *dev;
- bool voted_high;
+ bool voted_change;
};
static int
lockdep_assert_held(&shared_desc->mutex);
- if (value) {
- /* User wants to set value to high. */
- if (proxy->voted_high)
- /* Already voted for high, nothing to do. */
+ if (value != shared_desc->def_val) {
+ /* User wants to vote for a value change. */
+ if (proxy->voted_change)
+ /* Already voted for a change, nothing to do. */
goto out;
- /* Haven't voted for high yet. */
- if (!shared_desc->highcnt) {
+ /* Haven't voted for a value change yet. */
+ if (!shared_desc->votecnt) {
/*
- * Current value is low, need to actually set value
- * to high.
+ * Current value is default, need to actually set value
+ * to the opposite.
*/
- ret = gpiod_set_value_cansleep(desc, 1);
+ ret = gpiod_set_value_cansleep(desc, value);
if (ret)
goto out;
}
- shared_desc->highcnt++;
- proxy->voted_high = true;
+ shared_desc->votecnt++;
+ proxy->voted_change = true;
goto out;
}
- /* Desired value is low. */
- if (!proxy->voted_high)
- /* We didn't vote for high, nothing to do. */
+ /* Desired value is the default. */
+ if (!proxy->voted_change)
+ /* We didn't vote for change previously, nothing to do. */
goto out;
- /* We previously voted for high. */
- if (shared_desc->highcnt == 1) {
- /* This is the last remaining vote for high, set value to low. */
- ret = gpiod_set_value_cansleep(desc, 0);
+ /* We previously voted for change. */
+ if (shared_desc->votecnt == 1) {
+ /* This is the last remaining vote for change, set value to default. */
+ ret = gpiod_set_value_cansleep(desc, shared_desc->def_val);
if (ret)
goto out;
}
- shared_desc->highcnt--;
- proxy->voted_high = false;
+ shared_desc->votecnt--;
+ proxy->voted_change = false;
out:
- if (shared_desc->highcnt)
+ if (shared_desc->votecnt)
dev_dbg(proxy->dev,
- "Voted for value '%s', effective value is 'high', number of votes for 'high': %u\n",
- str_high_low(value), shared_desc->highcnt);
+ "Voted for value '%s', effective value is '%s', number of votes: %u\n",
+ str_high_low(value), str_high_low(!shared_desc->def_val),
+ shared_desc->votecnt);
else
- dev_dbg(proxy->dev, "Voted for value 'low', effective value is 'low'\n");
+ dev_dbg(proxy->dev, "Voted for value '%s', effective value is '%s'\n",
+ str_high_low(value), str_high_low(shared_desc->def_val));
return ret;
}
guard(mutex)(&shared_desc->mutex);
- if (proxy->voted_high) {
- ret = gpio_shared_proxy_set_unlocked(proxy, 0);
+ if (proxy->voted_change) {
+ ret = gpio_shared_proxy_set_unlocked(proxy, shared_desc->def_val);
if (ret)
dev_err(proxy->dev,
"Failed to unset the shared GPIO value on release: %d\n", ret);
if (ret)
return ret;
- if (value) {
- proxy->voted_high = true;
- shared_desc->highcnt = 1;
- } else {
- proxy->voted_high = false;
- shared_desc->highcnt = 0;
- }
+ shared_desc->def_val = value;
+ shared_desc->votecnt = 0;
+ proxy->voted_change = false;
return 0;
}
struct gpio_desc *desc;
unsigned long cfg;
unsigned int usecnt;
- unsigned int highcnt;
+ unsigned int votecnt;
+ int def_val;
struct mutex mutex; /* serializes all proxy operations on this descriptor */
};
blocks, total_allocated_on_err);
}
+static int __alloc_contig_aligned_retry(struct gpu_buddy *mm,
+ u64 unaligned_offset,
+ u64 size,
+ u64 min_block_size,
+ struct list_head *blocks)
+{
+ u64 aligned_offset = round_down(unaligned_offset, min_block_size);
+
+ return __gpu_buddy_alloc_range(mm, aligned_offset, size, NULL, blocks);
+}
+
static int __alloc_contig_try_harder(struct gpu_buddy *mm,
u64 size,
u64 min_block_size,
struct list_head *blocks)
{
- u64 rhs_offset, lhs_offset, lhs_size, filled;
+ u64 rhs_offset, lhs_offset, filled;
struct gpu_buddy_block *block;
unsigned int tree, order;
- LIST_HEAD(blocks_lhs);
- unsigned long pages;
u64 modify_size;
int err;
modify_size = rounddown_pow_of_two(size);
- pages = modify_size >> ilog2(mm->chunk_size);
- order = fls(pages) - 1;
+ order = ilog2(modify_size) - ilog2(mm->chunk_size);
if (order == 0)
return -ENOSPC;
while (iter) {
block = rbtree_get_free_block(iter);
- /* Allocate blocks traversing RHS */
rhs_offset = gpu_buddy_block_offset(block);
+
+ /* Allocate blocks traversing RHS */
err = __gpu_buddy_alloc_range(mm, rhs_offset, size,
&filled, blocks);
- if (!err || err != -ENOSPC)
+ if (err && err != -ENOSPC)
return err;
+ if (!err && IS_ALIGNED(rhs_offset, min_block_size))
+ return 0;
+ if (!err) {
+ /* Allocate the unaligned RHS offset using round_down */
+ gpu_buddy_free_list_internal(mm, blocks);
+ err = __alloc_contig_aligned_retry(mm, rhs_offset,
+ size,
+ min_block_size,
+ blocks);
+ if (!err)
+ return 0;
+ if (err != -ENOSPC) {
+ gpu_buddy_free_list_internal(mm, blocks);
+ return err;
+ }
+ goto next;
+ }
- lhs_size = max((size - filled), min_block_size);
- if (!IS_ALIGNED(lhs_size, min_block_size))
- lhs_size = round_up(lhs_size, min_block_size);
+ if (size - filled > rhs_offset)
+ goto next;
- /* Allocate blocks traversing LHS */
- lhs_offset = gpu_buddy_block_offset(block) - lhs_size;
- err = __gpu_buddy_alloc_range(mm, lhs_offset, lhs_size,
- NULL, &blocks_lhs);
- if (!err) {
- list_splice(&blocks_lhs, blocks);
+ lhs_offset = rhs_offset - (size - filled);
+
+ /* Allocate the unaligned LHS offset using round_down */
+ gpu_buddy_free_list_internal(mm, blocks);
+ err = __alloc_contig_aligned_retry(mm, lhs_offset, size,
+ min_block_size, blocks);
+ if (!err)
return 0;
- } else if (err != -ENOSPC) {
+ if (err != -ENOSPC) {
gpu_buddy_free_list_internal(mm, blocks);
return err;
}
- /* Free blocks for the next iteration */
+next:
gpu_buddy_free_list_internal(mm, blocks);
-
iter = rb_prev(iter);
}
}
}
#ifdef CONFIG_ACPI
+/**
+ * amdgpu_acpi_vfct_match() - Check if a VFCT entry matches the device
+ * @adev: AMDGPU device
+ * @vhdr: VFCT image header to check
+ *
+ * VFCT entries contain the PCI bus number as recorded during BIOS POST.
+ * On systems where the kernel renumbers PCI buses (e.g. pci=realloc or
+ * resource conflicts), the runtime bus number may differ from the POST
+ * value. Match by device identity (vendor + device + function) and use
+ * the bus number as a preference: exact bus match is preferred, but when
+ * the bus numbers disagree we accept the entry if the device identity
+ * matches.
+ *
+ * Returns: 0 on match, -ENODEV on no match
+ */
+static int amdgpu_acpi_vfct_match(struct amdgpu_device *adev,
+ VFCT_IMAGE_HEADER *vhdr)
+{
+ /* Vendor and device IDs must always match */
+ if (vhdr->VendorID != adev->pdev->vendor ||
+ vhdr->DeviceID != adev->pdev->device)
+ return -ENODEV;
+
+ if (vhdr->PCIDevice != PCI_SLOT(adev->pdev->devfn) ||
+ vhdr->PCIFunction != PCI_FUNC(adev->pdev->devfn))
+ return -ENODEV;
+
+ /* Exact bus number match - preferred */
+ if (vhdr->PCIBus == adev->pdev->bus->number)
+ return 0;
+
+ /* Bus mismatch but device identity matches (PCI renumbering case) */
+ dev_notice(adev->dev,
+ "VFCT bus number mismatch: table %u != runtime %u, matching by device identity (vendor 0x%04x device 0x%04x)\n",
+ vhdr->PCIBus, adev->pdev->bus->number,
+ adev->pdev->vendor, adev->pdev->device);
+ return 0;
+}
+
static bool amdgpu_acpi_vfct_bios(struct amdgpu_device *adev)
{
struct acpi_table_header *hdr;
acpi_size tbl_size;
UEFI_ACPI_VFCT *vfct;
unsigned int offset;
+ bool r = false;
if (!ACPI_SUCCESS(acpi_get_table("VFCT", 1, &hdr)))
return false;
tbl_size = hdr->length;
if (tbl_size < sizeof(UEFI_ACPI_VFCT)) {
dev_info(adev->dev, "ACPI VFCT table present but broken (too short #1),skipping\n");
- return false;
+ goto out;
}
vfct = (UEFI_ACPI_VFCT *)hdr;
offset += sizeof(VFCT_IMAGE_HEADER);
if (offset > tbl_size) {
dev_info(adev->dev, "ACPI VFCT image header truncated,skipping\n");
- return false;
+ goto out;
}
offset += vhdr->ImageLength;
if (offset > tbl_size) {
dev_info(adev->dev, "ACPI VFCT image truncated,skipping\n");
- return false;
+ goto out;
}
if (vhdr->ImageLength &&
- vhdr->PCIBus == adev->pdev->bus->number &&
- vhdr->PCIDevice == PCI_SLOT(adev->pdev->devfn) &&
- vhdr->PCIFunction == PCI_FUNC(adev->pdev->devfn) &&
- vhdr->VendorID == adev->pdev->vendor &&
- vhdr->DeviceID == adev->pdev->device) {
+ !amdgpu_acpi_vfct_match(adev, vhdr)) {
adev->bios = kmemdup(&vbios->VbiosContent,
vhdr->ImageLength,
GFP_KERNEL);
if (!check_atom_bios(adev, vhdr->ImageLength)) {
amdgpu_bios_release(adev);
- return false;
+ goto out;
}
adev->bios_size = vhdr->ImageLength;
- return true;
+ r = true;
+ goto out;
}
}
dev_info(adev->dev, "ACPI VFCT table present but broken (too short #2),skipping\n");
- return false;
+
+out:
+ acpi_put_table(hdr);
+ return r;
}
#else
static inline bool amdgpu_acpi_vfct_bios(struct amdgpu_device *adev)
drm_printf(p, "\nIB #%d 0x%llx %d dw\n", i,
coredump->ibs[i].gpu_addr,
coredump->ibs[i].ib_size_dw);
+
+ for (int j = 0; j < coredump->ibs[i].ib_size_dw; j++)
+ drm_printf(p, "0xffffffff\n");
}
return;
}
drm_printf(&p, "kernel: %s\n", init_utsname()->release);
drm_printf(&p, "module: " KBUILD_MODNAME "\n");
drm_printf(&p, "time: %ptSp\n", &coredump->reset_time);
+ drm_printf(&p, "pasid: %u\n", coredump->pasid);
+ drm_printf(&p, "vmid: %u\n", coredump->vmid);
if (coredump->reset_task_info.task.pid)
- drm_printf(&p, "process_name: %s PID: %d\n",
+ drm_printf(&p, "process_name: %s TGID: %d thread: %s PID: %d\n",
coredump->reset_task_info.process_name,
+ coredump->reset_task_info.tgid,
+ coredump->reset_task_info.task.comm,
coredump->reset_task_info.task.pid);
/* SOC Information */
amdgpu_vm_put_task_info(ti);
}
coredump->pasid = job->pasid;
+ coredump->vmid = job->vmid;
coredump->num_ibs = job->num_ibs;
for (i = 0; i < job->num_ibs; ++i) {
coredump->ibs[i].gpu_addr = job->ibs[i].gpu_addr;
char *formatted;
unsigned int pasid;
+ unsigned int vmid;
int num_ibs;
struct amdgpu_coredump_ib_info ibs[] __counted_by(num_ibs);
};
if (c->x86_vendor == X86_VENDOR_INTEL)
return false;
+
+ /*
+ * AMD Ryzen Pinnacle Ridge (Zen+, family 0x17 model 0x08) CPUs don't
+ * support PCIe dynamic speed switching.
+ * https://gitlab.freedesktop.org/drm/amd/-/work_items/5436
+ */
+ if (c->x86_vendor == X86_VENDOR_AMD && c->x86 == 0x17 &&
+ c->x86_model == 0x08)
+ return false;
#endif
return true;
}
* It's unclear if this is a platform-specific or GPU-specific issue.
* Disable ASPM on SI for the time being.
*/
- if (adev->family == AMDGPU_FAMILY_SI)
+ if (adev->family == AMDGPU_FAMILY_SI ||
+ (!(adev->pm.pp_feature & PP_PCIE_DPM_MASK) && adev->family == AMDGPU_FAMILY_VI))
return true;
#if IS_ENABLED(CONFIG_X86)
amdgpu_device_ip_block_add(adev, &psp_v14_0_ip_block);
break;
case IP_VERSION(15, 0, 0):
+ case IP_VERSION(15, 0, 9):
amdgpu_device_ip_block_add(adev, &psp_v15_0_ip_block);
break;
case IP_VERSION(15, 0, 8):
case IP_VERSION(15, 0, 0):
case IP_VERSION(15, 0, 5):
case IP_VERSION(15, 0, 8):
+ case IP_VERSION(15, 0, 9):
amdgpu_device_ip_block_add(adev, &smu_v15_0_ip_block);
break;
default:
case IP_VERSION(11, 5, 3):
case IP_VERSION(11, 5, 4):
case IP_VERSION(11, 5, 6):
+ adev->family = AMDGPU_FAMILY_GC_11_5_0;
+ break;
case IP_VERSION(11, 7, 0):
case IP_VERSION(11, 7, 1):
- adev->family = AMDGPU_FAMILY_GC_11_5_0;
+ adev->family = AMDGPU_FAMILY_GC_11_5_4;
break;
case IP_VERSION(12, 0, 0):
case IP_VERSION(12, 0, 1):
goto error_free;
}
- r = amdgpu_bo_pin(*bo_ptr, domain);
- if (r) {
- dev_err(adev->dev, "(%d) kernel bo pin failed\n", r);
- goto error_unreserve;
+ if (free) {
+ r = amdgpu_bo_pin(*bo_ptr, domain);
+ if (r) {
+ dev_err(adev->dev, "(%d) kernel bo pin failed\n", r);
+ goto error_unreserve;
+ }
}
r = amdgpu_ttm_alloc_gart(&(*bo_ptr)->tbo);
return 0;
error_unpin:
- amdgpu_bo_unpin(*bo_ptr);
+ if (free)
+ amdgpu_bo_unpin(*bo_ptr);
error_unreserve:
amdgpu_bo_unreserve(*bo_ptr);
psp->boot_time_tmr = false;
break;
case IP_VERSION(15, 0, 0):
+ case IP_VERSION(15, 0, 9):
psp_v15_0_0_set_psp_funcs(psp);
psp->boot_time_tmr = false;
break;
amdgpu_ip_version(adev, MP0_HWIP, 0) ==
IP_VERSION(15, 0, 0) ||
amdgpu_ip_version(adev, MP0_HWIP, 0) ==
- IP_VERSION(15, 0, 8)) &&
+ IP_VERSION(15, 0, 8) ||
+ amdgpu_ip_version(adev, MP0_HWIP, 0) ==
+ IP_VERSION(15, 0, 9)) &&
(ucode->ucode_id == AMDGPU_UCODE_ID_SDMA1 ||
ucode->ucode_id == AMDGPU_UCODE_ID_SDMA2 ||
ucode->ucode_id == AMDGPU_UCODE_ID_SDMA3))
return;
}
- /* Navi1x's workaround requires us to limit to a single SDMA sched
- * for ttm.
- */
hub = &adev->vmhub[AMDGPU_GFXHUB(0)];
- adev->mman.num_buffer_funcs_scheds = hub->sdma_invalidation_workaround ?
- 1 : n;
+
+ /*
+ * Allow using multiple SDMA schedulers only on GPUs where
+ * we are allowed to do concurrent VM flushes.
+ * This consideration is necessary because all GART windows
+ * are mapped in VMID 0 (the kernel VMID) so each buffer
+ * entity would flush VMID 0 concurrently.
+ *
+ * Also consider the SDMA invalidation workaround on
+ * Navi 1x GPUs, which also prevents us from using
+ * multiple SDMA engines on VMID 0 at the same time.
+ */
+ adev->mman.num_buffer_funcs_scheds =
+ (adev->vm_manager.concurrent_flush &&
+ !hub->sdma_invalidation_workaround) ? n : 1;
}
#if defined(CONFIG_DEBUG_FS)
amdgpu_userq_cleanup(queue);
mutex_unlock(&uq_mgr->userq_mutex);
+ /*
+ * A failed unmap means MES could not remove the hung queue and is now
+ * unresponsive. Recover the GPU here so the wedged MES does not fail
+ * the next, unrelated queue submission and trigger a reset attributed
+ * to an innocent workload.
+ */
+ if (r)
+ queue_work(adev->reset_domain->wq, &uq_mgr->reset_work);
+
cancel_delayed_work_sync(&queue->hang_detect_work);
uq_funcs->mqd_destroy(queue);
queue->userq_mgr = NULL;
/* TODO: We probably need a new lock for the queue state */
xa_for_each(&adev->userq_doorbell_xa, queue_id, queue) {
- if (queue->state != AMDGPU_USERQ_STATE_MAPPED)
- continue;
-
- userq_funcs = adev->userq_funcs[queue->queue_type];
- userq_funcs->unmap(queue);
- /* just mark all queues as hung at this point.
- * if unmap succeeds, we could map again
- * in amdgpu_userq_post_reset() if vram is not lost
+ if (queue->state == AMDGPU_USERQ_STATE_MAPPED) {
+ userq_funcs = adev->userq_funcs[queue->queue_type];
+ userq_funcs->unmap(queue);
+ /* just mark all queues as hung at this point.
+ * if unmap succeeds, we could map again
+ * in amdgpu_userq_post_reset() if vram is not lost
+ */
+ queue->state = AMDGPU_USERQ_STATE_HUNG;
+ }
+ /* Force-complete any pending fence regardless of queue state so
+ * that eviction/suspend and queue teardown waiters don't block
+ * forever on a fence that will never signal after the reset.
*/
- queue->state = AMDGPU_USERQ_STATE_HUNG;
amdgpu_userq_fence_driver_force_completion(queue);
}
}
job->oa_size);
}
- if (vm_flush_needed || pasid_mapping_needed || cleaner_shader_needed) {
- amdgpu_fence_emit(ring, job->hw_vm_fence, 0);
- fence = &job->hw_vm_fence->base;
- /* get a ref for the job */
- dma_fence_get(fence);
- }
+ amdgpu_fence_emit(ring, job->hw_vm_fence, 0);
+ fence = &job->hw_vm_fence->base;
+ /* get a ref for the job */
+ dma_fence_get(fence);
if (vm_flush_needed) {
mutex_lock(&id_mgr->lock);
kfree(ti);
}
-static inline struct amdgpu_vm *
-amdgpu_vm_get_vm_from_pasid(struct amdgpu_device *adev, u32 pasid)
-{
- struct amdgpu_vm *vm;
- unsigned long flags;
-
- xa_lock_irqsave(&adev->vm_manager.pasids, flags);
- vm = xa_load(&adev->vm_manager.pasids, pasid);
- xa_unlock_irqrestore(&adev->vm_manager.pasids, flags);
-
- return vm;
-}
-
/**
* amdgpu_vm_put_task_info - reference down the vm task_info ptr
*
struct amdgpu_task_info *
amdgpu_vm_get_task_info_pasid(struct amdgpu_device *adev, u32 pasid)
{
- return amdgpu_vm_get_task_info_vm(
- amdgpu_vm_get_vm_from_pasid(adev, pasid));
+ struct amdgpu_task_info *ti;
+ struct amdgpu_vm *vm;
+ unsigned long flags;
+
+ xa_lock_irqsave(&adev->vm_manager.pasids, flags);
+ vm = xa_load(&adev->vm_manager.pasids, pasid);
+ ti = amdgpu_vm_get_task_info_vm(vm);
+ xa_unlock_irqrestore(&adev->vm_manager.pasids, flags);
+
+ return ti;
}
static int amdgpu_vm_create_task_info(struct amdgpu_vm *vm)
is_compute_context = vm->is_compute_context;
if (is_compute_context) {
+ __label__ drm_exec_retry;
+
/* Release the root PD lock since svm_range_restore_pages
* might try to take it.
* TODO: rework svm_range_restore_pages so that this isn't
gfx_v10_0_get_tcc_info(adev);
adev->gfx.config.pa_sc_tile_steering_override =
gfx_v10_0_init_pa_sc_tile_steering_override(adev);
+ /* Program DB_RING_CONTROL for multiple GFX pipes
+ * Default power up value is 1.
+ * Possible values:
+ * 0 - split occlusion counters between gfx pipes
+ * 1 - all occlusion counters to pipe 0
+ * 2 - all occlusion counters to pipe 1
+ */
+ WREG32_FIELD15(GC, 0, DB_RING_CONTROL, COUNTER_CONTROL,
+ (adev->gfx.me.num_pipe_per_me > 1) ? 0 : 1);
/* XXX SH_MEM regs */
/* where to put LDS, scratch, GPUVM in FSA64 space */
case IP_VERSION(14, 0, 4):
case IP_VERSION(14, 0, 5):
case IP_VERSION(15, 0, 0):
+ case IP_VERSION(15, 0, 9):
return AMD_RESET_METHOD_MODE2;
default:
if (amdgpu_dpm_is_baco_supported(adev))
AMD_CG_SUPPORT_BIF_LS;
adev->pg_flags = AMD_PG_SUPPORT_VCN_DPG |
AMD_PG_SUPPORT_VCN |
- AMD_PG_SUPPORT_JPEG_DPG |
AMD_PG_SUPPORT_JPEG |
AMD_PG_SUPPORT_GFX_PG;
adev->external_rev_id = adev->rev_id + 0xF;
AMD_CG_SUPPORT_BIF_LS;
adev->pg_flags = AMD_PG_SUPPORT_VCN_DPG |
AMD_PG_SUPPORT_VCN |
- AMD_PG_SUPPORT_JPEG_DPG |
AMD_PG_SUPPORT_JPEG |
AMD_PG_SUPPORT_GFX_PG;
adev->external_rev_id = adev->rev_id + 0x40;
struct device_queue_manager *device_queue_manager_init(struct kfd_node *dev)
{
struct device_queue_manager *dqm;
+ int i;
pr_debug("Loading device queue manager\n");
deallocate_hiq_sdma_mqd(dev, &dqm->hiq_sdma_mqd);
out_free:
+ for (i = 0; i < KFD_MQD_TYPE_MAX; i++)
+ kfree(dqm->mqd_mgrs[i]);
+
kfree(dqm);
return NULL;
}
dqm_unlock(dqm);
return r;
}
+
+size_t mqd_size_from_queue_type(struct device_queue_manager *dqm, enum kfd_queue_type type)
+{
+ return dqm->mqd_mgrs[get_mqd_type_from_queue_type(type)]->mqd_size;
+}
+
#if defined(CONFIG_DEBUG_FS)
static void seq_reg_dump(struct seq_file *m,
bool kfd_dqm_is_queue_in_process(struct device_queue_manager *dqm,
struct qcm_process_device *qpd,
int doorbell_off, u32 *queue_format);
+size_t mqd_size_from_queue_type(struct device_queue_manager *dqm,
+ enum kfd_queue_type type);
static inline unsigned int get_sh_mem_bases_32(struct kfd_process_device *pdd)
{
}
if (restore_id) {
+ if (*restore_id >= KFD_SIGNAL_EVENT_LIMIT)
+ return -EINVAL;
+
id = idr_alloc(&p->event_idr, ev, *restore_id, *restore_id + 1,
GFP_KERNEL);
} else {
ret = allocate_event_notification_slot(p, ev, restore_id);
if (ret) {
- pr_warn("Signal event wasn't created because out of kernel memory\n");
+ pr_warn("Failed to create signal event notification slot\n");
return ret;
}
KFD_QUEUE_TYPE_SDMA,
KFD_QUEUE_TYPE_HIQ,
KFD_QUEUE_TYPE_SDMA_XGMI,
- KFD_QUEUE_TYPE_SDMA_BY_ENG_ID
+ KFD_QUEUE_TYPE_SDMA_BY_ENG_ID,
+ KFD_QUEUE_TYPE_MAX,
};
enum kfd_queue_format {
{
struct kfd_process *primary_process;
- if (process->context_id != KFD_CONTEXT_ID_PRIMARY)
+ if (process->context_id == KFD_CONTEXT_ID_PRIMARY)
return;
primary_process = kfd_lookup_process_by_mm(process->lead_thread->mm);
(*q)->process = pqm->process;
if (dev->kfd->shared_resources.enable_mes) {
+ if (!q_properties->wptr_bo) {
+ pr_debug("Queue initialization with shared MES requires queue buffers to be initialized\n");
+ return -EINVAL;
+ }
+
retval = amdgpu_amdkfd_alloc_kernel_mem(dev->adev,
AMDGPU_MES_GANG_CTX_SIZE,
AMDGPU_GEM_DOMAIN_GTT,
goto exit;
}
+ pdd = kfd_process_device_data_by_id(p, q_data->gpu_id);
+ if (!pdd) {
+ pr_err("Failed to get pdd\n");
+ ret = -EINVAL;
+ goto exit;
+ }
+
+ if (q_data->type >= KFD_QUEUE_TYPE_MAX) {
+ ret = -EINVAL;
+ goto exit;
+ }
+
+ if (q_data->mqd_size != mqd_size_from_queue_type(pdd->dev->dqm, q_data->type)) {
+ ret = -EINVAL;
+ goto exit;
+ }
+
*priv_data_offset += sizeof(*q_data);
q_extra_data_size = (uint64_t)q_data->ctl_stack_size + q_data->mqd_size;
*priv_data_offset += q_extra_data_size;
- pdd = kfd_process_device_data_by_id(p, q_data->gpu_id);
- if (!pdd) {
- pr_err("Failed to get pdd\n");
- ret = -EINVAL;
- goto exit;
- }
-
/*
* data stored in this order:
* mqd[xcc0], mqd[xcc1],..., ctl_stack[xcc0], ctl_stack[xcc1]...
memset(&qp, 0, sizeof(qp));
set_queue_properties_from_criu(&qp, q_data, NUM_XCC(pdd->dev->adev->gfx.xcc_mask));
- ret = kfd_queue_acquire_buffers(pdd, &qp);
- if (ret) {
- pr_debug("failed to acquire user queue buffers for CRIU\n");
- goto exit;
- }
-
print_queue_properties(&qp);
ret = pqm_create_queue(&p->pqm, pdd->dev, &qp, &queue_id, q_data, mqd, ctl_stack, NULL);
if (ret) {
- kfd_queue_unref_bo_vas(pdd, &qp);
- kfd_queue_release_buffers(pdd, &qp);
pr_err("Failed to create new queue err:%d\n", ret);
goto exit;
}
*/
#include <linux/slab.h>
+#include <linux/overflow.h>
#include "kfd_priv.h"
#include "kfd_topology.h"
#include "kfd_svm.h"
struct kfd_topology_device *topo_dev;
u64 expected_queue_size;
struct amdgpu_vm *vm;
- u32 total_cwsr_size;
+ u64 total_cwsr_size;
int err;
topo_dev = kfd_topology_device_by_id(pdd->dev->id);
goto out_err_unreserve;
}
- total_cwsr_size = (properties->ctx_save_restore_area_size +
- topo_dev->node_props.debug_memory_size) * NUM_XCC(pdd->dev->xcc_mask);
+ total_cwsr_size = (u64)properties->ctx_save_restore_area_size +
+ topo_dev->node_props.debug_memory_size;
+ if (check_mul_overflow(total_cwsr_size,
+ NUM_XCC(pdd->dev->xcc_mask),
+ &total_cwsr_size)) {
+ err = -EINVAL;
+ goto out_err_unreserve;
+ }
total_cwsr_size = ALIGN(total_cwsr_size, PAGE_SIZE);
err = kfd_queue_buffer_get(vm, (void *)properties->ctx_save_restore_area_address,
int kfd_queue_release_buffers(struct kfd_process_device *pdd, struct queue_properties *properties)
{
struct kfd_topology_device *topo_dev;
- u32 total_cwsr_size;
+ u64 total_cwsr_size;
kfd_queue_buffer_put(&properties->wptr_bo);
kfd_queue_buffer_put(&properties->rptr_bo);
topo_dev = kfd_topology_device_by_id(pdd->dev->id);
if (!topo_dev)
return -EINVAL;
- total_cwsr_size = (properties->ctx_save_restore_area_size +
- topo_dev->node_props.debug_memory_size) * NUM_XCC(pdd->dev->xcc_mask);
+ total_cwsr_size = (u64)properties->ctx_save_restore_area_size +
+ topo_dev->node_props.debug_memory_size;
+ if (check_mul_overflow(total_cwsr_size,
+ NUM_XCC(pdd->dev->xcc_mask),
+ &total_cwsr_size))
+ return -EINVAL;
total_cwsr_size = ALIGN(total_cwsr_size, PAGE_SIZE);
kfd_queue_buffer_svm_put(pdd, properties->ctx_save_restore_area_address, total_cwsr_size);
queue_work(system_percpu_wq, &offload_work->work);
}
-static void dm_vupdate_high_irq(void *interrupt_params)
-{
- struct common_irq_params *irq_params = interrupt_params;
- struct amdgpu_device *adev = irq_params->adev;
- struct amdgpu_crtc *acrtc;
- struct drm_device *drm_dev;
- struct drm_vblank_crtc *vblank;
- ktime_t frame_duration_ns, previous_timestamp;
- unsigned long flags;
- int vrr_active;
-
- acrtc = get_crtc_by_otg_inst(adev, irq_params->irq_src - IRQ_TYPE_VUPDATE);
-
- if (acrtc) {
- vrr_active = amdgpu_dm_crtc_vrr_active_irq(acrtc);
- drm_dev = acrtc->base.dev;
- vblank = drm_crtc_vblank_crtc(&acrtc->base);
- previous_timestamp = atomic64_read(&irq_params->previous_timestamp);
- frame_duration_ns = vblank->time - previous_timestamp;
-
- if (frame_duration_ns > 0) {
- trace_amdgpu_refresh_rate_track(acrtc->base.index,
- frame_duration_ns,
- ktime_divns(NSEC_PER_SEC, frame_duration_ns));
- atomic64_set(&irq_params->previous_timestamp, vblank->time);
- }
-
- drm_dbg_vbl(drm_dev,
- "crtc:%d, vupdate-vrr:%d\n", acrtc->crtc_id,
- vrr_active);
-
- /* Core vblank handling is done here after end of front-porch in
- * vrr mode, as vblank timestamping will give valid results
- * while now done after front-porch. This will also deliver
- * page-flip completion events that have been queued to us
- * if a pageflip happened inside front-porch.
- */
- if (vrr_active && acrtc->dm_irq_params.stream) {
- bool replay_en = acrtc->dm_irq_params.stream->link->replay_settings.replay_feature_enabled;
- bool psr_en = acrtc->dm_irq_params.stream->link->psr_settings.psr_feature_enabled;
- bool fs_active_var_en = acrtc->dm_irq_params.freesync_config.state
- == VRR_STATE_ACTIVE_VARIABLE;
-
- amdgpu_dm_crtc_handle_vblank(acrtc);
-
- /* BTR processing for pre-DCE12 ASICs */
- if (adev->family < AMDGPU_FAMILY_AI) {
- spin_lock_irqsave(&adev_to_drm(adev)->event_lock, flags);
- mod_freesync_handle_v_update(
- adev->dm.freesync_module,
- acrtc->dm_irq_params.stream,
- &acrtc->dm_irq_params.vrr_params);
-
- if (fs_active_var_en || (!fs_active_var_en && !replay_en && !psr_en)) {
- schedule_dc_vmin_vmax(adev,
- acrtc->dm_irq_params.stream,
- &acrtc->dm_irq_params.vrr_params.adjust);
- }
- spin_unlock_irqrestore(&adev_to_drm(adev)->event_lock, flags);
- }
- }
- }
-}
-
/**
- * dm_crtc_high_irq() - Handles CRTC interrupt
- * @interrupt_params: used for determining the CRTC instance
+ * dm_crtc_high_irq_handler() - Common OTG vblank/flip event handling
+ * @adev: amdgpu device
+ * @acrtc: the CRTC to service
*
- * Handles the CRTC/VSYNC interrupt by notfying DRM's VBLANK
- * event handler.
+ * Performs writeback completion, vblank event handling, CRC processing, VRR BTR
+ * updates and pageflip completion delivery.
+ *
+ * On DCN this is driven by VUPDATE_NO_LOCK (the register latch point) from
+ * dm_vupdate_high_irq(); on DCE it is driven by VLINE0 at the start of vblank
+ * from dm_crtc_high_irq().
*/
-static void dm_crtc_high_irq(void *interrupt_params)
+static void dm_crtc_high_irq_handler(struct amdgpu_device *adev,
+ struct amdgpu_crtc *acrtc)
{
- struct common_irq_params *irq_params = interrupt_params;
- struct amdgpu_device *adev = irq_params->adev;
struct drm_writeback_job *job;
- struct amdgpu_crtc *acrtc;
unsigned long flags;
int vrr_active;
-
- acrtc = get_crtc_by_otg_inst(adev, irq_params->irq_src - IRQ_TYPE_VBLANK);
- if (!acrtc)
- return;
+ bool is_dcn = amdgpu_ip_version(adev, DCE_HWIP, 0) != 0;
if (acrtc->wb_conn) {
spin_lock_irqsave(&acrtc->wb_conn->job_lock, flags);
vrr_active, acrtc->dm_irq_params.active_planes);
/**
- * Core vblank handling at start of front-porch is only possible
- * in non-vrr mode, as only there vblank timestamping will give
- * valid results while done in front-porch. Otherwise defer it
- * to dm_vupdate_high_irq after end of front-porch.
+ * Core vblank handling.
+ *
+ * On DCN this handler runs at VUPDATE_NO_LOCK, the register latch
+ * point, which is the correct place to timestamp both VRR and non-VRR
+ * vblanks.
+ *
+ * On DCE this handler runs at the start of front-porch, where only
+ * non-VRR timestamping is valid; VRR vblank is deferred to
+ * dm_vupdate_high_irq() after end of front-porch.
*/
- if (!vrr_active)
+ if (is_dcn || !vrr_active)
amdgpu_dm_crtc_handle_vblank(acrtc);
/**
}
/*
- * If there aren't any active_planes then DCH HUBP may be clock-gated.
- * In that case, pageflip completion interrupts won't fire and pageflip
- * completion events won't get delivered. Prevent this by sending
- * pending pageflip events from here if a flip is still pending.
+ * Deliver pageflip completion events (DCN only).
+ *
+ * Since GRPH_PFLIP is not used, VUPDATE_NO_LOCK is the flip latch
+ * point. Deliver any pending pageflip completion event from here,
+ * once HW has consumed the new address (the OTG no longer reports a
+ * pending flip).
*
- * If any planes are enabled, use dm_pflip_high_irq() instead, to
- * avoid race conditions between flip programming and completion,
- * which could cause too early flip completion events.
+ * Also handle the case here where there aren't any active planes and
+ * DCN HUBP may be clock-gated, so the flip-pending status may be
+ * undefined.
*/
- if (adev->family >= AMDGPU_FAMILY_RV &&
- acrtc->pflip_status == AMDGPU_FLIP_SUBMITTED &&
- acrtc->dm_irq_params.active_planes == 0) {
+ if (is_dcn && acrtc->pflip_status == AMDGPU_FLIP_SUBMITTED &&
+ acrtc->event) {
+
+ if (!dc_get_flip_pending_on_otg(adev->dm.dc, acrtc->otg_inst)) {
+ drm_crtc_send_vblank_event(&acrtc->base, acrtc->event);
+ acrtc->event = NULL;
+ drm_crtc_vblank_put(&acrtc->base);
+ acrtc->pflip_status = AMDGPU_FLIP_NONE;
+ }
+ /*
+ * If the flip is still pending, leave it armed and
+ * retry on the next vupdate.
+ */
+ } else if (is_dcn && acrtc->pflip_status == AMDGPU_FLIP_SUBMITTED &&
+ acrtc->dm_irq_params.active_planes == 0) {
+
if (acrtc->event) {
drm_crtc_send_vblank_event(&acrtc->base, acrtc->event);
acrtc->event = NULL;
spin_unlock_irqrestore(&adev_to_drm(adev)->event_lock, flags);
}
+static void dm_vupdate_high_irq(void *interrupt_params)
+{
+ struct common_irq_params *irq_params = interrupt_params;
+ struct amdgpu_device *adev = irq_params->adev;
+ struct amdgpu_crtc *acrtc;
+ struct drm_device *drm_dev;
+ struct drm_vblank_crtc *vblank;
+ ktime_t frame_duration_ns, previous_timestamp;
+ unsigned long flags;
+ int vrr_active;
+
+ acrtc = get_crtc_by_otg_inst(adev, irq_params->irq_src - IRQ_TYPE_VUPDATE);
+ if (!acrtc)
+ return;
+
+ vrr_active = amdgpu_dm_crtc_vrr_active_irq(acrtc);
+ drm_dev = acrtc->base.dev;
+ vblank = drm_crtc_vblank_crtc(&acrtc->base);
+ previous_timestamp = atomic64_read(&irq_params->previous_timestamp);
+ frame_duration_ns = vblank->time - previous_timestamp;
+
+ if (frame_duration_ns > 0) {
+ trace_amdgpu_refresh_rate_track(acrtc->base.index,
+ frame_duration_ns,
+ ktime_divns(NSEC_PER_SEC, frame_duration_ns));
+ atomic64_set(&irq_params->previous_timestamp, vblank->time);
+ }
+
+ drm_dbg_vbl(drm_dev,
+ "crtc:%d, vupdate-vrr:%d\n", acrtc->crtc_id,
+ vrr_active);
+
+ /*
+ * On DCN, VUPDATE_NO_LOCK is the single OTG interrupt used to deliver
+ * vblank and pageflip completion events; VSTARTUP and GRPH_PFLIP are
+ * not used. Run the full handler here.
+ */
+ if (amdgpu_ip_version(adev, DCE_HWIP, 0) != 0) {
+ dm_crtc_high_irq_handler(adev, acrtc);
+ return;
+ }
+
+ /* DCE only below. */
+
+ /* Core vblank handling is done here after end of front-porch in
+ * vrr mode, as vblank timestamping will give valid results
+ * while now done after front-porch. This will also deliver
+ * page-flip completion events that have been queued to us
+ * if a pageflip happened inside front-porch.
+ */
+ if (vrr_active && acrtc->dm_irq_params.stream) {
+ bool replay_en = acrtc->dm_irq_params.stream->link->replay_settings.replay_feature_enabled;
+ bool psr_en = acrtc->dm_irq_params.stream->link->psr_settings.psr_feature_enabled;
+ bool fs_active_var_en = acrtc->dm_irq_params.freesync_config.state
+ == VRR_STATE_ACTIVE_VARIABLE;
+
+ amdgpu_dm_crtc_handle_vblank(acrtc);
+
+ /* BTR processing for pre-DCE12 ASICs */
+ if (adev->family < AMDGPU_FAMILY_AI) {
+ spin_lock_irqsave(&adev_to_drm(adev)->event_lock, flags);
+ mod_freesync_handle_v_update(
+ adev->dm.freesync_module,
+ acrtc->dm_irq_params.stream,
+ &acrtc->dm_irq_params.vrr_params);
+
+ if (fs_active_var_en || (!fs_active_var_en && !replay_en && !psr_en)) {
+ schedule_dc_vmin_vmax(adev,
+ acrtc->dm_irq_params.stream,
+ &acrtc->dm_irq_params.vrr_params.adjust);
+ }
+ spin_unlock_irqrestore(&adev_to_drm(adev)->event_lock, flags);
+ }
+ }
+}
+
+/**
+ * dm_crtc_high_irq() - Handles CRTC interrupt
+ * @interrupt_params: used for determining the CRTC instance
+ *
+ * Handles the CRTC/VSYNC interrupt by notifying DRM's VBLANK event handler.
+ *
+ * Used on DCE (VLINE0, set to vblank start). On DCN the equivalent handling is
+ * driven by VUPDATE_NO_LOCK in dm_vupdate_high_irq().
+ */
+static void dm_crtc_high_irq(void *interrupt_params)
+{
+ struct common_irq_params *irq_params = interrupt_params;
+ struct amdgpu_device *adev = irq_params->adev;
+ struct amdgpu_crtc *acrtc;
+
+ acrtc = get_crtc_by_otg_inst(adev, irq_params->irq_src - IRQ_TYPE_VBLANK);
+ if (!acrtc)
+ return;
+
+ dm_crtc_high_irq_handler(adev, acrtc);
+}
+
#if defined(CONFIG_DRM_AMD_SECURE_DISPLAY)
/**
* dm_dcn_vertical_interrupt0_high_irq() - Handles OTG Vertical interrupt0 for
*/
if (!dc_interrupt_set(adev->dm.dc, irq_source, enable))
drm_warn(adev_to_drm(adev), "Failed to %sable vblank interrupt\n", enable ? "en" : "dis");
+
+ } else if (acrtc && state->stream_status[i].plane_count != 0) {
+ /* DCN only needs to toggle VUPDATE_NO_LOCK */
+ rc = amdgpu_dm_crtc_set_vupdate_irq(&acrtc->base, enable);
+ if (rc)
+ drm_warn(adev_to_drm(adev), "Failed to %sable vupdate interrupt\n",
+ enable ? "en" : "dis");
}
}
caps->ext_caps = &aconnector->dc_link->dpcd_sink_ext_caps;
caps->aux_support = false;
+ panel_backlight_quirk = drm_get_panel_backlight_quirk(aconnector->drm_edid);
+
if (caps->ext_caps->bits.oled == 1
/*
* ||
caps->aux_support = false;
else if (amdgpu_backlight == 1)
caps->aux_support = true;
+ else if (!IS_ERR_OR_NULL(panel_backlight_quirk) &&
+ panel_backlight_quirk->force_pwm)
+ caps->aux_support = false;
if (caps->aux_support)
aconnector->dc_link->backlight_control_type = BACKLIGHT_CONTROL_AMD_AUX;
else
caps->aux_min_input_signal = 1;
- panel_backlight_quirk =
- drm_get_panel_backlight_quirk(aconnector->drm_edid);
if (!IS_ERR_OR_NULL(panel_backlight_quirk)) {
if (panel_backlight_quirk->min_brightness) {
caps->min_input_signal =
* for acknowledging and handling.
*/
- /* Use VSTARTUP interrupt */
- for (i = DCN_1_0__SRCID__DC_D1_OTG_VSTARTUP;
- i <= DCN_1_0__SRCID__DC_D1_OTG_VSTARTUP + adev->mode_info.num_crtc - 1;
- i++) {
- r = amdgpu_irq_add_id(adev, SOC15_IH_CLIENTID_DCE, i, &adev->crtc_irq);
-
- if (r) {
- drm_err(adev_to_drm(adev), "Failed to add crtc irq id!\n");
- return r;
- }
-
- int_params.int_context = INTERRUPT_HIGH_IRQ_CONTEXT;
- int_params.irq_source =
- dc_interrupt_to_irq_source(dc, i, 0);
-
- if (int_params.irq_source == DC_IRQ_SOURCE_INVALID ||
- int_params.irq_source < DC_IRQ_SOURCE_VBLANK1 ||
- int_params.irq_source > DC_IRQ_SOURCE_VBLANK6) {
- drm_err(adev_to_drm(adev), "Failed to register vblank irq!\n");
- return -EINVAL;
- }
-
- c_irq_params = &adev->dm.vblank_params[int_params.irq_source - DC_IRQ_SOURCE_VBLANK1];
-
- c_irq_params->adev = adev;
- c_irq_params->irq_src = int_params.irq_source;
-
- if (!amdgpu_dm_irq_register_interrupt(adev, &int_params,
- dm_crtc_high_irq, c_irq_params))
- return -ENOMEM;
- }
-
/* Use otg vertical line interrupt */
#if defined(CONFIG_DRM_AMD_SECURE_DISPLAY)
for (i = 0; i <= adev->mode_info.num_crtc - 1; i++) {
return -ENOMEM;
}
- /* Use GRPH_PFLIP interrupt */
- for (i = DCN_1_0__SRCID__HUBP0_FLIP_INTERRUPT;
- i <= DCN_1_0__SRCID__HUBP0_FLIP_INTERRUPT + dc->caps.max_otg_num - 1;
- i++) {
- r = amdgpu_irq_add_id(adev, SOC15_IH_CLIENTID_DCE, i, &adev->pageflip_irq);
- if (r) {
- drm_err(adev_to_drm(adev), "Failed to add page flip irq id!\n");
- return r;
- }
-
- int_params.int_context = INTERRUPT_HIGH_IRQ_CONTEXT;
- int_params.irq_source =
- dc_interrupt_to_irq_source(dc, i, 0);
-
- if (int_params.irq_source == DC_IRQ_SOURCE_INVALID ||
- int_params.irq_source < DC_IRQ_SOURCE_PFLIP_FIRST ||
- int_params.irq_source > DC_IRQ_SOURCE_PFLIP_LAST) {
- drm_err(adev_to_drm(adev), "Failed to register pflip irq!\n");
- return -EINVAL;
- }
-
- c_irq_params = &adev->dm.pflip_params[int_params.irq_source - DC_IRQ_SOURCE_PFLIP_FIRST];
-
- c_irq_params->adev = adev;
- c_irq_params->irq_src = int_params.irq_source;
-
- if (!amdgpu_dm_irq_register_interrupt(adev, &int_params,
- dm_pflip_high_irq, c_irq_params))
- return -ENOMEM;
- }
-
/* HPD */
r = amdgpu_irq_add_id(adev, SOC15_IH_CLIENTID_DCE, DCN_1_0__SRCID__DC_HPD1_INT,
&adev->hpd_irq);
caps = &dm->backlight_caps[aconnector->bl_idx];
if (get_brightness_range(caps, &min, &max)) {
if (power_supply_is_system_supplied() > 0)
- props.brightness = DIV_ROUND_CLOSEST((max - min) * caps->ac_level, 100);
+ props.brightness = DIV_ROUND_CLOSEST(max * caps->ac_level, 100);
else
- props.brightness = DIV_ROUND_CLOSEST((max - min) * caps->dc_level, 100);
+ props.brightness = DIV_ROUND_CLOSEST(max * caps->dc_level, 100);
/* min is zero, so max needs to be adjusted */
- props.max_brightness = max - min;
+ props.max_brightness = max;
drm_dbg(drm, "Backlight caps: min: %d, max: %d, ac %d, dc %d\n", min, max,
caps->ac_level, caps->dc_level);
} else
if (acrtc_state) {
timing = &acrtc_state->stream->timing;
- if (amdgpu_ip_version(adev, DCE_HWIP, 0) >=
- IP_VERSION(3, 2, 0) &&
- !(adev->flags & AMD_IS_APU)) {
- /*
- * DGPUs NV3x and newer that support idle optimizations
- * experience intermittent flip-done timeouts on cursor
- * updates. Restore 5s offdelay behavior for now.
- *
- * Discussion on the issue:
- * https://lore.kernel.org/amd-gfx/20260217191632.1243826-1-sysdadmin@m1k.cloud/
- */
- config.offdelay_ms = 5000;
- config.disable_immediate = false;
- } else if (amdgpu_ip_version(adev, DCE_HWIP, 0) <
- IP_VERSION(3, 5, 0)) {
+ if (amdgpu_ip_version(adev, DCE_HWIP, 0) <
+ IP_VERSION(3, 5, 0) ||
+ !(adev->flags & AMD_IS_APU)) {
/*
* Older HW and DGPU have issues with instant off;
* use a 2 frame offdelay.
drm_crtc_vblank_on_config(&acrtc->base,
&config);
- /* Allow RX6xxx, RX7700, RX7800 GPUs to call amdgpu_irq_get.*/
+ /*
+ * Since pflip_high_irq is no longer registered for DCN, grab an
+ * extra reference to vupdate irq instead to workaround this
+ * issue:
+ * https://gitlab.freedesktop.org/drm/amd/-/work_items/3936
+ *
+ * The callbacks to drm_vblank_on/off should really take care of
+ * this though.
+ */
switch (amdgpu_ip_version(adev, DCE_HWIP, 0)) {
case IP_VERSION(3, 0, 0):
case IP_VERSION(3, 0, 2):
case IP_VERSION(3, 0, 3):
case IP_VERSION(3, 2, 0):
- if (amdgpu_irq_get(adev, &adev->pageflip_irq, irq_type))
- drm_err(dev, "DM_IRQ: Cannot get pageflip irq!\n");
+ if (amdgpu_irq_get(adev, &adev->vupdate_irq, irq_type))
+ drm_err(dev, "DM_IRQ: Cannot get vupdate irq!\n");
#if defined(CONFIG_DRM_AMD_SECURE_DISPLAY)
if (amdgpu_irq_get(adev, &adev->vline0_irq, irq_type))
drm_err(dev, "DM_IRQ: Cannot get vline0 irq!\n");
if (amdgpu_irq_put(adev, &adev->vline0_irq, irq_type))
drm_err(dev, "DM_IRQ: Cannot put vline0 irq!\n");
#endif
- if (amdgpu_irq_put(adev, &adev->pageflip_irq, irq_type))
- drm_err(dev, "DM_IRQ: Cannot put pageflip irq!\n");
+ if (amdgpu_irq_put(adev, &adev->vupdate_irq, irq_type))
+ drm_err(dev, "DM_IRQ: Cannot put vupdate irq!\n");
}
drm_crtc_vblank_off(&acrtc->base);
int irq_type =
amdgpu_display_crtc_idx_to_irq_type(adev, acrtc->crtc_id);
+ /* GRPH_PFLIP is not used on DCN; nothing to reapply. */
+ if (amdgpu_ip_version(adev, DCE_HWIP, 0) != 0)
+ return;
+
/**
* This reads the current state for the IRQ and force reapplies
* the setting to hardware.
struct dm_crtc_state *old_state,
struct dm_crtc_state *new_state)
{
+ struct amdgpu_device *adev = drm_to_adev(new_state->base.crtc->dev);
bool old_vrr_active = amdgpu_dm_crtc_vrr_active(old_state);
bool new_vrr_active = amdgpu_dm_crtc_vrr_active(new_state);
+ /* Only DCE gates vupdate on VRR, keep it enabled for DCN */
+ bool vrr_gates_vupdate = amdgpu_ip_version(adev, DCE_HWIP, 0) == 0;
+
if (!old_vrr_active && new_vrr_active) {
/* Transition VRR inactive -> active:
* While VRR is active, we must not disable vblank irq, as a
* We also need vupdate irq for the actual core vblank handling
* at end of vblank.
*/
- WARN_ON(amdgpu_dm_crtc_set_vupdate_irq(new_state->base.crtc, true) != 0);
+ if (vrr_gates_vupdate)
+ WARN_ON(amdgpu_dm_crtc_set_vupdate_irq(new_state->base.crtc, true) != 0);
WARN_ON(drm_crtc_vblank_get(new_state->base.crtc) != 0);
drm_dbg_driver(new_state->base.crtc->dev, "%s: crtc=%u VRR off->on: Get vblank ref\n",
__func__, new_state->base.crtc->base.id);
/* Transition VRR active -> inactive:
* Allow vblank irq disable again for fixed refresh rate.
*/
- WARN_ON(amdgpu_dm_crtc_set_vupdate_irq(new_state->base.crtc, false) != 0);
+ if (vrr_gates_vupdate)
+ WARN_ON(amdgpu_dm_crtc_set_vupdate_irq(new_state->base.crtc, false) != 0);
drm_crtc_vblank_put(new_state->base.crtc);
drm_dbg_driver(new_state->base.crtc->dev, "%s: crtc=%u VRR on->off: Drop vblank ref\n",
__func__, new_state->base.crtc->base.id);
}
}
+static void dm_arm_vblank_event(struct amdgpu_crtc *acrtc,
+ struct dm_crtc_state *acrtc_state,
+ bool pflip_update,
+ bool cursor_update)
+{
+ assert_spin_locked(&acrtc->base.dev->event_lock);
+
+ if (!acrtc->base.state->event || acrtc_state->active_planes == 0)
+ return;
+
+ if (pflip_update) {
+ drm_crtc_vblank_get(&acrtc->base);
+ WARN_ON(acrtc->pflip_status != AMDGPU_FLIP_NONE);
+ /* Arm flip completion handling and event delivery after programming. */
+ prepare_flip_isr(acrtc);
+ } else if (cursor_update) {
+ drm_crtc_vblank_get(&acrtc->base);
+ acrtc->event = acrtc->base.state->event;
+ acrtc->base.state->event = NULL;
+ }
+}
+
static void amdgpu_dm_commit_planes(struct drm_atomic_commit *state,
struct drm_device *dev,
struct amdgpu_display_manager *dm,
bool vrr_active = amdgpu_dm_crtc_vrr_active(acrtc_state);
bool cursor_update = false;
bool pflip_present = false;
+ bool immediate_flip = false;
+ bool flip_latched_during_prog = false;
bool dirty_rects_changed = false;
bool updated_planes_and_streams = false;
struct {
acrtc_state->update_type == UPDATE_TYPE_FAST &&
get_mem_type(old_plane_state->fb) == get_mem_type(fb);
+ immediate_flip |= bundle->flip_addrs[planes_count].flip_immediate;
+
timestamp_ns = ktime_get_ns();
bundle->flip_addrs[planes_count].flip_timestamp_in_us = div_u64(timestamp_ns, 1000);
bundle->surface_updates[planes_count].flip_addr = &bundle->flip_addrs[planes_count];
usleep_range(1000, 1100);
}
- /**
- * Prepare the flip event for the pageflip interrupt to handle.
- *
- * This only works in the case where we've already turned on the
- * appropriate hardware blocks (eg. HUBP) so in the transition case
- * from 0 -> n planes we have to skip a hardware generated event
- * and rely on sending it from software.
- */
- if (acrtc_attach->base.state->event &&
- acrtc_state->active_planes > 0) {
- drm_crtc_vblank_get(pcrtc);
-
- spin_lock_irqsave(&pcrtc->dev->event_lock, flags);
-
- WARN_ON(acrtc_attach->pflip_status != AMDGPU_FLIP_NONE);
- prepare_flip_isr(acrtc_attach);
-
- spin_unlock_irqrestore(&pcrtc->dev->event_lock, flags);
- }
-
if (acrtc_state->stream) {
if (acrtc_state->freesync_vrr_info_changed)
bundle->stream_update.vrr_infopacket =
&acrtc_state->stream->vrr_infopacket;
}
- } else if (cursor_update && acrtc_state->active_planes > 0) {
- spin_lock_irqsave(&pcrtc->dev->event_lock, flags);
- if (acrtc_attach->base.state->event) {
- drm_crtc_vblank_get(pcrtc);
- acrtc_attach->event = acrtc_attach->base.state->event;
- acrtc_attach->base.state->event = NULL;
+ }
+
+ /*
+ * DCE depends on a combination of GRPH_FLIP, VLINE0, and VUPDATE for
+ * event delivery. Only GRPH_FLIP handler can send pflip events, and it
+ * only fires if HW latched to the flip. Maintain legacy behavior by
+ * arming event before programming.
+ */
+ if (amdgpu_ip_version(dm->adev, DCE_HWIP, 0) == 0) {
+ scoped_guard(spinlock_irqsave, &pcrtc->dev->event_lock) {
+ dm_arm_vblank_event(acrtc_attach, acrtc_state,
+ pflip_present, cursor_update);
}
- spin_unlock_irqrestore(&pcrtc->dev->event_lock, flags);
}
/* Update the planes if changed or disable if we don't have any. */
acrtc_state->cursor_mode == DM_CURSOR_NATIVE_MODE)
amdgpu_dm_commit_cursors(state);
+ /*
+ * DCN specific vblank handling
+ * ============================
+ *
+ * With the event_lock held, arm the vblank event, and determine whether
+ * deliver it immediately, or in VUPDATE_NO_LOCK IRQ (i.e. HW latch
+ * point) handler. Do this *after* programming so that the IRQ handler
+ * will not deliver the event before HW laches onto the programmed
+ * values:
+ *
+ * Commit thread IRQ handler HW
+ * -----------------------------------------------------------------
+ * arm_vblank_event()
+ * vupdate()
+ * vupdate_handler()
+ * cook_timestamp()
+ * # prev flip already latched,
+ * # so flip_latched == true.
+ * if event_armed && flip_latched:
+ * send_vblank_event()
+ * # sent before latch, **BAD!**
+ * hw_program()
+ * vupdate()
+ * **latch**
+ *
+ * There's a consequence of arming after: it's possible for HW to latch
+ * between start of HW programming and acrtc->event/pflip_status arming.
+ * When this happens, the IRQ handler will send the event on the next
+ * immediate latch point, even though HW has already latched. This is
+ * handled by optimistically checking for HW latch after programming,
+ * and if latched, send the event immediately:
+ *
+ * Commit thread IRQ handler HW
+ * -----------------------------------------------------------------
+ * hw_program()
+ * vupdate()
+ * **latch**
+ * vupdate_handler()
+ * cook_timestamp()
+ * # event_armed == false
+ * # **no event sent!**
+ * arm_vblank_event()
+ * if flip_latched:
+ * **send_vblank_event()**
+ * disarm_vblank_event()
+ *
+ * The IRQ handler is expected to cook the timestamp, but we need to
+ * cook the timestamp before optimistic sending as well. That's because
+ * the following sequence is possible:
+ *
+ * Commit thread IRQ handler HW
+ * -----------------------------------------------------------------
+ * hw_program()
+ * arm_vblank_event()
+ * vupdate()
+ * **latch**
+ * if flip_latched:
+ * # Need cook before send!
+ * **cook_timestamp()**
+ * send_vblank_event()
+ * disarm_vblank_event()
+ * vupdate_handler()
+ * cook_timestamp()
+ * # event_armed == false
+ * # no event sent!
+ *
+ * Cooking twice is OK, since DRM scanout accurate timestamps report A)
+ * the previous vactive start if currently in vactive, or B) the next
+ * vactive start if currently in vblank (see &get_vblank_counter). 'A)'
+ * is what we want for the optimistic send, and for 'B)', we'll cook a
+ * timestamp no later than the next IRQ handler run.
+ *
+ * The more correct fix is to wrap programming and arming with the
+ * event_lock and thus serializing it with the IRQ handler. However,
+ * there are various sleep-waits within
+ * update_planes_and_stream_adapter() that makes spin locking illegal.
+ * And on full updates, it can take 1-2 frame-times to return (see
+ * commit_planes_for_stream).
+ *
+ * On DCE, GRPH_PFLIP IRQ is used and takes care of this.
+ */
+ if (amdgpu_ip_version(dm->adev, DCE_HWIP, 0) != 0) {
+ spin_lock_irqsave(&pcrtc->dev->event_lock, flags);
+
+ if (updated_planes_and_streams) {
+ flip_latched_during_prog =
+ !dc_get_flip_pending_on_otg(dm->dc, acrtc_attach->otg_inst);
+ }
+
+ dm_arm_vblank_event(acrtc_attach, acrtc_state,
+ pflip_present, cursor_update);
+
+ /*
+ * Deliver the event immediately on immediate flip, or on a
+ * update that has already latched.
+ */
+ if ((immediate_flip || flip_latched_during_prog) &&
+ acrtc_attach->pflip_status == AMDGPU_FLIP_SUBMITTED &&
+ acrtc_attach->event) {
+ drm_crtc_accurate_vblank_count(&acrtc_attach->base);
+ drm_crtc_send_vblank_event(&acrtc_attach->base,
+ acrtc_attach->event);
+ acrtc_attach->event = NULL;
+ drm_crtc_vblank_put(&acrtc_attach->base);
+ acrtc_attach->pflip_status = AMDGPU_FLIP_NONE;
+ }
+ spin_unlock_irqrestore(&pcrtc->dev->event_lock, flags);
+ }
+
cleanup:
kfree(bundle);
}
/* Release extra reference */
if (new_stream)
dc_stream_release(new_stream);
+ new_stream = NULL;
/*
* We want to do dc stream updates that do not require a
/* Overlay cursor not supported on HW before DCN
* DCN401/420 does not have the cursor-on-scaled-plane or cursor-on-yuv-plane restrictions
* as previous DCN generations, so enable native mode on DCN401/420
+ *
+ * Always set native cursor mode when the CRTC is disabled,
+ * to make sure it doesn't cause atomic commits to fail when
+ * they are trying to disable the CRTC.
*/
if (amdgpu_ip_version(adev, DCE_HWIP, 0) == IP_VERSION(4, 0, 1) ||
amdgpu_ip_version(adev, DCE_HWIP, 0) == IP_VERSION(4, 2, 0) ||
- amdgpu_ip_version(adev, DCE_HWIP, 0) == IP_VERSION(4, 2, 1)) {
+ amdgpu_ip_version(adev, DCE_HWIP, 0) == IP_VERSION(4, 2, 1) ||
+ !dm_crtc_state->base.enable) {
*cursor_mode = DM_CURSOR_NATIVE_MODE;
return 0;
}
drm_crtc_vblank_restore(crtc);
}
- if (dc_supports_vrr(dm->dc->ctx->dce_version)) {
+ /*
+ * On DCN, VUPDATE_NO_LOCK is the single OTG interrupt used to deliver
+ * vblank and pageflip completion events, so enable it whenever vblank
+ * is enabled. On DCE, vupdate is only needed in VRR mode.
+ */
+ if (amdgpu_ip_version(adev, DCE_HWIP, 0) != 0) {
+ rc = amdgpu_dm_crtc_set_vupdate_irq(crtc, enable);
+ } else if (dc_supports_vrr(dm->dc->ctx->dce_version)) {
if (enable) {
/* vblank irq on -> Only need vupdate irq in vrr mode */
if (amdgpu_dm_crtc_vrr_active(acrtc_state))
}
}
- if (rc)
- return rc;
-
- /* crtc vblank or vstartup interrupt */
- if (enable) {
- rc = amdgpu_irq_get(adev, &adev->crtc_irq, irq_type);
- drm_dbg_vbl(crtc->dev, "Get crtc_irq ret=%d\n", rc);
- } else {
- rc = amdgpu_irq_put(adev, &adev->crtc_irq, irq_type);
- drm_dbg_vbl(crtc->dev, "Put crtc_irq ret=%d\n", rc);
- }
-
if (rc)
return rc;
/*
- * hubp surface flip interrupt
- *
- * We have no guarantee that the frontend index maps to the same
- * backend index - some even map to more than one.
- *
- * TODO: Use a different interrupt or check DC itself for the mapping.
+ * VLINE0 (crtc_irq) and GRPH_PFLIP (pageflip_irq) are only used on
+ * DCE. On DCN, vblank and pageflip completion are delivered from
+ * VUPDATE_NO_LOCK (enabled above), so don't touch them here.
*/
- if (enable) {
- rc = amdgpu_irq_get(adev, &adev->pageflip_irq, irq_type);
- drm_dbg_vbl(crtc->dev, "Get pageflip_irq ret=%d\n", rc);
- } else {
- rc = amdgpu_irq_put(adev, &adev->pageflip_irq, irq_type);
- drm_dbg_vbl(crtc->dev, "Put pageflip_irq ret=%d\n", rc);
- }
+ if (amdgpu_ip_version(adev, DCE_HWIP, 0) == 0) {
+ /* crtc vblank or vstartup interrupt */
+ if (enable) {
+ rc = amdgpu_irq_get(adev, &adev->crtc_irq, irq_type);
+ drm_dbg_vbl(crtc->dev, "Get crtc_irq ret=%d\n", rc);
+ } else {
+ rc = amdgpu_irq_put(adev, &adev->crtc_irq, irq_type);
+ drm_dbg_vbl(crtc->dev, "Put crtc_irq ret=%d\n", rc);
+ }
- if (rc)
- return rc;
+ if (rc)
+ return rc;
+
+ /*
+ * hubp surface flip interrupt
+ *
+ * We have no guarantee that the frontend index maps to the same
+ * backend index - some even map to more than one.
+ *
+ * TODO: Use a different interrupt or check DC itself for the mapping.
+ */
+ if (enable) {
+ rc = amdgpu_irq_get(adev, &adev->pageflip_irq, irq_type);
+ drm_dbg_vbl(crtc->dev, "Get pageflip_irq ret=%d\n", rc);
+ } else {
+ rc = amdgpu_irq_put(adev, &adev->pageflip_irq, irq_type);
+ drm_dbg_vbl(crtc->dev, "Put pageflip_irq ret=%d\n", rc);
+ }
+
+ if (rc)
+ return rc;
+ }
#if defined(CONFIG_DRM_AMD_SECURE_DISPLAY)
/* crtc vline0 interrupt, only available on DCN+ */
continue;
edid = drm_edid_raw(drm_edid); // FIXME: Get rid of drm_edid_raw()
- if (!edid ||
- edid->extensions >= sizeof(sink->dc_edid.raw_edid) / EDID_LENGTH)
+ /*
+ * Use the length of the EDID property blob populated by
+ * drm_edid_connector_update() above. It reflects the true number
+ * of EDID blocks, including any HDMI Forum EDID Extension Override
+ * Data Block (HF-EEODB) count, which the raw byte 0x7e extension
+ * count can hide (e.g. HDMI 8K sinks).
+ */
+ if (!edid || !connector->edid_blob_ptr ||
+ connector->edid_blob_ptr->length > sizeof(sink->dc_edid.raw_edid))
return EDID_BAD_INPUT;
- sink->dc_edid.length = EDID_LENGTH * (edid->extensions + 1);
+ /*
+ * FIXME: amdgpu_dm today does not consider the HF-EEODB, which
+ * may contain additional mode info for sinks. This is a
+ * workaround until dc_edid is refactored out from DC into
+ * amdgpu_dm's ownership, allowing amdgpu_dm to use drm_edid
+ * directly
+ */
+ sink->dc_edid.length = connector->edid_blob_ptr->length;
memmove(sink->dc_edid.raw_edid, (uint8_t *)edid, sink->dc_edid.length);
/* We don't need the original edid anymore */
dal_irq_service_ack(dc->res_pool->irqs, src);
}
+/*
+ * dc_get_flip_pending_on_otg() - Check if a GRPH_FLIP is still pending on OTG
+ *
+ * @dc: display core context @otg_inst: OTG instance to query
+ *
+ * Reads the HUBP flip-pending status for the pipe(s) bound to @otg_inst,
+ * returning true if any of them has not yet latched its programmed surface
+ * address.
+ *
+ * Unlike dc_plane_get_status(), this does not take or mutate a dc_plane_state,
+ * so it is safe to call from interrupt context without racing a concurrent
+ * commit that may be updating plane state.
+ *
+ * Return: true if a flip is still pending on the OTG, false otherwise.
+ */
+bool dc_get_flip_pending_on_otg(struct dc *dc, int otg_inst)
+{
+ bool flip_pending = false;
+ int i;
+
+ if (!dc || !dc->current_state)
+ return false;
+
+ dc_exit_ips_for_hw_access(dc);
+
+ for (i = 0; i < dc->res_pool->pipe_count; i++) {
+ struct pipe_ctx *pipe_ctx = &dc->current_state->res_ctx.pipe_ctx[i];
+ struct hubp *hubp = pipe_ctx->plane_res.hubp;
+
+ if (!pipe_ctx->plane_state || !pipe_ctx->stream_res.tg)
+ continue;
+
+ if (pipe_ctx->stream_res.tg->inst != otg_inst)
+ continue;
+
+ if (hubp && hubp->funcs->hubp_is_flip_pending &&
+ hubp->funcs->hubp_is_flip_pending(hubp)) {
+ flip_pending = true;
+ break;
+ }
+ }
+
+ return flip_pending;
+}
+
void dc_power_down_on_boot(struct dc *dc)
{
if (dc->ctx->dce_environment != DCE_ENV_VIRTUAL_HW &&
bool dp_skip_DID2;
bool dp_skip_reset_segment;
bool dp_skip_fs_144hz;
+ /* Some DP bridges don't work with RBR and must use HBR. */
+ bool dp_skip_rbr;
bool dp_mot_reset_segment;
/* Some USB4 docks do not handle turning off MST DSC once it has been enabled. */
bool dpia_mst_dsc_always_on;
uint32_t ext_id);
bool dc_interrupt_set(struct dc *dc, enum dc_irq_source src, bool enable);
void dc_interrupt_ack(struct dc *dc, enum dc_irq_source src);
+bool dc_get_flip_pending_on_otg(struct dc *dc, int otg_inst);
enum dc_irq_source dc_get_hpd_irq_source_at_index(
struct dc *dc, uint32_t link_index);
*/
modulo_hz = REG_READ(MODULO[inst]);
if (modulo_hz) {
- temp = div_u64((uint64_t)clock_hz * dp_dto_ref_khz * 10, modulo_hz);
- ASSERT(temp / 100 <= 0xFFFFFFFFUL);
- *pixel_clk_100hz = (unsigned int)(temp / 100);
+ temp = clock_hz * dp_dto_ref_khz * 10;
+ ASSERT(temp <= UINT_MAX * modulo_hz * 100ULL);
+ *pixel_clk_100hz = div_u64(temp, modulo_hz * 100);
} else
*pixel_clk_100hz = 0;
} else {
* - target pix_clk_hz = (DPDTO INTEGER * DPDTO MODULO + DPDTO PHASE)
*/
temp = (unsigned long long)dp_dto_integer * modulo_hz + phase_hz;
-
- if (temp / 100 > 0xFFFFFFFFUL) {
+ if (temp > (UINT_MAX * 100ULL)) {
/* pixel rate 100hz should never be this high, if it is, throw an assert and return 0 */
BREAK_TO_DEBUGGER();
*pixel_clk_100hz = 0;
} else {
- *pixel_clk_100hz = (unsigned int)(temp / 100);
+ *pixel_clk_100hz = div_u64(temp, 100);
}
return true;
link->dpcd_caps.sink_count.bits.SINK_COUNT = 1;
/* NUTMEG requires that we use HBR, doesn't work with RBR. */
if (link->dpcd_caps.branch_dev_id == DP_BRANCH_DEVICE_ID_00001A)
- link->preferred_link_setting.link_rate = LINK_RATE_HIGH;
+ link->wa_flags.dp_skip_rbr = true;
}
return true;
if (req_bw > dp_link_bandwidth_kbps(link, &link->verified_link_cap))
return false;
- if (link->preferred_link_setting.link_rate != LINK_RATE_UNKNOWN)
- initial_link_setting.link_rate = link->preferred_link_setting.link_rate;
+ if (link->wa_flags.dp_skip_rbr) {
+ initial_link_setting.link_rate = LINK_RATE_HIGH;
+ current_link_setting.link_rate = LINK_RATE_HIGH;
+ }
/* search for the minimum link setting that:
* 1. is supported according to the link training result
for (i = 0; i < pool->stream_enc_count; i++) {
if (!res_ctx->is_stream_enc_acquired[i] &&
pool->stream_enc[i]) {
+ /* DP/MST needs a digital encoder; skip analog/no-DP encoders */
+ if (dc_is_dp_signal(stream->signal) &&
+ (!pool->stream_enc[i]->funcs ||
+ !pool->stream_enc[i]->funcs->dp_set_stream_attribute))
+ continue;
/* Store first available for MST second display
* in daisy chain use case
*/
* required for non DP connectors.
*/
- if (j >= 0 && link->connector_signal == SIGNAL_TYPE_DISPLAY_PORT)
+ if (j >= 0 && dc_is_dp_signal(stream->signal))
return pool->stream_enc[j];
return NULL;
dc->config.use_pipe_ctx_sync_logic = true;
dc->config.dc_mode_clk_limit_support = false;
dc->config.enable_windowed_mpo_odm = true;
+ dc->config.set_pipe_unlock_order = true; /* Need to ensure DET gets freed before allocating */
/* Use psp mailbox to enable assr */
dc->config.use_assr_psp_message = true;
/* dcn42 and afterward always support external panel replay */
#include "dcn35/dcn35_resource.h"
#include "dcn321/dcn321_resource.h"
#include "dcn401/dcn401_resource.h"
+#include "dcn42/dcn42_resource.h"
#include "dcn42/dcn42_resource_fpu.h"
#include "dcn10/dcn10_ipp.h"
#define regAPG9_APG_DBG_GEN_CONTROL 0x38ae
#define regAPG9_APG_DBG_GEN_CONTROL_BASE_IDX 2
+#define regHUBP0_HUBPREQ_DEBUG_DB 0x05f8
+#define regHUBP0_HUBPREQ_DEBUG_DB_BASE_IDX 2
+#define regHUBP0_HUBPREQ_DEBUG 0x05f9
+#define regHUBP0_HUBPREQ_DEBUG_BASE_IDX 2
+#define regHUBP1_HUBPREQ_DEBUG_DB 0x06d4
+#define regHUBP1_HUBPREQ_DEBUG_DB_BASE_IDX 2
+#define regHUBP1_HUBPREQ_DEBUG 0x06d5
+#define regHUBP1_HUBPREQ_DEBUG_BASE_IDX 2
+#define regHUBP2_HUBPREQ_DEBUG_DB 0x07b0
+#define regHUBP2_HUBPREQ_DEBUG_DB_BASE_IDX 2
+#define regHUBP2_HUBPREQ_DEBUG 0x07b1
+#define regHUBP2_HUBPREQ_DEBUG_BASE_IDX 2
+#define regHUBP3_HUBPREQ_DEBUG_DB 0x088c
+#define regHUBP3_HUBPREQ_DEBUG_DB_BASE_IDX 2
+#define regHUBP3_HUBPREQ_DEBUG 0x088d
+#define regHUBP3_HUBPREQ_DEBUG_BASE_IDX 2
+
enum dcn401_clk_src_array_id {
DCN401_CLK_SRC_PLL0,
DCN401_CLK_SRC_PLL1,
OPTC_COMMON_MASK_SH_LIST_DCN42B(_MASK)};
#define hubp_regs_init(id) \
- HUBP_REG_LIST_DCN42B_RI(id)
+ HUBP_REG_LIST_DCN42_RI(id)
static struct dcn_hubp2_registers hubp_regs[4];
.update_soc_for_wm_a = dcn30_update_soc_for_wm_a,
.add_phantom_pipes = dcn32_add_phantom_pipes,
.calculate_mall_ways_from_bytes = dcn32_calculate_mall_ways_from_bytes,
-#ifdef CONFIG_DRM_AMD_DC_DML21
.prepare_mcache_programming = dcn42b_prepare_mcache_programming,
-#endif
.build_pipe_pix_clk_params = dcn42b_build_pipe_pix_clk_params,
.get_power_profile = dcn401_get_power_profile,
.get_vstartup_for_pipe = dcn401_get_vstartup_for_pipe,
dc->config.use_pipe_ctx_sync_logic = true;
dc->config.dc_mode_clk_limit_support = false;
dc->config.enable_windowed_mpo_odm = true;
+ dc->config.set_pipe_unlock_order = true; /* Need to ensure DET gets freed before allocating */
/* Use psp mailbox to enable assr */
dc->config.use_assr_psp_message = true;
/* dcn42 and afterward always support external panel replay */
* DCCG_SRII(PHASE, DP_DTO, 3),
* DCCG_SRII(MODULO, DP_DTO, 3),
* SR(DSCCLK3_DTO_PARAM),
- * SR(HDMISTREAMCLK_CNTL),
* SR(SYMCLKD_CLOCK_ENABLE),
* SR(SYMCLKE_CLOCK_ENABLE)
*/
SR(PHYBSYMCLK_CLOCK_CNTL), \
SR(PHYCSYMCLK_CLOCK_CNTL), \
SR(DPSTREAMCLK_CNTL), \
+ SR(HDMISTREAMCLK_CNTL), \
SR(SYMCLK32_SE_CNTL), \
SR(SYMCLK32_LE_CNTL), \
DCCG_SRII(PIXEL_RATE_CNTL, OTG, 0), \
SRI_ARR(DC_ABM1_ACE_OFFSET_SLOPE_DATA, ABM, id), \
SRI_ARR(DC_ABM1_ACE_PWL_CNTL, ABM, id)
-/* HUBP */
-/* Not in DCN42B: HUBPREQ_DEBUG_DB and HUBPREQ_DEBUG */
-#define HUBP_REG_LIST_DCN42B_RI(id) \
- SRI_ARR(DCN_DMDATA_VM_CNTL, HUBPREQ, id), \
- SRI_ARR(FLIP_PARAMETERS_3, HUBPREQ, id), \
- SRI_ARR(FLIP_PARAMETERS_4, HUBPREQ, id), \
- SRI_ARR(FLIP_PARAMETERS_5, HUBPREQ, id), \
- SRI_ARR(FLIP_PARAMETERS_6, HUBPREQ, id), \
- SRI_ARR(VBLANK_PARAMETERS_5, HUBPREQ, id), \
- SRI_ARR(VBLANK_PARAMETERS_6, HUBPREQ, id), \
- HUBP_REG_LIST_DCN_VM_RI(id), \
- SRI_ARR(PREFETCH_SETTINGS, HUBPREQ, id), \
- SRI_ARR(PREFETCH_SETTINGS_C, HUBPREQ, id), \
- SRI_ARR(DCN_VM_SYSTEM_APERTURE_LOW_ADDR, HUBPREQ, id), \
- SRI_ARR(DCN_VM_SYSTEM_APERTURE_HIGH_ADDR, HUBPREQ, id), \
- SRI_ARR(CURSOR_SETTINGS, HUBPREQ, id), \
- SRI_ARR(CURSOR_SURFACE_ADDRESS_HIGH, CURSOR0_, id), \
- SRI_ARR(CURSOR_SURFACE_ADDRESS, CURSOR0_, id), \
- SRI_ARR(CURSOR_SIZE, CURSOR0_, id), \
- SRI_ARR(CURSOR_CONTROL, CURSOR0_, id), \
- SRI_ARR(CURSOR_POSITION, CURSOR0_, id), \
- SRI_ARR(CURSOR_HOT_SPOT, CURSOR0_, id), \
- SRI_ARR(CURSOR_DST_OFFSET, CURSOR0_, id), \
- SRI_ARR(DMDATA_ADDRESS_HIGH, CURSOR0_, id), \
- SRI_ARR(DMDATA_ADDRESS_LOW, CURSOR0_, id), \
- SRI_ARR(DMDATA_CNTL, CURSOR0_, id), \
- SRI_ARR(DMDATA_SW_CNTL, CURSOR0_, id), \
- SRI_ARR(DMDATA_QOS_CNTL, CURSOR0_, id), \
- SRI_ARR(DMDATA_SW_DATA, CURSOR0_, id), \
- SRI_ARR(DMDATA_STATUS, CURSOR0_, id), \
- SRI_ARR(FLIP_PARAMETERS_0, HUBPREQ, id), \
- SRI_ARR(FLIP_PARAMETERS_1, HUBPREQ, id), \
- SRI_ARR(FLIP_PARAMETERS_2, HUBPREQ, id), \
- SRI_ARR(DCN_CUR1_TTU_CNTL0, HUBPREQ, id), \
- SRI_ARR(DCN_CUR1_TTU_CNTL1, HUBPREQ, id), \
- SRI_ARR(DCSURF_FLIP_CONTROL2, HUBPREQ, id), \
- SRI_ARR(VMID_SETTINGS_0, HUBPREQ, id), \
- SRI_ARR(DCHUBP_CNTL, HUBP, id), \
- SRI_ARR(DCSURF_ADDR_CONFIG, HUBP, id), \
- SRI_ARR(DCSURF_TILING_CONFIG, HUBP, id), \
- SRI_ARR(DCSURF_SURFACE_PITCH, HUBPREQ, id), \
- SRI_ARR(DCSURF_SURFACE_PITCH_C, HUBPREQ, id), \
- SRI_ARR(DCSURF_SURFACE_CONFIG, HUBP, id), \
- SRI_ARR(DCSURF_FLIP_CONTROL, HUBPREQ, id), \
- SRI_ARR(DCSURF_PRI_VIEWPORT_DIMENSION, HUBP, id), \
- SRI_ARR(DCSURF_PRI_VIEWPORT_START, HUBP, id), \
- SRI_ARR(DCSURF_SEC_VIEWPORT_DIMENSION, HUBP, id), \
- SRI_ARR(DCSURF_SEC_VIEWPORT_START, HUBP, id), \
- SRI_ARR(DCSURF_PRI_VIEWPORT_DIMENSION_C, HUBP, id), \
- SRI_ARR(DCSURF_PRI_VIEWPORT_START_C, HUBP, id), \
- SRI_ARR(DCSURF_SEC_VIEWPORT_DIMENSION_C, HUBP, id), \
- SRI_ARR(DCSURF_SEC_VIEWPORT_START_C, HUBP, id), \
- SRI_ARR(DCSURF_PRIMARY_SURFACE_ADDRESS_HIGH, HUBPREQ, id), \
- SRI_ARR(DCSURF_PRIMARY_SURFACE_ADDRESS, HUBPREQ, id), \
- SRI_ARR(DCSURF_SECONDARY_SURFACE_ADDRESS_HIGH, HUBPREQ, id), \
- SRI_ARR(DCSURF_SECONDARY_SURFACE_ADDRESS, HUBPREQ, id), \
- SRI_ARR(DCSURF_PRIMARY_META_SURFACE_ADDRESS_HIGH, HUBPREQ, id), \
- SRI_ARR(DCSURF_PRIMARY_META_SURFACE_ADDRESS, HUBPREQ, id), \
- SRI_ARR(DCSURF_SECONDARY_META_SURFACE_ADDRESS_HIGH, HUBPREQ, id), \
- SRI_ARR(DCSURF_SECONDARY_META_SURFACE_ADDRESS, HUBPREQ, id), \
- SRI_ARR(DCSURF_PRIMARY_SURFACE_ADDRESS_HIGH_C, HUBPREQ, id), \
- SRI_ARR(DCSURF_PRIMARY_SURFACE_ADDRESS_C, HUBPREQ, id), \
- SRI_ARR(DCSURF_SECONDARY_SURFACE_ADDRESS_HIGH_C, HUBPREQ, id), \
- SRI_ARR(DCSURF_SECONDARY_SURFACE_ADDRESS_C, HUBPREQ, id), \
- SRI_ARR(DCSURF_PRIMARY_META_SURFACE_ADDRESS_HIGH_C, HUBPREQ, id), \
- SRI_ARR(DCSURF_PRIMARY_META_SURFACE_ADDRESS_C, HUBPREQ, id), \
- SRI_ARR(DCSURF_SECONDARY_META_SURFACE_ADDRESS_HIGH_C, HUBPREQ, id), \
- SRI_ARR(DCSURF_SECONDARY_META_SURFACE_ADDRESS_C, HUBPREQ, id), \
- SRI_ARR(DCSURF_SURFACE_INUSE, HUBPREQ, id), \
- SRI_ARR(DCSURF_SURFACE_INUSE_HIGH, HUBPREQ, id), \
- SRI_ARR(DCSURF_SURFACE_INUSE_C, HUBPREQ, id), \
- SRI_ARR(DCSURF_SURFACE_INUSE_HIGH_C, HUBPREQ, id), \
- SRI_ARR(DCSURF_SURFACE_EARLIEST_INUSE, HUBPREQ, id), \
- SRI_ARR(DCSURF_SURFACE_EARLIEST_INUSE_HIGH, HUBPREQ, id), \
- SRI_ARR(DCSURF_SURFACE_EARLIEST_INUSE_C, HUBPREQ, id), \
- SRI_ARR(DCSURF_SURFACE_EARLIEST_INUSE_HIGH_C, HUBPREQ, id), \
- SRI_ARR(DCSURF_SURFACE_CONTROL, HUBPREQ, id), \
- SRI_ARR(DCSURF_SURFACE_FLIP_INTERRUPT, HUBPREQ, id), \
- SRI_ARR(HUBPRET_CONTROL, HUBPRET, id), \
- SRI_ARR(HUBPRET_READ_LINE_STATUS, HUBPRET, id), \
- SRI_ARR(DCN_EXPANSION_MODE, HUBPREQ, id), \
- SRI_ARR(DCHUBP_REQ_SIZE_CONFIG, HUBP, id), \
- SRI_ARR(DCHUBP_REQ_SIZE_CONFIG_C, HUBP, id), \
- SRI_ARR(BLANK_OFFSET_0, HUBPREQ, id), \
- SRI_ARR(BLANK_OFFSET_1, HUBPREQ, id), \
- SRI_ARR(DST_DIMENSIONS, HUBPREQ, id), \
- SRI_ARR(DST_AFTER_SCALER, HUBPREQ, id), \
- SRI_ARR(VBLANK_PARAMETERS_0, HUBPREQ, id), \
- SRI_ARR(REF_FREQ_TO_PIX_FREQ, HUBPREQ, id), \
- SRI_ARR(VBLANK_PARAMETERS_1, HUBPREQ, id), \
- SRI_ARR(VBLANK_PARAMETERS_3, HUBPREQ, id), \
- SRI_ARR(NOM_PARAMETERS_4, HUBPREQ, id), \
- SRI_ARR(NOM_PARAMETERS_5, HUBPREQ, id), \
- SRI_ARR(PER_LINE_DELIVERY_PRE, HUBPREQ, id), \
- SRI_ARR(PER_LINE_DELIVERY, HUBPREQ, id), \
- SRI_ARR(VBLANK_PARAMETERS_2, HUBPREQ, id), \
- SRI_ARR(VBLANK_PARAMETERS_4, HUBPREQ, id), \
- SRI_ARR(NOM_PARAMETERS_6, HUBPREQ, id), \
- SRI_ARR(NOM_PARAMETERS_7, HUBPREQ, id), \
- SRI_ARR(DCN_TTU_QOS_WM, HUBPREQ, id), \
- SRI_ARR(DCN_GLOBAL_TTU_CNTL, HUBPREQ, id), \
- SRI_ARR(DCN_SURF0_TTU_CNTL0, HUBPREQ, id), \
- SRI_ARR(DCN_SURF0_TTU_CNTL1, HUBPREQ, id), \
- SRI_ARR(DCN_SURF1_TTU_CNTL0, HUBPREQ, id), \
- SRI_ARR(DCN_SURF1_TTU_CNTL1, HUBPREQ, id), \
- SRI_ARR(DCN_CUR0_TTU_CNTL0, HUBPREQ, id), \
- SRI_ARR(DCN_CUR0_TTU_CNTL1, HUBPREQ, id), \
- SRI_ARR(HUBP_CLK_CNTL, HUBP, id), \
- SRI_ARR(HUBPRET_READ_LINE_VALUE, HUBPRET, id), \
- SRI_ARR(DCHUBP_MALL_CONFIG, HUBP, id), \
- SRI_ARR(DCHUBP_VMPG_CONFIG, HUBP, id), \
- SRI_ARR(UCLK_PSTATE_FORCE, HUBPREQ, id), \
- SRI_ARR(HUBP_3DLUT_DLG_PARAM, CURSOR0_, id), \
- HUBP_3DLUT_FL_REG_LIST_DCN401(id)
struct dcn42b_resource_pool {
struct resource_pool base;
};
{
struct amdgpu_device *adev = (struct amdgpu_device *)handle;
- /* Check if the platform already manages the AC/DC switch via dedicated GPIO. */
- if (adev->pm.dpm.platform_caps & ATOM_PP_PLATFORM_CAP_HARDWAREDC)
- return;
-
- /* The SMU automatically notices DC, but needs to be notified when switching to AC. */
- if (adev->pm.ac_power)
+ /*
+ * Check if the platform already manages the AC/DC switch via dedicated GPIO.
+ * Otherwise SMU automatically notices DC, but needs to be notified of AC.
+ */
+ if (adev->pm.ac_power &&
+ (adev->pm.dpm.platform_caps & ATOM_PP_PLATFORM_CAP_HARDWAREDC))
amdgpu_si_send_msg_to_smc(adev, PPSMC_MSG_RunningOnAC);
+
+ /* Recompute clocks with updated max_limits. */
+ amdgpu_legacy_dpm_compute_clocks(adev);
}
static PPSMC_Result si_send_msg_to_smc_with_parameter(struct amdgpu_device *adev,
break;
}
- if (queue_thermal)
+ if (queue_thermal && amdgpu_dpm)
schedule_work(&adev->pm.dpm.thermal.work);
return 0;
hwmgr->od_enabled = false;
switch (hwmgr->chip_id) {
case CHIP_BONAIRE:
- /* R9 M380 in iMac 2015: SMU hangs when enabling MCLK DPM
- * R7 260X cards with old MC ucode: MCLK DPM is unstable
- */
- if (adev->pdev->subsystem_vendor == 0x106B ||
- adev->pdev->device == 0x6658) {
+ /* R9 M380 in iMac 2015: SMU hangs when enabling MCLK DPM */
+ if (adev->pdev->subsystem_vendor == 0x106B) {
dev_info(adev->dev, "disabling MCLK DPM on quirky ASIC");
adev->pm.pp_feature &= ~PP_MCLK_DPM_MASK;
hwmgr->feature_mask &= ~PP_MCLK_DPM_MASK;
static void smu7_notify_ac_dc(struct pp_hwmgr *hwmgr)
{
struct amdgpu_device *adev = (struct amdgpu_device *)(hwmgr->adev);
+ const struct amd_pm_funcs *pp_funcs = adev->powerplay.pp_funcs;
- /* Check if the platform already manages the AC/DC switch via dedicated GPIO. */
- if (phm_cap_enabled(hwmgr->platform_descriptor.platformCaps,
+ /*
+ * Check if the platform already manages the AC/DC switch via dedicated GPIO.
+ * Otherwise SMU automatically notices DC, but needs to be notified of AC.
+ */
+ if (adev->pm.ac_power &&
+ phm_cap_enabled(hwmgr->platform_descriptor.platformCaps,
PHM_PlatformCaps_AutomaticDCTransition))
- return;
-
- /* The SMU automatically notices DC, but needs to be notified when switching to AC. */
- if (adev->pm.ac_power)
smum_send_msg_to_smc(hwmgr, PPSMC_MSG_RunningOnAC, NULL);
+
+ /* Recompute clocks with updated max_limits. */
+ pp_funcs->pm_compute_clocks(adev->powerplay.pp_handle);
}
static const struct pp_hwmgr_func smu7_hwmgr_funcs = {
break;
case IP_VERSION(15, 0, 0):
case IP_VERSION(15, 0, 5):
+ case IP_VERSION(15, 0, 9):
smu_v15_0_0_set_ppt_funcs(smu);
break;
case IP_VERSION(15, 0, 8):
bitmap_zero(fea_cap->cap_map, SMU_FEATURE_CAP_ID__COUNT);
}
+static int smu_set_power_dep(struct smu_context *smu, bool enable)
+{
+ if (!smu->ppt_funcs->set_power_dep)
+ return 0;
+
+ return smu->ppt_funcs->set_power_dep(smu, enable);
+}
+
static int smu_sw_init(struct amdgpu_ip_block *ip_block)
{
struct amdgpu_device *adev = ip_block->adev;
if (!smu->ppt_funcs->get_fan_control_mode)
smu->adev->pm.no_fan = true;
+ smu_set_power_dep(smu, true);
+
return 0;
}
smu_fini_microcode(smu);
+ smu_set_power_dep(smu, false);
+
return 0;
}
bool pm_enabled;
bool is_apu;
+ /* Power dependency link from an integrated xHCI controller to the GPU */
+ struct device_link *usb_power_link;
+
uint32_t smc_driver_if_version;
uint32_t smc_fw_if_version;
uint32_t smc_fw_version;
int (*ras_send_msg)(struct smu_context *smu,
enum smu_message_type msg, uint32_t param, uint32_t *read_arg);
-
/**
* @get_ras_smu_drv: Get RAS smu driver interface
* Return: ras_smu_drv *
*/
int (*get_ras_smu_drv)(struct smu_context *smu, const struct ras_smu_drv **ras_smu_drv);
+
+ /**
+ * @set_power_dep: Create or destroy a power dependency link
+ * from an integrated xHCI controller to the GPU so that the GPU is
+ * resumed before the USB controller during PM resume. @enable is true
+ * to create the link and false to tear it down.
+ */
+ int (*set_power_dep)(struct smu_context *smu, bool enable);
};
typedef enum {
return 0;
}
+/*
+ * Link any xHCI controller sharing the GPU's PCIe root port as a consumer
+ * of the GPU so the GPU resumes first, avoiding an xHCI resume race.
+ */
+static int smu_v14_0_0_set_power_dep(struct smu_context *smu, bool enable)
+{
+ struct amdgpu_device *adev = smu->adev;
+ struct pci_dev *gpu_pdev = adev->pdev;
+ struct pci_dev *root_port, *usb_pdev = NULL;
+ struct device_link *link;
+
+ if (!enable) {
+ if (smu->usb_power_link) {
+ device_link_del(smu->usb_power_link);
+ smu->usb_power_link = NULL;
+ }
+ return 0;
+ }
+
+ root_port = pcie_find_root_port(gpu_pdev);
+ while ((usb_pdev = pci_get_class(PCI_CLASS_SERIAL_USB_XHCI, usb_pdev))) {
+ struct pci_dev *usb_root;
+
+ usb_root = pcie_find_root_port(usb_pdev);
+ if (usb_root != root_port)
+ continue;
+
+ /* Create device link: USB (consumer) depends on GPU (supplier) */
+ link = device_link_add(&usb_pdev->dev, &gpu_pdev->dev,
+ DL_FLAG_STATELESS | DL_FLAG_PM_RUNTIME);
+ if (link) {
+ smu->usb_power_link = link;
+ drm_info(adev_to_drm(adev), "USB controller %s D0 power state depends on %s\n",
+ pci_name(usb_pdev), pci_name(gpu_pdev));
+ /* Only create one link for the first USB controller found */
+ break;
+ }
+ }
+
+ pci_dev_put(usb_pdev);
+
+ return 0;
+}
+
static const struct pptable_funcs smu_v14_0_0_ppt_funcs = {
.check_fw_status = smu_v14_0_check_fw_status,
.check_fw_version = smu_cmn_check_fw_version,
.dpm_set_umsch_mm_enable = smu_v14_0_0_set_umsch_mm_enable,
.get_dpm_clock_table = smu_v14_0_common_get_dpm_table,
.set_mall_enable = smu_v14_0_common_set_mall_enable,
+ .set_power_dep = smu_v14_0_0_set_power_dep,
};
static void smu_v14_0_0_init_msg_ctl(struct smu_context *smu)
table_context->power_play_table;
PPTable_t *pptable = table_context->driver_pptable;
CustomSkuTable_t *skutable = &pptable->CustomSkuTable;
- int16_t od_percent_upper = 0, od_percent_lower = 0;
+ uint32_t pp_limit = smu->adev->pm.ac_power ?
+ skutable->SocketPowerLimitAc[PPT_THROTTLER_PPT0] :
+ skutable->SocketPowerLimitDc[PPT_THROTTLER_PPT0];
uint32_t msg_limit = pptable->SkuTable.MsgLimits.Power[PPT_THROTTLER_PPT0][POWER_SOURCE_AC];
- uint32_t power_limit;
+ uint32_t min_limit = min_t(uint32_t, pp_limit, msg_limit);
+ uint32_t max_limit = max_t(uint32_t, pp_limit, msg_limit);
+ int16_t od_percent_upper = 0, od_percent_lower = 0;
+ int ret;
- if (smu_v14_0_get_current_power_limit(smu, &power_limit))
- power_limit = smu->adev->pm.ac_power ?
- skutable->SocketPowerLimitAc[PPT_THROTTLER_PPT0] :
- skutable->SocketPowerLimitDc[PPT_THROTTLER_PPT0];
+ if (current_power_limit) {
+ ret = smu_v14_0_get_current_power_limit(smu, current_power_limit);
+ if (ret)
+ *current_power_limit = pp_limit;
+ }
- if (current_power_limit)
- *current_power_limit = power_limit;
if (default_power_limit)
- *default_power_limit = power_limit;
+ *default_power_limit = pp_limit;
if (powerplay_table) {
if (smu->od_enabled &&
}
dev_dbg(smu->adev->dev, "od percent upper:%d, od percent lower:%d (default power: %d)\n",
- od_percent_upper, od_percent_lower, power_limit);
+ od_percent_upper, od_percent_lower, pp_limit);
if (max_power_limit) {
- *max_power_limit = msg_limit * (100 + od_percent_upper);
+ *max_power_limit = max_limit * (100 + od_percent_upper);
*max_power_limit /= 100;
}
if (min_power_limit) {
- *min_power_limit = power_limit * (100 + od_percent_lower);
+ *min_power_limit = min_limit * (100 + od_percent_lower);
*min_power_limit /= 100;
}
switch (amdgpu_ip_version(adev, MP1_HWIP, 0)) {
case IP_VERSION(15, 0, 0):
+ case IP_VERSION(15, 0, 9):
if (!(adev->pm.pp_feature & PP_GFXOFF_MASK))
return 0;
if (enable)
smu_v15_0_common_get_dpm_ultimate_freq(smu, SMU_SOCCLK, NULL, &clk_limit);
break;
case SMU_FCLK:
- if (amdgpu_ip_version(smu->adev, MP1_HWIP, 0) == IP_VERSION(15, 0, 0))
+ if (amdgpu_ip_version(smu->adev, MP1_HWIP, 0) == IP_VERSION(15, 0, 0) ||
+ amdgpu_ip_version(smu->adev, MP1_HWIP, 0) == IP_VERSION(15, 0, 9))
smu_v15_0_common_get_dpm_ultimate_freq(smu, SMU_FCLK, NULL, &clk_limit);
else
clk_limit = SMU_15_0_UMD_PSTATE_FCLK;
lane_count = dp->link_train.lane_count;
for (lane = 0; lane < lane_count; lane++) {
voltage_swing = drm_dp_get_adjust_request_voltage(link_status, lane);
+ voltage_swing >>= DP_TRAIN_VOLTAGE_SWING_SHIFT;
pre_emphasis = drm_dp_get_adjust_request_pre_emphasis(link_status, lane);
+ pre_emphasis >>= DP_TRAIN_PRE_EMPHASIS_SHIFT;
training_lane = DPCD_VOLTAGE_SWING_SET(voltage_swing) |
DPCD_PRE_EMPHASIS_SET(pre_emphasis);
for (lane = 0; lane < lane_count; lane++) {
training_lane = analogix_dp_get_lane_link_training(dp, lane);
voltage_swing = drm_dp_get_adjust_request_voltage(link_status, lane);
+ voltage_swing >>= DP_TRAIN_VOLTAGE_SWING_SHIFT;
pre_emphasis = drm_dp_get_adjust_request_pre_emphasis(link_status, lane);
+ pre_emphasis >>= DP_TRAIN_PRE_EMPHASIS_SHIFT;
if (DPCD_VOLTAGE_SWING_GET(training_lane) == voltage_swing &&
DPCD_PRE_EMPHASIS_GET(training_lane) == pre_emphasis)
console_unlock();
}
+static int find_crtc_index_atomic(struct drm_fb_helper *helper)
+{
+ struct drm_device *dev = helper->dev;
+ int crtc_index = -EINVAL;
+ struct drm_modeset_acquire_ctx ctx;
+ struct drm_plane *plane;
+ int ret = 0;
+
+ drm_modeset_acquire_init(&ctx, 0);
+
+retry:
+ drm_for_each_plane(plane, dev) {
+ const struct drm_plane_state *plane_state;
+
+ if (plane->type != DRM_PLANE_TYPE_PRIMARY)
+ continue;
+
+ ret = drm_modeset_lock(&plane->mutex, &ctx);
+ if (ret)
+ goto err_drm_modeset_lock;
+ plane_state = plane->state;
+
+ if (plane_state->fb == helper->fb && plane_state->crtc) {
+ struct drm_crtc *crtc = plane_state->crtc;
+
+ ret = drm_modeset_lock(&crtc->mutex, &ctx);
+ if (ret)
+ goto err_drm_modeset_lock;
+ if (crtc->state->active)
+ crtc_index = crtc->index;
+ drm_modeset_unlock(&crtc->mutex);
+ }
+ drm_modeset_unlock(&plane->mutex);
+
+ if (crtc_index >= 0)
+ break;
+ }
+
+ drm_modeset_drop_locks(&ctx);
+ drm_modeset_acquire_fini(&ctx);
+
+ return crtc_index;
+
+err_drm_modeset_lock:
+ if (ret == -EDEADLK) {
+ drm_modeset_backoff(&ctx);
+ goto retry;
+ }
+ return ret;
+}
+
+static int find_crtc_index_legacy(struct drm_fb_helper *helper)
+{
+ struct drm_device *dev = helper->dev;
+ struct drm_crtc *crtc;
+
+ drm_for_each_crtc(crtc, dev) {
+ struct drm_plane *plane = crtc->primary;
+
+ if (!crtc->enabled)
+ continue;
+ if (!plane || plane->fb != helper->fb)
+ continue; /* CRTC doesn't display fbdev emulation */
+
+ return crtc->index;
+ }
+
+ return -EINVAL;
+}
+
+static int drm_fb_helper_find_crtc_index(struct drm_fb_helper *helper)
+{
+ struct drm_device *dev = helper->dev;
+ int crtc_index;
+
+ mutex_lock(&dev->mode_config.mutex);
+
+ if (drm_drv_uses_atomic_modeset(dev))
+ crtc_index = find_crtc_index_atomic(helper);
+ else
+ crtc_index = find_crtc_index_legacy(helper);
+
+ mutex_unlock(&dev->mode_config.mutex);
+
+ return crtc_index;
+}
+
static void drm_fb_helper_fb_dirty(struct drm_fb_helper *helper)
{
struct drm_device *dev = helper->dev;
struct drm_clip_rect *clip = &helper->damage_clip;
struct drm_clip_rect clip_copy;
+ int crtc_index;
unsigned long flags;
int ret;
mutex_lock(&helper->lock);
- drm_client_modeset_wait_for_vblank(&helper->client, 0);
+ crtc_index = drm_fb_helper_find_crtc_index(helper);
+ if (crtc_index >= 0)
+ drm_client_modeset_wait_for_vblank(&helper->client, crtc_index);
mutex_unlock(&helper->lock);
if (drm_WARN_ON_ONCE(dev, !helper->funcs->fb_dirty))
};
bool use_iova = dma_use_iova(&svm_pages->state);
- if (use_iova)
- dma_iova_destroy(dev, &svm_pages->state,
- svm_pages->state_offset,
- svm_pages->dma_addr[0].dir, 0);
+ /*
+ * IOVA is reserved for the whole range but only the linked
+ * system pages (state_offset bytes) need unlinking; free the
+ * entire reservation to avoid leaking the device-page part.
+ * On the error path state_offset is 0, so just free it.
+ */
+ if (use_iova) {
+ if (svm_pages->state_offset)
+ dma_iova_unlink(dev, &svm_pages->state, 0,
+ svm_pages->state_offset,
+ svm_pages->dma_addr[0].dir, 0);
+ dma_iova_free(dev, &svm_pages->state);
+ }
for (i = 0, j = 0; i < npages; j++) {
struct drm_pagemap_addr *addr = &svm_pages->dma_addr[j];
- if (!use_iova && addr->proto == DRM_INTERCONNECT_SYSTEM)
- dma_unmap_page(dev,
- addr->addr,
- PAGE_SIZE << addr->order,
- addr->dir);
- else if (dpagemap && dpagemap->ops->device_unmap)
+ if (addr->proto == DRM_INTERCONNECT_SYSTEM) {
+ /*
+ * Linked IOVA pages were already torn down by
+ * the dma_iova_unlink()/dma_iova_free() above;
+ * only the non-IOVA mappings need unmap here.
+ */
+ if (!use_iova)
+ dma_unmap_page(dev,
+ addr->addr,
+ PAGE_SIZE << addr->order,
+ addr->dir);
+ } else if (dpagemap && dpagemap->ops->device_unmap)
dpagemap->ops->device_unmap(dpagemap,
dev, addr);
i += 1 << addr->order;
/* Unlock and restart mapping to allocate memory. */
drm_gpusvm_notifier_unlock(gpusvm);
svm_pages->dma_addr =
- kvmalloc_objs(*svm_pages->dma_addr, npages);
+ kvzalloc_objs(*svm_pages->dma_addr, npages);
if (!svm_pages->dma_addr) {
err = -ENOMEM;
goto err_free;
err = -EAGAIN;
goto err_unmap;
}
+
+ /*
+ * Set the dpagemap as soon as the first
+ * device page is mapped so the err_unmap path
+ * can device_unmap() the device mappings that
+ * have already been created.
+ */
+ drm_pagemap_get(dpagemap);
+ drm_pagemap_put(svm_pages->dpagemap);
+ svm_pages->dpagemap = dpagemap;
}
svm_pages->dma_addr[j] =
dpagemap->ops->device_map(dpagemap,
goto err_unmap;
}
- if (pagemap) {
+ if (pagemap)
flags.has_devmem_pages = true;
- drm_pagemap_get(dpagemap);
- drm_pagemap_put(svm_pages->dpagemap);
- svm_pages->dpagemap = dpagemap;
- }
/* WRITE_ONCE pairs with READ_ONCE for opportunistic checks */
WRITE_ONCE(svm_pages->flags.__flags, flags.__flags);
return -EINVAL;
file_priv->supports_virtualized_cursor_plane = req->value;
break;
- case DRM_CLIENT_CAP_PLANE_COLOR_PIPELINE:
+ case DRM_CLIENT_CAP_PLANE_COLOR_PIPELINE: {
+ struct drm_plane *plane;
+ bool has_plane_with_color_pipeline = false;
+
if (!file_priv->atomic)
return -EINVAL;
if (req->value > 1)
return -EINVAL;
+ drm_for_each_plane(plane, dev) {
+ if (plane->color_pipeline_property) {
+ has_plane_with_color_pipeline = true;
+ break;
+ }
+ }
+ if (!has_plane_with_color_pipeline)
+ return -EOPNOTSUPP;
file_priv->plane_color_pipeline = req->value;
break;
+ }
default:
return -EINVAL;
}
};
static const struct drm_get_panel_backlight_quirk drm_panel_min_backlight_quirks[] = {
+ /* Lenovo Legion 5 15ARH05, AUX backlight non-functional, force PWM */
+ {
+ .dmi_match.field = DMI_SYS_VENDOR,
+ .dmi_match.value = "LENOVO",
+ .dmi_match_other.field = DMI_PRODUCT_VERSION,
+ .dmi_match_other.value = "Lenovo Legion 5 15ARH05",
+ .ident.panel_id = drm_edid_encode_panel_id('B', 'O', 'E', 0x08df),
+ .quirk = { .force_pwm = true, },
+ },
/* 13 inch matte panel */
{
.dmi_match.field = DMI_BOARD_VENDOR,
struct intel_display *display = to_intel_display(intel_dp);
bool force_retrain = intel_dp->link.force_retrain;
bool reprobe_needed = false;
+ int tries = 33;
- for (;;) {
+ while (--tries) {
u8 esi[4] = {};
u8 ack[4] = {};
bool new_irqs;
break;
}
+ if (!tries) {
+ drm_dbg_kms(display->drm, "DPRX ESI not clearing, device may be stuck\n");
+ reprobe_needed = true;
+ }
+
return !reprobe_needed;
}
else
val |= XELPDP_DDI_CLOCK_SELECT_PREP(display, XELPDP_DDI_CLOCK_SELECT_MAXPCLK);
- /* DP2.0 10G and 20G rates enable MPLLA*/
- if (port_clock == 1000000 || port_clock == 2000000)
- val |= XELPDP_SSC_ENABLE_PLLA;
- else
- val |= ltpll->ssc_enabled ? XELPDP_SSC_ENABLE_PLLB : 0;
+ val |= ltpll->ssc_enabled ? XELPDP_SSC_ENABLE_PLLA : 0;
intel_de_rmw(display, XELPDP_PORT_CLOCK_CTL(display, encoder->port),
XELPDP_LANE1_PHY_CLOCK_SELECT | XELPDP_FORWARD_CLOCK_UNGATE |
needs_panel_replay)
return 0;
- if (intel_vrr_always_use_vrr_tg(display))
- return 0;
-
return 1;
}
return 0;
}
-
-/* Divide a U16.16 fixed-point value by 2, staying in fixed-point domain */
-static inline u32 fp_16_16_div2(u32 fp)
-{
- return fp >> 1;
-}
-
-/* Convert a U16.16 fixed-point value to integer, rounding up */
-static inline int fp_16_16_to_int_ceil(u32 fp)
-{
- return DIV_ROUND_UP(fp, 1 << 16);
-}
-
static int skl_check_nv12_aux_surface(struct intel_plane_state *plane_state)
{
struct intel_display *display = to_intel_display(plane_state);
int max_height = intel_plane_max_height(plane, fb, uv_plane, rotation);
/*
- * LNL+ UV surface start/size =
- * ceiling(half of Y plane start/size). Use ceiling division
- * unconditionally; it is a no-op for even values.
+ * UV (chroma) start/size = ceiling(half of the *integer* Y plane
+ * start/size), i.e. the value the luma surface programs (src >> 16),
+ * not the raw U16.16. A bigjoiner seam mapped through the scaler can
+ * give a fractional luma src; ceiling that directly would round the
+ * chroma one column too far and read past the chroma surface.
*/
- int x = fp_16_16_to_int_ceil(fp_16_16_div2(plane_state->uapi.src.x1));
- int y = fp_16_16_to_int_ceil(fp_16_16_div2(plane_state->uapi.src.y1));
- int w = fp_16_16_to_int_ceil(fp_16_16_div2(drm_rect_width(&plane_state->uapi.src)));
- int h = fp_16_16_to_int_ceil(fp_16_16_div2(drm_rect_height(&plane_state->uapi.src)));
+ int luma_x = plane_state->uapi.src.x1 >> 16;
+ int luma_y = plane_state->uapi.src.y1 >> 16;
+ int luma_w = drm_rect_width(&plane_state->uapi.src) >> 16;
+ int luma_h = drm_rect_height(&plane_state->uapi.src) >> 16;
+ int x = DIV_ROUND_UP(luma_x, 2);
+ int y = DIV_ROUND_UP(luma_y, 2);
+ int w = DIV_ROUND_UP(luma_x + luma_w, 2) - x;
+ int h = DIV_ROUND_UP(luma_y + luma_h, 2) - y;
u32 offset;
/* FIXME not quite sure how/if these apply to the chroma plane */
return;
skl_ddb_entry_init(&crtc_state->wm.skl.plane_ddb[plane->id], 0, 0);
- skl_ddb_entry_init(&crtc_state->wm.skl.plane_ddb[plane->id], 0, 0);
+ skl_ddb_entry_init(&crtc_state->wm.skl.plane_ddb_y[plane->id], 0, 0);
crtc_state->wm.skl.plane_min_ddb[plane->id] = 0;
crtc_state->wm.skl.plane_interim_ddb[plane->id] = 0;
struct intel_engine_cs *engine;
if (copy_from_user(&ci, &user->engines[n], sizeof(ci))) {
- kfree(set.engines);
- return -EFAULT;
+ err = -EFAULT;
+ goto err;
}
memset(&set.engines[n], 0, sizeof(set.engines[n]));
drm_dbg(&i915->drm,
"Invalid engine[%d]: { class:%d, instance:%d }\n",
n, ci.engine_class, ci.engine_instance);
- kfree(set.engines);
- return -ENOENT;
+ err = -ENOENT;
+ goto err;
}
set.engines[n].type = I915_GEM_ENGINE_TYPE_PHYSICAL;
set_proto_ctx_engines_extensions,
ARRAY_SIZE(set_proto_ctx_engines_extensions),
&set);
- if (err) {
- kfree(set.engines);
- return err;
- }
+ if (err)
+ goto err_extensions;
pc->num_user_engines = set.num_engines;
pc->user_engines = set.engines;
return 0;
+
+err_extensions:
+ for (n = 0; n < set.num_engines; n++)
+ kfree(set.engines[n].siblings);
+err:
+ kfree(set.engines);
+
+ return err;
}
static int set_proto_ctx_sseu(struct drm_i915_file_private *fpriv,
pe = &pc->user_engines[idx];
/* Only render engine supports RPCS configuration. */
- if (pe->engine->class != RENDER_CLASS)
+ if (!pe->engine || pe->engine->class != RENDER_CLASS)
return -EINVAL;
sseu = &pe->sseu;
p = &prev->rb_right;
}
- if (IS_ENABLED(CONFIG_DRM_I915_SELFTESTS) &&
+ if (IS_ENABLED(CONFIG_DRM_I915_SELFTEST) &&
IS_ENABLED(CONFIG_DRM_I915_DEBUG_GEM)) {
struct intel_engine_cs *engine;
unsigned int isolation;
struct drm_i915_private *i915 = siblings[0]->i915;
struct virtual_engine *ve;
unsigned int n;
- int err;
+ int err = -ENOMEM;
ve = kzalloc_flex(*ve, siblings, count);
if (!ve)
- return ERR_PTR(-ENOMEM);
+ goto err;
ve->base.i915 = i915;
ve->base.gt = siblings[0]->gt;
intel_engine_init_execlists(&ve->base);
ve->base.sched_engine = i915_sched_engine_create(ENGINE_VIRTUAL);
- if (!ve->base.sched_engine) {
- err = -ENOMEM;
- goto err_put;
- }
+ if (!ve->base.sched_engine)
+ goto err_noput;
ve->base.sched_engine->private_data = &ve->base;
ve->base.cops = &virtual_context_ops;
intel_context_init(&ve->context, &ve->base);
ve->base.breadcrumbs = intel_breadcrumbs_create(NULL);
- if (!ve->base.breadcrumbs) {
- err = -ENOMEM;
+ if (!ve->base.breadcrumbs)
goto err_put;
- }
for (n = 0; n < count; n++) {
struct intel_engine_cs *sibling = siblings[n];
virtual_engine_initial_hint(ve);
return &ve->context;
+err_noput:
+ kfree(ve);
+ goto err;
+
err_put:
intel_context_put(&ve->context);
+err:
return ERR_PTR(err);
}
{
const u64 *a = A, *b = B;
- if (a < b)
+ if (*a < *b)
return -1;
- else if (a > b)
+ else if (*a > *b)
return 1;
else
return 0;
{
const u32 *a = A, *b = B;
- if (a < b)
+ if (*a < *b)
return -1;
- else if (a > b)
+ else if (*a > *b)
return 1;
else
return 0;
goto err_free_ctx;
ctx->vm_ctx = pvr_vm_context_lookup(pvr_file, args->vm_context_handle);
- if (IS_ERR(ctx->vm_ctx)) {
- err = PTR_ERR(ctx->vm_ctx);
+ if (!ctx->vm_ctx) {
+ err = -EINVAL;
goto err_free_ctx;
}
return 0;
}
-const struct kernel_param_ops pvr_fw_trace_init_mask_ops = {
+static const struct kernel_param_ops pvr_fw_trace_init_mask_ops = {
.set = pvr_fw_trace_init_mask_set,
.get = param_get_hexint,
};
* covered by a number of LPTEs, the LPTEs once again take
* control over their address range.
*
- * Determine how many LPTEs need to transition state.
+ * Transition each LPTE individually as each may have a
+ * different target state (sparse, invalid, or valid).
*/
- pgt->pte[ptei].s.spte_valid = false;
- for (ptes = 1, ptei++; ptei < lpti; ptes++, ptei++) {
+ for (ptei++; ptei < lpti; ptei++) {
if (pgt->pte[ptei].s.sptes)
break;
- pgt->pte[ptei].s.spte_valid = false;
}
- if (pgt->pte[pteb].s.sparse) {
- TRA(it, "LPTE %05x: U -> S %d PTEs", pteb, ptes);
- pair->func->sparse(vmm, pgt->pt[0], pteb, ptes);
- } else if (!pgt->pte[pteb].s.lpte_valid) {
- if (pair->func->invalid) {
- /* If the MMU supports it, restore the LPTE to the
- * INVALID state to tell the MMU there is no point
- * trying to fetch the corresponding SPTEs.
- */
- TRA(it, "LPTE %05x: U -> I %d PTEs", pteb, ptes);
- pair->func->invalid(vmm, pgt->pt[0], pteb, ptes);
+ while (pteb < ptei) {
+ pgt->pte[pteb].s.spte_valid = false;
+ if (pgt->pte[pteb].s.sparse) {
+ TRA(it, "LPTE %05x: U -> S", pteb);
+ pair->func->sparse(vmm, pgt->pt[0], pteb, 1);
+ } else if (!pgt->pte[pteb].s.lpte_valid) {
+ if (pair->func->invalid) {
+ TRA(it, "LPTE %05x: U -> I", pteb);
+ pair->func->invalid(vmm, pgt->pt[0], pteb, 1);
+ }
}
- } else {
- TRA(it, "LPTE %05x: V %d PTEs", pteb, ptes);
+ pteb++;
}
}
}
return ret;
#ifdef CONFIG_DEBUG_FS
- drmm_mutex_init(&ptdev->base, &ptdev->gems.lock);
+ ret = drmm_mutex_init(&ptdev->base, &ptdev->gems.lock);
+ if (ret)
+ return ret;
+
INIT_LIST_HEAD(&ptdev->gems.node);
#endif
}
if (hdr.size > iter.size) {
+ ret = -EINVAL;
drm_err(&ptdev->base, "Firmware image is truncated\n");
goto out;
}
sor->num_settings = sor->soc->num_settings;
sor->pmc = devm_tegra_pmc_get(&pdev->dev);
- if (IS_ERR(sor->pmc)) {
- err = PTR_ERR(sor->pmc);
- goto put_aux;
- }
+ if (IS_ERR(sor->pmc))
+ return PTR_ERR(sor->pmc);
np = of_parse_phandle(pdev->dev.of_node, "nvidia,dpaux", 0);
if (np) {
{
struct drm_exec exec;
- drm_exec_init(&exec, DRM_EXEC_INTERRUPTIBLE_WAIT, 0);
- drm_exec_until_all_locked(&exec)
{
- break;
+ __label__ drm_exec_retry;
+
+ drm_exec_init(&exec, DRM_EXEC_INTERRUPTIBLE_WAIT, 0);
+ drm_exec_until_all_locked(&exec)
+ {
+ break;
+ }
+ drm_exec_fini(&exec);
}
- drm_exec_fini(&exec);
- drm_exec_init(&exec, DRM_EXEC_INTERRUPTIBLE_WAIT, 0);
- drm_exec_until_all_locked(&exec)
{
- break;
+ __label__ drm_exec_retry;
+
+ drm_exec_init(&exec, DRM_EXEC_INTERRUPTIBLE_WAIT, 0);
+ drm_exec_until_all_locked(&exec)
+ {
+ break;
+ }
+ drm_exec_fini(&exec);
}
- drm_exec_fini(&exec);
KUNIT_SUCCEED(test);
}
return -EBUSY;
#ifdef CONFIG_X86
- /* Anything returned to the system needs to be cached. */
- if (tt->caching != ttm_cached)
- set_pages_array_wb(tt->pages, tt->num_pages);
+ /* Anything returned to the system needs to be cached. Walk allocations
+ * skipping NULL pages and issue set_pages_array_wb() per contiguous run.
+ */
+ if (tt->caching != ttm_cached) {
+ pgoff_t run_start = 0, run_count = 0;
+
+ for (i = 0; i < tt->num_pages; i += num_pages) {
+ page = tt->pages[i];
+ if (unlikely(!page || ttm_backup_page_ptr_is_handle(page))) {
+ if (run_count) {
+ set_pages_array_wb(&tt->pages[run_start],
+ run_count);
+ run_count = 0;
+ }
+ num_pages = 1;
+ continue;
+ }
+ num_pages = 1UL << ttm_pool_page_order(pool, page);
+ if (!run_count)
+ run_start = i;
+ run_count += num_pages;
+ }
+ if (run_count)
+ set_pages_array_wb(&tt->pages[run_start], run_count);
+ }
#endif
if (tt->dma_address || flags->purge) {
unsigned int order;
page = tt->pages[i];
- if (unlikely(!page)) {
+ if (unlikely(!page || ttm_backup_page_ptr_is_handle(page))) {
num_pages = 1;
continue;
}
if (unlikely(!page))
continue;
+ /* Already-handled entry from a previous attempt. */
+ if (unlikely(ttm_backup_page_ptr_is_handle(page)))
+ continue;
+
ttm_pool_split_for_swap(pool, page);
shandle = ttm_backup_backup_page(backup, page, flags->writeback, i,
sizeof(indirect_csd.wg_uniform_offsets));
info->indirect = drm_gem_object_lookup(file_priv, indirect_csd.indirect);
+ if (!info->indirect)
+ return -ENOENT;
return v3d_setup_csd_jobs_and_bos(file_priv, v3d, &indirect_csd.submit,
&info->job, &info->clean_job,
if (!vgdev->has_virgl_3d)
return;
- objs = virtio_gpu_array_alloc(1);
- if (!objs)
- return;
- virtio_gpu_array_add_obj(objs, obj);
+ if (vfpriv->context_created) {
+ objs = virtio_gpu_array_alloc(1);
+ if (!objs)
+ return;
+ virtio_gpu_array_add_obj(objs, obj);
- virtio_gpu_cmd_context_detach_resource(vgdev, vfpriv->ctx_id,
- objs);
+ virtio_gpu_cmd_context_detach_resource(vgdev, vfpriv->ctx_id,
+ objs);
+ }
virtio_gpu_notify(vgdev);
}
virtio_gpu_cmd_get_edids(vgdev);
virtio_gpu_cmd_get_display_info(vgdev);
virtio_gpu_notify(vgdev);
- drm_helper_hpd_irq_event(vgdev->ddev);
+ wait_event_timeout(vgdev->resp_wq,
+ !vgdev->display_info_pending,
+ 5 * HZ);
+ drm_kms_helper_hotplug_event(vgdev->ddev);
}
events_clear |= VIRTIO_GPU_EVENT_DISPLAY;
}
vgdev->display_info_pending = false;
spin_unlock(&vgdev->display_info_lock);
wake_up(&vgdev->resp_wq);
-
- if (!drm_helper_hpd_irq_event(vgdev->ddev))
- drm_kms_helper_hotplug_event(vgdev->ddev);
}
static void virtio_gpu_cmd_get_capset_info_cb(struct virtio_gpu_device *vgdev,
#include <kunit/test-bug.h>
#include <kunit/test.h>
-#include <kunit/test-bug.h>
#include <kunit/visibility.h>
#define PLATFORM_CASE(platform__, graphics_step__) \
xe_pm_runtime_get_noresume(xe);
}
+ /*
+ * Attach CCS BBs before submitting the copy job below so a VF
+ * migration racing the copy sees valid, up to date attach state.
+ */
+ if (IS_VF_CCS_READY(xe) &&
+ ((move_lacks_source && new_mem->mem_type == XE_PL_TT) ||
+ (old_mem_type == XE_PL_SYSTEM && new_mem->mem_type == XE_PL_TT)) &&
+ handle_system_ccs) {
+ ret = xe_sriov_vf_ccs_attach_bo(bo, new_mem);
+ if (ret) {
+ xe_pm_runtime_put(xe);
+ goto out;
+ }
+ }
+
if (move_lacks_source) {
u32 flags = 0;
ttm_bo_move_null(ttm_bo, new_mem);
}
- dma_fence_put(fence);
- xe_pm_runtime_put(xe);
-
/*
- * CCS meta data is migrated from TT -> SMEM. So, let us detach the
- * BBs from BO as it is no longer needed.
+ * Detach must wait for the copy above to complete: a VF migration
+ * racing an in-flight copy must still see valid CCS BBs, so don't
+ * tear them down until the copy fence has signaled.
*/
if (IS_VF_CCS_READY(xe) && old_mem_type == XE_PL_TT &&
- new_mem->mem_type == XE_PL_SYSTEM)
+ new_mem->mem_type == XE_PL_SYSTEM) {
+ dma_fence_wait(fence, false);
xe_sriov_vf_ccs_detach_bo(bo);
+ }
- if (IS_VF_CCS_READY(xe) &&
- ((move_lacks_source && new_mem->mem_type == XE_PL_TT) ||
- (old_mem_type == XE_PL_SYSTEM && new_mem->mem_type == XE_PL_TT)) &&
- handle_system_ccs)
- ret = xe_sriov_vf_ccs_attach_bo(bo);
+ dma_fence_put(fence);
+ xe_pm_runtime_put(xe);
out:
if ((!ttm_bo->resource || ttm_bo->resource->mem_type == XE_PL_SYSTEM) &&
backup = xe_bo_init_locked(xe, NULL, NULL, bo->ttm.base.resv, NULL, xe_bo_size(bo),
DRM_XE_GEM_CPU_CACHING_WB, ttm_bo_type_kernel,
XE_BO_FLAG_SYSTEM | XE_BO_FLAG_NEEDS_CPU_ACCESS |
- XE_BO_FLAG_PINNED, &exec);
+ XE_BO_FLAG_PINNED, NULL, &exec);
if (IS_ERR(backup)) {
drm_exec_retry_on_contention(&exec);
ret = PTR_ERR(backup);
xe_bo_size(bo),
DRM_XE_GEM_CPU_CACHING_WB, ttm_bo_type_kernel,
XE_BO_FLAG_SYSTEM | XE_BO_FLAG_NEEDS_CPU_ACCESS |
- XE_BO_FLAG_PINNED, &exec);
+ XE_BO_FLAG_PINNED, NULL, &exec);
if (IS_ERR(backup)) {
drm_exec_retry_on_contention(&exec);
ret = PTR_ERR(backup);
if (bo->ttm.base.import_attach)
drm_prime_gem_destroy(&bo->ttm.base, NULL);
+ if (bo->dma_buf)
+ dma_buf_put(bo->dma_buf);
drm_gem_object_release(&bo->ttm.base);
xe_assert(xe, list_empty(&ttm_bo->base.gpuva.list));
* @cpu_caching: The cpu caching used for system memory backing store.
* @type: The TTM buffer object type.
* @flags: XE_BO_FLAG_ flags.
+ * @dma_buf: The dma-buf to reference for the BO lifetime (imported BOs),
+ * or NULL.
* @exec: The drm_exec transaction to use for exhaustive eviction.
*
* Initialize or create an xe buffer object. On failure, any allocated buffer
struct xe_tile *tile, struct dma_resv *resv,
struct ttm_lru_bulk_move *bulk, size_t size,
u16 cpu_caching, enum ttm_bo_type type,
- u32 flags, struct drm_exec *exec)
+ u32 flags, struct dma_buf *dma_buf,
+ struct drm_exec *exec)
{
struct ttm_operation_ctx ctx = {
.interruptible = true,
placement = (type == ttm_bo_type_sg ||
bo->flags & XE_BO_FLAG_DEFER_BACKING) ? &sys_placement :
&bo->placement;
+
+ /*
+ * For imported BOs, keep the exporter dma-buf alive for the BO
+ * lifetime. Taken before ttm_bo_init_reserved() to also cover a
+ * creation failure there. Released in xe_ttm_bo_destroy().
+ */
+ if (dma_buf) {
+ get_dma_buf(dma_buf);
+ bo->dma_buf = dma_buf;
+ }
+
err = ttm_bo_init_reserved(&xe->ttm, &bo->ttm, type,
placement, alignment,
&ctx, NULL, resv, xe_ttm_bo_destroy);
vm && !xe_vm_in_fault_mode(vm) &&
flags & XE_BO_FLAG_USER ?
&vm->lru_bulk_move : NULL, size,
- cpu_caching, type, flags, exec);
+ cpu_caching, type, flags, NULL, exec);
if (IS_ERR(bo))
return bo;
struct xe_tile *tile, struct dma_resv *resv,
struct ttm_lru_bulk_move *bulk, size_t size,
u16 cpu_caching, enum ttm_bo_type type,
- u32 flags, struct drm_exec *exec);
+ u32 flags, struct dma_buf *dma_buf,
+ struct drm_exec *exec);
struct xe_bo *xe_bo_create_locked(struct xe_device *xe, struct xe_tile *tile,
struct xe_vm *vm, size_t size,
enum ttm_bo_type type, u32 flags,
struct xe_bo *backup_obj;
/** @parent_obj: Ref to parent bo if this a backup_obj */
struct xe_bo *parent_obj;
+ /** @dma_buf: Imported dma-buf ref to keep its resv alive. */
+ struct dma_buf *dma_buf;
/** @flags: flags for this buffer object */
u32 flags;
/** @vm: VM this BO is attached to, for extobj this will be NULL */
static const struct drm_driver admin_only_driver = {
.driver_features =
- XE_DISPLAY_DRIVER_FEATURES |
DRIVER_GEM | DRIVER_RENDER | DRIVER_GEM_GPUVA,
.open = xe_file_open,
.postclose = xe_file_close,
.major = DRIVER_MAJOR,
.minor = DRIVER_MINOR,
.patchlevel = DRIVER_PATCHLEVEL,
- XE_DISPLAY_DRIVER_OPS,
};
/**
WQ_MEM_RECLAIM);
xe->ordered_wq = alloc_ordered_workqueue("xe-ordered-wq", 0);
xe->unordered_wq = alloc_workqueue("xe-unordered-wq", WQ_PERCPU, 0);
- xe->destroy_wq = alloc_workqueue("xe-destroy-wq", WQ_PERCPU, 0);
+ xe->destroy_wq = alloc_workqueue("xe-destroy-wq", WQ_PERCPU | WQ_MEM_RECLAIM, 0);
if (!xe->ordered_wq || !xe->unordered_wq ||
!xe->preempt_fence_wq || !xe->destroy_wq) {
/*
/** @unordered_wq: used to serialize unordered work */
struct workqueue_struct *unordered_wq;
- /** @destroy_wq: used to serialize user destroy work, like queue */
+ /** @destroy_wq: used to serialize SVM pagemap destroy work */
struct workqueue_struct *destroy_wq;
/** @tiles: device tiles */
bo = xe_bo_init_locked(xe, NULL, NULL, resv, NULL, dma_buf->size,
0, /* Will require 1way or 2way for vm_bind */
- ttm_bo_type_sg, XE_BO_FLAG_SYSTEM, &exec);
+ ttm_bo_type_sg, XE_BO_FLAG_SYSTEM, dma_buf, &exec);
drm_exec_retry_on_contention(&exec);
if (IS_ERR(bo)) {
ret = PTR_ERR(bo);
goto err_exec;
}
- /* Wait behind rebinds */
+ /*
+ * Wait behind rebinds and any kernel operations (evictions, defrag
+ * moves, ...) on the VM and all external BOs. The VM's private BOs
+ * carry their kernel ops in the VM dma-resv KERNEL slot, while each
+ * external BO carries them in its own dma-resv KERNEL slot; both are
+ * covered by iterating every object locked by the exec, mirroring the
+ * drm_gpuvm_resv_add_fence() below.
+ */
if (!xe_vm_in_lr_mode(vm)) {
- err = xe_sched_job_add_deps(job,
- xe_vm_resv(vm),
- DMA_RESV_USAGE_KERNEL);
- if (err)
- goto err_put_job;
+ struct drm_gem_object *obj;
+
+ drm_exec_for_each_locked_object(exec, obj) {
+ err = xe_sched_job_add_deps(job, obj->resv,
+ DMA_RESV_USAGE_KERNEL);
+ if (err)
+ goto err_put_job;
+ }
}
for (i = 0; i < num_syncs && !err; i++)
#include <linux/workqueue.h>
#include "xe_gpu_scheduler_types.h"
+#include "xe_hw_fence_types.h"
struct dma_fence;
struct xe_exec_queue;
struct rcu_head rcu;
/** @sched: GPU scheduler for this xe_exec_queue */
struct xe_gpu_scheduler sched;
+ /**
+ * @name: Scheduler timeline name, kept with @sched until RCU free.
+ */
+ char name[MAX_FENCE_NAME_LEN];
/** @entity: Scheduler entity for this xe_exec_queue */
struct xe_sched_entity entity;
/**
#include <linux/circ_buf.h>
#include <linux/dma-fence-array.h>
+#include <drm/drm_drv.h>
#include <drm/drm_managed.h>
#include "abi/guc_actions_abi.h"
#include "xe_macros.h"
#include "xe_map.h"
#include "xe_mocs.h"
+#include "xe_module.h"
#include "xe_pm.h"
#include "xe_ring_ops_types.h"
#include "xe_sched_job.h"
static void guc_submit_sw_fini(struct drm_device *drm, void *arg)
{
struct xe_guc *guc = arg;
- struct xe_device *xe = guc_to_xe(guc);
struct xe_gt *gt = guc_to_gt(guc);
- int ret;
-
- ret = wait_event_timeout(guc->submission_state.fini_wq,
- xa_empty(&guc->submission_state.exec_queue_lookup),
- HZ * 5);
- drain_workqueue(xe->destroy_wq);
-
- xe_gt_assert(gt, ret);
+ xe_gt_assert(gt, xa_empty(&guc->submission_state.exec_queue_lookup));
xa_destroy(&guc->submission_state.exec_queue_lookup);
}
xa_init(&guc->submission_state.exec_queue_lookup);
- init_waitqueue_head(&guc->submission_state.fini_wq);
-
primelockdep(guc);
guc->submission_state.initialized = true;
xe_guc_id_mgr_release_locked(&guc->submission_state.idm,
q->guc->id, q->width);
- if (xa_empty(&guc->submission_state.exec_queue_lookup))
- wake_up(&guc->submission_state.fini_wq);
-
mutex_unlock(&guc->submission_state.lock);
}
{
struct xe_guc_exec_queue *ge = q->guc;
struct xe_guc *guc = exec_queue_to_guc(q);
+ struct drm_device *drm = &guc_to_xe(guc)->drm;
if (xe_exec_queue_is_multi_queue_secondary(q)) {
struct xe_exec_queue_group *group = q->multi_queue.group;
* (timeline name).
*/
kfree_rcu(ge, rcu);
+
+ drm_dev_put(drm);
}
-static void __guc_exec_queue_destroy_async(struct work_struct *w)
+static void guc_exec_queue_do_destroy(struct xe_exec_queue *q)
{
- struct xe_guc_exec_queue *ge =
- container_of(w, struct xe_guc_exec_queue, destroy_async);
- struct xe_exec_queue *q = ge->q;
+ struct xe_guc_exec_queue *ge = q->guc;
struct xe_guc *guc = exec_queue_to_guc(q);
+ struct xe_device *xe = guc_to_xe(guc);
+ struct drm_device *drm = &xe->drm;
+
+ /*
+ * guc_exec_queue_fini() drops the queue's drm_device ref.
+ * Keep the device alive until the PM-runtime guard unwinds.
+ */
+ drm_dev_get(drm);
+
+ scoped_guard(xe_pm_runtime, xe) {
+ trace_xe_exec_queue_destroy(q);
- guard(xe_pm_runtime)(guc_to_xe(guc));
- trace_xe_exec_queue_destroy(q);
+ /* Confirm no work left behind accessing device structures */
+ cancel_delayed_work_sync(&ge->sched.base.work_tdr);
- /* Confirm no work left behind accessing device structures */
- cancel_delayed_work_sync(&ge->sched.base.work_tdr);
+ xe_exec_queue_fini(q);
+ }
- xe_exec_queue_fini(q);
+ drm_dev_put(drm);
}
-static void guc_exec_queue_destroy_async(struct xe_exec_queue *q)
+static void __guc_exec_queue_destroy_async(struct work_struct *w)
{
- struct xe_guc *guc = exec_queue_to_guc(q);
- struct xe_device *xe = guc_to_xe(guc);
+ struct xe_guc_exec_queue *ge =
+ container_of(w, struct xe_guc_exec_queue, destroy_async);
+
+ guc_exec_queue_do_destroy(ge->q);
+}
+static void guc_exec_queue_destroy_async(struct xe_exec_queue *q)
+{
INIT_WORK(&q->guc->destroy_async, __guc_exec_queue_destroy_async);
/* We must block on kernel engines so slabs are empty on driver unload */
if (q->flags & EXEC_QUEUE_FLAG_PERMANENT || exec_queue_wedged(q))
- __guc_exec_queue_destroy_async(&q->guc->destroy_async);
+ guc_exec_queue_do_destroy(q);
else
- queue_work(xe->destroy_wq, &q->guc->destroy_async);
+ xe_destroy_wq_queue(&q->guc->destroy_async);
}
static void __guc_exec_queue_destroy(struct xe_guc *guc, struct xe_exec_queue *q)
{
struct xe_gpu_scheduler *sched;
struct xe_guc *guc = exec_queue_to_guc(q);
+ struct drm_device *drm = &guc_to_xe(guc)->drm;
struct workqueue_struct *submit_wq = NULL;
struct xe_guc_exec_queue *ge;
long timeout;
if (!ge)
return -ENOMEM;
+ drm_dev_get(drm);
+
q->guc = ge;
ge->q = q;
init_rcu_head(&ge->rcu);
xe_exec_queue_assign_name(q, q->guc->id);
+ strscpy(ge->name, q->name, sizeof(ge->name));
+
/*
* Use primary queue's submit_wq for all secondary queues of a
* multi queue group. This serialization avoids any locking around
err = xe_sched_init(&ge->sched, &drm_sched_ops, &xe_sched_ops,
submit_wq, xe_lrc_ring_size() / MAX_JOB_SIZE_BYTES, 64,
timeout, guc_to_gt(guc)->ordered_wq, NULL,
- q->name, gt_to_xe(q->gt)->drm.dev);
+ ge->name, gt_to_xe(q->gt)->drm.dev);
if (err)
goto err_release_id;
release_guc_id(guc, q);
err_free:
kfree(ge);
+ drm_dev_put(drm);
return err;
}
* even initialized - before that not even the lock is valid
*/
bool initialized;
- /** @submission_state.fini_wq: submit fini wait queue */
- wait_queue_head_t fini_wq;
} submission_state;
/** @hwconfig: Hardware config state */
* @tile: Tile whose migration context to be used.
* @q : Execution to be used along with migration context.
* @src_bo: The buffer object @src is currently bound to.
+ * @new_mem: The (not yet committed) destination resource @src_bo is being
+ * moved into; src_bo->ttm.resource is still the old resource.
* @read_write : Creates BB commands for CCS read/write.
*
* Creates batch buffer instructions to copy CCS metadata from CCS pool to
*/
int xe_migrate_ccs_rw_copy(struct xe_tile *tile, struct xe_exec_queue *q,
struct xe_bo *src_bo,
+ struct ttm_resource *new_mem,
enum xe_sriov_vf_ccs_rw_ctxs read_write)
{
bool src_is_pltt = read_write == XE_SRIOV_VF_CCS_READ_CTX;
bool dst_is_pltt = read_write == XE_SRIOV_VF_CCS_WRITE_CTX;
- struct ttm_resource *src = src_bo->ttm.resource;
+ struct ttm_resource *src = new_mem;
struct xe_migrate *m = tile->migrate;
struct xe_gt *gt = tile->primary_gt;
u32 batch_size, batch_size_allocated;
int xe_migrate_ccs_rw_copy(struct xe_tile *tile, struct xe_exec_queue *q,
struct xe_bo *src_bo,
+ struct ttm_resource *new_mem,
enum xe_sriov_vf_ccs_rw_ctxs read_write);
void xe_migrate_ccs_rw_copy_clear(struct xe_bo *src_bo,
#include <linux/init.h>
#include <linux/module.h>
+#include <linux/workqueue.h>
#include <drm/drm_module.h>
return 0;
}
+static struct workqueue_struct *xe_destroy_wq;
+
+static int __init xe_destroy_wq_module_init(void)
+{
+ xe_destroy_wq = alloc_workqueue("xe-guc-destroy-wq", WQ_UNBOUND, 0);
+ if (!xe_destroy_wq)
+ return -ENOMEM;
+ return 0;
+}
+
+static void xe_destroy_wq_module_exit(void)
+{
+ if (xe_destroy_wq)
+ destroy_workqueue(xe_destroy_wq);
+ xe_destroy_wq = NULL;
+}
+
+/**
+ * xe_destroy_wq_queue() - Queue work on the destroy workqueue
+ * @work: work item to queue
+ *
+ * The destroy workqueue has module lifetime and is used for GuC exec queue
+ * teardown that can outlive a single xe_device. SVM pagemap destroy uses the
+ * per-device xe->destroy_wq instead.
+ *
+ * Return: %true if @work was queued, %false if it was already pending.
+ */
+bool xe_destroy_wq_queue(struct work_struct *work)
+{
+ return queue_work(xe_destroy_wq, work);
+}
+
+/**
+ * xe_destroy_wq_flush() - Flush the destroy workqueue
+ *
+ * Drains all pending destroy work. Called from PCI remove to ensure
+ * teardown ordering before the device is destroyed.
+ */
+void xe_destroy_wq_flush(void)
+{
+ if (xe_destroy_wq)
+ flush_workqueue(xe_destroy_wq);
+}
+
struct init_funcs {
int (*init)(void);
void (*exit)(void);
.init = xe_sched_job_module_init,
.exit = xe_sched_job_module_exit,
},
+ {
+ .init = xe_destroy_wq_module_init,
+ .exit = xe_destroy_wq_module_exit,
+ },
{
.init = xe_register_pci_driver,
.exit = xe_unregister_pci_driver,
#include <linux/types.h>
+struct work_struct;
+
/* Module modprobe variables */
struct xe_modparam {
bool force_execlist;
extern struct xe_modparam xe_modparam;
+bool xe_destroy_wq_queue(struct work_struct *work);
+void xe_destroy_wq_flush(void);
+
#endif
struct xe_mmio *mmio = xe_root_tile_mmio(xe);
bool writable_override;
struct xe_reg reg;
- u32 test_bit;
+ u32 test_bit, test_val;
switch (xe->info.platform) {
case XE_CRESCENTISLAND:
reg = PCODE_SCRATCH(0);
test_bit = FDO_MODE;
+ test_val = FDO_MODE;
break;
case XE_BATTLEMAGE:
reg = HECI_FWSTS2(DG2_GSC_HECI2_BASE);
test_bit = HECI_FW_STATUS_2_NVM_ACCESS_MODE;
+ test_val = 0;
break;
case XE_PVC:
reg = HECI_FWSTS2(PVC_GSC_HECI2_BASE);
test_bit = HECI_FW_STATUS_2_NVM_ACCESS_MODE;
+ test_val = 0;
break;
case XE_DG2:
reg = HECI_FWSTS2(DG2_GSC_HECI2_BASE);
test_bit = HECI_FW_STATUS_2_NVM_ACCESS_MODE;
+ test_val = 0;
break;
case XE_DG1:
reg = HECI_FWSTS2(DG1_GSC_HECI2_BASE);
test_bit = HECI_FW_STATUS_2_NVM_ACCESS_MODE;
+ test_val = 0;
break;
default:
drm_err(&xe->drm, "Unknown platform\n");
return true;
}
- writable_override = !(xe_mmio_read32(mmio, reg) & test_bit);
+ writable_override = (xe_mmio_read32(mmio, reg) & test_bit) == test_val;
if (writable_override)
drm_info(&xe->drm, "NVM access overridden by jumper\n");
return writable_override;
xe->info.probe_display = IS_ENABLED(CONFIG_DRM_XE_DISPLAY) &&
xe_modparam.probe_display &&
- desc->has_display;
+ desc->has_display &&
+ !xe_device_is_admin_only(xe);
xe->info.force_execlist = xe_modparam.force_execlist;
xe_assert(xe, desc->max_gt_per_tile > 0);
return;
xe_device_remove(xe);
+
+ /*
+ * Preserve remove-time flush after moving destroy work to module
+ * lifetime.
+ */
+ xe_destroy_wq_flush();
xe_pm_fini(xe);
}
u64 *ptr = data;
u32 i;
+ /*
+ * @qword_ofs is the absolute entry offset within the page table, while
+ * @ptes is indexed relative to @update->ofs (its first entry). The GPU
+ * path (write_pgtable) splits a single update into MAX_PTE_PER_SDI-sized
+ * chunks, calling this with an advancing @qword_ofs but a fresh @data
+ * pointer per chunk, so translate back into a @ptes index rather than
+ * assuming the chunk starts at ptes[0].
+ */
for (i = 0; i < num_qwords; i++) {
+ u32 idx = qword_ofs - update->ofs + i;
+
if (map)
xe_map_wr(tile_to_xe(tile), map, (qword_ofs + i) *
- sizeof(u64), u64, ptes[i].pte);
+ sizeof(u64), u64, ptes[idx].pte);
else
- ptr[i] = ptes[i].pte;
+ ptr[i] = ptes[idx].pte;
}
}
pt_update_ops, rftree);
}
+#if IS_ENABLED(CONFIG_DRM_GPUSVM)
/*
* Acquire/release the svm notifier_lock around xe_pt_svm_userptr_pre_commit()
* and the matching late release in xe_pt_update_ops_run(). Read mode by
xe_svm_notifier_unlock(vm);
#endif
}
+#else
+static inline void xe_pt_svm_userptr_notifier_lock(struct xe_vm *vm) { }
+static inline void xe_pt_svm_userptr_notifier_unlock(struct xe_vm *vm) { }
+#endif
#if IS_ENABLED(CONFIG_DRM_GPUSVM)
#ifdef CONFIG_DRM_XE_USERPTR_INVAL_INJECT
xe_pt_update_ops_init(struct xe_vm_pgtable_update_ops *pt_update_ops)
{
init_llist_head(&pt_update_ops->deferred);
+ pt_update_ops->current_op = 0;
pt_update_ops->start = ~0x0ull;
pt_update_ops->last = 0x0ull;
+ pt_update_ops->needs_svm_lock = false;
+ pt_update_ops->needs_invalidation = false;
xe_page_reclaim_list_init(&pt_update_ops->prl);
}
/**
* xe_sriov_vf_ccs_attach_bo - Insert CCS read write commands in the BO.
* @bo: the &buffer object to which batch buffer commands will be added.
+ * @new_mem: the (not yet committed) destination resource @bo is being moved
+ * into; bo->ttm.resource is still the old resource at this point.
*
* This function shall be called only by VF. It inserts the PTEs and copy
* command instructions in the BO by calling xe_migrate_ccs_rw_copy()
*
* Returns: 0 if successful, negative error code on failure.
*/
-int xe_sriov_vf_ccs_attach_bo(struct xe_bo *bo)
+int xe_sriov_vf_ccs_attach_bo(struct xe_bo *bo, struct ttm_resource *new_mem)
{
struct xe_device *xe = xe_bo_device(bo);
enum xe_sriov_vf_ccs_rw_ctxs ctx_id;
xe_assert(xe, !bb);
ctx = &xe->sriov.vf.ccs.contexts[ctx_id];
- err = xe_migrate_ccs_rw_copy(tile, ctx->mig_q, bo, ctx_id);
+ err = xe_migrate_ccs_rw_copy(tile, ctx->mig_q, bo, new_mem, ctx_id);
+ if (err)
+ goto err_unwind;
+ }
+ return 0;
+
+err_unwind:
+ /*
+ * Clean up any contexts already attached. Can't reuse
+ * xe_sriov_vf_ccs_detach_bo() here as it requires both contexts
+ * attached before cleaning up either one.
+ */
+ for_each_ccs_rw_ctx(ctx_id) {
+ if (bo->bb_ccs[ctx_id])
+ xe_migrate_ccs_rw_copy_clear(bo, ctx_id);
}
return err;
}
#include "xe_sriov_vf_ccs_types.h"
struct drm_printer;
+struct ttm_resource;
struct xe_device;
struct xe_bo;
int xe_sriov_vf_ccs_init(struct xe_device *xe);
-int xe_sriov_vf_ccs_attach_bo(struct xe_bo *bo);
+int xe_sriov_vf_ccs_attach_bo(struct xe_bo *bo, struct ttm_resource *new_mem);
int xe_sriov_vf_ccs_detach_bo(struct xe_bo *bo);
int xe_sriov_vf_ccs_register_context(struct xe_device *xe);
void xe_sriov_vf_ccs_rebase(struct xe_device *xe);
.request_decompress = false,
.check_purged = true,
});
- if (!err && !xe_vma_has_no_bo(vma))
- err = xe_bo_migrate(xe_vma_bo(vma),
- region_to_mem_type[region],
- NULL,
- exec);
+ if (!err && !xe_vma_has_no_bo(vma)) {
+ struct xe_bo *bo = xe_vma_bo(vma);
+ u32 mem_type;
+
+ if (region == DRM_XE_CONSULT_MEM_ADVISE_PREF_LOC) {
+ unsigned int i;
+
+ mem_type = XE_PL_TT;
+ for (i = 0; i < bo->placement.num_placement; i++) {
+ if (mem_type_is_vram(bo->placements[i].mem_type)) {
+ mem_type = bo->placements[i].mem_type;
+ break;
+ }
+ }
+ } else {
+ mem_type = region_to_mem_type[region];
+ }
+
+ err = xe_bo_migrate(bo, mem_type, NULL, exec);
+ }
break;
}
default:
xe_device_is_l2_flush_optimized(xe) &&
(pat_index != 19 && coh_mode != XE_COH_2WAY))) {
err = -EINVAL;
- goto madv_fini;
+ goto free_vmas;
}
}
*/
/* Default WOPCM size is 2MB from Gen11, 1MB on previous platforms */
-/* FIXME: Larger size require for 2 tile PVC, do a proper probe sooner or later */
+/* FIXME: Larger size require for some platforms, do a proper probe sooner or later */
#define DGFX_WOPCM_SIZE SZ_4M
-/* FIXME: Larger size require for MTL, do a proper probe sooner or later */
+#define LNL_WOPCM_SIZE SZ_8M
#define MTL_WOPCM_SIZE SZ_4M
#define WOPCM_SIZE SZ_2M
u32 xe_wopcm_size(struct xe_device *xe)
{
- return IS_DGFX(xe) ? DGFX_WOPCM_SIZE :
- xe->info.platform == XE_METEORLAKE ? MTL_WOPCM_SIZE :
- WOPCM_SIZE;
+ if (xe->info.platform >= XE_LUNARLAKE)
+ return LNL_WOPCM_SIZE;
+ else if (IS_DGFX(xe))
+ return DGFX_WOPCM_SIZE;
+ else if (xe->info.platform == XE_METEORLAKE)
+ return MTL_WOPCM_SIZE;
+ else
+ return WOPCM_SIZE;
}
static u32 max_wopcm_size(struct xe_device *xe)
if (WARN_ON(!cache))
continue;
- mutex_lock(&mapping->cache->lock);
+ mutex_lock(&cache->lock);
WARN_ON(kref_read(&mapping->ref) != 1);
__host1x_bo_unpin(&mapping->ref);
- mutex_unlock(&mapping->cache->lock);
+ mutex_unlock(&cache->lock);
}
}
EXPORT_SYMBOL(host1x_bo_clear_cached_mappings);
#include <linux/kfifo.h>
#include <linux/minmax.h>
#include <linux/module.h>
+#include <linux/overflow.h>
#include "hid_bpf_dispatch.h"
const struct hid_ops *hid_ops;
hid_bpf_get_data(struct hid_bpf_ctx *ctx, unsigned int offset, const size_t rdwr_buf_size)
{
struct hid_bpf_ctx_kern *ctx_kern;
+ size_t end;
ctx_kern = container_of(ctx, struct hid_bpf_ctx_kern, ctx);
- if (rdwr_buf_size + offset > ctx->allocated_size)
+ if (check_add_overflow(rdwr_buf_size, offset, &end) ||
+ end > ctx->allocated_size)
return NULL;
return ctx_kern->data + offset;
struct hid_device *hid;
unsigned short keymap[ARRAY_SIZE(appleir_key_table)];
struct timer_list key_up_timer; /* timer for key up */
- spinlock_t lock; /* protects .current_key */
+ spinlock_t lock; /* protects .current_key, .removing */
int current_key; /* the currently pressed key */
int prev_key_idx; /* key index in a 2 packets message */
+ bool removing; /* set during teardown; gates input_dev access */
};
static int get_key(int data)
unsigned long flags;
spin_lock_irqsave(&appleir->lock, flags);
- if (appleir->current_key) {
+ if (!appleir->removing && appleir->current_key) {
key_up(hid, appleir, appleir->current_key);
appleir->current_key = 0;
}
int index;
spin_lock_irqsave(&appleir->lock, flags);
+ if (appleir->removing) {
+ spin_unlock_irqrestore(&appleir->lock, flags);
+ goto out;
+ }
/*
* If we already have a key down, take it up before marking
* this one down
appleir->prev_key_idx = 0;
if (!memcmp(data, keyrepeat, sizeof(keyrepeat))) {
- key_down(hid, appleir, appleir->current_key);
- /*
- * Remote doesn't do key up, either pull them up, in the test
- * above, or here set a timer which pulls them up after 1/8 s
- */
- mod_timer(&appleir->key_up_timer, jiffies + HZ / 8);
+ spin_lock_irqsave(&appleir->lock, flags);
+ if (!appleir->removing) {
+ key_down(hid, appleir, appleir->current_key);
+ /*
+ * Remote doesn't do key up, either pull them up, in
+ * the test above, or here set a timer which pulls them
+ * up after 1/8 s
+ */
+ mod_timer(&appleir->key_up_timer, jiffies + HZ / 8);
+ }
+ spin_unlock_irqrestore(&appleir->lock, flags);
goto out;
}
if (!memcmp(data, flatbattery, sizeof(flatbattery))) {
- battery_flat(appleir);
+ spin_lock_irqsave(&appleir->lock, flags);
+ if (!appleir->removing)
+ battery_flat(appleir);
+ spin_unlock_irqrestore(&appleir->lock, flags);
/* Fall through */
}
static void appleir_remove(struct hid_device *hid)
{
struct appleir *appleir = hid_get_drvdata(hid);
+ unsigned long flags;
+
+ /*
+ * Mark the driver as tearing down so that any concurrent raw_event
+ * (e.g. from a USB URB completion that hid_hw_stop() has not yet
+ * killed) and the key_up_timer softirq stop touching input_dev
+ * before hid_hw_stop() frees it via hidinput_disconnect().
+ */
+ spin_lock_irqsave(&appleir->lock, flags);
+ appleir->removing = true;
+ spin_unlock_irqrestore(&appleir->lock, flags);
+
+ timer_shutdown_sync(&appleir->key_up_timer);
hid_hw_stop(hid);
- timer_delete_sync(&appleir->key_up_timer);
}
static const struct hid_device_id appleir_devices[] = {
u8 *cdata = data;
int ret = 0;
+ if (report_enum->numbered && (size < 1 || bufsize < 1)) {
+ hid_warn_ratelimited(hid,
+ "Event data for numbered report is too short (%d vs %zu)\n",
+ size, bufsize);
+ return -EINVAL;
+ }
+
report = hid_get_report(report_enum, data);
if (!report)
return 0;
ret = letsketch_setup_input_tablet(data);
if (ret)
- return ret;
+ goto err_shutdown_timer;
ret = letsketch_setup_input_tablet_pad(data);
if (ret)
- return ret;
+ goto err_shutdown_timer;
+
+ ret = hid_hw_start(hdev, HID_CONNECT_HIDRAW);
+ if (ret)
+ goto err_shutdown_timer;
- return hid_hw_start(hdev, HID_CONNECT_HIDRAW);
+ return 0;
+
+err_shutdown_timer:
+ /*
+ * Drain any pending callback and permanently disable the timer
+ * before devm releases data: if hid_hw_start() enabled I/O on an
+ * always-poll-quirk device and then failed, raw_event may have
+ * armed the timer already.
+ */
+ timer_shutdown_sync(&data->inrange_timer);
+ return ret;
+}
+
+static void letsketch_remove(struct hid_device *hdev)
+{
+ struct letsketch_data *data = hid_get_drvdata(hdev);
+
+ /*
+ * hid_hw_stop() synchronously kills the URBs that deliver
+ * raw_event(), so once it returns no path can re-arm
+ * inrange_timer. timer_shutdown_sync() then drains any
+ * in-flight callback and permanently disables further
+ * mod_timer() calls before devm releases data.
+ */
+ hid_hw_stop(hdev);
+ timer_shutdown_sync(&data->inrange_timer);
}
static const struct hid_device_id letsketch_devices[] = {
.name = "letsketch",
.id_table = letsketch_devices,
.probe = letsketch_probe,
+ .remove = letsketch_remove,
.raw_event = letsketch_raw_event,
};
module_hid_driver(letsketch_driver);
};
MODULE_DEVICE_TABLE(hid, lg_g15_devices);
+static void lg_g15_remove(struct hid_device *hdev)
+{
+ struct lg_g15_data *g15 = hid_get_drvdata(hdev);
+
+ /*
+ * g15->work is only initialized for the models that schedule it
+ * (G15, G15 v2, G510). The G13 and Z-10 leave it zeroed, so only
+ * cancel it when it was set up.
+ */
+ if (g15 && g15->work.func)
+ cancel_work_sync(&g15->work);
+
+ hid_hw_stop(hdev);
+}
+
static struct hid_driver lg_g15_driver = {
.name = "lg-g15",
.id_table = lg_g15_devices,
.raw_event = lg_g15_raw_event,
.probe = lg_g15_probe,
+ .remove = lg_g15_remove,
};
module_hid_driver(lg_g15_driver);
output_report_enum = &hdev->report_enum[HID_OUTPUT_REPORT];
rep = output_report_enum->report_id_hash[REPORT_ID_DJ_SHORT];
- if (rep && (rep->maxfield < 1 ||
- rep->field[0]->report_count != DJREPORT_SHORT_LENGTH - 1)) {
+ if (rep && rep->maxfield < 1) {
+ hid_err(hdev, "Expected size of DJ short report is %d, but got 0",
+ DJREPORT_SHORT_LENGTH - 1);
+ return -EINVAL;
+ }
+
+ if (rep && rep->field[0]->report_count != DJREPORT_SHORT_LENGTH - 1) {
hid_err(hdev, "Expected size of DJ short report is %d, but got %d",
DJREPORT_SHORT_LENGTH - 1, rep->field[0]->report_count);
return -EINVAL;
* [1] https://gitlab.freedesktop.org/libevdev/hid-tools
*/
+#include <linux/bitmap.h>
#include <linux/bits.h>
#include <linux/device.h>
#include <linux/hid.h>
TOUCHPAD_REPORT_ALL = TOUCHPAD_REPORT_BUTTONS | TOUCHPAD_REPORT_CONTACTS,
};
-#define MT_IO_SLOTS_MASK GENMASK(7, 0) /* reserve first 8 bits for slot tracking */
-#define MT_IO_FLAGS_RUNNING 32
+#define MT_IO_FLAGS_RUNNING 0
static const bool mtrue = true; /* default for true */
static const bool mfalse; /* default for false */
struct timer_list release_timer; /* to release sticky fingers */
struct hid_haptic_device *haptic; /* haptic related configuration */
struct hid_device *hdev; /* hid_device we're attached to */
- unsigned long mt_io_flags; /* mt flags (MT_IO_FLAGS_RUNNING)
- * first 8 bits are reserved for keeping the slot
- * states, this is fine because we only support up
- * to 250 slots (MT_MAX_MAXCONTACT)
+ unsigned long mt_io_flags; /* mt flags (MT_IO_FLAGS_RUNNING) */
+ unsigned long *active_slots; /* bitmap of slots with an active
+ * contact, sized for maxcontacts
*/
__u8 inputmode_value; /* InputMode HID feature value */
__u8 maxcontacts;
for_each_set_bit(slotnum, app->pending_palm_slots, td->maxcontacts) {
clear_bit(slotnum, app->pending_palm_slots);
- clear_bit(slotnum, &td->mt_io_flags);
+ clear_bit(slotnum, td->active_slots);
input_mt_slot(input, slotnum);
input_mt_report_slot_inactive(input);
input_event(input, EV_ABS, ABS_MT_TOUCH_MAJOR, major);
input_event(input, EV_ABS, ABS_MT_TOUCH_MINOR, minor);
- set_bit(slotnum, &td->mt_io_flags);
+ set_bit(slotnum, td->active_slots);
} else {
- clear_bit(slotnum, &td->mt_io_flags);
+ clear_bit(slotnum, td->active_slots);
}
return 0;
* defect.
*/
if (app->quirks & MT_QUIRK_STICKY_FINGERS) {
- if (td->mt_io_flags & MT_IO_SLOTS_MASK)
+ if (!bitmap_empty(td->active_slots, td->maxcontacts))
mod_timer(&td->release_timer,
jiffies + msecs_to_jiffies(100));
else
if (td->is_pressurepad)
__set_bit(INPUT_PROP_PRESSUREPAD, input->propbit);
+ if (!td->active_slots) {
+ td->active_slots = devm_kcalloc(&td->hdev->dev,
+ BITS_TO_LONGS(td->maxcontacts),
+ sizeof(long),
+ GFP_KERNEL);
+ if (!td->active_slots)
+ return -ENOMEM;
+ }
+
app->pending_palm_slots = devm_kcalloc(&hi->input->dev,
BITS_TO_LONGS(td->maxcontacts),
sizeof(long),
for (i = 0; i < mt->num_slots; i++) {
input_mt_slot(input_dev, i);
input_mt_report_slot_inactive(input_dev);
- clear_bit(i, &td->mt_io_flags);
+ clear_bit(i, td->active_slots);
}
input_mt_sync_frame(input_dev);
input_sync(input_dev);
*/
if (test_and_set_bit_lock(MT_IO_FLAGS_RUNNING, &td->mt_io_flags))
return;
- if (td->mt_io_flags & MT_IO_SLOTS_MASK)
+ if (!bitmap_empty(td->active_slots, td->maxcontacts))
mt_release_contacts(hdev);
clear_bit_unlock(MT_IO_FLAGS_RUNNING, &td->mt_io_flags);
}
struct picolcd_pending *work;
struct hid_report *report = picolcd_out_report(report_id, hdev);
unsigned long flags;
- int i, j, k;
+ int i, j;
+ unsigned int k;
if (!report || !data)
return NULL;
}
EXPORT_SYMBOL_GPL(sensor_hub_get_feature);
+int sensor_hub_input_attr_read_values(struct hid_sensor_hub_device *hsdev,
+ u32 usage_id, u32 attr_usage_id,
+ u32 report_id,
+ enum sensor_hub_read_flags flag,
+ u32 buffer_size, u8 *buffer)
+{
+ struct sensor_hub_data *data = hid_get_drvdata(hsdev->hdev);
+ struct hid_report *report;
+ unsigned long flags;
+ long cycles;
+ int ret;
+
+ report = sensor_hub_report(report_id, hsdev->hdev, HID_INPUT_REPORT);
+ if (!report)
+ return -EINVAL;
+
+ mutex_lock(hsdev->mutex_ptr);
+ if (flag == SENSOR_HUB_SYNC) {
+ memset(&hsdev->pending, 0, sizeof(hsdev->pending));
+ init_completion(&hsdev->pending.ready);
+ hsdev->pending.usage_id = usage_id;
+ hsdev->pending.attr_usage_id = attr_usage_id;
+ hsdev->pending.max_raw_size = buffer_size;
+ hsdev->pending.raw_data = buffer;
+
+ spin_lock_irqsave(&data->lock, flags);
+ hsdev->pending.status = true;
+ spin_unlock_irqrestore(&data->lock, flags);
+ }
+ mutex_lock(&data->mutex);
+ hid_hw_request(hsdev->hdev, report, HID_REQ_GET_REPORT);
+ mutex_unlock(&data->mutex);
+ ret = 0;
+ if (flag == SENSOR_HUB_SYNC) {
+ cycles = wait_for_completion_interruptible_timeout(&hsdev->pending.ready,
+ HZ * 5);
+ if (cycles == 0)
+ ret = -ETIMEDOUT;
+ else if (cycles < 0)
+ ret = cycles;
+
+ hsdev->pending.status = false;
+ }
+ mutex_unlock(hsdev->mutex_ptr);
+
+ return ret;
+}
+EXPORT_SYMBOL_GPL(sensor_hub_input_attr_read_values);
int sensor_hub_input_attr_get_raw_value(struct hid_sensor_hub_device *hsdev,
u32 usage_id,
struct hid_collection *collection = NULL;
void *priv = NULL;
struct hid_sensor_hub_device *hsdev = NULL;
+ u32 copy_size;
+ u32 avail;
hid_dbg(hdev, "sensor_hub_raw_event report id:0x%x size:%d type:%d\n",
report->id, size, report->type);
hsdev->pending.attr_usage_id ==
report->field[i]->logical)) {
hid_dbg(hdev, "data was pending ...\n");
- hsdev->pending.raw_data = kmemdup(ptr, sz, GFP_ATOMIC);
- if (hsdev->pending.raw_data)
- hsdev->pending.raw_size = sz;
- else
- hsdev->pending.raw_size = 0;
- complete(&hsdev->pending.ready);
+ if (hsdev->pending.max_raw_size) {
+ if (hsdev->pending.index < hsdev->pending.max_raw_size) {
+ avail = hsdev->pending.max_raw_size - hsdev->pending.index;
+ copy_size = clamp(sz, 0U, avail);
+
+ memcpy(hsdev->pending.raw_data + hsdev->pending.index,
+ ptr, copy_size);
+ hsdev->pending.index += copy_size;
+ if (hsdev->pending.index >= hsdev->pending.max_raw_size) {
+ hsdev->pending.raw_size = hsdev->pending.index;
+ complete(&hsdev->pending.ready);
+ }
+ }
+ } else {
+ hsdev->pending.raw_data = kmemdup(ptr, sz, GFP_ATOMIC);
+ if (hsdev->pending.raw_data)
+ hsdev->pending.raw_size = sz;
+ else
+ hsdev->pending.raw_size = 0;
+ complete(&hsdev->pending.ready);
+ }
}
if (callback->capture_sample) {
if (report->field[i]->logical)
static inline void i2c_imx_isr_read_block_data_len(struct imx_i2c_struct *i2c_imx)
{
u8 len = imx_i2c_read_reg(i2c_imx, IMX_I2C_I2DR);
+ unsigned int temp;
if (len == 0 || len > I2C_SMBUS_BLOCK_MAX) {
+ /*
+ * SMBus 3.1 6.5.7: support count byte of 0.
+ * I2C_SMBUS_BLOCK_MAX case should not hold the SDA either.
+ * So NACK it (TXAK) to not hold the bus.
+ */
+ temp = imx_i2c_read_reg(i2c_imx, IMX_I2C_I2CR);
+ temp |= I2CR_TXAK;
+ imx_i2c_write_reg(temp, i2c_imx, IMX_I2C_I2CR);
+
+ if (len == 0) {
+ i2c_imx->msg->buf[i2c_imx->msg_buf_idx++] = 0;
+ i2c_imx->msg->len = 2;
+ return;
+ }
+
i2c_imx->isr_result = -EPROTO;
i2c_imx->state = IMX_I2C_STATE_FAILED;
wake_up(&i2c_imx->queue);
+ return;
}
i2c_imx->msg->len += len;
i2c_imx->msg->buf[i2c_imx->msg_buf_idx++] = len;
int i, result;
unsigned int temp;
int block_data = msgs->flags & I2C_M_RECV_LEN;
+ int block_err = 0;
result = i2c_imx_prepare_read(i2c_imx, msgs, false);
if (result)
*/
if ((!i) && block_data) {
len = imx_i2c_read_reg(i2c_imx, IMX_I2C_I2DR);
- if ((len == 0) || (len > I2C_SMBUS_BLOCK_MAX))
- return -EPROTO;
+ if ((len == 0) || (len > I2C_SMBUS_BLOCK_MAX)) {
+ /*
+ * SMBus 3.1 6.5.7: support count byte of 0.
+ * I2C_SMBUS_BLOCK_MAX case should not hold the SDA either.
+ */
+ if (len > I2C_SMBUS_BLOCK_MAX)
+ block_err = -EPROTO;
+ temp = imx_i2c_read_reg(i2c_imx, IMX_I2C_I2CR);
+ temp |= I2CR_TXAK;
+ imx_i2c_write_reg(temp, i2c_imx, IMX_I2C_I2CR);
+ msgs->buf[0] = 0;
+ msgs->len = 2;
+ continue;
+ }
dev_dbg(&i2c_imx->adapter.dev,
"<%s> read length: 0x%X\n",
__func__, len);
"<%s> read byte: B%d=0x%X\n",
__func__, i, msgs->buf[i]);
}
- return 0;
+ return block_err;
}
static int i2c_imx_read(struct imx_i2c_struct *i2c_imx, struct i2c_msg *msgs,
status = readl(i2c->base + SPACEMIT_ISR);
if (!status)
- return IRQ_HANDLED;
+ return IRQ_NONE;
i2c->status = status;
tmp_res->io = devm_platform_get_and_ioremap_resource(pdev, type, &tmp_res->params);
if (IS_ERR(tmp_res->io)) {
+ int ret = PTR_ERR(tmp_res->io);
+
devm_kfree(dev, tmp_res);
- return PTR_ERR(tmp_res->io);
+ return ret;
}
tmp_res->type = type;
i2c->auto_restart = i2c->dev_comp->auto_restart;
/* checking if we can skip restart and optimize using WRRD mode */
- if (i2c->auto_restart && num == 2) {
+ if (num == 2) {
if (!(msgs[0].flags & I2C_M_RD) && (msgs[1].flags & I2C_M_RD) &&
msgs[0].addr == msgs[1].addr) {
i2c->auto_restart = 0;
if (samples && count > samples)
count = samples;
+ count = min_t(u8, count, BMC150_ACCEL_FIFO_LENGTH);
+
ret = bmc150_accel_fifo_transfer(data, (u8 *)buffer, count);
if (ret)
return ret;
if (mask == IIO_CHAN_INFO_SCALE) {
/* Check no integer component */
if (val)
- return -EINVAL;
- ret = kxsd9_write_scale(indio_dev, val2);
+ ret = -EINVAL;
+ else
+ ret = kxsd9_write_scale(indio_dev, val2);
}
pm_runtime_put_autosuspend(st->dev);
depends on SPI
depends on GPIOLIB
select IIO_BUFFER
+ select IIO_TRIGGERED_BUFFER
select IIO_KFIFO_BUF
select REGMAP_SPI
depends on COMMON_CLK
config AD7380
tristate "Analog Devices AD7380 ADC driver"
depends on SPI_MASTER
+ select REGMAP
select SPI_OFFLOAD
select IIO_BUFFER
select IIO_BUFFER_DMAENGINE
depends on SPI
select CRC8
select IIO_BUFFER
+ select IIO_TRIGGERED_BUFFER
select IIO_BACKEND
help
Say yes here to build support for Analog Devices AD777X family
if (irq < 0)
return irq;
+ init_completion(&st->completion);
+
retval = devm_request_irq(&pdev->dev, irq, lpc32xx_adc_isr, 0,
LPC32XXAD_NAME, st);
if (retval < 0) {
platform_set_drvdata(pdev, iodev);
- init_completion(&st->completion);
-
iodev->name = LPC32XXAD_NAME;
iodev->info = &lpc32xx_adc_iio_info;
iodev->modes = INDIO_DIRECT_MODE;
writel(0, NXP_SAR_ADC_IMR(info->regs));
}
-static void nxp_sar_adc_wait_for(struct nxp_sar_adc *info, unsigned int cycles)
+static void nxp_sar_adc_wait_for(struct nxp_sar_adc *info, u64 cycles)
{
u64 rate;
rate = clk_get_rate(info->clk);
if (rate)
- ndelay(div64_u64(NSEC_PER_SEC, rate * cycles));
+ ndelay(div64_u64(NSEC_PER_SEC * cycles, rate));
}
static bool nxp_sar_adc_set_enabled(struct nxp_sar_adc *info, bool enable)
st = iio_priv(indio_dev);
st->dev = dev;
+ init_completion(&st->completion);
mutex_init(&st->lock);
/*
spear_adc_configure(st);
- init_completion(&st->completion);
-
indio_dev->name = SPEAR_ADC_MOD_NAME;
indio_dev->info = &spear_adc_info;
indio_dev->modes = INDIO_DIRECT_MODE;
if (ret)
return ret;
- return i2c_smbus_write_byte(st->client, ADS1119_CMD_START_SYNC);
+ ret = i2c_smbus_write_byte(st->client, ADS1119_CMD_START_SYNC);
+ if (ret)
+ pm_runtime_put_autosuspend(dev);
+
+ return ret;
}
static int ads1119_triggered_buffer_postdisable(struct iio_dev *indio_dev)
ads124s_priv->reset_gpio = devm_gpiod_get_optional(&spi->dev,
"reset", GPIOD_OUT_LOW);
if (IS_ERR(ads124s_priv->reset_gpio))
- dev_info(&spi->dev, "Reset GPIO not defined\n");
+ return dev_err_probe(&spi->dev, PTR_ERR(ads124s_priv->reset_gpio),
+ "Failed to get reset GPIO\n");
ads124s_priv->chip_info = &ads124s_chip_info_tbl[spi_id->driver_data];
u8 *outdata;
struct st_sensor_data *sdata = iio_priv(indio_dev);
unsigned int byte_for_channel;
+ u32 tmp;
byte_for_channel = DIV_ROUND_UP(ch->scan_type.realbits +
ch->scan_type.shift, 8);
if (err < 0)
return err;
- if (byte_for_channel == 1)
- *data = (s8)*outdata;
- else if (byte_for_channel == 2)
- *data = (s16)get_unaligned_le16(outdata);
- else if (byte_for_channel == 3)
- *data = (s32)sign_extend32(get_unaligned_le24(outdata), 23);
+ if (byte_for_channel == 1) {
+ tmp = *outdata;
+ } else if (byte_for_channel == 2) {
+ if (ch->scan_type.endianness == IIO_BE)
+ tmp = get_unaligned_be16(outdata);
+ else
+ tmp = get_unaligned_le16(outdata);
+ } else if (byte_for_channel == 3) {
+ if (ch->scan_type.endianness == IIO_BE)
+ tmp = get_unaligned_be24(outdata);
+ else
+ tmp = get_unaligned_le24(outdata);
+ } else {
+ return -EINVAL;
+ }
+ *data = sign_extend32(tmp, BYTES_TO_BITS(byte_for_channel) - 1);
return 0;
}
vdd_uV = ret;
- ret = devm_regulator_get_enable_read_voltage(dev, "vref");
- if (ret > 0) {
- vref_uV = ret;
+ if (device_property_present(dev, "vref-supply")) {
+ vref_uV = devm_regulator_get_enable_read_voltage(dev, "vref");
+ if (vref_uV < 0)
+ return vref_uV;
+
+ if (vref_uV == 0)
+ return dev_err_probe(dev, -EINVAL, "Vref is 0 uV.\n");
+
data->use_vref = true;
} else {
vref_uV = 0;
- dev_dbg(dev, "using internal band gap as voltage reference.\n");
- dev_dbg(dev, "Vref is unavailable.\n");
+ dev_dbg(dev, "Using internal band gap as voltage reference.\n");
}
- if (chip_features->have_ext_vref1) {
- ret = devm_regulator_get_enable_read_voltage(dev, "vref1");
- if (ret > 0) {
- vref1_uV = ret;
- data->use_vref1 = true;
- } else {
- vref1_uV = 0;
- dev_dbg(dev, "using internal band gap as voltage reference 1.\n");
- dev_dbg(dev, "Vref1 is unavailable.\n");
- }
+ if (chip_features->have_ext_vref1 &&
+ device_property_present(dev, "vref1-supply")) {
+ vref1_uV = devm_regulator_get_enable_read_voltage(dev, "vref1");
+ if (vref1_uV < 0)
+ return vref1_uV;
+
+ if (vref1_uV == 0)
+ return dev_err_probe(dev, -EINVAL, "Vref1 is 0 uV.\n");
+
+ data->use_vref1 = true;
+ } else {
+ vref1_uV = 0;
+ dev_dbg(dev, "Using internal band gap as voltage reference 1.\n");
}
ret = mcp47feb02_init_ctrl_regs(data);
else
ret = devm_request_irq(&adis->spi->dev, adis->spi->irq,
&iio_trigger_generic_data_rdy_poll,
- adis->irq_flag,
+ adis->irq_flag | IRQF_NO_THREAD,
indio_dev->name,
adis->trig);
if (ret)
ret = devm_request_irq(&indio_dev->dev, irq,
&iio_trigger_generic_data_rdy_poll,
- irq_type, "bmi160", data->trig);
+ irq_type | IRQF_NO_THREAD,
+ "bmi160", data->trig);
if (ret)
return ret;
accel_st->filter = INV_ICM42600_FILTER_AVG_16X;
/*
- * clock period is 32kHz (31250ns)
+ * clock period is 8kHz (125000ns)
* jitter is +/- 2% (20 per mille)
*/
- ts_chip.clock_period = 31250;
+ ts_chip.clock_period = 125000;
ts_chip.jitter = 20;
ts_chip.init_period = inv_icm42600_odr_to_period(st->conf.accel.odr);
inv_sensors_timestamp_init(&accel_st->ts, &ts_chip);
/* compute watermark value in bytes */
wm_size = watermark * packet_size;
+ st->fifo.watermark.value = watermark;
/* changing FIFO watermark requires to turn off watermark interrupt */
ret = regmap_update_bits_check(st->map, INV_ICM42600_REG_INT_SOURCE0,
st->fifo.nb.accel = 0;
st->fifo.nb.total = 0;
- /* compute maximum FIFO read size */
+ /* compute maximum FIFO read size (watermark for max = 0 interrupt case) */
if (max == 0)
- max_count = sizeof(st->fifo.data);
- else
- max_count = max * inv_icm42600_get_packet_size(st->fifo.en);
+ max = st->fifo.watermark.value;
+ max_count = max * inv_icm42600_get_packet_size(st->fifo.en);
/* read FIFO count value */
raw_fifo_count = (__be16 *)st->buffer;
st->fifo.watermark.eff_gyro = 1;
st->fifo.watermark.eff_accel = 1;
+ st->fifo.watermark.value = 1;
/*
* Default FIFO configuration (bits 7 to 5)
unsigned int accel;
unsigned int eff_gyro;
unsigned int eff_accel;
+ unsigned int value;
} watermark;
size_t count;
struct {
}
/*
- * clock period is 32kHz (31250ns)
+ * clock period is 8kHz (125000ns)
* jitter is +/- 2% (20 per mille)
*/
- ts_chip.clock_period = 31250;
+ ts_chip.clock_period = 125000;
ts_chip.jitter = 20;
ts_chip.init_period = inv_icm42600_odr_to_period(st->conf.accel.odr);
inv_sensors_timestamp_init(&gyro_st->ts, &ts_chip);
return -ENODEV;
}
+ hw->settings = &st_lsm6dsx_sensor_settings[i];
+
+ if (hw->settings->shub_settings.page_mux.addr) {
+ /*
+ * If the IMU has the shub page selected on init, for example
+ * after a CPU watchdog reset while the page is selected, the
+ * regular register space is shadowed. While the regular
+ * register space is shadowed, the registers needed for
+ * initializing the IMU are not available.
+ *
+ * Unconditionally clear the shub page selection to ensure
+ * normal register access.
+ */
+ err = st_lsm6dsx_set_page(hw, false);
+ if (err < 0) {
+ dev_err(hw->dev, "failed to clear shub page\n");
+ return err;
+ }
+ }
+
err = regmap_read(hw->regmap, ST_LSM6DSX_REG_WHOAMI_ADDR, &data);
if (err < 0) {
dev_err(hw->dev, "failed to read whoami register\n");
}
*name = st_lsm6dsx_sensor_settings[i].id[j].name;
- hw->settings = &st_lsm6dsx_sensor_settings[i];
return 0;
}
goto unlock;
}
+ kfifo_reset_out(&ev_int->det_events);
+
iio_device_get(indio_dev);
fd = anon_inode_getfd("iio:event", &iio_event_chrdev_fileops,
if (fd < 0) {
clear_bit(IIO_BUSY_BIT_POS, &ev_int->flags);
iio_device_put(indio_dev);
- } else {
- kfifo_reset_out(&ev_int->det_events);
}
-
unlock:
mutex_unlock(&iio_dev_opaque->mlock);
return fd;
config AL3000A
tristate "AL3000a ambient light sensor"
+ select REGMAP_I2C
depends on I2C
help
Say Y here if you want to build a driver for the Dyna Image AL3000a
config AL3010
tristate "AL3010 ambient light sensor"
+ select REGMAP_I2C
depends on I2C
help
Say Y here if you want to build a driver for the Dyna Image AL3010
config AL3320A
tristate "AL3320A ambient light sensor"
+ select REGMAP_I2C
depends on I2C
help
Say Y here if you want to build a driver for the Dyna Image AL3320A
};
static const int al3010_scales[][2] = {
- {0, 1187200}, {0, 296800}, {0, 74200}, {0, 18600}
+ { 1, 187200 }, { 0, 296800 }, { 0, 74200 }, { 0, 18600 },
};
static const struct regmap_config al3010_regmap_config = {
case IIO_LIGHT:
ret = gp2ap002_get_lux(gp2ap002);
if (ret < 0)
- return ret;
+ goto out;
*val = ret;
ret = IIO_VAL_INT;
goto out;
NULL, tsl2591_event_handler,
IRQF_TRIGGER_FALLING | IRQF_ONESHOT,
"tsl2591_irq", indio_dev);
- if (ret) {
- dev_err_probe(&client->dev, ret, "IRQ request error\n");
- return -EINVAL;
- }
+ if (ret)
+ return ret;
indio_dev->info = &tsl2591_info;
} else {
indio_dev->info = &tsl2591_info_no_irq;
long mask)
{
struct dev_rot_state *rot_state = iio_priv(indio_dev);
+ struct hid_sensor_hub_device *hsdev = rot_state->common_attributes.hsdev;
+ struct hid_sensor_hub_attribute_info *info = &rot_state->quaternion;
+ u32 usage_id = HID_USAGE_SENSOR_ORIENT_QUATERNION;
+ union {
+ s16 val16[4];
+ s32 val32[4];
+ } raw_buf;
int ret_type;
int i;
switch (mask) {
case IIO_CHAN_INFO_RAW:
if (size >= 4) {
- for (i = 0; i < 4; ++i)
- vals[i] = rot_state->scan.sampled_vals[i];
+ if (info->size <= 0 || info->size > sizeof(raw_buf))
+ return -EINVAL;
+
+ hid_sensor_power_state(&rot_state->common_attributes, true);
+
+ ret_type = sensor_hub_input_attr_read_values(hsdev,
+ hsdev->usage,
+ usage_id,
+ info->report_id,
+ SENSOR_HUB_SYNC,
+ info->size,
+ (u8 *)&raw_buf);
+
+ hid_sensor_power_state(&rot_state->common_attributes, false);
+
+ if (ret_type < 0)
+ return ret_type;
+
+ switch (info->size) {
+ case sizeof(raw_buf.val16):
+ for (i = 0; i < ARRAY_SIZE(raw_buf.val16); i++)
+ vals[i] = raw_buf.val16[i];
+ break;
+ case sizeof(raw_buf.val32):
+ for (i = 0; i < ARRAY_SIZE(raw_buf.val32); i++)
+ vals[i] = raw_buf.val32[i];
+ break;
+ default:
+ return -EINVAL;
+ }
+
ret_type = IIO_VAL_INT_MULTIPLE;
*val_len = 4;
} else
case IIO_CHAN_INFO_PROCESSED:
pm_runtime_get_sync(data->dev);
ret = mpl115_comp_pressure(data, val, val2);
+ pm_runtime_put_autosuspend(data->dev);
if (ret < 0)
return ret;
- pm_runtime_put_autosuspend(data->dev);
return IIO_VAL_INT_PLUS_MICRO;
case IIO_CHAN_INFO_RAW:
pm_runtime_get_sync(data->dev);
/* temperature -5.35 C / LSB, 472 LSB is 25 C */
ret = mpl115_read_temp(data);
+ pm_runtime_put_autosuspend(data->dev);
if (ret < 0)
return ret;
- pm_runtime_put_autosuspend(data->dev);
*val = ret >> 6;
return IIO_VAL_INT;
obj-$(CONFIG_MCP9600) += mcp9600.o
obj-$(CONFIG_MLX90614) += mlx90614.o
obj-$(CONFIG_MLX90632) += mlx90632.o
-obj-$(CONFIG_MLX90632) += mlx90635.o
+obj-$(CONFIG_MLX90635) += mlx90635.o
obj-$(CONFIG_TMP006) += tmp006.o
obj-$(CONFIG_TMP007) += tmp007.o
obj-$(CONFIG_TMP117) += tmp117.o
list_for_each_entry(current_id, &ips_node->id_list, id_list_entry) {
if (!memcmp(current_id->id.route.addr.dev_addr.dst_dev_addr,
- neigh->ha, ETH_ALEN))
+ neigh->ha, neigh->dev->addr_len))
continue;
cma_id_get(current_id);
if (!queue_work(cma_wq, ¤t_id->id.net_work))
change_mad_state(mad_send_wr, IB_MAD_STATE_EARLY_RESP);
}
+static bool is_kernel_rmpp_data_response(struct ib_mad_agent_private *agent,
+ struct ib_mad_recv_wc *mad_recv_wc)
+{
+ const struct ib_mad_hdr *mad_hdr = &mad_recv_wc->recv_buf.mad->mad_hdr;
+ struct ib_rmpp_mad *rmpp_mad;
+
+ if (!ib_mad_kernel_rmpp_agent(&agent->agent) ||
+ !ib_response_mad(mad_hdr) ||
+ !ib_is_mad_class_rmpp(mad_hdr->mgmt_class))
+ return false;
+
+ rmpp_mad = (struct ib_rmpp_mad *)mad_recv_wc->recv_buf.mad;
+
+ return (ib_get_rmpp_flags(&rmpp_mad->rmpp_hdr) &
+ IB_MGMT_RMPP_FLAG_ACTIVE) &&
+ rmpp_mad->rmpp_hdr.rmpp_type == IB_MGMT_RMPP_TYPE_DATA;
+}
+
static void ib_mad_complete_recv(struct ib_mad_agent_private *mad_agent_priv,
struct ib_mad_recv_wc *mad_recv_wc)
{
}
list_add(&mad_recv_wc->recv_buf.list, &mad_recv_wc->rmpp_list);
+ if (is_kernel_rmpp_data_response(mad_agent_priv, mad_recv_wc)) {
+ spin_lock_irqsave(&mad_agent_priv->lock, flags);
+ mad_send_wr = ib_find_send_mad(mad_agent_priv, mad_recv_wc);
+ spin_unlock_irqrestore(&mad_agent_priv->lock, flags);
+
+ if (!mad_send_wr) {
+ ib_free_recv_mad(mad_recv_wc);
+ deref_mad_agent(mad_agent_priv);
+ return;
+ }
+ }
+
if (ib_mad_kernel_rmpp_agent(&mad_agent_priv->agent)) {
mad_recv_wc = ib_process_rmpp_recv_wc(mad_agent_priv,
mad_recv_wc);
struct ib_cq *cq;
int ret;
+ if (WARN_ON_ONCE(!cq_attr->cqe))
+ return ERR_PTR(-EINVAL);
+
cq = rdma_zalloc_drv_obj(device, ib_cq);
if (!cq)
return ERR_PTR(-ENOMEM);
- if (WARN_ON_ONCE(!cq_attr->cqe))
- return ERR_PTR(-EINVAL);
-
cq->device = device;
cq->comp_handler = comp_handler;
cq->event_handler = event_handler;
const struct ib_recv_wr *wr = recv_wr;
struct erdma_qp *qp = to_eqp(ibqp);
unsigned long flags;
- int ret;
+ int ret = 0;
spin_lock_irqsave(&qp->lock, flags);
mhop.bt_chunk_size;
for (i = 0; i < table->num_hem; ++i) {
- obj = i * buf_chunk_size / table->obj_size;
+ obj = (u64)i * buf_chunk_size / table->obj_size;
if (table->hem[i])
hns_roce_table_mhop_put(hr_dev, table, obj, 0);
}
iwqp->ctx_info.qp_compl_ctx = req.user_compl_ctx;
iwqp->user_mode = 1;
- if (req.user_wqe_bufs) {
- spin_lock_irqsave(&ucontext->qp_reg_mem_list_lock, flags);
- iwqp->iwpbl = irdma_get_pbl((unsigned long)req.user_wqe_bufs,
- &ucontext->qp_reg_mem_list);
- spin_unlock_irqrestore(&ucontext->qp_reg_mem_list_lock, flags);
- if (!iwqp->iwpbl) {
- ret = -ENODATA;
- ibdev_dbg(&iwdev->ibdev, "VERBS: no pbl info\n");
- return ret;
- }
+ spin_lock_irqsave(&ucontext->qp_reg_mem_list_lock, flags);
+ iwqp->iwpbl = irdma_get_pbl((unsigned long)req.user_wqe_bufs,
+ &ucontext->qp_reg_mem_list);
+ spin_unlock_irqrestore(&ucontext->qp_reg_mem_list_lock, flags);
+
+ if (!iwqp->iwpbl) {
+ ret = -ENODATA;
+ ibdev_dbg(&iwdev->ibdev, "VERBS: no pbl info\n");
+ return ret;
}
if (!ucontext->use_raw_attrs) {
u32 pg_idx;
for (pg_idx = 0; pg_idx < npages; pg_idx++) {
- if ((*arr + (pg_size * pg_idx)) != arr[pg_idx])
+ if ((*arr + ((u64)pg_size * pg_idx)) != arr[pg_idx])
return false;
}
for (i = 0; i < lvl2->leaf_cnt; i++, leaf++) {
arr = leaf->addr;
- if ((*start_addr + (i * pg_size * PBLE_PER_PAGE)) != *arr)
+ if ((*start_addr + ((u64)i * pg_size * PBLE_PER_PAGE)) != *arr)
return false;
ret = irdma_check_mem_contiguous(arr, leaf->cnt, pg_size);
if (!ret)
if (flags & ~(IB_MR_REREG_TRANS | IB_MR_REREG_PD | IB_MR_REREG_ACCESS))
return ERR_PTR(-EOPNOTSUPP);
+ if (iwmr->type != IRDMA_MEMREG_TYPE_MEM)
+ return ERR_PTR(-EINVAL);
+
ret = ib_umem_check_rereg(iwmr->region, flags, new_access);
if (ret)
return ERR_PTR(ret);
int mana_ib_post_send(struct ib_qp *ibqp, const struct ib_send_wr *wr,
const struct ib_send_wr **bad_wr)
{
- int err;
+ int err = 0;
struct mana_ib_qp *qp = container_of(ibqp, struct mana_ib_qp, ibqp);
for (; wr; wr = wr->next) {
return err;
}
-static int handle_reg_mr_integrity(struct mlx5_ib_dev *dev,
- struct mlx5_ib_qp *qp,
- const struct ib_send_wr *wr,
- struct mlx5_wqe_ctrl_seg **ctrl, void **seg,
- int *size, void **cur_edge,
- unsigned int *idx, int nreq, u8 fence,
- u8 next_fence)
+static noinline_for_stack int handle_reg_mr_integrity(struct mlx5_ib_dev *dev,
+ struct mlx5_ib_qp *qp,
+ const struct ib_send_wr *wr,
+ struct mlx5_wqe_ctrl_seg **ctrl,
+ void **seg,
+ int *size, void **cur_edge,
+ unsigned int *idx, int nreq,
+ u8 fence,
+ u8 next_fence)
{
struct mlx5_ib_mr *mr;
struct mlx5_ib_mr *pi_mr;
struct siw_ucontext *uctx =
rdma_udata_to_drv_context(udata, struct siw_ucontext,
base_ucontext);
+ struct siw_uresp_create_qp uresp = {};
unsigned long flags;
int num_sqe, num_rqe, rv = 0;
size_t length;
spin_lock_init(&qp->rq_lock);
spin_lock_init(&qp->orq_lock);
- rv = siw_qp_add(sdev, qp);
- if (rv)
- goto err_atomic;
-
-
/* All queue indices are derived from modulo operations
* on a free running 'get' (consumer) and 'put' (producer)
* unsigned counter. Having queue sizes at power of two
if (qp->sendq == NULL) {
rv = -ENOMEM;
- goto err_out_xa;
+ goto err_out;
}
if (attrs->sq_sig_type != IB_SIGNAL_REQ_WR) {
if (attrs->sq_sig_type == IB_SIGNAL_ALL_WR)
qp->attrs.flags |= SIW_SIGNAL_ALL_WR;
else {
rv = -EINVAL;
- goto err_out_xa;
+ goto err_out;
}
}
qp->pd = pd;
if (qp->recvq == NULL) {
rv = -ENOMEM;
- goto err_out_xa;
+ goto err_out;
}
qp->attrs.rq_size = num_rqe;
}
qp->attrs.state = SIW_QP_STATE_IDLE;
if (udata) {
- struct siw_uresp_create_qp uresp = {};
-
uresp.num_sqe = num_sqe;
uresp.num_rqe = num_rqe;
- uresp.qp_id = qp_id(qp);
if (qp->sendq) {
length = num_sqe * sizeof(struct siw_sqe);
length, &uresp.sq_key);
if (!qp->sq_entry) {
rv = -ENOMEM;
- goto err_out_xa;
+ goto err_out;
}
}
if (!qp->rq_entry) {
uresp.sq_key = SIW_INVAL_UOBJ_KEY;
rv = -ENOMEM;
- goto err_out_xa;
+ goto err_out;
}
}
+ }
+ qp->tx_cpu = siw_get_tx_cpu(sdev);
+ if (qp->tx_cpu < 0) {
+ rv = -EINVAL;
+ goto err_out;
+ }
+ init_completion(&qp->qp_free);
+
+ rv = siw_qp_add(sdev, qp);
+ if (rv)
+ goto err_out_tx;
+
+ if (udata) {
+ uresp.qp_id = qp_id(qp);
if (udata->outlen < sizeof(uresp)) {
rv = -EINVAL;
if (rv)
goto err_out_xa;
}
- qp->tx_cpu = siw_get_tx_cpu(sdev);
- if (qp->tx_cpu < 0) {
- rv = -EINVAL;
- goto err_out_xa;
- }
+
INIT_LIST_HEAD(&qp->devq);
spin_lock_irqsave(&sdev->lock, flags);
list_add_tail(&qp->devq, &sdev->qp_list);
spin_unlock_irqrestore(&sdev->lock, flags);
- init_completion(&qp->qp_free);
-
return 0;
err_out_xa:
xa_erase(&sdev->qp_xa, qp_id(qp));
+err_out_tx:
+ siw_put_tx_cpu(qp->tx_cpu);
+err_out:
if (uctx) {
rdma_user_mmap_entry_remove(qp->sq_entry);
rdma_user_mmap_entry_remove(qp->rq_entry);
pad->dev = idev;
pad->mdev = mdev;
+ maple_set_drvdata(mdev, pad);
+
idev->open = dc_pad_open;
idev->close = dc_pad_close;
goto fail;
mdev->driver = mdrv;
- maple_set_drvdata(mdev, pad);
return 0;
kbd->dev = idev;
memcpy(kbd->keycode, dc_kbd_keycode, sizeof(kbd->keycode));
+ maple_set_drvdata(mdev, kbd);
+
idev->name = mdev->product_name;
idev->evbit[0] = BIT(EV_KEY) | BIT(EV_REP);
idev->keycode = kbd->keycode;
mdev->driver = mdrv;
- maple_set_drvdata(mdev, kbd);
-
return error;
fail_register:
static int dc_mouse_open(struct input_dev *dev)
{
- struct dc_mouse *mse = maple_get_drvdata(to_maple_dev(&dev->dev));
+ struct dc_mouse *mse = input_get_drvdata(dev);
maple_getcond_callback(mse->mdev, dc_mouse_callback, HZ/50,
MAPLE_FUNC_MOUSE);
static void dc_mouse_close(struct input_dev *dev)
{
- struct dc_mouse *mse = maple_get_drvdata(to_maple_dev(&dev->dev));
+ struct dc_mouse *mse = input_get_drvdata(dev);
maple_getcond_callback(mse->mdev, dc_mouse_callback, 0,
MAPLE_FUNC_MOUSE);
mse->dev = input_dev;
mse->mdev = mdev;
+ maple_set_drvdata(mdev, mse);
+
+ input_set_drvdata(input_dev, mse);
input_dev->evbit[0] = BIT_MASK(EV_KEY) | BIT_MASK(EV_REL);
input_dev->keybit[BIT_WORD(BTN_MOUSE)] = BIT_MASK(BTN_LEFT) |
BIT_MASK(BTN_RIGHT) | BIT_MASK(BTN_MIDDLE);
goto fail_register;
mdev->driver = mdrv;
- maple_set_drvdata(mdev, mse);
return error;
fail_register:
input_free_device(input_dev);
+ maple_set_drvdata(mdev, NULL);
fail_nomem:
kfree(mse);
fail:
unsigned int x;
unsigned int y;
- if (touch->id > MMS114_MAX_TOUCH) {
+ if (touch->id == 0 || touch->id > MMS114_MAX_TOUCH) {
dev_err(&client->dev, "Wrong touch id (%d)\n", touch->id);
return;
}
struct dm_io_region io_reg = {
.bdev = c->bdev,
.sector = block_to_sector(c, block),
- .count = block_to_sector(c, count),
+ .count = likely(c->sectors_per_block_bits >= 0) ?
+ count << c->sectors_per_block_bits :
+ count * (c->block_size >> SECTOR_SHIFT),
};
if (WARN_ON_ONCE(dm_bufio_in_request()))
int r;
struct era_metadata *md = kzalloc_obj(*md);
- if (!md)
- return NULL;
+ if (!md) {
+ DMERR("could not allocate metadata struct");
+ return ERR_PTR(-ENOMEM);
+ }
md->bdev = bdev;
md->block_size = block_size;
static void remap_to_origin(struct era *era, struct bio *bio)
{
bio_set_dev(bio, era->origin_dev->bdev);
+ bio->bi_iter.bi_sector = dm_target_offset(era->ti, bio->bi_iter.bi_sector);
}
/*
if (r) {
ti->error = "Error opening metadata device";
era_destroy(era);
- return -EINVAL;
+ return r;
}
r = dm_get_device(ti, argv[1], BLK_OPEN_READ | BLK_OPEN_WRITE,
if (r) {
ti->error = "Error opening data device";
era_destroy(era);
- return -EINVAL;
+ return r;
}
r = sscanf(argv[2], "%u%c", &era->sectors_per_block, &dummy);
if (r) {
ti->error = "could not set max io len";
era_destroy(era);
- return -EINVAL;
+ return r;
}
if (!valid_block_size(era->sectors_per_block)) {
static int era_map(struct dm_target *ti, struct bio *bio)
{
struct era *era = ti->private;
- dm_block_t block = get_block(era, bio);
+ dm_block_t block;
/*
* All bios get remapped to the origin device. We do this now, but
* block is marked in this era.
*/
remap_to_origin(era, bio);
+ block = get_block(era, bio);
/*
* REQ_PREFLUSH bios carry no data, so we're not interested in them.
err = get_key_size(&argv[1]);
if (err < 0) {
ti->error = "Cannot parse key size";
- return -EINVAL;
+ err = -EINVAL;
+ goto bad;
}
ctx->key_size = err;
if (ctx->iv_offset & ((ctx->sector_size >> SECTOR_SHIFT) - 1)) {
ti->error = "Wrong alignment of iv_offset sector";
err = -EINVAL;
+ goto bad;
}
ctx->max_dun = (ctx->iv_offset + ti->len - 1) >>
*metadata_offset = 0;
}
- if (unlikely(!is_power_of_2(ic->tag_size)))
- hash_offset = (hash_offset + to_copy) % ic->tag_size;
-
total_size -= to_copy;
} while (unlikely(total_size));
if (unlikely((bio->bi_opf & REQ_PREFLUSH) != 0))
return DM_MAPIO_REMAPPED;
+ if (unlikely(!dm_integrity_check_limits(ic, bio->bi_iter.bi_sector, bio)))
+ return DM_MAPIO_KILL;
+
retry:
if (!dio->integrity_payload) {
unsigned digest_size, extra_size;
dio->bio_details.bi_iter = bio->bi_iter;
- if (unlikely(!dm_integrity_check_limits(ic, bio->bi_iter.bi_sector, bio))) {
- return DM_MAPIO_KILL;
- }
-
bio->bi_iter.bi_sector += ic->start + SB_SECTORS;
bip = bio_integrity_alloc(bio, GFP_NOIO, 1);
struct bio_vec bv = bio_iter_iovec(bio, dio->bio_details.bi_iter);
const char *mem = integrity_kmap(ic, bv.bv_page);
if (ic->tag_size < ic->tuple_size)
- memset(dio->integrity_payload + pos + ic->tag_size, 0, ic->tuple_size - ic->tuple_size);
+ memset(dio->integrity_payload + pos + ic->tag_size, 0, ic->tuple_size - ic->tag_size);
integrity_sector_checksum(ic, &dio->ahash_req, dio->bio_details.bi_iter.bi_sector, mem, bv.bv_offset, dio->integrity_payload + pos);
integrity_kunmap(ic, mem);
pos += ic->tuple_size;
ti->error = "Journal mac mismatch";
goto bad;
}
+ if (ic->fix_hmac && !(ic->sb->flags & cpu_to_le32(SB_FLAG_FIXED_HMAC)) && ic->journal_mac_alg.key_string) {
+ /*
+ * If this happens, it may be either because someone tampered
+ * with the device or it may be due to a bug in the
+ * integritysetup tool.
+ *
+ * In the latter case, upgrade to integritysetup 2.8.7 and use
+ * the argument --integrity-legacy-hmac when using the open
+ * command.
+ */
+ r = -EINVAL;
+ ti->error = "fix_hmac is on the command line but not in the superblock";
+ goto bad;
+ }
get_provided_data_sectors(ic);
if (!ic->provided_data_sectors) {
struct vers_iter *info = param;
/* Check space - it might have changed since the first iteration */
- if ((char *)info->vers + sizeof(tt->version) + strlen(tt->name) + 1 > info->end) {
+ if ((char *)info->vers + sizeof(struct dm_target_versions) + strlen(tt->name) + 1 > info->end) {
info->flags = DM_BUFFER_FULL_FLAG;
return;
}
/* resume device */
r = dm_resume(md);
if (r)
- goto err_destroy_table;
+ goto err_hash_remove;
DMINFO("%s (%s) is ready", md->disk->disk_name, dmi->name);
dm_put(md);
*/
bitset_size = dm_round_up(region_count, BITS_PER_LONG);
bitset_size >>= BYTE_SHIFT;
+ /* Handle dm_round_up rollover on 32-bit systems */
+ if (!bitset_size)
+ bitset_size = 1UL << (BITS_PER_LONG - BYTE_SHIFT);
lc->bitset_uint32_count = bitset_size / sizeof(*lc->clean_bits);
argc--;
if (!strcmp(arg, "cache_mode")) {
+ if (!argc) {
+ *error = "Missing value for cache_mode";
+ return -EINVAL;
+ }
arg = dm_shift_arg(as);
if (!strcmp(arg, "writeback")) {
opts->cache_mode = PCACHE_CACHE_MODE_WRITEBACK;
}
argc--;
} else if (!strcmp(arg, "data_crc")) {
+ if (!argc) {
+ *error = "Missing value for data_crc";
+ return -EINVAL;
+ }
arg = dm_shift_arg(as);
if (!strcmp(arg, "true")) {
opts->data_crc = true;
*/
last = raw_cpu_ptr(stats->last);
stats_aux->merged =
- (bi_sector == (READ_ONCE(last->last_sector) &&
- ((bi_rw == WRITE) ==
- (READ_ONCE(last->last_rw) == WRITE))
- ));
+ bi_sector == READ_ONCE(last->last_sector) &&
+ (bi_rw == WRITE) == (READ_ONCE(last->last_rw) == WRITE);
WRITE_ONCE(last->last_sector, end_sector);
WRITE_ONCE(last->last_rw, bi_rw);
} else
result = jiffies_to_msecs(j & 0x3fffff);
if (j >= 1 << 22) {
mult = jiffies_to_msecs(1 << 22);
- result += (unsigned long long)mult * (unsigned long long)jiffies_to_msecs((j >> 22) & 0x3fffff);
+ result += (unsigned long long)mult * ((j >> 22) & 0x3fffff);
}
if (j >= 1ULL << 44)
- result += (unsigned long long)mult * (unsigned long long)mult * (unsigned long long)jiffies_to_msecs(j >> 44);
+ result += (unsigned long long)mult * (unsigned long long)(1 << 22) * (j >> 44);
return result;
}
uint32_t time;
dm_block_t root;
dm_block_t details_root;
+ dm_block_t held_root;
struct list_head thin_devices;
uint64_t trans_id;
unsigned long flags;
*/
pmd->root = le64_to_cpu(disk_super->data_mapping_root);
pmd->details_root = le64_to_cpu(disk_super->device_details_root);
+ pmd->held_root = le64_to_cpu(disk_super->held_root);
__setup_btree_details(pmd);
dm_bm_unlock(sblock);
pmd->time = le32_to_cpu(disk_super->time);
pmd->root = le64_to_cpu(disk_super->data_mapping_root);
pmd->details_root = le64_to_cpu(disk_super->device_details_root);
+ pmd->held_root = le64_to_cpu(disk_super->held_root);
pmd->trans_id = le64_to_cpu(disk_super->trans_id);
pmd->flags = le32_to_cpu(disk_super->flags);
pmd->data_block_size = le32_to_cpu(disk_super->data_block_size);
disk_super->time = cpu_to_le32(pmd->time);
disk_super->data_mapping_root = cpu_to_le64(pmd->root);
disk_super->device_details_root = cpu_to_le64(pmd->details_root);
+ disk_super->held_root = cpu_to_le64(pmd->held_root);
disk_super->trans_id = cpu_to_le64(pmd->trans_id);
disk_super->flags = cpu_to_le32(pmd->flags);
{
int r, inc;
struct thin_disk_superblock *disk_super;
- struct dm_block *copy, *sblock;
+ struct dm_block *copy;
dm_block_t held_root;
+ if (pmd->held_root) {
+ DMWARN("Pool metadata snapshot already exists: release this before taking another.");
+ return -EBUSY;
+ }
+
/*
* We commit to ensure the btree roots which we increment in a
* moment are up to date.
dm_sm_inc_block(pmd->metadata_sm, THIN_SUPERBLOCK_LOCATION);
r = dm_tm_shadow_block(pmd->tm, THIN_SUPERBLOCK_LOCATION,
&sb_validator, ©, &inc);
- if (r)
+ if (r) {
+ dm_sm_dec_block(pmd->metadata_sm, THIN_SUPERBLOCK_LOCATION);
return r;
+ }
BUG_ON(!inc);
held_root = dm_block_location(copy);
disk_super = dm_block_data(copy);
- if (le64_to_cpu(disk_super->held_root)) {
- DMWARN("Pool metadata snapshot already exists: release this before taking another.");
-
- dm_tm_dec(pmd->tm, held_root);
- dm_tm_unlock(pmd->tm, copy);
- return -EBUSY;
- }
-
/*
* Wipe the spacemap since we're not publishing this.
*/
dm_tm_inc(pmd->tm, le64_to_cpu(disk_super->device_details_root));
dm_tm_unlock(pmd->tm, copy);
- /*
- * Write the held root into the superblock.
- */
- r = superblock_lock(pmd, &sblock);
- if (r) {
- dm_tm_dec(pmd->tm, held_root);
- return r;
- }
+ pmd->held_root = held_root;
- disk_super = dm_block_data(sblock);
- disk_super->held_root = cpu_to_le64(held_root);
- dm_bm_unlock(sblock);
return 0;
}
{
int r;
struct thin_disk_superblock *disk_super;
- struct dm_block *sblock, *copy;
+ struct dm_block *copy;
dm_block_t held_root;
- r = superblock_lock(pmd, &sblock);
- if (r)
- return r;
-
- disk_super = dm_block_data(sblock);
- held_root = le64_to_cpu(disk_super->held_root);
- disk_super->held_root = cpu_to_le64(0);
-
- dm_bm_unlock(sblock);
+ held_root = pmd->held_root;
if (!held_root) {
DMWARN("No pool metadata snapshot found: nothing to release.");
if (r)
return r;
+ pmd->held_root = 0;
+
disk_super = dm_block_data(copy);
dm_btree_del(&pmd->info, le64_to_cpu(disk_super->data_mapping_root));
dm_btree_del(&pmd->details_info, le64_to_cpu(disk_super->device_details_root));
- dm_sm_dec_block(pmd->metadata_sm, held_root);
-
dm_tm_unlock(pmd->tm, copy);
+ dm_sm_dec_block(pmd->metadata_sm, held_root);
+
return 0;
}
static int __get_metadata_snap(struct dm_pool_metadata *pmd,
dm_block_t *result)
{
- int r;
- struct thin_disk_superblock *disk_super;
- struct dm_block *sblock;
-
- r = dm_bm_read_lock(pmd->bm, THIN_SUPERBLOCK_LOCATION,
- &sb_validator, &sblock);
- if (r)
- return r;
-
- disk_super = dm_block_data(sblock);
- *result = le64_to_cpu(disk_super->held_root);
-
- dm_bm_unlock(sblock);
+ *result = pmd->held_root;
return 0;
}
PTR_ERR(bbuf));
/* assume the block is corrupted */
- if (neras && *neras <= v->fec->roots)
+ if (neras && *neras < v->fec->roots)
fio->erasures[(*neras)++] = i;
continue;
* skip if we have already found the theoretical
* maximum number (i.e. fec->roots) of erasures
*/
- if (neras && *neras <= v->fec->roots &&
+ if (neras && *neras < v->fec->roots &&
fec_is_erasure(v, io, want_digest, bbuf))
fio->erasures[(*neras)++] = i;
}
/* per-bio data */
struct dm_verity_fec_io {
struct rs_control *rs; /* Reed-Solomon state */
- int erasures[DM_VERITY_FEC_MAX_ROOTS + 1]; /* erasures for decode_rs8 */
+ int erasures[DM_VERITY_FEC_MAX_ROOTS]; /* erasures for decode_rs8 */
u8 *output; /* buffer for corrected output */
unsigned int level; /* recursion level */
unsigned int nbufs; /* number of buffers allocated */
table = dm_get_live_table(md, &srcu_idx);
- if (table->num_targets != 1)
+ if (!table || table->num_targets != 1)
goto out;
ti = dm_table_get_target(table, 0);
#define DM_MSG_PREFIX "verity"
-#define DM_VERITY_ENV_LENGTH 42
+#define DM_VERITY_ENV_LENGTH 46
#define DM_VERITY_ENV_VAR_NAME "DM_VERITY_ERR_BLOCK_NR"
#define DM_VERITY_DEFAULT_PREFETCH_SIZE 262144
char *envp[] = { verity_env, NULL };
const char *type_str = "";
struct mapped_device *md = dm_table_get_md(v->ti->table);
+ int ce;
/* Corruption should be visible in device status in all modes */
v->hash_failed = true;
- if (v->corrupted_errs >= DM_VERITY_MAX_CORRUPTED_ERRS)
- goto out;
-
- v->corrupted_errs++;
+ ce = atomic_read(&v->corrupted_errs);
+ do {
+ if (ce >= DM_VERITY_MAX_CORRUPTED_ERRS)
+ goto out;
+ } while (!atomic_try_cmpxchg(&v->corrupted_errs, &ce, ce + 1));
switch (type) {
case DM_VERITY_BLOCK_TYPE_DATA:
DMERR_LIMIT("%s: %s block %llu is corrupted", v->data_dev->name,
type_str, block);
- if (v->corrupted_errs == DM_VERITY_MAX_CORRUPTED_ERRS) {
+ if (ce + 1 == DM_VERITY_MAX_CORRUPTED_ERRS) {
DMERR("%s: reached maximum errors", v->data_dev->name);
dm_audit_log_target(DM_MSG_PREFIX, "max-corrupted-errors", v->ti, 0);
}
continue;
} else if (!strcasecmp(arg_name, DM_VERITY_OPT_TASKLET_VERIFY)) {
+ if (v->use_bh_wq)
+ continue;
v->use_bh_wq = true;
static_branch_inc(&use_bh_wq_enabled);
continue;
unsigned int digest_size; /* digest size for the current hash algorithm */
enum verity_mode mode; /* mode for handling verification errors */
enum verity_mode error_mode;/* mode for handling I/O errors */
- unsigned int corrupted_errs;/* Number of errors for corrupted blocks */
+ atomic_t corrupted_errs;/* Number of errors for corrupted blocks */
struct workqueue_struct *verify_wq;
return ERR_PTR(-ENOMEM);
refcount_set(&td->count, 1);
- bdev_file = bdev_file_open_by_dev(dev, mode, _dm_claim_ptr, NULL);
+ /*
+ * Open the backing device with kernel rather than caller
+ * credentials. Otherwise the caller's credentials would be
+ * pinned in bdev_file->f_cred until the table device is closed.
+ * That would keep the caller's thread keyring alive long beyond the
+ * lifetime of the caller, breaking userspace expectation (e.g.
+ * cryptsetup(8) leaking the LUKS volume key).
+ */
+ scoped_with_kernel_creds()
+ bdev_file = bdev_file_open_by_dev(dev, mode, _dm_claim_ptr, NULL);
if (IS_ERR(bdev_file)) {
r = PTR_ERR(bdev_file);
goto out_free_td;
return 0;
msb->zone_count = msb->block_count / MS_BLOCKS_IN_ZONE;
+ if (msb->block_count > MS_MAX_ZONES * MS_BLOCKS_IN_ZONE) {
+ pr_err("Too many blocks: %d\n", msb->block_count);
+ return -EINVAL;
+ }
msb->logical_block_count = msb->zone_count * 496 - 2;
msb->used_blocks_bitmap = bitmap_zalloc(msb->block_count, GFP_KERNEL);
{
struct mmc_rpmb_data *rpmb = dev_get_drvdata(dev);
- rpmb_dev_unregister(rpmb->rdev);
mmc_blk_put(rpmb->md);
ida_free(&mmc_rpmb_ida, rpmb->id);
kfree(rpmb);
}
static void mmc_blk_remove_rpmb_part(struct mmc_rpmb_data *rpmb)
-
{
+ rpmb_dev_unregister(rpmb->rdev);
cdev_device_del(&rpmb->chrdev, &rpmb->dev);
put_device(&rpmb->dev);
}
{
if (!mem)
return;
- while (mem->cnt--)
- __free_pages(mem->arr[mem->cnt].page,
- mem->arr[mem->cnt].order);
+ for (unsigned int i = 0; i < mem->cnt; i++)
+ __free_pages(mem->arr[i].page,
+ mem->arr[i].order);
kfree(mem);
}
unsigned long page_cnt = 0;
unsigned long limit = nr_free_buffer_pages() >> 4;
struct mmc_test_mem *mem;
+ unsigned int idx = 0;
if (max_page_cnt > limit)
max_page_cnt = limit;
goto out_free;
break;
}
- mem->arr[mem->cnt].page = page;
- mem->arr[mem->cnt].order = order;
- mem->cnt += 1;
+ mem->arr[idx].page = page;
+ mem->arr[idx].order = order;
+ idx += 1;
if (max_page_cnt <= (1UL << order))
break;
max_page_cnt -= 1UL << order;
page_cnt += 1UL << order;
- if (mem->cnt >= max_segs) {
+ if (idx >= mem->cnt) {
if (page_cnt < min_page_cnt)
goto out_free;
break;
}
}
+ mem->cnt = idx;
+
return mem;
out_free:
+ mem->cnt = idx;
mmc_test_free_mem(mem);
return NULL;
}
dev_dbg(mmc_dev(host->mmc), "change pinctrl state for uhs %d\n", uhs);
- if (IS_ERR(imx_data->pinctrl) ||
- IS_ERR(imx_data->pins_100mhz) ||
- IS_ERR(imx_data->pins_200mhz))
+ if (IS_ERR(imx_data->pinctrl))
return -EINVAL;
switch (uhs) {
case MMC_TIMING_UHS_SDR50:
case MMC_TIMING_UHS_DDR50:
+ if (IS_ERR(imx_data->pins_100mhz))
+ return -EINVAL;
pinctrl = imx_data->pins_100mhz;
break;
case MMC_TIMING_UHS_SDR104:
case MMC_TIMING_MMC_HS200:
case MMC_TIMING_MMC_HS400:
+ if (IS_ERR(imx_data->pins_200mhz))
+ return -EINVAL;
pinctrl = imx_data->pins_200mhz;
break;
default:
return pinctrl_select_state(imx_data->pinctrl, pinctrl);
}
+static void esdhc_set_dll_override(struct sdhci_host *host)
+{
+ struct sdhci_pltfm_host *pltfm_host = sdhci_priv(host);
+ struct pltfm_imx_data *imx_data = sdhci_pltfm_priv(pltfm_host);
+ struct esdhc_platform_data *boarddata = &imx_data->boarddata;
+ u32 v;
+
+ if (!boarddata->delay_line)
+ return;
+
+ v = boarddata->delay_line << ESDHC_DLL_OVERRIDE_VAL_SHIFT |
+ (1 << ESDHC_DLL_OVERRIDE_EN_SHIFT);
+ if (is_imx53_esdhc(imx_data))
+ v <<= 1;
+ writel(v, host->ioaddr + ESDHC_DLL_CTRL);
+}
+
/*
* For HS400 eMMC, there is a data_strobe line. This signal is generated
* by the device and used for data output and CRC status response output
u32 m;
struct sdhci_pltfm_host *pltfm_host = sdhci_priv(host);
struct pltfm_imx_data *imx_data = sdhci_pltfm_priv(pltfm_host);
- struct esdhc_platform_data *boarddata = &imx_data->boarddata;
/* disable ddr mode and disable HS400 mode */
m = readl(host->ioaddr + ESDHC_MIX_CTRL);
m |= ESDHC_MIX_CTRL_DDREN;
writel(m, host->ioaddr + ESDHC_MIX_CTRL);
imx_data->is_ddr = 1;
- if (boarddata->delay_line) {
- u32 v;
- v = boarddata->delay_line <<
- ESDHC_DLL_OVERRIDE_VAL_SHIFT |
- (1 << ESDHC_DLL_OVERRIDE_EN_SHIFT);
- if (is_imx53_esdhc(imx_data))
- v <<= 1;
- writel(v, host->ioaddr + ESDHC_DLL_CTRL);
- }
+ esdhc_set_dll_override(host);
break;
case MMC_TIMING_MMC_HS400:
m |= ESDHC_MIX_CTRL_DDREN | ESDHC_MIX_CTRL_HS400_EN;
* 2, make sure the pm_runtime_force_resume() in sdhci_esdhc_resume() really
* invoke its ->runtime_resume callback (needs_force_resume = 1).
*/
- pm_runtime_get_sync(dev);
+ ret = pm_runtime_resume_and_get(dev);
+ if (ret)
+ return ret;
if ((imx_data->socdata->flags & ESDHC_FLAG_STATE_LOST_IN_LPMODE) &&
(host->tuning_mode != SDHCI_TUNING_MODE_1)) {
* to save the tuning delay value just in case the usdhc
* lost power during system PM.
*/
- if (mmc_card_keep_power(host->mmc) && mmc_card_wake_sdio_irq(host->mmc) &&
- esdhc_is_usdhc(imx_data))
+ if (mmc_card_keep_power(host->mmc) && esdhc_is_usdhc(imx_data))
sdhc_esdhc_tuning_save(host);
+ /* The irqs of imx are not shared. It is safe to disable */
+ disable_irq(host->irq);
+
if (device_may_wakeup(dev)) {
- /* The irqs of imx are not shared. It is safe to disable */
- disable_irq(host->irq);
- ret = sdhci_enable_irq_wakeups(host);
- if (!ret)
+ if (!sdhci_enable_irq_wakeups(host))
dev_warn(dev, "Failed to enable irq wakeup\n");
} else {
/*
* other function like GPIO function to save power in PM,
* which finally block the SDIO wakeup function.
*/
- ret = pinctrl_pm_select_sleep_state(dev);
- if (ret)
- return ret;
+ if (pinctrl_pm_select_sleep_state(dev))
+ dev_warn(dev, "Failed to select sleep pinctrl state\n");
}
- ret = mmc_gpio_set_cd_wake(host->mmc, true);
+ if (mmc_gpio_set_cd_wake(host->mmc, true))
+ dev_warn(dev, "Failed to enable cd wake\n");
/*
* Make sure invoke runtime_suspend to gate off clock.
*/
pm_runtime_force_suspend(dev);
- return ret;
+ return 0;
}
static int sdhci_esdhc_resume(struct device *dev)
struct pltfm_imx_data *imx_data = sdhci_pltfm_priv(pltfm_host);
int ret;
- pm_runtime_force_resume(dev);
+ if (!device_may_wakeup(dev)) {
+ ret = esdhc_change_pinstate(host, host->timing);
+ if (ret)
+ dev_warn(dev, "Failed to restore pinctrl state\n");
+ }
- ret = mmc_gpio_set_cd_wake(host->mmc, false);
+ ret = pm_runtime_force_resume(dev);
if (ret)
return ret;
+ mmc_gpio_set_cd_wake(host->mmc, false);
+
/* re-initialize hw state in case it's lost in low power mode */
sdhci_esdhc_imx_hwinit(host);
- if (host->irq_wake_enabled) {
+ if (host->irq_wake_enabled)
sdhci_disable_irq_wakeups(host);
- enable_irq(host->irq);
- }
+
+ enable_irq(host->irq);
/*
* restore the saved tuning delay value for the device which keep
* power during system PM.
*/
- if (mmc_card_keep_power(host->mmc) && mmc_card_wake_sdio_irq(host->mmc) &&
- esdhc_is_usdhc(imx_data))
+ if (mmc_card_keep_power(host->mmc) && esdhc_is_usdhc(imx_data)) {
sdhc_esdhc_tuning_restore(host);
+ /*
+ * Restore DLL override for DDR modes. hwinit unconditionally
+ * clears ESDHC_DLL_CTRL, but the card is still in DDR mode.
+ */
+ if (host->timing == MMC_TIMING_UHS_DDR50 ||
+ host->timing == MMC_TIMING_MMC_DDR52)
+ esdhc_set_dll_override(host);
+ }
+
pm_runtime_put_autosuspend(dev);
- return ret;
+ return 0;
}
static int sdhci_esdhc_runtime_suspend(struct device *dev)
return err;
priv->bus_clk = devm_clk_get(dev, "bus");
- if (!IS_ERR(priv->bus_clk))
- clk_prepare_enable(priv->bus_clk);
+ if (!IS_ERR(priv->bus_clk)) {
+ err = clk_prepare_enable(priv->bus_clk);
+ if (err)
+ goto err_clk;
+ }
}
err = mmc_of_parse(host->mmc);
if (err)
- goto err_clk;
+ goto err_bus_clk;
sdhci_get_of_property(pdev);
if (pltfm_data->init) {
err = pltfm_data->init(&pdev->dev, host, priv);
if (err)
- goto err_clk;
+ goto err_bus_clk;
}
#ifdef CONFIG_ACPI
err_rpm:
pm_runtime_disable(dev);
pm_runtime_put_noidle(dev);
+err_bus_clk:
+ clk_disable_unprepare(priv->bus_clk);
err_clk:
clk_disable_unprepare(pltfm_host->clk);
- clk_disable_unprepare(priv->bus_clk);
clk_bulk_disable_unprepare(priv->num_other_clks, priv->other_clks);
return err;
}
return linear_length;
}
-static void __vub300_command_response(struct vub300_mmc_host *vub300,
+static bool __vub300_command_response(struct vub300_mmc_host *vub300,
struct mmc_command *cmd,
struct mmc_data *data, int data_length)
{
msecs_to_jiffies(msec_timeout));
if (respretval == 0) { /* TIMED OUT */
/* we don't know which of "out" and "res" if any failed */
- int result;
vub300->usb_timed_out = 1;
usb_kill_urb(vub300->command_out_urb);
usb_kill_urb(vub300->command_res_urb);
cmd->error = -ETIMEDOUT;
- result = usb_lock_device_for_reset(vub300->udev,
- vub300->interface);
- if (result == 0) {
- result = usb_reset_device(vub300->udev);
- usb_unlock_device(vub300->udev);
- }
+ return true;
} else if (respretval < 0) {
/* we don't know which of "out" and "res" if any failed */
usb_kill_urb(vub300->command_out_urb);
} else {
cmd->error = -EINVAL;
}
+
+ return false;
}
static void construct_request_response(struct vub300_mmc_host *vub300,
struct mmc_request *req = vub300->req;
struct mmc_command *cmd = vub300->cmd;
struct mmc_data *data = vub300->data;
+ bool reset_device;
int data_length;
mutex_lock(&vub300->cmd_mutex);
init_completion(&vub300->command_complete);
data_length = __command_read_data(vub300, cmd, data);
else
data_length = __command_write_data(vub300, cmd, data);
- __vub300_command_response(vub300, cmd, data, data_length);
+ reset_device = __vub300_command_response(vub300, cmd,
+ data, data_length);
vub300->req = NULL;
vub300->cmd = NULL;
vub300->data = NULL;
if (cmd->error == -ENOMEDIUM)
check_vub300_port_status(vub300);
mutex_unlock(&vub300->cmd_mutex);
+ if (reset_device) {
+ int result;
+
+ result = usb_lock_device_for_reset(vub300->udev,
+ vub300->interface);
+ if (result == 0) {
+ result = usb_reset_device(vub300->udev);
+ usb_unlock_device(vub300->udev);
+ }
+ }
mmc_request_done(vub300->mmc, req);
kref_put(&vub300->kref, vub300_delete);
return;
interface_to_InterfaceNumber(interface));
retval = mmc_add_host(mmc);
if (retval)
- goto err_delete_timer;
+ goto err_stop_io;
return 0;
-err_delete_timer:
- timer_delete_sync(&vub300->inactivity_timer);
+err_stop_io:
+ vub300->interface = NULL;
+ kref_put(&vub300->kref, vub300_delete);
+
+ return retval;
+
err_free_host:
mmc_free_host(mmc);
/*
data = dev_get_platdata(&spi->dev);
- flash->caps = of_device_get_match_data(&spi->dev);
+ flash->caps = spi_get_device_match_data(spi);
if (!flash->caps)
flash->caps = &mchp23k256_caps;
If unsure, say N.
+config MTD_UCLINUX
+ bool "Generic uClinux RAM/ROM filesystem support"
+ depends on (MTD_RAM=y || MTD_ROM=y) && (!MMU || COLDFIRE)
+ help
+ Map driver to support image based filesystems for uClinux.
+
config MTD_PLATRAM
tristate "Map driver for platform device RAM (mtd-ram)"
select MTD_RAM
obj-$(CONFIG_MTD_SCx200_DOCFLASH)+= scx200_docflash.o
obj-$(CONFIG_MTD_SOLUTIONENGINE)+= solutionengine.o
obj-$(CONFIG_MTD_PCI) += pci.o
+obj-$(CONFIG_MTD_UCLINUX) += uclinux.o
obj-$(CONFIG_MTD_SCB2_FLASH) += scb2_flash.o
obj-$(CONFIG_MTD_PLATRAM) += plat-ram.o
obj-$(CONFIG_MTD_VMU) += vmu-flash.o
--- /dev/null
+/****************************************************************************/
+
+/*
+ * uclinux.c -- generic memory mapped MTD driver for uclinux
+ *
+ * (C) Copyright 2002, Greg Ungerer (gerg@snapgear.com)
+ *
+ * License: GPL
+ */
+
+/****************************************************************************/
+
+#include <linux/moduleparam.h>
+#include <linux/types.h>
+#include <linux/init.h>
+#include <linux/kernel.h>
+#include <linux/fs.h>
+#include <linux/mm.h>
+#include <linux/major.h>
+#include <linux/mtd/mtd.h>
+#include <linux/mtd/map.h>
+#include <linux/mtd/partitions.h>
+#include <asm/io.h>
+#include <asm/sections.h>
+
+/****************************************************************************/
+
+#ifdef CONFIG_MTD_ROM
+#define MAP_NAME "rom"
+#else
+#define MAP_NAME "ram"
+#endif
+
+static struct map_info uclinux_ram_map = {
+ .name = MAP_NAME,
+ .size = 0,
+};
+
+static unsigned long physaddr = -1;
+module_param(physaddr, ulong, S_IRUGO);
+
+static struct mtd_info *uclinux_ram_mtdinfo;
+
+/****************************************************************************/
+
+static const struct mtd_partition uclinux_romfs[] = {
+ { .name = "ROMfs" }
+};
+
+#define NUM_PARTITIONS ARRAY_SIZE(uclinux_romfs)
+
+/****************************************************************************/
+
+static int uclinux_point(struct mtd_info *mtd, loff_t from, size_t len,
+ size_t *retlen, void **virt, resource_size_t *phys)
+{
+ struct map_info *map = mtd->priv;
+ *virt = map->virt + from;
+ if (phys)
+ *phys = map->phys + from;
+ *retlen = len;
+ return(0);
+}
+
+/****************************************************************************/
+
+static int __init uclinux_mtd_init(void)
+{
+ struct mtd_info *mtd;
+ struct map_info *mapp;
+
+ mapp = &uclinux_ram_map;
+
+ if (physaddr == -1)
+ mapp->phys = (resource_size_t)__bss_stop;
+ else
+ mapp->phys = physaddr;
+
+ if (!mapp->size)
+ mapp->size = PAGE_ALIGN(ntohl(*((unsigned long *)(mapp->phys + 8))));
+ mapp->bankwidth = 4;
+
+ printk("uclinux[mtd]: probe address=0x%x size=0x%x\n",
+ (int) mapp->phys, (int) mapp->size);
+
+ /*
+ * The filesystem is guaranteed to be in direct mapped memory. It is
+ * directly following the kernels own bss region. Following the same
+ * mechanism used by architectures setting up traditional initrds we
+ * use phys_to_virt to get the virtual address of its start.
+ */
+ mapp->virt = phys_to_virt(mapp->phys);
+
+ if (mapp->virt == 0) {
+ printk("uclinux[mtd]: no virtual mapping?\n");
+ return(-EIO);
+ }
+
+ simple_map_init(mapp);
+
+ mtd = do_map_probe("map_" MAP_NAME, mapp);
+ if (!mtd) {
+ printk("uclinux[mtd]: failed to find a mapping?\n");
+ return(-ENXIO);
+ }
+
+ mtd->owner = THIS_MODULE;
+ mtd->_point = uclinux_point;
+ mtd->priv = mapp;
+
+ uclinux_ram_mtdinfo = mtd;
+ mtd_device_register(mtd, uclinux_romfs, NUM_PARTITIONS);
+
+ return(0);
+}
+device_initcall(uclinux_mtd_init);
+
+/****************************************************************************/
if (item->concat) {
mtd_device_unregister(mtd);
kfree(mtd->name);
- mtd_concat_destroy(mtd);
mtd_virt_concat_put_mtd_devices(item->concat);
+ mtd_concat_destroy(mtd);
}
}
}
if (concat->mtd.name) {
del_mtd_device(&concat->mtd);
kfree(concat->mtd.name);
- mtd_concat_destroy(&concat->mtd);
mtd_virt_concat_put_mtd_devices(item->concat);
+ mtd_concat_destroy(&concat->mtd);
}
for (idx = 0; idx < item->count; idx++)
if (concat->mtd.name) {
ret = memcmp(concat->mtd.name, name, name_sz);
- if (ret == 0)
+ if (ret == 0) {
+ kfree(name);
continue;
+ }
}
mtd = mtd_concat_create(concat->subdev, concat->num_subdev, name);
if (!mtd) {
device_destroy(&mtd_class, index + 1);
}
+/*
+ * No-op device release used in add_mtd_device() error paths.
+ * Prevents mtd_release() from being called via device_release(),
+ * which would free the mtd_info that the caller still manages.
+ */
+static void mtd_dev_release_nop(struct device *dev)
+{
+}
+
static void mtd_device_release(struct kref *kref)
{
struct mtd_info *mtd = container_of(kref, struct mtd_info, refcnt);
mtd_check_of_node(mtd);
of_node_get(mtd_get_of_node(mtd));
error = device_register(&mtd->dev);
- if (error) {
- put_device(&mtd->dev);
+ if (error)
goto fail_added;
- }
/* Add the nvmem provider */
error = mtd_nvmem_add(mtd);
return 0;
fail_nvmem_add:
- device_unregister(&mtd->dev);
+ device_del(&mtd->dev);
fail_added:
+ /*
+ * Clear type and set nop release to prevent mtd_release() ->
+ * release_mtd_partition() -> free_partition() from freeing mtd.
+ * The caller handles cleanup on failure.
+ */
+ mtd->dev.type = NULL;
+ mtd->dev.release = mtd_dev_release_nop;
+ put_device(&mtd->dev);
of_node_put(mtd_get_of_node(mtd));
fail_devname:
idr_remove(&mtd_idr, i);
part->name, parent_size - child->part.offset,
child->part.size);
/* register to preserve ordering */
+ child->part.offset = 0;
+ child->part.size = 0;
+ child->erasesize = parent->erasesize;
goto out_register;
}
}
if (length <= 0)
return -EINVAL;
+ if (offset < 0 || offset >= (long long)parent_size)
+ return -EINVAL;
+
+ if ((u64)offset + (u64)length > parent_size)
+ return -EINVAL;
memset(&part, 0, sizeof(part));
part.name = name;
part.size = length;
char *page_buf;
char *oob_buf;
+ struct dentry *debugfs_stats;
};
struct mtdswap_oobdata {
if (IS_ERR_OR_NULL(root))
return -1;
- debugfs_create_file("mtdswap_stats", S_IRUSR, root, d, &mtdswap_fops);
+ d->debugfs_stats = debugfs_create_file("mtdswap_stats", 0400, root,
+ d, &mtdswap_fops);
return 0;
}
{
struct mtdswap_dev *d = MTDSWAP_MBD_TO_MTDSWAP(dev);
+ debugfs_remove(d->debugfs_stats);
del_mtd_blktrans_dev(dev);
mtdswap_cleanup(d);
kfree(d);
[ECC_DECIRQ_STA] = 0x144,
};
-static inline void mtk_ecc_wait_idle(struct mtk_ecc *ecc,
- enum mtk_ecc_operation op)
+static inline int mtk_ecc_wait_idle(struct mtk_ecc *ecc,
+ enum mtk_ecc_operation op)
{
struct device *dev = ecc->dev;
u32 val;
if (ret)
dev_warn(dev, "%s NOT idle\n",
op == ECC_ENCODE ? "encoder" : "decoder");
+
+ return ret;
}
static irqreturn_t mtk_ecc_irq(int irq, void *id)
{
struct platform_device *pdev;
struct mtk_ecc *ecc;
+ int ret;
pdev = of_find_device_by_node(np);
if (!pdev)
return ERR_PTR(-EPROBE_DEFER);
}
- clk_prepare_enable(ecc->clk);
+ ret = clk_prepare_enable(ecc->clk);
+ if (ret) {
+ put_device(&pdev->dev);
+ return ERR_PTR(ret);
+ }
+
mtk_ecc_hw_init(ecc);
return ecc;
return ret;
}
- mtk_ecc_wait_idle(ecc, op);
+ ret = mtk_ecc_wait_idle(ecc, op);
+ if (ret) {
+ mutex_unlock(&ecc->lock);
+ return ret;
+ }
ret = mtk_ecc_config(ecc, config);
if (ret) {
if (ret)
goto timeout;
- mtk_ecc_wait_idle(ecc, ECC_ENCODE);
+ ret = mtk_ecc_wait_idle(ecc, ECC_ENCODE);
+ if (ret)
+ goto timeout;
/* Program ECC bytes to OOB: per sector oob = FDM + ECC + SPARE */
len = (config->strength * ecc->caps->parity_bits + 7) >> 3;
} while (!(status & S5PC110_DMA_TRANS_STATUS_TD) &&
time_before(jiffies, timeout));
+ if (!(status & S5PC110_DMA_TRANS_STATUS_TD))
+ return -ETIMEDOUT;
+
writel(S5PC110_DMA_TRANS_CMD_TDC, base + S5PC110_DMA_TRANS_CMD);
return 0;
writel(S5PC110_DMA_TRANS_CMD_TR, base + S5PC110_DMA_TRANS_CMD);
- wait_for_completion_timeout(&onenand->complete, msecs_to_jiffies(20));
+ if (!wait_for_completion_timeout(&onenand->complete,
+ msecs_to_jiffies(20)))
+ return -ETIMEDOUT;
return 0;
}
config MTD_NAND_NDFC
tristate "IBM/MCC 4xx NAND controller"
depends on 44x || COMPILE_TEST
+ depends on OF
select MTD_NAND_ECC_SW_HAMMING
select MTD_NAND_ECC_SW_HAMMING_SMC
help
return check_erased_page(chip, buf);
}
- if (ctrl->nand_stat != IFC_NAND_EVTER_STAT_OPC)
+ if (!ctrl->nand_stat) {
mtd->ecc_stats.failed++;
+ return -ETIMEDOUT;
+ }
+
+ if (ctrl->nand_stat != IFC_NAND_EVTER_STAT_OPC) {
+ mtd->ecc_stats.failed++;
+ return -EIO;
+ }
return nctrl->max_bitflips;
}
{
struct platform_device *pdev;
struct ingenic_ecc *ecc;
+ int ret;
pdev = of_find_device_by_node(np);
if (!pdev)
}
ecc = platform_get_drvdata(pdev);
- clk_prepare_enable(ecc->clk);
+ ret = clk_prepare_enable(ecc->clk);
+ if (ret) {
+ put_device(&pdev->dev);
+ return ERR_PTR(ret);
+ }
return ecc;
}
struct lpc32xx_nand_host *host = nand_get_controller_data(chip);
struct dma_async_tx_descriptor *desc;
int flags = DMA_CTRL_ACK | DMA_PREP_INTERRUPT;
+ unsigned long time_left;
int res;
sg_init_one(&host->sgl, mem, len);
flags);
if (!desc) {
dev_err(mtd->dev.parent, "Failed to prepare slave sg\n");
+ res = -ENXIO;
goto out1;
}
dmaengine_submit(desc);
dma_async_issue_pending(host->dma_chan);
- wait_for_completion_timeout(&host->comp_dma, msecs_to_jiffies(1000));
+ time_left = wait_for_completion_timeout(&host->comp_dma,
+ msecs_to_jiffies(1000));
+ if (!time_left) {
+ dmaengine_terminate_sync(host->dma_chan);
+ res = -ETIMEDOUT;
+ goto out1;
+ }
dma_unmap_sg(host->dma_chan->device->dev, &host->sgl, 1,
DMA_BIDIRECTIONAL);
out1:
dma_unmap_sg(host->dma_chan->device->dev, &host->sgl, 1,
DMA_BIDIRECTIONAL);
- return -ENXIO;
+ return res;
}
static int lpc32xx_read_page(struct nand_chip *chip, uint8_t *buf,
struct dma_async_tx_descriptor *desc;
int flags = DMA_CTRL_ACK | DMA_PREP_INTERRUPT;
int res;
+ unsigned long time_left;
host->dma_slave_config.direction = dir;
host->dma_slave_config.src_addr = dma;
dmaengine_submit(desc);
dma_async_issue_pending(host->dma_chan);
- wait_for_completion_timeout(&host->comp, msecs_to_jiffies(1000));
+ time_left = wait_for_completion_timeout(&host->comp,
+ msecs_to_jiffies(1000));
+ if (!time_left) {
+ dmaengine_terminate_sync(host->dma_chan);
+ res = -ETIMEDOUT;
+ } else {
+ res = 0;
+ }
dma_unmap_sg(host->dma_chan->device->dev, &host->sgl, 1,
DMA_BIDIRECTIONAL);
- return 0;
+ return res;
out1:
dma_unmap_sg(host->dma_chan->device->dev, &host->sgl, 1,
DMA_BIDIRECTIONAL);
const __be32 *reg;
u32 ccr;
u32 cs;
- int err, len;
+ int err, len = 0;
/* Read the reg property to get the chip select */
reg = of_get_property(ofdev->dev.of_node, "reg", &len);
bool disable_ecc = false;
bool ecc_failed = false;
unsigned int retry_mode = 0;
- int ret;
+ int ret = 0;
old_stats = mtd->ecc_stats;
nla_total_size(sizeof(__u16)) + /* IFLA_AMT_GATEWAY_PORT */
nla_total_size(sizeof(__u32)) + /* IFLA_AMT_LINK */
nla_total_size(sizeof(__u32)) + /* IFLA_MAX_TUNNELS */
- nla_total_size(sizeof(struct iphdr)) + /* IFLA_AMT_DISCOVERY_IP */
- nla_total_size(sizeof(struct iphdr)) + /* IFLA_AMT_REMOTE_IP */
- nla_total_size(sizeof(struct iphdr)); /* IFLA_AMT_LOCAL_IP */
+ nla_total_size(sizeof(__be32)) + /* IFLA_AMT_DISCOVERY_IP */
+ nla_total_size(sizeof(__be32)) + /* IFLA_AMT_REMOTE_IP */
+ nla_total_size(sizeof(__be32)); /* IFLA_AMT_LOCAL_IP */
}
static int amt_fill_info(struct sk_buff *skb, const struct net_device *dev)
When one end receives the packet it appears on its pair and vice
versa. The vxcan can be used for cross namespace communication.
- In opposite to vcan loopback devices the vxcan only forwards CAN
- frames to its pair and does *not* provide a local echo of sent
- CAN frames. To disable a potential echo in af_can.c the vxcan driver
- announces IFF_ECHO in the interface flags. To have a clean start
- in each namespace the CAN GW hop counter is set to zero.
+ To have a clean start in each namespace the CAN GW hop counter is
+ set to zero.
This driver can also be built as a module. If so, the module
will be called vxcan.
/* Copyright (C) 2007, 2011 Wolfgang Grandegger <wg@grandegger.com>
*
* Copyright (C) 2016-2025 PEAK System-Technik GmbH
- * Author: Stéphane Grosjean <stephane.grosjean@hms-networks.com>
+ * Author: Stéphane Grosjean <s.grosjean@peak-system.fr>
*/
#include <linux/can.h>
/* CAN driver for PEAK System micro-CAN based adapters
*
* Copyright (C) 2003-2025 PEAK System-Technik GmbH
- * Author: Stéphane Grosjean <stephane.grosjean@hms-networks.com>
+ * Author: Stéphane Grosjean <s.grosjean@peak-system.fr>
*/
#ifndef PEAK_CANFD_USER_H
#define PEAK_CANFD_USER_H
* Derived from the PCAN project file driver/src/pcan_pci.c:
*
* Copyright (C) 2001-2025 PEAK System-Technik GmbH
- * Author: Stéphane Grosjean <stephane.grosjean@hms-networks.com>
+ * Author: Stéphane Grosjean <s.grosjean@peak-system.fr>
*/
#include <linux/kernel.h>
#include "peak_canfd_user.h"
-MODULE_AUTHOR("Stéphane Grosjean <stephane.grosjean@hms-networks.com>");
+MODULE_AUTHOR("Stéphane Grosjean <s.grosjean@peak-system.fr>");
MODULE_DESCRIPTION("Socket-CAN driver for PEAK PCAN PCIe/M.2 FD family cards");
MODULE_LICENSE("GPL v2");
* Derived from the PCAN project file driver/src/pcan_pci.c:
*
* Copyright (C) 2001-2025 PEAK System-Technik GmbH
- * Author: Stéphane Grosjean <stephane.grosjean@hms-networks.com>
+ * Author: Stéphane Grosjean <s.grosjean@peak-system.fr>
*/
#include <linux/kernel.h>
#include "sja1000.h"
-MODULE_AUTHOR("Stéphane Grosjean <stephane.grosjean@hms-networks.com>");
+MODULE_AUTHOR("Stéphane Grosjean <s.grosjean@peak-system.fr>");
MODULE_DESCRIPTION("Socket-CAN driver for PEAK PCAN PCI family cards");
MODULE_LICENSE("GPL v2");
* Derived from the PCAN project file driver/src/pcan_pccard.c
*
* Copyright (C) 2006-2025 PEAK System-Technik GmbH
- * Author: Stéphane Grosjean <stephane.grosjean@hms-networks.com>
+ * Author: Stéphane Grosjean <s.grosjean@peak-system.fr>
*/
#include <linux/kernel.h>
#include <linux/module.h>
#include <linux/can/dev.h>
#include "sja1000.h"
-MODULE_AUTHOR("Stéphane Grosjean <stephane.grosjean@hms-networks.com>");
+MODULE_AUTHOR("Stéphane Grosjean <s.grosjean@peak-system.fr>");
MODULE_DESCRIPTION("CAN driver for PEAK-System PCAN-PC Cards");
MODULE_LICENSE("GPL v2");
netdev = dev->nets[i]->netdev;
netdev_info(netdev, "unregister\n");
unregister_netdev(netdev);
- free_candev(netdev);
}
}
unlink_all_urbs(dev);
+ for (i = 0; i < dev->net_count; i++) {
+ if (dev->nets[i])
+ free_candev(dev->nets[i]->netdev);
+ }
kfree(dev);
}
}
* Derived from the PCAN project file driver/src/pcan_usb.c
*
* Copyright (C) 2003-2025 PEAK System-Technik GmbH
- * Author: Stéphane Grosjean <stephane.grosjean@hms-networks.com>
+ * Author: Stéphane Grosjean <s.grosjean@peak-system.fr>
*
* Many thanks to Klaus Hitschler <klaus.hitschler@gmx.de>
*/
* Derived from the PCAN project file driver/src/pcan_usb_core.c
*
* Copyright (C) 2003-2025 PEAK System-Technik GmbH
- * Author: Stéphane Grosjean <stephane.grosjean@hms-networks.com>
+ * Author: Stéphane Grosjean <s.grosjean@peak-system.fr>
*
* Many thanks to Klaus Hitschler <klaus.hitschler@gmx.de>
*/
#include "pcan_usb_core.h"
-MODULE_AUTHOR("Stéphane Grosjean <stephane.grosjean@hms-networks.com>");
+MODULE_AUTHOR("Stéphane Grosjean <s.grosjean@peak-system.fr>");
MODULE_DESCRIPTION("CAN driver for PEAK-System USB adapters");
MODULE_LICENSE("GPL v2");
* Derived from the PCAN project file driver/src/pcan_usb_core.c
*
* Copyright (C) 2003-2025 PEAK System-Technik GmbH
- * Author: Stéphane Grosjean <stephane.grosjean@hms-networks.com>
+ * Author: Stéphane Grosjean <s.grosjean@peak-system.fr>
*
* Many thanks to Klaus Hitschler <klaus.hitschler@gmx.de>
*/
* CAN driver for PEAK System PCAN-USB FD / PCAN-USB Pro FD adapter
*
* Copyright (C) 2013-2025 PEAK System-Technik GmbH
- * Author: Stéphane Grosjean <stephane.grosjean@hms-networks.com>
+ * Author: Stéphane Grosjean <s.grosjean@peak-system.fr>
*/
#include <linux/ethtool.h>
#include <linux/module.h>
* Derived from the PCAN project file driver/src/pcan_usbpro.c
*
* Copyright (C) 2003-2025 PEAK System-Technik GmbH
- * Author: Stéphane Grosjean <stephane.grosjean@hms-networks.com>
+ * Author: Stéphane Grosjean <s.grosjean@peak-system.fr>
*/
#include <linux/ethtool.h>
#include <linux/module.h>
* Derived from the PCAN project file driver/src/pcan_usbpro_fw.h
*
* Copyright (C) 2003-2025 PEAK System-Technik GmbH
- * Author: Stéphane Grosjean <stephane.grosjean@hms-networks.com>
+ * Author: Stéphane Grosjean <s.grosjean@peak-system.fr>
*/
#ifndef PCAN_USB_PRO_H
#define PCAN_USB_PRO_H
if (!aux_priv)
goto next_auxdev;
+ edev = kzalloc_obj(*edev);
+ if (!edev)
+ goto aux_priv_free;
+ aux_priv->edev = edev;
+ bnxt_set_edev_info(edev, bp);
+
+ ulp = kzalloc_obj(*ulp);
+ if (!ulp)
+ goto edev_free;
+ edev->ulp_tbl = ulp;
+ aux_priv->id = idx;
+
aux_dev = &aux_priv->aux_dev;
aux_dev->id = bp->auxdev_id;
aux_dev->name = bnxt_aux_devices[idx].name;
aux_dev->dev.release = bnxt_aux_dev_release;
rc = auxiliary_device_init(aux_dev);
- if (rc) {
- kfree(aux_priv);
- goto next_auxdev;
- }
+ if (rc)
+ goto ulp_free;
bp->aux_priv[idx] = aux_priv;
/* From this point, all cleanup will happen via the .release
* callback & any error unwinding will need to include a call
* to auxiliary_device_uninit.
*/
- edev = kzalloc_obj(*edev);
- if (!edev)
- goto aux_dev_uninit;
-
- aux_priv->edev = edev;
- bnxt_set_edev_info(edev, bp);
-
- ulp = kzalloc_obj(*ulp);
- if (!ulp)
- goto aux_dev_uninit;
-
- edev->ulp_tbl = ulp;
bp->edev[idx] = edev;
if (idx == BNXT_AUXDEV_RDMA)
bp->ulp_num_msix_want = bnxt_set_dflt_ulp_msix(bp);
- aux_priv->id = idx;
bnxt_auxdev_set_state(bp, idx, BNXT_ADEV_STATE_INIT);
continue;
-aux_dev_uninit:
- auxiliary_device_uninit(aux_dev);
+ulp_free:
+ kfree(ulp);
+edev_free:
+ kfree(edev);
+aux_priv_free:
+ kfree(aux_priv);
next_auxdev:
if (idx == BNXT_AUXDEV_RDMA)
bp->flags &= ~BNXT_FLAG_ROCE_CAP;
dma_free_coherent(dev, size, bp->queues[0].rx_ring, bp->queues[0].rx_ring_dma);
for (q = 0, queue = bp->queues; q < bp->num_queues; ++q, ++queue) {
- kfree(queue->tx_skb);
- queue->tx_skb = NULL;
+ if (queue->tx_skb) {
+ unsigned int dropped = 0, tail;
+
+ for (tail = queue->tx_tail; tail != queue->tx_head;
+ tail++) {
+ if (macb_tx_skb(queue, tail)->skb)
+ dropped++;
+ macb_tx_unmap(bp, macb_tx_skb(queue, tail), 0);
+ }
+
+ queue->stats.tx_dropped += dropped;
+ bp->dev->stats.tx_dropped += dropped;
+
+ kfree(queue->tx_skb);
+ queue->tx_skb = NULL;
+ }
+
+ queue->tx_head = 0;
+ queue->tx_tail = 0;
queue->tx_ring = NULL;
queue->rx_ring = NULL;
}
static int octeon_enable_sriov(struct octeon_device *oct)
{
unsigned int num_vfs_alloced = oct->sriov_info.num_vfs_alloced;
- struct pci_dev *vfdev;
int err;
- u32 u;
if (OCTEON_CN23XX_PF(oct) && num_vfs_alloced) {
err = pci_enable_sriov(oct->pci_dev,
return err;
}
oct->sriov_info.sriov_enabled = 1;
-
- /* init lookup table that maps DPI ring number to VF pci_dev
- * struct pointer
- */
- u = 0;
- vfdev = pci_get_device(PCI_VENDOR_ID_CAVIUM,
- OCTEON_CN23XX_VF_VID, NULL);
- while (vfdev) {
- if (vfdev->is_virtfn &&
- (vfdev->physfn == oct->pci_dev)) {
- oct->sriov_info.dpiring_to_vfpcidev_lut[u] =
- vfdev;
- u += oct->sriov_info.rings_per_vf;
- }
- vfdev = pci_get_device(PCI_VENDOR_ID_CAVIUM,
- OCTEON_CN23XX_VF_VID, vfdev);
- }
}
return num_vfs_alloced;
static int lio_pci_sriov_disable(struct octeon_device *oct)
{
- int u;
-
if (pci_vfs_assigned(oct->pci_dev)) {
dev_err(&oct->pci_dev->dev, "VFs are still assigned to VMs.\n");
return -EPERM;
pci_disable_sriov(oct->pci_dev);
- u = 0;
- while (u < MAX_POSSIBLE_VFS) {
- oct->sriov_info.dpiring_to_vfpcidev_lut[u] = NULL;
- u += oct->sriov_info.rings_per_vf;
- }
-
oct->sriov_info.num_vfs_alloced = 0;
dev_info(&oct->pci_dev->dev, "oct->pf_num:%d disabled VFs\n",
oct->pf_num);
struct lio_trusted_vf trusted_vf;
- /*lookup table that maps DPI ring number to VF pci_dev struct pointer*/
- struct pci_dev *dpiring_to_vfpcidev_lut[MAX_POSSIBLE_VFS];
-
u64 vf_macaddr[MAX_POSSIBLE_VFS];
u16 vf_vlantci[MAX_POSSIBLE_VFS];
#include "octeon_mailbox.h"
#include "cn23xx_pf_device.h"
+static struct pci_dev *lio_vf_pci_dev_by_qno(struct octeon_device *oct, u32 q_no)
+{
+ struct pci_dev *vfdev = NULL;
+ int vfidx;
+
+ if (!oct->sriov_info.rings_per_vf)
+ return NULL;
+
+ if (q_no % oct->sriov_info.rings_per_vf)
+ return NULL;
+
+ vfidx = q_no / oct->sriov_info.rings_per_vf;
+ if (vfidx >= oct->sriov_info.num_vfs_alloced)
+ return NULL;
+
+ while ((vfdev = pci_get_device(PCI_VENDOR_ID_CAVIUM,
+ OCTEON_CN23XX_VF_VID, vfdev))) {
+ if (pci_physfn(vfdev) && pci_physfn(vfdev) == oct->pci_dev &&
+ pci_iov_vf_id(vfdev) == vfidx)
+ return vfdev;
+ }
+
+ return NULL;
+}
+
/**
* octeon_mbox_read:
* @mbox: Pointer mailbox
struct octeon_mbox_cmd *mbox_cmd)
{
struct octeon_device *oct = mbox->oct_dev;
+ struct pci_dev *vfdev;
switch (mbox_cmd->msg.s.cmd) {
case OCTEON_VF_ACTIVE:
dev_info(&oct->pci_dev->dev,
"got a request for FLR from VF that owns DPI ring %u\n",
mbox->q_no);
- pcie_flr(oct->sriov_info.dpiring_to_vfpcidev_lut[mbox->q_no]);
+ vfdev = lio_vf_pci_dev_by_qno(oct, mbox->q_no);
+ if (!vfdev)
+ break;
+
+ pcie_flr(vfdev);
+ pci_dev_put(vfdev);
break;
case OCTEON_PF_CHANGED_VF_MACADDR:
rvu_npc_install_allmulti_entry(rvu, pcifunc, nixlf,
pfvf->rx_chan_base);
} else {
- if (!nix_rx_multicast)
+ if (!nix_rx_multicast && !is_vf(pcifunc))
rvu_npc_enable_allmulti_entry(rvu, pcifunc, nixlf, false);
}
pfvf->rx_chan_base,
pfvf->rx_chan_cnt);
else
- if (!nix_rx_multicast)
+ if (!nix_rx_multicast && !is_vf(pcifunc))
rvu_npc_enable_promisc_entry(rvu, pcifunc, nixlf, false);
return 0;
otx2_sq_free_sqbs(pf);
for (qidx = 0; qidx < otx2_get_total_tx_queues(pf); qidx++) {
sq = &qset->sq[qidx];
- /* Skip freeing Qos queues if they are not initialized */
- if (!sq->sqe)
- continue;
- qmem_free(pf->dev, sq->sqe);
- qmem_free(pf->dev, sq->sqe_ring);
- qmem_free(pf->dev, sq->cpt_resp);
- qmem_free(pf->dev, sq->tso_hdrs);
- qmem_free(pf->dev, sq->timestamps);
- kfree(sq->sg);
+ /* sq->sqe is not initialized for unused QoS queues */
+ if (sq->sqe) {
+ qmem_free(pf->dev, sq->sqe);
+ qmem_free(pf->dev, sq->sqe_ring);
+ qmem_free(pf->dev, sq->cpt_resp);
+ qmem_free(pf->dev, sq->tso_hdrs);
+ qmem_free(pf->dev, sq->timestamps);
+ kfree(sq->sg);
+ }
kfree(sq->sqb_ptrs);
}
}
return err;
err_free_nix_queues:
- otx2_free_sq_res(pf);
otx2_free_cq_res(pf);
otx2_ctx_disable(mbox, NIX_AQ_CTYPE_RQ, false);
err_free_txsch:
otx2_txschq_stop(pf);
err_free_sq_ptrs:
- otx2_sq_free_sqbs(pf);
+ otx2_free_sq_res(pf);
err_free_rq_ptrs:
otx2_free_aura_ptr(pf, AURA_NIX_RQ);
otx2_ctx_disable(mbox, NPA_AQ_CTYPE_POOL, true);
static int otx2_do_set_vf_mac(struct otx2_nic *pf, int vf, const u8 *mac)
{
+ struct npc_get_field_status_req *freq;
+ struct npc_get_field_status_rsp *frsp;
struct npc_install_flow_req *req;
int err;
mutex_lock(&pf->mbox.lock);
+
+ /* Skip installing the DMAC filter if the hardware parser profile
+ * does not support DMAC extraction.
+ */
+ freq = otx2_mbox_alloc_msg_npc_get_field_status(&pf->mbox);
+ if (!freq) {
+ err = -ENOMEM;
+ goto out;
+ }
+
+ freq->field = NPC_DMAC;
+ err = otx2_sync_mbox_msg(&pf->mbox);
+ if (err)
+ goto out;
+
+ frsp = (struct npc_get_field_status_rsp *)otx2_mbox_get_rsp
+ (&pf->mbox.mbox, 0, &freq->hdr);
+ if (IS_ERR(frsp)) {
+ err = PTR_ERR(frsp);
+ goto out;
+ }
+
+ if (!frsp->enable) {
+ netdev_warn(pf->netdev,
+ "VF %d MAC filter not installed: DMAC extraction not supported by parser profile\n",
+ vf);
+ err = -EOPNOTSUPP;
+ goto out;
+ }
+
req = otx2_mbox_alloc_msg_npc_install_flow(&pf->mbox);
if (!req) {
err = -ENOMEM;
if (!is_valid_ether_addr(mac))
return -EINVAL;
- config = &pf->vf_configs[vf];
- ether_addr_copy(config->mac, mac);
-
ret = otx2_do_set_vf_mac(pf, vf, mac);
- if (ret == 0)
- dev_info(&pdev->dev,
- "Load/Reload VF driver\n");
+ if (ret == 0) {
+ config = &pf->vf_configs[vf];
+ ether_addr_copy(config->mac, mac);
+ dev_info(&pdev->dev, "Load/Reload VF driver\n");
+ }
return ret;
}
struct ethtool_fec_hist_range *fec_ranges;
};
+static inline u16 mlx5e_stats_nch_read(const struct mlx5e_priv *priv)
+{
+ /* Pairs with smp_store_release in mlx5e_stats_nch_write(). */
+ return smp_load_acquire(&priv->stats_nch);
+}
+
+static inline void mlx5e_stats_nch_write(struct mlx5e_priv *priv, u16 n)
+{
+ /* Pairs with smp_load_acquire in mlx5e_stats_nch_read(). */
+ smp_store_release(&priv->stats_nch, n);
+}
+
struct mlx5e_dev {
struct net_device *netdev;
struct devlink_port dl_port;
static void mlx5e_hv_vhca_fill_stats(struct mlx5e_priv *priv, void *data,
int buf_len)
{
+ u16 nch = mlx5e_stats_nch_read(priv);
int ch, i = 0;
- for (ch = 0; ch < priv->stats_nch; ch++) {
+ for (ch = 0; ch < nch; ch++) {
void *buf = data + i;
if (WARN_ON_ONCE(buf +
}
static int mlx5e_hv_vhca_stats_buf_size(struct mlx5e_priv *priv)
+{
+ u16 nch = mlx5e_stats_nch_read(priv);
+
+ return sizeof(struct mlx5e_hv_vhca_per_ring_stats) * nch;
+}
+
+static int mlx5e_hv_vhca_stats_buf_max_size(struct mlx5e_priv *priv)
{
return (sizeof(struct mlx5e_hv_vhca_per_ring_stats) *
- priv->stats_nch);
+ max(priv->max_nch, priv->stats_nch));
}
static void mlx5e_hv_vhca_stats_work(struct work_struct *work)
sagent = container_of(dwork, struct mlx5e_hv_vhca_stats_agent, work);
priv = container_of(sagent, struct mlx5e_priv, stats_agent);
buf_len = mlx5e_hv_vhca_stats_buf_size(priv);
- agent = sagent->agent;
+ agent = READ_ONCE(sagent->agent);
buf = sagent->buf;
memset(buf, 0, buf_len);
sagent = &priv->stats_agent;
block->version = MLX5_HV_VHCA_STATS_VERSION;
- block->rings = priv->stats_nch;
+ block->rings = mlx5e_stats_nch_read(priv);
if (!block->command) {
cancel_delayed_work_sync(&priv->stats_agent.work);
void mlx5e_hv_vhca_stats_create(struct mlx5e_priv *priv)
{
- int buf_len = mlx5e_hv_vhca_stats_buf_size(priv);
+ int buf_len = mlx5e_hv_vhca_stats_buf_max_size(priv);
struct mlx5_hv_vhca_agent *agent;
priv->stats_agent.buf = kvzalloc(buf_len, GFP_KERNEL);
if (!priv->stats_agent.buf)
return;
+ INIT_DELAYED_WORK(&priv->stats_agent.work, mlx5e_hv_vhca_stats_work);
+
agent = mlx5_hv_vhca_agent_create(priv->mdev->hv_vhca,
MLX5_HV_VHCA_AGENT_STATS,
mlx5e_hv_vhca_stats_control, NULL,
mlx5e_hv_vhca_stats_cleanup,
- priv);
+ priv,
+ &priv->stats_agent.agent);
if (IS_ERR_OR_NULL(agent)) {
if (IS_ERR(agent))
agent);
kvfree(priv->stats_agent.buf);
- return;
+ priv->stats_agent.buf = NULL;
}
-
- priv->stats_agent.agent = agent;
- INIT_DELAYED_WORK(&priv->stats_agent.work, mlx5e_hv_vhca_stats_work);
}
void mlx5e_hv_vhca_stats_destroy(struct mlx5e_priv *priv)
{
- if (IS_ERR_OR_NULL(priv->stats_agent.agent))
+ struct mlx5_hv_vhca_agent *agent;
+
+ agent = READ_ONCE(priv->stats_agent.agent);
+ if (IS_ERR_OR_NULL(agent))
return;
- mlx5_hv_vhca_agent_destroy(priv->stats_agent.agent);
+ mlx5_hv_vhca_agent_destroy(agent);
+ WRITE_ONCE(priv->stats_agent.agent, NULL);
kvfree(priv->stats_agent.buf);
+ priv->stats_agent.buf = NULL;
}
GFP_KERNEL, cpu_to_node(cpu));
if (!priv->channel_stats[ix])
return -ENOMEM;
- priv->stats_nch++;
+ mlx5e_stats_nch_write(priv, priv->stats_nch + 1);
return 0;
}
void mlx5e_fold_sw_stats64(struct mlx5e_priv *priv, struct rtnl_link_stats64 *s)
{
+ u16 nch = mlx5e_stats_nch_read(priv);
int i;
- for (i = 0; i < priv->stats_nch; i++) {
+ for (i = 0; i < nch; i++) {
struct mlx5e_channel_stats *channel_stats = priv->channel_stats[i];
struct mlx5e_rq_stats *xskrq_stats = &channel_stats->xskrq;
struct mlx5e_rq_stats *rq_stats = &channel_stats->rq;
struct mlx5e_rq_stats *xskrq_stats;
struct mlx5e_rq_stats *rq_stats;
- if (mlx5e_is_uplink_rep(priv) || !priv->stats_nch)
+ if (mlx5e_is_uplink_rep(priv) || !mlx5e_stats_nch_read(priv))
return;
channel_stats = priv->channel_stats[i];
struct mlx5e_priv *priv = netdev_priv(dev);
struct mlx5e_sq_stats *sq_stats;
- if (!priv->stats_nch)
+ if (!mlx5e_stats_nch_read(priv))
return;
/* no special case needed for ptp htb etc since txq2sq_stats is kept up
struct netdev_queue_stats_tx *tx)
{
struct mlx5e_priv *priv = netdev_priv(dev);
+ u16 nch = mlx5e_stats_nch_read(priv);
struct mlx5e_ptp *ptp_channel;
int i, tc;
rx->hw_gro_wire_packets = 0;
rx->hw_gro_wire_bytes = 0;
- for (i = priv->channels.params.num_channels; i < priv->stats_nch; i++) {
+ for (i = priv->channels.params.num_channels; i < nch; i++) {
struct netdev_queue_stats_rx rx_i = {0};
mlx5e_get_queue_stats_rx(dev, i, &rx_i);
tx->stop = 0;
tx->wake = 0;
- for (i = 0; i < priv->stats_nch; i++) {
+ for (i = 0; i < nch; i++) {
struct mlx5e_channel_stats *channel_stats = priv->channel_stats[i];
/* handle two cases:
static MLX5E_DECLARE_STATS_GRP_OP_UPDATE_STATS(sw)
{
struct mlx5e_sw_stats *s = &priv->stats.sw;
+ u16 nch = mlx5e_stats_nch_read(priv);
int i;
memset(s, 0, sizeof(*s));
for (i = 0; i < priv->channels.num; i++) /* for active channels only */
mlx5e_stats_update_stats_rq_page_pool(priv->channels.c[i]);
- for (i = 0; i < priv->stats_nch; i++) {
+ for (i = 0; i < nch; i++) {
struct mlx5e_channel_stats *channel_stats =
priv->channel_stats[i];
static MLX5E_DECLARE_STATS_GRP_OP_NUM_STATS(channels)
{
- int max_nch = priv->stats_nch;
+ int max_nch = mlx5e_stats_nch_read(priv);
return (NUM_RQ_STATS * max_nch) +
(NUM_CH_STATS * max_nch) +
static MLX5E_DECLARE_STATS_GRP_OP_FILL_STRS(channels)
{
+ int max_nch = mlx5e_stats_nch_read(priv);
bool is_xsk = priv->xsk.ever_used;
- int max_nch = priv->stats_nch;
int i, j, tc;
for (i = 0; i < max_nch; i++)
static MLX5E_DECLARE_STATS_GRP_OP_FILL_STATS(channels)
{
+ int max_nch = mlx5e_stats_nch_read(priv);
bool is_xsk = priv->xsk.ever_used;
- int max_nch = priv->stats_nch;
int i, j, tc;
for (i = 0; i < max_nch; i++)
mlx5_devcom_for_each_peer_entry(devcom, peer_esw, pos) {
i = mlx5_lag_get_dev_seq(peer_esw->dev);
+ if (i < 0)
+ continue;
+
list_for_each_entry_safe(flow, tmp, &esw->offloads.peer_flows[i], peer[i])
mlx5e_tc_del_fdb_peers_flow(flow);
}
static void mlx5i_grp_sw_update_stats(struct mlx5e_priv *priv)
{
+ u16 nch = mlx5e_stats_nch_read(priv);
struct rtnl_link_stats64 s = {};
int i, j;
- for (i = 0; i < priv->stats_nch; i++) {
+ for (i = 0; i < nch; i++) {
struct mlx5e_channel_stats *channel_stats;
struct mlx5e_rq_stats *rq_stats;
struct mlx5_lag *ldev = mpesww->lag;
devcom = mlx5_lag_get_devcom_comp(ldev);
- if (!devcom)
- return;
+ if (!devcom) {
+ mpesww->result = -ENODEV;
+ goto complete;
+ }
mlx5_devcom_comp_lock(devcom);
mlx5_mpesw_sd_devcoms_lock(ldev);
mutex_unlock(&ldev->lock);
mlx5_mpesw_sd_devcoms_unlock(ldev);
mlx5_devcom_comp_unlock(devcom);
+complete:
complete(&mpesww->comp);
}
}
return 0;
err:
- mlx5_lag_for_each_reverse(j, i, 0, ldev, filter) {
+ mlx5_lag_for_each_reverse(j, i - 1, 0, ldev, filter) {
struct mlx5_eswitch *slave_esw;
if (j == master_idx)
return mlx5_hv_vhca_agent_create(hv_vhca, MLX5_HV_VHCA_AGENT_CONTROL,
NULL,
mlx5_hv_vhca_control_agent_invalidate,
- NULL, NULL);
+ NULL, NULL, NULL);
}
static void mlx5_hv_vhca_control_agent_destroy(struct mlx5_hv_vhca_agent *agent)
void (*invalidate)(struct mlx5_hv_vhca_agent*,
u64 block_mask),
void (*cleaup)(struct mlx5_hv_vhca_agent *agent),
- void *priv)
+ void *priv,
+ struct mlx5_hv_vhca_agent **ctx_update)
{
struct mlx5_hv_vhca_agent *agent;
agent->invalidate = invalidate;
agent->cleanup = cleaup;
+ if (ctx_update)
+ WRITE_ONCE(*ctx_update, agent);
+
mutex_lock(&hv_vhca->agents_lock);
hv_vhca->agents[type] = agent;
mutex_unlock(&hv_vhca->agents_lock);
void (*invalidate)(struct mlx5_hv_vhca_agent*,
u64 block_mask),
void (*cleanup)(struct mlx5_hv_vhca_agent *agent),
- void *context);
+ void *context,
+ struct mlx5_hv_vhca_agent **ctx_update);
void mlx5_hv_vhca_agent_destroy(struct mlx5_hv_vhca_agent *agent);
int mlx5_hv_vhca_agent_write(struct mlx5_hv_vhca_agent *agent,
void (*invalidate)(struct mlx5_hv_vhca_agent*,
u64 block_mask),
void (*cleanup)(struct mlx5_hv_vhca_agent *agent),
- void *context)
+ void *context,
+ struct mlx5_hv_vhca_agent **ctx_update)
{
return NULL;
}
int reformat_type)
{
mutex_lock(&tun_entropy->lock);
- if (reformat_type == MLX5_REFORMAT_TYPE_L2_TO_VXLAN)
+ if (reformat_type == MLX5_REFORMAT_TYPE_L2_TO_VXLAN ||
+ reformat_type == MLX5_REFORMAT_TYPE_L2_TO_L3_TUNNEL)
tun_entropy->num_enabling_entries--;
else if (reformat_type == MLX5_REFORMAT_TYPE_L2_TO_NVGRE &&
--tun_entropy->num_disabling_entries == 0)
if (refcount_dec_and_test(&idx_data->usecount)) {
xa_erase(&st->idx_xa, st_index);
+ kfree(idx_data);
/* We leave PCI config space as was before, no mkey will refer to it */
}
kfree(admin->cache.keystream);
kfree(admin->cache.maskstream);
kfree(admin->cache.actionstream);
- mutex_destroy(&admin->lock);
kfree(admin);
}
if (!admin)
return ERR_PTR(-ENOMEM);
- mutex_init(&admin->lock);
+ admin->vctrl = ctrl;
INIT_LIST_HEAD(&admin->list);
INIT_LIST_HEAD(&admin->rules);
INIT_LIST_HEAD(&admin->enabled);
ctrl->ops = &lan966x_vcap_ops;
INIT_LIST_HEAD(&ctrl->list);
+ mutex_init(&ctrl->lock);
for (int i = 0; i < ARRAY_SIZE(lan966x_vcap_inst_cfg); ++i) {
cfg = &lan966x_vcap_inst_cfg[i];
lan966x_vcap_admin_free(admin);
}
+ mutex_destroy(&ctrl->lock);
kfree(ctrl);
}
{
if (!admin)
return;
- mutex_destroy(&admin->lock);
kfree(admin->cache.keystream);
kfree(admin->cache.maskstream);
kfree(admin->cache.actionstream);
INIT_LIST_HEAD(&admin->list);
INIT_LIST_HEAD(&admin->rules);
INIT_LIST_HEAD(&admin->enabled);
- mutex_init(&admin->lock);
+ admin->vctrl = ctrl;
admin->vtype = cfg->vtype;
admin->vinst = cfg->vinst;
admin->ingress = cfg->ingress;
ctrl->ops = &sparx5_vcap_ops;
INIT_LIST_HEAD(&ctrl->list);
+ mutex_init(&ctrl->lock);
for (idx = 0; idx < ARRAY_SIZE(sparx5_vcap_inst_cfg); ++idx) {
cfg = &consts->vcaps_cfg[idx];
admin = sparx5_vcap_admin_alloc(sparx5, ctrl, cfg);
list_del(&admin->list);
sparx5_vcap_admin_free(admin);
}
+ mutex_destroy(&ctrl->lock);
kfree(ctrl);
}
return false;
}
+void vcap_lock(struct vcap_admin *admin)
+{
+ mutex_lock(&admin->vctrl->lock);
+}
+
+void vcap_unlock(struct vcap_admin *admin)
+{
+ mutex_unlock(&admin->vctrl->lock);
+}
+
/* Find a rule with a provided rule id return a locked vcap */
static struct vcap_rule_internal *
vcap_get_locked_rule(struct vcap_control *vctrl, u32 id)
/* Look for the rule id in all vcaps */
list_for_each_entry(admin, &vctrl->list, list) {
- mutex_lock(&admin->lock);
+ vcap_lock(admin);
list_for_each_entry(ri, &admin->rules, list)
if (ri->data.id == id)
return ri;
- mutex_unlock(&admin->lock);
+ vcap_unlock(admin);
}
return NULL;
}
/* Look for the rule id in all vcaps */
list_for_each_entry(admin, &vctrl->list, list) {
- mutex_lock(&admin->lock);
+ vcap_lock(admin);
list_for_each_entry(ri, &admin->rules, list) {
if (ri->data.cookie == cookie) {
id = ri->data.id;
break;
}
}
- mutex_unlock(&admin->lock);
+ vcap_unlock(admin);
if (id)
return id;
}
int count = 0;
list_for_each_entry(elem, &admin->rules, list) {
- mutex_lock(&admin->lock);
+ vcap_lock(admin);
if (elem->data.vcap_chain_id >= min_cid &&
elem->data.vcap_chain_id < max_cid)
++count;
- mutex_unlock(&admin->lock);
+ vcap_unlock(admin);
}
return count;
}
if (ret)
return ret;
/* Insert the new rule in the list of vcap rules */
- mutex_lock(&ri->admin->lock);
+ vcap_lock(ri->admin);
vcap_rule_set_state(ri);
ret = vcap_insert_rule(ri, &move);
goto out;
}
out:
- mutex_unlock(&ri->admin->lock);
+ vcap_unlock(ri->admin);
return ret;
}
EXPORT_SYMBOL_GPL(vcap_add_rule);
if (vctrl->vcaps[admin->vtype].rows == 0)
return ERR_PTR(-EINVAL);
- mutex_lock(&admin->lock);
+ vcap_lock(admin);
/* Check if a rule with this id already exists */
if (vcap_rule_exists(vctrl, id)) {
err = -EINVAL;
goto out_free;
}
- mutex_unlock(&admin->lock);
+ vcap_unlock(admin);
return (struct vcap_rule *)ri;
out_free:
kfree(ri);
out_unlock:
- mutex_unlock(&admin->lock);
+ vcap_unlock(admin);
return ERR_PTR(err);
}
return ERR_PTR(-ENOENT);
rule = vcap_decode_rule(elem);
- mutex_unlock(&elem->admin->lock);
+ vcap_unlock(elem->admin);
return rule;
}
EXPORT_SYMBOL_GPL(vcap_get_rule);
err = vcap_write_counter(ri, &ctr);
out:
- mutex_unlock(&ri->admin->lock);
+ vcap_unlock(ri->admin);
return err;
}
EXPORT_SYMBOL_GPL(vcap_mod_rule);
admin->last_used_addr = elem->addr;
}
- mutex_unlock(&admin->lock);
+ vcap_unlock(admin);
return err;
}
EXPORT_SYMBOL_GPL(vcap_del_rule);
if (ret)
return ret;
- mutex_lock(&admin->lock);
+ vcap_lock(admin);
list_for_each_entry_safe(ri, next_ri, &admin->rules, list) {
vctrl->ops->init(ri->ndev, admin, ri->addr, ri->size);
list_del(&ri->list);
list_del(&eport->list);
kfree(eport);
}
- mutex_unlock(&admin->lock);
+ vcap_unlock(admin);
return 0;
}
continue;
/* Found the admin, now find the offloadable rules */
- mutex_lock(&admin->lock);
+ vcap_lock(admin);
list_for_each_entry(ri, &admin->rules, list) {
/* Is the rule in the lookup defined by the chain */
if (!(ri->data.vcap_chain_id >= chain &&
if (err)
break;
}
- mutex_unlock(&admin->lock);
+ vcap_unlock(admin);
if (err)
break;
}
continue;
/* Found the admin, now find the rules on the chain */
- mutex_lock(&admin->lock);
+ vcap_lock(admin);
list_for_each_entry(ri, &admin->rules, list) {
if (ri->data.vcap_chain_id != chain)
continue;
if (err)
break;
}
- mutex_unlock(&admin->lock);
+ vcap_unlock(admin);
if (err)
break;
}
eport->cookie = cookie;
eport->src_cid = src_cid;
eport->dst_cid = dst_cid;
- mutex_lock(&admin->lock);
+ vcap_lock(admin);
list_add_tail(&eport->list, &admin->enabled);
- mutex_unlock(&admin->lock);
+ vcap_unlock(admin);
if (vcap_path_exist(vctrl, ndev, src_cid)) {
/* Enable chained lookups */
dst_cid = vcap_get_next_chain(vctrl, ndev, dst_cid);
}
- mutex_lock(&found->lock);
+ vcap_lock(found);
list_del(&eport->list);
- mutex_unlock(&found->lock);
+ vcap_unlock(found);
kfree(eport);
return 0;
}
return -EINVAL;
}
- mutex_lock(&ri->admin->lock);
+ vcap_lock(ri->admin);
err = vcap_write_counter(ri, ctr);
- mutex_unlock(&ri->admin->lock);
+ vcap_unlock(ri->admin);
return err;
}
return -EINVAL;
}
- mutex_lock(&ri->admin->lock);
+ vcap_lock(ri->admin);
err = vcap_read_counter(ri, ctr);
- mutex_unlock(&ri->admin->lock);
+ vcap_unlock(ri->admin);
return err;
}
/* Iterate all rules in each VCAP instance */
list_for_each_entry(admin, &vctrl->list, list) {
- mutex_lock(&admin->lock);
+ vcap_lock(admin);
list_for_each_entry(ri, &admin->rules, list) {
if (ri->data.cookie != cookie)
continue;
if (err)
goto unlock;
}
- mutex_unlock(&admin->lock);
+ vcap_unlock(admin);
}
return err;
unlock:
- mutex_unlock(&admin->lock);
+ vcap_unlock(admin);
return err;
}
EXPORT_SYMBOL_GPL(vcap_get_rule_count_by_cookie);
struct list_head list; /* for insertion in vcap_control */
struct list_head rules; /* list of rules */
struct list_head enabled; /* list of enabled ports */
- struct mutex lock; /* control access to rules */
+ struct vcap_control *vctrl; /* the control instance owning this vcap */
enum vcap_type vtype; /* type of vcap */
int vinst; /* instance number within the same type */
int first_cid; /* first chain id in this vcap */
const struct vcap_info *vcaps; /* client supplied vcap models */
const struct vcap_statistics *stats; /* client supplied vcap stats */
struct list_head list; /* list of vcap instances */
+ struct mutex lock; /* serialize access to all vcap instances */
};
#endif /* __VCAP_API__ */
};
int ret;
- mutex_lock(&info->admin->lock);
+ vcap_lock(info->admin);
ret = vcap_show_admin(info->vctrl, info->admin, &out);
- mutex_unlock(&info->admin->lock);
+ vcap_unlock(info->admin);
return ret;
}
DEFINE_SHOW_ATTRIBUTE(vcap_debugfs);
};
int ret;
- mutex_lock(&info->admin->lock);
+ vcap_lock(info->admin);
ret = vcap_show_admin_raw(info->vctrl, info->admin, &out);
- mutex_unlock(&info->admin->lock);
+ vcap_unlock(info->admin);
return ret;
}
DEFINE_SHOW_ATTRIBUTE(vcap_raw_debugfs);
{
/* Initialize the shared objects */
INIT_LIST_HEAD(&test_vctrl.list);
+ mutex_init(&test_vctrl.lock);
INIT_LIST_HEAD(&admin->list);
INIT_LIST_HEAD(&admin->rules);
INIT_LIST_HEAD(&admin->enabled);
- mutex_init(&admin->lock);
+ admin->vctrl = &test_vctrl;
list_add_tail(&admin->list, &test_vctrl.list);
memset(test_updateaddr, 0, sizeof(test_updateaddr));
test_updateaddridx = 0;
{
/* Initialize the shared objects */
INIT_LIST_HEAD(&test_vctrl.list);
+ mutex_init(&test_vctrl.lock);
INIT_LIST_HEAD(&admin->list);
INIT_LIST_HEAD(&admin->rules);
INIT_LIST_HEAD(&admin->enabled);
- mutex_init(&admin->lock);
+ admin->vctrl = &test_vctrl;
list_add_tail(&admin->list, &test_vctrl.list);
memset(test_updateaddr, 0, sizeof(test_updateaddr));
test_updateaddridx = 0;
/* Check that the control has a valid set of callbacks */
int vcap_api_check(struct vcap_control *ctrl);
+/* Serialize access to the vcap instances of a control */
+void vcap_lock(struct vcap_admin *admin);
+void vcap_unlock(struct vcap_admin *admin);
/* Erase the VCAP cache area used or encoding and decoding */
void vcap_erase_cache(struct vcap_rule_internal *ri);
}
static void *mana_get_rxfrag(struct mana_rxq *rxq, struct device *dev,
- dma_addr_t *da, bool *from_pool)
+ dma_addr_t *da, bool *from_pool,
+ struct page **pp_page, u32 *dma_sync_offset)
{
struct page *page;
u32 offset;
void *va;
+
*from_pool = false;
+ *pp_page = NULL;
+ *dma_sync_offset = 0;
/* Don't use fragments for jumbo frames or XDP where it's 1 fragment
* per page.
va = page_to_virt(page) + offset;
*da = page_pool_get_dma_addr(page) + offset + rxq->headroom;
*from_pool = true;
+ *pp_page = page;
+ *dma_sync_offset = offset + rxq->headroom;
return va;
}
/* Allocate frag for rx buffer, and save the old buf */
static void mana_refill_rx_oob(struct device *dev, struct mana_rxq *rxq,
- struct mana_recv_buf_oob *rxoob, void **old_buf,
- bool *old_fp)
+ struct mana_recv_buf_oob *rxoob, u32 pktlen,
+ void **old_buf, bool *old_fp)
{
+ struct page *pp_page;
+ u32 dma_sync_offset;
bool from_pool;
dma_addr_t da;
void *va;
- va = mana_get_rxfrag(rxq, dev, &da, &from_pool);
+ va = mana_get_rxfrag(rxq, dev, &da, &from_pool, &pp_page,
+ &dma_sync_offset);
if (!va)
return;
- if (!rxoob->from_pool || rxq->frag_count == 1)
+ if (!rxoob->from_pool || rxq->frag_count == 1) {
dma_unmap_single(dev, rxoob->sgl[0].address, rxq->datasize,
DMA_FROM_DEVICE);
+ } else {
+ /* The page pool maps the whole page and only syncs for device
+ * automatically (PP_FLAG_DMA_SYNC_DEV). Sync the received bytes
+ * for the CPU before they are read: this is required if DMA
+ * is incoherent or bounce buffers are used.
+ */
+ page_pool_dma_sync_for_cpu(rxq->page_pool, rxoob->pp_page,
+ rxoob->dma_sync_offset, pktlen);
+ }
*old_buf = rxoob->buf_va;
*old_fp = rxoob->from_pool;
rxoob->buf_va = va;
rxoob->sgl[0].address = da;
rxoob->from_pool = from_pool;
+ rxoob->pp_page = pp_page;
+ rxoob->dma_sync_offset = dma_sync_offset;
}
static void mana_process_rx_cqe(struct mana_rxq *rxq, struct mana_cq *cq,
rxbuf_oob = &rxq->rx_oobs[curr];
WARN_ON_ONCE(rxbuf_oob->wqe_inf.wqe_size_in_bu != 1);
- mana_refill_rx_oob(dev, rxq, rxbuf_oob, &old_buf, &old_fp);
+ if (unlikely(pktlen > rxq->datasize)) {
+ /* Increase it even if mana_rx_skb() isn't called. */
+ rxq->rx_cq.work_done++;
- /* Unsuccessful refill will have old_buf == NULL.
- * In this case, mana_rx_skb() will drop the packet.
- */
- mana_rx_skb(old_buf, old_fp, oob, rxq, i);
+ ++ndev->stats.rx_dropped;
+ netdev_warn_once(ndev,
+ "Dropped oversized RX packet: len=%u, datasize=%u\n",
+ pktlen, rxq->datasize);
+
+ /* Reuse the RX buffer since rxbuf_oob is unchanged. */
+ } else {
+
+ mana_refill_rx_oob(dev, rxq, rxbuf_oob, pktlen,
+ &old_buf, &old_fp);
+
+ /* Unsuccessful refill will have old_buf == NULL.
+ * In this case, mana_rx_skb() will drop the packet.
+ */
+ mana_rx_skb(old_buf, old_fp, oob, rxq, i);
+ }
mana_move_wq_tail(rxq->gdma_rq,
rxbuf_oob->wqe_inf.wqe_size_in_bu);
struct mana_rxq *rxq, struct device *dev)
{
struct mana_port_context *mpc = netdev_priv(rxq->ndev);
+ struct page *pp_page = NULL;
+ u32 dma_sync_offset = 0;
bool from_pool = false;
dma_addr_t da;
void *va;
if (mpc->rxbufs_pre)
va = mana_get_rxbuf_pre(rxq, &da);
else
- va = mana_get_rxfrag(rxq, dev, &da, &from_pool);
+ va = mana_get_rxfrag(rxq, dev, &da, &from_pool, &pp_page,
+ &dma_sync_offset);
if (!va)
return -ENOMEM;
rx_oob->buf_va = va;
rx_oob->from_pool = from_pool;
+ rx_oob->pp_page = pp_page;
+ rx_oob->dma_sync_offset = dma_sync_offset;
rx_oob->sgl[0].address = da;
rx_oob->sgl[0].size = rxq->datasize;
*
* Return: 0 on success, negative errno on failure
**/
-static int mucse_read_mbx_pf(struct mucse_hw *hw, u32 *msg, u16 size)
+static int mucse_read_mbx_pf(struct mucse_hw *hw, __le32 *msg, u16 size)
{
- const int size_in_words = size / sizeof(u32);
+ const int size_in_words = size / sizeof(__le32);
struct mucse_mbx_info *mbx = &hw->mbx;
+ int off = MUCSE_MBX_FWPF_SHM;
int err;
err = mucse_obtain_mbx_lock_pf(hw);
if (err)
return err;
+ /* memcpy_fromio() is unsuitable: the mailbox uses 32-bit MMIO
+ * registers, not byte-addressable RAM. readl() guarantees
+ * the required 32-bit access width.
+ */
for (int i = 0; i < size_in_words; i++)
- msg[i] = mbx_data_rd32(mbx, MUCSE_MBX_FWPF_SHM + 4 * i);
+ msg[i] = cpu_to_le32(mbx_data_rd32(mbx, off + 4 * i));
/* Hw needs write data_reg at last */
mbx_data_wr32(mbx, MUCSE_MBX_FWPF_SHM, 0);
/* flush reqs as we have read this request data */
* Return: 0 if it successfully received a message notification and
* copied it into the receive buffer, negative errno on failure
**/
-int mucse_poll_and_read_mbx(struct mucse_hw *hw, u32 *msg, u16 size)
+int mucse_poll_and_read_mbx(struct mucse_hw *hw, __le32 *msg, u16 size)
{
int err;
* Return: 0 if it successfully copied message into the buffer,
* negative errno on failure
**/
-static int mucse_write_mbx_pf(struct mucse_hw *hw, u32 *msg, u16 size)
+static int mucse_write_mbx_pf(struct mucse_hw *hw, const __le32 *msg, u16 size)
{
- const int size_in_words = size / sizeof(u32);
+ const int size_in_words = size / sizeof(__le32);
struct mucse_mbx_info *mbx = &hw->mbx;
int err;
if (err)
return err;
+ /* memcpy_toio() would decompose into arbitrary-width accesses;
+ * the mailbox requires 32-bit MMIO writes via writel().
+ */
for (int i = 0; i < size_in_words; i++)
- mbx_data_wr32(mbx, MUCSE_MBX_FWPF_SHM + i * 4, msg[i]);
+ mbx_data_wr32(mbx, MUCSE_MBX_FWPF_SHM + i * 4,
+ le32_to_cpu(msg[i]));
/* flush acks as we are overwriting the message buffer */
hw->mbx.fw_ack = mucse_mbx_get_fwack(mbx);
* Return: 0 if it successfully copied message into the buffer and
* received an ack to that message within delay * timeout_cnt period
**/
-int mucse_write_and_wait_ack_mbx(struct mucse_hw *hw, u32 *msg, u16 size)
+int mucse_write_and_wait_ack_mbx(struct mucse_hw *hw, const __le32 *msg,
+ u16 size)
{
int err;
#define MUCSE_MBX_REQ BIT(0) /* Request a req to mailbox */
#define MUCSE_MBX_PFU BIT(3) /* PF owns the mailbox buffer */
-int mucse_write_and_wait_ack_mbx(struct mucse_hw *hw, u32 *msg, u16 size);
+int mucse_write_and_wait_ack_mbx(struct mucse_hw *hw,
+ const __le32 *msg, u16 size);
void mucse_init_mbx_params_pf(struct mucse_hw *hw);
-int mucse_poll_and_read_mbx(struct mucse_hw *hw, u32 *msg, u16 size);
+int mucse_poll_and_read_mbx(struct mucse_hw *hw, __le32 *msg, u16 size);
#endif /* _RNPGBE_MBX_H */
* Return: 0 on success, negative errno on failure
**/
static int mucse_fw_send_cmd_wait_resp(struct mucse_hw *hw,
- struct mbx_fw_cmd_req *req,
- struct mbx_fw_cmd_reply *reply)
+ union mbx_fw_cmd_req_u *req,
+ union mbx_fw_cmd_reply_u *reply)
{
- int len = le16_to_cpu(req->datalen);
+ int len = le16_to_cpu(req->r.datalen);
int retry_cnt = 3;
int err;
mutex_lock(&hw->mbx.lock);
- err = mucse_write_and_wait_ack_mbx(hw, (u32 *)req, len);
+ err = mucse_write_and_wait_ack_mbx(hw, req->dwords, len);
if (err)
goto out;
do {
- err = mucse_poll_and_read_mbx(hw, (u32 *)reply,
- sizeof(*reply));
+ err = mucse_poll_and_read_mbx(hw, reply->dwords,
+ sizeof(reply->r));
if (err)
goto out;
/* mucse_write_and_wait_ack_mbx return 0 means fw has
* received request, wait for the expect opcode
* reply with 'retry_cnt' times.
*/
- } while (--retry_cnt >= 0 && reply->opcode != req->opcode);
+ } while (--retry_cnt >= 0 && reply->r.opcode != req->r.opcode);
out:
mutex_unlock(&hw->mbx.lock);
if (!err && retry_cnt < 0)
return -ETIMEDOUT;
- if (!err && reply->error_code)
+ if (!err && reply->r.error_code)
return -EIO;
return err;
**/
static int mucse_mbx_get_info(struct mucse_hw *hw)
{
- struct mbx_fw_cmd_req req = {
- .datalen = cpu_to_le16(MUCSE_MBX_REQ_HDR_LEN),
- .opcode = cpu_to_le16(GET_HW_INFO),
+ union mbx_fw_cmd_req_u req = {
+ .r = {
+ .datalen = cpu_to_le16(MUCSE_MBX_REQ_HDR_LEN),
+ .opcode = cpu_to_le16(GET_HW_INFO),
+ },
};
- struct mbx_fw_cmd_reply reply = {};
+ union mbx_fw_cmd_reply_u reply = {};
int err;
err = mucse_fw_send_cmd_wait_resp(hw, &req, &reply);
if (!err)
hw->pfvfnum = FIELD_GET(GENMASK_U16(7, 0),
- le16_to_cpu(reply.hw_info.pfnum));
+ le16_to_cpu(reply.r.hw_info.pfnum));
return err;
}
**/
int mucse_mbx_powerup(struct mucse_hw *hw, bool is_powerup)
{
- struct mbx_fw_cmd_req req = {
- .datalen = cpu_to_le16(sizeof(req.powerup) +
- MUCSE_MBX_REQ_HDR_LEN),
- .opcode = cpu_to_le16(POWER_UP),
- .powerup = {
- /* fw needs this to reply correct cmd */
- .version = cpu_to_le32(GENMASK_U32(31, 0)),
- .status = cpu_to_le32(is_powerup ? 1 : 0),
+ union mbx_fw_cmd_req_u req = {
+ .r = {
+ .datalen = cpu_to_le16(sizeof(req.r.powerup) +
+ MUCSE_MBX_REQ_HDR_LEN),
+ .opcode = cpu_to_le16(POWER_UP),
+ .powerup = {
+ /* fw needs this to reply correct cmd */
+ .version = cpu_to_le32(GENMASK_U32(31, 0)),
+ .status = cpu_to_le32(is_powerup ? 1 : 0),
+ },
},
};
int len, err;
- len = le16_to_cpu(req.datalen);
+ len = le16_to_cpu(req.r.datalen);
mutex_lock(&hw->mbx.lock);
- err = mucse_write_and_wait_ack_mbx(hw, (u32 *)&req, len);
+ err = mucse_write_and_wait_ack_mbx(hw, req.dwords, len);
mutex_unlock(&hw->mbx.lock);
return err;
**/
int mucse_mbx_reset_hw(struct mucse_hw *hw)
{
- struct mbx_fw_cmd_req req = {
- .datalen = cpu_to_le16(MUCSE_MBX_REQ_HDR_LEN),
- .opcode = cpu_to_le16(RESET_HW),
+ union mbx_fw_cmd_req_u req = {
+ .r = {
+ .datalen = cpu_to_le16(MUCSE_MBX_REQ_HDR_LEN),
+ .opcode = cpu_to_le16(RESET_HW),
+ },
};
- struct mbx_fw_cmd_reply reply = {};
+ union mbx_fw_cmd_reply_u reply = {};
return mucse_fw_send_cmd_wait_resp(hw, &req, &reply);
}
u8 *mac_addr,
int port)
{
- struct mbx_fw_cmd_req req = {
- .datalen = cpu_to_le16(sizeof(req.get_mac_addr) +
- MUCSE_MBX_REQ_HDR_LEN),
- .opcode = cpu_to_le16(GET_MAC_ADDRESS),
- .get_mac_addr = {
- .port_mask = cpu_to_le32(BIT(port)),
- .pfvf_num = cpu_to_le32(pfvfnum),
+ union mbx_fw_cmd_req_u req = {
+ .r = {
+ .datalen = cpu_to_le16(sizeof(req.r.get_mac_addr) +
+ MUCSE_MBX_REQ_HDR_LEN),
+ .opcode = cpu_to_le16(GET_MAC_ADDRESS),
+ .get_mac_addr = {
+ .port_mask = cpu_to_le32(BIT(port)),
+ .pfvf_num = cpu_to_le32(pfvfnum),
+ },
},
};
- struct mbx_fw_cmd_reply reply = {};
+ union mbx_fw_cmd_reply_u reply = {};
int err;
err = mucse_fw_send_cmd_wait_resp(hw, &req, &reply);
if (err)
return err;
- if (le32_to_cpu(reply.mac_addr.ports) & BIT(port))
- memcpy(mac_addr, reply.mac_addr.addrs[port].mac, ETH_ALEN);
+ if (le32_to_cpu(reply.r.mac_addr.ports) & BIT(port))
+ memcpy(mac_addr, reply.r.mac_addr.addrs[port].mac, ETH_ALEN);
else
return -ENODATA;
};
} __packed;
+/* Union wrappers to expose struct as __le32 dword array for mailbox
+ * transport, eliminating the need for pointer casts. The __packed
+ * structs have no padding, so dwords[] overlays the fields exactly.
+ */
+union mbx_fw_cmd_req_u {
+ struct mbx_fw_cmd_req r;
+ __le32 dwords[sizeof(struct mbx_fw_cmd_req) / sizeof(__le32)];
+};
+
+union mbx_fw_cmd_reply_u {
+ struct mbx_fw_cmd_reply r;
+ __le32 dwords[sizeof(struct mbx_fw_cmd_reply) / sizeof(__le32)];
+};
+
int mucse_mbx_sync_fw(struct mucse_hw *hw);
int mucse_mbx_powerup(struct mucse_hw *hw, bool is_powerup);
int mucse_mbx_reset_hw(struct mucse_hw *hw);
struct sk_buff *skb;
skb = qede_build_skb(rxq, bd, len, pad);
+ if (unlikely(!skb))
+ return NULL;
+
bd->page_offset += rxq->rx_buf_seg_size;
if (bd->page_offset == PAGE_SIZE) {
}
skb = qede_build_skb(rxq, bd, len, pad);
+ if (unlikely(!skb))
+ return NULL;
if (unlikely(qede_realloc_rx_buffer(rxq, bd))) {
/* Incr page ref count to reuse on allocation failure so
consume_skb(skb);
} else {
- __rmnet_map_ingress_handler(skb, port);
+ if (rmnet_map_validate_packet_len(skb, port))
+ __rmnet_map_ingress_handler(skb, port);
+ else
+ kfree_skb(skb);
}
}
void rmnet_map_tx_aggregate_exit(struct rmnet_port *port);
void rmnet_map_update_ul_agg_config(struct rmnet_port *port, u32 size,
u32 count, u32 time);
+u32 rmnet_map_validate_packet_len(struct sk_buff *skb, struct rmnet_port *port);
#endif /* _RMNET_MAP_H_ */
return map_header;
}
-/* Deaggregates a single packet
- * A whole new buffer is allocated for each portion of an aggregated frame.
- * Caller should keep calling deaggregate() on the source skb until 0 is
- * returned, indicating that there are no more packets to deaggregate. Caller
- * is responsible for freeing the original skb.
- */
-struct sk_buff *rmnet_map_deaggregate(struct sk_buff *skb,
- struct rmnet_port *port)
+u32 rmnet_map_validate_packet_len(struct sk_buff *skb, struct rmnet_port *port)
{
struct rmnet_map_v5_csum_header *next_hdr = NULL;
struct rmnet_map_header *maph;
void *data = skb->data;
- struct sk_buff *skbn;
- u8 nexthdr_type;
u32 packet_len;
- if (skb->len == 0)
- return NULL;
+ if (skb->len < sizeof(*maph))
+ return 0;
maph = (struct rmnet_map_header *)skb->data;
+
+ /* Some hardware can send us empty frames. Catch them */
+ if (!maph->pkt_len)
+ return 0;
+
packet_len = ntohs(maph->pkt_len) + sizeof(*maph);
if (port->data_format & RMNET_FLAGS_INGRESS_MAP_CKSUMV4) {
packet_len += sizeof(struct rmnet_map_dl_csum_trailer);
- } else if (port->data_format & RMNET_FLAGS_INGRESS_MAP_CKSUMV5) {
- if (!(maph->flags & MAP_CMD_FLAG)) {
- packet_len += sizeof(*next_hdr);
- if (maph->flags & MAP_NEXT_HEADER_FLAG)
- next_hdr = data + sizeof(*maph);
- else
- /* Mapv5 data pkt without csum hdr is invalid */
- return NULL;
- }
+ } else if ((port->data_format & RMNET_FLAGS_INGRESS_MAP_CKSUMV5) &&
+ !(maph->flags & MAP_CMD_FLAG)) {
+ /* Mapv5 data pkt without csum hdr is invalid */
+ if (!(maph->flags & MAP_NEXT_HEADER_FLAG))
+ return 0;
+
+ packet_len += sizeof(*next_hdr);
+ next_hdr = data + sizeof(*maph);
}
- if (((int)skb->len - (int)packet_len) < 0)
- return NULL;
+ if (skb->len < packet_len)
+ return 0;
- /* Some hardware can send us empty frames. Catch them */
- if (!maph->pkt_len)
- return NULL;
+ if (next_hdr &&
+ u8_get_bits(next_hdr->header_info, MAPV5_HDRINFO_HDR_TYPE_FMASK) !=
+ RMNET_MAP_HEADER_TYPE_CSUM_OFFLOAD)
+ return 0;
- if (next_hdr) {
- nexthdr_type = u8_get_bits(next_hdr->header_info,
- MAPV5_HDRINFO_HDR_TYPE_FMASK);
- if (nexthdr_type != RMNET_MAP_HEADER_TYPE_CSUM_OFFLOAD)
- return NULL;
- }
+ return packet_len;
+}
+
+/* Deaggregates a single packet
+ * A whole new buffer is allocated for each portion of an aggregated frame.
+ * Caller should keep calling deaggregate() on the source skb until 0 is
+ * returned, indicating that there are no more packets to deaggregate. Caller
+ * is responsible for freeing the original skb.
+ */
+struct sk_buff *rmnet_map_deaggregate(struct sk_buff *skb,
+ struct rmnet_port *port)
+{
+ struct sk_buff *skbn;
+ u32 packet_len;
+
+ packet_len = rmnet_map_validate_packet_len(skb, port);
+ if (!packet_len)
+ return NULL;
skbn = alloc_skb(packet_len + RMNET_MAP_DEAGGR_SPACING, GFP_ATOMIC);
if (!skbn)
}
unprotected_len = skb->len;
- eth = eth_hdr(skb);
+ eth = skb_eth_hdr(skb);
sci_present = macsec_send_sci(secy);
hh = skb_push(skb, macsec_extra_len(sci_present));
memmove(hh, eth, 2 * ETH_ALEN);
ops = macsec_get_ops(netdev_priv(dev), &ctx);
if (!ops) {
err = -EOPNOTSUPP;
- goto clear_allmulti;
+ goto clear_promisc;
}
ctx.secy = &macsec->secy;
err = macsec_offload(ops->mdo_dev_open, &ctx);
if (err)
- goto clear_allmulti;
+ goto clear_promisc;
}
if (netif_carrier_ok(real_dev))
netif_carrier_on(dev);
return 0;
+clear_promisc:
+ if (dev->flags & IFF_PROMISC)
+ dev_set_promiscuity(real_dev, -1);
clear_allmulti:
if (dev->flags & IFF_ALLMULTI)
dev_set_allmulti(real_dev, -1);
config MDIO_MSCC_MIIM
tristate "Microsemi MIIM interface support"
- depends on HAS_IOMEM && REGMAP_MMIO
+ depends on HAS_IOMEM
+ select REGMAP_MMIO
help
This driver supports the MIIM (MDIO) interface found in the network
switches of the Microsemi SoCs; it is recommended to switch on
struct list_head clist; /* link in list of channels per unit */
spinlock_t upl; /* protects `ppp' and 'bridge' */
struct channel __rcu *bridge; /* "bridged" ppp channel */
+ struct rcu_head rcu; /* for RCU-deferred free of the channel */
#ifdef CONFIG_PPP_MULTILINK
u8 avail; /* flag used in multilink stuff */
u8 had_frag; /* >= 1 fragments have been sent */
return err;
}
+/* Purge after the grace period: a late ppp_input() may still queue an
+ * skb on pch->file.rq before the last RCU reader drains.
+ */
+static void ppp_release_channel_free(struct rcu_head *rcu)
+{
+ struct channel *pch = container_of(rcu, struct channel, rcu);
+
+ skb_queue_purge(&pch->file.xq);
+ skb_queue_purge(&pch->file.rq);
+ kfree(pch);
+}
+
/*
* Drop a reference to a ppp channel and free its memory if the refcount reaches
* zero.
pr_err("ppp: destroying undead channel %p !\n", pch);
return;
}
- skb_queue_purge(&pch->file.xq);
- skb_queue_purge(&pch->file.rq);
- kfree(pch);
+ call_rcu(&pch->rcu, ppp_release_channel_free);
}
static void __exit ppp_cleanup(void)
device_destroy(&ppp_class, MKDEV(PPP_MAJOR, 0));
class_unregister(&ppp_class);
unregister_pernet_device(&ppp_net_ops);
+ rcu_barrier(); /* wait for RCU callbacks before module unload */
}
/*
return;
}
+static void lan78xx_update_vlan_filter(struct lan78xx_priv *pdata,
+ struct net_device *netdev,
+ netdev_features_t features)
+{
+ if ((features & NETIF_F_HW_VLAN_CTAG_FILTER) &&
+ !(netdev->flags & IFF_PROMISC))
+ pdata->rfe_ctl |= RFE_CTL_VLAN_FILTER_;
+ else
+ pdata->rfe_ctl &= ~RFE_CTL_VLAN_FILTER_;
+}
+
static void lan78xx_set_multicast(struct net_device *netdev)
{
struct lan78xx_net *dev = netdev_priv(netdev);
}
}
+ lan78xx_update_vlan_filter(pdata, dev->net, dev->net->features);
+
if (netdev_mc_count(dev->net)) {
struct netdev_hw_addr *ha;
int i;
else
pdata->rfe_ctl &= ~RFE_CTL_VLAN_STRIP_;
- if (features & NETIF_F_HW_VLAN_CTAG_FILTER)
- pdata->rfe_ctl |= RFE_CTL_VLAN_FILTER_;
- else
- pdata->rfe_ctl &= ~RFE_CTL_VLAN_FILTER_;
+ lan78xx_update_vlan_filter(pdata, netdev, features);
spin_unlock_irqrestore(&pdata->rfe_ctl_lock, flags);
skb_trim(skb, skb->len - sizeof *trailer);
if ((packet_len & 0x01) == 0) {
- if (skb->data [packet_len] != PAD_BYTE) {
+ if (packet_len >= skb->len || skb->data[packet_len] != PAD_BYTE) {
dev->net->stats.rx_frame_errors++;
netdev_dbg(dev->net, "bad pad\n");
return 0;
return -EINVAL;
}
+ if (mgmt_frame_len < offsetof(struct ieee80211_mgmt, u)) {
+ bphy_err(drvr, "Event %s (%d) frame too small. Ignore\n",
+ brcmf_fweh_event_name(e->event_code), e->event_code);
+ return -EINVAL;
+ }
+
wdev = &ifp->vif->wdev;
WARN_ON(!wdev);
bus->sdiodev = sdiodev;
sdiodev->bus = bus;
skb_queue_head_init(&bus->glom);
+ INIT_WORK(&bus->datawork, brcmf_sdio_dataworker);
bus->txbound = BRCMF_TXBOUND;
bus->rxbound = BRCMF_RXBOUND;
bus->txminmax = BRCMF_TXMINMAX;
goto fail;
}
brcmf_sdiod_freezer_count(sdiodev);
- INIT_WORK(&bus->datawork, brcmf_sdio_dataworker);
bus->brcmf_wq = wq;
/* attempt to attach to the dongle */
if (err) {
printk(KERN_WARNING DRV_NAME
"Error calling pci_enable_device.\n");
+ free_libipw(dev, 0);
+ pci_iounmap(pci_dev, ioaddr);
return err;
}
if (err) {
printk(KERN_WARNING DRV_NAME
"Error calling pci_set_dma_mask.\n");
- pci_disable_device(pci_dev);
- return err;
+ goto fail;
}
err = pci_request_regions(pci_dev, DRV_NAME);
if (err) {
printk(KERN_WARNING DRV_NAME
"Error calling pci_request_regions.\n");
- pci_disable_device(pci_dev);
- return err;
+ goto fail;
}
/* We disable the RETRY_TIMEOUT register (0x41) to keep
ieee->host_mc_decrypt : ieee->host_decrypt;
if (can_be_decrypted) {
- if (skb->len >= hdrlen + 3) {
+ if (skb->len >= hdrlen + 4) {
/* Top two-bits of byte 3 are the key index */
keyidx = skb->data[hdrlen + 3] >> 6;
}
int trimlen = 0;
/* Top two-bits of byte 3 are the key index */
- if (skb->len >= hdrlen + 3)
+ if (skb->len >= hdrlen + 4)
keyidx = skb->data[hdrlen + 3] >> 6;
/* To strip off any security data which appears before the
if (le16_to_cpu(eeprom->v2.len) != priv->eeprom_slice_size)
return;
+ if (eeprom->v2.data + priv->eeprom_slice_size >
+ skb_tail_pointer(skb))
+ return;
+
memcpy(priv->eeprom, eeprom->v2.data, priv->eeprom_slice_size);
} else {
if (le16_to_cpu(eeprom->v1.len) != priv->eeprom_slice_size)
return;
+ if (eeprom->v1.data + priv->eeprom_slice_size >
+ skb_tail_pointer(skb))
+ return;
+
memcpy(priv->eeprom, eeprom->v1.data, priv->eeprom_slice_size);
}
} else {
/* No main firmware needed for this helper --> success! */
lbs_fw_loaded(priv, 0, firmware, NULL);
+ release_firmware(firmware);
}
}
if (priv->wdev->iftype == NL80211_IFTYPE_MONITOR) {
struct tx_radiotap_hdr *rtap_hdr = (void *)skb->data;
+ if (skb->len < sizeof(*rtap_hdr) + 4 + ETH_ALEN) {
+ lbs_deb_tx("tx err: short monitor frame %u\n", skb->len);
+ dev->stats.tx_dropped++;
+ dev->stats.tx_errors++;
+ goto free;
+ }
+
/* set txpd fields from the radiotap header */
txpd->tx_control = cpu_to_le32(convert_radiotap_rate_to_mv(rtap_hdr->rate));
{
lbtf_deb_enter(LBTF_DEB_MAIN);
lbtf_free_cmd_buffer(priv);
- timer_delete(&priv->command_timer);
+ timer_delete_sync(&priv->command_timer);
lbtf_deb_leave(LBTF_DEB_MAIN);
}
return -EOPNOTSUPP;
}
- if (!priv->auth_flag) {
+ if (!(priv->auth_flag & HOST_MLME_AUTH_PENDING)) {
ret = mwifiex_remain_on_chan_cfg(priv, HostCmd_ACT_GEN_SET,
req->bss->channel,
AUTH_TX_DEFAULT_WAIT_TIME);
/* Send a Media Connected event, according to the Spec */
priv->media_connected = true;
- priv->adapter->ps_state = PS_STATE_AWAKE;
priv->adapter->pps_uapsd_mode = false;
priv->adapter->tx_lock_flag = false;
*/
static int rt2400pci_validate_eeprom(struct rt2x00_dev *rt2x00dev)
{
- struct eeprom_93cx6 eeprom;
+ struct eeprom_93cx6 eeprom = {};
u32 reg;
u16 word;
u8 *mac;
*/
static int rt2500pci_validate_eeprom(struct rt2x00_dev *rt2x00dev)
{
- struct eeprom_93cx6 eeprom;
+ struct eeprom_93cx6 eeprom = {};
u32 reg;
u16 word;
u8 *mac;
static int rt2800pci_read_eeprom_pci(struct rt2x00_dev *rt2x00dev)
{
- struct eeprom_93cx6 eeprom;
+ struct eeprom_93cx6 eeprom = {};
u32 reg;
reg = rt2x00mmio_register_read(rt2x00dev, E2PROM_CSR);
GFP_KERNEL);
if (!rt2x00dev->drv_data) {
retval = -ENOMEM;
- goto exit;
+ return retval;
}
}
alloc_ordered_workqueue("%s", 0, wiphy_name(rt2x00dev->hw->wiphy));
if (!rt2x00dev->workqueue) {
retval = -ENOMEM;
- goto exit;
+ goto exit_free_drv_data;
}
INIT_WORK(&rt2x00dev->intf_work, rt2x00lib_intf_scheduled);
exit:
rt2x00lib_remove_dev(rt2x00dev);
+ return retval;
+
+exit_free_drv_data:
+ clear_bit(DEVICE_STATE_PRESENT, &rt2x00dev->flags);
+
+ kfree(rt2x00dev->drv_data);
+ rt2x00dev->drv_data = NULL;
+
return retval;
}
EXPORT_SYMBOL_GPL(rt2x00lib_probe_dev);
*/
static int rt61pci_validate_eeprom(struct rt2x00_dev *rt2x00dev)
{
- struct eeprom_93cx6 eeprom;
+ struct eeprom_93cx6 eeprom = {};
u32 reg;
u16 word;
u8 *mac;
struct ieee80211_vif *vif;
struct sk_buff *mac_bcn;
u8 vap_id = 0, i;
+ unsigned int tailroom;
u16 tim_offset = 0;
for (i = 0; i < RSI_MAX_VIFS; i++) {
if (mac_bcn->data[tim_offset + 2] == 0)
bcn_frm->frame_info |= cpu_to_le16(RSI_DATA_DESC_DTIM_BEACON);
+ tailroom = skb_tailroom(skb);
+ if (tailroom < FRAME_DESC_SZ ||
+ mac_bcn->len > tailroom - FRAME_DESC_SZ) {
+ dev_kfree_skb(mac_bcn);
+ return -EMSGSIZE;
+ }
+
memcpy(&skb->data[FRAME_DESC_SZ], mac_bcn->data, mac_bcn->len);
skb_put(skb, mac_bcn->len + FRAME_DESC_SZ);
} else {
memcpy(&set_key->key[0][0], data, key_len);
}
- memcpy(set_key->tx_mic_key, &data[16], 8);
- memcpy(set_key->rx_mic_key, &data[24], 8);
+ if (cipher == WLAN_CIPHER_SUITE_TKIP) {
+ memcpy(set_key->tx_mic_key, &data[16], 8);
+ memcpy(set_key->rx_mic_key, &data[24], 8);
+ }
} else {
memset(&set_key[FRAME_DESC_SZ], 0, frame_len - FRAME_DESC_SZ);
}
return -ENOMEM;
}
+ if (probereq_skb->len > MAX_BGSCAN_PROBE_REQ_LEN) {
+ dev_kfree_skb(probereq_skb);
+ dev_kfree_skb(skb);
+ return -EINVAL;
+ }
+
memcpy(&skb->data[frame_len], probereq_skb->data, probereq_skb->len);
bgscan->probe_req_length = cpu_to_le16(probereq_skb->len);
}
}
+static struct ieee80211_rate *
+mac80211_hwsim_get_tx_rate(struct ieee80211_hw *hw,
+ struct ieee80211_tx_info *info)
+{
+ if (info->control.rates[0].flags &
+ (IEEE80211_TX_RC_MCS | IEEE80211_TX_RC_VHT_MCS))
+ return NULL;
+
+ return ieee80211_get_tx_rate(hw, info);
+}
+
static void mac80211_hwsim_monitor_rx(struct ieee80211_hw *hw,
struct sk_buff *tx_skb,
struct ieee80211_channel *chan)
struct hwsim_radiotap_hdr *hdr;
u16 flags, bitrate;
struct ieee80211_tx_info *info = IEEE80211_SKB_CB(tx_skb);
- struct ieee80211_rate *txrate = ieee80211_get_tx_rate(hw, info);
+ struct ieee80211_rate *txrate = mac80211_hwsim_get_tx_rate(hw, info);
if (!txrate)
bitrate = 0;
spin_lock_bh(&data->tsf_offset_lock);
- txrate = ieee80211_get_tx_rate(data->hw, info);
+ txrate = mac80211_hwsim_get_tx_rate(data->hw, info);
if (txrate)
bitrate = txrate->bitrate;
skb->data = skb->head;
skb_reset_tail_pointer(skb);
+ len = min(len, skb_end_offset(skb));
skb_put(skb, len);
hwsim_virtio_handle_cmd(skb);
#include "pmc.h"
+static const struct amd_pmc_bit_map soc15_ip_blk_v3[] = {
+ {"VDDCR", BIT(0)},
+ {"VDDCR_LP", BIT(1)},
+ {"LSOCV", BIT(2)},
+ {"DISPLAY", BIT(3)},
+ {"VCN", BIT(4)},
+ {"JPEG", BIT(5)},
+ {"UMSCH", BIT(6)},
+ {"VPE", BIT(7)},
+ {"MPM", BIT(8)},
+ {"NPU", BIT(9)},
+ {"USB_HC0", BIT(10)},
+ {"eUSB_HC0", BIT(11)},
+ {"RT0_ADP_HC1", BIT(12)},
+ {"RT1_ADP_HC1", BIT(13)},
+ {"RT2_ADP_HC2", BIT(14)},
+ {"USB4_RT0", BIT(15)},
+ {"USB4_RT1", BIT(16)},
+ {"USB4-RT2", BIT(17)},
+ {"LAPIC", BIT(18)},
+};
+
static const struct amd_pmc_bit_map soc15_ip_blk_v2[] = {
{"DISPLAY", BIT(0)},
{"CPU", BIT(1)},
.smu_msg = AMD_PMC_REGISTER_MSG_1AH_80H,
.smu_arg = AMD_PMC_REGISTER_ARG_1AH_80H,
.smu_rsp = AMD_PMC_REGISTER_RSP_1AH_80H,
- .num_ips = ARRAY_SIZE(soc15_ip_blk),
+ .num_ips = ARRAY_SIZE(soc15_ip_blk_v3),
.scratch_reg = AMD_PMC_SCRATCH_REG_1AH,
- .ips_ptr = soc15_ip_blk,
+ .ips_ptr = soc15_ip_blk_v3,
.os_hint = MSG_OS_HINT_RN,
};
} else if (delay_suspend == 1) {
if (!intermediate_wakeup)
dev_info(pdev->dev, "Delaying suspend by 2.5s because delay_suspend=1. If this solves problems on your machine, please report this whole line to: platform-driver-x86@vger.kernel.org so it can be automatically detected as affected in the future. System Vendor: \"%s\" Product Name: \"%s\" Product Family: \"%s\" Board Vendor: \"%s\" Board Name: \"%s\"\n",
- dmi_get_system_info(DMI_SYS_VENDOR),
- dmi_get_system_info(DMI_PRODUCT_NAME),
- dmi_get_system_info(DMI_PRODUCT_FAMILY),
- dmi_get_system_info(DMI_BOARD_VENDOR),
- dmi_get_system_info(DMI_BOARD_NAME));
+ dmi_get_system_info(DMI_SYS_VENDOR) ?: "(Unknown)",
+ dmi_get_system_info(DMI_PRODUCT_NAME) ?: "(Unknown)",
+ dmi_get_system_info(DMI_PRODUCT_FAMILY) ?: "(Unknown)",
+ dmi_get_system_info(DMI_BOARD_VENDOR) ?: "(Unknown)",
+ dmi_get_system_info(DMI_BOARD_NAME) ?: "(Unknown)");
return true;
}
return false;
.driver_data = &(struct power_data) {
.ac_data = &(struct power_limits) {
.ppt_pl1_spl_min = 30,
- .ppt_pl1_spl_max = 90,
+ .ppt_pl1_spl_max = 120,
.ppt_pl2_sppt_min = 65,
- .ppt_pl2_sppt_def = 110,
- .ppt_pl2_sppt_max = 125,
+ .ppt_pl2_sppt_def = 140,
+ .ppt_pl2_sppt_max = 145,
.ppt_pl3_fppt_min = 65,
- .ppt_pl3_fppt_def = 110,
- .ppt_pl3_fppt_max = 125,
+ .ppt_pl3_fppt_def = 140,
+ .ppt_pl3_fppt_max = 145,
.nv_temp_target_min = 75,
.nv_temp_target_max = 87,
.nv_dynamic_boost_min = 5,
enum platform_profile_option profile;
int ret;
+ /* Skip event device */
+ if (!data->pp_dev)
+ return 0;
+
ret = laptop_profile_get(data->pp_dev, &profile);
if (ret == 0)
data->saved_profile = profile;
{
struct bitland_mifs_wmi_data *data = dev_get_drvdata(dev);
+ /* Skip event device */
+ if (!data->pp_dev)
+ return 0;
+
dev_dbg(dev, "Resuming, restoring profile %d\n", data->saved_profile);
return laptop_profile_set(dev, data->saved_profile);
}
* register.
*/
u32 mipi_phy_rst_mask;
+
+ /*
+ * VC8000E reset de-assertion edge and AXI clock may have a timing issue.
+ * Workaround: Set bit2 (vc8000e_clk_en) of BLK_CLK_EN_CSR to 0 to gate off
+ * both AXI clock and VC8000E clock sent to VC8000E and AXI clock sent to
+ * VPU_NOC m_v_2 interface during VC8000E power up(VC8000E reset is
+ * de-asserted by HW)
+ */
+ bool is_errata_err050531;
};
#define DOMAIN_MAX_CLKS 4
dev_err(bc->dev, "failed to enable clocks\n");
goto bus_put;
}
- regmap_set_bits(bc->regmap, BLK_CLK_EN, data->clk_mask);
+
+ if (data->is_errata_err050531)
+ regmap_clear_bits(bc->regmap, BLK_CLK_EN, data->clk_mask);
+ else
+ regmap_set_bits(bc->regmap, BLK_CLK_EN, data->clk_mask);
/* power up upstream GPC domain */
ret = pm_runtime_get_sync(domain->power_dev);
goto clk_disable;
}
+ if (data->is_errata_err050531)
+ regmap_set_bits(bc->regmap, BLK_CLK_EN, data->clk_mask);
+
/* wait for reset to propagate */
udelay(5);
.clk_mask = BIT(2),
.path_names = (const char *[]){"vc8000e"},
.num_paths = 1,
+ .is_errata_err050531 = true,
},
};
+static int imx8mp_vpu_power_notifier(struct notifier_block *nb,
+ unsigned long action, void *data)
+{
+ struct imx8m_blk_ctrl *bc = container_of(nb, struct imx8m_blk_ctrl,
+ power_nb);
+
+ if (action == GENPD_NOTIFY_ON) {
+ /*
+ * On power up we have no software backchannel to the GPC to
+ * wait for the ADB handshake to happen, so we just delay for a
+ * bit. On power down the GPC driver waits for the handshake.
+ */
+
+ udelay(5);
+
+ /* set "fuse" bits to enable the VPUs */
+ regmap_set_bits(bc->regmap, 0x8, 0xffffffff);
+ regmap_set_bits(bc->regmap, 0xc, 0xffffffff);
+ regmap_set_bits(bc->regmap, 0x10, 0xffffffff);
+ regmap_set_bits(bc->regmap, 0x14, 0xffffffff);
+ }
+
+ return NOTIFY_OK;
+}
+
static const struct imx8m_blk_ctrl_data imx8mp_vpu_blk_ctl_dev_data = {
.max_reg = 0x18,
- .power_notifier_fn = imx8mm_vpu_power_notifier,
+ .power_notifier_fn = imx8mp_vpu_power_notifier,
.domains = imx8mp_vpu_blk_ctl_domain_data,
.num_domains = ARRAY_SIZE(imx8mp_vpu_blk_ctl_domain_data),
};
#define PRIO(X) (X)
+#define BLK_CTRL_NO_PARENT UINT_MAX
+
struct imx93_blk_ctrl_domain;
struct imx93_blk_ctrl {
u32 cfg_prio;
};
+struct imx93_blk_ctrl_subdomain_link {
+ struct generic_pm_domain *parent;
+ struct generic_pm_domain *subdomain;
+};
+
struct imx93_blk_ctrl_domain_data {
const char *name;
const char * const *clk_names;
int num_clks;
u32 rst_mask;
u32 clk_mask;
+ u32 parent;
int num_qos;
struct imx93_blk_ctrl_qos qos[DOMAIN_MAX_QOS];
};
pm_genpd_remove(genpd);
}
+static void imx93_release_subdomain(void *data)
+{
+ struct imx93_blk_ctrl_subdomain_link *link = data;
+
+ pm_genpd_remove_subdomain(link->parent, link->subdomain);
+}
+
static struct lock_class_key blk_ctrl_genpd_lock_class;
static int imx93_blk_ctrl_probe(struct platform_device *pdev)
bc->onecell_data.domains[i] = &domain->genpd;
}
+ for (i = 0; i < bc_data->num_domains; i++) {
+ struct imx93_blk_ctrl_domain *domain = &bc->domains[i];
+ const struct imx93_blk_ctrl_domain_data *data = domain->data;
+ struct imx93_blk_ctrl_subdomain_link *link;
+
+ if (bc_data->skip_mask & BIT(i) ||
+ data->parent == BLK_CTRL_NO_PARENT)
+ continue;
+
+ link = devm_kzalloc(dev, sizeof(*link), GFP_KERNEL);
+ if (!link)
+ return -ENOMEM;
+
+ link->parent = &bc->domains[data->parent].genpd;
+ link->subdomain = &domain->genpd;
+
+ ret = pm_genpd_add_subdomain(&bc->domains[data->parent].genpd,
+ &domain->genpd);
+ if (ret)
+ return dev_err_probe(dev, ret, "failed to add subdomain %s\n",
+ domain->genpd.name);
+
+ ret = devm_add_action_or_reset(dev, imx93_release_subdomain, link);
+ if (ret)
+ return dev_err_probe(dev, ret,
+ "failed to add subdomain release callback\n");
+ }
+
ret = devm_pm_runtime_enable(dev);
if (ret)
return dev_err_probe(dev, ret, "failed to enable pm-runtime\n");
.name = "mediablk-mipi-dsi",
.clk_names = (const char *[]){ "dsi" },
.num_clks = 1,
- .rst_mask = BIT(11) | BIT(12),
- .clk_mask = BIT(11) | BIT(12),
+ .rst_mask = BIT(11),
+ .clk_mask = BIT(11),
+ .parent = IMX93_MEDIABLK_PD_MIPI_PHY,
},
[IMX93_MEDIABLK_PD_MIPI_CSI] = {
.name = "mediablk-mipi-csi",
.num_clks = 2,
.rst_mask = BIT(9) | BIT(10),
.clk_mask = BIT(9) | BIT(10),
+ .parent = IMX93_MEDIABLK_PD_MIPI_PHY,
},
[IMX93_MEDIABLK_PD_PXP] = {
.name = "mediablk-pxp",
.num_clks = 1,
.rst_mask = BIT(7) | BIT(8),
.clk_mask = BIT(7) | BIT(8),
+ .parent = BLK_CTRL_NO_PARENT,
.num_qos = 2,
.qos = {
{
.num_clks = 2,
.rst_mask = BIT(4) | BIT(5) | BIT(6),
.clk_mask = BIT(4) | BIT(5) | BIT(6),
+ .parent = BLK_CTRL_NO_PARENT,
.num_qos = 1,
.qos = {
{
.num_clks = 1,
.rst_mask = BIT(2) | BIT(3),
.clk_mask = BIT(2) | BIT(3),
+ .parent = BLK_CTRL_NO_PARENT,
.num_qos = 4,
.qos = {
{
}
}
},
+ [IMX93_MEDIABLK_PD_MIPI_PHY] = {
+ .name = "mediablk-mipi-phy",
+ .clk_names = NULL,
+ .num_clks = 0,
+ .rst_mask = BIT(12),
+ .clk_mask = BIT(12),
+ .parent = BLK_CTRL_NO_PARENT,
+ },
};
static const struct regmap_range imx93_media_blk_ctl_yes_ranges[] = {
return ret;
};
-static int scpsys_hwv_power_off(struct generic_pm_domain *genpd)
+static int scpsys_hwv_power_off_internal(struct scpsys_domain *pd)
{
- struct scpsys_domain *pd = container_of(genpd, struct scpsys_domain, genpd);
const struct scpsys_hwv_domain_data *hwv = pd->hwv_data;
struct scpsys *scpsys = pd->scpsys;
u32 val;
return ret;
};
+static int scpsys_hwv_power_off(struct generic_pm_domain *genpd)
+{
+ struct scpsys_domain *pd = container_of(genpd, struct scpsys_domain, genpd);
+
+ return scpsys_hwv_power_off_internal(pd);
+}
+
static int scpsys_ctl_pwrseq_on(struct scpsys_domain *pd)
{
struct scpsys *scpsys = pd->scpsys;
return ret;
}
-static int scpsys_power_off(struct generic_pm_domain *genpd)
+static int scpsys_power_off_internal(struct scpsys_domain *pd)
{
- struct scpsys_domain *pd = container_of(genpd, struct scpsys_domain, genpd);
struct scpsys *scpsys = pd->scpsys;
bool tmp;
int ret;
return 0;
}
+static int scpsys_power_off(struct generic_pm_domain *genpd)
+{
+ struct scpsys_domain *pd = container_of(genpd, struct scpsys_domain, genpd);
+
+ return scpsys_power_off_internal(pd);
+}
+
static struct
generic_pm_domain *scpsys_add_one_domain(struct scpsys *scpsys, struct device_node *node)
{
* late_init time.
*/
if (MTK_SCPD_CAPS(pd, MTK_SCPD_KEEP_DEFAULT_OFF)) {
- if (scpsys_domain_is_on(pd))
+ bool domain_is_on;
+
+ if (scpsys->soc_data->type == SCPSYS_MTCMOS_TYPE_HW_VOTER)
+ domain_is_on = scpsys_hwv_domain_is_enable_done(pd);
+ else
+ domain_is_on = scpsys_domain_is_on(pd);
+
+ if (domain_is_on)
dev_warn(scpsys->dev,
"%pOF: A default off power domain has been ON\n", node);
} else {
static void scpsys_remove_one_domain(struct scpsys_domain *pd)
{
+ struct scpsys *scpsys = pd->scpsys;
int ret;
/*
dev_err(pd->scpsys->dev,
"failed to remove domain '%s' : %d - state may be inconsistent\n",
pd->genpd.name, ret);
- if (scpsys_domain_is_on(pd))
- scpsys_power_off(&pd->genpd);
+
+ if (scpsys->soc_data->type == SCPSYS_MTCMOS_TYPE_HW_VOTER) {
+ if (scpsys_hwv_domain_is_enable_done(pd))
+ scpsys_hwv_power_off_internal(pd);
+ } else {
+ if (scpsys_domain_is_on(pd))
+ scpsys_power_off_internal(pd);
+ }
clk_bulk_put(pd->num_clks, pd->clks);
clk_bulk_put(pd->num_subsys_clks, pd->subsys_clks);
ret = regulator_lock_nested(rdev1, ww_ctx);
WARN_ON(ret);
ret = regulator_lock_nested(rdev2, ww_ctx);
- if (ret != -EDEADLOCK) {
+ if (ret != -EDEADLK) {
WARN_ON(ret);
goto exit;
}
swap(held, contended);
ret = regulator_lock_nested(contended, ww_ctx);
- if (ret != -EDEADLOCK) {
+ if (ret != -EDEADLK) {
WARN_ON(ret);
break;
}
#define LTC3676_DVBxA_REF_SELECT BIT(5)
#define LTC3676_DVBxB_PGOOD_MASK BIT(5)
-#define LTC3676_IRQSTAT_PGOOD_TIMEOUT BIT(3)
-#define LTC3676_IRQSTAT_UNDERVOLT_WARN BIT(4)
-#define LTC3676_IRQSTAT_UNDERVOLT_FAULT BIT(5)
-#define LTC3676_IRQSTAT_THERMAL_WARN BIT(6)
-#define LTC3676_IRQSTAT_THERMAL_FAULT BIT(7)
+#define LTC3676_IRQSTAT_PGOOD_TIMEOUT BIT(2)
+#define LTC3676_IRQSTAT_UNDERVOLT_WARN BIT(3)
+#define LTC3676_IRQSTAT_UNDERVOLT_FAULT BIT(4)
+#define LTC3676_IRQSTAT_THERMAL_WARN BIT(5)
+#define LTC3676_IRQSTAT_THERMAL_FAULT BIT(6)
enum ltc3676_reg {
LTC3676_SW1,
{ .compatible = "mediatek,mt6316d-regulator", .data = (void *)MT6316_TYPE_4PHASE },
{ /* sentinel */ }
};
+MODULE_DEVICE_TABLE(of, mt6316_regulator_match);
static struct spmi_driver mt6316_regulator_driver = {
.driver = {
{ .compatible = "mediatek,mt6363-regulator" },
{ /* sentinel */ }
};
+MODULE_DEVICE_TABLE(of, mt6363_regulator_match);
static struct platform_driver mt6363_regulator_driver = {
.driver = {
case IMX8MQ_RESET_PCIE_CTRL_APPS_EN:
case IMX8MQ_RESET_PCIE2_CTRL_APPS_EN:
+ case IMX8MQ_RESET_MIPI_CSI1_CORE_RESET:
+ case IMX8MQ_RESET_MIPI_CSI1_PHY_REF_RESET:
+ case IMX8MQ_RESET_MIPI_CSI1_ESC_RESET:
+ case IMX8MQ_RESET_MIPI_CSI2_CORE_RESET:
+ case IMX8MQ_RESET_MIPI_CSI2_PHY_REF_RESET:
+ case IMX8MQ_RESET_MIPI_CSI2_ESC_RESET:
case IMX8MQ_RESET_MIPI_DSI_PCLK_RESET_N:
case IMX8MQ_RESET_MIPI_DSI_ESC_RESET_N:
case IMX8MQ_RESET_MIPI_DSI_DPI_RESET_N:
data->membase = ioremap(res.start, size);
if (!data->membase) {
ret = -ENOMEM;
- goto err_alloc;
+ goto err_mem_region;
}
spin_lock_init(&data->lock);
return reset_controller_register(&data->rcdev);
+err_mem_region:
+ release_mem_region(res.start, size);
err_alloc:
kfree(data);
return ret;
[RESET_APMU_SDH0] = RESET_DATA(APMU_SDH0_CLK_RES_CTRL, 0, BIT(1)),
[RESET_APMU_SDH1] = RESET_DATA(APMU_SDH1_CLK_RES_CTRL, 0, BIT(1)),
[RESET_APMU_SDH2] = RESET_DATA(APMU_SDH2_CLK_RES_CTRL, 0, BIT(1)),
- [RESET_APMU_USB2_AHB] = RESET_DATA(APMU_USB_CLK_RES_CTRL, 0, BIT(1)),
+ [RESET_APMU_USB2_AHB] = RESET_DATA(APMU_USB_CLK_RES_CTRL, 0, BIT(0)),
[RESET_APMU_USB2_VCC] = RESET_DATA(APMU_USB_CLK_RES_CTRL, 0, BIT(2)),
[RESET_APMU_USB2_PHY] = RESET_DATA(APMU_USB_CLK_RES_CTRL, 0, BIT(3)),
[RESET_APMU_USB3_A_AHB] = RESET_DATA(APMU_USB_CLK_RES_CTRL, 0, BIT(5)),
}
-static void
+static noinline_for_stack void
bfa_fcs_fdmi_get_portattr(struct bfa_fcs_lport_fdmi_s *fdmi,
struct bfa_fcs_fdmi_port_attr_s *port_attr)
{
wqcb = efct_hw_reqtag_alloc(hw, efct_hw_wq_process_abort, io_to_abort);
if (!wqcb) {
efc_log_err(hw->os, "can't allocate request tag\n");
+ io_to_abort->abort_in_progress = false;
+ kref_put(&io_to_abort->ref, io_to_abort->release);
return -ENOSPC;
}
if (cmnd->fc_flags & FCP_CFL_LEN_MASK) {
efc_log_err(efct, "Additional CDB not supported\n");
+ efct_scsi_io_free(io);
return -EIO;
}
/*
/* Wait for functions invoked through call_rcu(&scmd->rcu, ...) */
rcu_barrier();
+ cancel_work_sync(&shost->eh_work);
if (shost->tmf_work_q)
destroy_workqueue(shost->tmf_work_q);
if (shost->ehandler)
INIT_LIST_HEAD(&shost->starved_list);
init_waitqueue_head(&shost->host_wait);
mutex_init(&shost->scan_mutex);
+ INIT_WORK(&shost->eh_work, scsi_rcu_eh_wakeup);
index = ida_alloc(&host_index_ida, GFP_KERNEL);
if (index < 0) {
if (phys_disk->in_reset) {
cmd->result = DID_RESET << 16;
+ atomic_dec(&phys_disk->ioaccel_cmds_out);
+ scsi_dma_unmap(cmd);
+ if (use_sg > h->ioaccel_maxsg)
+ hpsa_unmap_ioaccel2_sg_chain_block(h, cp);
return -1;
}
mempool_free(mboxq, phba->mbox_mem_pool);
goto out_free_bsmbx;
}
+ mempool_free(mboxq, phba->mbox_mem_pool);
/*
* 1 for cmd, 1 for rsp, NVME adds an extra one
goto out_free_sg_dma_buf;
}
- mempool_free(mboxq, phba->mbox_mem_pool);
-
/* Verify OAS is supported */
lpfc_sli4_oas_verify(phba);
}
}
+void scsi_rcu_eh_wakeup(struct work_struct *work)
+{
+ struct Scsi_Host *shost = container_of(work, struct Scsi_Host, eh_work);
+ unsigned long flags;
+ unsigned int busy;
+
+ /*
+ * Ensure any running scsi_dec_host_busy has completed its rcu section
+ * so changes to host state and host_eh_scheduled are visible to all
+ * future calls of scsi_dec_host_busy
+ */
+ synchronize_rcu();
+
+ busy = scsi_host_busy(shost);
+
+ spin_lock_irqsave(shost->host_lock, flags);
+ scsi_eh_wakeup(shost, busy);
+ spin_unlock_irqrestore(shost->host_lock, flags);
+}
+
/**
* scsi_schedule_eh - schedule EH for SCSI host
* @shost: SCSI host to invoke error handling on.
if (scsi_host_set_state(shost, SHOST_RECOVERY) == 0 ||
scsi_host_set_state(shost, SHOST_CANCEL_RECOVERY) == 0) {
shost->host_eh_scheduled++;
- scsi_eh_wakeup(shost, scsi_host_busy(shost));
+ queue_work(shost->tmf_work_q, &shost->eh_work);
}
spin_unlock_irqrestore(shost->host_lock, flags);
return sdev;
}
-/*
- * pktcdvd should have been integrated into the SCSI layers, but for historical
- * reasons like the old IDE driver it isn't. This export allows it to safely
- * probe if a given device is a SCSI one and only attach to that.
- */
-#ifdef CONFIG_CDROM_PKTCDVD_MODULE
-EXPORT_SYMBOL_GPL(scsi_device_from_queue);
-#endif
/**
* scsi_block_requests - Utility function used by low-level drivers to prevent
extern int scsi_error_handler(void *host);
extern enum scsi_disposition scsi_decide_disposition(struct scsi_cmnd *cmd);
extern void scsi_eh_wakeup(struct Scsi_Host *shost, unsigned int busy);
+extern void scsi_rcu_eh_wakeup(struct work_struct *work);
extern void scsi_eh_scmd_add(struct scsi_cmnd *);
void scsi_eh_ready_devs(struct Scsi_Host *shost,
struct list_head *work_q,
/* scsi_lib.c */
extern void scsi_device_unbusy(struct scsi_device *sdev, struct scsi_cmnd *cmd);
+extern struct scsi_device *scsi_device_from_queue(struct request_queue *q);
extern void scsi_queue_insert(struct scsi_cmnd *cmd,
enum scsi_qc_status reason);
extern void scsi_io_completion(struct scsi_cmnd *, unsigned int);
if (val >= SG_MAX_QUEUE)
break;
rinfo[val].req_state = srp->done + 1;
- rinfo[val].problem =
- srp->header.masked_status &
- srp->header.host_status &
- srp->header.driver_status;
+ rinfo[val].problem = srp->header.masked_status ||
+ srp->header.host_status ||
+ srp->header.driver_status;
if (srp->done)
rinfo[val].duration =
srp->header.duration;
u32 tegra_read_straps(void)
{
- WARN(!chipid, "Tegra ABP MISC not yet available\n");
+ WARN(!apbmisc_base, "Tegra ABP MISC not yet available\n");
return strapping;
}
/* Clear interrupt */
writel(irq_status, cqspi->iobase + CQSPI_REG_IRQSTATUS);
- if (cqspi->use_dma_read && ddata && ddata->get_dma_status)
- irq_status = ddata->get_dma_status(cqspi);
- else if (cqspi->slow_sram)
+ if (cqspi->use_dma_read && ddata && ddata->get_dma_status) {
+ if (ddata->get_dma_status(cqspi)) {
+ complete(&cqspi->transfer_complete);
+ return IRQ_HANDLED;
+ }
+ } else if (cqspi->slow_sram) {
irq_status &= CQSPI_IRQ_MASK_RD_SLOW_SRAM | CQSPI_IRQ_MASK_WR;
- else
+ } else {
irq_status &= CQSPI_IRQ_MASK_RD | CQSPI_IRQ_MASK_WR;
+ }
if (irq_status)
complete(&cqspi->transfer_complete);
static inline bool dw_spi_dma_tx_busy(struct dw_spi *dws)
{
- return !(dw_readl(dws, DW_SPI_SR) & DW_SPI_SR_TF_EMPT);
+ return (dw_readl(dws, DW_SPI_SR) &
+ (DW_SPI_SR_BUSY | DW_SPI_SR_TF_EMPT)) != DW_SPI_SR_TF_EMPT;
}
static int dw_spi_dma_wait_tx_done(struct dw_spi *dws,
cnt = (_TIMESTAMP_ + _BEACON_ITERVAL_ + _CAPABILITY_);
while (cnt < in_len) {
+ if (cnt + 2 > in_len)
+ break;
+ if (cnt + 2 + in_ie[cnt + 1] > in_len)
+ break;
authmode = in_ie[cnt];
if (authmode == WLAN_EID_BSS_AC_ACCESS_DELAY &&
+ in_ie[cnt + 1] >= 8 &&
(!memcmp(&in_ie[cnt + 6], wapi_oui1, 4) ||
!memcmp(&in_ie[cnt + 6], wapi_oui2, 4))) {
if (wapi_ie)
cnt = (_TIMESTAMP_ + _BEACON_ITERVAL_ + _CAPABILITY_);
while (cnt < in_len) {
+ if (cnt + 2 > in_len)
+ break;
+ if (cnt + 2 + in_ie[cnt + 1] > in_len)
+ break;
authmode = in_ie[cnt];
if ((authmode == WLAN_EID_VENDOR_SPECIFIC) &&
+ in_ie[cnt + 1] >= 4 &&
(!memcmp(&in_ie[cnt + 2], &wpa_oui[0], 4))) {
if (wpa_ie)
memcpy(wpa_ie, &in_ie[cnt], in_ie[cnt + 1] + 2);
if (len_attr)
*len_attr = 0;
+ if (wps_ielen < 6)
+ return attr_ptr;
+
if ((wps_ie[0] != WLAN_EID_VENDOR_SPECIFIC) ||
(memcmp(wps_ie + 2, wps_oui, 4))) {
return attr_ptr;
while (attr_ptr - wps_ie < wps_ielen) {
/* 4 = 2(Attribute ID) + 2(Length) */
+ if (attr_ptr + 4 > wps_ie + wps_ielen)
+ break;
u16 attr_id = get_unaligned_be16(attr_ptr);
u16 attr_data_len = get_unaligned_be16(attr_ptr + 2);
u16 attr_len = attr_data_len + 4;
if ((pmlmeinfo->state&0x03) != WIFI_FW_AP_STATE)
return _FAIL;
+ if (len < WLAN_HDR_A3_LEN)
+ return _FAIL;
+
sa = GetAddr2Ptr(pframe);
auth_mode = psecuritypriv->dot11AuthAlgrthm;
prxattrib->hdrlen = WLAN_HDR_A3_LEN;
prxattrib->encrypt = _WEP40_;
+ if (len < WLAN_HDR_A3_LEN + 8)
+ return _FAIL;
+
iv = pframe+prxattrib->hdrlen;
prxattrib->key_index = ((iv[3]>>6)&0x3);
p = rtw_get_ie(pframe + WLAN_HDR_A3_LEN + 4 + _AUTH_IE_OFFSET_, WLAN_EID_CHALLENGE, (int *)&ie_len,
len - WLAN_HDR_A3_LEN - _AUTH_IE_OFFSET_ - 4);
- if (!p || ie_len <= 0) {
+ if (!p || ie_len != 128) {
status = WLAN_STATUS_CHALLENGE_FAIL;
goto auth_fail;
}
/* to handle HT, WMM, rate adaptive, update MAC reg */
/* for not to handle the synchronous IO in the tasklet */
for (i = (6 + WLAN_HDR_A3_LEN); i < pkt_len;) {
+ if (i + sizeof(*pIE) > pkt_len)
+ break;
pIE = (struct ndis_80211_var_ie *)(pframe + i);
+ if (i + sizeof(*pIE) + pIE->length > pkt_len)
+ break;
switch (pIE->element_id) {
case WLAN_EID_VENDOR_SPECIFIC:
/* vendor specific IE, such as WPA, WMM, WPS */
for (i = sizeof(struct ndis_802_11_fix_ie); i < pmlmeinfo->network.ie_length;) {
+ if (i + sizeof(*pIE) > pmlmeinfo->network.ie_length)
+ break;
pIE = (struct ndis_80211_var_ie *)(pmlmeinfo->network.ies + i);
+ if (i + sizeof(*pIE) + pIE->length > pmlmeinfo->network.ie_length)
+ break;
switch (pIE->element_id) {
case WLAN_EID_VENDOR_SPECIFIC:
/* sizeof(struct ndis_802_11_fix_ie) */
for (i = _FIXED_IE_LENGTH_; i < pnetwork->ie_length;) {
+ if (i + sizeof(*pIE) > pnetwork->ie_length)
+ break;
pIE = (struct ndis_80211_var_ie *)(pnetwork->ies + i);
+ if (i + sizeof(*pIE) + pIE->length > pnetwork->ie_length)
+ break;
switch (pIE->element_id) {
case WLAN_EID_VENDOR_SPECIFIC:/* Get WMM IE. */
pmlmeinfo->HT_caps_enable = 1;
- for (i = 0; i < (pIE->length); i++) {
+ for (i = 0; i < umin(pIE->length,
+ sizeof(pmlmeinfo->HT_caps.u.HT_cap)); i++) {
if (i != 2) {
/* Commented by Albert 2010/07/12 */
/* Got the endian issue here. */
len = pkt_len - (_BEACON_IE_OFFSET_ + WLAN_HDR_A3_LEN);
for (i = 0; i < len;) {
+ if (i + sizeof(*pIE) > len)
+ break;
pIE = (struct ndis_80211_var_ie *)(pframe + (_BEACON_IE_OFFSET_ + WLAN_HDR_A3_LEN) + i);
+ if (i + sizeof(*pIE) + pIE->length > len)
+ break;
switch (pIE->element_id) {
case WLAN_EID_VENDOR_SPECIFIC:
break;
}
- i += (pIE->length + 2);
+ i += sizeof(*pIE) + pIE->length;
}
}
for (i = sizeof(struct ndis_802_11_fix_ie); i < pmlmeinfo->network.ie_length;) {
pIE = (struct ndis_80211_var_ie *)(pmlmeinfo->network.ies + i);
+ if (i + sizeof(*pIE) > pmlmeinfo->network.ie_length)
+ break;
+ if (i + sizeof(*pIE) + pIE->length > pmlmeinfo->network.ie_length)
+ break;
+
switch (pIE->element_id) {
case WLAN_EID_VENDOR_SPECIFIC:
- if ((!memcmp(pIE->data, RTW_WPA_OUI, 4)) && (!memcmp((pIE->data + 12), WPA_TKIP_CIPHER, 4)))
+ if (pIE->length >= 16 &&
+ !memcmp(pIE->data, RTW_WPA_OUI, 4) &&
+ !memcmp((pIE->data + 12), WPA_TKIP_CIPHER, 4))
return true;
break;
case WLAN_EID_RSN:
- if (!memcmp((pIE->data + 8), RSN_TKIP_CIPHER, 4))
+ if (pIE->length >= 12 &&
+ !memcmp((pIE->data + 8), RSN_TKIP_CIPHER, 4))
return true;
break;
break;
}
- i += (pIE->length + 2);
+ i += sizeof(*pIE) + pIE->length;
}
return false;
pwpa = rtw_get_wpa_ie(buf, &wpa_ielen, ielen);
if (pwpa && wpa_ielen > 0) {
+ if (wpa_ielen + 2 > sizeof(padapter->securitypriv.supplicant_ie)) {
+ ret = -EINVAL;
+ goto exit;
+ }
if (rtw_parse_wpa_ie(pwpa, wpa_ielen + 2, &group_cipher, &pairwise_cipher, NULL) == _SUCCESS) {
padapter->securitypriv.dot11AuthAlgrthm = dot11AuthAlgrthm_8021X;
padapter->securitypriv.ndisauthtype = Ndis802_11AuthModeWPAPSK;
pwpa2 = rtw_get_wpa2_ie(buf, &wpa2_ielen, ielen);
if (pwpa2 && wpa2_ielen > 0) {
+ if (wpa2_ielen + 2 > sizeof(padapter->securitypriv.supplicant_ie)) {
+ ret = -EINVAL;
+ goto exit;
+ }
if (rtw_parse_wpa2_ie(pwpa2, wpa2_ielen + 2, &group_cipher, &pairwise_cipher, NULL) == _SUCCESS) {
padapter->securitypriv.dot11AuthAlgrthm = dot11AuthAlgrthm_8021X;
padapter->securitypriv.ndisauthtype = Ndis802_11AuthModeWPA2PSK;
int _rtw_pktfile_read(struct pkt_file *pfile, u8 *rmem, unsigned int rlen)
{
int ret;
+ unsigned int remain = rtw_remainder_len(pfile);
- if (rtw_remainder_len(pfile) < rlen)
- return -EINVAL;
+ /* clamp to bytes remaining; the coalesce loop relies on short reads */
+ if (rlen > remain)
+ rlen = remain;
if (rmem) {
ret = skb_copy_bits(pfile->pkt, pfile->buf_len - pfile->pkt_len, rmem, rlen);
{
struct list_head *pos = NULL;
struct list_head *tmplist;
+ struct vme_lm_resource *lm;
struct vme_master_resource *master_image;
struct vme_slave_resource *slave_image;
int i;
vme_unregister_bridge(fake_bridge);
fake_crcsr_exit(fake_bridge);
+ /* resources are stored in link list */
+ list_for_each_safe(pos, tmplist, &fake_bridge->lm_resources) {
+ lm = list_entry(pos, struct vme_lm_resource, list);
+ list_del(pos);
+ kfree(lm);
+ }
+
/* resources are stored in link list */
list_for_each_safe(pos, tmplist, &fake_bridge->slave_resources) {
slave_image = list_entry(pos, struct vme_slave_resource, list);
{
struct list_head *pos = NULL;
struct list_head *tmplist;
+ struct vme_lm_resource *lm;
struct vme_master_resource *master_image;
struct vme_slave_resource *slave_image;
struct vme_dma_resource *dma_ctrlr;
tsi148_crcsr_exit(tsi148_bridge, pdev);
+ /* resources are stored in link list */
+ list_for_each_safe(pos, tmplist, &tsi148_bridge->lm_resources) {
+ lm = list_entry(pos, struct vme_lm_resource, list);
+ list_del(pos);
+ kfree(lm);
+ }
+
/* resources are stored in link list */
list_for_each_safe(pos, tmplist, &tsi148_bridge->dma_resources) {
dma_ctrlr = list_entry(pos, struct vme_dma_resource, list);
{
void *image_ptr;
+ /*
+ * The slave window (image_size) can exceed the fixed kern_buf
+ * (size_buf == PCI_BUF_SIZE), so bound the copy to kern_buf.
+ * *ppos is >= 0 here (checked by the caller), so the
+ * subtraction below cannot wrap.
+ */
+ if (*ppos >= image[minor].size_buf)
+ return 0;
+ if (count > image[minor].size_buf - *ppos)
+ count = image[minor].size_buf - *ppos;
+
image_ptr = image[minor].kern_buf + *ppos;
if (copy_to_user(buf, image_ptr, (unsigned long)count))
return -EFAULT;
{
void *image_ptr;
+ /*
+ * The slave window (image_size) can exceed the fixed kern_buf
+ * (size_buf == PCI_BUF_SIZE), so bound the copy to kern_buf.
+ * *ppos is >= 0 here (checked by the caller), so the
+ * subtraction below cannot wrap.
+ */
+ if (*ppos >= image[minor].size_buf)
+ return 0;
+ if (count > image[minor].size_buf - *ppos)
+ count = image[minor].size_buf - *ppos;
+
image_ptr = image[minor].kern_buf + *ppos;
if (copy_from_user(image_ptr, buf, (unsigned long)count))
return -EFAULT;
static bool iscsi_parse_pr_out_transport_id(
struct se_portal_group *se_tpg,
char *buf,
+ u32 buf_len,
u32 *out_tid_len,
char **port_nexus_ptr,
char *i_str)
{
char *p;
+ u32 tid_len;
int i;
- u8 format_code = (buf[0] & 0xc0);
+ u8 format_code;
+
+ /*
+ * The 4-byte iSCSI TransportID header (FORMAT CODE + 2-byte ADDITIONAL
+ * LENGTH) must be present before any of it can be parsed.
+ */
+ if (buf_len < 4)
+ return false;
+
+ format_code = buf[0] & 0xc0;
/*
* Check for FORMAT CODE 00b or 01b from spc4r17, section 7.5.4.6:
*
return false;
}
/*
- * If the caller wants the TransportID Length, we set that value for the
- * entire iSCSI Tarnsport ID now.
+ * Reconstruct the self-described TransportID length from the ADDITIONAL
+ * LENGTH field plus the 4-byte header. Reject it if it is below the
+ * spc4r17 section 7.5.4.6 minimum (ADDITIONAL LENGTH shall be at least
+ * 20) or if it runs past the bytes actually received, so that every
+ * access below stays inside the TransportID.
*/
- if (out_tid_len) {
- /* The shift works thanks to integer promotion rules */
- *out_tid_len = get_unaligned_be16(&buf[2]);
- /* Add four bytes for iSCSI Transport ID header */
- *out_tid_len += 4;
- }
+ tid_len = get_unaligned_be16(&buf[2]) + 4;
+ if (tid_len < 24 || tid_len > buf_len)
+ return false;
+ if (out_tid_len)
+ *out_tid_len = tid_len;
/*
* Check for ',i,0x' separator between iSCSI Name and iSCSI Initiator
* format.
*/
if (format_code == 0x40) {
- p = strstr(&buf[4], ",i,0x");
+ p = strnstr(&buf[4], ",i,0x", tid_len - 4);
if (!p) {
- pr_err("Unable to locate \",i,0x\" separator"
- " for Initiator port identifier: %s\n",
- &buf[4]);
+ pr_err("Unable to locate \",i,0x\" separator in iSCSI TransportID\n");
+ return false;
+ }
+ /*
+ * The iSCSI name runs from &buf[4] up to the separator; reject it
+ * if it cannot fit in i_str[TRANSPORT_IQN_LEN].
+ */
+ if (p - &buf[4] >= TRANSPORT_IQN_LEN) {
+ pr_err("iSCSI Initiator port name too long in TransportID\n");
return false;
}
*p = '\0'; /* Terminate iSCSI Name */
p += 5; /* Skip over ",i,0x" separator */
+ /*
+ * The ISID must follow the separator. A ",i,0x" sitting at the
+ * very end of the TransportID leaves no ISID and would point the
+ * port nexus at buf + tid_len, i.e. past the descriptor, which
+ * the registration code then reads as the ISID string.
+ */
+ if (p >= buf + tid_len) {
+ pr_err("Missing ISID in iSCSI Initiator port TransportID\n");
+ return false;
+ }
*port_nexus_ptr = p;
/*
* Go ahead and do the lower case conversion of the received
* for comparison against the running iSCSI session's ISID from
* iscsi_target.c:lio_sess_get_initiator_sid()
*/
- for (i = 0; i < 12; i++) {
+ for (i = 0; i < 12 && p < buf + tid_len; i++) {
/*
* The first ISCSI INITIATOR SESSION ID field byte
* containing an ASCII null character terminates the
*p = tolower(*p);
p++;
}
- } else
+ strscpy(i_str, &buf[4], TRANSPORT_IQN_LEN);
+ } else {
*port_nexus_ptr = NULL;
-
- strscpy(i_str, &buf[4], TRANSPORT_IQN_LEN);
+ /*
+ * FORMAT CODE 00b: the name occupies buf[4..tid_len-1]. The
+ * declared length tid_len - 4 must fit in i_str[TRANSPORT_IQN_LEN].
+ * (For 01b the same tid_len bound would be over-restrictive: the
+ * descriptor also carries the separator and ISID, so a legal
+ * <=223-byte name gives tid_len up to 244.)
+ */
+ if (tid_len - 4 >= TRANSPORT_IQN_LEN) {
+ pr_err("iSCSI Initiator port name too long in TransportID\n");
+ return false;
+ }
+ strscpy(i_str, &buf[4], tid_len - 4);
+ }
return true;
}
}
bool target_parse_pr_out_transport_id(struct se_portal_group *tpg,
- char *buf, u32 *out_tid_len, char **port_nexus_ptr, char *i_str)
+ char *buf, u32 buf_len, u32 *out_tid_len,
+ char **port_nexus_ptr, char *i_str)
{
+ /*
+ * The fixed-length SAS/SRP/FCP/SBP TransportIDs are 24 bytes; the iSCSI
+ * format is variable and bounds itself against buf_len below.
+ */
+ if (tpg->proto_id != SCSI_PROTOCOL_ISCSI && buf_len < 24)
+ return false;
+
switch (tpg->proto_id) {
case SCSI_PROTOCOL_SAS:
/*
sbp_parse_pr_out_transport_id(buf, i_str);
break;
case SCSI_PROTOCOL_ISCSI:
- return iscsi_parse_pr_out_transport_id(tpg, buf, out_tid_len,
- port_nexus_ptr, i_str);
+ return iscsi_parse_pr_out_transport_id(tpg, buf, buf_len,
+ out_tid_len, port_nexus_ptr, i_str);
default:
pr_err("Unknown proto_id: 0x%02x\n", tpg->proto_id);
return false;
struct t10_pr_registration *pr_reg, int *format_code,
unsigned char *buf);
bool target_parse_pr_out_transport_id(struct se_portal_group *tpg,
- char *buf, u32 *out_tid_len, char **port_nexus_ptr, char *i_str);
+ char *buf, u32 buf_len, u32 *out_tid_len,
+ char **port_nexus_ptr, char *i_str);
/* target_core_hba.c */
struct se_hba *core_alloc_hba(const char *, u32, u32);
iport_ptr = NULL;
tid_found = target_parse_pr_out_transport_id(tmp_tpg,
- ptr, &tid_len, &iport_ptr, i_str);
+ ptr, tpdl, &tid_len, &iport_ptr, i_str);
if (!tid_found)
continue;
/*
goto out;
}
tid_found = target_parse_pr_out_transport_id(dest_se_tpg,
- &buf[24], &tmp_tid_len, &iport_ptr, initiator_str);
+ &buf[24], tid_len, &tmp_tid_len, &iport_ptr, initiator_str);
if (!tid_found) {
pr_err("SPC-3 PR REGISTER_AND_MOVE: Unable to locate"
" initiator_str from Transport ID\n");
goto out;
}
- transport_kunmap_data_sg(cmd);
- buf = NULL;
-
pr_debug("SPC-3 PR [%s] Extracted initiator %s identifier: %s"
" %s\n", dest_tf_ops->fabric_name, (iport_ptr != NULL) ?
"port" : "device", initiator_str, (iport_ptr != NULL) ?
core_scsi3_update_and_write_aptpl(cmd->se_dev, aptpl);
core_scsi3_put_pr_reg(dest_pr_reg);
+ /*
+ * iport_ptr aliases the PR-OUT parameter list mapped above, so the
+ * buffer is unmapped only here on success (and at out: on error).
+ */
+ transport_kunmap_data_sg(cmd);
return 0;
out:
if (buf)
#include <linux/module.h>
#include <linux/pci.h>
#include <linux/rational.h>
+#include <linux/util_macros.h>
#include <linux/dma/hsu.h>
.freq = 133333333,
.base_baud = 115200,
.bar = 1,
- .setup = dnv_setup,
- .exit = dnv_exit,
+ /*
+ * Errata:
+ * HSUART May Stop Functioning when DMA is Active.
+ *
+ * - Denverton document #572409, rev 3.4, DNV60
+ * - Ice Lake Xeon D document #714070, ICXD65
+ * - Snowridge document #731931, SNR44
+ */
+ .setup = PTR_IF(false, dnv_setup),
+ .exit = PTR_IF(false, dnv_exit),
};
static const struct pci_device_id pci_ids[] = {
dev_err(p->port.dev, "teardown incomplete\n");
}
}
+
+ dma->rx_running = 0;
if (!count)
goto out;
ret = tty_insert_flip_string(tty_port, dma->rx_buf, count);
- dma->rx_running = 0;
p->port.icount.rx += ret;
p->port.icount.buf_overrun += count - ret;
out:
tx_ready = wait_for_lsr(up, bits);
- /* Wait up to 1s for flow control if necessary */
- if (uart_console_hwflow_active(&up->port)) {
+ /*
+ * Wait up to 1s for flow control if necessary.
+ * When 'no_console_suspend' is active (in the window between
+ * suspend() and resume()), flow control is temporarily ignored
+ * because the canary workaround is not reliable in all situations,
+ * leading to flow control timeouts for every character.
+ */
+ if (uart_console_hwflow_active(&up->port) && !up->canary) {
for (tmout = 1000000; tmout; tmout--) {
unsigned int msr = serial_in(up, UART_MSR);
up->msr_saved_flags |= msr & MSR_SAVE_FLAGS;
return 0;
}
+static int max310x_gpio_get_direction(struct gpio_chip *chip, unsigned int offset)
+{
+ struct max310x_port *s = gpiochip_get_data(chip);
+ struct uart_port *port = &s->p[offset / 4].port;
+ unsigned int val;
+
+ val = max310x_port_read(port, MAX310X_GPIOCFG_REG);
+
+ return val & BIT(offset % 4) ? GPIO_LINE_DIRECTION_OUT : GPIO_LINE_DIRECTION_IN;
+}
+
static int max310x_gpio_direction_input(struct gpio_chip *chip, unsigned int offset)
{
struct max310x_port *s = gpiochip_get_data(chip);
s->gpio.owner = THIS_MODULE;
s->gpio.parent = dev;
s->gpio.label = devtype->name;
+ s->gpio.get_direction = max310x_gpio_get_direction;
s->gpio.direction_input = max310x_gpio_direction_input;
s->gpio.get = max310x_gpio_get;
s->gpio.direction_output= max310x_gpio_direction_output;
data |= MSM_UART_MR1_AUTO_RFR_LEVEL0 & rfr_level;
msm_write(port, data, MSM_UART_MR1);
- if (msm_port->is_uartdm) {
+ /* Disable DMA for console to prevent PIO/DMA collisions */
+ if (msm_port->is_uartdm && !uart_console(port)) {
msm_request_tx_dma(msm_port, msm_port->uart.mapbase);
msm_request_rx_dma(msm_port, msm_port->uart.mapbase);
}
/*
* Compute xterm-style modifier parameter for CSI sequences.
* Returns 1 + (shift ? 1 : 0) + (alt ? 2 : 0) + (ctrl ? 4 : 0)
+ *
+ * Only the canonical modifier weights are counted. The left/right variants
+ * (KG_SHIFTL, KG_SHIFTR, KG_CTRLL, KG_CTRLR) and KG_ALTGR are commonly
+ * repurposed as keymap layout-group or level selectors rather than as plain
+ * modifiers (for instance XKB-derived keymaps select the layout group with
+ * KG_SHIFTL/KG_SHIFTR), so counting them would encode a spurious modifier.
*/
static int csi_modifier_param(void)
{
int mod = 1;
- if (shift_state & (BIT(KG_SHIFT) | BIT(KG_SHIFTL) | BIT(KG_SHIFTR)))
+ if (shift_state & BIT(KG_SHIFT))
mod += 1;
- if (shift_state & (BIT(KG_ALT) | BIT(KG_ALTGR)))
+ if (shift_state & BIT(KG_ALT))
mod += 2;
- if (shift_state & (BIT(KG_CTRL) | BIT(KG_CTRLL) | BIT(KG_CTRLR)))
+ if (shift_state & BIT(KG_CTRL))
mod += 4;
return mod;
}
TP_STRUCT__entry(
__field(struct ufs_hba *, hba)
+ __string(dev_name, dev_name(hba->dev))
__field(int, state)
),
TP_fast_assign(
+ __assign_str(dev_name);
__entry->hba = hba;
__entry->state = state;
),
TP_printk("%s: gating state changed to %s",
- dev_name(__entry->hba->dev),
+ __get_str(dev_name),
__print_symbolic(__entry->state, UFSCHD_CLK_GATING_STATES))
);
TP_STRUCT__entry(
__field(struct ufs_hba *, hba)
+ __string(dev_name, dev_name(hba->dev))
__string(state, state)
__string(clk, clk)
__field(u32, prev_state)
TP_fast_assign(
__entry->hba = hba;
+ __assign_str(dev_name);
__assign_str(state);
__assign_str(clk);
__entry->prev_state = prev_state;
),
TP_printk("%s: %s %s from %u to %u Hz",
- dev_name(__entry->hba->dev), __get_str(state), __get_str(clk),
+ __get_str(dev_name), __get_str(state), __get_str(clk),
__entry->prev_state, __entry->curr_state)
);
TP_STRUCT__entry(
__field(struct ufs_hba *, hba)
+ __string(dev_name, dev_name(hba->dev))
__string(state, state)
),
TP_fast_assign(
__entry->hba = hba;
+ __assign_str(dev_name);
__assign_str(state);
),
TP_printk("%s: auto bkops - %s",
- dev_name(__entry->hba->dev), __get_str(state))
+ __get_str(dev_name), __get_str(state))
);
DECLARE_EVENT_CLASS(ufshcd_profiling_template,
TP_STRUCT__entry(
__field(struct ufs_hba *, hba)
+ __string(dev_name, dev_name(hba->dev))
__string(profile_info, profile_info)
__field(s64, time_us)
__field(int, err)
TP_fast_assign(
__entry->hba = hba;
+ __assign_str(dev_name);
__assign_str(profile_info);
__entry->time_us = time_us;
__entry->err = err;
),
TP_printk("%s: %s: took %lld usecs, err %d",
- dev_name(__entry->hba->dev), __get_str(profile_info),
+ __get_str(dev_name), __get_str(profile_info),
__entry->time_us, __entry->err)
);
__field(s64, usecs)
__field(int, err)
__field(struct ufs_hba *, hba)
+ __string(dev_name, dev_name(hba->dev))
__field(int, dev_state)
__field(int, link_state)
),
__entry->usecs = usecs;
__entry->err = err;
__entry->hba = hba;
+ __assign_str(dev_name);
__entry->dev_state = dev_state;
__entry->link_state = link_state;
),
TP_printk(
"%s: took %lld usecs, dev_state: %s, link_state: %s, err %d",
- dev_name(__entry->hba->dev),
+ __get_str(dev_name),
__entry->usecs,
__print_symbolic(__entry->dev_state, UFS_PWR_MODES),
__print_symbolic(__entry->link_state, UFS_LINK_STATES),
TP_STRUCT__entry(
__field(struct scsi_device *, sdev)
__field(struct ufs_hba *, hba)
+ __string(dev_name, dev_name(&sdev->sdev_dev))
__field(enum ufs_trace_str_t, str_t)
__field(unsigned int, tag)
__field(u32, doorbell)
),
TP_fast_assign(
+ __assign_str(dev_name);
__entry->sdev = sdev;
__entry->hba = hba;
__entry->str_t = str_t;
TP_printk(
"%s: %s: tag: %u, DB: 0x%x, size: %d, IS: %u, LBA: %llu, opcode: 0x%x (%s), group_id: 0x%x, hwq_id: %d",
show_ufs_cmd_trace_str(__entry->str_t),
- dev_name(&__entry->sdev->sdev_dev), __entry->tag,
+ __get_str(dev_name), __entry->tag,
__entry->doorbell, __entry->transfer_len, __entry->intr,
__entry->lba, (u32)__entry->opcode, str_opcode(__entry->opcode),
(u32)__entry->group_id, __entry->hwq_id
TP_STRUCT__entry(
__field(struct ufs_hba *, hba)
+ __string(dev_name, dev_name(hba->dev))
__field(enum ufs_trace_str_t, str_t)
__field(u32, cmd)
__field(u32, arg1)
TP_fast_assign(
__entry->hba = hba;
+ __assign_str(dev_name);
__entry->str_t = str_t;
__entry->cmd = cmd;
__entry->arg1 = arg1;
TP_printk(
"%s: %s: cmd: 0x%x, arg1: 0x%x, arg2: 0x%x, arg3: 0x%x",
- show_ufs_cmd_trace_str(__entry->str_t), dev_name(__entry->hba->dev),
+ show_ufs_cmd_trace_str(__entry->str_t), __get_str(dev_name),
__entry->cmd, __entry->arg1, __entry->arg2, __entry->arg3
)
);
TP_STRUCT__entry(
__field(struct ufs_hba *, hba)
+ __string(dev_name, dev_name(hba->dev))
__field(enum ufs_trace_str_t, str_t)
__array(unsigned char, hdr, 12)
__array(unsigned char, tsf, 16)
TP_fast_assign(
__entry->hba = hba;
+ __assign_str(dev_name);
__entry->str_t = str_t;
memcpy(__entry->hdr, hdr, sizeof(__entry->hdr));
memcpy(__entry->tsf, tsf, sizeof(__entry->tsf));
TP_printk(
"%s: %s: HDR:%s, %s:%s",
- show_ufs_cmd_trace_str(__entry->str_t), dev_name(__entry->hba->dev),
+ show_ufs_cmd_trace_str(__entry->str_t), __get_str(dev_name),
__print_hex(__entry->hdr, sizeof(__entry->hdr)),
show_ufs_cmd_trace_tsf(__entry->tsf_t),
__print_hex(__entry->tsf, sizeof(__entry->tsf))
TP_STRUCT__entry(
__field(struct ufs_hba *, hba)
+ __string(dev_name, dev_name(hba->dev))
__field(u16, status)
),
TP_fast_assign(
__entry->hba = hba;
+ __assign_str(dev_name);
__entry->status = status;
),
TP_printk("%s: status 0x%x",
- dev_name(__entry->hba->dev), __entry->status
+ __get_str(dev_name), __entry->status
)
);
static void uea_upload_pre_firmware(const struct firmware *fw_entry,
void *context)
{
- struct usb_device *usb = context;
+ struct usb_interface *intf = context;
+ struct usb_device *usb = interface_to_usbdev(intf);
+ struct completion *fw_done = usb_get_intfdata(intf);
const u8 *pfw;
u8 value;
u32 crc = 0;
uea_err(usb, "firmware is corrupted\n");
err:
release_firmware(fw_entry);
+ complete(fw_done);
}
/*
* uea_load_firmware - Load usb firmware for pre-firmware devices.
*/
-static int uea_load_firmware(struct usb_device *usb, unsigned int ver)
+static int uea_load_firmware(struct usb_interface *intf, unsigned int ver)
{
int ret;
char *fw_name = EAGLE_FIRMWARE;
+ struct usb_device *usb = interface_to_usbdev(intf);
uea_info(usb, "pre-firmware device, uploading firmware\n");
}
ret = request_firmware_nowait(THIS_MODULE, 1, fw_name, &usb->dev,
- GFP_KERNEL, usb,
+ GFP_KERNEL, intf,
uea_upload_pre_firmware);
if (ret)
uea_err(usb, "firmware %s is not available\n", fw_name);
usb_reset_device(usb);
- if (UEA_IS_PREFIRM(id))
- return uea_load_firmware(usb, UEA_CHIP_VERSION(id));
+ if (UEA_IS_PREFIRM(id)) {
+ struct completion *fw_done;
+
+ /* Wait for the firmware load to be done, in .disconnect() */
+ fw_done = kzalloc_obj(*fw_done);
+ if (!fw_done)
+ return -ENOMEM;
+
+ init_completion(fw_done);
+ usb_set_intfdata(intf, fw_done);
+
+ ret = uea_load_firmware(intf, UEA_CHIP_VERSION(id));
+ if (ret)
+ kfree(fw_done);
+
+ return ret;
+ }
ret = usbatm_usb_probe(intf, id, &uea_usbatm_driver);
if (ret == 0) {
usbatm_usb_disconnect(intf);
mutex_unlock(&uea_mutex);
uea_info(usb, "ADSL device removed\n");
+ } else if (usb->config->desc.bNumInterfaces == 1) {
+ struct completion *fw_done = usb_get_intfdata(intf);
+
+ uea_dbg(usb, "pre-firmware device, waiting firmware upload\n");
+ wait_for_completion(fw_done);
+ uea_dbg(usb, "pre-firmware device, finished waiting\n");
+ kfree(fw_done);
}
}
}
}
+ cdnsp_free_stream_ctx(pdev, pep);
+
cleanup_stream_rings:
kfree(pep->stream_info.stream_rings);
{ USB_DEVICE(0x1901, 0x0006), /* GE Healthcare Patient Monitor UI Controller */
.driver_info = DISABLE_ECHO, /* DISABLE ECHO in termios flag */
},
+ { USB_DEVICE(0x1965, 0x0017), /* Uniden BC125AT */
+ .driver_info = NO_UNION_NORMAL, /* has no union descriptor */
+ },
{ USB_DEVICE(0x1965, 0x0018), /* Uniden UBC125XLT */
.driver_info = NO_UNION_NORMAL, /* has no union descriptor */
},
ulpi->dev.parent = dev; /* needed early for ops */
ulpi->dev.bus = &ulpi_bus;
ulpi->dev.type = &ulpi_dev_type;
+
+ device_initialize(&ulpi->dev);
+
dev_set_name(&ulpi->dev, "%s.ulpi", dev_name(dev));
ACPI_COMPANION_SET(&ulpi->dev, ACPI_COMPANION(dev));
ret = ulpi_of_register(ulpi);
- if (ret) {
- kfree(ulpi);
+ if (ret)
return ret;
- }
ret = ulpi_read_id(ulpi);
- if (ret) {
- of_node_put(ulpi->dev.of_node);
- kfree(ulpi);
+ if (ret)
return ret;
- }
- ret = device_register(&ulpi->dev);
- if (ret) {
- put_device(&ulpi->dev);
+ ret = device_add(&ulpi->dev);
+ if (ret)
return ret;
- }
root = debugfs_create_dir(dev_name(&ulpi->dev), ulpi_root);
debugfs_create_file("regs", 0444, root, ulpi, &ulpi_regs_fops);
ulpi->ops = ops;
ret = ulpi_register(dev, ulpi);
- if (ret)
+ if (ret) {
+ put_device(&ulpi->dev);
return ERR_PTR(ret);
-
+ }
return ulpi;
}
delay = HUB_LONG_RESET_TIME;
}
- dev_err(&port_dev->dev, "Cannot enable. Maybe the USB cable is bad?\n");
+ dev_err_ratelimited(&port_dev->dev, "Cannot enable. Maybe the USB cable is bad?\n");
done:
if (status == 0) {
/* CarrolTouch 4500U */
{ USB_DEVICE(0x04e7, 0x0030), .driver_info = USB_QUIRK_RESET_RESUME },
+ /* Samsung T5 EVO Portable SSD */
+ { USB_DEVICE(0x04e8, 0x6200), .driver_info = USB_QUIRK_NO_LPM },
+
/* Samsung Android phone modem - ID conflict with SPH-I500 */
{ USB_DEVICE(0x04e8, 0x6601), .driver_info =
USB_QUIRK_CONFIG_INTF_STRINGS },
/* VLI disk */
{ USB_DEVICE(0x2109, 0x0711), .driver_info = USB_QUIRK_NO_LPM },
+ /* VIA Labs, Inc. USB2.0 Hub */
+ { USB_DEVICE(0x2109, 0x2817), .driver_info = USB_QUIRK_NO_LPM },
+
/* Raydium Touchscreen */
{ USB_DEVICE(0x2386, 0x3114), .driver_info = USB_QUIRK_NO_LPM },
if (dwc->enable_usb2_transceiver_delay) {
for (index = 0; index < dwc->num_usb2_ports; index++) {
- reg = dwc3_readl(dwc->regs, DWC3_GUSB2PHYCFG(index));
+ reg = dwc3_readl(dwc, DWC3_GUSB2PHYCFG(index));
reg |= DWC3_GUSB2PHYCFG_XCVRDLY;
- dwc3_writel(dwc->regs, DWC3_GUSB2PHYCFG(index), reg);
+ dwc3_writel(dwc, DWC3_GUSB2PHYCFG(index), reg);
}
}
}
ret = priv->drvdata->usb_init(priv);
if (ret)
- return ret;
+ goto err_rearm;
/* Init PHYs */
for (i = 0 ; i < PHY_COUNT ; ++i) {
ret = phy_init(priv->phys[i]);
if (ret)
- return ret;
+ goto err_rearm;
}
/* Set PHY Power */
for (i = 0 ; i < PHY_COUNT ; ++i) {
ret = phy_power_on(priv->phys[i]);
if (ret)
- return ret;
+ goto err_rearm;
}
if (priv->vbus && priv->otg_phy_mode == PHY_MODE_USB_HOST) {
ret = regulator_enable(priv->vbus);
if (ret)
- return ret;
+ goto err_rearm;
}
if (priv->drvdata->usb_post_init) {
ret = priv->drvdata->usb_post_init(priv);
if (ret)
- return ret;
+ goto err_rearm;
}
return 0;
+
+err_rearm:
+ reset_control_rearm(priv->reset);
+ return ret;
}
static const struct dev_pm_ops dwc3_meson_g12a_dev_pm_ops = {
}
}
+static bool dwc3_prepare_disconnect_gadget(struct dwc3 *dwc,
+ struct usb_gadget_driver **driver,
+ struct usb_gadget **gadget)
+{
+ if (!dwc->async_callbacks || !dwc->gadget_driver ||
+ !dwc->gadget_driver->disconnect)
+ return false;
+
+ *driver = dwc->gadget_driver;
+ *gadget = dwc->gadget;
+
+ return true;
+}
+
static void dwc3_disconnect_gadget(struct dwc3 *dwc)
{
- if (dwc->async_callbacks && dwc->gadget_driver->disconnect) {
+ struct usb_gadget_driver *driver;
+ struct usb_gadget *gadget;
+
+ if (dwc3_prepare_disconnect_gadget(dwc, &driver, &gadget)) {
spin_unlock(&dwc->lock);
- dwc->gadget_driver->disconnect(dwc->gadget);
+ driver->disconnect(gadget);
spin_lock(&dwc->lock);
}
}
+static void dwc3_disconnect_gadget_sleepable(struct dwc3 *dwc)
+{
+ struct usb_gadget_driver *driver;
+ struct usb_gadget *gadget;
+ unsigned long flags;
+
+ spin_lock_irqsave(&dwc->lock, flags);
+ if (!dwc3_prepare_disconnect_gadget(dwc, &driver, &gadget)) {
+ spin_unlock_irqrestore(&dwc->lock, flags);
+ return;
+ }
+
+ spin_unlock_irqrestore(&dwc->lock, flags);
+ driver->disconnect(gadget);
+}
+
static void dwc3_suspend_gadget(struct dwc3 *dwc)
{
if (dwc->async_callbacks && dwc->gadget_driver->suspend) {
int dwc3_gadget_suspend(struct dwc3 *dwc)
{
- unsigned long flags;
int ret;
ret = dwc3_gadget_soft_disconnect(dwc);
return -EAGAIN;
}
- spin_lock_irqsave(&dwc->lock, flags);
- if (dwc->gadget_driver)
- dwc3_disconnect_gadget(dwc);
- spin_unlock_irqrestore(&dwc->lock, flags);
+ dwc3_disconnect_gadget_sleepable(dwc);
return 0;
}
return 0;
fail:
- iso_sched_free(stream, sched);
- urb->hcpriv = NULL;
return status;
}
else
usb_hcd_unlink_urb_from_ep(fotg210_to_hcd(fotg210), urb);
done_not_linked:
+ if (status < 0) {
+ iso_sched_free(stream, urb->hcpriv);
+ urb->hcpriv = NULL;
+ }
spin_unlock_irqrestore(&fotg210->lock, flags);
done:
return status;
if (cdev->config)
config = cdev->config;
else
- config = list_first_entry(
+ config = list_first_entry_or_null(
&cdev->configs,
- struct usb_configuration, list);
+ struct usb_configuration,
+ list);
if (!config)
goto done;
static void ffs_release_dev(struct ffs_dev *ffs_dev);
static int ffs_ready(struct ffs_data *ffs);
static void ffs_closed(struct ffs_data *ffs);
+static void ffs_reset_work(struct work_struct *work);
/* Misc helper functions ****************************************************/
mutex_unlock(&epfile->dmabufs_mutex);
- __ffs_epfile_read_buffer_free(epfile);
ffs_data_closed(epfile->ffs);
return 0;
resv_dir = epfile->in ? DMA_RESV_USAGE_READ : DMA_RESV_USAGE_WRITE;
dma_resv_add_fence(dmabuf->resv, &fence->base, resv_dir);
+ dma_fence_put(&fence->base);
dma_resv_unlock(dmabuf->resv);
/* Now that the dma_fence is in place, queue the transfer. */
init_waitqueue_head(&ffs->ev.waitq);
init_waitqueue_head(&ffs->wait);
init_completion(&ffs->ep0req_completion);
+ INIT_WORK(&ffs->reset_work, ffs_reset_work);
/* XXX REVISIT need to update it in some places, or do we? */
ffs->ev.can_stall = 1;
sprintf(epfile->name, "ep%02x", ffs->eps_addrmap[i]);
else
sprintf(epfile->name, "ep%u", i);
+ epfile->in = (ffs->eps_addrmap[i] & USB_ENDPOINT_DIR_MASK) ? 1 : 0;
err = ffs_sb_create_file(ffs->sb, epfile->name,
epfile, &ffs_epfile_operations);
if (err) {
for (; count; --count, ++epfile) {
BUG_ON(mutex_is_locked(&epfile->mutex));
+ __ffs_epfile_read_buffer_free(epfile);
simple_remove_by_name(root, epfile->name, clear_one);
}
ret = usb_ep_enable(ep->ep);
if (!ret) {
epfile->ep = ep;
- epfile->in = usb_endpoint_dir_in(ep->ep->desc);
epfile->isoc = usb_endpoint_xfer_isoc(ep->ep->desc);
} else {
break;
if (ffs->state == FFS_DEACTIVATED) {
ffs->state = FFS_CLOSING;
spin_unlock_irqrestore(&ffs->eps_lock, flags);
- INIT_WORK(&ffs->reset_work, ffs_reset_work);
schedule_work(&ffs->reset_work);
return -ENODEV;
}
if (ffs->state == FFS_DEACTIVATED) {
ffs->state = FFS_CLOSING;
spin_unlock_irqrestore(&ffs->eps_lock, flags);
- INIT_WORK(&ffs->reset_work, ffs_reset_work);
schedule_work(&ffs->reset_work);
return;
}
ret = 0;
/* Change the printer status to show that it's on-line. */
dev->printer_status |= PRINTER_SELECTED;
+ kref_get(&dev->kref);
}
spin_unlock_irqrestore(&dev->lock, flags);
- kref_get(&dev->kref);
-
return ret;
}
static int rndis_query_response(struct rndis_params *params,
rndis_query_msg_type *buf)
{
+ u32 BufLength, BufOffset;
rndis_query_cmplt_type *resp;
rndis_resp_t *r;
if (!params->dev)
return -ENOTSUPP;
+ BufLength = le32_to_cpu(buf->InformationBufferLength);
+ BufOffset = le32_to_cpu(buf->InformationBufferOffset);
+ if ((BufLength > RNDIS_MAX_TOTAL_SIZE) ||
+ (BufOffset > RNDIS_MAX_TOTAL_SIZE) ||
+ (BufOffset + 8 >= RNDIS_MAX_TOTAL_SIZE))
+ return -EINVAL;
+
/*
* we need more memory:
* gen_ndis_query_resp expects enough space for
resp->RequestID = buf->RequestID; /* Still LE in msg buffer */
if (gen_ndis_query_resp(params, le32_to_cpu(buf->OID),
- le32_to_cpu(buf->InformationBufferOffset)
- + 8 + (u8 *)buf,
- le32_to_cpu(buf->InformationBufferLength),
- r)) {
+ BufOffset + 8 + (u8 *)buf,
+ BufLength, r)) {
/* OID not supported */
resp->Status = cpu_to_le32(RNDIS_STATUS_NOT_SUPPORTED);
resp->MessageLength = cpu_to_le32(sizeof *resp);
/* tmp points to a struct rndis_packet_msg_type */
__le32 *tmp = (void *)skb->data;
+ /* Need at least MessageType, MessageLength, DataOffset, DataLength */
+ if (skb->len < 16) {
+ dev_kfree_skb_any(skb);
+ return -EINVAL;
+ }
+
/* MessageType, MessageLength */
if (cpu_to_le32(RNDIS_MSG_PACKET)
!= get_unaligned(tmp++)) {
/**
* struct usb_udc - describes one usb device controller
* @driver: the gadget driver pointer. For use by the class code
- * @dev: the child device to the actual controller
* @gadget: the gadget. For use by the class code
+ * @gadget_release: the gadget's release routine
+ * @dev: the child device to the actual controller
* @list: for use by the udc class driver
* @vbus: for udcs who care about vbus status, this value is real vbus status;
* for udcs who do not care about vbus status, this value is always true
struct usb_udc {
struct usb_gadget_driver *driver;
struct usb_gadget *gadget;
+ void (*gadget_release)(struct device *dev);
struct device dev;
struct list_head list;
bool vbus;
dev_vdbg(dev, "%s\n", __func__);
}
+static void usb_gadget_release(struct device *dev)
+{
+ struct usb_gadget *gadget = dev_to_usb_gadget(dev);
+ struct usb_udc *udc = gadget->udc;
+ /* Cache the gadget's release routine to prevent UAF */
+ void (*release)(struct device *dev) = udc->gadget_release;
+
+ put_device(&udc->dev);
+ release(dev);
+}
+
/**
* usb_initialize_gadget - initialize a gadget and its embedded struct device
* @parent: the parent device to this udc. Usually the controller driver's
mutex_init(&udc->connect_lock);
udc->started = false;
+ /*
+ * Align decoupled lifecycles: take a UDC reference to ensure it
+ * remains allocated until the gadget is released, requiring an
+ * override of the gadget's release routine to drop it.
+ */
+ udc->gadget_release = gadget->dev.release;
+ gadget->dev.release = usb_gadget_release;
+ get_device(&udc->dev);
mutex_lock(&udc_lock);
list_add_tail(&udc->list, &udc_list);
mutex_lock(&udc_lock);
list_del(&udc->list);
mutex_unlock(&udc_lock);
+ /*
+ * Revert the override and drop the UDC reference to prevent
+ * leaking the UDC if the gadget was statically allocated.
+ */
+ gadget->dev.release = udc->gadget_release;
+ put_device(&udc->dev);
err_put_udc:
put_device(&udc->dev);
status = 1; /* and give it back immediately */
iso_sched_free(stream, sched);
sched = NULL;
+ urb->hcpriv = NULL;
}
}
urb->error_count = skip / period;
return status;
fail:
- iso_sched_free(stream, sched);
- urb->hcpriv = NULL;
return status;
}
usb_hcd_unlink_urb_from_ep(ehci_to_hcd(ehci), urb);
}
done_not_linked:
+ if (status < 0) {
+ iso_sched_free(stream, urb->hcpriv);
+ urb->hcpriv = NULL;
+ }
spin_unlock_irqrestore(&ehci->lock, flags);
done:
return status;
usb_hcd_unlink_urb_from_ep(ehci_to_hcd(ehci), urb);
}
done_not_linked:
+ if (status < 0) {
+ iso_sched_free(stream, urb->hcpriv);
+ urb->hcpriv = NULL;
+ }
spin_unlock_irqrestore(&ehci->lock, flags);
done:
return status;
remove_debug_file(sl811);
usb_remove_hcd(hcd);
+ device_wakeup_disable(hcd->self.controller);
/* some platforms may use IORESOURCE_IO */
res = platform_get_resource(dev, IORESOURCE_MEM, 1);
static void xhci_dbc_set_state(struct xhci_dbc *dbc, enum dbc_state new_state)
{
+ if (dbc->state == new_state)
+ return;
+
+ switch (new_state) {
+ case DS_ENABLED:
+ /*
+ * DbC pm usage is 1 here, both when moved from disconnect or
+ * configured states, or when setting initial DbC enable state.
+ * Just enable pending put
+ */
+ dev_dbg(dbc->dev, "DbC set pending_rpm_put = 1\n");
+ dbc->pending_rpm_put = 1;
+ break;
+ case DS_CONNECTED:
+ if (dbc->pending_rpm_put)
+ /* DbC pm usage still 1, just remove pending put */
+ dbc->pending_rpm_put = 0;
+ else
+ /* DbC pm usage was put to 0, call get */
+ pm_runtime_get(dbc->dev);
+ break;
+ default:
+ break;
+ }
+
dbc->state_timestamp = jiffies;
dbc->state = new_state;
}
WARN_ON(!dbc);
- pm_runtime_get_sync(dbc->dev); /* note this was self.controller */
+ pm_runtime_get(dbc->dev);
spin_lock_irqsave(&dbc->lock, flags);
ret = xhci_do_dbc_start(dbc);
static void xhci_dbc_stop(struct xhci_dbc *dbc)
{
unsigned long flags;
+ bool need_rpm_put = false;
WARN_ON(!dbc);
spin_lock_irqsave(&dbc->lock, flags);
writel(0, &dbc->regs->control);
+
+ if (dbc->state == DS_CONNECTED || dbc->state == DS_CONFIGURED ||
+ dbc->pending_rpm_put)
+ need_rpm_put = true;
+
+ dbc->pending_rpm_put = 0;
+
xhci_dbc_set_state(dbc, DS_DISABLED);
spin_unlock_irqrestore(&dbc->lock, flags);
xhci_dbc_mem_cleanup(dbc);
- pm_runtime_put(dbc->dev); /* note, was self.controller */
+ if (need_rpm_put)
+ pm_runtime_put(dbc->dev);
}
static void
dev_info(dbc->dev, "DbC connected\n");
} else if (!(ctrl & DBC_CTRL_DBC_ENABLE)) {
dev_err(dbc->dev, "unexpected DbC disable, xHC reset?\n");
+ } else if (dbc->pending_rpm_put &&
+ time_is_before_jiffies(dbc->state_timestamp +
+ msecs_to_jiffies(DBC_AUTOSUSPEND_DELAY))) {
+ dbc->pending_rpm_put = 0;
+ dev_dbg(dbc->dev, "DbC Enabled state for 15 seconds, allow rpm suspend\n");
+ pm_runtime_put(dbc->dev);
}
return EVT_DONE;
if (dbc->state >= ARRAY_SIZE(dbc_state_strings))
return sysfs_emit(buf, "unknown\n");
+ if (dbc->resume_required)
+ return sysfs_emit(buf, "suspended\n");
+
return sysfs_emit(buf, "%s\n", dbc_state_strings[dbc->state]);
}
dbc = xhci->dbc;
if (sysfs_streq(buf, "enable")) {
+ pm_runtime_get_sync(dbc->dev);
+
mutex_lock(&dbc->enable_mutex);
+ /*
+ * DbC may already be enabled here if xhci was suspended with
+ * dbc->resume_required set, and resumed by pm_runtime_get_sync()
+ * above. In this case we end up calling xhci_dbc_start() twice,
+ * second time returns an error but is harmless
+ */
xhci_dbc_start(dbc);
+
mutex_unlock(&dbc->enable_mutex);
+ pm_runtime_put(dbc->dev);
} else if (sysfs_streq(buf, "disable")) {
mutex_lock(&dbc->enable_mutex);
+
+ dbc->resume_required = 0;
xhci_dbc_stop(dbc);
+
mutex_unlock(&dbc->enable_mutex);
} else {
return -EINVAL;
#define DBC_POLL_INTERVAL_MAX 5000 /* milliseconds */
#define DBC_XFER_INACTIVITY_TIMEOUT 10 /* milliseconds */
#define DBC_ENUMERATION_TIMEOUT 2000 /* milliseconds */
+#define DBC_AUTOSUSPEND_DELAY 15000 /* milliseconds */
+
/*
* Private structure for DbC hardware state:
*/
unsigned long xfer_timestamp;
unsigned long state_timestamp;
unsigned resume_required:1;
+ unsigned pending_rpm_put:1;
struct dbc_ep eps[2];
const struct dbc_driver *driver;
if (sg_alloc_table_from_pages(sgt, pages, n_pages, 0, sz, GFP_KERNEL))
goto err;
+ kvfree(pages);
+
/*
* Save first segment dma address to sg dma_address field for the sideband
* client to have access to the IOVA of the ring.
struct xhci_virt_device *vdev;
struct xhci_command *command;
struct xhci_input_control_ctx *ctrl_ctx;
+ struct xhci_stream_info *stream_info[EP_CTX_PER_DEV];
unsigned int ep_index;
unsigned long flags;
u32 changed_ep_bitmask;
if (ret < 0)
return ret;
+ /*
+ * dma_free_coherent() called by xhci_free_stream_info() may sleep,
+ * so save stream_info pointers and clear references under lock,
+ * then free the memory outside lock.
+ */
spin_lock_irqsave(&xhci->lock, flags);
for (i = 0; i < num_eps; i++) {
ep_index = xhci_get_endpoint_index(&eps[i]->desc);
- xhci_free_stream_info(xhci, vdev->eps[ep_index].stream_info);
+ stream_info[i] = vdev->eps[ep_index].stream_info;
vdev->eps[ep_index].stream_info = NULL;
/* FIXME Unset maxPstreams in endpoint context and
* update deq ptr to point to normal string ring.
}
spin_unlock_irqrestore(&xhci->lock, flags);
+ for (i = 0; i < num_eps; i++)
+ xhci_free_stream_info(xhci, stream_info[i]);
+
return 0;
}
mutex_unlock(&dev->lock);
destruction:
mutex_unlock(&chaoskey_list_lock);
- usb_dbg(interface, "release success");
return rv;
}
/* structure to hold all of our device specific stuff */
struct usb_idmouse {
+ struct kref kref;
struct usb_device *udev; /* save off the usb device pointer */
struct usb_interface *interface; /* the interface for this device */
return 0;
}
-static inline void idmouse_delete(struct usb_idmouse *dev)
+static inline void idmouse_delete(struct kref *kref)
{
+ struct usb_idmouse *dev = container_of(kref, struct usb_idmouse, kref);
+
kfree(dev->bulk_in_buffer);
kfree(dev);
}
/* increment our usage count for the driver */
++dev->open;
+ kref_get(&dev->kref);
+
/* save our object in the file's private structure */
file->private_data = dev;
/* lock our device */
mutex_lock(&dev->lock);
-
--dev->open;
+ mutex_unlock(&dev->lock);
+
+ kref_put(&dev->kref, idmouse_delete);
- if (!dev->present) {
- /* the device was unplugged before the file was released */
- mutex_unlock(&dev->lock);
- idmouse_delete(dev);
- } else {
- mutex_unlock(&dev->lock);
- }
return 0;
}
if (dev == NULL)
return -ENOMEM;
+ kref_init(&dev->kref);
mutex_init(&dev->lock);
dev->udev = udev;
dev->interface = interface;
result = usb_find_bulk_in_endpoint(iface_desc, &endpoint);
if (result) {
dev_err(&interface->dev, "Unable to find bulk-in endpoint.\n");
- idmouse_delete(dev);
- return result;
+ goto err_put_kref;
}
dev->orig_bi_size = usb_endpoint_maxp(endpoint);
dev->bulk_in_endpointAddr = endpoint->bEndpointAddress;
dev->bulk_in_buffer = kmalloc(IMGSIZE + dev->bulk_in_size, GFP_KERNEL);
if (!dev->bulk_in_buffer) {
- idmouse_delete(dev);
- return -ENOMEM;
+ result = -ENOMEM;
+ goto err_put_kref;
}
/* allow device read, write and ioctl */
if (result) {
/* something prevented us from registering this device */
dev_err(&interface->dev, "Unable to allocate minor number.\n");
- idmouse_delete(dev);
- return result;
+ goto err_put_kref;
}
/* be noisy */
dev_info(&interface->dev,"%s now attached\n",DRIVER_DESC);
return 0;
+
+err_put_kref:
+ kref_put(&dev->kref, idmouse_delete);
+
+ return result;
}
static void idmouse_disconnect(struct usb_interface *interface)
/* prevent device read, write and ioctl */
dev->present = 0;
- /* if the device is opened, idmouse_release will clean this up */
- if (!dev->open) {
- mutex_unlock(&dev->lock);
- idmouse_delete(dev);
- } else {
- /* unlock */
- mutex_unlock(&dev->lock);
- }
+ mutex_unlock(&dev->lock);
+
+ kref_put(&dev->kref, idmouse_delete);
dev_info(&interface->dev, "disconnected\n");
}
/* Structure to hold all of our device specific stuff */
struct iowarrior {
+ struct kref kref;
struct mutex mutex; /* locks this structure */
struct usb_device *udev; /* save off the usb device pointer */
struct usb_interface *interface; /* the interface for this device */
/*
* iowarrior_delete
*/
-static inline void iowarrior_delete(struct iowarrior *dev)
+static inline void iowarrior_delete(struct kref *kref)
{
+ struct iowarrior *dev = container_of(kref, struct iowarrior, kref);
+
kfree(dev->int_in_buffer);
usb_free_urb(dev->int_in_urb);
kfree(dev->read_queue);
}
/* increment our usage count for the driver */
++dev->opened;
+
+ kref_get(&dev->kref);
+
/* save our object in the file's private structure */
file->private_data = dev;
retval = 0;
static int iowarrior_release(struct inode *inode, struct file *file)
{
struct iowarrior *dev;
- int retval = 0;
dev = file->private_data;
if (!dev)
/* lock our device */
mutex_lock(&dev->mutex);
+ dev->opened = 0; /* we're closing now */
- if (dev->opened <= 0) {
- retval = -ENODEV; /* close called more than once */
- mutex_unlock(&dev->mutex);
- } else {
- dev->opened = 0; /* we're closing now */
- retval = 0;
- if (dev->present) {
- /*
- The device is still connected so we only shutdown
- pending read-/write-ops.
- */
- usb_kill_urb(dev->int_in_urb);
- wake_up_interruptible(&dev->read_wait);
- wake_up_interruptible(&dev->write_wait);
- mutex_unlock(&dev->mutex);
- } else {
- /* The device was unplugged, cleanup resources */
- mutex_unlock(&dev->mutex);
- iowarrior_delete(dev);
- }
+ if (dev->present) {
+ usb_kill_urb(dev->int_in_urb);
+ wake_up_interruptible(&dev->read_wait);
+ wake_up_interruptible(&dev->write_wait);
}
- return retval;
+ mutex_unlock(&dev->mutex);
+
+ kref_put(&dev->kref, iowarrior_delete);
+
+ return 0;
}
static __poll_t iowarrior_poll(struct file *file, poll_table * wait)
if (!dev)
return retval;
+ kref_init(&dev->kref);
mutex_init(&dev->mutex);
atomic_set(&dev->intr_idx, 0);
return retval;
error:
- iowarrior_delete(dev);
+ kref_put(&dev->kref, iowarrior_delete);
+
return retval;
}
/* prevent device read, write and ioctl */
dev->present = 0;
+ /* write urbs are not stopped on close() so kill unconditionally */
+ usb_kill_anchored_urbs(&dev->submitted);
+
if (dev->opened) {
- /* There is a process that holds a filedescriptor to the device ,
- so we only shutdown read-/write-ops going on.
- Deleting the device is postponed until close() was called.
- */
usb_kill_urb(dev->int_in_urb);
- usb_kill_anchored_urbs(&dev->submitted);
wake_up_interruptible(&dev->read_wait);
wake_up_interruptible(&dev->write_wait);
- mutex_unlock(&dev->mutex);
- } else {
- /* no process is using the device, cleanup now */
- mutex_unlock(&dev->mutex);
- iowarrior_delete(dev);
}
+
+ mutex_unlock(&dev->mutex);
+
+ kref_put(&dev->kref, iowarrior_delete);
}
/* usb specific object needed to register this driver with the usb subsystem */
/* Structure to hold all of our device specific stuff */
struct ld_usb {
+ struct kref kref;
struct mutex mutex; /* locks this structure */
struct usb_interface *intf; /* save off the usb interface pointer */
unsigned long disconnected:1;
/*
* ld_usb_delete
*/
-static void ld_usb_delete(struct ld_usb *dev)
+static void ld_usb_delete(struct kref *kref)
{
+ struct ld_usb *dev = container_of(kref, struct ld_usb, kref);
+
/* free data structures */
usb_free_urb(dev->interrupt_in_urb);
usb_free_urb(dev->interrupt_out_urb);
goto unlock_exit;
}
+ kref_get(&dev->kref);
+
/* save device in the file's private structure */
file->private_data = dev;
mutex_lock(&dev->mutex);
- if (dev->open_count != 1) {
- retval = -ENODEV;
+ if (dev->disconnected)
goto unlock_exit;
- }
- if (dev->disconnected) {
- /* the device was unplugged before the file was released */
- mutex_unlock(&dev->mutex);
- /* unlock here as ld_usb_delete frees dev */
- ld_usb_delete(dev);
- goto exit;
- }
/* wait until write transfer is finished */
if (dev->interrupt_out_busy)
unlock_exit:
mutex_unlock(&dev->mutex);
-
+ kref_put(&dev->kref, ld_usb_delete);
exit:
return retval;
}
dev = kzalloc_obj(*dev);
if (!dev)
goto exit;
+
+ kref_init(&dev->kref);
mutex_init(&dev->mutex);
spin_lock_init(&dev->rbsl);
dev->intf = intf;
return retval;
error:
- ld_usb_delete(dev);
+ kref_put(&dev->kref, ld_usb_delete);
return retval;
}
mutex_lock(&dev->mutex);
- /* if the device is not opened, then we clean up right now */
- if (!dev->open_count) {
- mutex_unlock(&dev->mutex);
- ld_usb_delete(dev);
- } else {
- dev->disconnected = 1;
+ dev->disconnected = 1;
+
+ if (dev->open_count) {
/* wake up pollers */
wake_up_interruptible_all(&dev->read_wait);
wake_up_interruptible_all(&dev->write_wait);
- mutex_unlock(&dev->mutex);
}
+ mutex_unlock(&dev->mutex);
+
+ kref_put(&dev->kref, ld_usb_delete);
+
dev_info(&intf->dev, "LD USB Device #%d now disconnected\n",
(minor - USB_LD_MINOR_BASE));
}
/* Structure to hold all of our device specific stuff */
struct lego_usb_tower {
+ struct kref kref;
struct mutex lock; /* locks this structure */
struct usb_device *udev; /* save off the usb device pointer */
unsigned char minor; /* the starting minor number for this device */
/* local function prototypes */
static ssize_t tower_read(struct file *file, char __user *buffer, size_t count, loff_t *ppos);
static ssize_t tower_write(struct file *file, const char __user *buffer, size_t count, loff_t *ppos);
-static inline void tower_delete(struct lego_usb_tower *dev);
static int tower_open(struct inode *inode, struct file *file);
static int tower_release(struct inode *inode, struct file *file);
static __poll_t tower_poll(struct file *file, poll_table *wait);
/*
* tower_delete
*/
-static inline void tower_delete(struct lego_usb_tower *dev)
+static inline void tower_delete(struct kref *kref)
{
+ struct lego_usb_tower *dev = container_of(kref, struct lego_usb_tower, kref);
+
/* free data structures */
usb_free_urb(dev->interrupt_in_urb);
usb_free_urb(dev->interrupt_out_urb);
dev->open_count = 1;
+ kref_get(&dev->kref);
+
unlock_exit:
mutex_unlock(&dev->lock);
mutex_lock(&dev->lock);
- if (dev->disconnected) {
- /* the device was unplugged before the file was released */
-
- /* unlock here as tower_delete frees dev */
- mutex_unlock(&dev->lock);
- tower_delete(dev);
- goto exit;
- }
+ if (dev->disconnected)
+ goto out_unlock;
/* wait until write transfer is finished */
if (dev->interrupt_out_busy) {
dev->open_count = 0;
+out_unlock:
mutex_unlock(&dev->lock);
+ kref_put(&dev->kref, tower_delete);
exit:
return retval;
}
if (!dev)
goto exit;
+ kref_init(&dev->kref);
mutex_init(&dev->lock);
dev->udev = usb_get_dev(udev);
spin_lock_init(&dev->read_buffer_lock);
return retval;
error:
- tower_delete(dev);
+ kref_put(&dev->kref, tower_delete);
return retval;
}
mutex_lock(&dev->lock);
- /* if the device is not opened, then we clean up right now */
- if (!dev->open_count) {
- mutex_unlock(&dev->lock);
- tower_delete(dev);
- } else {
- dev->disconnected = 1;
+ dev->disconnected = 1;
+
+ if (dev->open_count) {
/* wake up pollers */
wake_up_interruptible_all(&dev->read_wait);
wake_up_interruptible_all(&dev->write_wait);
- mutex_unlock(&dev->lock);
}
+ mutex_unlock(&dev->lock);
+
+ kref_put(&dev->kref, tower_delete);
+
dev_info(&interface->dev, "LEGO USB Tower #%d now disconnected\n",
(minor - LEGO_USB_TOWER_MINOR_BASE));
}
if (ibuf_len < bpkt_len)
return -ENOSPC;
+ /* The device must not claim more payload than it actually sent. */
+ if (bpkt_len > act - sizeof(*bpkt))
+ return -EPROTO;
+
memcpy(ibuf, bpkt->data, bpkt_len);
return bpkt_len;
static void usbio_disconnect(struct usb_interface *intf)
{
struct usbio_device *usbio = usb_get_intfdata(intf);
- struct usbio_client *client;
+ struct usbio_client *client, *next;
/* Wakeup any clients waiting for a reply */
usbio->rxdat_len = 0;
usb_kill_urb(usbio->urb);
usb_free_urb(usbio->urb);
- list_for_each_entry_reverse(client, &usbio->cli_list, link) {
+ list_for_each_entry_safe_reverse(client, next, &usbio->cli_list, link) {
auxiliary_device_delete(&client->auxdev);
auxiliary_device_uninit(&client->auxdev);
}
* here. */
ret = get_1284_register(pp, 0, ®, GFP_KERNEL);
dev_dbg(&intf->dev, "reg: %7ph\n", priv->reg);
- if (ret < 0)
+ if (ret < 0) {
+ priv->pp = NULL;
+ parport_del_port(pp);
goto probe_abort;
+ }
ret = usb_find_last_int_in_endpoint(interface, &epd);
if (!ret) {
if (mtu3_prepare_transfer(mep)) {
ret = -EAGAIN;
+ usb_gadget_unmap_request(&mtu->g, req, mep->is_in);
goto error;
}
len &= ~3;
memcpy(oob_port->write_urb->transfer_buffer, buf, len);
oob_port->write_urb->transfer_buffer_length = len;
+
ret = usb_submit_urb(oob_port->write_urb, GFP_ATOMIC);
- if (ret == 0) {
- oob_priv->dp_write_urb_in_use = 1;
- count -= len;
- buf += len;
- }
+ if (ret)
+ break;
+
+ oob_priv->dp_write_urb_in_use = 1;
+ count -= len;
+ buf += len;
}
spin_unlock_irqrestore(&oob_priv->dp_port_lock, flags);
if (ret)
int len;
struct digi_port *priv = usb_get_serial_port_data(port);
unsigned char *data = port->write_urb->transfer_buffer;
+ unsigned long expire;
unsigned long flags;
dev_dbg(&port->dev, "digi_write_inb_command: TOP: port=%d, count=%d\n",
priv->dp_port_num, count);
if (timeout)
- timeout += jiffies;
- else
- timeout = ULONG_MAX;
+ expire = jiffies + timeout;
spin_lock_irqsave(&priv->dp_port_lock, flags);
while (count > 0 && ret == 0) {
- while (priv->dp_write_urb_in_use &&
- time_before(jiffies, timeout)) {
+ while (priv->dp_write_urb_in_use) {
+ if (timeout && time_after(jiffies, expire)) {
+ ret = -ETIMEDOUT;
+ break;
+ }
cond_wait_interruptible_timeout_irqrestore(
&priv->write_wait, DIGI_RETRY_TIMEOUT,
&priv->dp_port_lock, flags);
spin_lock_irqsave(&priv->dp_port_lock, flags);
}
+ if (ret)
+ break;
+
/* len must be a multiple of 4 and small enough to */
/* guarantee the write will send buffered data first, */
/* so commands are in order with data and not split */
unsigned char buf[32];
struct digi_port *priv = usb_get_serial_port_data(port);
struct ktermios not_termios;
+ int throttled;
/* be sure the device is started up */
if (digi_startup_device(port->serial) != 0)
not_termios.c_iflag = ~tty->termios.c_iflag;
digi_set_termios(tty, port, ¬_termios);
}
+
+ spin_lock_irq(&priv->dp_port_lock);
+ throttled = priv->dp_throttle_restart;
+ priv->dp_throttled = 0;
+ priv->dp_throttle_restart = 0;
+ spin_unlock_irq(&priv->dp_port_lock);
+
+ if (throttled) {
+ ret = usb_submit_urb(port->read_urb, GFP_KERNEL);
+ if (ret) {
+ dev_err(&port->dev, "failed to submit read urb: %d\n", ret);
+ return ret;
+ }
+ }
+
return 0;
}
if (count == room)
schedule_work(&priv->unthrottle_work);
- return count;
+ return 0;
}
static void keyspan_pda_write_bulk_callback(struct urb *urb)
{ USB_DEVICE(TELIT_VENDOR_ID, TELIT_PRODUCT_CC864_SINGLE) },
{ USB_DEVICE(TELIT_VENDOR_ID, TELIT_PRODUCT_DE910_DUAL) },
{ USB_DEVICE(TELIT_VENDOR_ID, TELIT_PRODUCT_UE910_V2) },
+ { USB_DEVICE_AND_INTERFACE_INFO(TELIT_VENDOR_ID, 0x0990, 0xff, 0xff, 0x30), /* Telit FE990D50 (RNDIS) */
+ .driver_info = NCTRL(6) },
+ { USB_DEVICE_AND_INTERFACE_INFO(TELIT_VENDOR_ID, 0x0990, 0xff, 0xff, 0x40) },
+ { USB_DEVICE_AND_INTERFACE_INFO(TELIT_VENDOR_ID, 0x0990, 0xff, 0xff, 0x60) },
+ { USB_DEVICE_AND_INTERFACE_INFO(TELIT_VENDOR_ID, 0x0991, 0xff, 0xff, 0x30), /* Telit FE990D50 (rmnet) */
+ .driver_info = NCTRL(5) },
+ { USB_DEVICE_AND_INTERFACE_INFO(TELIT_VENDOR_ID, 0x0991, 0xff, 0xff, 0x40) },
+ { USB_DEVICE_AND_INTERFACE_INFO(TELIT_VENDOR_ID, 0x0991, 0xff, 0xff, 0x60) },
+ { USB_DEVICE_AND_INTERFACE_INFO(TELIT_VENDOR_ID, 0x0992, 0xff, 0xff, 0x30), /* Telit FE990D50 (MBIM) */
+ .driver_info = NCTRL(6) },
+ { USB_DEVICE_AND_INTERFACE_INFO(TELIT_VENDOR_ID, 0x0992, 0xff, 0xff, 0x40) },
+ { USB_DEVICE_AND_INTERFACE_INFO(TELIT_VENDOR_ID, 0x0992, 0xff, 0xff, 0x60) },
+ { USB_DEVICE_AND_INTERFACE_INFO(TELIT_VENDOR_ID, 0x0993, 0xff, 0xff, 0x30), /* Telit FE990D50 (ECM) */
+ .driver_info = NCTRL(6) },
+ { USB_DEVICE_AND_INTERFACE_INFO(TELIT_VENDOR_ID, 0x0993, 0xff, 0xff, 0x40) },
+ { USB_DEVICE_AND_INTERFACE_INFO(TELIT_VENDOR_ID, 0x0993, 0xff, 0xff, 0x60) },
{ USB_DEVICE_INTERFACE_CLASS(TELIT_VENDOR_ID, 0x1031, 0xff), /* Telit LE910C1-EUX */
.driver_info = NCTRL(0) | RSVD(3) },
{ USB_DEVICE_INTERFACE_CLASS(TELIT_VENDOR_ID, 0x1033, 0xff), /* Telit LE910C1-EUX (ECM) */
/*US_DEBUG(usb_stor_show_command(us, srb)); */
scsi_set_resid(srb, 0);
- if (unlikely(!(info->SD_Status & SD_Ready) || (info->MS_Status & MS_Ready)))
+ if (unlikely(!(info->SD_Status & SD_Ready) &&
+ !(info->MS_Status & MS_Ready)))
result = ene_init(us);
if (result == USB_STOR_XFER_GOOD) {
result = USB_STOR_TRANSPORT_ERROR;
US_FL_INITIAL_READ10 | US_FL_WRITE_CACHE |
US_FL_NO_ATA_1X | US_FL_NO_REPORT_OPCODES |
US_FL_MAX_SECTORS_240 | US_FL_NO_REPORT_LUNS |
- US_FL_ALWAYS_SYNC);
+ US_FL_ALWAYS_SYNC | US_FL_NO_SAME);
p = quirks;
while (*p) {
if (anx7411_typec_check_connection(plat))
dev_err(dev, "check status\n");
- pm_runtime_enable(dev);
+ ret = devm_pm_runtime_enable(dev);
+ if (ret)
+ goto free_wq;
return 0;
return -EINVAL;
ret = port->ops->pd_set(port, pd);
+ put_device(&pd->dev);
if (ret)
return ret;
static void *typec_mux_match(const struct fwnode_handle *fwnode,
const char *id, void *data)
{
- struct typec_mux_dev **mux_devs = data;
struct device *dev;
- int i;
/*
* Device graph (OF graph) does not give any means to identify the
dev = class_find_device(&typec_mux_class, NULL, fwnode,
mux_fwnode_match);
- /* Skip duplicates */
- for (i = 0; i < TYPEC_MUX_MAX_DEVS; i++)
- if (to_typec_mux_dev(dev) == mux_devs[i]) {
- put_device(dev);
- return NULL;
- }
-
-
return dev ? to_typec_mux_dev(dev) : ERR_PTR(-EPROBE_DEFER);
}
return ERR_PTR(-ENOMEM);
count = fwnode_connection_find_matches(fwnode, "mode-switch",
- (void **)mux_devs,
- typec_mux_match,
+ NULL, typec_mux_match,
(void **)mux_devs,
ARRAY_SIZE(mux_devs));
if (count <= 0) {
CONN_STATUS_1_DP_HPD_LEVEL;
switch (state->mode) {
+ case TYPEC_DP_STATE_D:
+ cfg0 |= CONN_STATUS_0_USB_3_1_CONNECTED;
+ fallthrough;
case TYPEC_DP_STATE_C:
cfg1 |= CONN_STATUS_1_DP_SINK_REQUESTED |
CONN_STATUS_1_DP_PIN_ASSIGNMENT_C_D;
- fallthrough;
- case TYPEC_DP_STATE_D:
- cfg1 |= CONN_STATUS_0_USB_3_1_CONNECTED;
break;
default: /* MODE_E */
break;
return 0;
}
+static void rt1711h_unregister_tcpci_port(void *tcpci);
+
static int rt1711h_probe(struct i2c_client *client)
{
int ret;
if (IS_ERR_OR_NULL(chip->tcpci))
return PTR_ERR(chip->tcpci);
+ ret = devm_add_action_or_reset(chip->dev, rt1711h_unregister_tcpci_port, chip->tcpci);
+ if (ret)
+ return ret;
+
ret = devm_request_threaded_irq(chip->dev, client->irq, NULL,
rt1711h_irq,
IRQF_ONESHOT | IRQF_TRIGGER_LOW,
return 0;
}
-static void rt1711h_remove(struct i2c_client *client)
+static void rt1711h_unregister_tcpci_port(void *tcpci)
{
- struct rt1711h_chip *chip = i2c_get_clientdata(client);
-
- tcpci_unregister_port(chip->tcpci);
+ tcpci_unregister_port(tcpci);
}
static const struct rt1711h_chip_info rt1711h = {
.of_match_table = rt1711h_of_match,
},
.probe = rt1711h_probe,
- .remove = rt1711h_remove,
.id_table = rt1711h_id,
};
module_i2c_driver(rt1711h_i2c_driver);
return;
}
+ if (pmdata->svid_index < 0 || pmdata->svid_index >= pmdata->nsvids) {
+ tcpm_log(port, "Invalid SVID index %d", pmdata->svid_index);
+ return;
+ }
+
for (i = 1; i < cnt; i++) {
if (pmdata->altmodes >= ALTMODE_DISCOVERY_MAX) {
/* Already logged in svdm_consume_svids() */
if (svdm_version < 0)
return svdm_version;
- header = VDO(altmode->svid, vdo ? 2 : 1, svdm_version, CMD_ENTER_MODE);
+ header = VDO(altmode->svid, 1, svdm_version, CMD_ENTER_MODE);
header |= VDO_OPOS(altmode->mode);
return tcpm_queue_vdm_unlocked(port, header, vdo, vdo ? 1 : 0, TCPC_TX_SOP);
if (svdm_version < 0)
return svdm_version;
- header = VDO(altmode->svid, vdo ? 2 : 1, svdm_version, CMD_ENTER_MODE);
+ header = VDO(altmode->svid, 1, svdm_version, CMD_ENTER_MODE);
header |= VDO_OPOS(altmode->mode);
return tcpm_queue_vdm_unlocked(port, header, vdo, vdo ? 1 : 0, TCPC_TX_SOP_PRIME);
* that Multi-function is preferred.
*/
if (DP_CAP_CAPABILITY(cap) & DP_CAP_UFP_D) {
- dp->data.status |= DP_STATUS_CON_UFP_D;
+ dp->data.status |= DP_STATUS_CON_DFP_D;
if (DP_CAP_UFP_D_PIN_ASSIGN(cap) & BIT(DP_PIN_ASSIGN_D))
dp->data.status |= DP_STATUS_PREFER_MULTI_FUNC;
} else {
- dp->data.status |= DP_STATUS_CON_DFP_D;
+ dp->data.status |= DP_STATUS_CON_UFP_D;
if (DP_CAP_DFP_D_PIN_ASSIGN(cap) & BIT(DP_PIN_ASSIGN_D))
dp->data.status |= DP_STATUS_PREFER_MULTI_FUNC;
static int ucsi_displayport_configure(struct ucsi_dp *dp)
{
- u32 pins = DP_CONF_GET_PIN_ASSIGN(dp->data.conf);
u64 command;
if (!dp->override)
return 0;
- command = UCSI_CMD_SET_NEW_CAM(dp->con->num, 1, dp->offset, pins);
+ command = UCSI_CMD_SET_NEW_CAM(dp->con->num, 1, dp->offset, dp->data.conf);
return ucsi_send_command(dp->con->ucsi, command, NULL, 0);
}
}
}
+int ucsi_suspend(struct ucsi *ucsi)
+{
+ int i;
+
+ /*
+ * Cancel pending work so it cannot access the firmware after the ACPI
+ * EC is stopped for suspend; state is re-read on resume.
+ */
+ cancel_delayed_work_sync(&ucsi->work);
+
+ if (!ucsi->connector)
+ return 0;
+
+ for (i = 0; i < ucsi->cap.num_connectors; i++)
+ cancel_work_sync(&ucsi->connector[i].work);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(ucsi_suspend);
+
int ucsi_resume(struct ucsi *ucsi)
{
if (ucsi->connector)
void *msg_out, size_t msg_out_size);
void ucsi_altmode_update_active(struct ucsi_connector *con);
+int ucsi_suspend(struct ucsi *ucsi);
int ucsi_resume(struct ucsi *ucsi);
void ucsi_notify_common(struct ucsi *ucsi, u32 cci);
ucsi_acpi_notify);
}
+static int ucsi_acpi_suspend(struct device *dev)
+{
+ struct ucsi_acpi *ua = dev_get_drvdata(dev);
+
+ return ucsi_suspend(ua->ucsi);
+}
+
static int ucsi_acpi_resume(struct device *dev)
{
struct ucsi_acpi *ua = dev_get_drvdata(dev);
return ucsi_resume(ua->ucsi);
}
-static DEFINE_SIMPLE_DEV_PM_OPS(ucsi_acpi_pm_ops, NULL, ucsi_acpi_resume);
+static DEFINE_SIMPLE_DEV_PM_OPS(ucsi_acpi_pm_ops, ucsi_acpi_suspend,
+ ucsi_acpi_resume);
static const struct acpi_device_id ucsi_acpi_match[] = {
{ "PNP0CA0", 0 },
cancel_work_sync(&uc->work);
pm_runtime_disable(uc->dev);
ucsi_unregister(uc->ucsi);
- ucsi_destroy(uc->ucsi);
free_irq(uc->irq, uc);
+ ucsi_destroy(uc->ucsi);
}
static const struct of_device_id ucsi_ccg_of_match_table[] = {
struct auxiliary_device *bridge;
struct typec_mux *typec_mux;
+ struct typec_mux_state state;
+ struct typec_altmode dp_alt;
int idx;
enum gaokun_ucsi_ccx ccx;
static void gaokun_ucsi_handle_usb_mode(struct gaokun_ucsi_port *port)
{
struct gaokun_ucsi *uec = port->ucsi;
- struct typec_mux_state state = {};
- struct typec_altmode dp_alt = {};
int idx = port->idx, ret;
/*
* For every typec port on this platform, the only mode-switch is
* controlled by its qmp combo phy which consumes svid and mode only.
*/
- dp_alt.svid = port->svid;
- state.mode = port->mode;
- state.alt = &dp_alt;
+ port->dp_alt.svid = port->svid;
+ port->state.mode = port->mode;
+ port->state.alt = &port->dp_alt;
if (idx >= uec->num_ports) {
dev_warn(uec->dev, "altmode port out of range: %d\n", idx);
return;
}
- ret = typec_mux_set(port->typec_mux, &state);
+ ret = typec_mux_set(port->typec_mux, &port->state);
if (ret)
dev_err(uec->dev, "failed to set mux %d\n", ret);
struct vrequest {
struct usb_request req;
- struct vudc *udc;
struct list_head req_entry; /* Request queue */
};
static int vep_dequeue(struct usb_ep *_ep, struct usb_request *_req)
{
struct vep *ep;
- struct vrequest *req;
struct vudc *udc;
struct vrequest *lst;
unsigned long flags;
return ret;
ep = to_vep(_ep);
- req = to_vrequest(_req);
- udc = req->udc;
+ udc = ep_to_vudc(ep);
if (!udc->driver)
return -ESHUTDOWN;
unsigned int capacity;
capacity = virtqueue_get_vring_size(vb->reporting_vq);
- if (capacity < PAGE_REPORTING_CAPACITY) {
- err = -ENOSPC;
- goto out_unregister_oom;
- }
vb->pr_dev_info.order = PAGE_REPORTING_ORDER_UNSPECIFIED;
vb->pr_dev_info.order = 5;
#endif
+ vb->pr_dev_info.capacity = capacity;
err = page_reporting_register(&vb->pr_dev_info);
if (err)
goto out_unregister_oom;
xenbus_unmap_ring_vfree(info->dev, info->ring.sring);
}
+/*
+ * Send the error response for a request that did not reach the target core
+ * and return its tag. Free the tag before the response drops the v2p
+ * reference that keeps the session alive, and snapshot what the response
+ * needs since returning the tag can let the slot be reused.
+ */
+static void scsiback_resp_and_free(struct vscsibk_pend *pending_req,
+ int32_t result)
+{
+ struct vscsibk_info *info = pending_req->info;
+ struct v2p_entry *v2p = pending_req->v2p;
+ struct se_session *se_sess = v2p->tpg->tpg_nexus->tvn_se_sess;
+ u16 rqid = pending_req->rqid;
+
+ target_free_tag(se_sess, &pending_req->se_cmd);
+ scsiback_send_response(info, NULL, result, 0, rqid);
+ kref_put(&v2p->kref, scsiback_free_translation_entry);
+}
+
static void scsiback_device_action(struct vscsibk_pend *pending_req,
enum tcm_tmreq_table act, int tag)
{
return;
err:
- scsiback_do_resp_with_sense(NULL, err, 0, pending_req);
+ scsiback_resp_and_free(pending_req, err);
}
/*
case VSCSIIF_ACT_SCSI_CDB:
if (scsiback_gnttab_data_map(&ring_req, pending_req)) {
scsiback_fast_flush_area(pending_req);
- scsiback_do_resp_with_sense(NULL,
- DID_ERROR << 16, 0, pending_req);
- transport_generic_free_cmd(&pending_req->se_cmd, 0);
+ scsiback_resp_and_free(pending_req,
+ DID_ERROR << 16);
} else {
scsiback_cmd_exec(pending_req);
}
break;
default:
pr_err_ratelimited("invalid request\n");
- scsiback_do_resp_with_sense(NULL, DID_ERROR << 16, 0,
- pending_req);
- transport_generic_free_cmd(&pending_req->se_cmd, 0);
+ scsiback_resp_and_free(pending_req, DID_ERROR << 16);
break;
}
if (node) {
ASSERT(list_empty(&node->list));
ASSERT(list_empty(&node->lower));
- ASSERT(node->eb == NULL, "node->eb->start=%llu", node->eb->start);
+ ASSERT(node->eb == NULL, "node->eb->start=%llu level=%d owner=%llu",
+ node->eb ? node->eb->start : 0,
+ node->eb ? btrfs_header_level(node->eb) : 0,
+ node->eb ? btrfs_header_owner(node->eb) : 0);
cache->nr_nodes--;
btrfs_put_root(node->root);
kfree(node);
if (inode->flags & BTRFS_INODE_NODATACOW ||
inode->flags & BTRFS_INODE_NODATASUM)
return false;
+ if (btrfs_is_data_reloc_root(inode->root))
+ return false;
return true;
}
btrfs_set_header_flag(eb, BTRFS_HEADER_FLAG_WRITTEN);
percpu_counter_add_batch(&fs_info->dirty_metadata_bytes,
- -eb->len,
+ -(s64)eb->len,
fs_info->dirty_metadata_batch);
ret = true;
} else {
return;
buffer_tree_clear_mark(eb, PAGECACHE_TAG_DIRTY);
- percpu_counter_add_batch(&fs_info->dirty_metadata_bytes, -eb->len,
+ percpu_counter_add_batch(&fs_info->dirty_metadata_bytes, -(s64)eb->len,
fs_info->dirty_metadata_batch);
for (int i = 0; i < num_extent_folios(eb); i++) {
goto next;
}
- flags = em->flags;
/*
* In case we split the extent map, we want to preserve the
* EXTENT_FLAG_LOGGING flag on our extent map, but we don't want
* it on the new extent maps.
*/
- em->flags &= ~(EXTENT_FLAG_PINNED | EXTENT_FLAG_LOGGING);
+ flags = em->flags & ~EXTENT_FLAG_LOGGING;
+ em->flags &= ~EXTENT_FLAG_PINNED;
modified = !list_empty(&em->list);
/*
const unsigned int nblocks = orig_len >> fs_info->sectorsize_bits;
int ret = 0;
u32 bio_offset = 0;
+ bool using_commit_root = false;
if ((inode->flags & BTRFS_INODE_NODATASUM) ||
test_bit(BTRFS_FS_STATE_NO_DATA_CSUMS, &fs_info->fs_state))
* from across transactions.
*/
if (bbio->csum_search_commit_root) {
+ using_commit_root = true;
path->search_commit_root = true;
path->skip_locking = true;
down_read(&fs_info->commit_root_sem);
* assume this is the case.
*/
if (count == 0) {
+ /*
+ * If an extent is relocated in the current transaction
+ * then relocation writes a new csum without updating
+ * the extent map generation. Until the next commit, we
+ * will see a hole in that case, so we need to fallback
+ * to searching the transaction csum root.
+ *
+ * Note that a commit root lookup of a referenced extent can
+ * only miss, not return a stale csum. A freed extent's csum
+ * is deleted in the same transaction and its bytenr is not
+ * reusable until that transaction has committed and the
+ * extent is unpinned.
+ */
+ if (using_commit_root) {
+ up_read(&fs_info->commit_root_sem);
+ using_commit_root = false;
+ path->search_commit_root = false;
+ path->skip_locking = false;
+ btrfs_release_path(path);
+ continue;
+ }
+
memset(csum_dst, 0, csum_size);
count = 1;
bio_offset += count * sectorsize;
}
- if (bbio->csum_search_commit_root)
+ if (using_commit_root)
up_read(&fs_info->commit_root_sem);
return ret;
}
u32 crc = ~(u32)0;
unsigned offset = 0;
+ if (index >= io_ctl->num_pages)
+ return -EIO;
+
if (index == 0)
offset = sizeof(u32) * io_ctl->num_pages;
stat->result_mask |= STATX_SUBVOL;
spin_lock(&BTRFS_I(inode)->lock);
- delalloc_bytes = BTRFS_I(inode)->new_delalloc_bytes;
+ delalloc_bytes = S_ISREG(inode->i_mode) ?
+ BTRFS_I(inode)->new_delalloc_bytes : 0;
inode_bytes = inode_get_bytes(inode);
spin_unlock(&BTRFS_I(inode)->lock);
stat->blocks = (ALIGN(inode_bytes, blocksize) +
int ret;
const char *comp = NULL;
u32 inode_flags;
+ bool prop_set = false;
if (btrfs_root_readonly(root))
return -EROFS;
if (comp) {
ret = btrfs_set_prop(trans, inode, "btrfs.compression",
comp, strlen(comp), 0);
- if (unlikely(ret)) {
- btrfs_abort_transaction(trans, ret);
+ if (ret)
goto out_end_trans;
- }
+ prop_set = true;
} else {
ret = btrfs_set_prop(trans, inode, "btrfs.compression", NULL, 0, 0);
- if (unlikely(ret && ret != -ENODATA)) {
- btrfs_abort_transaction(trans, ret);
+ prop_set = (ret == 0);
+ /* If ret == -ENODATA ignore and proceed to update inode item. */
+ if (ret && ret != -ENODATA)
goto out_end_trans;
- }
}
update_flags:
inode_inc_iversion(&inode->vfs_inode);
inode_set_ctime_current(&inode->vfs_inode);
ret = btrfs_update_inode(trans, inode);
+ /*
+ * If we set a property or deleted one, we must abort if we fail to
+ * update the inode, to avoid persisting an inconsistent state.
+ */
+ if (unlikely(ret && prop_set))
+ btrfs_abort_transaction(trans, ret);
out_end_trans:
btrfs_end_transaction(trans);
ret = -ENOENT;
goto out;
}
+ ret = 0;
}
out:
size_t max_segment_len = workspace_buf_length(fs_info);
int ret;
- if (unlikely(srclen < LZO_LEN || srclen > max_segment_len + LZO_LEN * 2))
+ if (unlikely(srclen < LZO_LEN || srclen > max_segment_len + LZO_LEN * 2)) {
+ btrfs_err(fs_info, "invalid lzo header length, has %zu expect (%u, %zu)",
+ srclen, LZO_LEN, max_segment_len + LZO_LEN * 2);
return -EUCLEAN;
+ }
in_len = get_unaligned_le32(data_in);
- if (unlikely(in_len != srclen))
+ if (unlikely(in_len != srclen)) {
+ btrfs_err(fs_info, "invalid lzo header length, has %zu expect %zu",
+ in_len, srclen);
return -EUCLEAN;
+ }
data_in += LZO_LEN;
in_len = get_unaligned_le32(data_in);
- if (unlikely(in_len != srclen - LZO_LEN * 2))
+ if (unlikely(in_len != srclen - LZO_LEN * 2)) {
+ btrfs_err(fs_info, "invalid lzo segment length, has %zu expect %zu",
+ in_len, srclen - LZO_LEN * 2);
return -EUCLEAN;
+ }
data_in += LZO_LEN;
out_len = sectorsize;
nr = btrfs_header_nritems(l);
btrfs_info(fs_info,
- "leaf %llu gen %llu total ptrs %d free space %d owner %llu",
+ "leaf %llu gen %llu total ptrs %d free space %d owner %lld",
btrfs_header_bytenr(l), btrfs_header_generation(l), nr,
- btrfs_leaf_free_space(l), btrfs_header_owner(l));
+ btrfs_leaf_free_space(l), (s64)btrfs_header_owner(l));
print_eb_refs_lock(l);
for (i = 0 ; i < nr ; i++) {
char key_buf[KEY_TYPE_BUF_SIZE];
return;
}
btrfs_info(fs_info,
- "node %llu level %d gen %llu total ptrs %d free spc %u owner %llu",
+ "node %llu level %d gen %llu total ptrs %d free spc %u owner %lld",
btrfs_header_bytenr(c), level, btrfs_header_generation(c),
nr, (u32)BTRFS_NODEPTRS_PER_BLOCK(fs_info) - nr,
- btrfs_header_owner(c));
+ (s64)btrfs_header_owner(c));
print_eb_refs_lock(c);
for (i = 0; i < nr; i++) {
btrfs_node_key_to_cpu(c, &key, i);
return ret;
}
+ ret = handler->validate(inode, value, value_len);
+ if (ret)
+ return ret;
ret = btrfs_setxattr(trans, &inode->vfs_inode, handler->xattr_name, value,
value_len, flags);
if (ret)
return ret;
ret = handler->apply(inode, value, value_len);
- if (ret) {
- btrfs_setxattr(trans, &inode->vfs_inode, handler->xattr_name, NULL,
- 0, flags);
+ /* We validated before, so it should not fail here. */
+ ASSERT(ret == 0);
+ if (unlikely(ret)) {
+ int ret2;
+
+ /* Try to delete xattr, if not possible abort transaction. */
+ ret2 = btrfs_setxattr(trans, &inode->vfs_inode, handler->xattr_name,
+ NULL, 0, flags);
+ if (unlikely(ret2))
+ btrfs_abort_transaction(trans, ret2);
return ret;
}
btrfs_err(fs_info,
"Duplicate root found for start=%llu while inserting into relocation tree",
node->bytenr);
+ kfree(node);
return -EEXIST;
}
ret = btrfs_insert_root(trans, fs_info->tree_root,
&root_key, root_item);
- if (ret)
- goto abort;
+ if (unlikely(ret)) {
+ btrfs_abort_transaction(trans, ret);
+ return ERR_PTR(ret);
+ }
reloc_root = btrfs_read_tree_root(fs_info->tree_root, &root_key);
if (IS_ERR(reloc_root)) {
- ret = PTR_ERR(reloc_root);
- goto abort;
+ btrfs_abort_transaction(trans, PTR_ERR(reloc_root));
+ return ERR_CAST(reloc_root);
}
set_bit(BTRFS_ROOT_SHAREABLE, &reloc_root->state);
btrfs_set_root_last_trans(reloc_root, trans->transid);
return reloc_root;
-
-abort:
- btrfs_abort_transaction(trans, ret);
- return ERR_PTR(ret);
}
/*
leaf = path->nodes[0];
fi = btrfs_item_ptr(leaf, path->slots[0],
struct btrfs_file_extent_item);
+ if (unlikely(btrfs_file_extent_type(leaf, fi) == BTRFS_FILE_EXTENT_INLINE)) {
+ btrfs_print_leaf(leaf);
+ btrfs_err(fs_info,
+ "unexpected inline file extent item for data reloc inode %llu key offset %llu",
+ btrfs_ino(BTRFS_I(reloc_inode)), bytenr);
+ return -EUCLEAN;
+ }
/*
* The cluster-boundary key searched above is always written by
return 0;
}
+static void clear_reloc_root(struct btrfs_root *root)
+{
+ root->reloc_root = NULL;
+ /*
+ * Need barrier to ensure clear_bit() only happens after
+ * root->reloc_root = NULL. Pairs with have_reloc_root().
+ */
+ smp_wmb();
+ clear_bit(BTRFS_ROOT_DEAD_RELOC_TREE, &root->state);
+}
+
static int clean_dirty_subvols(struct reloc_control *rc)
{
struct btrfs_root *root;
struct btrfs_root *reloc_root = root->reloc_root;
list_del_init(&root->reloc_dirty_list);
- root->reloc_root = NULL;
- /*
- * Need barrier to ensure clear_bit() only happens after
- * root->reloc_root = NULL. Pairs with have_reloc_root.
- */
- smp_wmb();
- clear_bit(BTRFS_ROOT_DEAD_RELOC_TREE, &root->state);
+ clear_reloc_root(root);
if (reloc_root) {
/*
* btrfs_drop_snapshot drops our ref we hold for
* corruption, e.g. bad reloc tree key offset.
*/
ret = -EINVAL;
+ btrfs_put_root(root);
goto out;
}
ret = merge_reloc_root(rc, root);
- btrfs_put_root(root);
if (ret) {
- if (list_empty(&reloc_root->root_list))
+ /*
+ * Clear the reloc root since below we will call
+ * free_reloc_roots(), otherwise we leave
+ * root->reloc_root pointing to a freed reloc
+ * root and trigger a use-after-free during
+ * unmount or elsewhere.
+ */
+ clear_reloc_root(root);
+ btrfs_put_root(root);
+ /*
+ * We are adding the reloc_root to the local
+ * reloc_roots list, so we add a ref for this
+ * list which will be dropped below by the call
+ * to free_reloc_roots().
+ */
+ if (list_empty(&reloc_root->root_list)) {
list_add_tail(&reloc_root->root_list,
&reloc_roots);
+ btrfs_grab_root(reloc_root);
+ }
+ /* Now drop the ref for root->reloc_root. */
+ btrfs_put_root(reloc_root);
goto out;
}
+ btrfs_put_root(root);
} else {
if (!IS_ERR(root)) {
if (root->reloc_root == reloc_root) {
- root->reloc_root = NULL;
+ clear_reloc_root(root);
+ /* Drop the ref for root->reloc_root. */
btrfs_put_root(reloc_root);
}
- clear_bit(BTRFS_ROOT_DEAD_RELOC_TREE,
- &root->state);
btrfs_put_root(root);
}
folio_mark_dirty(folio);
}
+static void folio_clear_tags(struct folio *folio)
+{
+ struct address_space *mapping = folio_mapping(folio);
+ XA_STATE(xas, &mapping->i_pages, folio->index);
+ unsigned long flags;
+
+ ASSERT(folio_test_locked(folio));
+ ASSERT(mapping);
+ ASSERT(mapping_use_writeback_tags(mapping));
+
+ xas_lock_irqsave(&xas, flags);
+ xas_load(&xas);
+ xas_clear_mark(&xas, PAGECACHE_TAG_DIRTY);
+ xas_clear_mark(&xas, PAGECACHE_TAG_TOWRITE);
+ xas_unlock_irqrestore(&xas, flags);
+}
+
/*
* Extra clear_and_test function for subpage dirty bitmap.
*
unsigned int start_bit = subpage_calc_start_bit(fs_info, folio,
writeback, start, len);
unsigned long flags;
- bool keep_write;
spin_lock_irqsave(&bfs->lock, flags);
bitmap_set(bfs->bitmaps, start_bit, len >> fs_info->sectorsize_bits);
* folio. Doing so can cause WB_SYNC_ALL writepages() to overlook it,
* assume writeback is complete, and exit too early — violating sync
* ordering guarantees.
+ *
+ * Instead we manually clear the DIRTY and TOWRITE tags after the folio
+ * is no longer dirty.
*/
- keep_write = folio_test_dirty(folio);
if (!folio_test_writeback(folio))
- __folio_start_writeback(folio, keep_write);
+ __folio_start_writeback(folio, true);
+ if (!folio_test_dirty(folio))
+ folio_clear_tags(folio);
spin_unlock_irqrestore(&bfs->lock, flags);
}
iref = (struct btrfs_inode_ref *)ptr;
namelen = btrfs_inode_ref_name_len(leaf, iref);
+ if (unlikely(namelen == 0 || namelen > BTRFS_NAME_LEN)) {
+ inode_ref_err(leaf, slot,
+ "invalid inode ref name length, has %u expect [1, %u]",
+ namelen, BTRFS_NAME_LEN);
+ return -EUCLEAN;
+ }
if (unlikely(ptr + sizeof(*iref) + namelen > end)) {
inode_ref_err(leaf, slot,
"inode ref overflow, ptr %lu end %lu namelen %u",
static struct vfsmount *erofs_ishare_mnt;
-static inline bool erofs_is_ishare_inode(struct inode *inode)
-{
- /* assumed FS_ONDEMAND is excluded with FS_PAGE_CACHE_SHARE feature */
- return inode->i_sb->s_type == &erofs_anon_fs_type;
-}
-
static int erofs_ishare_iget5_eq(struct inode *inode, void *data)
{
struct erofs_inode_fingerprint *fp1 = &EROFS_I(inode)->fingerprint;
struct inode *realinode;
*need_iput = false;
- if (!erofs_is_ishare_inode(inode))
+ if (inode->i_sb != erofs_ishare_mnt->mnt_sb)
return inode;
vi_share = EROFS_I(inode);
.get_parent = erofs_get_parent,
};
-static void erofs_set_sysfs_name(struct super_block *sb)
-{
- struct erofs_sb_info *sbi = EROFS_SB(sb);
-
- if (erofs_is_fileio_mode(sbi))
- super_set_sysfs_name_generic(sb, "%s",
- bdi_dev_name(sb->s_bdi));
- else
- super_set_sysfs_name_id(sb);
-}
-
static int erofs_fc_fill_super(struct super_block *sb, struct fs_context *fc)
{
struct inode *inode;
err = super_setup_bdi(sb);
if (err)
return err;
+
+ snprintf(sb->s_id, sizeof(sb->s_id),
+ "%u:%u", MAJOR(sb->s_dev), MINOR(sb->s_dev));
} else {
if (!sb_set_blocksize(sb, PAGE_SIZE)) {
errorfc(fc, "failed to set initial blksize");
return -EINVAL;
}
-
sbi->dif0.dax_dev = fs_dax_get_by_bdev(sb->s_bdev,
&sbi->dif0.dax_part_off, NULL, NULL);
}
if (err)
return err;
- erofs_set_sysfs_name(sb);
+ super_set_sysfs_name_id(sb);
err = erofs_register_sysfs(sb);
if (err)
return err;
",user_xattr" : ",nouser_xattr");
if (IS_ENABLED(CONFIG_EROFS_FS_POSIX_ACL))
seq_puts(seq, test_opt(opt, POSIX_ACL) ? ",acl" : ",noacl");
- if (IS_ENABLED(CONFIG_EROFS_FS_ZIP))
+ if (IS_ENABLED(CONFIG_EROFS_FS_ZIP) && sbi->available_compr_algs)
seq_printf(seq, ",cache_strategy=%s",
erofs_param_cache_strategy[opt->cache_strategy].name);
if (test_opt(opt, DAX_ALWAYS))
map->m_algorithmformat, EROFS_I(inode)->nid);
return -EFSCORRUPTED;
}
- if (EROFS_MAP_FULL(map->m_flags) && map->m_llen < map->m_plen) {
+ if (EROFS_MAP_FULL(map->m_flags) && map->m_llen < map->m_plen &&
+ map->m_la + map->m_llen < inode->i_size) {
erofs_err(inode->i_sb, "too much compressed data @ la %llu of nid %llu",
map->m_la, EROFS_I(inode)->nid);
return -EFSCORRUPTED;
return s;
}
-static inline sector_t fat_clus_to_blknr(struct msdos_sb_info *sbi, int clus)
+static inline sector_t fat_clus_to_blknr(const struct msdos_sb_info *sbi, int clus)
{
return ((sector_t)clus - FAT_START_ENT) * sbi->sec_per_clus
+ sbi->data_start;
}
-static inline void fat_get_blknr_offset(struct msdos_sb_info *sbi,
+static inline void fat_get_blknr_offset(const struct msdos_sb_info *sbi,
loff_t i_pos, sector_t *blknr, int *offset)
{
*blknr = i_pos >> sbi->dir_per_block_bits;
static void fat_clus_to_blknr_test(struct kunit *test)
{
- struct msdos_sb_info sbi = {
+ static const struct msdos_sb_info sbi = {
.sec_per_clus = 4,
.data_start = 100,
};
static void fat_get_blknr_offset_test(struct kunit *test)
{
- struct msdos_sb_info sbi = {
+ static const struct msdos_sb_info sbi = {
.dir_per_block = 16,
.dir_per_block_bits = 4,
};
mask |= MAY_READ;
if ((openflags & O_ACCMODE) != O_RDONLY)
mask |= MAY_WRITE;
+ if (openflags & O_TRUNC)
+ mask |= MAY_WRITE;
}
return mask;
}
/*
- * Record the page as unstable (an extra writeback period) and mark its
- * inode as dirty.
+ * Record the request's range as unstable (an extra writeback period) and
+ * mark its inode as dirty.
*/
-static inline void nfs_folio_mark_unstable(struct folio *folio,
+static inline void nfs_folio_mark_unstable(struct nfs_page *req,
struct nfs_commit_info *cinfo)
{
+ struct folio *folio = nfs_page_to_folio(req);
+
if (folio && !cinfo->dreq) {
struct inode *inode = folio->mapping->host;
- long nr = folio_nr_pages(folio);
+ long nr = DIV_ROUND_UP(req->wb_bytes, PAGE_SIZE);
- /* This page is really still in write-back - just that the
+ /* This range is really still in write-back - just that the
* writeback is happening on the server now.
*/
node_stat_mod_folio(folio, NR_WRITEBACK, nr);
nfs_request_add_commit_list_locked(req, list, cinfo);
mutex_unlock(&NFS_I(cinfo->inode)->commit_mutex);
- nfs_folio_mark_unstable(nfs_page_to_folio(req), cinfo);
+ nfs_folio_mark_unstable(req, cinfo);
return;
out_resched:
mutex_unlock(&NFS_I(cinfo->inode)->commit_mutex);
mutex_lock(&NFS_I(cinfo->inode)->commit_mutex);
nfs_request_add_commit_list_locked(req, &cinfo->mds->list, cinfo);
mutex_unlock(&NFS_I(cinfo->inode)->commit_mutex);
- nfs_folio_mark_unstable(nfs_page_to_folio(req), cinfo);
+ nfs_folio_mark_unstable(req, cinfo);
}
EXPORT_SYMBOL_GPL(nfs_request_add_commit_list);
nfs_request_add_commit_list(req, cinfo);
}
-static void nfs_folio_clear_commit(struct folio *folio)
+static void nfs_folio_clear_commit(struct nfs_page *req)
{
+ struct folio *folio = nfs_page_to_folio(req);
+
if (folio) {
- long nr = folio_nr_pages(folio);
+ long nr = DIV_ROUND_UP(req->wb_bytes, PAGE_SIZE);
node_stat_mod_folio(folio, NR_WRITEBACK, -nr);
bdi_wb_stat_mod(folio->mapping->host, WB_WRITEBACK, -nr);
nfs_request_remove_commit_list(req, cinfo);
}
mutex_unlock(&NFS_I(inode)->commit_mutex);
- nfs_folio_clear_commit(nfs_page_to_folio(req));
+ nfs_folio_clear_commit(req);
}
}
req = nfs_list_entry(page_list->next);
nfs_list_remove_request(req);
nfs_mark_request_commit(req, lseg, cinfo, ds_commit_idx);
- nfs_folio_clear_commit(nfs_page_to_folio(req));
+ nfs_folio_clear_commit(req);
nfs_unlock_and_release_request(req);
}
}
req = nfs_list_entry(data->pages.next);
nfs_list_remove_request(req);
folio = nfs_page_to_folio(req);
- nfs_folio_clear_commit(folio);
+ nfs_folio_clear_commit(req);
dprintk("NFS: commit (%s/%llu %d@%lld)",
nfs_req_openctx(req)->dentry->d_sb->s_id,
if (error)
return error;
- nfsd4_cancel_copy_by_sb(net, path.dentry->d_sb);
error = nlmsvc_unlock_all_by_sb(path.dentry->d_sb);
mutex_lock(&nfsd_mutex);
- if (nn->nfsd_serv)
+ if (nn->nfsd_serv) {
+ nfsd4_cancel_copy_by_sb(net, path.dentry->d_sb);
nfsd4_revoke_states(nn, path.dentry->d_sb);
- else
+ } else {
error = -EINVAL;
+ }
mutex_unlock(&nfsd_mutex);
path_put(&path);
.wbc = wbc,
.ops = &ntfs_writeback_ops,
};
+ bool need_iput = false;
+ int ret;
if (NVolShutdown(ni->vol))
return -EIO;
return -EOPNOTSUPP;
}
- return iomap_writepages(&wpc);
+ /*
+ * Prevent eviction in writeback to avoid deadlock in
+ * ntfs_drop_big_inode().
+ */
+ if ((ni->type == AT_DATA || ni->type == AT_INDEX_ALLOCATION) &&
+ igrab(inode))
+ need_iput = true;
+
+ ret = iomap_writepages(&wpc);
+
+ if (need_iput)
+ iput(inode);
+
+ return ret;
}
static int ntfs_swap_activate(struct swap_info_struct *sis,
err = -EIO;
goto err_out;
}
- WARN_ON(!ctx->attr->non_resident);
+ if (unlikely(!ctx->attr->non_resident)) {
+ err = -EIO;
+ goto err_out;
+ }
}
a = ctx->attr;
/*
ret = ntfs_attr_map_whole_runlist(ni);
if (ret) {
up_write(&ni->runlist.lock);
+ kfree(hole_rl);
return ret;
}
s64 old_data_size;
s64 vcn_start, vcn_end, vcn_uninit, vcn, try_alloc_cnt;
s64 lcn, alloc_cnt;
+ s64 rl_lcn, rl_length, rl_vcn;
int err = 0;
struct runlist_element *rl;
bool balloc;
while (vcn < vcn_uninit) {
down_read(&ni->runlist.lock);
rl = ntfs_attr_find_vcn_nolock(ni, vcn, NULL);
- up_read(&ni->runlist.lock);
if (IS_ERR(rl)) {
+ up_read(&ni->runlist.lock);
err = PTR_ERR(rl);
goto out;
}
+ rl_lcn = rl->lcn;
+ rl_length = rl->length;
+ rl_vcn = rl->vcn;
+ up_read(&ni->runlist.lock);
- if (rl->lcn > 0) {
- vcn += rl->length - (vcn - rl->vcn);
- } else if (rl->lcn == LCN_DELALLOC || rl->lcn == LCN_HOLE) {
- try_alloc_cnt = min(rl->length - (vcn - rl->vcn),
+ if (rl_lcn > 0) {
+ vcn += rl_length - (vcn - rl_vcn);
+ } else if (rl_lcn == LCN_DELALLOC || rl_lcn == LCN_HOLE) {
+ try_alloc_cnt = min(rl_length - (vcn - rl_vcn),
vcn_uninit - vcn);
- if (rl->lcn == LCN_DELALLOC) {
+ if (rl_lcn == LCN_DELALLOC) {
vcn += try_alloc_cnt;
continue;
}
if (err)
goto out;
- err = ntfs_dio_zero_range(VFS_I(ni),
- lcn << vol->cluster_size_bits,
- alloc_cnt << vol->cluster_size_bits);
- if (err > 0)
- goto out;
+ if (balloc) {
+ err = ntfs_dio_zero_range(VFS_I(ni),
+ lcn << vol->cluster_size_bits,
+ alloc_cnt <<
+ vol->cluster_size_bits);
+ if (err > 0)
+ goto out;
+ }
if (signal_pending(current))
goto out;
struct ntfs_inode *attr_ni;
int err;
+ /*
+ * generic_shutdown_super() clears SB_ACTIVE before evicting cached
+ * inodes. Do not look up the attribute-list inode after SB_ACTIVE has
+ * been cleared; it may already be I_FREEING, and waiting on it can
+ * self-deadlock.
+ */
+ if (!(VFS_I(base_ni)->i_sb->s_flags & SB_ACTIVE))
+ return -EIO;
+
attr_vi = ntfs_attr_iget(VFS_I(base_ni), AT_ATTRIBUTE_LIST, AT_UNNAMED, 0);
if (IS_ERR(attr_vi)) {
err = PTR_ERR(attr_vi);
__le16 I30[5] = { cpu_to_le16('$'), cpu_to_le16('I'),
cpu_to_le16('3'), cpu_to_le16('0'), 0 };
+static inline u64 ntfs_check_mref(u64 mref)
+{
+ if (IS_ERR_MREF(mref))
+ return ERR_MREF(-EIO);
+ return mref;
+}
+
/*
* ntfs_lookup_inode_by_name - find an inode in a directory given its name
* @dir_ni: ntfs inode of the directory in which to search for the name
mref = le64_to_cpu(ie->data.dir.indexed_file);
ntfs_attr_put_search_ctx(ctx);
unmap_mft_record(dir_ni);
- return mref;
+ return ntfs_check_mref(mref);
}
/*
* For a case insensitive mount, we also perform a case
if (name) {
ntfs_attr_put_search_ctx(ctx);
unmap_mft_record(dir_ni);
- return name->mref;
+ return ntfs_check_mref(name->mref);
}
ntfs_debug("Entry not found.");
err = -ENOENT;
mref = le64_to_cpu(ie->data.dir.indexed_file);
kfree(kaddr);
iput(ia_vi);
- return mref;
+ return ntfs_check_mref(mref);
}
/*
* For a case insensitive mount, we also perform a case
if (name) {
kfree(kaddr);
iput(ia_vi);
- return name->mref;
+ return ntfs_check_mref(name->mref);
}
ntfs_debug("Entry not found.");
err = -ENOENT;
ret = ntfs_inode_attr_pwrite(VFS_I(icx->ia_ni),
ntfs_ib_vcn_to_pos(icx, vcn), icx->block_size,
(u8 *)ib, icx->sync_write);
+
+ /* Perform data restoration before returning */
+ post_write_mst_fixup((struct ntfs_record *)ib);
+
if (ret != icx->block_size) {
ntfs_debug("Failed to write index block %lld, inode %llu",
vcn, (unsigned long long)icx->idx_ni->mft_no);
icx->ib = NULL;
icx->ib_dirty = false;
} else {
- post_write_mst_fixup((struct ntfs_record *)icx->ib);
icx->sync_write = false;
}
!S_ISFIFO(vi->i_mode) && !S_ISSOCK(vi->i_mode) && !S_ISLNK(vi->i_mode))
vi->i_flags |= S_IMMUTABLE;
+ /*
+ * System files such as $Bitmap and $MFT are maintained by the driver
+ * itself, and writing them from userspace corrupts the volume.
+ * Always make them immutable regardless of the sys_immutable option.
+ * Directories are skipped so the root and $Extend stay usable.
+ */
+ if (ni->mft_no < FILE_first_user && S_ISREG(vi->i_mode))
+ vi->i_flags |= S_IMMUTABLE;
+
/*
* The number of 512-byte blocks used on disk (for stat). This is in so
* far inaccurate as it doesn't account for any named streams or other
s64 vcn = ntfs_pidx_to_cluster(vol, folio->index);
s64 end_vcn = ntfs_bytes_to_cluster(vol, ni->allocated_size);
unsigned int folio_sz;
- struct runlist_element *rl = NULL;
loff_t i_size = i_size_read(vi);
ntfs_debug("Entering for inode 0x%llx, attribute type 0x%x, folio index 0x%lx.",
&tni, &ref_inos[nr_ref_inos])) {
unsigned int mft_record_off = 0;
s64 vcn_off = vcn;
+ s64 rl_len = 0;
/*
* The record should be written. If a locked ntfs
}
if (vol->cluster_size < folio_size(folio)) {
+ struct runlist_element *rl;
+
down_write(&ni->runlist.lock);
rl = ntfs_attr_vcn_to_rl(ni, vcn_off, &lcn);
+ if (!IS_ERR(rl))
+ rl_len = rl->length - (vcn_off - rl->vcn);
up_write(&ni->runlist.lock);
if (IS_ERR(rl) || lcn < 0) {
err = -EIO;
if (vol->cluster_size == NTFS_BLOCK_SIZE &&
(mft_record_off ||
- (rl && rl->length - (vcn_off - rl->vcn) == 1) ||
+ rl_len == 1 ||
mft_ofs + NTFS_BLOCK_SIZE >= PAGE_SIZE))
folio_sz = NTFS_BLOCK_SIZE;
else
struct ntfs_volume *vol = NTFS_SB(sb);
struct ntfs_inode *old_ni, *new_ni = NULL;
struct ntfs_inode *old_dir_ni = NTFS_I(old_dir), *new_dir_ni = NTFS_I(new_dir);
+ bool new_dir_first = false;
if (NVolShutdown(old_dir_ni->vol))
return -EIO;
old_inode = old_dentry->d_inode;
new_inode = new_dentry->d_inode;
old_ni = NTFS_I(old_inode);
+ if (new_inode)
+ new_ni = NTFS_I(new_inode);
+ if (old_dir != new_dir)
+ new_dir_first = is_subdir(new_dentry->d_parent,
+ old_dentry->d_parent);
if (!(vol->vol_flags & VOLUME_IS_DIRTY))
ntfs_set_volume_flags(vol, VOLUME_IS_DIRTY);
mutex_lock_nested(&old_ni->mrec_lock, NTFS_INODE_MUTEX_NORMAL);
- mutex_lock_nested(&old_dir_ni->mrec_lock, NTFS_INODE_MUTEX_PARENT);
+ if (new_ni)
+ mutex_lock_nested(&new_ni->mrec_lock, NTFS_INODE_MUTEX_NORMAL_2);
+
+ if (old_dir == new_dir) {
+ mutex_lock_nested(&old_dir_ni->mrec_lock, NTFS_INODE_MUTEX_PARENT);
+ } else if (new_dir_first) {
+ mutex_lock_nested(&new_dir_ni->mrec_lock, NTFS_INODE_MUTEX_PARENT);
+ mutex_lock_nested(&old_dir_ni->mrec_lock, NTFS_INODE_MUTEX_PARENT_2);
+ } else {
+ mutex_lock_nested(&old_dir_ni->mrec_lock, NTFS_INODE_MUTEX_PARENT);
+ mutex_lock_nested(&new_dir_ni->mrec_lock, NTFS_INODE_MUTEX_PARENT_2);
+ }
- if (NInoBeingDeleted(old_ni) || NInoBeingDeleted(old_dir_ni)) {
+ if (NInoBeingDeleted(old_ni) || NInoBeingDeleted(old_dir_ni) ||
+ (new_ni && NInoBeingDeleted(new_ni)) ||
+ (old_dir != new_dir && NInoBeingDeleted(new_dir_ni))) {
err = -ENOENT;
- goto unlock_old;
+ goto err_out;
}
is_dir = S_ISDIR(old_inode->i_mode);
if (new_inode) {
- new_ni = NTFS_I(new_inode);
- mutex_lock_nested(&new_ni->mrec_lock, NTFS_INODE_MUTEX_NORMAL_2);
- if (old_dir != new_dir) {
- mutex_lock_nested(&new_dir_ni->mrec_lock, NTFS_INODE_MUTEX_PARENT_2);
- if (NInoBeingDeleted(new_dir_ni)) {
- err = -ENOENT;
- goto err_out;
- }
- }
-
- if (NInoBeingDeleted(new_ni)) {
- err = -ENOENT;
- goto err_out;
- }
-
if (is_dir) {
struct mft_record *ni_mrec;
err = ntfs_delete(new_ni, new_dir_ni, uname_new, new_name_len, false);
if (err)
goto err_out;
- } else {
- if (old_dir != new_dir) {
- mutex_lock_nested(&new_dir_ni->mrec_lock, NTFS_INODE_MUTEX_PARENT_2);
- if (NInoBeingDeleted(new_dir_ni)) {
- err = -ENOENT;
- goto err_out;
- }
- }
}
err = __ntfs_link(old_ni, new_dir_ni, uname_new, new_name_len);
inode_inc_iversion(new_dir);
err_out:
- if (old_dir != new_dir)
+ if (old_dir == new_dir) {
+ mutex_unlock(&old_dir_ni->mrec_lock);
+ } else if (new_dir_first) {
+ mutex_unlock(&old_dir_ni->mrec_lock);
mutex_unlock(&new_dir_ni->mrec_lock);
- if (new_inode)
+ } else {
+ mutex_unlock(&new_dir_ni->mrec_lock);
+ mutex_unlock(&old_dir_ni->mrec_lock);
+ }
+ if (new_ni)
mutex_unlock(&new_ni->mrec_lock);
-
-unlock_old:
- mutex_unlock(&old_dir_ni->mrec_lock);
mutex_unlock(&old_ni->mrec_lock);
if (uname_new)
kmem_cache_free(ntfs_name_cache, uname_new);
u |= 1 << KPF_MMAP;
if (is_anon) {
u |= 1 << KPF_ANON;
- if (mapping & FOLIO_MAPPING_KSM)
+ if ((mapping & FOLIO_MAPPING_FLAGS) == FOLIO_MAPPING_KSM)
u |= 1 << KPF_KSM;
}
static void rdtgroup_destroy_root(void);
+static void mon_put_kn_priv(void);
+
struct dentry *debugfs_resctrl;
/*
*
* On resource group creation via a mkdir, an extra kernfs_node reference is
* taken to ensure that the rdtgroup structure remains accessible for the
- * rdtgroup_kn_unlock() calls where it is removed.
+ * rdtgroup_kn_unlock() calls where it is removed. The default group is
+ * statically allocated: it does not have an extra reference but will have
+ * RDT_DELETED set on unmount to support safe access to its associated files
+ * via rdtgroup_kn_lock_live/rdtgroup_kn_unlock().
*
- * Drop the extra reference here, then free the rdtgroup structure.
+ * For all but the default group: drop the extra reference, then free the
+ * rdtgroup structure.
*
* Return: void
*/
static void rdtgroup_remove(struct rdtgroup *rdtgrp)
{
+ if (rdtgrp == &rdtgroup_default)
+ return;
kernfs_put(rdtgrp->kn);
kfree(rdtgrp);
}
goto out;
}
+ /* Avoid races from pending operations from a previous mount */
+ if (atomic_read(&rdtgroup_default.waitcount) != 0) {
+ ret = -EBUSY;
+ goto out;
+ }
+
ret = setup_rmid_lru_list();
if (ret)
goto out;
kernfs_remove(kn_mondata);
out_mongrp:
if (resctrl_arch_mon_capable()) {
+ mon_put_kn_priv();
rdtgroup_unassign_cntrs(&rdtgroup_default);
kernfs_remove(kn_mongrp);
}
if (rdtgrp == &rdtgroup_default)
continue;
- if (rdtgrp->mode == RDT_MODE_PSEUDO_LOCKSETUP ||
- rdtgrp->mode == RDT_MODE_PSEUDO_LOCKED)
- rdtgroup_pseudo_lock_remove(rdtgrp);
-
/*
* Give any CPUs back to the default group. We cannot copy
* cpu_online_mask because a CPU might have executed the
rdtgroup_unassign_cntrs(rdtgrp);
- free_rmid(rdtgrp->closid, rdtgrp->mon.rmid);
+ if (rdtgrp->mode == RDT_MODE_PSEUDO_LOCKSETUP ||
+ rdtgrp->mode == RDT_MODE_PSEUDO_LOCKED) {
+ rdtgroup_pseudo_lock_remove(rdtgrp);
+ } else {
+ /* Pseudo-locked group's RMID is freed during setup. */
+ free_rmid(rdtgrp->closid, rdtgrp->mon.rmid);
+ }
kernfs_remove(rdtgrp->kn);
list_del(&rdtgrp->rdtgroup_list);
mon_put_kn_priv();
rdt_pseudo_lock_release();
rdtgroup_default.mode = RDT_MODE_SHAREABLE;
+ rdtgroup_default.flags = RDT_DELETED;
closid_exit();
schemata_list_destroy();
rdtgroup_destroy_root();
ctx->kfc.root = rdt_root;
rdtgroup_default.kn = kernfs_root_to_node(rdt_root);
+ rdtgroup_default.flags = 0;
return 0;
}
struct smb3_fs_context *ctx;
atomic_t active;
atomic_t mnt_cifs_flags;
+ atomic_t outstanding_rreq; /* nr of rreqs not yet fully deinitialized */
struct delayed_work prune_tlinks;
struct rcu_head rcu;
*/
fattr->cf_mode &= ~07777;
fattr->cf_mode |=
- le32_to_cpu(ppace[i]->sid.sub_auth[2]);
+ le32_to_cpu(ppace[i]->sid.sub_auth[2]) & 07777;
break;
} else {
if (compare_sids(&(ppace[i]->sid), pownersid) == 0) {
/* Wait for all opened files to release */
flush_workqueue(deferredclose_wq);
+ /*
+ * Wait for all in-flight netfs I/O requests to finish their
+ * cleanup_work so that any cifsFileInfo final puts they queue
+ * to fileinfo_put_wq/serverclose_wq have been queued, then
+ * drain the workqueue so the cfile dentry refs are dropped to
+ * avoid the busy dentry warning.
+ */
+ wait_var_event(&cifs_sb->outstanding_rreq,
+ !atomic_read(&cifs_sb->outstanding_rreq));
+ flush_workqueue(serverclose_wq);
+ flush_workqueue(fileinfo_put_wq);
+
/* finally release root dentry */
dput(cifs_sb->root);
cifs_sb->root = NULL;
}
}
- /* force revalidate of size and timestamps of target file now
- that target is updated on the server */
- CIFS_I(target_inode)->time = 0;
+ /*
+ * On success, duplicate_extents already updated the target inode attrs
+ * or marked them stale if the refresh failed. On failure, mark attrs
+ * stale because EOF may have changed before the clone failed.
+ */
+ if (rc)
+ CIFS_I(target_inode)->time = 0;
unlock:
/* although unlocking in the reverse order from locking is not
strictly necessary here it is a little cleaner to be consistent */
bool adjust_tz;
bool reparse_point;
bool contains_posix_file_info;
+ bool unknown_nlink;
struct {
/* ioctl response buffer */
struct {
#ifdef CONFIG_CIFS_ALLOW_INSECURE_LEGACY
#endif /* CONFIG_CIFS_ALLOW_INSECURE_LEGACY */
-void cifs_autodisable_serverino(struct cifs_sb_info *cifs_sb);
+void cifs_autodisable_serverino(struct cifs_sb_info *cifs_sb, const char *reason, int rc);
bool couldbe_mf_symlink(const struct cifs_fattr *fattr);
int check_mf_symlink(unsigned int xid, struct cifs_tcon *tcon,
struct cifs_sb_info *cifs_sb, struct cifs_fattr *fattr,
return rc;
}
-static int
-CIFSSMBSetPathInfoFB(const unsigned int xid, struct cifs_tcon *tcon,
- const char *fileName, const FILE_BASIC_INFO *data,
- const struct nls_table *nls_codepage,
- struct cifs_sb_info *cifs_sb)
-{
- int oplock = 0;
- struct cifs_open_parms oparms;
- struct cifs_fid fid;
- int rc;
-
- oparms = (struct cifs_open_parms) {
- .tcon = tcon,
- .cifs_sb = cifs_sb,
- .desired_access = GENERIC_WRITE,
- .create_options = cifs_create_options(cifs_sb, 0),
- .disposition = FILE_OPEN,
- .path = fileName,
- .fid = &fid,
- };
-
- rc = CIFS_open(xid, &oparms, &oplock, NULL);
- if (rc)
- goto out;
-
- rc = CIFSSMBSetFileInfo(xid, tcon, data, fid.netfid, current->tgid);
- CIFSSMBClose(xid, tcon, fid.netfid);
-out:
-
- return rc;
-}
-
int
CIFSSMBSetPathInfo(const unsigned int xid, struct cifs_tcon *tcon,
const char *fileName, const FILE_BASIC_INFO *data,
if (rc == -EAGAIN)
goto SetTimesRetry;
- if (rc == -EOPNOTSUPP)
- return CIFSSMBSetPathInfoFB(xid, tcon, fileName, data,
- nls_codepage, cifs_sb);
-
return rc;
}
spin_lock_init(&cifs_sb->tlink_tree_lock);
cifs_sb->tlink_tree = RB_ROOT;
+ atomic_set(&cifs_sb->outstanding_rreq, 0);
cifs_dbg(FYI, "file mode: %04ho dir mode: %04ho\n",
ctx->file_mode, ctx->dir_mode);
* After reconnecting to a different server, unique ids won't match anymore, so we disable
* serverino. This prevents dentry revalidation to think the dentry are stale (ESTALE).
*/
- cifs_autodisable_serverino(cifs_sb);
+ cifs_autodisable_serverino(cifs_sb, "DFS failover may potentially connect to a different server, inode numbers won't match anymore", 0);
/*
* Force the use of prefix path to support failover on DFS paths that resolve to targets
* that have different prefix paths.
{
struct cache_dfs_tgt *t;
- t = kmalloc_obj(*t, GFP_ATOMIC);
+ t = kmalloc_obj(*t, GFP_KERNEL);
if (!t)
return ERR_PTR(-ENOMEM);
- t->name = kstrdup(name, GFP_ATOMIC);
+ t->name = kstrdup(name, GFP_KERNEL);
if (!t->name) {
kfree(t);
return ERR_PTR(-ENOMEM);
target = READ_ONCE(ce->tgthint);
if (target) {
- th = kstrdup(target->name, GFP_ATOMIC);
+ th = kstrdup(target->name, GFP_KERNEL);
if (!th)
return -ENOMEM;
}
memset(ref, 0, sizeof(*ref));
- ref->path_name = kstrdup(path, GFP_ATOMIC);
+ ref->path_name = kstrdup(path, GFP_KERNEL);
if (!ref->path_name)
return -ENOMEM;
- ref->node_name = kstrdup(target, GFP_ATOMIC);
+ ref->node_name = kstrdup(target, GFP_KERNEL);
if (!ref->node_name) {
rc = -ENOMEM;
goto err_free_path;
* After reconnecting to a different server, unique ids won't match anymore, so we disable
* serverino. This prevents dentry revalidation to think the dentry are stale (ESTALE).
*/
- cifs_autodisable_serverino(cifs_sb);
+ cifs_autodisable_serverino(cifs_sb, "DFS failover may potentially connect to a different server, inode numbers won't match anymore", 0);
/*
* Force the use of prefix path to support failover on DFS paths that resolve to targets
* that have different prefix paths.
return smb_EIO1(smb_eio_trace_not_netfs_writeback, rreq->origin);
}
+ atomic_inc(&cifs_sb->outstanding_rreq);
return 0;
}
/* we do not want atime to be less than mtime, it broke some apps */
atime = inode_set_atime_to_ts(inode, current_time(inode));
mtime = inode_get_mtime(inode);
- if (timespec64_compare(&atime, &mtime))
+ if (timespec64_compare(&atime, &mtime) < 0)
inode_set_atime_to_ts(inode, inode_get_mtime(inode));
}
static void cifs_free_request(struct netfs_io_request *rreq)
{
struct cifs_io_request *req = container_of(rreq, struct cifs_io_request, rreq);
+ struct cifs_sb_info *cifs_sb = CIFS_SB(rreq->inode->i_sb);
if (req->cfile)
cifsFileInfo_put(req->cfile);
+
+ if (atomic_dec_and_test(&cifs_sb->outstanding_rreq))
+ wake_up_var(&cifs_sb->outstanding_rreq);
}
static void cifs_free_subrequest(struct netfs_io_subrequest *subreq)
struct cifs_tcon *tcon = cifs_sb_master_tcon(cifs_sb);
memset(fattr, 0, sizeof(*fattr));
+ if (data->unknown_nlink)
+ fattr->cf_flags |= CIFS_FATTR_UNKNOWN_NLINK;
fattr->cf_cifsattrs = le32_to_cpu(info->Attributes);
if (info->DeletePending)
fattr->cf_flags |= CIFS_FATTR_DELETE_PENDING;
fattr->cf_uniqueid = CIFS_I(*inode)->uniqueid;
else {
fattr->cf_uniqueid = iunique(sb, ROOT_I);
- cifs_autodisable_serverino(cifs_sb);
+ cifs_autodisable_serverino(cifs_sb, "Cannot retrieve inode number via get_srv_inum", rc);
}
return;
}
fattr->cf_flags &= ~CIFS_FATTR_INO_COLLISION;
if (inode_has_hashed_dentries(inode)) {
- cifs_autodisable_serverino(CIFS_SB(sb));
+ cifs_autodisable_serverino(CIFS_SB(sb), "Inode number collision detected", 0);
iput(inode);
fattr->cf_uniqueid = iunique(sb, ROOT_I);
goto retry_iget5_locked;
iget_root:
if (!rc) {
if (fattr.cf_flags & CIFS_FATTR_JUNCTION) {
+ cifs_dbg(VFS, "Removing junction mark and disabling 'serverino' to prevent inode collisions\n");
fattr.cf_flags &= ~CIFS_FATTR_JUNCTION;
- cifs_autodisable_serverino(cifs_sb);
+ cifs_autodisable_serverino(cifs_sb, "Cannot retrieve attributes for junction point", rc);
}
inode = cifs_iget(sb, &fattr);
}
}
skip_invalidate:
- clear_bit_unlock(CIFS_INO_LOCK, flags);
- smp_mb__after_atomic();
- wake_up_bit(flags, CIFS_INO_LOCK);
+ clear_and_wake_up_bit(CIFS_INO_LOCK, flags);
return rc;
}
struct cifs_open_parms oparms;
struct cifs_io_parms io_parms = {0};
int buf_type = CIFS_NO_BUFFER;
- struct cifs_open_info_data query_data;
+ struct cifs_open_info_data query_data = {};
oparms = (struct cifs_open_parms) {
.tcon = tcon,
int buf_type = CIFS_NO_BUFFER;
__le16 *utf16_path;
__u8 oplock = SMB2_OPLOCK_LEVEL_NONE;
- struct smb2_file_all_info *pfile_info = NULL;
+ struct cifs_open_info_data data = {};
oparms = (struct cifs_open_parms) {
.tcon = tcon,
if (utf16_path == NULL)
return -ENOMEM;
- pfile_info = kzalloc(sizeof(struct smb2_file_all_info) + PATH_MAX * 2,
- GFP_KERNEL);
-
- if (pfile_info == NULL) {
- kfree(utf16_path);
- return -ENOMEM;
- }
-
- rc = SMB2_open(xid, &oparms, utf16_path, &oplock, pfile_info, NULL,
+ rc = SMB2_open(xid, &oparms, utf16_path, &oplock, &data, NULL,
NULL, NULL);
if (rc)
goto qmf_out_open_fail;
- if (pfile_info->EndOfFile != cpu_to_le64(CIFS_MF_SYMLINK_FILE_SIZE)) {
+ if (data.fi.EndOfFile != cpu_to_le64(CIFS_MF_SYMLINK_FILE_SIZE)) {
/* it's not a symlink */
rc = -ENOENT; /* Is there a better rc to return? */
goto qmf_out;
SMB2_close(xid, tcon, fid.persistent_fid, fid.volatile_fid);
qmf_out_open_fail:
kfree(utf16_path);
- kfree(pfile_info);
return rc;
}
}
void
-cifs_autodisable_serverino(struct cifs_sb_info *cifs_sb)
+cifs_autodisable_serverino(struct cifs_sb_info *cifs_sb, const char *reason, int rc)
{
unsigned int sbflags = cifs_sb_flags(cifs_sb);
atomic_andnot(CIFS_MOUNT_SERVER_INUM, &cifs_sb->mnt_cifs_flags);
cifs_sb->mnt_cifs_serverino_autodisabled = true;
+ if (rc)
+ cifs_dbg(VFS, "%s: %d\n", reason, rc);
+ else
+ cifs_dbg(VFS, "%s\n", reason);
cifs_dbg(VFS, "Autodisabling the use of server inode numbers on %s\n",
tcon ? tcon->tree_name : "new server");
cifs_dbg(VFS, "The server doesn't seem to support them properly or the files might be on different servers (DFS)\n");
node->ref_flag = le16_to_cpu(ref->ReferralEntryFlags);
/* copy DfsPath */
+ if (le16_to_cpu(ref->DfsPathOffset) > data_end - (char *)ref) {
+ rc = -EINVAL;
+ goto parse_DFS_referrals_exit;
+ }
temp = (char *)ref + le16_to_cpu(ref->DfsPathOffset);
max_len = data_end - temp;
node->path_name = cifs_strndup_from_utf16(temp, max_len,
}
/* copy link target UNC */
+ if (le16_to_cpu(ref->NetworkAddressOffset) > data_end - (char *)ref) {
+ rc = -EINVAL;
+ goto parse_DFS_referrals_exit;
+ }
temp = (char *)ref + le16_to_cpu(ref->NetworkAddressOffset);
max_len = data_end - temp;
node->node_name = cifs_strndup_from_utf16(temp, max_len,
if (rc == 0) {
cifsFile->invalidHandle = false;
} else if (rc == -EOPNOTSUPP && (sbflags & CIFS_MOUNT_SERVER_INUM)) {
- cifs_autodisable_serverino(cifs_sb);
+ cifs_autodisable_serverino(cifs_sb, "Cannot retrieve inode number via query_dir_first", rc);
goto ffirst_retry;
}
error_exit:
fattr.cf_uniqueid = de.ino;
} else {
fattr.cf_uniqueid = iunique(sb, ROOT_I);
- cifs_autodisable_serverino(cifs_sb);
+ cifs_autodisable_serverino(cifs_sb, "Cannot retrieve inode number from readdir", 0);
}
if ((sbflags & CIFS_MOUNT_MF_SYMLINKS) && couldbe_mf_symlink(&fattr))
char *sym = NULL;
struct kvec iov;
bool directory;
+ int path_len;
int rc = 0;
if (strlen(symname) > REPARSE_SYM_PATH_MAX)
if (!(sbflags & CIFS_MOUNT_POSIX_PATHS) && symname[0] == '/')
sym[0] = sym[1] = sym[2] = sym[5] = '_';
- path = cifs_convert_path_to_utf16(sym, cifs_sb);
+ /*
+ * On a POSIX paths mount the symlink target is stored verbatim, so
+ * convert it with cifs_strndup_to_utf16(). cifs_convert_path_to_utf16()
+ * must not be used here: it strips a leading path separator (it is
+ * meant for share-relative SMB paths), which would corrupt an absolute
+ * POSIX symlink target such as "/foo/bar". Using NO_MAP_UNI_RSVD also
+ * matches the readback path in smb2_parse_native_symlink().
+ */
+ if (sbflags & CIFS_MOUNT_POSIX_PATHS)
+ path = cifs_strndup_to_utf16(sym, strlen(sym), &path_len,
+ cifs_sb->local_nls,
+ NO_MAP_UNI_RSVD);
+ else
+ path = cifs_convert_path_to_utf16(sym, cifs_sb);
+
if (!path) {
rc = -ENOMEM;
goto out;
*/
#include <linux/bsearch.h>
+
+#include <kunit/visibility.h>
+
#include "cifsproto.h"
#include "smb1proto.h"
#include "smberr.h"
}
#if IS_ENABLED(CONFIG_SMB1_KUNIT_TESTS)
-#define EXPORT_SYMBOL_FOR_SMB_TEST(sym) \
- EXPORT_SYMBOL_FOR_MODULES(sym, "smb1maperror_test")
-
const struct ntstatus_to_dos_err *
search_ntstatus_to_dos_map_test(__u32 ntstatus)
{
return search_ntstatus_to_dos_map(ntstatus);
}
-EXPORT_SYMBOL_FOR_SMB_TEST(search_ntstatus_to_dos_map_test);
+EXPORT_SYMBOL_IF_KUNIT(search_ntstatus_to_dos_map_test);
const struct ntstatus_to_dos_err *
ntstatus_to_dos_map_test = ntstatus_to_dos_map;
-EXPORT_SYMBOL_FOR_SMB_TEST(ntstatus_to_dos_map_test);
+EXPORT_SYMBOL_IF_KUNIT(ntstatus_to_dos_map_test);
unsigned int ntstatus_to_dos_num = ARRAY_SIZE(ntstatus_to_dos_map);
-EXPORT_SYMBOL_FOR_SMB_TEST(ntstatus_to_dos_num);
+EXPORT_SYMBOL_IF_KUNIT(ntstatus_to_dos_num);
const struct smb_to_posix_error *
search_mapping_table_ERRDOS_test(__u16 smb_err)
{
return search_mapping_table_ERRDOS(smb_err);
}
-EXPORT_SYMBOL_FOR_SMB_TEST(search_mapping_table_ERRDOS_test);
+EXPORT_SYMBOL_IF_KUNIT(search_mapping_table_ERRDOS_test);
const struct smb_to_posix_error *
mapping_table_ERRDOS_test = mapping_table_ERRDOS;
-EXPORT_SYMBOL_FOR_SMB_TEST(mapping_table_ERRDOS_test);
+EXPORT_SYMBOL_IF_KUNIT(mapping_table_ERRDOS_test);
unsigned int mapping_table_ERRDOS_num = ARRAY_SIZE(mapping_table_ERRDOS);
-EXPORT_SYMBOL_FOR_SMB_TEST(mapping_table_ERRDOS_num);
+EXPORT_SYMBOL_IF_KUNIT(mapping_table_ERRDOS_num);
const struct smb_to_posix_error *
search_mapping_table_ERRSRV_test(__u16 smb_err)
{
return search_mapping_table_ERRSRV(smb_err);
}
-EXPORT_SYMBOL_FOR_SMB_TEST(search_mapping_table_ERRSRV_test);
+EXPORT_SYMBOL_IF_KUNIT(search_mapping_table_ERRSRV_test);
const struct smb_to_posix_error *
mapping_table_ERRSRV_test = mapping_table_ERRSRV;
-EXPORT_SYMBOL_FOR_SMB_TEST(mapping_table_ERRSRV_test);
+EXPORT_SYMBOL_IF_KUNIT(mapping_table_ERRSRV_test);
unsigned int mapping_table_ERRSRV_num = ARRAY_SIZE(mapping_table_ERRSRV);
-EXPORT_SYMBOL_FOR_SMB_TEST(mapping_table_ERRSRV_num);
+EXPORT_SYMBOL_IF_KUNIT(mapping_table_ERRSRV_num);
#endif
MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("KUnit tests of SMB1 maperror");
+MODULE_IMPORT_NS("EXPORTED_FOR_KUNIT_TESTING");
cifs_is_path_accessible(const unsigned int xid, struct cifs_tcon *tcon,
struct cifs_sb_info *cifs_sb, const char *full_path)
{
- int rc;
- FILE_ALL_INFO *file_info;
+ int rc = -EOPNOTSUPP;
+ FILE_ALL_INFO file_info;
- file_info = kmalloc_obj(FILE_ALL_INFO);
- if (file_info == NULL)
- return -ENOMEM;
+ if (tcon->ses->capabilities & CAP_NT_SMBS)
+ rc = CIFSSMBQPathInfo(xid, tcon, full_path, &file_info,
+ 0 /* not legacy */, cifs_sb->local_nls,
+ cifs_remap(cifs_sb));
- rc = CIFSSMBQPathInfo(xid, tcon, full_path, file_info,
- 0 /* not legacy */, cifs_sb->local_nls,
- cifs_remap(cifs_sb));
+ /*
+ * Non-UNICODE variant of fallback functions below expands wildcards,
+ * so they cannot be used for querying paths with wildcard characters.
+ * Therefore for such paths returns -ENOENT as they cannot exist.
+ */
+ if ((rc == -EOPNOTSUPP || rc == -EINVAL) &&
+ !(tcon->ses->capabilities & CAP_UNICODE) &&
+ strpbrk(full_path, "*?\"><"))
+ rc = -ENOENT;
if (rc == -EOPNOTSUPP || rc == -EINVAL)
- rc = SMBQueryInformation(xid, tcon, full_path, file_info,
+ rc = SMBQueryInformation(xid, tcon, full_path, &file_info,
cifs_sb->local_nls, cifs_remap(cifs_sb));
- kfree(file_info);
return rc;
}
struct cifs_open_parms oparms;
struct cifsFileInfo *open_file;
FILE_BASIC_INFO new_buf;
- struct cifs_open_info_data query_data;
+ struct cifs_open_info_data query_data = {};
__le64 write_time = buf->LastWriteTime;
struct cifsInodeInfo *cinode = CIFS_I(inode);
struct cifs_sb_info *cifs_sb = CIFS_SB(inode->i_sb);
__le16 *smb2_path;
__u8 smb2_oplock;
struct cifs_open_info_data *data = buf;
- struct smb2_file_all_info file_info = {};
- struct smb2_file_all_info *smb2_data = data ? &file_info : NULL;
struct kvec err_iov = {};
int err_buftype = CIFS_NO_BUFFER;
struct cifs_fid *fid = oparms->fid;
}
smb2_oplock = SMB2_OPLOCK_LEVEL_BATCH;
- rc = SMB2_open(xid, oparms, smb2_path, &smb2_oplock, smb2_data, NULL, &err_iov,
+ rc = SMB2_open(xid, oparms, smb2_path, &smb2_oplock, data, NULL, &err_iov,
&err_buftype);
if (rc == -EACCES && retry_without_read_attributes) {
free_rsp_buf(err_buftype, err_iov.iov_base);
memset(&err_iov, 0, sizeof(err_iov));
err_buftype = CIFS_NO_BUFFER;
oparms->desired_access &= ~FILE_READ_ATTRIBUTES;
- rc = SMB2_open(xid, oparms, smb2_path, &smb2_oplock, smb2_data, NULL, &err_iov,
+ rc = SMB2_open(xid, oparms, smb2_path, &smb2_oplock, data, NULL, &err_iov,
&err_buftype);
}
if (rc && data) {
oparms->path,
&data->symlink_target);
if (!rc) {
- memset(smb2_data, 0, sizeof(*smb2_data));
+ memset(&data->fi, 0, sizeof(data->fi));
oparms->create_options |= OPEN_REPARSE_POINT;
- rc = SMB2_open(xid, oparms, smb2_path, &smb2_oplock, smb2_data,
+ rc = SMB2_open(xid, oparms, smb2_path, &smb2_oplock, data,
NULL, NULL, NULL);
oparms->create_options &= ~OPEN_REPARSE_POINT;
}
rc = 0;
}
- if (smb2_data) {
+ if (data) {
/* if open response does not have IndexNumber field - get it */
- if (smb2_data->IndexNumber == 0) {
+ if (data->fi.IndexNumber == 0) {
rc = SMB2_get_srv_num(xid, oparms->tcon,
fid->persistent_fid,
fid->volatile_fid,
- &smb2_data->IndexNumber);
+ &data->fi.IndexNumber);
if (rc) {
/*
* let get_inode_info disable server inode
* numbers
*/
- smb2_data->IndexNumber = 0;
+ data->fi.IndexNumber = 0;
rc = 0;
}
}
- memcpy(&data->fi, smb2_data, sizeof(data->fi));
}
*oplock = smb2_oplock;
*
*/
#include <linux/errno.h>
+
+#include <kunit/visibility.h>
+
#include "cifsproto.h"
#include "cifs_debug.h"
#include "smb2proto.h"
}
#if IS_ENABLED(CONFIG_SMB_KUNIT_TESTS)
-#define EXPORT_SYMBOL_FOR_SMB_TEST(sym) \
- EXPORT_SYMBOL_FOR_MODULES(sym, "smb2maperror_test")
-
const struct status_to_posix_error *smb2_get_err_map_test(__u32 smb2_status)
{
return smb2_get_err_map(smb2_status);
}
-EXPORT_SYMBOL_FOR_SMB_TEST(smb2_get_err_map_test);
+EXPORT_SYMBOL_IF_KUNIT(smb2_get_err_map_test);
const struct status_to_posix_error *smb2_error_map_table_test = smb2_error_map_table;
-EXPORT_SYMBOL_FOR_SMB_TEST(smb2_error_map_table_test);
+EXPORT_SYMBOL_IF_KUNIT(smb2_error_map_table_test);
unsigned int smb2_error_map_num = ARRAY_SIZE(smb2_error_map_table);
-EXPORT_SYMBOL_FOR_SMB_TEST(smb2_error_map_num);
+EXPORT_SYMBOL_IF_KUNIT(smb2_error_map_num);
#endif
MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("KUnit tests of SMB2 maperror");
+MODULE_IMPORT_NS("EXPORTED_FOR_KUNIT_TESTING");
#include "nterr.h"
#include "cached_dir.h"
+static unsigned int __smb2_calc_size(void *buf, bool *have_data,
+ bool *data_area_overlap);
+
static int
check_smb2_hdr(struct smb2_hdr *shdr, __u64 mid)
{
int command;
__u32 calc_len; /* calculated length */
__u64 mid;
+ bool have_data;
+ bool data_area_overlap;
/* If server is a channel, select the primary channel */
pserver = SERVER_IS_CHAN(server) ? server->primary_server : server;
}
}
- calc_len = smb2_calc_size(buf);
+ have_data = false;
+ data_area_overlap = false;
+ calc_len = __smb2_calc_size(buf, &have_data, &data_area_overlap);
+
+ /* Reject responses whose data area overlaps the fixed area. */
+ if (data_area_overlap)
+ return 1;
/* For SMB2_IOCTL, OutputOffset and OutputLength are optional, so might
* be 0, and not a real miscalculation */
/* Windows 7 server returns 24 bytes more */
if (calc_len + 24 == len && command == SMB2_OPLOCK_BREAK_HE)
return 0;
- /* server can return one byte more due to implied bcc[0] */
- if (calc_len == len + 1)
+ /*
+ * Server can return one byte more due to implied bcc[0].
+ * Allow it only when there is no data area; if data_length > 0
+ * the +1 gap indicates an overreported data length rather than
+ * the bcc[0] omission.
+ */
+ if (calc_len == len + 1 && !have_data)
return 0;
/*
}
/*
- * Calculate the size of the SMB message based on the fixed header
- * portion, the number of word parameters and the data portion of the message.
+ * Calculate the size of the SMB message based on the fixed header, fixed
+ * parameter area, and variable data area.
+ *
+ * If have_data is not NULL, it is set when a non-empty data area is found.
+ * If data_area_overlap is not NULL, it is set when the data area overlaps
+ * the fixed area.
*/
-unsigned int
-smb2_calc_size(void *buf)
+static unsigned int
+__smb2_calc_size(void *buf, bool *have_data, bool *data_area_overlap)
{
struct smb2_pdu *pdu = buf;
struct smb2_hdr *shdr = &pdu->hdr;
int offset; /* the offset from the beginning of SMB to data area */
- int data_length; /* the length of the variable length data area */
+ int data_length = 0; /* the length of the variable length data area */
/* Structure Size has already been checked to make sure it is 64 */
int len = le16_to_cpu(shdr->StructureSize);
+ if (have_data)
+ *have_data = false;
+ if (data_area_overlap)
+ *data_area_overlap = false;
+
/*
* StructureSize2, ie length of fixed parameter area has already
* been checked to make sure it is the correct length.
if (offset + 1 < len) {
cifs_dbg(VFS, "data area offset %d overlaps SMB2 header %d\n",
offset + 1, len);
+ if (data_area_overlap)
+ *data_area_overlap = true;
data_length = 0;
+ goto calc_size_exit;
} else {
len = offset + data_length;
}
}
calc_size_exit:
cifs_dbg(FYI, "SMB2 len %d\n", len);
+ if (have_data)
+ *have_data = (data_length > 0);
return len;
}
+unsigned int
+smb2_calc_size(void *buf)
+{
+ return __smb2_calc_size(buf, NULL, NULL);
+}
+
/* Note: caller must free return buffer */
__le16 *
cifs_convert_path_to_utf16(const char *from, struct cifs_sb_info *cifs_sb)
if (le32_to_cpu(io_rsp->OutputCount) < qi.input_buffer_length)
qi.input_buffer_length = le32_to_cpu(io_rsp->OutputCount);
if (qi.input_buffer_length > 0 &&
- le32_to_cpu(io_rsp->OutputOffset) + qi.input_buffer_length
- > rsp_iov[1].iov_len) {
+ size_add(le32_to_cpu(io_rsp->OutputOffset),
+ qi.input_buffer_length) > rsp_iov[1].iov_len) {
rc = -EFAULT;
goto out;
}
u64 len, u64 dest_off)
{
int rc;
+ int qrc;
unsigned int ret_data_len;
struct inode *inode;
+ struct smb2_file_all_info file_inf;
struct duplicate_extents_to_file dup_ext_buf;
+ struct timespec64 ts;
struct cifs_tcon *tcon = tlink_tcon(trgtfile->tlink);
+ u64 asize;
/* server fileays advertise duplicate extent support with this flag */
if ((le32_to_cpu(tcon->fsAttrInfo.Attributes) &
if (ret_data_len > 0)
cifs_dbg(FYI, "Non-zero response length in duplicate extents\n");
+ if (rc == 0) {
+ qrc = SMB2_query_info(xid, tcon, trgtfile->fid.persistent_fid,
+ trgtfile->fid.volatile_fid, &file_inf);
+ spin_lock(&inode->i_lock);
+ if (qrc == 0) {
+ asize = le64_to_cpu(file_inf.AllocationSize);
+ CIFS_I(inode)->time = jiffies;
+ if (file_inf.LastWriteTime) {
+ ts = cifs_NTtimeToUnix(file_inf.LastWriteTime);
+ inode_set_mtime_to_ts(inode, ts);
+ }
+ if (file_inf.ChangeTime) {
+ ts = cifs_NTtimeToUnix(file_inf.ChangeTime);
+ inode_set_ctime_to_ts(inode, ts);
+ }
+ if (file_inf.LastAccessTime) {
+ ts = cifs_NTtimeToUnix(file_inf.LastAccessTime);
+ inode_set_atime_to_ts(inode, ts);
+ }
+ inode->i_blocks = CIFS_INO_BLOCKS(asize);
+ } else {
+ CIFS_I(inode)->time = 0; /* force reval */
+ }
+ spin_unlock(&inode->i_lock);
+ }
+
duplicate_extents_out:
if (rc)
trace_smb3_clone_err(xid, srcfile->fid.volatile_fid,
goto out;
filemap_invalidate_lock(inode->i_mapping);
+ /*
+ * Flush dirty data first, otherwise a dirty folio spanning the punched
+ * range may be written back after the ioctl and refill the hole.
+ */
+ rc = filemap_write_and_wait_range(inode->i_mapping, offset,
+ offset + len - 1);
+ if (rc < 0)
+ goto unlock;
+
/*
* We implement the punch hole through ioctl, so we need remove the page
* caches first, otherwise the data may be inconsistent with the server.
char *buf)
{
struct cifs_io_parms io_parms = {0};
- int nbytes;
+ unsigned int nbytes;
int rc = 0;
struct kvec iov[2];
rc = SMB2_write(xid, &io_parms, &nbytes, iov, 1);
if (rc)
break;
+ if (!nbytes)
+ return -EIO;
if (nbytes > len)
return -EINVAL;
- buf += nbytes;
off += nbytes;
len -= nbytes;
}
loff_t off, loff_t len)
{
struct file_allocated_range_buffer in_data, *out_data = NULL, *tmp_data;
+ struct inode *inode = d_inode(cfile->dentry);
u32 out_data_len;
char *buf = NULL;
+ u64 range_start, range_len, range_end;
loff_t l;
int rc;
+ buf = kvzalloc(min_t(loff_t, len, SMB2_MAX_BUFFER_SIZE), GFP_KERNEL);
+ if (!buf) {
+ rc = -ENOMEM;
+ goto out;
+ }
+
+ if (off >= i_size_read(inode)) {
+ rc = smb3_simple_fallocate_write_range(xid, tcon, cfile,
+ off, len, buf);
+ goto out;
+ }
+
in_data.file_offset = cpu_to_le64(off);
in_data.length = cpu_to_le64(len);
rc = SMB2_ioctl(xid, tcon, cfile->fid.persistent_fid,
if (rc)
goto out;
- buf = kzalloc(1024 * 1024, GFP_KERNEL);
- if (buf == NULL) {
- rc = -ENOMEM;
- goto out;
- }
-
tmp_data = out_data;
while (len) {
/*
goto out;
}
- if (off < le64_to_cpu(tmp_data->file_offset)) {
+ range_start = le64_to_cpu(tmp_data->file_offset);
+ range_len = le64_to_cpu(tmp_data->length);
+ if (check_add_overflow(range_start, range_len, &range_end) ||
+ range_end > S64_MAX) {
+ rc = -EINVAL;
+ goto out;
+ }
+
+ if (off < range_start) {
/*
* We are at a hole. Write until the end of the region
* or until the next allocated data,
* whichever comes next.
*/
- l = le64_to_cpu(tmp_data->file_offset) - off;
+ l = range_start - off;
if (len < l)
l = len;
rc = smb3_simple_fallocate_write_range(xid, tcon,
* until the end of the data or the end of the region
* we are supposed to fallocate, whichever comes first.
*/
- l = le64_to_cpu(tmp_data->length);
- if (len < l)
- l = len;
- off += l;
- len -= l;
+ if (off < range_end) {
+ l = range_end - off;
+ if (len < l)
+ l = len;
+ off += l;
+ len -= l;
+ }
tmp_data = &tmp_data[1];
out_data_len -= sizeof(struct file_allocated_range_buffer);
out:
kfree(out_data);
- kfree(buf);
+ kvfree(buf);
return rc;
}
struct cifsFileInfo *cfile = file->private_data;
long rc = -EOPNOTSUPP;
unsigned int xid;
- loff_t new_eof;
+ loff_t old_eof, new_eof;
+ struct smb2_file_all_info file_inf;
+ u64 asize;
+ int qrc;
xid = get_xid();
inode = d_inode(cfile->dentry);
cifsi = CIFS_I(inode);
+ old_eof = i_size_read(inode);
trace_smb3_falloc_enter(xid, cfile->fid.persistent_fid, tcon->tid,
tcon->ses->Suid, off, len);
/* if file not oplocked can't be sure whether asking to extend size */
if (!CIFS_CACHE_READ(cifsi))
- if (keep_size == false) {
+ if (!keep_size) {
trace_smb3_falloc_err(xid, cfile->fid.persistent_fid,
tcon->tid, tcon->ses->Suid, off, len, rc);
free_xid(xid);
/*
* Extending the file
*/
- if ((keep_size == false) && i_size_read(inode) < off + len) {
+ if (!keep_size && old_eof < off + len) {
rc = inode_newsize_ok(inode, off + len);
if (rc)
goto out;
+ /*
+ * A small range at or beyond EOF can be allocated by writing
+ * zeroes. For off > old_eof, this preserves the intervening
+ * hole instead of allocating from offset 0.
+ */
+ if (off > old_eof ||
+ (off == old_eof && old_eof != 0 &&
+ (cifsi->cifsAttrs & FILE_ATTRIBUTE_SPARSE_FILE))) {
+ if (len > 1024 * 1024) {
+ rc = -EOPNOTSUPP;
+ goto out;
+ }
+
+ rc = smb3_simple_fallocate_range(xid, tcon, cfile,
+ off, len);
+ if (rc) {
+ spin_lock(&inode->i_lock);
+ cifsi->time = 0;
+ spin_unlock(&inode->i_lock);
+ goto out;
+ }
+
+ new_eof = off + len;
+ netfs_resize_file(&cifsi->netfs, new_eof, true);
+ cifs_setsize(inode, new_eof);
+
+ qrc = SMB2_query_info(xid, tcon,
+ cfile->fid.persistent_fid,
+ cfile->fid.volatile_fid, &file_inf);
+ spin_lock(&inode->i_lock);
+ if (qrc == 0) {
+ asize = le64_to_cpu(file_inf.AllocationSize);
+ inode->i_blocks = CIFS_INO_BLOCKS(asize);
+ } else {
+ cifsi->time = 0;
+ }
+ spin_unlock(&inode->i_lock);
+ goto out;
+ }
+
if (cifsi->cifsAttrs & FILE_ATTRIBUTE_SPARSE_FILE)
smb2_set_sparse(xid, tcon, cfile, inode, false);
new_eof = off + len;
+
+ qrc = SMB2_query_info(xid, tcon,
+ cfile->fid.persistent_fid,
+ cfile->fid.volatile_fid, &file_inf);
+ if (qrc == 0)
+ asize = le64_to_cpu(file_inf.AllocationSize);
+
+ /*
+ * FILE_ALLOCATION_INFORMATION can only describe allocation up to
+ * new_eof. Some servers may accept it without allocating blocks,
+ * so refresh AllocationSize before updating i_blocks.
+ */
+ if (off == 0 || off == old_eof) {
+ if (qrc || asize < new_eof) {
+ rc = SMB2_set_allocation(xid, tcon,
+ cfile->fid.persistent_fid,
+ cfile->fid.volatile_fid,
+ cfile->pid, new_eof);
+ if (rc)
+ goto out;
+ }
+ }
+
rc = SMB2_set_eof(xid, tcon, cfile->fid.persistent_fid,
cfile->fid.volatile_fid, cfile->pid, new_eof);
- if (rc == 0) {
- netfs_resize_file(&cifsi->netfs, new_eof, true);
- cifs_setsize(inode, new_eof);
+ if (rc)
+ goto out;
+
+ netfs_resize_file(&cifsi->netfs, new_eof, true);
+ cifs_setsize(inode, new_eof);
+
+ qrc = SMB2_query_info(xid, tcon,
+ cfile->fid.persistent_fid,
+ cfile->fid.volatile_fid, &file_inf);
+ spin_lock(&inode->i_lock);
+ if (qrc == 0) {
+ asize = le64_to_cpu(file_inf.AllocationSize);
+ if (asize >= new_eof)
+ inode->i_blocks = CIFS_INO_BLOCKS(asize);
+ } else {
+ cifsi->time = 0;
}
+ spin_unlock(&inode->i_lock);
goto out;
}
{
struct TCP_Server_Info *server = tcon->ses->server;
struct cifs_open_parms oparms;
- struct cifs_open_info_data idata;
+ struct cifs_open_info_data idata = {};
struct cifs_io_parms io_parms = {};
struct cifs_sb_info *cifs_sb = CIFS_SB(inode->i_sb);
struct cifs_fid fid;
data = (u8 *)symname_utf16;
break;
case S_IFSOCK:
- type_len = 8;
- strscpy(type, "LnxSOCK");
- data = (u8 *)&pdev;
- data_len = sizeof(pdev);
+ /* SFU socket is system file with one zero byte */
+ type_len = 1;
+ type[0] = '\0';
break;
case S_IFIFO:
- type_len = 8;
- strscpy(type, "LnxFIFO");
- data = (u8 *)&pdev;
- data_len = sizeof(pdev);
+ /* SFU fifo is system file which is empty */
+ type_len = 0;
break;
default:
rc = -EPERM;
int
SMB2_open(const unsigned int xid, struct cifs_open_parms *oparms, __le16 *path,
- __u8 *oplock, struct smb2_file_all_info *buf,
+ __u8 *oplock, struct cifs_open_info_data *buf,
struct create_posix_rsp *posix,
struct kvec *err_iov, int *buftype)
{
int rc = 0;
int flags = 0;
int retries = 0, cur_sleep = 0;
+ struct smb2_file_all_info *file_info = buf ? &buf->fi : NULL;
replay_again:
/* reinitialize for possible replay */
oparms->fid->mid = le64_to_cpu(rsp->hdr.MessageId);
#endif /* CIFS_DEBUG2 */
- if (buf) {
- buf->CreationTime = rsp->CreationTime;
- buf->LastAccessTime = rsp->LastAccessTime;
- buf->LastWriteTime = rsp->LastWriteTime;
- buf->ChangeTime = rsp->ChangeTime;
- buf->AllocationSize = rsp->AllocationSize;
- buf->EndOfFile = rsp->EndofFile;
- buf->Attributes = rsp->FileAttributes;
- buf->NumberOfLinks = cpu_to_le32(1);
- buf->DeletePending = 0; /* successful open = not delete pending */
+ if (file_info) {
+ file_info->CreationTime = rsp->CreationTime;
+ file_info->LastAccessTime = rsp->LastAccessTime;
+ file_info->LastWriteTime = rsp->LastWriteTime;
+ file_info->ChangeTime = rsp->ChangeTime;
+ file_info->AllocationSize = rsp->AllocationSize;
+ file_info->EndOfFile = rsp->EndofFile;
+ file_info->Attributes = rsp->FileAttributes;
+ file_info->NumberOfLinks = cpu_to_le32(1);
+ buf->unknown_nlink = true;
+ file_info->DeletePending = 0; /* successful open = not delete pending */
}
rc = smb2_parse_contexts(server, &rsp_iov, &oparms->fid->epoch,
- oparms->fid->lease_key, oplock, buf, posix);
+ oparms->fid->lease_key, oplock, file_info, posix);
trace_smb3_open_done(xid, rsp->PersistentFileId, tcon->tid, ses->Suid,
oparms->create_options, oparms->desired_access,
0, 1, &data, &size);
}
+int
+SMB2_set_allocation(const unsigned int xid, struct cifs_tcon *tcon,
+ u64 persistent_fid, u64 volatile_fid, u32 pid,
+ loff_t allocation_size)
+{
+ struct smb2_file_alloc_info info;
+ void *data;
+ unsigned int size;
+
+ info.AllocationSize = cpu_to_le64(allocation_size);
+
+ data = &info;
+ size = sizeof(struct smb2_file_alloc_info);
+
+ return send_set_info(xid, tcon, persistent_fid, volatile_fid,
+ pid, FILE_ALLOCATION_INFORMATION, SMB2_O_INFO_FILE,
+ 0, 1, &data, &size);
+}
+
int
SMB2_set_acl(const unsigned int xid, struct cifs_tcon *tcon,
u64 persistent_fid, u64 volatile_fid,
struct cifs_tcon *tcon, const struct nls_table *cp);
int SMB2_tdis(const unsigned int xid, struct cifs_tcon *tcon);
int SMB2_open(const unsigned int xid, struct cifs_open_parms *oparms,
- __le16 *path, __u8 *oplock, struct smb2_file_all_info *buf,
+ __le16 *path, __u8 *oplock, struct cifs_open_info_data *buf,
struct create_posix_rsp *posix, struct kvec *err_iov,
int *buftype);
int SMB2_open_init(struct cifs_tcon *tcon, struct TCP_Server_Info *server,
int SMB2_set_eof(const unsigned int xid, struct cifs_tcon *tcon,
u64 persistent_fid, u64 volatile_fid, u32 pid,
loff_t new_eof);
+int SMB2_set_allocation(const unsigned int xid, struct cifs_tcon *tcon,
+ u64 persistent_fid, u64 volatile_fid, u32 pid,
+ loff_t allocation_size);
int SMB2_set_info_init(struct cifs_tcon *tcon, struct TCP_Server_Info *server,
struct smb_rqst *rqst, u64 persistent_fid,
u64 volatile_fid, u32 pid, u8 info_class, u8 info_type,
__le64 EndOfFile; /* new end of file value */
} __packed; /* level 20 Set */
+/* See MS-FSCC 2.4.4 */
+struct smb2_file_alloc_info { /* encoding of request for level 19 */
+ __le64 AllocationSize;
+} __packed;
+
/* See MS-FSCC 2.4.15 */
typedef struct {
__le32 NextEntryOffset;
* @blen: NTLMv2 blob length
* @domain_name: domain name
* @cryptkey: session crypto key
+ * @sess_key: derived session key output buffer
*
* Return: 0 on success, error number on error
*/
int ksmbd_auth_ntlmv2(struct ksmbd_conn *conn, struct ksmbd_session *sess,
struct ntlmv2_resp *ntlmv2, int blen, char *domain_name,
- char *cryptkey)
+ char *cryptkey, char *sess_key)
{
char ntlmv2_hash[CIFS_ENCPWD_SIZE];
char ntlmv2_rsp[CIFS_HMAC_MD5_HASH_SIZE];
- char sess_key[SMB2_NTLMV2_SESSKEY_SIZE];
+ char base_key[SMB2_NTLMV2_SESSKEY_SIZE];
struct hmac_md5_ctx ctx;
int rc;
/* Generate the session key */
hmac_md5_usingrawkey(ntlmv2_hash, CIFS_HMAC_MD5_HASH_SIZE,
ntlmv2_rsp, CIFS_HMAC_MD5_HASH_SIZE,
- sess_key);
+ base_key);
if (crypto_memneq(ntlmv2->ntlmv2_hash, ntlmv2_rsp,
CIFS_HMAC_MD5_HASH_SIZE)) {
goto out;
}
- memcpy(sess->sess_key, sess_key, sizeof(sess_key));
+ memcpy(sess_key, base_key, sizeof(base_key));
rc = 0;
out:
memzero_explicit(ntlmv2_hash, sizeof(ntlmv2_hash));
memzero_explicit(ntlmv2_rsp, sizeof(ntlmv2_rsp));
- memzero_explicit(sess_key, sizeof(sess_key));
+ memzero_explicit(base_key, sizeof(base_key));
return rc;
}
* @blob_len: length of the @authblob message
* @conn: connection
* @sess: session of connection
+ * @sess_key: derived session key output buffer
*
* Return: 0 on success, error number on error
*/
int ksmbd_decode_ntlmssp_auth_blob(struct authenticate_message *authblob,
int blob_len, struct ksmbd_conn *conn,
- struct ksmbd_session *sess)
+ struct ksmbd_session *sess, char *sess_key)
{
char *domain_name;
unsigned int nt_off, dn_off;
ret = ksmbd_auth_ntlmv2(conn, sess,
(struct ntlmv2_resp *)((char *)authblob + nt_off),
nt_len - CIFS_ENCPWD_SIZE,
- domain_name, conn->ntlmssp.cryptkey);
+ domain_name, conn->ntlmssp.cryptkey, sess_key);
kfree(domain_name);
if (ret)
return ret;
if (!ctx_arc4)
return -ENOMEM;
- arc4_setkey(ctx_arc4, sess->sess_key, SMB2_NTLMV2_SESSKEY_SIZE);
- arc4_crypt(ctx_arc4, sess->sess_key,
+ arc4_setkey(ctx_arc4, sess_key, SMB2_NTLMV2_SESSKEY_SIZE);
+ arc4_crypt(ctx_arc4, sess_key,
(char *)authblob + sess_key_off, sess_key_len);
kfree_sensitive(ctx_arc4);
}
#ifdef CONFIG_SMB_SERVER_KERBEROS5
int ksmbd_krb5_authenticate(struct ksmbd_session *sess, char *in_blob,
- int in_len, char *out_blob, int *out_len)
+ int in_len, char *out_blob, int *out_len,
+ char *sess_key)
{
struct ksmbd_spnego_authen_response *resp;
struct ksmbd_login_response_ext *resp_ext = NULL;
} else {
if (!ksmbd_compare_user(sess->user, user)) {
ksmbd_debug(AUTH, "different user tried to reuse session\n");
- retval = -EPERM;
+ retval = -EKEYREJECTED;
ksmbd_free_user(user);
goto out;
}
ksmbd_free_user(user);
}
- memcpy(sess->sess_key, resp->payload, resp->session_key_len);
+ memcpy(sess_key, resp->payload, resp->session_key_len);
memcpy(out_blob, resp->payload + resp->session_key_len,
resp->spnego_blob_len);
*out_len = resp->spnego_blob_len;
}
#else
int ksmbd_krb5_authenticate(struct ksmbd_session *sess, char *in_blob,
- int in_len, char *out_blob, int *out_len)
+ int in_len, char *out_blob, int *out_len,
+ char *sess_key)
{
return -EOPNOTSUPP;
}
bool binding;
};
-static void generate_key(struct ksmbd_conn *conn, struct ksmbd_session *sess,
+static void generate_key(struct ksmbd_conn *conn, const char *sess_key,
struct kvec label, struct kvec context, __u8 *key,
unsigned int key_size)
{
unsigned char prfhash[SMB2_HMACSHA256_SIZE];
struct hmac_sha256_ctx ctx;
- hmac_sha256_init_usingrawkey(&ctx, sess->sess_key,
+ hmac_sha256_init_usingrawkey(&ctx, sess_key,
SMB2_NTLMV2_SESSKEY_SIZE);
hmac_sha256_update(&ctx, i, 4);
hmac_sha256_update(&ctx, label.iov_base, label.iov_len);
const struct derivation *signing)
{
struct channel *chann;
- char *key;
+ char *key, *sess_key;
chann = lookup_chann_list(sess, conn);
if (!chann)
return 0;
- if (conn->dialect >= SMB30_PROT_ID && signing->binding)
+ if (conn->dialect >= SMB30_PROT_ID && signing->binding) {
key = chann->smb3signingkey;
- else
+ sess_key = chann->sess_key;
+ } else {
key = sess->smb3signingkey;
+ sess_key = sess->sess_key;
+ }
- generate_key(conn, sess, signing->label, signing->context, key,
+ generate_key(conn, sess_key, signing->label, signing->context, key,
SMB3_SIGN_KEY_SIZE);
if (!(conn->dialect >= SMB30_PROT_ID && signing->binding))
struct ksmbd_session *sess,
const struct derivation_twin *ptwin)
{
- generate_key(conn, sess, ptwin->encryption.label,
+ generate_key(conn, sess->sess_key, ptwin->encryption.label,
ptwin->encryption.context, sess->smb3encryptionkey,
SMB3_ENC_DEC_KEY_SIZE);
- generate_key(conn, sess, ptwin->decryption.label,
+ generate_key(conn, sess->sess_key, ptwin->decryption.label,
ptwin->decryption.context,
sess->smb3decryptionkey, SMB3_ENC_DEC_KEY_SIZE);
void ksmbd_copy_gss_neg_header(void *buf);
int ksmbd_auth_ntlmv2(struct ksmbd_conn *conn, struct ksmbd_session *sess,
struct ntlmv2_resp *ntlmv2, int blen, char *domain_name,
- char *cryptkey);
+ char *cryptkey, char *sess_key);
int ksmbd_decode_ntlmssp_auth_blob(struct authenticate_message *authblob,
int blob_len, struct ksmbd_conn *conn,
- struct ksmbd_session *sess);
+ struct ksmbd_session *sess, char *sess_key);
int ksmbd_decode_ntlmssp_neg_blob(struct negotiate_message *negblob,
int blob_len, struct ksmbd_conn *conn);
unsigned int
ksmbd_build_ntlmssp_challenge_blob(struct challenge_message *chgblob,
struct ksmbd_conn *conn);
int ksmbd_krb5_authenticate(struct ksmbd_session *sess, char *in_blob,
- int in_len, char *out_blob, int *out_len);
+ int in_len, char *out_blob, int *out_len,
+ char *sess_key);
void ksmbd_sign_smb2_pdu(struct ksmbd_conn *conn, char *key, struct kvec *iov,
int n_vec, char *sig);
void ksmbd_sign_smb3_pdu(struct ksmbd_conn *conn, char *key, struct kvec *iov,
down_write(&sess->chann_lock);
xa_for_each(&sess->ksmbd_chann_list, index, chann) {
xa_erase(&sess->ksmbd_chann_list, index);
- kfree(chann);
+ kfree_sensitive(chann);
}
xa_destroy(&sess->ksmbd_chann_list);
if (!chann)
return -ENOENT;
- kfree(chann);
+ kfree_sensitive(chann);
return 0;
}
{
struct ksmbd_session *sess;
unsigned long id;
+ struct hlist_node *tmp;
+ int bkt;
down_write(&sessions_table_lock);
- if (conn->binding) {
- int bkt;
- struct hlist_node *tmp;
-
- hash_for_each_safe(sessions_table, bkt, tmp, sess, hlist) {
- if (!ksmbd_chann_del(conn, sess) &&
- xa_empty(&sess->ksmbd_chann_list)) {
- hash_del(&sess->hlist);
- down_write(&conn->session_lock);
- xa_erase(&conn->sessions, sess->id);
- up_write(&conn->session_lock);
- if (atomic_dec_and_test(&sess->refcnt))
- ksmbd_session_destroy(sess);
- }
+ hash_for_each_safe(sessions_table, bkt, tmp, sess, hlist) {
+ if (!ksmbd_chann_del(conn, sess) &&
+ xa_empty(&sess->ksmbd_chann_list)) {
+ hash_del(&sess->hlist);
+ down_write(&conn->session_lock);
+ xa_erase(&conn->sessions, sess->id);
+ up_write(&conn->session_lock);
+ if (atomic_dec_and_test(&sess->refcnt))
+ ksmbd_session_destroy(sess);
}
}
struct ksmbd_file_table;
struct channel {
+ char sess_key[CIFS_KEY_SIZE];
__u8 smb3signingkey[SMB3_SIGN_KEY_SIZE];
struct ksmbd_conn *conn;
};
out:
ksmbd_free_work_struct(work);
ksmbd_conn_r_count_dec(conn);
+ ksmbd_conn_put(conn);
}
/**
br_info->open_trunc = opinfo->open_trunc;
work->request_buf = (char *)br_info;
- work->conn = conn;
+ work->conn = ksmbd_conn_get(conn);
work->sess = opinfo->sess;
ksmbd_conn_r_count_inc(conn);
out:
ksmbd_free_work_struct(work);
ksmbd_conn_r_count_dec(conn);
+ ksmbd_conn_put(conn);
}
/**
memcpy(br_info->lease_key, lease->lease_key, SMB2_LEASE_KEY_SIZE);
work->request_buf = (char *)br_info;
- work->conn = conn;
+ work->conn = ksmbd_conn_get(conn);
work->sess = opinfo->sess;
ksmbd_conn_r_count_inc(conn);
else
conn->ops->set_rsp_status(work,
STATUS_USER_SESSION_DELETED);
+ if (conn->ops->is_sign_req(work, conn->ops->get_cmd_val(work))) {
+ struct smb2_hdr *rsp_hdr;
+
+ rsp_hdr = ksmbd_resp_buf_curr(work);
+ rsp_hdr->Flags |= SMB2_FLAGS_SIGNED;
+ }
goto send;
} else if (rc > 0) {
rc = conn->ops->get_ksmbd_tcon(work);
if (work->sess &&
(work->sess->sign || smb3_11_final_sess_setup_resp(work) ||
- conn->ops->is_sign_req(work, command)))
- conn->ops->set_sign_rsp(work);
+ conn->ops->is_sign_req(work, command))) {
+ if (command == SMB2_SESSION_SETUP_HE &&
+ work->sess->dialect >= SMB30_PROT_ID &&
+ conn->dialect < SMB30_PROT_ID)
+ smb3_set_sign_rsp(work);
+ else
+ conn->ops->set_sign_rsp(work);
+ }
} while (is_chained == true);
send:
return 1;
}
+ if (len < __SMB2_HEADER_STRUCTURE_SIZE + sizeof(__le16)) {
+ ksmbd_debug(SMB, "Message is too small for StructureSize2\n");
+ return 1;
+ }
+
if (smb2_req_struct_sizes[command] != pdu->StructureSize2) {
if (!(command == SMB2_OPLOCK_BREAK_HE &&
(le16_to_cpu(pdu->StructureSize2) == OP_BREAK_STRUCT_SIZE_20 ||
(FILE_ATTRIBUTE_MASK & ~(FILE_ATTRIBUTE_INTEGRITY_STREAM | \
FILE_ATTRIBUTE_NO_SCRUB_DATA))
+/* Windows reports automatic write-time updates at roughly 15 ms resolution. */
+#define KSMBD_WRITE_TIME_RESOLUTION (15ULL * 10000)
+
/**
* check_session_id() - check for valid session id in smb header
* @conn: connection instance
return chann;
}
+#define KSMBD_MAX_CHANNELS 32
+
+static int register_session_channel(struct ksmbd_session *sess,
+ struct ksmbd_conn *conn,
+ const char *sess_key)
+{
+ struct channel *chann, *old;
+ unsigned long index;
+ unsigned int count = 0;
+ int rc = 0;
+
+ down_write(&sess->chann_lock);
+ if (xa_load(&sess->ksmbd_chann_list, (long)conn))
+ goto out;
+
+ xa_for_each(&sess->ksmbd_chann_list, index, chann)
+ count++;
+ if (count >= KSMBD_MAX_CHANNELS) {
+ rc = -ENOSPC;
+ goto out;
+ }
+
+ chann = kmalloc_obj(struct channel, KSMBD_DEFAULT_GFP);
+ if (!chann) {
+ rc = -ENOMEM;
+ goto out;
+ }
+
+ chann->conn = conn;
+ memcpy(chann->sess_key, sess_key, sizeof(chann->sess_key));
+ old = xa_store(&sess->ksmbd_chann_list, (long)conn, chann,
+ KSMBD_DEFAULT_GFP);
+ if (xa_is_err(old)) {
+ kfree_sensitive(chann);
+ rc = xa_err(old);
+ }
+out:
+ up_write(&sess->chann_lock);
+ return rc;
+}
+
/**
* smb2_get_ksmbd_tcon() - get tree connection information using a tree id.
* @work: smb work
{
struct ksmbd_conn *conn = work->conn;
struct ksmbd_session *sess = work->sess;
- struct channel *chann = NULL, *old;
struct ksmbd_user *user;
+ char channel_key[CIFS_KEY_SIZE] = {};
+ char *auth_key = conn->binding ? channel_key : sess->sess_key;
u64 prev_id;
+ bool binding = conn->binding;
int sz, rc;
ksmbd_debug(SMB, "authenticate phase\n");
if (!ksmbd_compare_user(sess->user, user)) {
ksmbd_free_user(user);
- return -EPERM;
+ return -EKEYREJECTED;
}
ksmbd_free_user(user);
} else {
sz = conn->mechTokenLen;
else
sz = le16_to_cpu(req->SecurityBufferLength);
- rc = ksmbd_decode_ntlmssp_auth_blob(authblob, sz, conn, sess);
+ rc = ksmbd_decode_ntlmssp_auth_blob(authblob, sz, conn, sess,
+ auth_key);
if (rc) {
set_user_flag(sess->user, KSMBD_USER_FLAG_BAD_PASSWORD);
ksmbd_debug(SMB, "authentication failed\n");
- return -EPERM;
+ rc = -EPERM;
+ goto out;
}
}
binding_session:
if (conn->dialect >= SMB30_PROT_ID) {
- chann = lookup_chann_list(sess, conn);
- if (!chann) {
- chann = kmalloc_obj(struct channel, KSMBD_DEFAULT_GFP);
- if (!chann)
- return -ENOMEM;
-
- chann->conn = conn;
- down_write(&sess->chann_lock);
- old = xa_store(&sess->ksmbd_chann_list, (long)conn, chann,
- KSMBD_DEFAULT_GFP);
- up_write(&sess->chann_lock);
- if (xa_is_err(old)) {
- kfree(chann);
- return xa_err(old);
- }
- }
+ rc = register_session_channel(sess, conn, auth_key);
+ if (rc)
+ goto out;
}
if (conn->ops->generate_signingkey) {
rc = conn->ops->generate_signingkey(sess, conn);
if (rc) {
ksmbd_debug(SMB, "SMB3 signing key generation failed\n");
- return -EINVAL;
+ rc = -EINVAL;
+ goto out;
}
}
if (!ksmbd_conn_lookup_dialect(conn)) {
pr_err("fail to verify the dialect\n");
- return -ENOENT;
+ rc = -ENOENT;
+ goto out;
}
- return 0;
+ rc = 0;
+out:
+ if (binding)
+ memzero_explicit(channel_key, sizeof(channel_key));
+ return rc;
}
#ifdef CONFIG_SMB_SERVER_KERBEROS5
struct ksmbd_conn *conn = work->conn;
struct ksmbd_session *sess = work->sess;
char *in_blob, *out_blob;
- struct channel *chann = NULL, *old;
+ char channel_key[CIFS_KEY_SIZE] = {};
+ char *auth_key = conn->binding ? channel_key : sess->sess_key;
u64 prev_sess_id;
+ bool binding = conn->binding;
int in_len, out_len;
int retval;
(le16_to_cpu(rsp->SecurityBufferOffset) + 4);
retval = ksmbd_krb5_authenticate(sess, in_blob, in_len,
- out_blob, &out_len);
+ out_blob, &out_len, auth_key);
if (retval) {
ksmbd_debug(SMB, "krb5 authentication failed\n");
- return -EINVAL;
+ if (retval != -EKEYREJECTED)
+ retval = -EINVAL;
+ goto out;
}
/* Check previous session */
binding_session:
if (conn->dialect >= SMB30_PROT_ID) {
- chann = lookup_chann_list(sess, conn);
- if (!chann) {
- chann = kmalloc_obj(struct channel, KSMBD_DEFAULT_GFP);
- if (!chann)
- return -ENOMEM;
-
- chann->conn = conn;
- down_write(&sess->chann_lock);
- old = xa_store(&sess->ksmbd_chann_list, (long)conn,
- chann, KSMBD_DEFAULT_GFP);
- up_write(&sess->chann_lock);
- if (xa_is_err(old)) {
- kfree(chann);
- return xa_err(old);
- }
- }
+ retval = register_session_channel(sess, conn, auth_key);
+ if (retval)
+ goto out;
}
if (conn->ops->generate_signingkey) {
retval = conn->ops->generate_signingkey(sess, conn);
if (retval) {
ksmbd_debug(SMB, "SMB3 signing key generation failed\n");
- return -EINVAL;
+ retval = -EINVAL;
+ goto out;
}
}
if (!ksmbd_conn_lookup_dialect(conn)) {
pr_err("fail to verify the dialect\n");
- return -ENOENT;
+ retval = -ENOENT;
+ goto out;
}
- return 0;
+ retval = 0;
+out:
+ if (binding)
+ memzero_explicit(channel_key, sizeof(channel_key));
+ return retval;
}
#else
static int krb5_authenticate(struct ksmbd_work *work,
} else if ((conn->dialect < SMB30_PROT_ID ||
server_conf.flags & KSMBD_GLOBAL_FLAG_SMB3_MULTICHANNEL) &&
(req->Flags & SMB2_SESSION_REQ_FLAG_BINDING)) {
- sess = NULL;
+ sess = ksmbd_session_lookup_slowpath(le64_to_cpu(req->hdr.SessionId));
+ if (sess) {
+ int sign_ret;
+
+ work->sess = sess;
+ if (sess->dialect >= SMB30_PROT_ID)
+ sign_ret = smb3_check_sign_req(work);
+ else
+ sign_ret = smb2_check_sign_req(work);
+ if (sess->state != SMB2_SESSION_VALID ||
+ !(req->hdr.Flags & SMB2_FLAGS_SIGNED) ||
+ !sign_ret) {
+ ksmbd_user_session_put(sess);
+ work->sess = NULL;
+ sess = NULL;
+ }
+ }
rc = -EACCES;
goto out_err;
} else {
sess = ksmbd_session_lookup(conn,
le64_to_cpu(req->hdr.SessionId));
+ if (!sess) {
+ sess = ksmbd_session_lookup_slowpath(le64_to_cpu(req->hdr.SessionId));
+ if (sess && !lookup_chann_list(sess, conn)) {
+ ksmbd_user_session_put(sess);
+ sess = NULL;
+ }
+ }
if (!sess) {
rc = -ENOENT;
goto out_err;
rsp->hdr.Status = STATUS_REQUEST_NOT_ACCEPTED;
else if (rc == -EFAULT)
rsp->hdr.Status = STATUS_NETWORK_SESSION_EXPIRED;
- else if (rc == -ENOMEM)
+ else if (rc == -ENOMEM || rc == -ENOSPC)
rsp->hdr.Status = STATUS_INSUFFICIENT_RESOURCES;
else if (rc == -EOPNOTSUPP)
rsp->hdr.Status = STATUS_NOT_SUPPORTED;
+ else if (rc == -EKEYREJECTED)
+ rsp->hdr.Status = STATUS_ACCESS_DENIED;
else if (rc)
rsp->hdr.Status = STATUS_LOGON_FAILURE;
+ if ((rsp->hdr.Status == STATUS_USER_SESSION_DELETED ||
+ (rsp->hdr.Status == STATUS_INVALID_PARAMETER &&
+ (req->Flags & SMB2_SESSION_REQ_FLAG_BINDING))) &&
+ (req->hdr.Flags & SMB2_FLAGS_SIGNED))
+ rsp->hdr.Flags |= SMB2_FLAGS_SIGNED;
if (conn->mechToken) {
kfree(conn->mechToken);
}
if (rc < 0) {
+ if (sess && conn->dialect == SMB311_PROT_ID &&
+ (req->Flags & SMB2_SESSION_REQ_FLAG_BINDING)) {
+ struct preauth_session *preauth_sess;
+
+ preauth_sess = ksmbd_preauth_session_lookup(conn, sess->id);
+ if (preauth_sess) {
+ list_del(&preauth_sess->preauth_entry);
+ kfree(preauth_sess);
+ }
+ }
+
/*
* SecurityBufferOffset should be set to zero
* in session setup error response.
sess->last_active = jiffies;
sess->state = SMB2_SESSION_EXPIRED;
}
- ksmbd_user_session_put(sess);
- work->sess = NULL;
+ /*
+ * Keep the binding session reference until the response is
+ * signed and sent. Error responses for a signed binding
+ * request are signed with the existing session signing key.
+ */
+ if (!(req->Flags & SMB2_SESSION_REQ_FLAG_BINDING) ||
+ work->sess != sess) {
+ ksmbd_user_session_put(sess);
+ work->sess = NULL;
+ }
if (try_delay) {
ksmbd_conn_set_need_reconnect(conn);
ssleep(5);
time = ksmbd_UnixTimeToNT(stat.atime);
rsp->LastAccessTime = cpu_to_le64(time);
time = ksmbd_UnixTimeToNT(stat.mtime);
+ fp->open_mtime = time;
rsp->LastWriteTime = cpu_to_le64(time);
rsp->ChangeTime = cpu_to_le64(fp->change_time);
/*
time = ksmbd_UnixTimeToNT(stat.atime);
rsp->LastAccessTime = cpu_to_le64(time);
time = ksmbd_UnixTimeToNT(stat.mtime);
+ if (time > fp->open_mtime &&
+ time - fp->open_mtime < KSMBD_WRITE_TIME_RESOLUTION)
+ time = fp->open_mtime;
rsp->LastWriteTime = cpu_to_le64(time);
rsp->ChangeTime = cpu_to_le64(fp->change_time);
ksmbd_fd_put(work, fp);
pr_err("failed to store stream name in xattr: %d\n",
rc);
rc = -EINVAL;
- goto out;
}
-
+ kfree(xattr_stream_name);
goto out;
}
*/
loff_t alloc_blks;
+ u64 alloc_size;
struct inode *inode;
struct kstat stat;
int rc;
if (rc)
return rc;
- alloc_blks = (le64_to_cpu(file_alloc_info->AllocationSize) + 511) >> 9;
+ /*
+ * AllocationSize is fully client-controlled (the caller only
+ * validates the fixed 8-byte buffer length). Reject values that
+ * would overflow the "round up to 512-byte blocks" conversion
+ * below instead of silently wrapping it to a tiny block count,
+ * which would truncate the file to a size the client never
+ * asked for.
+ */
+ alloc_size = le64_to_cpu(file_alloc_info->AllocationSize);
+ if (alloc_size > MAX_LFS_FILESIZE - 511)
+ return -EINVAL;
+
+ alloc_blks = (alloc_size + 511) >> 9;
inode = file_inode(fp->filp);
if (alloc_blks > stat.blocks) {
rsp->hdr.Status = STATUS_INVALID_PARAMETER;
else if (rc == -EMSGSIZE)
rsp->hdr.Status = STATUS_INFO_LENGTH_MISMATCH;
+ else if (rc == -ENOSPC || rc == -EFBIG)
+ rsp->hdr.Status = STATUS_DISK_FULL;
else if (rc == -ESHARE)
rsp->hdr.Status = STATUS_SHARING_VIOLATION;
else if (rc == -ENOENT)
goto out;
}
+ /*
+ * ksmbd_vfs_read() fills only nbytes; the [nbytes, ALIGN(nbytes, 8))
+ * tail of the un-zeroed buffer is transmitted as compound-response
+ * alignment padding, leaking uninitialized kernel memory to the
+ * client. Zero just that tail.
+ */
+ if (nbytes & 7)
+ memset(aux_payload_buf + nbytes, 0, ALIGN(nbytes, 8) - nbytes);
+
if ((nbytes == 0 && length != 0) || nbytes < mincount) {
kvfree(aux_payload_buf);
rsp->hdr.Status = STATUS_END_OF_FILE;
if ((rcv_hdr2->Flags & SMB2_FLAGS_SIGNED) &&
command != SMB2_NEGOTIATE_HE &&
- command != SMB2_SESSION_SETUP_HE &&
command != SMB2_OPLOCK_BREAK_HE)
return true;
} else {
chann = lookup_chann_list(work->sess, conn);
if (!chann) {
- return 0;
+ if (le16_to_cpu(hdr->Command) != SMB2_SESSION_SETUP_HE ||
+ !(hdr->Flags & SMB2_FLAGS_SIGNED))
+ return 0;
+ signing_key = work->sess->smb3signingkey;
+ } else {
+ signing_key = chann->smb3signingkey;
}
- signing_key = chann->smb3signingkey;
}
if (!signing_key) {
struct channel *chann;
char signature[SMB2_CMACAES_SIZE];
struct kvec *iov;
+ u16 command = conn->ops->get_cmd_val(work);
int n_vec = 1;
char *signing_key;
hdr = ksmbd_resp_buf_curr(work);
- if (conn->binding == false &&
- le16_to_cpu(hdr->Command) == SMB2_SESSION_SETUP_HE) {
+ if (command == SMB2_SESSION_SETUP_HE &&
+ (!conn->binding || hdr->Status != STATUS_SUCCESS)) {
signing_key = work->sess->smb3signingkey;
} else {
chann = lookup_chann_list(work->sess, work->conn);
}
if (le16_to_cpu(rsp->Command) == SMB2_SESSION_SETUP_HE && sess) {
- __u8 *hash_value;
+ ksmbd_conn_lock(conn);
if (conn->binding) {
struct preauth_session *preauth_sess;
preauth_sess = ksmbd_preauth_session_lookup(conn, sess->id);
- if (!preauth_sess)
- return;
- hash_value = preauth_sess->Preauth_HashValue;
- } else {
- hash_value = sess->Preauth_HashValue;
- if (!hash_value)
- return;
+ if (preauth_sess)
+ ksmbd_gen_preauth_integrity_hash(conn,
+ work->response_buf,
+ preauth_sess->Preauth_HashValue);
+ } else if (sess->Preauth_HashValue) {
+ ksmbd_gen_preauth_integrity_hash(conn, work->response_buf,
+ sess->Preauth_HashValue);
}
- ksmbd_gen_preauth_integrity_hash(conn, work->response_buf,
- hash_value);
+ ksmbd_conn_unlock(conn);
}
}
__le32 EASize;
} __packed;
-struct smb2_file_alloc_info {
- __le64 AllocationSize;
-} __packed;
-
struct smb2_file_disposition_info {
__u8 DeletePending;
} __packed;
struct smb_sid *psid, uint sidtype,
struct smb_fattr *fattr)
{
+ const struct smb_sid *sid_prefix;
int rc = -EINVAL;
/*
kuid_t uid;
uid_t id;
+ /* Only the server domain RID has a local uid representation. */
+ sid_prefix = &server_conf.domain_sid;
+ if (psid->num_subauth != sid_prefix->num_subauth + 1 ||
+ compare_sids(psid, sid_prefix))
+ return -EINVAL;
+
id = le32_to_cpu(psid->sub_auth[psid->num_subauth - 1]);
uid = KUIDT_INIT(id);
uid = from_vfsuid(idmap, &init_user_ns, VFSUIDT_INIT(uid));
kgid_t gid;
gid_t id;
+ /* Local gids are represented by S-1-22-2-<gid>. */
+ sid_prefix = &sid_unix_groups;
+ if (psid->num_subauth != sid_prefix->num_subauth + 1 ||
+ compare_sids(psid, sid_prefix))
+ return -EINVAL;
+
id = le32_to_cpu(psid->sub_auth[psid->num_subauth - 1]);
gid = KGIDT_INIT(id);
gid = from_vfsgid(idmap, &init_user_ns, VFSGIDT_INIT(gid));
rc = sid_to_id(idmap, owner_sid_ptr, SIDOWNER, fattr);
if (rc) {
- pr_err("%s: Error %d mapping Owner SID to uid\n",
- __func__, rc);
+ ksmbd_debug(SMB, "Owner SID has no Unix uid mapping\n");
owner_sid_ptr = NULL;
+ rc = 0;
}
}
}
rc = sid_to_id(idmap, group_sid_ptr, SIDUNIX_GROUP, fattr);
if (rc) {
- pr_err("%s: Error %d mapping Group SID to gid\n",
- __func__, rc);
+ ksmbd_debug(SMB, "Group SID has no Unix gid mapping\n");
group_sid_ptr = NULL;
+ rc = 0;
}
}
__u64 change_time;
__u64 allocation_size;
__u64 itime;
+ __u64 open_mtime;
bool is_nt_open;
bool attrib_only;
if (dfp)
return;
- xfs_defer_alloc(&tp->t_dfops, &xfs_barrier_defer_type);
+ dfp = xfs_defer_alloc(&tp->t_dfops, &xfs_barrier_defer_type);
trace_xfs_defer_add_item(tp->t_mountp, dfp, NULL);
}
#include "xfs_rtalloc.h"
#include "xfs_rtbitmap.h"
#include "xfs_rtgroup.h"
+#include "xfs_bmap_util.h"
#include "scrub/xfs_scrub.h"
#include "scrub/scrub.h"
#include "scrub/common.h"
unsigned int next_bno;
};
-/* CoW staging extent. */
-struct xrep_cow_extent {
- xfs_fsblock_t fsbno;
- xfs_extlen_t len;
-};
-
/*
* Mark the part of the file range that corresponds to the given physical
* space. Caller must ensure that the physical range is within xc->irec.
xfs_gbno_to_fsb(cur->bc_group, rec_bno), rec_len);
}
+/*
+ * Trim the start and end of the current mapping by up to 1/4 of the length
+ * and mark that as "bad" to test the cow fork repair mechanism.
+ */
+static inline int
+xrep_cow_debug_replacement(
+ struct xrep_cow *xc)
+{
+ xfs_fsblock_t fsbno = xc->irec.br_startblock;
+ xfs_extlen_t len = xc->irec.br_blockcount;
+ uint32_t trim;
+
+ /* get_random_u32_below requires a nonzero argument */
+ trim = len > 4 ? get_random_u32_below(len / 4) : 0;
+ len -= trim;
+
+ trim = len > 4 ? get_random_u32_below(len / 4) : 0;
+ fsbno += trim;
+ len -= trim;
+
+ return xrep_cow_mark_file_range(xc, fsbno, len);
+}
+
/*
* Find any part of the CoW fork mapping that isn't a single-owner CoW staging
* extent and mark the corresponding part of the file range in the bitmap.
* If userspace is forcing us to rebuild the CoW fork or someone turned
* on the debugging knob, replace everything in the CoW fork.
*/
- if ((sc->sm->sm_flags & XFS_SCRUB_IFLAG_FORCE_REBUILD) ||
- XFS_TEST_ERROR(sc->mp, XFS_ERRTAG_FORCE_SCRUB_REPAIR))
+ if (XFS_TEST_ERROR(sc->mp, XFS_ERRTAG_FORCE_SCRUB_REPAIR))
+ error = xrep_cow_debug_replacement(xc);
+ else if (sc->sm->sm_flags & XFS_SCRUB_IFLAG_FORCE_REBUILD)
error = xrep_cow_mark_file_range(xc, xc->irec.br_startblock,
xc->irec.br_blockcount);
* turned on the debugging knob, replace everything in the
* CoW fork and then scan for staging extents in the refcountbt.
*/
- if ((sc->sm->sm_flags & XFS_SCRUB_IFLAG_FORCE_REBUILD) ||
- XFS_TEST_ERROR(sc->mp, XFS_ERRTAG_FORCE_SCRUB_REPAIR))
+ if (XFS_TEST_ERROR(sc->mp, XFS_ERRTAG_FORCE_SCRUB_REPAIR))
+ error = xrep_cow_debug_replacement(xc);
+ else if (sc->sm->sm_flags & XFS_SCRUB_IFLAG_FORCE_REBUILD)
error = xrep_cow_mark_file_range(xc, xc->irec.br_startblock,
xc->irec.br_blockcount);
STATIC int
xrep_cow_alloc(
struct xfs_scrub *sc,
- xfs_extlen_t maxlen,
- struct xrep_cow_extent *repl)
+ struct xfs_bmbt_irec *del)
{
struct xfs_alloc_arg args = {
.tp = sc->tp,
.mp = sc->mp,
.oinfo = XFS_RMAP_OINFO_SKIP_UPDATE,
.minlen = 1,
- .maxlen = maxlen,
+ .maxlen = del->br_blockcount,
.prod = 1,
.resv = XFS_AG_RESV_NONE,
.datatype = XFS_ALLOC_USERDATA,
};
int error;
- error = xfs_trans_reserve_more(sc->tp, maxlen, 0);
+ error = xfs_trans_reserve_more(sc->tp, del->br_blockcount, 0);
if (error)
return error;
xfs_refcount_alloc_cow_extent(sc->tp, false, args.fsbno, args.len);
- repl->fsbno = args.fsbno;
- repl->len = args.len;
+ del->br_startblock = args.fsbno;
+ del->br_blockcount = args.len;
return 0;
}
STATIC int
xrep_cow_alloc_rt(
struct xfs_scrub *sc,
- xfs_extlen_t maxlen,
- struct xrep_cow_extent *repl)
+ struct xfs_bmbt_irec *del)
{
- xfs_rtxlen_t maxrtx = xfs_rtb_to_rtx(sc->mp, maxlen);
+ xfs_fsblock_t fsbno;
+ xfs_rtxlen_t maxrtx =
+ min(U32_MAX, xfs_blen_to_rtbxlen(sc->mp, del->br_blockcount));
+ xfs_extlen_t len;
int error;
error = xfs_trans_reserve_more(sc->tp, 0, maxrtx);
return error;
error = xfs_rtallocate_rtgs(sc->tp, NULLRTBLOCK, 1, maxrtx, 1, false,
- false, &repl->fsbno, &repl->len);
+ false, &fsbno, &len);
if (error)
return error;
- xfs_refcount_alloc_cow_extent(sc->tp, true, repl->fsbno, repl->len);
+ xfs_refcount_alloc_cow_extent(sc->tp, true, fsbno, len);
+
+ del->br_startblock = fsbno;
+ del->br_blockcount = len;
return 0;
}
xrep_cow_find_mapping(
struct xrep_cow *xc,
struct xfs_iext_cursor *icur,
- xfs_fileoff_t startoff,
- struct xfs_bmbt_irec *got)
+ xfs_fileoff_t badoff,
+ xfs_extlen_t badlen,
+ struct xfs_bmbt_irec *got,
+ struct xfs_bmbt_irec *rep)
{
struct xfs_inode *ip = xc->sc->ip;
struct xfs_ifork *ifp = xfs_ifork_ptr(ip, XFS_COW_FORK);
- if (!xfs_iext_lookup_extent(ip, ifp, startoff, icur, got))
+ if (!xfs_iext_lookup_extent(ip, ifp, badoff, icur, got))
goto bad;
+ memcpy(rep, got, sizeof(*rep));
- if (got->br_startoff > startoff)
- goto bad;
-
- if (got->br_blockcount == 0)
+ if (got->br_startoff > badoff)
goto bad;
if (isnullstartblock(got->br_startblock))
if (xfs_bmap_is_written_extent(got))
goto bad;
- return 0;
-bad:
- ASSERT(0);
- return -EFSCORRUPTED;
-}
+ if (got->br_startoff < badoff) {
+ const int64_t delta = badoff - got->br_startoff;
-#define REPLACE_LEFT_SIDE (1U << 0)
-#define REPLACE_RIGHT_SIDE (1U << 1)
-
-/*
- * Given a CoW fork mapping @got and a replacement mapping @repl, remap the
- * beginning of @got with the space described by @rep.
- */
-static inline void
-xrep_cow_replace_mapping(
- struct xfs_inode *ip,
- struct xfs_iext_cursor *icur,
- const struct xfs_bmbt_irec *got,
- const struct xrep_cow_extent *repl)
-{
- struct xfs_bmbt_irec new = *got; /* struct copy */
-
- ASSERT(repl->len > 0);
- ASSERT(!isnullstartblock(got->br_startblock));
+ rep->br_blockcount -= delta;
+ rep->br_startoff += delta;
+ rep->br_startblock += delta;
+ }
- trace_xrep_cow_replace_mapping(ip, got, repl->fsbno, repl->len);
+ if (got->br_startoff + got->br_blockcount > badoff + badlen) {
+ const int64_t delta = (got->br_startoff + got->br_blockcount) -
+ (badoff + badlen);
- if (got->br_blockcount == repl->len) {
- /*
- * The new extent is a complete replacement for the existing
- * extent. Update the COW fork record.
- */
- new.br_startblock = repl->fsbno;
- xfs_iext_update_extent(ip, BMAP_COWFORK, icur, &new);
- return;
+ rep->br_blockcount -= delta;
}
- /*
- * The new extent can replace the beginning of the COW fork record.
- * Move the left side of @got upwards, then insert the new record.
- */
- new.br_startoff += repl->len;
- new.br_startblock += repl->len;
- new.br_blockcount -= repl->len;
- xfs_iext_update_extent(ip, BMAP_COWFORK, icur, &new);
-
- new.br_startoff = got->br_startoff;
- new.br_startblock = repl->fsbno;
- new.br_blockcount = repl->len;
- xfs_iext_insert(ip, icur, &new, BMAP_COWFORK);
+ if (got->br_blockcount == 0)
+ goto bad;
+
+ return 0;
+bad:
+ ASSERT(0);
+ return -EFSCORRUPTED;
}
/*
xfs_extlen_t *blockcount)
{
struct xfs_iext_cursor icur;
- struct xrep_cow_extent repl;
- struct xfs_bmbt_irec got;
+ struct xfs_bmbt_irec got, rep;
struct xfs_scrub *sc = xc->sc;
- xfs_fileoff_t nextoff;
- xfs_extlen_t alloc_len;
+ xfs_fsblock_t old_fsbno;
int error;
/*
- * Put the existing CoW fork mapping in @got. If @got ends before
- * @rep, truncate @rep so we only replace one extent mapping at a time.
+ * Put the existing CoW fork mapping in @got, and put in @rep the
+ * contents of @got trimmed to @startoff/@blockcount. We only want
+ * to replace the bad region, and only one mapping at a time.
*/
- error = xrep_cow_find_mapping(xc, &icur, startoff, &got);
+ error = xrep_cow_find_mapping(xc, &icur, startoff, *blockcount, &got,
+ &rep);
if (error)
return error;
- nextoff = min(startoff + *blockcount,
- got.br_startoff + got.br_blockcount);
+ old_fsbno = rep.br_startblock;
/*
* Allocate a replacement extent. If we don't fill all the blocks,
* shorten the quantity that will be deleted in this step.
*/
- alloc_len = min_t(xfs_fileoff_t, XFS_MAX_BMBT_EXTLEN,
- nextoff - startoff);
if (XFS_IS_REALTIME_INODE(sc->ip))
- error = xrep_cow_alloc_rt(sc, alloc_len, &repl);
+ error = xrep_cow_alloc_rt(sc, &rep);
else
- error = xrep_cow_alloc(sc, alloc_len, &repl);
+ error = xrep_cow_alloc(sc, &rep);
if (error)
return error;
* Replace the old mapping with the new one, and commit the metadata
* changes made so far.
*/
- xrep_cow_replace_mapping(sc->ip, &icur, &got, &repl);
+ xfs_bmap_replace_cow_mapping(sc->ip, &icur, &got, &rep);
xfs_inode_set_cowblocks_tag(sc->ip);
error = xfs_defer_finish(&sc->tp);
/* Note the old CoW staging extents; we'll reap them all later. */
if (XFS_IS_REALTIME_INODE(sc->ip))
- error = xrtb_bitmap_set(&xc->old_cowfork_rtblocks,
- got.br_startblock, repl.len);
+ error = xrtb_bitmap_set(&xc->old_cowfork_rtblocks, old_fsbno,
+ rep.br_blockcount);
else
- error = xfsb_bitmap_set(&xc->old_cowfork_fsblocks,
- got.br_startblock, repl.len);
+ error = xfsb_bitmap_set(&xc->old_cowfork_fsblocks, old_fsbno,
+ rep.br_blockcount);
if (error)
return error;
- *blockcount = repl.len;
+ *blockcount = rep.br_blockcount;
return 0;
}
goto out_scanlock;
}
+ /* The handle encoded in the parent pointer must match. */
+ if (VFS_I(dp)->i_generation != be32_to_cpu(dl->pptr_rec.p_gen)) {
+ trace_xchk_dirpath_badgen(dl->sc, dp, path->path_nr,
+ path->nr_steps, &dl->xname, &dl->pptr_rec);
+ error = -EFSCORRUPTED;
+ goto out_scanlock;
+ }
+
/* We've reached the root directory; the path is ok. */
if (parent_ino == dl->root_ino) {
xchk_dirpath_set_outcome(dl, path, XCHK_DIRPATH_OK);
goto out_scanlock;
}
- /* The handle encoded in the parent pointer must match. */
- if (VFS_I(dp)->i_generation != be32_to_cpu(dl->pptr_rec.p_gen)) {
- trace_xchk_dirpath_badgen(dl->sc, dp, path->path_nr,
- path->nr_steps, &dl->xname, &dl->pptr_rec);
- error = -EFSCORRUPTED;
- goto out_scanlock;
- }
-
/* Parent pointer must point up to a directory. */
if (!S_ISDIR(VFS_I(dp)->i_mode)) {
trace_xchk_dirpath_nondir_parent(dl->sc, dp, path->path_nr,
if (error)
return error;
- cursor->id = dq->q_id + 1;
+ cursor->id = (uint64_t)dq->q_id + 1;
*dqpp = dq;
return 1;
}
if (nrecs == 0 || xfs_bmdr_space_calc(nrecs) > dfork_size)
return true;
- if (level == 0 || level >= XFS_BM_MAXLEVELS(sc->mp, whichfork))
+ if (level == 0 || level > XFS_BM_MAXLEVELS(sc->mp, whichfork))
return true;
dmxr = xfs_bmdr_maxrecs(dfork_size, 0);
struct xfs_inode *ip,
struct timespec64 *ts)
{
- ts->tv_nsec = clamp_t(long, ts->tv_nsec, 0, NSEC_PER_SEC);
+ ts->tv_nsec = clamp_t(long, ts->tv_nsec, 0, NSEC_PER_SEC - 1);
*ts = timestamp_truncate(*ts, VFS_I(ip));
}
xchk_setup_rgsuperblock(
struct xfs_scrub *sc)
{
+ if (xchk_need_intent_drain(sc))
+ xchk_fsgates_enable(sc, XCHK_FSGATES_DRAIN);
return xchk_trans_alloc(sc, 0);
}
struct xfs_scrub *sc)
{
xfs_rgnumber_t rgno = sc->sm->sm_agno;
+ unsigned int flags;
int error;
/*
if (!xchk_xref_process_error(sc, 0, 0, &error))
return error;
- error = xchk_rtgroup_lock(sc, &sc->sr, XFS_RTGLOCK_BITMAP_SHARED);
+ if (xfs_has_rtrmapbt(sc->mp))
+ flags = XFS_RTGLOCK_BITMAP | XFS_RTGLOCK_RMAP;
+ else
+ flags = XFS_RTGLOCK_BITMAP_SHARED;
+
+ error = xchk_rtgroup_lock(sc, &sc->sr, flags);
if (error)
return error;
xrep_rgsuperblock(
struct xfs_scrub *sc)
{
+ struct xfs_buf *sb_bp;
+
ASSERT(rtg_rgno(sc->sr.rtg) == 0);
+ sb_bp = xfs_trans_getsb(sc->tp);
xfs_log_sb(sc->tp);
+ xfs_log_rtsb(sc->tp, sb_bp);
return 0;
}
#endif /* CONFIG_XFS_ONLINE_REPAIR */
* the last free extent we saw and the last possible extent in the rt
* group.
*/
- last_rgbno = rtg->rtg_extents * mp->m_sb.sb_rextsize - 1;
+ last_rgbno = rtg->rtg_extents * mp->m_sb.sb_rextsize;
if (rtb->next_free_rgbno < last_rgbno)
xchk_xref_has_rt_owner(sc, rtb->next_free_rgbno,
last_rgbno - rtb->next_free_rgbno);
/* CoW lookup returned a shared extent record? */
if (rc.rc_domain != XFS_REFC_DOMAIN_COW)
- xchk_btree_xref_set_corrupt(sc, sc->sa.refc_cur, 0);
+ xchk_btree_xref_set_corrupt(sc, sc->sr.refc_cur, 0);
/* Must be at least as long as what was passed in */
if (rc.rc_blockcount < len)
return false;
if (irec->rm_flags & XFS_RMAP_UNWRITTEN)
return false;
+ if (irec->rm_owner == XFS_RMAP_OWN_COW ||
+ irec->rm_owner == XFS_RMAP_OWN_FS)
+ return false;
return true;
}
return false;
if (r1->rm_flags != r2->rm_flags)
return false;
+ if (r1->rm_owner == XFS_RMAP_OWN_COW ||
+ r1->rm_owner == XFS_RMAP_OWN_FS)
+ return true;
return r1->rm_offset + r1->rm_blockcount == r2->rm_offset;
}
xfs_rgbno_to_rtb(sc->sr.rtg, irec->rm_startblock),
irec->rm_blockcount);
if (irec->rm_owner == XFS_RMAP_OWN_COW)
- xchk_xref_is_cow_staging(sc, irec->rm_startblock,
+ xchk_xref_is_rt_cow_staging(sc, irec->rm_startblock,
irec->rm_blockcount);
else
xchk_rtrmapbt_xref_rtrefc(sc, irec);
struct xchk_relax {
unsigned long next_resched;
unsigned int resched_nr;
- bool interruptible;
+ bool killable;
};
/* Yield to the scheduler at most 10x per second. */
(struct xchk_relax){ \
.next_resched = XCHK_RELAX_NEXT, \
.resched_nr = 0, \
- .interruptible = true, \
+ .killable = true, \
}
/*
widget->next_resched = XCHK_RELAX_NEXT;
}
- if (widget->interruptible && fatal_signal_pending(current))
+ if (widget->killable && fatal_signal_pending(current))
return -EINTR;
return 0;
__entry->blockcount)
);
-TRACE_EVENT(xrep_cow_replace_mapping,
- TP_PROTO(struct xfs_inode *ip, const struct xfs_bmbt_irec *irec,
- xfs_fsblock_t new_startblock, xfs_extlen_t new_blockcount),
- TP_ARGS(ip, irec, new_startblock, new_blockcount),
- TP_STRUCT__entry(
- __field(dev_t, dev)
- __field(xfs_ino_t, ino)
- __field(xfs_fsblock_t, startblock)
- __field(xfs_fileoff_t, startoff)
- __field(xfs_filblks_t, blockcount)
- __field(xfs_exntst_t, state)
- __field(xfs_fsblock_t, new_startblock)
- __field(xfs_extlen_t, new_blockcount)
- ),
- TP_fast_assign(
- __entry->dev = ip->i_mount->m_super->s_dev;
- __entry->ino = I_INO(ip);
- __entry->startoff = irec->br_startoff;
- __entry->startblock = irec->br_startblock;
- __entry->blockcount = irec->br_blockcount;
- __entry->state = irec->br_state;
- __entry->new_startblock = new_startblock;
- __entry->new_blockcount = new_blockcount;
- ),
- TP_printk("dev %d:%d ino 0x%llx startoff 0x%llx startblock 0x%llx fsbcount 0x%llx state 0x%x new_startblock 0x%llx new_fsbcount 0x%x",
- MAJOR(__entry->dev), MINOR(__entry->dev),
- __entry->ino,
- __entry->startoff,
- __entry->startblock,
- __entry->blockcount,
- __entry->state,
- __entry->new_startblock,
- __entry->new_blockcount)
-);
-
TRACE_EVENT(xrep_cow_free_staging,
TP_PROTO(const struct xfs_perag *pag, xfs_agblock_t agbno,
xfs_extlen_t blockcount),
xfarray_sortinfo_lo(si)[0] = 0;
xfarray_sortinfo_hi(si)[0] = array->nr - 1;
si->relax = INIT_XCHK_RELAX;
- if (flags & XFARRAY_SORT_KILLABLE)
- si->relax.interruptible = false;
+ si->relax.killable = !!(flags & XFARRAY_SORT_KILLABLE);
trace_xfarray_sort(si, nr_bytes);
*infop = si;
xfs_trans_cancel(tp);
goto out_unlock_ilock;
}
+
+/*
+ * Given a CoW fork mapping @got and a replacement mapping @rep, map the space
+ * described by @rep into the cow fork, pushing aside @got as necessary. @icur
+ * must point to iext tree leaf containing @got.
+ */
+void
+xfs_bmap_replace_cow_mapping(
+ struct xfs_inode *ip,
+ struct xfs_iext_cursor *icur,
+ struct xfs_bmbt_irec *got,
+ struct xfs_bmbt_irec *rep)
+{
+ struct xfs_ifork *ifp = xfs_ifork_ptr(ip, XFS_COW_FORK);
+ xfs_fileoff_t rep_endoff =
+ rep->br_startoff + rep->br_blockcount;
+ xfs_fileoff_t got_endoff =
+ got->br_startoff + got->br_blockcount;
+ uint32_t state = BMAP_COWFORK;
+
+ ASSERT(rep->br_blockcount > 0);
+ ASSERT(!isnullstartblock(got->br_startblock));
+ ASSERT(got->br_startoff <= rep->br_startoff);
+ ASSERT(got_endoff >= rep_endoff);
+
+ trace_xfs_bmap_replace_cow_mapping(ip, got, rep);
+
+ if (got->br_startoff == rep->br_startoff)
+ state |= BMAP_LEFT_FILLING;
+ if (got_endoff == rep_endoff)
+ state |= BMAP_RIGHT_FILLING;
+
+ switch (state & (BMAP_LEFT_FILLING | BMAP_RIGHT_FILLING)) {
+ case BMAP_LEFT_FILLING | BMAP_RIGHT_FILLING:
+ /*
+ * Replacement matches the whole mapping, update the record.
+ */
+ xfs_iext_update_extent(ip, state, icur, rep);
+ break;
+ case BMAP_LEFT_FILLING:
+ /*
+ * Replace the first part of the mapping: Update the cursor
+ * position with the new mapping, then add a record with the
+ * tail of the old mapping.
+ */
+ got->br_startoff = rep_endoff;
+ got->br_blockcount -= rep->br_blockcount;
+ got->br_startblock += rep->br_blockcount;
+
+ xfs_iext_update_extent(ip, state, icur, rep);
+ xfs_iext_next(ifp, icur);
+ xfs_iext_insert(ip, icur, got, state);
+ break;
+ case BMAP_RIGHT_FILLING:
+ /*
+ * Replacing the last part of the mapping. Shorten the current
+ * mapping then add a record with the new mapping.
+ */
+ got->br_blockcount -= rep->br_blockcount;
+
+ xfs_iext_update_extent(ip, state, icur, got);
+ xfs_iext_next(ifp, icur);
+ xfs_iext_insert(ip, icur, rep, state);
+ break;
+ case 0:
+ /*
+ * Replacing the middle of the extent. Shorten the current
+ * mapping, add a new record with the new mapping, and add a
+ * second new record with the tail of the old mapping.
+ */
+ got->br_blockcount = rep->br_startoff - got->br_startoff;
+
+ struct xfs_bmbt_irec new = {
+ .br_startoff = rep_endoff,
+ .br_blockcount = got_endoff - rep_endoff,
+ .br_state = got->br_state,
+ .br_startblock = got->br_startblock +
+ rep->br_blockcount +
+ got->br_blockcount,
+ };
+
+ xfs_iext_update_extent(ip, state, icur, got);
+ xfs_iext_next(ifp, icur);
+ xfs_iext_insert(ip, icur, rep, state);
+ xfs_iext_next(ifp, icur);
+ xfs_iext_insert(ip, icur, &new, state);
+ break;
+ }
+}
int xfs_flush_unmap_range(struct xfs_inode *ip, xfs_off_t offset,
xfs_off_t len);
+void xfs_bmap_replace_cow_mapping(struct xfs_inode *ip,
+ struct xfs_iext_cursor *icur, struct xfs_bmbt_irec *got,
+ struct xfs_bmbt_irec *rep);
+
#endif /* __XFS_BMAP_UTIL_H__ */
static void
xlog_cil_push_now(
struct xlog *log,
- xfs_lsn_t push_seq,
+ xfs_csn_t push_seq,
bool async)
{
struct xfs_cil *cil = log->l_cilp;
list_for_each_entry_safe(item, n, &sort_list, ri_list) {
enum xlog_recover_reorder fate = XLOG_REORDER_ITEM_LIST;
+ /* a committed item with no regions has a NULL ri_buf[0] */
+ if (!item->ri_cnt || !item->ri_buf) {
+ xfs_warn(log->l_mp,
+ "%s: committed log item has no regions",
+ __func__);
+ error = -EFSCORRUPTED;
+ break;
+ }
+
item->ri_ops = xlog_find_item_ops(item);
if (!item->ri_ops) {
xfs_warn(log->l_mp,
"%s: unrecognized type of log operation (%d)",
__func__, ITEM_TYPE(item));
- ASSERT(0);
- /*
- * return the remaining items back to the transaction
- * item list so they can be freed in caller.
- */
- if (!list_empty(&sort_list))
- list_splice_init(&sort_list, &trans->r_itemq);
error = -EFSCORRUPTED;
break;
}
}
}
- ASSERT(list_empty(&sort_list));
+ /*
+ * Return the remaining items back to the transaction item list so they
+ * can be freed in caller. This should only happen when we encounter
+ * an error.
+ */
+ if (!list_empty(&sort_list)) {
+ ASSERT(error);
+ list_splice_init(&sort_list, &trans->r_itemq);
+ }
if (!list_empty(&buffer_list))
list_splice(&buffer_list, &trans->r_itemq);
if (!list_empty(&item_list))
struct xfs_mount *mp = ip->i_mount;
struct xfs_trans *tp;
xfs_filblks_t resaligned;
+ unsigned int seq_before = READ_ONCE(ip->i_df.if_seq);
unsigned int dblocks = 0, rblocks = 0;
int nimaps;
int error;
*lockmode = XFS_ILOCK_EXCL;
+ /*
+ * The data fork mapping may have changed while we dropped the ILOCK
+ * (a racing O_DIRECT writer under IOLOCK_SHARED can complete a full
+ * CoW cycle including xfs_reflink_end_cow(), which remaps this offset
+ * and drops the refcount of the old shared block). Re-read it so the
+ * shared-status recheck below and the caller's in-place iomap both
+ * operate on the current mapping rather than a stale physical block.
+ */
+ if (seq_before != READ_ONCE(ip->i_df.if_seq)) {
+ nimaps = 1;
+ error = xfs_bmapi_read(ip, imap->br_startoff,
+ imap->br_blockcount, imap, &nimaps, 0);
+ if (error)
+ goto out_trans_cancel;
+ }
+
error = xfs_find_trim_cow_extent(ip, imap, cmap, shared, &found);
if (error || !*shared)
goto out_trans_cancel;
bool found;
do {
+ unsigned int seq_before = READ_ONCE(ip->i_df.if_seq);
+
xfs_iunlock(ip, *lockmode);
*lockmode = 0;
*lockmode = XFS_ILOCK_EXCL;
+ /*
+ * The data fork mapping may have changed while we dropped the
+ * ILOCK (a racing O_DIRECT writer under IOLOCK_SHARED can
+ * complete a full CoW cycle including xfs_reflink_end_cow(),
+ * which remaps this offset and drops the refcount of the old
+ * shared block). Re-read it so the shared-status recheck
+ * below and the caller's in-place iomap both operate on the
+ * current mapping rather than a stale physical block.
+ */
+ if (seq_before != READ_ONCE(ip->i_df.if_seq)) {
+ nimaps = 1;
+ error = xfs_bmapi_read(ip, imap->br_startoff,
+ imap->br_blockcount, imap, &nimaps, 0);
+ if (error)
+ goto out_trans_cancel;
+ }
+
error = xfs_find_trim_cow_extent(ip, imap, cmap, shared,
&found);
if (error || !*shared)
.default_groups = xfs_zoned_groups,
};
+int
+xfs_zoned_sysfs_init(struct xfs_mount *mp)
+{
+ if (!IS_ENABLED(CONFIG_XFS_RT) || !xfs_has_zoned(mp))
+ return 0;
+
+ return xfs_sysfs_init(&mp->m_zoned_kobj, &xfs_zoned_ktype,
+ &mp->m_kobj, "zoned");
+}
+
+void
+xfs_zoned_sysfs_del(struct xfs_mount *mp)
+{
+ if (IS_ENABLED(CONFIG_XFS_RT) && xfs_has_zoned(mp))
+ xfs_sysfs_del(&mp->m_zoned_kobj);
+}
+
int
xfs_mount_sysfs_init(
struct xfs_mount *mp)
if (error)
goto out_remove_error_dir;
- if (IS_ENABLED(CONFIG_XFS_RT) && xfs_has_zoned(mp)) {
- /* .../xfs/<dev>/zoned/ */
- error = xfs_sysfs_init(&mp->m_zoned_kobj, &xfs_zoned_ktype,
- &mp->m_kobj, "zoned");
- if (error)
- goto out_remove_error_dir;
- }
-
return 0;
out_remove_error_dir:
struct xfs_error_cfg *cfg;
int i, j;
- if (IS_ENABLED(CONFIG_XFS_RT) && xfs_has_zoned(mp))
- xfs_sysfs_del(&mp->m_zoned_kobj);
-
for (i = 0; i < XFS_ERR_CLASS_MAX; i++) {
for (j = 0; j < XFS_ERR_ERRNO_MAX; j++) {
cfg = &mp->m_error_cfg[i][j];
}
int xfs_mount_sysfs_init(struct xfs_mount *mp);
+int xfs_zoned_sysfs_init(struct xfs_mount *mp);
+void xfs_zoned_sysfs_del(struct xfs_mount *mp);
void xfs_mount_sysfs_del(struct xfs_mount *mp);
#endif /* __XFS_SYSFS_H__ */
__entry->error)
);
+TRACE_EVENT(xfs_bmap_replace_cow_mapping,
+ TP_PROTO(struct xfs_inode *ip, const struct xfs_bmbt_irec *got,
+ const struct xfs_bmbt_irec *rep),
+ TP_ARGS(ip, got, rep),
+ TP_STRUCT__entry(
+ __field(dev_t, dev)
+ __field(xfs_ino_t, ino)
+ __field(xfs_fsblock_t, startblock)
+ __field(xfs_fileoff_t, startoff)
+ __field(xfs_filblks_t, blockcount)
+ __field(xfs_exntst_t, state)
+ __field(xfs_fileoff_t, new_startoff)
+ __field(xfs_fsblock_t, new_startblock)
+ __field(xfs_extlen_t, new_blockcount)
+ __field(xfs_exntst_t, new_state)
+ ),
+ TP_fast_assign(
+ __entry->dev = ip->i_mount->m_super->s_dev;
+ __entry->ino = I_INO(ip);
+ __entry->startoff = got->br_startoff;
+ __entry->startblock = got->br_startblock;
+ __entry->blockcount = got->br_blockcount;
+ __entry->state = got->br_state;
+ __entry->new_startoff = rep->br_startoff;
+ __entry->new_startblock = rep->br_startblock;
+ __entry->new_blockcount = rep->br_blockcount;
+ __entry->new_state = rep->br_state;
+ ),
+ TP_printk("dev %d:%d ino 0x%llx startoff 0x%llx startblock 0x%llx fsbcount 0x%llx state 0x%x new_startoff 0x%llx new_startblock 0x%llx new_fsbcount 0x%x new_state 0x%x",
+ MAJOR(__entry->dev), MINOR(__entry->dev),
+ __entry->ino,
+ __entry->startoff,
+ __entry->startblock,
+ __entry->blockcount,
+ __entry->state,
+ __entry->new_startoff,
+ __entry->new_startblock,
+ __entry->new_blockcount,
+ __entry->new_state)
+);
+
#endif /* _TRACE_XFS_H */
#undef TRACE_INCLUDE_PATH
#include "xfs_rtbitmap.h"
#include "xfs_rtrmap_btree.h"
#include "xfs_zone_alloc.h"
+#include "xfs_sysfs.h"
#include "xfs_zone_priv.h"
#include "xfs_zones.h"
#include "xfs_trace.h"
if (error)
goto out_free_zone_info;
+ error = xfs_zoned_sysfs_init(mp);
+ if (error)
+ goto out_zone_gc_unmount;
+
xfs_info(mp, "%u zones of %u blocks (%u max open zones)",
mp->m_sb.sb_rgcount, iz.zone_capacity, mp->m_max_open_zones);
trace_xfs_zones_mount(mp);
return 0;
+out_zone_gc_unmount:
+ xfs_zone_gc_unmount(mp);
out_free_zone_info:
xfs_free_zone_info(mp->m_zone_info);
return error;
xfs_unmount_zones(
struct xfs_mount *mp)
{
+ xfs_zoned_sysfs_del(mp);
xfs_zone_gc_unmount(mp);
xfs_free_zone_info(mp->m_zone_info);
}
#define drm_exec_for_each_locked_object_reverse(exec, obj) \
__drm_exec_for_each_locked_object_reverse(exec, obj, __UNIQUE_ID(drm_exec))
-/*
- * Helper to drm_exec_until_all_locked(). Don't use directly.
- *
- * Since labels can't be defined local to the loop's body we use a jump pointer
- * to make sure that the retry is only used from within the loop's body.
- */
-#define __drm_exec_until_all_locked(exec, _label) \
-_label: \
- for (void *const __maybe_unused __drm_exec_retry_ptr = &&_label; \
- drm_exec_cleanup(exec);)
-
/**
* drm_exec_until_all_locked - loop until all GEM objects are locked
* @exec: drm_exec object
* Core functionality of the drm_exec object. Loops until all GEM objects are
* locked and no more contention exists. At the beginning of the loop it is
* guaranteed that no GEM object is locked.
+ *
+ * A global label name drm_exec_retry is used, if you need to use more than one
+ * instance of this macro in the same function the label needs to be made local
+ * to the block with the __label__ keyword.
*/
#define drm_exec_until_all_locked(exec) \
- __drm_exec_until_all_locked(exec, __UNIQUE_ID(drm_exec))
+ for (bool const __maybe_unused __drm_exec_loop = false; \
+ drm_exec_cleanup(exec);) \
+ if (false) { \
+drm_exec_retry: __maybe_unused; \
+ continue; \
+ } else
/**
* drm_exec_retry_on_contention - restart the loop to grap all locks
*
* Control flow helper to continue when a contention was detected and we need to
* clean up and re-start the loop to prepare all GEM objects.
+ * The __drm_exec_loop check exists to prevent usage outside of an
+ * drm_exec_until_all_locked() loop.
*/
#define drm_exec_retry_on_contention(exec) \
do { \
if (unlikely(drm_exec_is_contended(exec))) \
- goto *__drm_exec_retry_ptr; \
- } while (0)
+ goto drm_exec_retry; \
+ } while (__drm_exec_loop)
/**
* drm_exec_is_contended - check for contention
*
* Unconditionally retry the loop to lock all objects. For consistency,
* the exec object needs to be newly initialized.
+ * The __drm_exec_loop check exists to prevent usage outside of an
+ * drm_exec_until_all_locked() loop.
*/
#define drm_exec_retry(_exec) \
do { \
WARN_ON((_exec)->contended != DRM_EXEC_DUMMY); \
- goto *__drm_exec_retry_ptr; \
- } while (0)
+ goto drm_exec_retry; \
+ } while (__drm_exec_loop)
/**
* drm_exec_ticket - return the ww_acquire_ctx for this exec context
struct drm_panel_backlight_quirk {
u16 min_brightness;
u32 brightness_mask;
+ bool force_pwm;
};
const struct drm_panel_backlight_quirk *
#define IMX93_MEDIABLK_PD_PXP 2
#define IMX93_MEDIABLK_PD_LCDIF 3
#define IMX93_MEDIABLK_PD_ISI 4
+#define IMX93_MEDIABLK_PD_MIPI_PHY 5
#endif
#define USB0_RESET 35
#define USB1_RESET 36
#define NAND_RESET 37
-/* 38 is empty */
+#define COMBOPHY_RESET 38
#define SDMMC_RESET 39
#define EMAC0_OCP_RESET 40
#define EMAC1_OCP_RESET 41
#define FFA_EMAD_HAS_IMPDEF_FIELD(version) ((version) >= FFA_VERSION_1_2)
#define FFA_MEM_REGION_HAS_EP_MEM_OFFSET(version) ((version) > FFA_VERSION_1_0)
+/* The layout changed from FFA_VERSION_1_0 and the region includes an
+ * ep_mem_offset.
+ */
+#define FFA_MEM_REGION_SZ(version) (!FFA_MEM_REGION_HAS_EP_MEM_OFFSET((version)) ?\
+ offsetof(struct ffa_mem_region, ep_mem_offset) :\
+ sizeof(struct ffa_mem_region))
+
static inline u32 ffa_emad_size_get(u32 ffa_version)
{
u32 sz;
if (!FFA_MEM_REGION_HAS_EP_MEM_OFFSET(ffa_version))
offset += offsetof(struct ffa_mem_region, ep_mem_offset);
else
- offset += sizeof(struct ffa_mem_region);
+ offset += buf->ep_mem_offset;
return offset;
}
struct btf *btf_parse_vmlinux(void);
struct btf *bpf_prog_get_target_btf(const struct bpf_prog *prog);
u32 *btf_kfunc_flags(const struct btf *btf, u32 kfunc_btf_id, const struct bpf_prog *prog);
+int btf_kfunc_check_flag(const struct btf *btf, u32 kfunc_btf_id, u32 flag);
bool btf_kfunc_is_allowed(const struct btf *btf, u32 kfunc_btf_id, const struct bpf_prog *prog);
u32 *btf_kfunc_is_modify_return(const struct btf *btf, u32 kfunc_btf_id,
const struct bpf_prog *prog);
* CAN driver for PEAK System micro-CAN based adapters
*
* Copyright (C) 2003-2025 PEAK System-Technik GmbH
- * Author: Stéphane Grosjean <stephane.grosjean@hms-networks.com>
+ * Author: Stéphane Grosjean <s.grosjean@peak-system.fr>
*/
#ifndef PUCAN_H
#define PUCAN_H
/* SPDX-License-Identifier: GPL-2.0 */
/*
* DAMON api
- *
- * Author: SeongJae Park <sj@kernel.org>
*/
#ifndef _DAMON_H_
* including damon_call() and damos_walk().
*
* @ops: Set of monitoring operations for given use cases.
+ * @probes: Head of probes (&damon_probe) list.
* @addr_unit: Scale factor for core to ops address conversion.
* @min_region_sz: Minimum region size.
* @pause: Pause kdamond main loop.
* @adaptive_targets: Head of monitoring targets (&damon_target) list.
* @schemes: Head of schemes (&damos) list.
+ * @rnd_state: Per-ctx PRNG state for damon_rand().
*/
struct damon_ctx {
struct damon_attrs attrs;
struct list_head adaptive_targets;
struct list_head schemes;
- /* Per-ctx PRNG state for damon_rand(); kdamond is the sole consumer. */
struct rnd_state rnd_state;
};
static inline unsigned int damon_max_nr_accesses(const struct damon_attrs *attrs)
{
- /* {aggr,sample}_interval are unsigned long, hence could overflow */
- return min(attrs->aggr_interval / attrs->sample_interval,
+ unsigned long sample_interval;
+ unsigned long max_nr_accesses;
+
+ sample_interval = attrs->sample_interval ? : 1;
+ max_nr_accesses = min(attrs->aggr_interval / sample_interval,
(unsigned long)UINT_MAX);
+ return max_nr_accesses ? : 1;
}
return mnt_idmap(file->f_path.mnt);
}
+static inline bool file_owner_or_capable(const struct file *file)
+{
+ return inode_owner_or_capable(file_mnt_idmap(file), file_inode(file));
+}
+
/**
* is_idmapped_mnt - check whether a mount is mapped
* @mnt: the mount to check
#include <linux/compiler.h> /* For __pure */
bool __pure glob_match(char const *pat, char const *str);
+bool __pure glob_match_len(char const *pat, char const *str, size_t len);
#endif /* _LINUX_GLOB_H */
* @attr_usage_id: Usage Id of a field, e.g. X-axis for a gyro.
* @raw_size: Response size for a read request.
* @raw_data: Place holder for received response.
+ * @index: Current write index into raw_data for multi-byte reads.
+ * @max_raw_size: Total buffer size for multi-byte reads; 0 for single-value reads.
*/
struct sensor_hub_pending {
bool status;
u32 attr_usage_id;
int raw_size;
u8 *raw_data;
+ u32 index;
+ u32 max_raw_size;
};
/**
bool is_signed
);
+/**
+ * sensor_hub_input_attr_read_values() - Synchronous multi-byte read request
+ * @hsdev: Hub device instance.
+ * @usage_id: Attribute usage id of parent physical device as per spec
+ * @attr_usage_id: Attribute usage id as per spec
+ * @report_id: Report id to look for
+ * @flag: Synchronous or asynchronous read
+ * @buffer_size: Size of the buffer in bytes
+ * @buffer: Buffer to store the read data
+ *
+ * Issues a synchronous or asynchronous read request for an input attribute,
+ * accumulating data into the provided buffer until it is full.
+ * Return: 0 on success, -ETIMEDOUT if the device did not respond, or a
+ * negative error code.
+ */
+int sensor_hub_input_attr_read_values(struct hid_sensor_hub_device *hsdev,
+ u32 usage_id, u32 attr_usage_id,
+ u32 report_id,
+ enum sensor_hub_read_flags flag,
+ u32 buffer_size, u8 *buffer);
+
/**
* sensor_hub_set_feature() - Feature set request
* @hsdev: Hub device instance.
const struct ieee80211_multi_link_elem *mle = (const void *)data;
u8 fixed = sizeof(*mle);
u8 common = 0;
- bool check_common_len = false;
+ u8 common_len;
u16 control;
if (!data || len < fixed)
switch (u16_get_bits(control, IEEE80211_ML_CONTROL_TYPE)) {
case IEEE80211_ML_CONTROL_TYPE_BASIC:
common += sizeof(struct ieee80211_mle_basic_common_info);
- check_common_len = true;
if (control & IEEE80211_MLC_BASIC_PRES_LINK_ID)
common += 1;
if (control & IEEE80211_MLC_BASIC_PRES_BSS_PARAM_CH_CNT)
common += sizeof(struct ieee80211_mle_preq_common_info);
if (control & IEEE80211_MLC_PREQ_PRES_MLD_ID)
common += 1;
- check_common_len = true;
break;
case IEEE80211_ML_CONTROL_TYPE_RECONF:
+ common += 1;
if (control & IEEE80211_MLC_RECONF_PRES_MLD_MAC_ADDR)
common += ETH_ALEN;
if (control & IEEE80211_MLC_RECONF_PRES_EML_CAPA)
break;
case IEEE80211_ML_CONTROL_TYPE_TDLS:
common += sizeof(struct ieee80211_mle_tdls_common_info);
- check_common_len = true;
break;
case IEEE80211_ML_CONTROL_TYPE_PRIO_ACCESS:
common = ETH_ALEN + 1;
if (len < fixed + common)
return false;
- if (!check_common_len)
- return true;
+ common_len = mle->variable[0];
- /* if present, common length is the first octet there */
- return mle->variable[0] >= common;
+ return common_len >= common && common_len <= len - fixed;
}
/**
#define __in_dev_put(idev) refcount_dec(&(idev)->refcnt)
#define in_dev_hold(idev) refcount_inc(&(idev)->refcnt)
+static inline bool in_dev_hold_safe(struct in_device *idev)
+{
+ return refcount_inc_not_zero(&idev->refcnt);
+}
+
#endif /* __KERNEL__ */
static __inline__ __be32 inet_make_mask(int logmask)
* page_ext_iter_begin() - Prepare for iterating through page extensions.
* @iter: page extension iterator.
* @pfn: PFN of the page we're interested in.
+ * @count: maximum number of page extensions to return.
*
* Must be called with RCU read lock taken.
*
* Return: NULL if no page_ext exists for this page.
*/
static inline struct page_ext *page_ext_iter_begin(struct page_ext_iter *iter,
- unsigned long pfn)
+ unsigned long pfn, unsigned long count)
{
+ if (!count)
+ return NULL;
+
iter->index = 0;
iter->start_pfn = pfn;
iter->page_ext = page_ext_lookup(pfn);
/**
* page_ext_iter_next() - Get next page extension
* @iter: page extension iterator.
+ * @count: maximum number of page extensions to return.
*
* Must be called with RCU read lock taken.
*
* Return: NULL if no next page_ext exists.
*/
-static inline struct page_ext *page_ext_iter_next(struct page_ext_iter *iter)
+static inline struct page_ext *page_ext_iter_next(struct page_ext_iter *iter,
+ unsigned long count)
{
unsigned long pfn;
if (WARN_ON_ONCE(!iter->page_ext))
return NULL;
- iter->index++;
+ if (++iter->index >= count)
+ return NULL;
pfn = iter->start_pfn + iter->index;
if (page_ext_iter_next_fast_possible(pfn))
* IMPORTANT: must be called with RCU read lock taken.
*/
#define for_each_page_ext(__page, __pgcount, __page_ext, __iter) \
- for (__page_ext = page_ext_iter_begin(&__iter, page_to_pfn(__page));\
- __page_ext && __iter.index < __pgcount; \
- __page_ext = page_ext_iter_next(&__iter))
+ for (__page_ext = page_ext_iter_begin(&__iter, page_to_pfn(__page), __pgcount); \
+ __page_ext; \
+ __page_ext = page_ext_iter_next(&__iter, __pgcount))
#else /* !CONFIG_PAGE_EXTENSION */
struct page_ext;
#include <linux/mmzone.h>
#include <linux/scatterlist.h>
-/* This value should always be a power of 2, see page_reporting_cycle() */
#define PAGE_REPORTING_CAPACITY 32
#define PAGE_REPORTING_ORDER_UNSPECIFIED -1
/* Minimal order of page reporting */
unsigned int order;
+
+ /* Max pages per report batch; 0 (default) means PAGE_REPORTING_CAPACITY */
+ unsigned int capacity;
};
/* Tear-down and bring-up for page reporting devices */
psock_set_prog(&progs->skb_verdict, NULL);
}
-/* for tcp only, sk is locked */
-static inline ssize_t sk_psock_msg_inq(struct sock *sk)
-{
- struct sk_psock *psock;
- ssize_t inq = 0;
-
- psock = sk_psock_get(sk);
- if (likely(psock)) {
- inq = sk_psock_get_msg_len_nolock(psock);
- sk_psock_put(sk, psock);
- }
- return inq;
-}
-
/* for udp only, sk is not locked */
static inline ssize_t sk_msg_first_len(struct sock *sk)
{
const struct rpc_timeout *);
void rpc_shutdown_client(struct rpc_clnt *);
+void rpc_hold_client(struct rpc_clnt *);
void rpc_release_client(struct rpc_clnt *);
void rpc_task_release_transport(struct rpc_task *);
void rpc_task_release_client(struct rpc_task *);
{ \
} \
static inline bool \
+ __trace_##name##_enabled(void) \
+ { \
+ return static_branch_unlikely(&__tracepoint_##name.key);\
+ } \
+ static inline bool \
trace_##name##_enabled(void) \
{ \
if (IS_ENABLED(CONFIG_LOCKDEP)) { \
WARN_ONCE(!rcu_is_watching(), \
"RCU not watching for tracepoint"); \
} \
- return static_branch_unlikely(&__tracepoint_##name.key);\
+ return __trace_##name##_enabled(); \
}
#define __DECLARE_TRACE(name, proto, args, cond, data_proto) \
{ \
} \
static inline bool \
+ __trace_##name##_enabled(void) \
+ { \
+ return false; \
+ } \
+ static inline bool \
trace_##name##_enabled(void) \
{ \
return false; \
refcount_inc(&idev->refcnt);
}
+static inline bool in6_dev_hold_safe(struct inet6_dev *idev)
+{
+ return refcount_inc_not_zero(&idev->refcnt);
+}
+
/* called with rcu_read_lock held */
static inline bool ip6_ignore_linkdown(const struct net_device *dev)
{
#define hci_iso_flags_pack(pb, ts) ((pb & 0x03) | ((ts & 0x01) << 2))
/* ISO data length and flags pack/unpack */
-#define hci_iso_data_len_pack(h, f) ((__u16) ((h) | ((f) << 14)))
-#define hci_iso_data_len(h) ((h) & 0x3fff)
+#define hci_iso_data_len_pack(h, f) ((__u16) (((h) & 0x0fff) | \
+ (((f) & 0x3) << 14)))
+#define hci_iso_data_len(h) ((h) & 0x0fff)
#define hci_iso_data_flags(h) ((h) >> 14)
/* codec transport types */
HCI_CONN_AUTH_FAILURE,
HCI_CONN_PER_ADV,
HCI_CONN_BIG_CREATED,
+ HCI_CONN_CREATE,
HCI_CONN_CREATE_CIS,
HCI_CONN_CREATE_BIG_SYNC,
HCI_CONN_BIG_SYNC,
bool persistent);
void mgmt_device_connected(struct hci_dev *hdev, struct hci_conn *conn,
u8 *name, u8 name_len);
+u8 hci_to_mgmt_reason(u8 err);
void mgmt_device_disconnected(struct hci_dev *hdev, bdaddr_t *bdaddr,
u8 link_type, u8 addr_type, u8 reason,
bool mgmt_connected);
struct l2cap_ops {
char *name;
- struct l2cap_chan *(*new_connection) (struct l2cap_chan *chan);
+ int (*new_connection)(struct l2cap_chan *chan,
+ struct l2cap_chan *new_chan);
int (*recv) (struct l2cap_chan * chan,
struct sk_buff *skb);
void (*teardown) (struct l2cap_chan *chan, int err);
return (seq + 1) % (chan->tx_win_max + 1);
}
-static inline struct l2cap_chan *l2cap_chan_no_new_connection(struct l2cap_chan *chan)
+static inline int l2cap_chan_no_new_connection(struct l2cap_chan *chan,
+ struct l2cap_chan *new_chan)
{
- return NULL;
+ return -EOPNOTSUPP;
}
static inline int l2cap_chan_no_recv(struct l2cap_chan *chan, struct sk_buff *skb)
void l2cap_chan_busy(struct l2cap_chan *chan, int busy);
void l2cap_chan_rx_avail(struct l2cap_chan *chan, ssize_t rx_avail);
int l2cap_chan_check_security(struct l2cap_chan *chan, bool initiator);
-void l2cap_chan_set_defaults(struct l2cap_chan *chan);
+void l2cap_chan_set_defaults(struct l2cap_chan *chan, struct l2cap_chan *pchan);
int l2cap_ertm_init(struct l2cap_chan *chan);
void l2cap_chan_add(struct l2cap_conn *conn, struct l2cap_chan *chan);
void __l2cap_chan_add(struct l2cap_conn *conn, struct l2cap_chan *chan);
enum ieee80211_bss_type conn_bss_type;
u32 conn_owner_nlportid;
- struct work_struct disconnect_wk;
+ struct wiphy_work disconnect_wk;
u8 disconnect_bssid[ETH_ALEN];
struct list_head event_list;
struct list_head pmsr_list;
spinlock_t pmsr_lock;
- struct work_struct pmsr_free_wk;
+ struct wiphy_work pmsr_free_wk;
unsigned long unprot_beacon_reported;
static inline size_t guehdr_priv_flags_len(__be32 flags)
{
- return 0;
+ return (flags & GUE_PFLAG_REMCSUM) ? GUE_PLEN_REMCSUM : 0;
}
/* Validate standard and private flags. Returns non-zero (meaning invalid)
void (*state_transition)(struct ip_vs_conn *cp, int direction,
const struct sk_buff *skb,
- struct ip_vs_proto_data *pd);
+ struct ip_vs_proto_data *pd,
+ unsigned int iph_len);
int (*register_app)(struct netns_ipvs *ipvs, struct ip_vs_app *inc);
void *buf_va;
bool from_pool; /* allocated from a page pool */
+ /* head page of the page_pool fragment; valid only when
+ * from_pool && frag_count > 1.
+ */
+ struct page *pp_page;
+ /* Fragment offset plus rxq->headroom, passed to
+ * page_pool_dma_sync_for_cpu().
+ */
+ u32 dma_sync_offset;
/* SGL of the buffer going to be sent as part of the work request. */
u32 num_sge;
tun_num:2,
in_vlan_ingress:2;
u16 mtu;
+ u32 dst_cookie;
+ struct dst_entry *dst_cache;
+
union {
struct {
- struct dst_entry *dst_cache;
u32 ifidx;
- u32 dst_cookie;
};
struct {
u32 ifidx;
void nf_flow_offload_add(struct nf_flowtable *flowtable,
struct flow_offload *flow);
+void nf_flow_offload_refresh(struct nf_flowtable *flowtable,
+ struct flow_offload *flow);
void nf_flow_offload_del(struct nf_flowtable *flowtable,
struct flow_offload *flow);
void nf_flow_offload_stats(struct nf_flowtable *flowtable,
__u8 auth_random[sizeof(struct sctp_paramhdr) +
SCTP_AUTH_RANDOM_LENGTH];
- __u8 auth_hmacs[SCTP_AUTH_NUM_HMACS * sizeof(__u16) + 2];
+ __u8 auth_hmacs[sizeof(struct sctp_paramhdr) +
+ SCTP_AUTH_NUM_HMACS * sizeof(__u16)];
__u8 auth_chunks[sizeof(struct sctp_paramhdr) + SCTP_AUTH_MAX_CHUNKS];
/* This is a shim for my peer's INIT packet, followed by
return false;
}
-static inline int tcf_pedit_nkeys(const struct tc_action *a)
+/* Must be called with act->tcfa_lock held to ensure consistency of parallel
+ * reads of the same action's pedit keys (e.g. flow_offload count vs fill).
+ * Note, this is only used for pedit offload.
+ */
+static inline int tcf_pedit_nkeys_locked(const struct tc_action *a)
{
- struct tcf_pedit_parms *parms;
- int nkeys;
-
- rcu_read_lock();
- parms = to_pedit_parms(a);
- nkeys = parms->tcfp_nkeys;
- rcu_read_unlock();
-
- return nkeys;
+ lockdep_assert_held(&a->tcfa_lock);
+ return rcu_dereference_protected(to_pedit(a)->parms,
+ lockdep_is_held(&a->tcfa_lock))->tcfp_nkeys;
}
static inline u32 tcf_pedit_htype(const struct tc_action *a, int index)
*/
struct net_device *real_dev;
unsigned long offload_handle;
+ /* Snapshot the attached device index for dump paths. */
+ int ifindex;
u8 dir : 2;
u8 type : 2;
u8 flags : 2;
void scsi_cdl_check(struct scsi_device *sdev);
int scsi_cdl_enable(struct scsi_device *sdev, bool enable);
-extern struct scsi_device *scsi_device_from_queue(struct request_queue *q);
extern int __must_check scsi_device_get(struct scsi_device *);
extern void scsi_device_put(struct scsi_device *);
extern struct scsi_device *scsi_device_lookup(struct Scsi_Host *,
*/
struct device *dma_dev;
+ /* Used for an rcu-synchronizing eh wakeup */
+ struct work_struct eh_work;
+
/* Delay for runtime autosuspend */
int rpm_autosuspend_delay;
bool tegra_pmc_cpu_is_powered(unsigned int cpuid);
int tegra_pmc_cpu_power_on(unsigned int cpuid);
int tegra_pmc_cpu_remove_clamping(unsigned int cpuid);
-bool tegra_pmc_core_domain_state_synced(void);
#if defined(CONFIG_SOC_TEGRA_PMC) && defined(CONFIG_PM_SLEEP)
enum tegra_suspend_mode tegra_pmc_get_suspend_mode(void);
{
}
#endif
+#endif
+
+#if defined(CONFIG_ARM) && defined(CONFIG_SOC_TEGRA_PMC)
+bool tegra_pmc_core_domain_state_synced(void);
#else
/* needed for COMPILE_TEST */
static inline bool tegra_pmc_core_domain_state_synced(void)
/* SPDX-License-Identifier: GPL-2.0 */
#undef TRACE_SYSTEM
-#define TRACE_SYSTEM memory_failure
+/*
+ * For historical versions, memory_failure_event is in ras subsystem,
+ * some user programs depend on it.
+ */
+#define TRACE_SYSTEM ras
#define TRACE_INCLUDE_FILE memory-failure
#if !defined(_TRACE_MEMORY_FAILURE_H) || defined(TRACE_HEADER_MULTI_READ)
__u64 buf_size; /* in - size of buffer
* out - on EOVERFLOW: needed size
* to store item */
- __u64 buf[]; /* out - found items */
+ __u8 buf[]; /* out - found items */
};
/* With a @src_length of zero, the range from @src_offset->EOF is cloned! */
if (ctx->bpf_ops)
return -EBUSY;
+ if (ops->priv)
+ return -EBUSY;
if (WARN_ON_ONCE(!ops->loop_step))
return -EINVAL;
const char __user *fname;
int err;
- if (sqe->off || sqe->len || sqe->buf_index || sqe->splice_fd_in)
+ if (sqe->off || sqe->len || sqe->buf_index || sqe->splice_fd_in ||
+ sqe->addr3 || sqe->__pad2[0])
return -EINVAL;
if (unlikely(req->flags & REQ_F_FIXED_FILE))
return -EBADF;
iov = kmalloc_objs(struct iovec, nr_avail);
if (unlikely(!iov))
return -ENOMEM;
- if (arg->mode & KBUF_MODE_FREE)
- kfree(arg->iovs);
arg->iovs = iov;
nr_iovs = nr_avail;
} else if (nr_avail < nr_iovs) {
buf = io_ring_head_to_buf(br, ++head, bl->mask);
} while (--nr_iovs);
+ if (arg->iovs != org_iovs && (arg->mode & KBUF_MODE_FREE))
+ kfree(org_iovs);
+
if (head == tail)
req->flags |= REQ_F_BL_EMPTY;
ptr = io_uring_validate_mmap_request(filp, pgoff);
if (IS_ERR(ptr))
- return -ENOMEM;
+ return PTR_ERR(ptr);
/*
* Some architectures have strong cache aliasing requirements.
io_req_task_work_add_remote(req, IOU_F_TWQ_LAZY_WAKE);
}
+static int io_msg_ring_cqe_flags(struct io_ring_ctx *target_ctx,
+ const struct io_msg *msg, u32 *flags)
+{
+ *flags = 0;
+
+ if (!(msg->flags & IORING_MSG_RING_FLAGS_PASS))
+ return 0;
+
+ *flags = msg->cqe_flags;
+ if ((*flags & IORING_CQE_F_32) &&
+ !(target_ctx->flags & (IORING_SETUP_CQE32 |
+ IORING_SETUP_CQE_MIXED)))
+ return -EINVAL;
+
+ return 0;
+}
+
static int io_msg_data_remote(struct io_ring_ctx *target_ctx,
struct io_msg *msg)
{
struct io_kiocb *target;
- u32 flags = 0;
+ u32 flags;
+ int ret;
- target = kmem_cache_alloc(req_cachep, GFP_KERNEL | __GFP_NOWARN | __GFP_ZERO) ;
+ ret = io_msg_ring_cqe_flags(target_ctx, msg, &flags);
+ if (ret)
+ return ret;
+
+ target = kmem_cache_alloc(req_cachep, GFP_KERNEL | __GFP_NOWARN | __GFP_ZERO);
if (unlikely(!target))
return -ENOMEM;
- if (msg->flags & IORING_MSG_RING_FLAGS_PASS)
- flags = msg->cqe_flags;
-
io_msg_remote_post(target_ctx, target, msg->len, flags, msg->user_data);
return 0;
}
if (io_msg_need_remote(target_ctx))
return io_msg_data_remote(target_ctx, msg);
- if (msg->flags & IORING_MSG_RING_FLAGS_PASS)
- flags = msg->cqe_flags;
+ ret = io_msg_ring_cqe_flags(target_ctx, msg, &flags);
+ if (ret)
+ return ret;
ret = -EOVERFLOW;
if (target_ctx->flags & IORING_SETUP_IOPOLL) {
*/
static void io_ctx_mark_taskrun(struct io_ring_ctx *ctx)
{
+ lockdep_assert_in_rcu_read_lock();
+
if (ctx->flags & IORING_SETUP_TASKRUN_FLAG) {
- struct io_rings *rings;
+ struct io_rings *rings = rcu_dereference(ctx->rings_rcu);
- guard(rcu)();
- rings = rcu_dereference(ctx->rings_rcu);
atomic_or(IORING_SQ_TASKRUN, &rings->sq_flags);
}
}
struct io_ring_ctx *ctx = req->ctx;
int nr_wait;
+ /* pairs with synchronize_rcu() in io_ring_exit_work() */
+ guard(rcu)();
+
/*
* We don't know how many requests there are in the link and whether
* they can even be queued lazily, fall back to non-lazy.
}
/*
- * Mark this command as concelable, then io_uring_try_cancel_uring_cmd()
+ * Mark this command as cancelable, then io_uring_try_cancel_uring_cmd()
* will try to cancel this issued command by sending ->uring_cmd() with
* issue_flags of IO_URING_F_CANCEL.
*
}
io_req_uring_cleanup(req, issue_flags);
if (req->flags & REQ_F_IOPOLL) {
- /* order with io_iopoll_req_issued() checking ->iopoll_complete */
+ /* order with io_do_iopoll() checking ->iopoll_completed */
smp_store_release(&req->iopoll_completed, 1);
} else if (issue_flags & IO_URING_F_COMPLETE_DEFER) {
if (WARN_ON_ONCE(issue_flags & IO_URING_F_UNLOCKED))
#include <net/ip.h>
#include <net/ipv6.h>
#include <linux/sctp.h>
+#include <linux/overflow.h>
#include "audit.h"
* do the multicast send and rotate records from the
* main queue to the retry/hold queues */
wait_event_freezable(kauditd_wait,
- (skb_queue_len(&audit_queue) ? 1 : 0));
+ (skb_queue_len_lockless(&audit_queue) ? 1 : 0));
}
return 0;
s.rate_limit = audit_rate_limit;
s.backlog_limit = audit_backlog_limit;
s.lost = atomic_read(&audit_lost);
- s.backlog = skb_queue_len(&audit_queue);
+ s.backlog = skb_queue_len_lockless(&audit_queue);
s.feature_bitmap = AUDIT_FEATURE_BITMAP_ALL;
s.backlog_wait_time = audit_backlog_wait_time;
s.backlog_wait_time_actual = atomic_read(&audit_backlog_wait_time_actual);
/* can't block with the ctrl lock, so penalize the sender now */
if (audit_backlog_limit &&
- (skb_queue_len(&audit_queue) > audit_backlog_limit)) {
+ (skb_queue_len_lockless(&audit_queue) > audit_backlog_limit)) {
DECLARE_WAITQUEUE(wait, current);
/* wake kauditd to try and flush the queue */
long stime = audit_backlog_wait_time;
while (audit_backlog_limit &&
- (skb_queue_len(&audit_queue) > audit_backlog_limit)) {
+ (skb_queue_len_lockless(&audit_queue) > audit_backlog_limit)) {
/* wake kauditd to try and flush the queue */
wake_up_interruptible(&kauditd_wait);
} else {
if (audit_rate_check() && printk_ratelimit())
pr_warn("audit_backlog=%d > audit_backlog_limit=%d\n",
- skb_queue_len(&audit_queue),
+ skb_queue_len_lockless(&audit_queue),
audit_backlog_limit);
audit_log_lost("backlog limit exceeded");
return NULL;
void audit_log_n_hex(struct audit_buffer *ab, const unsigned char *buf,
size_t len)
{
- int i, avail, new_len;
+ int avail;
+ size_t i, new_len;
unsigned char *ptr;
struct sk_buff *skb;
BUG_ON(!ab->skb);
skb = ab->skb;
avail = skb_tailroom(skb);
- new_len = len<<1;
+
+ if (check_shl_overflow(len, 1, &new_len)) {
+ audit_log_format(ab, "?");
+ return;
+ }
+
if (new_len >= avail) {
/* Round the buffer request up to the next multiple */
new_len = AUDIT_BUFSIZ*(((new_len-avail)/AUDIT_BUFSIZ) + 1);
return btf_kfunc_id_set_contains(btf, hook, kfunc_btf_id);
}
+/*
+ * Check a single KF_* @flag on a kfunc across all of its hook sets.
+ * Returns:
+ * * 1 if @flag is set
+ * * 0 if @flag is not set
+ * * -EINVAL if @flag is set inconsistently across the sets
+ * * -ENOENT if kfunc_btf_id is not a registered kfunc
+ */
+int btf_kfunc_check_flag(const struct btf *btf, u32 kfunc_btf_id, u32 flag)
+{
+ enum btf_kfunc_hook hook;
+ int res = -ENOENT;
+ bool is_set;
+ u32 *flags;
+
+ for (hook = 0; hook < BTF_KFUNC_HOOK_MAX; hook++) {
+ flags = btf_kfunc_id_set_contains(btf, hook, kfunc_btf_id);
+ if (!flags)
+ continue;
+ is_set = *flags & flag;
+ if (res < 0)
+ res = is_set;
+ else if (res != is_set)
+ return -EINVAL;
+ }
+
+ return res;
+}
+
u32 *btf_kfunc_is_modify_return(const struct btf *btf, u32 kfunc_btf_id,
const struct bpf_prog *prog)
{
#define KF_IMPL_SUFFIX "_impl"
-static const struct btf_type *find_kfunc_impl_proto(struct bpf_verifier_env *env,
+static const struct btf_type *find_kfunc_impl_proto(struct bpf_verifier_log *log,
struct btf *btf,
const char *func_name)
{
- char *buf = env->tmp_str_buf;
const struct btf_type *func;
+ char buf[KSYM_NAME_LEN];
s32 impl_id;
int len;
- len = snprintf(buf, TMP_STR_BUF_LEN, "%s%s", func_name, KF_IMPL_SUFFIX);
- if (len < 0 || len >= TMP_STR_BUF_LEN) {
- verbose(env, "function name %s%s is too long\n", func_name, KF_IMPL_SUFFIX);
+ len = snprintf(buf, sizeof(buf), "%s%s", func_name, KF_IMPL_SUFFIX);
+ if (len < 0 || len >= sizeof(buf)) {
+ bpf_log(log, "function name %s%s is too long\n",
+ func_name, KF_IMPL_SUFFIX);
return NULL;
}
impl_id = btf_find_by_name_kind(btf, buf, BTF_KIND_FUNC);
if (impl_id <= 0) {
- verbose(env, "cannot find function %s in BTF\n", buf);
+ bpf_log(log, "cannot find function %s in BTF\n", buf);
return NULL;
}
* can be found through the counterpart _impl kfunc.
*/
if (kfunc_flags && (*kfunc_flags & KF_IMPLICIT_ARGS))
- func_proto = find_kfunc_impl_proto(env, btf, func_name);
+ func_proto = find_kfunc_impl_proto(&env->log, btf, func_name);
else
func_proto = btf_type_by_id(btf, func->type);
static int __check_buffer_access(struct bpf_verifier_env *env,
const char *buf_info,
const struct bpf_reg_state *reg,
- argno_t argno, int off, int size)
+ argno_t argno, int off, int size,
+ u32 *access_end)
{
- if (off < 0) {
- verbose(env,
- "%s invalid %s buffer access: off=%d, size=%d\n",
- reg_arg_name(env, argno), buf_info, off, size);
- return -EACCES;
- }
+ s64 start;
+
if (!tnum_is_const(reg->var_off)) {
char tn_buf[48];
return -EACCES;
}
+ start = (s64)reg->var_off.value + off;
+ if (start < 0) {
+ verbose(env,
+ "%s invalid negative %s buffer offset: off=%d, var_off=%lld\n",
+ reg_arg_name(env, argno), buf_info, off, (s64)reg->var_off.value);
+ return -EACCES;
+ }
+
+ *access_end = start + size;
return 0;
}
const struct bpf_reg_state *reg,
argno_t argno, int off, int size)
{
+ u32 access_end;
int err;
- err = __check_buffer_access(env, "tracepoint", reg, argno, off, size);
+ err = __check_buffer_access(env, "tracepoint", reg, argno, off, size, &access_end);
if (err)
return err;
- env->prog->aux->max_tp_access = max(reg->var_off.value + off + size,
- env->prog->aux->max_tp_access);
+ env->prog->aux->max_tp_access = max(access_end, env->prog->aux->max_tp_access);
return 0;
}
u32 *max_access)
{
const char *buf_info = type_is_rdonly_mem(reg->type) ? "rdonly" : "rdwr";
+ u32 access_end;
int err;
- err = __check_buffer_access(env, buf_info, reg, argno, off, size);
+ err = __check_buffer_access(env, buf_info, reg, argno, off, size, &access_end);
if (err)
return err;
- *max_access = max(reg->var_off.value + off + size, *max_access);
+ *max_access = max(access_end, *max_access);
return 0;
}
return -EINVAL;
}
+/*
+ * Resolve the prototype describing a trace target's real ABI. A
+ * KF_IMPLICIT_ARGS kfunc has its injected args stripped from the public
+ * prototype, so use the _impl prototype; other targets use their own.
+ */
+static const struct btf_type *
+btf_attach_func_proto(struct bpf_verifier_log *log, struct btf *btf, u32 func_id)
+{
+ const struct btf_type *func;
+ struct module *mod = NULL;
+ const char *name;
+ int implicit;
+
+ func = btf_type_by_id(btf, func_id);
+ if (!func || !btf_type_is_func(func))
+ return NULL;
+ name = btf_name_by_offset(btf, func->name_off);
+
+ /*
+ * btf_kfunc_check_flag() reads kfunc_set_tab, which for a module is
+ * stable only once it is live; hold a module ref across the read to
+ * exclude a concurrent module load.
+ */
+ if (btf_is_module(btf)) {
+ mod = btf_try_get_module(btf);
+ if (!mod)
+ return NULL;
+ }
+ implicit = btf_kfunc_check_flag(btf, func_id, KF_IMPLICIT_ARGS);
+ module_put(mod);
+
+ if (implicit == -EINVAL) {
+ bpf_log(log, "kfunc %s has inconsistent KF_IMPLICIT_ARGS\n", name);
+ return NULL;
+ }
+ if (implicit > 0)
+ return find_kfunc_impl_proto(log, btf, name);
+
+ return btf_type_by_id(btf, func->type);
+}
+
int bpf_check_attach_target(struct bpf_verifier_log *log,
const struct bpf_prog *prog,
const struct bpf_prog *tgt_prog,
if (prog_extension &&
btf_check_type_match(log, prog, btf, t))
return -EINVAL;
- t = btf_type_by_id(btf, t->type);
- if (!btf_type_is_func_proto(t))
+ t = btf_attach_func_proto(log, btf, btf_id);
+ if (!t || !btf_type_is_func_proto(t))
return -EINVAL;
if ((prog->aux->saved_dst_prog_type || prog->aux->saved_dst_attach_type) &&
tname = btf_name_by_offset(btf, t->name_off);
if (!tname)
return -EINVAL;
- if (!btf_type_is_func(t))
- return -EINVAL;
- t = btf_type_by_id(btf, t->type);
- if (!btf_type_is_func_proto(t))
+ t = btf_attach_func_proto(NULL, btf, btf_id);
+ if (!t || !btf_type_is_func_proto(t))
return -EINVAL;
err = btf_distill_func_proto(NULL, btf, t, tname, &tgt_info->fmodel);
if (err < 0)
migrate = is_memory_migrate(cs);
- mpol_rebind_mm(mm, &cs->mems_allowed);
+ /*
+ * For v1 we can have empty effective_mems, but we cannot
+ * attach any tasks (see cpuset_can_attach_check()). For v2,
+ * effective_mems is guaranteed to not be empty.
+ */
+ mpol_rebind_mm(mm, &cs->effective_mems);
if (migrate)
cpuset_migrate_mm(mm, &cs->old_mems_allowed, &newmems);
else
int err = 0;
unsigned long pagenum;
+ guard(mutex)(&rb->aux_mutex);
+
/*
* We map this as a VM_PFNMAP VMA.
*
__unhash_process(post, tsk, group_dead);
write_sequnlock(&sig->stats_lock);
- tsk->sighand = NULL;
+ /*
+ * Ensure that all preceeding state is visible. Pairs with
+ * the smp_acquire__after_ctrl_dep() in the sighand == NULL
+ * path of lock_task_sighand().
+ */
+ smp_store_release(&tsk->sighand, NULL);
spin_unlock(&sighand->siglock);
__cleanup_sighand(sighand);
*/
DEFINE_PER_CPU(struct rq *, scx_locked_rq_state);
+static void switch_rq_lock(struct rq *from, struct rq *to)
+{
+ bool tracked = scx_locked_rq() == from;
+
+ if (tracked)
+ update_locked_rq(NULL);
+ raw_spin_rq_unlock(from);
+ raw_spin_rq_lock(to);
+ if (tracked)
+ update_locked_rq(to);
+}
+
/*
* Flipped on enable per sch->is_cid_type. Declared in internal.h so
* subsystem inlines can read it.
deactivate_task(src_rq, p, 0);
set_task_cpu(p, cpu_of(dst_rq));
- raw_spin_rq_unlock(src_rq);
- raw_spin_rq_lock(dst_rq);
+ switch_rq_lock(src_rq, dst_rq);
/*
* We want to pass scx-specific enq_flags but activate_task() will
* no to the BPF scheduler initiated migrations while offline.
*
* The caller must ensure that @p and @rq are on different CPUs.
+ * If enforce == true, caller must hold @p's rq lock.
*/
static bool task_can_run_on_remote_rq(struct scx_sched *sch,
struct task_struct *p, struct rq *rq,
{
s32 cpu = cpu_of(rq);
+ /*
+ * To prevent races with @p still running on its old CPU while switching
+ * out, make sure we're holding @p's rq lock so as not to risk
+ * erroneously killing the BPF scheduler.
+ */
+ if (enforce)
+ lockdep_assert_rq_held(task_rq(p));
+
WARN_ON_ONCE(task_cpu(p) == cpu);
/*
return;
}
- if (src_rq != dst_rq &&
- unlikely(!task_can_run_on_remote_rq(sch, p, dst_rq, true))) {
- dispatch_enqueue(sch, rq, find_global_dsq(sch, task_cpu(p)), p,
- enq_flags | SCX_ENQ_CLEAR_OPSS | SCX_ENQ_GDSQ_FALLBACK);
- return;
- }
-
/*
* @p is on a possibly remote @src_rq which we need to lock to move the
* task. If dequeue is in progress, it'd be locking @src_rq and waiting
/* switch to @src_rq lock */
if (locked_rq != src_rq) {
- raw_spin_rq_unlock(locked_rq);
+ switch_rq_lock(locked_rq, src_rq);
locked_rq = src_rq;
- raw_spin_rq_lock(src_rq);
}
/* task_rq couldn't have changed if we're still the holding cpu */
if (likely(p->scx.holding_cpu == raw_smp_processor_id()) &&
!WARN_ON_ONCE(src_rq != task_rq(p))) {
+ bool fallback = false;
/*
* If @p is staying on the same rq, there's no need to go
* through the full deactivate/activate cycle. Optimize by
p->scx.holding_cpu = -1;
dispatch_enqueue(sch, dst_rq, &dst_rq->scx.local_dsq, p,
enq_flags);
+ } else if (unlikely(!task_can_run_on_remote_rq(sch, p, dst_rq, true))) {
+ p->scx.holding_cpu = -1;
+ fallback = true;
+ dispatch_enqueue(sch, src_rq, find_global_dsq(sch, task_cpu(p)),
+ p, enq_flags | SCX_ENQ_GDSQ_FALLBACK);
} else {
move_remote_task_to_local_dsq(p, enq_flags,
src_rq, dst_rq);
}
/* if the destination CPU is idle, wake it up */
- if (sched_class_above(p->sched_class, dst_rq->curr->sched_class))
+ if (!fallback && sched_class_above(p->sched_class, dst_rq->curr->sched_class))
resched_curr(dst_rq);
}
/* switch back to @rq lock */
- if (locked_rq != rq) {
- raw_spin_rq_unlock(locked_rq);
- raw_spin_rq_lock(rq);
- }
+ if (locked_rq != rq)
+ switch_rq_lock(locked_rq, rq);
}
/**
/*
* @p is getting newly scheduled or got kicked after someone updated its
- * slice. Refresh whether tick can be stopped. See scx_can_stop_tick().
+ * slice. Update SCX_RQ_CAN_STOP_TICK to reflect whether the tick can be
+ * stopped. See scx_can_stop_tick().
+ *
+ * Moreover, refresh the load_avgs just when transitioning in and out of
+ * nohz. In the future, we might want to add a mechanism to update
+ * load_avgs periodically on tick-stopped CPUs.
*/
- if ((p->scx.slice == SCX_SLICE_INF) !=
- (bool)(rq->scx.flags & SCX_RQ_CAN_STOP_TICK)) {
- if (p->scx.slice == SCX_SLICE_INF)
+ if (p->scx.slice == SCX_SLICE_INF) {
+ if (!(rq->scx.flags & SCX_RQ_CAN_STOP_TICK)) {
+ /*
+ * Bypass mode always assigns finite slices, so @p
+ * can't have an infinite slice while bypassing.
+ * Therefore, sched_update_tick_dependency() can safely
+ * evaluate the outgoing task.
+ */
rq->scx.flags |= SCX_RQ_CAN_STOP_TICK;
- else
- rq->scx.flags &= ~SCX_RQ_CAN_STOP_TICK;
+ sched_update_tick_dependency(rq);
- sched_update_tick_dependency(rq);
+ update_other_load_avgs(rq);
+ }
+ } else {
+ if (rq->scx.flags & SCX_RQ_CAN_STOP_TICK) {
+ rq->scx.flags &= ~SCX_RQ_CAN_STOP_TICK;
+ update_other_load_avgs(rq);
+ }
/*
- * For now, let's refresh the load_avgs just when transitioning
- * in and out of nohz. In the future, we might want to add a
- * mechanism which calls the following periodically on
- * tick-stopped CPUs.
+ * @rq still references the outgoing scheduling context. A finite
+ * slice is sufficient by itself to require the tick.
*/
- update_other_load_avgs(rq);
+ if (tick_nohz_full_cpu(cpu_of(rq)))
+ tick_nohz_dep_set_cpu(cpu_of(rq), TICK_DEP_BIT_SCHED);
}
}
* sched_class, %SCX_OPS_ENQ_LAST must be set. Tell
* ops.enqueue() that @p is the only one available for this cpu,
* which should trigger an explicit follow-up scheduling event.
+ *
+ * Core scheduling can force this CPU idle while @p stays
+ * runnable. @p's cookie then won't match the core's, so skip
+ * the warning in that case.
*/
if (next && sched_class_above(&ext_sched_class, next->sched_class)) {
- WARN_ON_ONCE(!(sch->ops.flags & SCX_OPS_ENQ_LAST));
+ WARN_ON_ONCE(sched_cpu_cookie_match(rq, p) &&
+ !(sch->ops.flags & SCX_OPS_ENQ_LAST));
do_enqueue_task(rq, p, SCX_ENQ_LAST, -1);
} else {
do_enqueue_task(rq, p, 0, -1);
SCX_CALL_OP_TASK(sch, disable, rq, p);
scx_set_task_state(p, SCX_TASK_READY);
+ /*
+ * Reset the SCX-managed fields when @p leaves the BPF scheduler's
+ * control, after ops.disable() has observed their final values.
+ */
+ p->scx.dsq_vtime = 0;
+ p->scx.slice = 0;
+
/*
* Verify the task is not in BPF scheduler's custody. If flag
* transitions are consistent, the flag should always be clear
if (task_dead_and_done(p))
return;
+ /*
+ * When switching sched_class away from SCX, reweight_task_scx()
+ * is called _after_ scx_disable_task(). Skip calling ops.set_weight()
+ * since the BPF scheduler may have already forgotten the task in
+ * ops.disable().
+ * p->scx.weight will be recalculated in scx_enable_task() if the task
+ * ever returns to SCX class.
+ */
+ if (scx_get_task_state(p) != SCX_TASK_ENABLED)
+ return;
+
p->scx.weight = sched_weight_to_cgroup(scale_load_down(lw->weight));
if (SCX_HAS_OP(sch, set_weight))
SCX_CALL_OP_TASK(sch, set_weight, rq, p, p->scx.weight);
if (p->sched_class != &ext_sched_class)
return true;
+ /*
+ * @rq->curr may still reference an outgoing EXT task after it has been
+ * dequeued. If no EXT tasks are accounted on @rq, ignore its stale
+ * slice state. If another task is dispatched from a DSQ,
+ * set_next_task_scx() will update the dependency for the incoming task.
+ */
+ if (!rq->scx.nr_running)
+ return true;
+
if (scx_bypassing(sch, cpu_of(rq)))
return false;
cgroup_put(sch_cgroup(sch));
if (sch->sub_kset)
kobject_put(&sch->sub_kset->kobj);
+ if (scx_parent(sch))
+ kobject_put(&scx_parent(sch)->kobj);
#endif /* CONFIG_EXT_SUB_SCHED */
for_each_possible_cpu(cpu) {
int cpu;
for_each_possible_cpu(cpu) {
- struct scx_kick_syncs **ksyncs = per_cpu_ptr(&scx_kick_syncs, cpu);
+ struct scx_kick_syncs __rcu **ksyncs = per_cpu_ptr(&scx_kick_syncs, cpu);
struct scx_kick_syncs *to_free;
to_free = rcu_replace_pointer(*ksyncs, NULL, true);
* can exceed percpu allocator limits on large machines.
*/
for_each_possible_cpu(cpu) {
- struct scx_kick_syncs **ksyncs = per_cpu_ptr(&scx_kick_syncs, cpu);
+ struct scx_kick_syncs __rcu **ksyncs = per_cpu_ptr(&scx_kick_syncs, cpu);
struct scx_kick_syncs *new_ksyncs;
WARN_ON_ONCE(rcu_access_pointer(*ksyncs));
sch->ops = *cmd->ops;
}
- rcu_assign_pointer(ops->priv, sch);
-
- sch->kobj.kset = scx_kset;
- INIT_LIST_HEAD(&sch->all);
-
#ifdef CONFIG_EXT_SUB_SCHED
char *buf = kzalloc(PATH_MAX, GFP_KERNEL);
if (!buf) {
sch->cgrp = cgrp;
INIT_LIST_HEAD(&sch->children);
INIT_LIST_HEAD(&sch->sibling);
+#endif /* CONFIG_EXT_SUB_SCHED */
- if (parent)
+ /*
+ * Publishing makes @sch visible to scx_prog_sched() readers. Failure
+ * paths after this point must free @sch through kobject_put() whose
+ * release path defers the actual freeing by an RCU grace period.
+ */
+ rcu_assign_pointer(ops->priv, sch);
+
+ sch->kobj.kset = scx_kset;
+ INIT_LIST_HEAD(&sch->all);
+
+#ifdef CONFIG_EXT_SUB_SCHED
+ if (parent) {
+ /*
+ * Pin @parent for @sch's lifetime. The kobject hierarchy pins
+ * it only via @parent->sub_kset, which is dropped during
+ * disable. Released in scx_sched_free_rcu_work().
+ */
+ kobject_get(&parent->kobj);
ret = kobject_init_and_add(&sch->kobj, &scx_ktype,
&parent->sub_kset->kobj,
"sub-%llu", cgroup_id(cgrp));
- else
+ } else {
ret = kobject_init_and_add(&sch->kobj, &scx_ktype, NULL, "root");
+ }
if (ret < 0) {
RCU_INIT_POINTER(ops->priv, NULL);
#ifdef CONFIG_EXT_SUB_SCHED
err_free_lb_resched:
- RCU_INIT_POINTER(ops->priv, NULL);
free_cpumask_var(sch->bypass_lb_resched_cpumask);
#endif
err_free_lb_cpumask:
* run past the BPF allocation. Skip for cid-form.
*/
if (!sch->is_cid_type && (ops->cpu_acquire || ops->cpu_release))
- pr_warn("ops->cpu_acquire/release() are deprecated, use sched_switch TP instead\n");
+ pr_warn_ratelimited("ops->cpu_acquire/release() are deprecated, use sched_switch TP instead\n");
/*
* Sub-scheduler support is tied to the cid-form struct_ops. A sub-sched
percpu_up_write(&scx_fork_rwsem);
err_disable:
mutex_unlock(&scx_enable_mutex);
+ /*
+ * Some enable failures only return an errno (e.g. -ENOMEM from an
+ * allocation) without calling scx_error(). Record it so
+ * scx_flush_disable_work() runs the disable and ops.exit() fires.
+ */
+ scx_error(sch, "scx_sub_enable() failed (%d)", ret);
scx_flush_disable_work(sch);
cmd->ret = 0;
}
off + size <= offsetofend(struct task_struct, scx.slice)) ||
(off >= offsetof(struct task_struct, scx.dsq_vtime) &&
off + size <= offsetofend(struct task_struct, scx.dsq_vtime))) {
- pr_warn("sched_ext: Writing directly to p->scx.slice/dsq_vtime is deprecated, use scx_bpf_task_set_slice/dsq_vtime()");
+ pr_warn_ratelimited("sched_ext: Writing directly to p->scx.slice/dsq_vtime is deprecated, use scx_bpf_task_set_slice/dsq_vtime()\n");
return SCALAR_VALUE;
}
in_balance = this_rq->scx.flags & SCX_RQ_IN_BALANCE;
if (in_balance) {
- if (this_rq != src_rq) {
- raw_spin_rq_unlock(this_rq);
- raw_spin_rq_lock(src_rq);
- }
+ if (this_rq != src_rq)
+ switch_rq_lock(this_rq, src_rq);
} else {
raw_spin_rq_lock(src_rq);
}
dispatched = true;
out:
if (in_balance) {
- if (this_rq != locked_rq) {
- raw_spin_rq_unlock(locked_rq);
- raw_spin_rq_lock(this_rq);
- }
+ if (this_rq != locked_rq)
+ switch_rq_lock(locked_rq, this_rq);
} else {
raw_spin_rq_unlock_irqrestore(locked_rq, flags);
}
* The sched_ext core uses a "lock dancing" protocol coordinated by
* p->scx.holding_cpu. When moving a task to a different rq:
*
- * 1. Verify task can be moved (CPU affinity, migration_disabled, etc.)
- * 2. Set p->scx.holding_cpu to the current CPU
- * 3. Set task state to %SCX_OPSS_NONE; dequeue waits while DISPATCHING
+ * 1. Set p->scx.holding_cpu to the current CPU
+ * 2. Set task state to %SCX_OPSS_NONE; dequeue waits while DISPATCHING
* is set, so clearing DISPATCHING first prevents the circular wait
* (safe to lock the rq we need)
- * 4. Unlock the current CPU's rq
- * 5. Lock src_rq (where the task currently lives)
- * 6. Verify p->scx.holding_cpu == current CPU, if not, dequeue won the
+ * 3. Unlock the current CPU's rq
+ * 4. Lock src_rq (where the task currently lives)
+ * 5. Verify p->scx.holding_cpu == current CPU, if not, dequeue won the
* race (dequeue clears holding_cpu to -1 when it takes the task), in
* this case migration is aborted
- * 7. If src_rq == dst_rq: clear holding_cpu and enqueue directly
+ * 6. If src_rq == dst_rq: clear holding_cpu and enqueue directly
* into dst_rq's local DSQ (no lock swap needed)
- * 8. Otherwise: call move_remote_task_to_local_dsq(), which releases
- * src_rq, locks dst_rq, and performs the deactivate/activate
- * migration cycle (dst_rq is held on return)
+ * 7. Otherwise, verify under src_rq lock that the task can be moved to dst_rq
+ * (CPU affinity, migration_disabled, etc.). If not, clear holding_cpu,
+ * leave the task on src_rq, and enqueue it on the fallback DSQ.
+ * 8. Otherwise (i.e. if the task can be moved to dst_rq), call
+ * move_remote_task_to_local_dsq(), which releases src_rq, locks dst_rq,
+ * and performs the deactivate/activate migration cycle
+ * (dst_rq is held on return)
* 9. Unlock dst_rq and re-lock the current CPU's rq to restore
* the lock state expected by the caller
*
rcu_read_lock();
for (;;) {
sighand = rcu_dereference(tsk->sighand);
- if (unlikely(sighand == NULL))
+ if (unlikely(sighand == NULL)) {
+ /*
+ * Pairs with the smp_store_release() in
+ * __exit_signal(). It ensures that all state
+ * modifications to the task preceeding the store are
+ * visible to the callers of lock_task_sighand().
+ */
+ smp_acquire__after_ctrl_dep();
break;
+ }
/*
* This sighand can be already freed and even reused, but
trigger_base_recalc_expires(timer, p);
}
+/*
+ * Lookup the task via timer->it.cpu.pid and attempt to lock the task's sighand.
+ *
+ * This can race with the reaping of the task:
+ *
+ * CPU0 CPU1
+ *
+ * // Finds task
+ * p = pid_task(pid, pid_type); __exit_signal(p)
+ * lock(p, sighand);
+ * posix_cpu_timers*_exit();
+ * sighand = lock_task_sighand(p); unhash_task(p);
+ * p->sighand = NULL;
+ * unlock(sighand);
+ *
+ * In this case sighand is NULL, which means the task and the associated timer
+ * queue cannot be longer accessed safely.
+ *
+ * __exit_signal() invokes posix_cpu_timers_exit() and if the thread group is
+ * dead it also invokes posix_cpu_timers_group_exit(). These functions delete
+ * all pending timers from the related timer queues. The POSIX timers (k_itimer)
+ * themself are still accessible, but not longer connected to the task.
+ *
+ * exec() works slightly differently. The task which exec()'s terminates all
+ * other threads in the thread group and runs __exit_signal() on them. As the
+ * thread group is not dead they only clean up the per task timers via
+ * posix_cpu_timers_exit().
+ *
+ * As the TGID on exec() stays the same per process timers stay queued, if they
+ * are armed. This works without a problem when exec() is done by the thread
+ * group leader. If a non-leader thread exec()'s this can end up in the
+ * following scenario:
+ *
+ * CPU0 CPU1
+ * // Returns old leader
+ * p = pid_task(pid, pid_type); de_thread()
+ * switch_leader()
+ * release_task(old leader)
+ * __exit_signal()
+ * old_leader->sighand = NULL;
+ * // Returns NULL
+ * sighand = lock_task_sighand(p)
+ *
+ * That's problematic for several functions:
+ *
+ * - posix_cpu_timer_del(): If the timer is still enqueued on the task the
+ * underlying k_itimer will be freed which results in a UAF in
+ * run_posix_cpu_timers() or on timerqueue related add/delete operations.
+ * If the timer is not enqueued, the failure is harmless
+ *
+ * - posix_cpu_timer_set(): Independent of the enqueued state that results in a
+ * transient failure which is user space visible (-ESRCH) for regular posix
+ * timers. But for the use case in do_cpu_nanosleep() it's the same UAF
+ * problem just that the timer is allocated on the stack.
+ *
+ * - posix_cpu_timer_rearm(): Timer is not enqueued at that point, but this
+ * silently ignores the rearm request, which is a functional problem as the
+ * timer wont expire anymore.
+ */
+static struct task_struct *timer_lock_sighand(struct k_itimer *timer, unsigned long *flags)
+{
+ enum pid_type type = clock_pid_type(timer->it_clock);
+ struct cpu_timer *ctmr = &timer->it.cpu;
+
+ guard(rcu)();
+
+ for (;;) {
+ struct task_struct *t = pid_task(timer->it.cpu.pid, type);
+
+ /* Fail if the task cannot be found. */
+ if (!t)
+ break;
+
+ /* Try to lock the task's sighand */
+ if (lock_task_sighand(t, flags))
+ return t;
+
+ /*
+ * The next PID lookup might either fail or return the new
+ * leader. This is correct for both exit() and exec().
+ */
+ }
+
+ /*
+ * If the timer is still enqueued, warn. There is nothing safe to do
+ * here as there might be two timers in there which are removed in
+ * parallel and that will cause more damage than good. This should never
+ * happen!
+ *
+ * Ensure that the stores to the timer and timerqueue are visible:
+ *
+ * __exit_signal()
+ * posix_cpu_timers*_exit()
+ * write_seqlock(seqlock)
+ * smp_wmb(); <-------
+ * __unhash_process() | !pid_task()
+ * ----> smp_rmb();
+ * WARN_ON_ONCE(...)
+ */
+ smp_rmb();
+ WARN_ON_ONCE(ctmr->head || timerqueue_node_queued(&ctmr->node));
+ return NULL;
+}
/*
* Clean up a CPU-clock timer that is about to be destroyed.
*/
static int posix_cpu_timer_del(struct k_itimer *timer)
{
- struct cpu_timer *ctmr = &timer->it.cpu;
- struct sighand_struct *sighand;
struct task_struct *p;
unsigned long flags;
int ret = 0;
- rcu_read_lock();
- p = cpu_timer_task_rcu(timer);
- if (!p)
- goto out;
+ p = timer_lock_sighand(timer, &flags);
- /*
- * Protect against sighand release/switch in exit/exec and process/
- * thread timer list entry concurrent read/writes.
- */
- sighand = lock_task_sighand(p, &flags);
- if (unlikely(sighand == NULL)) {
- /*
- * This raced with the reaping of the task. The exit cleanup
- * should have removed this timer from the timer queue.
- */
- WARN_ON_ONCE(ctmr->head || timerqueue_node_queued(&ctmr->node));
- } else {
+ if (likely(p)) {
if (timer->it.cpu.firing) {
/*
* Prevent signal delivery. The timer cannot be dequeued
unlock_task_sighand(p, &flags);
}
-out:
- rcu_read_unlock();
-
if (!ret) {
- put_pid(ctmr->pid);
+ put_pid(timer->it.cpu.pid);
timer->it_status = POSIX_TIMER_DISARMED;
}
return ret;
clockid_t clkid = CPUCLOCK_WHICH(timer->it_clock);
struct cpu_timer *ctmr = &timer->it.cpu;
u64 old_expires, new_expires, now;
- struct sighand_struct *sighand;
struct task_struct *p;
unsigned long flags;
int ret = 0;
- rcu_read_lock();
- p = cpu_timer_task_rcu(timer);
- if (!p) {
- /*
- * If p has just been reaped, we can no
- * longer get any information about it at all.
- */
- rcu_read_unlock();
+ p = timer_lock_sighand(timer, &flags);
+ /*
+ * If p has just been reaped, we can no longer get any information about
+ * it at all.
+ */
+ if (!p)
return -ESRCH;
- }
/*
* Use the to_ktime conversion because that clamps the maximum
*/
new_expires = ktime_to_ns(timespec64_to_ktime(new->it_value));
- /*
- * Protect against sighand release/switch in exit/exec and p->cpu_timers
- * and p->signal->cpu_timers read/write in arm_timer()
- */
- sighand = lock_task_sighand(p, &flags);
- /*
- * If p has just been reaped, we can no
- * longer get any information about it at all.
- */
- if (unlikely(sighand == NULL)) {
- rcu_read_unlock();
- return -ESRCH;
- }
-
/* Retrieve the current expiry time before disarming the timer */
old_expires = cpu_timer_getexpires(ctmr);
/* Retry if the timer expiry is running concurrently */
if (unlikely(ret)) {
unlock_task_sighand(p, &flags);
- goto out;
+ return ret;
}
/* Convert relative expiry time to absolute */
*/
if (!sigev_none && new_expires && now >= new_expires)
cpu_timer_fire(timer);
-out:
- rcu_read_unlock();
return ret;
}
static bool posix_cpu_timer_rearm(struct k_itimer *timer)
{
clockid_t clkid = CPUCLOCK_WHICH(timer->it_clock);
- struct sighand_struct *sighand;
struct task_struct *p;
unsigned long flags;
u64 now;
- guard(rcu)();
- p = cpu_timer_task_rcu(timer);
- if (!p)
- return true;
-
- /* Protect timer list r/w in arm_timer() */
- sighand = lock_task_sighand(p, &flags);
- if (unlikely(sighand == NULL))
+ p = timer_lock_sighand(timer, &flags);
+ if (unlikely(!p))
return true;
/*
if (event->type_len > RINGBUF_TYPE_DATA_TYPE_LEN_MAX)
return length;
length -= RB_EVNT_HDR_SIZE;
- if (length > RB_MAX_SMALL_DATA + sizeof(event->array[0]))
+ if (length > RB_MAX_SMALL_DATA + sizeof(event->array[0]) ||
+ RB_FORCE_8BYTE_ALIGNMENT)
length -= sizeof(event->array[0]);
return length;
}
size_t len;
int i;
- if (!trace_desc)
- return NULL;
-
- if (cpu >= trace_desc->nr_cpus)
+ if (!trace_desc || !trace_desc->nr_cpus)
return NULL;
end = (struct ring_buffer_desc *)((void *)trace_desc + trace_desc->struct_len);
rpos = reader->read;
pos += event_size;
- if (rpos >= event_size)
+ if (rpos >= size)
break;
event = rb_reader_event(cpu_buffer);
/* Pipe tracepoints to printk */
static struct trace_iterator *tracepoint_print_iter;
-int tracepoint_printk;
+static int tracepoint_printk;
static bool tracepoint_printk_stop_on_boot __initdata;
static bool traceoff_after_boot __initdata;
static DEFINE_STATIC_KEY_FALSE(tracepoint_printk_key);
RING_BUFFER_ALL_CPUS);
if (ret < 0)
return ret;
- ret = 0;
}
list_for_each_entry(t, &tr->tracers, list) {
if (!slash)
return false;
- if (strncmp(ep->event_system, argv[0], slash - argv[0]))
+ if (strncmp(ep->event_system, argv[0], slash - argv[0]) ||
+ ep->event_system[slash - argv[0]] != '\0')
return false;
if (strcmp(ep->event_name, slash + 1))
return false;
return 0;
}
-static int regex_match_glob(char *str, struct regex *r, int len __maybe_unused)
+static int regex_match_glob(char *str, struct regex *r, int len)
{
- if (glob_match(r->pattern, str))
- return 1;
- return 0;
+ return glob_match_len(r->pattern, str, len) ? 1 : 0;
}
/**
seq_buf_puts(&s, "__data_loc ");
seq_buf_puts(&s, field->type);
- if (WARN_ON_ONCE(!seq_buf_buffer_left(&s)))
+ if (WARN_ON_ONCE(!seq_buf_buffer_left(&s))) {
+ kfree(type);
goto free;
+ }
s.buffer[s.len] = '\0';
kfree(field->type);
if (cmd_version > 1 && n_fields_this_loop >= 1) {
synth_err(SYNTH_ERR_INVALID_CMD, errpos(field_str));
ret = -EINVAL;
- goto err_free_arg;
+ goto err_free_field;
}
if (n_fields == SYNTH_FIELDS_MAX) {
synth_err(SYNTH_ERR_TOO_MANY_FIELDS, 0);
ret = -EINVAL;
- goto err_free_arg;
+ goto err_free_field;
}
fields[n_fields++] = field;
kfree(saved_fields);
return ret;
+ err_free_field:
+ free_synth_field(field);
err_free_arg:
argv_free(argv);
err:
/* Track enable bit, flags, etc. Aligned for bitops. */
unsigned long values;
+
+ /* Defer the event put and enabler free past an RCU grace period. */
+ struct rcu_work put_rwork;
};
/* Bits 0-5 are for the bit to update upon enable/disable (0-63 allowed) */
return NULL;
};
-static void user_event_enabler_destroy(struct user_event_enabler *enabler,
- bool locked)
+static void delayed_user_event_enabler_put(struct work_struct *work)
{
- list_del_rcu(&enabler->mm_enablers_link);
+ struct user_event_enabler *enabler = container_of(to_rcu_work(work),
+ struct user_event_enabler, put_rwork);
/* No longer tracking the event via the enabler */
- user_event_put(enabler->event, locked);
+ user_event_put(enabler->event, false);
+ /* Run from queue_rcu_work(), the RCU grace period has elapsed */
kfree(enabler);
}
+static void user_event_enabler_destroy(struct user_event_enabler *enabler)
+{
+ list_del_rcu(&enabler->mm_enablers_link);
+
+ /*
+ * The enabler is removed from an RCU-traversed list
+ * (user_event_mm_dup() walks mm->enablers under rcu_read_lock() only),
+ * and readers there dereference enabler->event and take a new ref on
+ * it. Both the put of that event reference and the free of the enabler
+ * therefore have to wait for a grace period so no reader can be looking
+ * at the enabler or racing the last put of its event.
+ *
+ * The put itself must not run in RCU context: when it drops the last
+ * reference user_event_put() takes event_mutex, which cannot be taken
+ * from a softirq/RCU callback. Defer both to a work item scheduled
+ * after a grace period via queue_rcu_work().
+ */
+ INIT_RCU_WORK(&enabler->put_rwork, delayed_user_event_enabler_put);
+ queue_rcu_work(system_percpu_wq, &enabler->put_rwork);
+}
+
static int user_event_mm_fault_in(struct user_event_mm *mm, unsigned long uaddr,
int attempt)
{
/* User asked for enabler to be removed during fault */
if (test_bit(ENABLE_VAL_FREEING_BIT, ENABLE_BITOPS(enabler))) {
- user_event_enabler_destroy(enabler, true);
+ user_event_enabler_destroy(enabler);
goto out;
}
struct user_event_enabler *enabler, *next;
list_for_each_entry_safe(enabler, next, &mm->enablers, mm_enablers_link)
- user_event_enabler_destroy(enabler, false);
+ user_event_enabler_destroy(enabler);
mmdrop(mm->mm);
kfree(mm);
flags |= enabler->values & ENABLE_VAL_COMPAT_MASK;
if (!test_bit(ENABLE_VAL_FAULTING_BIT, ENABLE_BITOPS(enabler)))
- user_event_enabler_destroy(enabler, true);
+ user_event_enabler_destroy(enabler);
/* Removed at least one */
ret = 0;
ftrace_func_t func;
u32 new_flags;
- /* Do nothing if already set. */
- if (!!set == !!(tr->current_trace_flags->val & bit))
+ /* We can change this flag only when current tracer is function. */
+ if (tr->current_trace != &function_trace)
return 0;
- /* We can change this flag only when not running. */
- if (tr->current_trace != &function_trace)
+ /* Do nothing if already set. */
+ if (!!set == !!(tr->current_trace_flags->val & bit))
return 0;
new_flags = (tr->current_trace_flags->val & ~bit) | (set ? bit : 0);
if (!found)
return;
- kvfree_rcu_mightsleep(inst);
+ /* Do a full sync to ensure that tr remains valid, not just inst */
+ synchronize_rcu();
+ kvfree(inst);
}
/*
#else
#define trace(point, args) \
do { \
- if (trace_##point##_enabled()) { \
+ if (__trace_##point##_enabled()) { \
bool exit_rcu = false; \
if (in_nmi()) \
break; \
lockdep_assert_held(&dyn_event_ops_mutex);
- if (!trace_probe_log.argv)
+ if (!trace_probe_log.argv || !trace_probe_log.argc)
return;
/* Recalculate the length and allocate buffer */
}
/* When len=0, we just calculate the needed length */
-#define LEN_OR_ZERO (len ? len - pos : 0)
+#define LEN_OR_ZERO (len > pos ? len - pos : 0)
static int __set_print_fmt(struct trace_probe *tp, char *buf, int len,
enum probe_print_type ptype)
{
bool trace_probe_match_command_args(struct trace_probe *tp,
int argc, const char **argv)
{
- char buf[MAX_ARGSTR_LEN + 1];
int i;
if (tp->nr_args < argc)
return false;
for (i = 0; i < argc; i++) {
- snprintf(buf, sizeof(buf), "%s=%s",
- tp->args[i].name, tp->args[i].comm);
- if (strcmp(buf, argv[i]))
+ int len = strlen(tp->args[i].name);
+
+ if (strncmp(argv[i], tp->args[i].name, len) ||
+ argv[i][len] != '=' ||
+ strcmp(argv[i] + len + 1, tp->args[i].comm))
return false;
}
return true;
int trace_remote_alloc_buffer(struct trace_buffer_desc *desc, size_t desc_size, size_t buffer_size,
const struct cpumask *cpumask)
{
+ size_t min_desc_size = trace_buffer_desc_size(buffer_size, cpumask_weight(cpumask));
unsigned int nr_pages = max(DIV_ROUND_UP(buffer_size, PAGE_SIZE), 2UL) + 1;
- void *desc_end = desc + desc_size;
struct ring_buffer_desc *rb_desc;
int cpu, ret = -ENOMEM;
- if (desc_size < struct_size(desc, __data, 0))
+ if (desc_size < min_desc_size)
return -EINVAL;
desc->nr_cpus = 0;
- desc->struct_len = struct_size(desc, __data, 0);
+ desc->struct_len = min_desc_size;
- rb_desc = (struct ring_buffer_desc *)&desc->__data[0];
+ rb_desc = __first_ring_buffer_desc(desc);
for_each_cpu(cpu, cpumask) {
unsigned int id;
- if ((void *)rb_desc + struct_size(rb_desc, page_va, nr_pages) > desc_end) {
- ret = -EINVAL;
- goto err;
- }
-
rb_desc->cpu = cpu;
rb_desc->nr_page_va = 0;
rb_desc->meta_va = (unsigned long)__get_free_page(GFP_KERNEL);
if (!rb_desc->meta_va)
goto err;
+ desc->nr_cpus++;
+
for (id = 0; id < nr_pages; id++) {
rb_desc->page_va[id] = (unsigned long)__get_free_page(GFP_KERNEL);
if (!rb_desc->page_va[id])
rb_desc->nr_page_va++;
}
- desc->nr_cpus++;
- desc->struct_len += offsetof(struct ring_buffer_desc, page_va);
- desc->struct_len += struct_size(rb_desc, page_va, rb_desc->nr_page_va);
rb_desc = __next_ring_buffer_desc(rb_desc);
}
no_cut = bug->flags & BUGFLAG_NO_CUT_HERE;
has_args = bug->flags & BUGFLAG_ARGS;
-#ifdef CONFIG_KUNIT
/*
* Before the once logic so suppressed warnings do not consume
* the single-fire budget of WARN_ON_ONCE().
*/
if (warning && kunit_is_suppressed_warning(true))
return BUG_TRAP_TYPE_WARN;
-#endif
disable_trace_on_warning();
config CRYPTO_LIB_AES
tristate
- # Select dependencies of modes that are part of libaes.
- select CRYPTO_LIB_UTILS if CRYPTO_LIB_AES_CBC_MACS
+ select CRYPTO_LIB_UTILS
config CRYPTO_LIB_AES_ARCH
bool
}
EXPORT_SYMBOL_GPL(hmac_md5_usingrawkey);
-#ifdef md5_mod_init_arch
-static int __init md5_mod_init(void)
-{
- md5_mod_init_arch();
- return 0;
-}
-subsys_initcall(md5_mod_init);
-
-static void __exit md5_mod_exit(void)
-{
-}
-module_exit(md5_mod_exit);
-#endif
-
MODULE_DESCRIPTION("MD5 and HMAC-MD5 library functions");
MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("glob(7) matching");
MODULE_LICENSE("Dual MIT/GPL");
+static bool __pure glob_match_str(char const *pat, char const *str,
+ char const *str_end);
+
/**
* glob_match - Shell-style pattern matching, like !fnmatch(pat, str, 0)
* @pat: Shell-style pattern to match, e.g. "*.[ch]".
* An opening bracket without a matching close is matched literally.
*/
bool __pure glob_match(char const *pat, char const *str)
+{
+ return glob_match_str(pat, str, NULL);
+}
+EXPORT_SYMBOL(glob_match);
+
+/**
+ * glob_match_len - glob match against a length-bounded string
+ * @pat: Shell-style pattern to match.
+ * @str: String to match. Need not be NUL-terminated.
+ * @len: Number of bytes of @str that may be read.
+ *
+ * Like glob_match(), but @str is only read up to @len bytes, so it can be
+ * used on buffers that are not NUL-terminated (e.g. trace event fields).
+ * A NUL byte within @len still terminates the string.
+ */
+bool __pure glob_match_len(char const *pat, char const *str, size_t len)
+{
+ return glob_match_str(pat, str, str + len);
+}
+EXPORT_SYMBOL(glob_match_len);
+
+static bool __pure glob_match_str(char const *pat, char const *str,
+ char const *str_end)
{
/*
* Backtrack to previous * on mismatch and retry starting one
* on mismatch, or true after matching the trailing nul bytes.
*/
for (;;) {
- unsigned char c = *str++;
+ unsigned char c = (str_end && str >= str_end) ? '\0' : *str;
unsigned char d = *pat++;
+ str++;
+
switch (d) {
case '?': /* Wildcard: anything but nul */
if (c == '\0')
}
}
}
-EXPORT_SYMBOL(glob_match);
iter->walker.tbl = rht_dereference_rcu(ht->tbl, ht);
iter->slot = 0;
iter->skip = 0;
+ iter->p = NULL;
return -EAGAIN;
}
int order = folio_order(dst);
struct page *page = &dst->page;
- if (folio_put_testzero(dst)) {
- free_pages_prepare(page, order);
+ if (folio_put_testzero(dst) && free_pages_prepare(page, order)) {
list_add(&dst->lru, &cc->freepages[order]);
cc->nr_freepages += 1 << order;
}
cc->nr_migratepages += 1 << order;
/*
- * someone else has referenced the page, we cannot take it back to our
- * free list.
+ * someone else has referenced the page or free_pages_prepare() fails,
+ * we cannot take it back to our free list.
*/
}
// SPDX-License-Identifier: GPL-2.0
/*
* Data Access Monitor
- *
- * Author: SeongJae Park <sj@kernel.org>
*/
#define pr_fmt(fmt) "damon: " fmt
{
struct damon_region *r, *next;
unsigned int i;
+ unsigned long last_end;
int err;
+ for (i = 0; i < nr_ranges; i++) {
+ unsigned long start, end;
+
+ start = ALIGN_DOWN(ranges[i].start, min_region_sz);
+ end = ALIGN(ranges[i].end, min_region_sz);
+ if (start >= end)
+ return -EINVAL;
+ if (i > 0 && last_end > start)
+ return -EINVAL;
+ last_end = end;
+ }
+
/* Remove regions which are not in the new ranges */
damon_for_each_region_safe(r, next, t) {
for (i = 0; i < nr_ranges; i++) {
// SPDX-License-Identifier: GPL-2.0
/*
* DAMON-based LRU-lists Sorting
- *
- * Author: SeongJae Park <sj@kernel.org>
*/
#define pr_fmt(fmt) "damon-lru-sort: " fmt
// SPDX-License-Identifier: GPL-2.0
/*
* Common Code for DAMON Modules
- *
- * Author: SeongJae Park <sj@kernel.org>
*/
#include <linux/damon.h>
/* SPDX-License-Identifier: GPL-2.0 */
/*
* Common Code for DAMON Modules
- *
- * Author: SeongJae Park <sj@kernel.org>
*/
#include <linux/moduleparam.h>
// SPDX-License-Identifier: GPL-2.0
/*
* Common Code for Data Access Monitoring
- *
- * Author: SeongJae Park <sj@kernel.org>
*/
#include <linux/migrate.h>
* Transform it to fit in [0, DAMOS_MAX_SCORE]
*/
hotness = hotness * DAMOS_MAX_SCORE / DAMON_MAX_SUBSCORE;
+ hotness = max(min(hotness, DAMOS_MAX_SCORE), 0);
return hotness;
}
/* SPDX-License-Identifier: GPL-2.0 */
/*
* Common Code for Data Access Monitoring
- *
- * Author: SeongJae Park <sj@kernel.org>
*/
#include <linux/damon.h>
// SPDX-License-Identifier: GPL-2.0
/*
* DAMON Code for The Physical Address Space
- *
- * Author: SeongJae Park <sj@kernel.org>
*/
#define pr_fmt(fmt) "damon-pa: " fmt
// SPDX-License-Identifier: GPL-2.0
/*
* DAMON-based page reclamation
- *
- * Author: SeongJae Park <sj@kernel.org>
*/
#define pr_fmt(fmt) "damon-reclaim: " fmt
// SPDX-License-Identifier: GPL-2.0
/*
* Common Code for DAMON Sysfs Interface
- *
- * Author: SeongJae Park <sj@kernel.org>
*/
#include <linux/slab.h>
/* SPDX-License-Identifier: GPL-2.0 */
/*
* Common Code for DAMON Sysfs Interface
- *
- * Author: SeongJae Park <sj@kernel.org>
*/
#include <linux/damon.h>
// SPDX-License-Identifier: GPL-2.0
/*
* DAMON sysfs Interface
- *
- * Copyright (c) 2022 SeongJae Park <sj@kernel.org>
*/
#include <linux/slab.h>
err = damon_sysfs_access_pattern_add_range_dir(access_pattern,
&access_pattern->sz, "sz");
if (err)
- goto put_sz_out;
+ return err;
err = damon_sysfs_access_pattern_add_range_dir(access_pattern,
&access_pattern->nr_accesses, "nr_accesses");
if (err)
- goto put_nr_accesses_sz_out;
+ goto put_sz_out;
err = damon_sysfs_access_pattern_add_range_dir(access_pattern,
&access_pattern->age, "age");
if (err)
- goto put_age_nr_accesses_sz_out;
+ goto put_nr_accesses_sz_out;
return 0;
-put_age_nr_accesses_sz_out:
- kobject_put(&access_pattern->age->kobj);
- access_pattern->age = NULL;
put_nr_accesses_sz_out:
kobject_put(&access_pattern->nr_accesses->kobj);
access_pattern->nr_accesses = NULL;
goto put_filters_watermarks_quotas_access_pattern_out;
err = damon_sysfs_scheme_set_tried_regions(scheme);
if (err)
- goto put_tried_regions_out;
+ goto put_stats_out;
return 0;
-put_tried_regions_out:
- kobject_put(&scheme->tried_regions->kobj);
- scheme->tried_regions = NULL;
+put_stats_out:
+ kobject_put(&scheme->stats->kobj);
+ scheme->stats = NULL;
put_filters_watermarks_quotas_access_pattern_out:
kobject_put(&scheme->ops_filters->kobj);
scheme->ops_filters = NULL;
// SPDX-License-Identifier: GPL-2.0
/*
* DAMON sysfs Interface
- *
- * Copyright (c) 2022 SeongJae Park <sj@kernel.org>
*/
#include <linux/pid.h>
/* SPDX-License-Identifier: GPL-2.0 */
/*
* Data Access Monitor Unit Tests
- *
- * Copyright 2019 Amazon.com, Inc. or its affiliates. All rights reserved.
- *
- * Author: SeongJae Park <sj@kernel.org>
*/
#ifdef CONFIG_DAMON_KUNIT_TEST
/* SPDX-License-Identifier: GPL-2.0 */
/*
* Data Access Monitor Unit Tests
- *
- * Author: SeongJae Park <sj@kernel.org>
*/
#ifdef CONFIG_DAMON_SYSFS_KUNIT_TEST
/* SPDX-License-Identifier: GPL-2.0 */
/*
* Data Access Monitor Unit Tests
- *
- * Copyright 2019 Amazon.com, Inc. or its affiliates. All rights reserved.
- *
- * Author: SeongJae Park <sj@kernel.org>
*/
#ifdef CONFIG_DAMON_VADDR_KUNIT_TEST
// SPDX-License-Identifier: GPL-2.0
/*
* DAMON Code for Virtual Address Spaces
- *
- * Author: SeongJae Park <sj@kernel.org>
*/
#define pr_fmt(fmt) "damon-va: " fmt
{
if (f->f_mode & FMODE_WRITE)
return true;
- if (inode_owner_or_capable(file_mnt_idmap(f), file_inode(f)))
+ if (file_owner_or_capable(f))
return true;
return file_permission(f, MAY_WRITE) == 0;
}
(1L << PG_dropbehind) |
LRU_GEN_MASK | LRU_REFS_MASK));
- if (handle_hwpoison &&
- page_range_has_hwpoisoned(new_head, new_nr_pages))
- folio_set_has_hwpoisoned(new_folio);
-
new_folio->mapping = folio->mapping;
new_folio->index = folio->index + i;
folio_set_large_rmappable(new_folio);
}
+ /*
+ * PG_has_hwpoisoned is on the 2nd page, so set it after
+ * the compound head is prepped.
+ */
+ if (handle_hwpoison &&
+ page_range_has_hwpoisoned(new_head, new_nr_pages))
+ folio_set_has_hwpoisoned(new_folio);
+
if (folio_test_young(folio))
folio_set_young(new_folio);
if (folio_test_idle(folio))
atomic_set(&object->use_count, 1);
object->excess_ref = 0;
object->count = 0; /* white color initially */
- object->checksum = 0;
+ object->checksum = ~0;
object->del_state = 0;
/* task information */
}
raw_spin_lock_irqsave(&object->lock, flags);
- object->checksum = 0;
+ object->checksum = ~0;
raw_spin_unlock_irqrestore(&object->lock, flags);
put_object(object);
}
for_each_possible_cpu(cpu) {
void *ptr = per_cpu_ptr((void __percpu *)object->pointer, cpu);
- object->checksum ^= crc32(0, kasan_reset_tag((void *)ptr), object->size);
+ object->checksum = crc32(object->checksum,
+ kasan_reset_tag((void *)ptr), object->size);
}
} else {
object->checksum = crc32(0, kasan_reset_tag((void *)object->pointer), object->size);
* otherwise we'd be including shared non-exclusive mappings, which
* opens a side channel.
*/
- return inode_owner_or_capable(&nop_mnt_idmap,
- file_inode(vma->vm_file)) ||
+ return file_owner_or_capable(vma->vm_file) ||
file_permission(vma->vm_file, MAY_WRITE) == 0;
}
* for writing; otherwise we'd be including shared non-exclusive
* mappings, which opens a side channel.
*/
- return inode_owner_or_capable(&nop_mnt_idmap,
- file_inode(vma->vm_file)) ||
+ return file_owner_or_capable(vma->vm_file) ||
file_permission(vma->vm_file, MAY_WRITE) == 0;
}
* any pages that may have already been present from the previous
* list processed. This should result in us reporting all pages on
* an idle system in about 30 seconds.
- *
- * The division here should be cheap since PAGE_REPORTING_CAPACITY
- * should always be a power of 2.
*/
- budget = DIV_ROUND_UP(area->nr_free, PAGE_REPORTING_CAPACITY * 16);
+ budget = DIV_ROUND_UP(area->nr_free, prdev->capacity * 16);
/* loop through free list adding unreported pages to sg list */
list_for_each_entry_safe(page, next, list, lru) {
spin_unlock_irq(&zone->lock);
/* begin processing pages in local list */
- err = prdev->report(prdev, sgl, PAGE_REPORTING_CAPACITY);
+ err = prdev->report(prdev, sgl, prdev->capacity);
/* reset offset since the full list was reported */
- *offset = PAGE_REPORTING_CAPACITY;
+ *offset = prdev->capacity;
/* update budget to reflect call to report function */
budget--;
spin_lock_irq(&zone->lock);
/* flush reported pages from the sg list */
- page_reporting_drain(prdev, sgl, PAGE_REPORTING_CAPACITY, !err);
+ page_reporting_drain(prdev, sgl, prdev->capacity, !err);
/*
* Reset next to first entry, the old next isn't valid
page_reporting_process_zone(struct page_reporting_dev_info *prdev,
struct scatterlist *sgl, struct zone *zone)
{
- unsigned int order, mt, leftover, offset = PAGE_REPORTING_CAPACITY;
+ unsigned int order, mt, leftover, offset = prdev->capacity;
unsigned long watermark;
int err = 0;
/* Generate minimum watermark to be able to guarantee progress */
watermark = low_wmark_pages(zone) +
- (PAGE_REPORTING_CAPACITY << page_reporting_order);
+ (prdev->capacity << page_reporting_order);
/*
* Cancel request if insufficient free memory or if we failed
}
/* report the leftover pages before going idle */
- leftover = PAGE_REPORTING_CAPACITY - offset;
+ leftover = prdev->capacity - offset;
if (leftover) {
sgl = &sgl[offset];
err = prdev->report(prdev, sgl, leftover);
atomic_set(&prdev->state, state);
/* allocate scatterlist to store pages being reported on */
- sgl = kmalloc_objs(*sgl, PAGE_REPORTING_CAPACITY);
+ sgl = kmalloc_objs(*sgl, prdev->capacity);
if (!sgl)
goto err_out;
- sg_init_table(sgl, PAGE_REPORTING_CAPACITY);
+ sg_init_table(sgl, prdev->capacity);
for_each_zone(zone) {
err = page_reporting_process_zone(prdev, sgl, zone);
page_reporting_order = pageblock_order;
}
+ if (!prdev->capacity || prdev->capacity > PAGE_REPORTING_CAPACITY)
+ prdev->capacity = PAGE_REPORTING_CAPACITY;
+
/* initialize state and work structures */
atomic_set(&prdev->state, PAGE_REPORTING_IDLE);
INIT_DELAYED_WORK(&prdev->work, &page_reporting_process);
*/
pmde = pmdp_get_lockless(pvmw->pmd);
- if (pmd_trans_huge(pmde) || pmd_is_migration_entry(pmde)) {
+ if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) &&
+ (pmd_trans_huge(pmde) || pmd_is_migration_entry(pmde) ||
+ pmd_is_device_private_entry(pmde))) {
pvmw->ptl = pmd_lock(mm, pvmw->pmd);
pmde = *pvmw->pmd;
- if (!pmd_present(pmde)) {
+ if (pmd_is_migration_entry(pmde)) {
softleaf_t entry;
- if (!thp_migration_supported() ||
- !(pvmw->flags & PVMW_MIGRATION))
+ if (!(pvmw->flags & PVMW_MIGRATION))
return not_found(pvmw);
entry = softleaf_from_pmd(pmde);
+ if (!check_pmd(softleaf_to_pfn(entry), pvmw))
+ return not_found(pvmw);
+ return true;
+ } else if (pmd_is_device_private_entry(pmde)) {
+ softleaf_t entry;
- if (!softleaf_is_migration(entry) ||
- !check_pmd(softleaf_to_pfn(entry), pvmw))
+ if (pvmw->flags & PVMW_MIGRATION)
+ return not_found(pvmw);
+ entry = softleaf_from_pmd(pmde);
+ if (!check_pmd(softleaf_to_pfn(entry), pvmw))
return not_found(pvmw);
return true;
+ } else if (!pmd_present(pmde)) {
+ return not_found(pvmw);
}
if (likely(pmd_trans_huge(pmde))) {
if (pvmw->flags & PVMW_MIGRATION)
return not_found(pvmw);
return true;
}
- /* THP pmd was split under us: handle on pte level */
+ /* THP/device-private pmd was split under us: handle on pte level */
spin_unlock(pvmw->ptl);
pvmw->ptl = NULL;
} else if (!pmd_present(pmde)) {
- const softleaf_t entry = softleaf_from_pmd(pmde);
-
- if (softleaf_is_device_private(entry)) {
- pvmw->ptl = pmd_lock(mm, pvmw->pmd);
- return true;
- }
-
if ((pvmw->flags & PVMW_SYNC) &&
thp_vma_suitable_order(vma, pvmw->address,
PMD_ORDER) &&
return 0;
}
-void free_shrinker_info(struct mem_cgroup *memcg)
+static void __free_shrinker_info(struct mem_cgroup *memcg)
{
struct mem_cgroup_per_node *pn;
struct shrinker_info *info;
int nid;
+ lockdep_assert_held(&shrinker_mutex);
+
for_each_node(nid) {
pn = memcg->nodeinfo[nid];
info = rcu_dereference_protected(pn->shrinker_info, true);
}
}
+void free_shrinker_info(struct mem_cgroup *memcg)
+{
+ mutex_lock(&shrinker_mutex);
+ __free_shrinker_info(memcg);
+ mutex_unlock(&shrinker_mutex);
+}
+
int alloc_shrinker_info(struct mem_cgroup *memcg)
{
int nid, ret = 0;
return ret;
err:
+ __free_shrinker_info(memcg);
mutex_unlock(&shrinker_mutex);
- free_shrinker_info(memcg);
return -ENOMEM;
}
}
shrinker->debugfs_entry = entry;
- debugfs_create_file("count", 0440, entry, shrinker,
- &shrinker_debugfs_count_fops);
- debugfs_create_file("scan", 0220, entry, shrinker,
- &shrinker_debugfs_scan_fops);
+ if (shrinker->count_objects)
+ debugfs_create_file("count", 0440, entry, shrinker,
+ &shrinker_debugfs_count_fops);
+ if (shrinker->scan_objects)
+ debugfs_create_file("scan", 0220, entry, shrinker,
+ &shrinker_debugfs_scan_fops);
return 0;
}
if (r < 0)
return NULL;
+ flush_cache_vmap(start, end);
+
return pfn_to_page(pfn);
}
{
const struct vm_uffd_ops *ops = vma_uffd_ops(vma);
- if (vma->vm_flags & VM_DROPPABLE)
+ if (vma->vm_flags & (VM_DROPPABLE | VM_SHADOW_STACK))
+ return false;
+
+ if (!is_vm_hugetlb_page(vma) && (vma->vm_flags & VM_SPECIAL))
return false;
vm_flags &= __VM_UFFD_FLAGS;
* the one with the lowest address
*/
if (tmp_max == max && max_orig_node &&
- batadv_compare_eth(candidate->orig, max_orig_node->orig))
+ memcmp(candidate->orig, max_orig_node->orig, ETH_ALEN) >= 0)
goto out;
ret = true;
mtu = min_t(unsigned int, mtu, BATADV_FRAG_MAX_FRAG_SIZE);
max_fragment_size = mtu - header_size;
- if (skb->len == 0 || max_fragment_size == 0)
- return -EINVAL;
+ if (skb->len == 0 || max_fragment_size == 0) {
+ ret = -EINVAL;
+ goto free_skb;
+ }
num_fragments = (skb->len - 1) / max_fragment_size + 1;
max_fragment_size = (skb->len - 1) / num_fragments + 1;
*/
if (skb_has_frag_list(skb) && __skb_linearize(skb)) {
ret = -ENOMEM;
- goto free_skb;
+ goto put_primary_if;
}
/* Create one header to be copied to all fragments */
void batadv_mesh_free(struct net_device *mesh_iface)
{
struct batadv_priv *bat_priv = netdev_priv(mesh_iface);
+ struct batadv_meshif_vlan *vlan;
WRITE_ONCE(bat_priv->mesh_state, BATADV_MESH_DEACTIVATING);
batadv_mcast_free(bat_priv);
+ /* destroy the "untagged" VLAN */
+ vlan = batadv_meshif_vlan_get(bat_priv, BATADV_NO_FLAGS);
+ if (vlan) {
+ batadv_meshif_destroy_vlan(bat_priv, vlan);
+ batadv_meshif_vlan_put(vlan);
+ }
+
/* Free the TT and the originator tables only after having terminated
* all the other depending components which may use these structures for
* their purposes.
switch (ethhdr->h_proto) {
case htons(ETH_P_8021Q):
- vhdr = skb_header_pointer(skb, offset + sizeof(*vhdr),
+ vhdr = skb_header_pointer(skb, offset,
sizeof(*vhdr), &vhdr_tmp);
if (!vhdr)
return;
if (READ_ONCE(bat_priv->mesh_state) != BATADV_MESH_ACTIVE)
goto dropped;
+ if (!pskb_may_pull(skb, ETH_HLEN))
+ goto dropped;
+
/* reset control block to avoid left overs from previous users */
memset(skb->cb, 0, sizeof(struct batadv_skb_cb));
* @bat_priv: the bat priv with all the mesh interface information
* @vlan: the object to remove
*/
-static void batadv_meshif_destroy_vlan(struct batadv_priv *bat_priv,
- struct batadv_meshif_vlan *vlan)
+void batadv_meshif_destroy_vlan(struct batadv_priv *bat_priv,
+ struct batadv_meshif_vlan *vlan)
{
/* explicitly remove the associated TT local entry because it is marked
* with the NOPURGE flag
static void batadv_meshif_destroy_netlink(struct net_device *mesh_iface,
struct list_head *head)
{
- struct batadv_priv *bat_priv = netdev_priv(mesh_iface);
struct batadv_hard_iface *hard_iface;
- struct batadv_meshif_vlan *vlan;
while (!list_empty(&mesh_iface->adj_list.lower)) {
hard_iface = netdev_adjacent_get_private(mesh_iface->adj_list.lower.next);
batadv_hardif_disable_interface(hard_iface);
}
- /* destroy the "untagged" VLAN */
- vlan = batadv_meshif_vlan_get(bat_priv, BATADV_NO_FLAGS);
- if (vlan) {
- batadv_meshif_destroy_vlan(bat_priv, vlan);
- batadv_meshif_vlan_put(vlan);
- }
-
unregister_netdevice_queue(mesh_iface, head);
}
bool batadv_meshif_is_valid(const struct net_device *net_dev);
extern struct rtnl_link_ops batadv_link_ops;
int batadv_meshif_create_vlan(struct batadv_priv *bat_priv, unsigned short vid);
+void batadv_meshif_destroy_vlan(struct batadv_priv *bat_priv,
+ struct batadv_meshif_vlan *vlan);
void batadv_meshif_vlan_release(struct kref *ref);
struct batadv_meshif_vlan *batadv_meshif_vlan_get(struct batadv_priv *bat_priv,
unsigned short vid);
{
struct batadv_tvlv_mcast_tracker *mcast_tracker;
struct batadv_neigh_node *neigh_node;
- unsigned long offset, num_dests_off;
struct sk_buff *nexthop_skb;
unsigned char *skb_net_hdr;
bool local_recv = false;
unsigned int tvlv_len;
+ unsigned long offset;
bool xmitted = false;
u8 *dest, *next_dest;
u16 num_dests;
/* (at least) TVLV part needs to be linearized */
SKB_LINEAR_ASSERT(skb);
- /* check if num_dests is within skb length */
- num_dests_off = offsetof(struct batadv_tvlv_mcast_tracker, num_dests);
- if (num_dests_off > skb_network_header_len(skb))
+ /* check if batadv_tvlv_mcast_tracker header is within skb length */
+ if (sizeof(*mcast_tracker) > skb_network_header_len(skb))
return -EINVAL;
skb_net_hdr = skb_network_header(skb);
out:
batadv_hardif_put(primary_if);
- if (ret && tt_req_node) {
+ if (!ret && tt_req_node) {
spin_lock_bh(&bat_priv->tt.req_list_lock);
if (!hlist_unhashed(&tt_req_node->list)) {
hlist_del_init(&tt_req_node->list);
u16 tvlv_value_len)
{
struct batadv_tvlv_tt_data *tt_data;
- u16 tt_vlan_len, tt_num_entries;
+ u16 tt_num_entries;
+ size_t tt_vlan_len;
char tt_flag;
bool ret;
if (!chan)
return NULL;
- l2cap_chan_set_defaults(chan);
+ l2cap_chan_set_defaults(chan, NULL);
chan->chan_type = L2CAP_CHAN_CONN_ORIENTED;
chan->mode = L2CAP_MODE_LE_FLOWCTL;
ifup(dev->netdev);
}
-static inline struct l2cap_chan *chan_new_conn_cb(struct l2cap_chan *pchan)
-{
- struct l2cap_chan *chan;
-
- chan = chan_create();
- if (!chan)
- return NULL;
-
- chan->ops = pchan->ops;
-
- BT_DBG("chan %p pchan %p", chan, pchan);
-
- return chan;
-}
-
static void unregister_dev(struct lowpan_btle_dev *dev)
{
struct hci_dev *hdev = READ_ONCE(dev->hdev);
struct lowpan_btle_dev *dev = NULL;
struct lowpan_peer *peer;
int err = -ENOENT;
- bool last = false, remove = true;
+ bool last = false;
BT_DBG("chan %p conn %p", chan, chan->conn);
- if (chan->conn && chan->conn->hcon) {
- if (!is_bt_6lowpan(chan->conn->hcon))
- return;
-
- /* If conn is set, then the netdev is also there and we should
- * not remove it.
- */
- remove = false;
- }
-
spin_lock(&devices_lock);
list_for_each_entry_rcu(entry, &bt_6lowpan_devices, list) {
ifdown(dev->netdev);
- if (remove) {
- INIT_WORK(&entry->delete_netdev, delete_netdev);
- schedule_work(&entry->delete_netdev);
- }
+ INIT_WORK(&entry->delete_netdev, delete_netdev);
+ schedule_work(&entry->delete_netdev);
} else {
spin_unlock(&devices_lock);
}
static const struct l2cap_ops bt_6lowpan_chan_ops = {
.name = "L2CAP 6LoWPAN channel",
- .new_connection = chan_new_conn_cb,
.recv = chan_recv_cb,
.close = chan_close_cb,
.state_change = chan_state_change_cb,
hci_dev_lock(hdev);
hcon = hci_conn_hash_lookup_le(hdev, addr, le_addr_type);
- hci_dev_unlock(hdev);
- hci_dev_put(hdev);
-
- if (!hcon)
+ if (!hcon) {
+ hci_dev_unlock(hdev);
+ hci_dev_put(hdev);
return -ENOENT;
+ }
- *conn = (struct l2cap_conn *)hcon->l2cap_data;
+ *conn = l2cap_conn_hold_unless_zero(hcon->l2cap_data);
BT_DBG("conn %p dst %pMR type %u", *conn, &hcon->dst, hcon->dst_type);
+ hci_dev_unlock(hdev);
+ hci_dev_put(hdev);
+
return 0;
}
} while (nchans);
}
-struct set_enable {
- struct work_struct work;
- bool flag;
-};
-
-static void do_enable_set(struct work_struct *work)
+static void do_enable_set(bool flag)
{
- struct set_enable *set_enable = container_of(work,
- struct set_enable, work);
-
- if (!set_enable->flag || enable_6lowpan != set_enable->flag)
+ if (!flag || enable_6lowpan != flag)
/* Disconnect existing connections if 6lowpan is
* disabled
*/
disconnect_all_peers();
- enable_6lowpan = set_enable->flag;
+ enable_6lowpan = flag;
mutex_lock(&set_lock);
if (listen_chan) {
listen_chan = bt_6lowpan_listen();
mutex_unlock(&set_lock);
-
- kfree(set_enable);
}
static int lowpan_enable_set(void *data, u64 val)
{
- struct set_enable *set_enable;
-
- set_enable = kzalloc_obj(*set_enable);
- if (!set_enable)
- return -ENOMEM;
-
- set_enable->flag = !!val;
- INIT_WORK(&set_enable->work, do_enable_set);
-
- schedule_work(&set_enable->work);
+ do_enable_set(!!val);
return 0;
}
if (conn) {
struct lowpan_peer *peer;
- if (!is_bt_6lowpan(conn->hcon))
+ if (!is_bt_6lowpan(conn->hcon)) {
+ l2cap_conn_put(conn);
return -EINVAL;
+ }
peer = lookup_peer(conn);
if (peer) {
BT_DBG("6LoWPAN connection already exists");
+ l2cap_conn_put(conn);
return -EALREADY;
}
BT_DBG("conn %p dst %pMR type %d user %u", conn,
&conn->hcon->dst, conn->hcon->dst_type,
addr_type);
+ l2cap_conn_put(conn);
}
ret = bt_6lowpan_connect(&addr, addr_type);
return ret;
ret = bt_6lowpan_disconnect(conn, addr_type);
+ if (conn)
+ l2cap_conn_put(conn);
if (ret < 0)
return ret;
restart:
for (sk = bt_accept_get(parent, NULL); sk; sk = next) {
- /* Prevent early freeing of sk due to unlink and sock_kill */
+ /* The reference from bt_accept_get() keeps sk alive. */
lock_sock(sk);
/* Check sk has not already been unlinked via
next = bt_accept_get(parent, sk);
- /* sk is safely in the parent list so reduce reference count */
- sock_put(sk);
-
/* FIXME: Is this check still needed */
if (sk->sk_state == BT_CLOSED) {
bt_accept_unlink(sk);
release_sock(sk);
+ sock_put(sk);
continue;
}
if (newsock)
sock_graft(sk, newsock);
- /* Hand the caller a reference taken while sk is
- * still locked. bt_accept_unlink() just dropped
- * the accept-queue reference; without this hold a
- * concurrent teardown (e.g. l2cap_conn_del() ->
- * l2cap_sock_kill()) could free sk between
- * release_sock() and the caller using it. Every
- * caller drops this with sock_put() when done.
- */
- sock_hold(sk);
-
release_sock(sk);
if (next)
sock_put(next);
}
release_sock(sk);
+ sock_put(sk);
}
return NULL;
return 0;
}
-static struct device *bnep_get_device(struct bnep_session *session)
+static struct l2cap_conn *bnep_get_conn(struct bnep_session *session)
{
- struct l2cap_conn *conn = l2cap_pi(session->sock->sk)->chan->conn;
+ struct l2cap_chan *chan = l2cap_pi(session->sock->sk)->chan;
+ struct l2cap_conn *conn;
- if (!conn || !conn->hcon)
- return NULL;
+ l2cap_chan_lock(chan);
+ conn = chan->conn;
+ if (conn)
+ l2cap_conn_get(conn);
+ l2cap_chan_unlock(chan);
- return &conn->hcon->dev;
+ return conn;
}
static const struct device_type bnep_type = {
u32 valid_flags = BIT(BNEP_SETUP_RESPONSE);
struct net_device *dev;
struct bnep_session *s, *ss;
+ struct l2cap_conn *conn = NULL;
u8 dst[ETH_ALEN], src[ETH_ALEN];
int err;
bnep_set_default_proto_filter(s);
#endif
- SET_NETDEV_DEV(dev, bnep_get_device(s));
+ conn = bnep_get_conn(s);
+ if (!conn) {
+ err = -ENOTCONN;
+ goto failed;
+ }
+
+ SET_NETDEV_DEV(dev, &conn->hcon->dev);
SET_NETDEV_DEVTYPE(dev, &bnep_type);
err = register_netdev(dev);
+ l2cap_conn_put(conn);
+ conn = NULL;
if (err)
goto failed;
return 0;
failed:
+ if (conn)
+ l2cap_conn_put(conn);
up_write(&bnep_session_sem);
free_netdev(dev);
return err;
conn->abort_reason = reason;
- /* If the connection is pending check the command opcode since that
- * might be blocking on hci_cmd_sync_work while waiting its respective
- * event so we need to hci_cmd_sync_cancel to cancel it.
- *
- * hci_connect_le serializes the connection attempts so only one
- * connection can be in BT_CONNECT at time.
+ /* Cancel the connect attempt. A return of 0 means the create command
+ * was still queued and got dequeued, so there is nothing to disconnect.
*/
- if (conn->state == BT_CONNECT && READ_ONCE(hdev->req_status) == HCI_REQ_PEND) {
- switch (hci_skb_event(hdev->sent_cmd)) {
- case HCI_EV_CONN_COMPLETE:
- case HCI_EV_LE_CONN_COMPLETE:
- case HCI_EV_LE_ENHANCED_CONN_COMPLETE:
- case HCI_EVT_LE_CIS_ESTABLISHED:
- hci_cmd_sync_cancel(hdev, ECANCELED);
- break;
- }
- /* Cancel connect attempt if still queued/pending */
- } else if (!hci_cancel_connect_sync(hdev, conn)) {
+ if (!hci_cancel_connect_sync(hdev, conn))
return 0;
- }
/* Run immediately if on cmd_sync_work since this may be called
* as a result to MGMT_OP_DISCONNECT/MGMT_OP_UNPAIR which does
}
mgmt_device_disconnected(hdev, &conn->dst, conn->type, conn->dst_type,
- cp->reason, mgmt_conn);
+ hci_to_mgmt_reason(cp->reason), mgmt_conn);
hci_disconn_cfm(conn, cp->reason);
hci_dev_unlock(hdev);
}
-static u8 hci_to_mgmt_reason(u8 err)
-{
- switch (err) {
- case HCI_ERROR_CONNECTION_TIMEOUT:
- return MGMT_DEV_DISCONN_TIMEOUT;
- case HCI_ERROR_REMOTE_USER_TERM:
- case HCI_ERROR_REMOTE_LOW_RESOURCES:
- case HCI_ERROR_REMOTE_POWER_OFF:
- return MGMT_DEV_DISCONN_REMOTE;
- case HCI_ERROR_LOCAL_HOST_TERM:
- return MGMT_DEV_DISCONN_LOCAL_HOST;
- default:
- return MGMT_DEV_DISCONN_UNKNOWN;
- }
-}
-
static void hci_disconn_complete_evt(struct hci_dev *hdev, void *data,
struct sk_buff *skb)
{
* In this kind of scenario skip the update and let the random
* address be updated at the next cycle.
*/
+ rcu_read_lock();
+
if (bacmp(&hdev->random_addr, BDADDR_ANY) &&
(hci_dev_test_flag(hdev, HCI_LE_ADV) ||
hci_lookup_le_connect(hdev))) {
bt_dev_dbg(hdev, "Deferring random address update");
hci_dev_set_flag(hdev, HCI_RPA_EXPIRED);
+ rcu_read_unlock();
return 0;
}
+ rcu_read_unlock();
+
return __hci_cmd_sync_status(hdev, HCI_OP_LE_SET_RANDOM_ADDR,
6, rpa, HCI_CMD_TIMEOUT);
}
/* Cannot disable addr resolution if scanning is enabled or
* when initiating an LE connection.
*/
+ rcu_read_lock();
+
if (hci_dev_test_flag(hdev, HCI_LE_SCAN) ||
hci_lookup_le_connect(hdev)) {
+ rcu_read_unlock();
bt_dev_err(hdev, "Command not allowed when scan/LE connect");
return -EPERM;
}
+ rcu_read_unlock();
+
/* Cannot disable addr resolution if advertising is enabled. */
err = hci_pause_advertising_sync(hdev);
if (err) {
if (hci_dev_test_flag(hdev, HCI_PA_SYNC)) {
struct hci_conn *conn;
+ rcu_read_lock();
+
conn = hci_conn_hash_lookup_create_pa_sync(hdev);
if (conn) {
struct conn_params pa;
bacpy(&pa.addr, &conn->dst);
pa.addr_type = conn->dst_type;
+ rcu_read_unlock();
+
/* Clear first since there could be addresses left
* behind.
*/
err = hci_le_add_accept_list_sync(hdev, &pa,
&num_entries);
goto done;
+ } else {
+ rcu_read_unlock();
}
}
* the controller.
*/
list_for_each_entry_safe(b, t, &hdev->le_accept_list, list) {
- if (hci_conn_hash_lookup_le(hdev, &b->bdaddr, b->bdaddr_type))
+ rcu_read_lock();
+
+ if (hci_conn_hash_lookup_le(hdev, &b->bdaddr, b->bdaddr_type)) {
+ rcu_read_unlock();
continue;
+ }
- /* Pointers not dereferenced, no locks needed */
pend_conn = hci_pend_le_action_lookup(&hdev->pend_le_conns,
&b->bdaddr,
b->bdaddr_type);
&b->bdaddr,
b->bdaddr_type);
+ rcu_read_unlock();
+
/* If the device is not likely to connect or report,
* remove it from the acceptlist.
*/
if (sent) {
struct hci_conn *conn;
+ rcu_read_lock();
+
conn = hci_conn_hash_lookup_ba(hdev, PA_LINK,
&sent->bdaddr);
if (conn) {
phy++;
}
+ rcu_read_unlock();
+
if (num_phy)
goto done;
+ } else {
+ rcu_read_unlock();
}
}
}
/* If there is at least one pending LE connection, we should
* keep the background scan running.
*/
+ bool exists;
/* If controller is connecting, we should not start scanning
* since some controllers are not able to scan and connect at
* the same time.
*/
- if (hci_lookup_le_connect(hdev))
+ rcu_read_lock();
+ exists = hci_lookup_le_connect(hdev);
+ rcu_read_unlock();
+ if (exists)
return 0;
bt_dev_dbg(hdev, "start background scanning");
}
static bool disconnected_accept_list_entries(struct hci_dev *hdev)
+ __must_hold(&hdev->lock)
{
struct bdaddr_list *b;
if (hdev->scanning_paused)
return 0;
+ hci_dev_lock(hdev);
+
if (hci_dev_test_flag(hdev, HCI_CONNECTABLE) ||
disconnected_accept_list_entries(hdev))
scan = SCAN_PAGE;
else
scan = SCAN_DISABLED;
+ hci_dev_unlock(hdev);
+
if (hci_dev_test_flag(hdev, HCI_DISCOVERABLE))
scan |= SCAN_INQUIRY;
bt_dev_dbg(hdev, "conn %p", conn);
+ /* Hold a reference so conn stays valid for the HCI_CONN_CREATE
+ * clear_bit() at done.
+ */
+ hci_conn_get(conn);
+
clear_bit(HCI_CONN_SCANNING, &conn->flags);
conn->state = BT_CONNECT;
hdev->le_scan_type == LE_SCAN_ACTIVE &&
!hci_dev_test_flag(hdev, HCI_LE_SIMULTANEOUS_ROLES)) {
hci_conn_del(conn);
+ hci_conn_put(conn);
return -EBUSY;
}
if (!hci_dev_test_flag(hdev, HCI_LE_SIMULTANEOUS_ROLES))
hci_pause_advertising_sync(hdev);
+ hci_dev_lock(hdev);
+
params = hci_conn_params_lookup(hdev, &conn->dst, conn->dst_type);
if (params) {
conn->le_conn_min_interval = params->conn_min_interval;
conn->le_supv_timeout = hdev->le_supv_timeout;
}
+ hci_dev_unlock(hdev);
+
/* If controller is scanning, we stop it since some controllers are
* not able to scan and connect at the same time. Also set the
* HCI_LE_SCAN_INTERRUPTED flag so that the command complete
&own_addr_type);
if (err)
goto done;
+
+ /* Mark create connection in flight so hci_cancel_connect_sync() can
+ * cancel it while blocking on the connection complete event.
+ */
+ set_bit(HCI_CONN_CREATE, &conn->flags);
+
/* Send command LE Extended Create Connection if supported */
if (use_ext_conn(hdev)) {
err = hci_le_ext_create_conn_sync(hdev, conn, own_addr_type);
conn->conn_timeout, NULL);
done:
+ clear_bit(HCI_CONN_CREATE, &conn->flags);
+
if (err == -ETIMEDOUT)
hci_le_connect_cancel_sync(hdev, conn, 0x00);
/* Re-enable advertising after the connection attempt is finished. */
hci_resume_advertising_sync(hdev);
+ hci_conn_put(conn);
return err;
}
else
cp.role_switch = 0x00;
- return __hci_cmd_sync_status_sk(hdev, HCI_OP_CREATE_CONN,
- sizeof(cp), &cp,
- HCI_EV_CONN_COMPLETE,
- conn->conn_timeout, NULL);
+ /* Hold a reference so conn stays valid for the HCI_CONN_CREATE
+ * clear_bit() below.
+ */
+ hci_conn_get(conn);
+
+ /* Mark create connection in flight so hci_cancel_connect_sync() can
+ * cancel it while blocking on the connection complete event.
+ */
+ set_bit(HCI_CONN_CREATE, &conn->flags);
+
+ err = __hci_cmd_sync_status_sk(hdev, HCI_OP_CREATE_CONN,
+ sizeof(cp), &cp,
+ HCI_EV_CONN_COMPLETE,
+ conn->conn_timeout, NULL);
+
+ clear_bit(HCI_CONN_CREATE, &conn->flags);
+ hci_conn_put(conn);
+
+ return err;
}
int hci_connect_acl_sync(struct hci_dev *hdev, struct hci_conn *conn)
return (err == -EEXIST) ? 0 : err;
}
-int hci_cancel_connect_sync(struct hci_dev *hdev, struct hci_conn *conn)
+static int hci_acl_cancel_create_conn_sync(struct hci_dev *hdev,
+ struct hci_conn *conn)
{
- if (conn->state != BT_OPEN)
- return -EINVAL;
+ struct hci_cmd_sync_work_entry *entry;
+ int err = -EBUSY;
+ /* cmd_sync_work_lock makes the HCI_CONN_CREATE test and the cancel
+ * atomic against the worker, which takes this lock to dequeue every
+ * entry: while it is held no other command can become pending, so
+ * hci_cmd_sync_cancel() cannot cancel an unrelated command.
+ */
+ mutex_lock(&hdev->cmd_sync_work_lock);
+
+ /* In flight: this connection owns the pending request, cancel it. */
+ if (test_bit(HCI_CONN_CREATE, &conn->flags)) {
+ hci_cmd_sync_cancel(hdev, ECANCELED);
+ goto unlock;
+ }
+
+ /* Still queued: a successful dequeue means it never started, so there
+ * is nothing to disconnect.
+ */
+ entry = _hci_cmd_sync_lookup_entry(hdev, hci_acl_create_conn_sync, conn,
+ NULL);
+ if (entry) {
+ _hci_cmd_sync_cancel_entry(hdev, entry, -ECANCELED);
+ err = 0;
+ }
+
+unlock:
+ mutex_unlock(&hdev->cmd_sync_work_lock);
+ return err;
+}
+
+static int hci_le_cancel_create_conn_sync(struct hci_dev *hdev,
+ struct hci_conn *conn)
+{
+ struct hci_cmd_sync_work_entry *entry;
+ int err = -EBUSY;
+
+ /* cmd_sync_work_lock keeps the HCI_CONN_CREATE test and the cancel
+ * atomic against the cmd_sync worker.
+ */
+ mutex_lock(&hdev->cmd_sync_work_lock);
+
+ if (test_bit(HCI_CONN_CREATE, &conn->flags)) {
+ hci_cmd_sync_cancel(hdev, ECANCELED);
+ goto unlock;
+ }
+
+ entry = _hci_cmd_sync_lookup_entry(hdev, hci_le_create_conn_sync, conn,
+ create_le_conn_complete);
+ if (entry) {
+ _hci_cmd_sync_cancel_entry(hdev, entry, -ECANCELED);
+ err = 0;
+ }
+
+unlock:
+ mutex_unlock(&hdev->cmd_sync_work_lock);
+ return err;
+}
+
+static int hci_cis_cancel_create_conn_sync(struct hci_dev *hdev,
+ struct hci_conn *conn)
+{
+ /* LE Create CIS is shared by the whole CIG and cannot be dequeued
+ * per-connection, so only an in-flight command can be cancelled.
+ * cmd_sync_work_lock keeps the test and the cancel atomic against the
+ * cmd_sync worker.
+ */
+ mutex_lock(&hdev->cmd_sync_work_lock);
+
+ if (test_bit(HCI_CONN_CREATE_CIS, &conn->flags))
+ hci_cmd_sync_cancel(hdev, ECANCELED);
+
+ mutex_unlock(&hdev->cmd_sync_work_lock);
+ return -EBUSY;
+}
+
+int hci_cancel_connect_sync(struct hci_dev *hdev, struct hci_conn *conn)
+{
switch (conn->type) {
case ACL_LINK:
- return !hci_cmd_sync_dequeue_once(hdev,
- hci_acl_create_conn_sync,
- conn, NULL);
+ return hci_acl_cancel_create_conn_sync(hdev, conn);
case LE_LINK:
- return !hci_cmd_sync_dequeue_once(hdev, hci_le_create_conn_sync,
- conn, create_le_conn_complete);
+ return hci_le_cancel_create_conn_sync(hdev, conn);
+ case CIS_LINK:
+ return hci_cis_cancel_create_conn_sync(hdev, conn);
+ default:
+ return -ENOENT;
}
-
- return -ENOENT;
}
int hci_le_conn_update_sync(struct hci_dev *hdev, struct hci_conn *conn,
}
static int hci_le_past_params_sync(struct hci_dev *hdev, struct hci_conn *conn,
- struct hci_conn *acl, struct bt_iso_qos *qos)
+ u16 acl_handle, struct bt_iso_qos *qos)
{
struct hci_cp_le_past_params cp;
int err;
memset(&cp, 0, sizeof(cp));
- cp.handle = cpu_to_le16(acl->handle);
+ cp.handle = cpu_to_le16(acl_handle);
/* An HCI_LE_Periodic_Advertising_Sync_Transfer_Received event is sent
* to the Host. HCI_LE_Periodic_Advertising_Report events will be
* enabled with duplicate filtering enabled.
* 2. Check if that HCI_CONN_FLAG_PAST has been set which indicates that
* user really intended to use PAST.
*/
+ hci_dev_lock(hdev);
+
le = hci_conn_hash_lookup_le(hdev, &conn->dst, conn->dst_type);
if (le) {
struct hci_conn_params *params;
+ hci_conn_flags_t flags = 0;
+ u16 le_handle = le->handle;
params = hci_conn_params_lookup(hdev, &le->dst, le->dst_type);
- if (params && params->flags & HCI_CONN_FLAG_PAST) {
- err = hci_le_past_params_sync(hdev, conn, le, qos);
+ if (params)
+ flags = params->flags;
+
+ hci_dev_unlock(hdev);
+
+ if (flags & HCI_CONN_FLAG_PAST) {
+ err = hci_le_past_params_sync(hdev, conn, le_handle,
+ qos);
if (!err)
goto done;
}
+ } else {
+ hci_dev_unlock(hdev);
}
/* SID has not been set listen for HCI_EV_LE_EXT_ADV_REPORT to update
{
int err;
struct hci_dev *hdev;
+ struct iso_conn *conn;
bdaddr_t src, dst;
u8 src_type;
hci_dev_lock(hdev);
lock_sock(sk);
+ /* The socket lock was dropped for hci_get_route(), so the connection
+ * may have been torn down meanwhile: iso_chan_del() clears conn and
+ * the broadcast teardown path can clear conn->hcon on its own. Check
+ * both before dereferencing conn->hcon.
+ */
+ conn = iso_pi(sk)->conn;
+ if (!conn || !conn->hcon)
+ goto unlock;
+
if (!test_and_set_bit(BT_SK_BIG_SYNC, &iso_pi(sk)->flags)) {
- err = hci_conn_big_create_sync(hdev, iso_pi(sk)->conn->hcon,
+ err = hci_conn_big_create_sync(hdev, conn->hcon,
&iso_pi(sk)->qos,
iso_pi(sk)->sync_handle,
iso_pi(sk)->bc_num_bis,
bt_dev_err(hdev, "hci_big_create_sync: %d", err);
}
+unlock:
release_sock(sk);
hci_dev_unlock(hdev);
hci_dev_put(hdev);
switch (pb) {
case ISO_START:
case ISO_SINGLE:
- if (conn->rx_len) {
+ if (conn->rx_skb || conn->rx_len) {
BT_ERR("Unexpected start frame (len %d)", skb->len);
kfree_skb(conn->rx_skb);
conn->rx_skb = NULL;
break;
case ISO_CONT:
- BT_DBG("Cont: frag len %d (expecting %d)", skb->len,
+ case ISO_END:
+ BT_DBG("%s: frag len %d (expecting %d)",
+ (pb == ISO_END) ? "End" : "Cont", skb->len,
conn->rx_len);
- if (!conn->rx_len) {
- BT_ERR("Unexpected continuation frame (len %d)",
- skb->len);
+ if (!conn->rx_skb) {
+ BT_ERR("Unexpected ISO %s frame (len %d)",
+ (pb == ISO_END) ? "End" : "Cont", skb->len);
goto drop;
}
skb_copy_from_linear_data(skb, skb_put(conn->rx_skb, skb->len),
skb->len);
conn->rx_len -= skb->len;
- break;
-
- case ISO_END:
- if (!conn->rx_len) {
- BT_ERR("Unexpected end frame (len %d)", skb->len);
- goto drop;
- }
- skb_copy_from_linear_data(skb, skb_put(conn->rx_skb, skb->len),
- skb->len);
- conn->rx_len -= skb->len;
+ if (pb == ISO_CONT)
+ break;
if (!conn->rx_len) {
struct sk_buff *rx_skb = conn->rx_skb;
*/
conn->rx_skb = NULL;
iso_recv_frame(conn, rx_skb);
+ } else {
+ BT_ERR("ISO fragment incomplete (len %d, expected %d)",
+ skb->len, conn->rx_len);
+ kfree_skb(conn->rx_skb);
+ conn->rx_skb = NULL;
+ conn->rx_len = 0;
+ goto drop;
}
break;
}
}
EXPORT_SYMBOL_GPL(l2cap_chan_put);
-void l2cap_chan_set_defaults(struct l2cap_chan *chan)
+/* Initialise @chan with default values, inheriting from the parent channel
+ * @pchan when it is given.
+ */
+void l2cap_chan_set_defaults(struct l2cap_chan *chan, struct l2cap_chan *pchan)
{
chan->fcs = L2CAP_FCS_CRC16;
chan->max_tx = L2CAP_DEFAULT_MAX_TX;
chan->retrans_timeout = L2CAP_DEFAULT_RETRANS_TO;
chan->monitor_timeout = L2CAP_DEFAULT_MONITOR_TO;
+ if (pchan) {
+ BT_DBG("chan %p pchan %p", chan, pchan);
+
+ chan->chan_type = pchan->chan_type;
+ chan->imtu = pchan->imtu;
+ chan->omtu = pchan->omtu;
+ chan->mode = pchan->mode;
+ chan->fcs = pchan->fcs;
+ chan->max_tx = pchan->max_tx;
+ chan->tx_win = pchan->tx_win;
+ chan->tx_win_max = pchan->tx_win_max;
+ chan->sec_level = pchan->sec_level;
+ chan->conf_state = pchan->conf_state;
+ chan->flags = pchan->flags;
+ chan->tx_credits = pchan->tx_credits;
+ chan->rx_credits = pchan->rx_credits;
+
+ if (chan->chan_type == L2CAP_CHAN_FIXED) {
+ chan->scid = pchan->scid;
+ chan->dcid = pchan->scid;
+ }
+
+ return;
+ }
+
chan->conf_state = 0;
set_bit(CONF_NOT_COMPLETE, &chan->conf_state);
disable_delayed_work_sync(&conn->info_timer);
disable_delayed_work_sync(&conn->id_addr_timer);
+ cancel_work_sync(&conn->pending_rx_work);
+
mutex_lock(&conn->lock);
kfree_skb(conn->rx_skb);
skb_queue_purge(&conn->pending_rx);
-
- /* We can not call flush_work(&conn->pending_rx_work) here since we
- * might block if we are running on a worker from the same workqueue
- * pending_rx_work is waiting on.
- */
- if (work_pending(&conn->pending_rx_work))
- cancel_work_sync(&conn->pending_rx_work);
-
ida_destroy(&conn->tx_ida);
l2cap_unregister_all_users(conn);
return NULL;
}
-static inline int l2cap_get_conf_opt(void **ptr, int *type, int *olen,
- unsigned long *val)
+static inline int l2cap_get_conf_opt(void **ptr, void *end, int *type,
+ int *olen, unsigned long *val)
{
struct l2cap_conf_opt *opt = *ptr;
int len;
+ /* opt->len is attacker-controlled. Validate that the full option
+ * (header + value) actually fits in the buffer before touching
+ * opt->val, otherwise the switch below reads past the end of the
+ * caller's buffer.
+ */
+ if (end - *ptr < L2CAP_CONF_OPT_SIZE)
+ return -EINVAL;
+
len = L2CAP_CONF_OPT_SIZE + opt->len;
+ if (end - *ptr < len)
+ return -EINVAL;
+
*ptr += len;
*type = opt->type;
void *ptr = rsp->data;
void *endptr = data + data_size;
void *req = chan->conf_req;
+ void *req_end = req + chan->conf_len;
int len = chan->conf_len;
int type, hint, olen;
unsigned long val;
BT_DBG("chan %p", chan);
while (len >= L2CAP_CONF_OPT_SIZE) {
- len -= l2cap_get_conf_opt(&req, &type, &olen, &val);
- if (len < 0)
+ int ret = l2cap_get_conf_opt(&req, req_end, &type, &olen, &val);
+
+ if (ret < 0)
break;
+ len -= ret;
hint = type & L2CAP_CONF_HINT;
type &= L2CAP_CONF_MASK;
struct l2cap_conf_req *req = data;
void *ptr = req->data;
void *endptr = data + size;
+ void *rsp_end = rsp + len;
int type, olen;
unsigned long val;
struct l2cap_conf_rfc rfc = { .mode = L2CAP_MODE_BASIC };
BT_DBG("chan %p, rsp %p, len %d, req %p", chan, rsp, len, data);
while (len >= L2CAP_CONF_OPT_SIZE) {
- len -= l2cap_get_conf_opt(&rsp, &type, &olen, &val);
- if (len < 0)
+ int ret = l2cap_get_conf_opt(&rsp, rsp_end, &type, &olen, &val);
+
+ if (ret < 0)
break;
+ len -= ret;
switch (type) {
case L2CAP_CONF_MTU:
{
int type, olen;
unsigned long val;
+ void *rsp_end = rsp + len;
/* Use sane default values in case a misbehaving remote device
* did not send an RFC or extended window size option.
*/
return;
while (len >= L2CAP_CONF_OPT_SIZE) {
- len -= l2cap_get_conf_opt(&rsp, &type, &olen, &val);
- if (len < 0)
+ int ret = l2cap_get_conf_opt(&rsp, rsp_end, &type, &olen, &val);
+
+ if (ret < 0)
break;
+ len -= ret;
switch (type) {
case L2CAP_CONF_RFC:
return 0;
}
+/* Allocate and initialise a channel for an incoming connection.
+ *
+ * The channel inherits its configuration from @pchan and is linked into @conn
+ * before ->new_connection() runs, so the conn list reference keeps it alive if
+ * the callback exposes it (e.g. via the socket accept queue) before this
+ * returns. The l2cap_chan_create() reference is taken over by the subsystem on
+ * success and dropped here on failure.
+ */
+static struct l2cap_chan *l2cap_new_connection(struct l2cap_conn *conn,
+ struct l2cap_chan *pchan)
+{
+ struct l2cap_chan *chan;
+
+ chan = l2cap_chan_create();
+ if (!chan)
+ return NULL;
+
+ l2cap_chan_set_defaults(chan, pchan);
+ chan->ops = pchan->ops;
+
+ __l2cap_chan_add(conn, chan);
+
+ if (pchan->ops->new_connection &&
+ pchan->ops->new_connection(pchan, chan) < 0) {
+ l2cap_chan_del(chan, 0);
+ l2cap_chan_put(chan);
+ return NULL;
+ }
+
+ return chan;
+}
+
static void l2cap_connect(struct l2cap_conn *conn, struct l2cap_cmd_hdr *cmd,
u8 *data, u8 rsp_code)
{
goto response;
}
- chan = pchan->ops->new_connection(pchan);
+ chan = l2cap_new_connection(conn, pchan);
if (!chan)
goto response;
chan->psm = psm;
chan->dcid = scid;
- __l2cap_chan_add(conn, chan);
-
dcid = chan->scid;
__set_chan_timer(chan, chan->ops->get_sndtimeo(chan));
case L2CAP_ECHO_RSP:
case L2CAP_INFO_RSP:
case L2CAP_CONN_PARAM_UPDATE_RSP:
+ case L2CAP_LE_CONN_RSP:
case L2CAP_ECRED_CONN_RSP:
case L2CAP_ECRED_RECONF_RSP:
/* First do a lookup since the remote may send bogus ids that
goto response_unlock;
}
- chan = pchan->ops->new_connection(pchan);
+ chan = l2cap_new_connection(conn, pchan);
if (!chan) {
result = L2CAP_CR_LE_NO_MEM;
goto response_unlock;
chan->omtu = mtu;
chan->remote_mps = mps;
- __l2cap_chan_add(conn, chan);
-
l2cap_le_flowctl_init(chan, __le16_to_cpu(req->credits));
dcid = chan->scid;
continue;
}
- chan = pchan->ops->new_connection(pchan);
+ chan = l2cap_new_connection(conn, pchan);
if (!chan) {
result = L2CAP_CR_LE_NO_MEM;
continue;
chan->omtu = mtu;
chan->remote_mps = mps;
- __l2cap_chan_add(conn, chan);
-
l2cap_ecred_init(chan, __le16_to_cpu(req->credits));
/* Init response */
struct l2cap_conn *conn = chan->conn;
struct l2cap_le_credits pkt;
u16 return_credits = l2cap_le_rx_credits(chan);
+ int ident;
if (chan->mode != L2CAP_MODE_LE_FLOWCTL &&
chan->mode != L2CAP_MODE_EXT_FLOWCTL)
pkt.cid = cpu_to_le16(chan->scid);
pkt.credits = cpu_to_le16(return_credits);
- chan->ident = l2cap_get_ident(conn);
+ ident = l2cap_get_ident(conn);
+
+ l2cap_send_cmd(conn, ident, L2CAP_LE_CREDITS, sizeof(pkt), &pkt);
- l2cap_send_cmd(conn, chan->ident, L2CAP_LE_CREDITS, sizeof(pkt), &pkt);
+ /* L2CAP_LE_CREDITS has no response so the ident is never released by
+ * l2cap_put_ident() - release it right away, otherwise the tx_ida
+ * range is exhausted after 254 packets and from then on credits are
+ * sent with the invalid ident 0, which some remote stacks ignore,
+ * stalling the channel.
+ */
+ if (ident > 0)
+ ida_free(&conn->tx_ida, ident);
}
void l2cap_chan_rx_avail(struct l2cap_chan *chan, ssize_t rx_avail)
goto next;
l2cap_chan_lock(pchan);
- chan = pchan->ops->new_connection(pchan);
+ chan = l2cap_new_connection(conn, pchan);
if (chan) {
bacpy(&chan->src, &hcon->src);
bacpy(&chan->dst, &hcon->dst);
chan->src_type = bdaddr_src_type(hcon);
chan->dst_type = dst_type;
-
- __l2cap_chan_add(conn, chan);
}
l2cap_chan_unlock(pchan);
return c;
}
+EXPORT_SYMBOL(l2cap_conn_hold_unless_zero);
int l2cap_recv_acldata(struct hci_dev *hdev, u16 handle,
struct sk_buff *skb, u16 flags)
static const struct proto_ops l2cap_sock_ops;
static void l2cap_sock_init(struct sock *sk, struct sock *parent);
static struct sock *l2cap_sock_alloc(struct net *net, struct socket *sock,
- int proto, gfp_t prio, int kern);
+ int proto, gfp_t prio, int kern,
+ struct l2cap_chan *chan);
static void l2cap_sock_cleanup_listen(struct sock *parent);
bool l2cap_is_socket(struct socket *sock)
return err;
}
+/* Release the sock's ref on chan and clear the pointer so that the ref is
+ * dropped exactly once even if both l2cap_sock_kill() and
+ * l2cap_sock_destruct() run. Setting chan->data to NULL first stops any other
+ * task from dereferencing the now-dead sock pointer.
+ */
+static void l2cap_sock_put_chan(struct sock *sk)
+{
+ struct l2cap_chan *chan = l2cap_pi(sk)->chan;
+
+ if (!chan)
+ return;
+
+ chan->data = NULL;
+ l2cap_pi(sk)->chan = NULL;
+ l2cap_chan_put(chan);
+}
+
/* Kill socket (only if zapped and orphan)
* Must be called on unlocked socket, with l2cap channel lock.
*/
BT_DBG("sk %p state %s", sk, state_to_string(sk->sk_state));
- /* Sock is dead, so set chan data to NULL, avoid other task use invalid
- * sock pointer.
- */
- l2cap_pi(sk)->chan->data = NULL;
- /* Kill poor orphan */
+ l2cap_sock_put_chan(sk);
- l2cap_chan_put(l2cap_pi(sk)->chan);
+ /* Kill poor orphan */
sock_set_flag(sk, SOCK_DEAD);
sock_put(sk);
}
/* Close not yet accepted channels.
*
- * bt_accept_dequeue() now returns sk with an extra reference held
- * (taken while sk was still locked) so a concurrent l2cap_conn_del()
+ * bt_accept_dequeue() returns sk with its temporary queue-walk
+ * reference held, so a concurrent l2cap_conn_del()
* -> l2cap_sock_kill() cannot free sk under us.
*
* cleanup_listen() runs under the parent sk lock, so unlike
}
}
-static struct l2cap_chan *l2cap_sock_new_connection_cb(struct l2cap_chan *chan)
+static int l2cap_sock_new_connection_cb(struct l2cap_chan *chan,
+ struct l2cap_chan *new_chan)
{
struct sock *sk, *parent = chan->data;
if (!parent)
- return NULL;
+ return -EINVAL;
lock_sock(parent);
if (sk_acceptq_is_full(parent)) {
BT_DBG("backlog full %d", parent->sk_ack_backlog);
release_sock(parent);
- return NULL;
+ return -ENOBUFS;
}
sk = l2cap_sock_alloc(sock_net(parent), NULL, BTPROTO_L2CAP,
- GFP_ATOMIC, 0);
+ GFP_ATOMIC, 0, new_chan);
if (!sk) {
release_sock(parent);
- return NULL;
- }
+ return -ENOMEM;
+ }
bt_sock_reclassify_lock(sk, BTPROTO_L2CAP);
l2cap_sock_init(sk, parent);
+ /* The conn list reference taken by l2cap_new_connection() keeps new_chan
+ * alive once release_sock() lets another task free this socket.
+ */
bt_accept_enqueue(parent, sk, false);
release_sock(parent);
- return l2cap_pi(sk)->chan;
+ return 0;
}
static int l2cap_sock_recv_cb(struct l2cap_chan *chan, struct sk_buff *skb)
BT_DBG("sk %p", sk);
- if (l2cap_pi(sk)->chan) {
- l2cap_pi(sk)->chan->data = NULL;
- l2cap_chan_put(l2cap_pi(sk)->chan);
- }
+ l2cap_sock_put_chan(sk);
list_for_each_entry_safe(rx_busy, next, &l2cap_pi(sk)->rx_busy, list) {
kfree_skb(rx_busy->skb);
BT_DBG("sk %p", sk);
if (parent) {
- struct l2cap_chan *pchan = l2cap_pi(parent)->chan;
-
sk->sk_type = parent->sk_type;
bt_sk(sk)->flags = bt_sk(parent)->flags;
- chan->chan_type = pchan->chan_type;
- chan->imtu = pchan->imtu;
- chan->omtu = pchan->omtu;
- chan->conf_state = pchan->conf_state;
- chan->mode = pchan->mode;
- chan->fcs = pchan->fcs;
- chan->max_tx = pchan->max_tx;
- chan->tx_win = pchan->tx_win;
- chan->tx_win_max = pchan->tx_win_max;
- chan->sec_level = pchan->sec_level;
- chan->flags = pchan->flags;
- chan->tx_credits = pchan->tx_credits;
- chan->rx_credits = pchan->rx_credits;
-
- if (chan->chan_type == L2CAP_CHAN_FIXED) {
- chan->scid = pchan->scid;
- chan->dcid = pchan->scid;
- }
-
+ /* Channel configuration is inherited from the parent by
+ * l2cap_new_connection().
+ */
security_sk_clone(parent, sk);
} else {
switch (sk->sk_type) {
chan->mode = L2CAP_MODE_BASIC;
}
- l2cap_chan_set_defaults(chan);
+ l2cap_chan_set_defaults(chan, NULL);
}
/* Default config options */
};
static struct sock *l2cap_sock_alloc(struct net *net, struct socket *sock,
- int proto, gfp_t prio, int kern)
+ int proto, gfp_t prio, int kern,
+ struct l2cap_chan *chan)
{
struct sock *sk;
- struct l2cap_chan *chan;
sk = bt_sock_alloc(net, sock, &l2cap_proto, proto, prio, kern);
if (!sk)
INIT_LIST_HEAD(&l2cap_pi(sk)->rx_busy);
- chan = l2cap_chan_create();
- if (!chan) {
- sk_free(sk);
- if (sock)
- sock->sk = NULL;
- return NULL;
- }
-
- l2cap_chan_hold(chan);
-
+ /* The sock takes ownership of the caller's reference on chan. */
l2cap_pi(sk)->chan = chan;
return sk;
int kern)
{
struct sock *sk;
+ struct l2cap_chan *chan;
BT_DBG("sock %p", sock);
sock->ops = &l2cap_sock_ops;
- sk = l2cap_sock_alloc(net, sock, protocol, GFP_ATOMIC, kern);
- if (!sk)
+ chan = l2cap_chan_create();
+ if (!chan)
return -ENOMEM;
+ sk = l2cap_sock_alloc(net, sock, protocol, GFP_ATOMIC, kern, chan);
+ if (!sk) {
+ l2cap_chan_put(chan);
+ return -ENOMEM;
+ }
+
l2cap_sock_init(sk, NULL);
bt_sock_link(&l2cap_sk_list, sk);
return 0;
struct mgmt_cp_unpair_device *cp = cmd->param;
struct hci_conn *conn;
+ hci_dev_lock(hdev);
+
if (cp->addr.type == BDADDR_BREDR)
conn = hci_conn_hash_lookup_ba(hdev, ACL_LINK,
&cp->addr.bdaddr);
conn = hci_conn_hash_lookup_le(hdev, &cp->addr.bdaddr,
le_addr_type(cp->addr.type));
+ if (conn)
+ hci_conn_get(conn);
+
+ hci_dev_unlock(hdev);
+
if (!conn)
return 0;
* will clean up the connection no matter the error.
*/
hci_abort_conn(conn, HCI_ERROR_REMOTE_USER_TERM);
+ hci_conn_put(conn);
return 0;
}
struct mgmt_cp_disconnect *cp = cmd->param;
struct hci_conn *conn;
+ hci_dev_lock(hdev);
+
if (cp->addr.type == BDADDR_BREDR)
conn = hci_conn_hash_lookup_ba(hdev, ACL_LINK,
&cp->addr.bdaddr);
conn = hci_conn_hash_lookup_le(hdev, &cp->addr.bdaddr,
le_addr_type(cp->addr.type));
+ if (conn)
+ hci_conn_get(conn);
+
+ hci_dev_unlock(hdev);
+
if (!conn)
return -ENOTCONN;
* will clean up the connection no matter the error.
*/
hci_abort_conn(conn, HCI_ERROR_REMOTE_USER_TERM);
+ hci_conn_put(conn);
return 0;
}
if (monitor->state == ADV_MONITOR_STATE_NOT_REGISTERED)
monitor->state = ADV_MONITOR_STATE_REGISTERED;
hci_update_passive_scan(hdev);
+ } else {
+ hci_free_adv_monitor(hdev, monitor);
}
mgmt_cmd_complete(cmd->sk, cmd->hdev->id, cmd->opcode,
rp.max_tx_power = HCI_TX_POWER_INVALID;
}
+ if (conn)
+ hci_conn_put(conn);
+
mgmt_cmd_complete(cmd->sk, cmd->hdev->id, MGMT_OP_GET_CONN_INFO, status,
&rp, sizeof(rp));
int err;
__le16 handle;
+ hci_dev_lock(hdev);
+
/* Make sure we are still connected */
if (cp->addr.type == BDADDR_BREDR)
conn = hci_conn_hash_lookup_ba(hdev, ACL_LINK,
else
conn = hci_conn_hash_lookup_ba(hdev, LE_LINK, &cp->addr.bdaddr);
- if (!conn || conn->state != BT_CONNECTED)
+ if (!conn || conn->state != BT_CONNECTED) {
+ hci_dev_unlock(hdev);
return MGMT_STATUS_NOT_CONNECTED;
+ }
- cmd->user_data = conn;
+ cmd->user_data = hci_conn_get(conn);
handle = cpu_to_le16(conn->handle);
+ hci_dev_unlock(hdev);
+
/* Refresh RSSI each time */
err = hci_read_rssi_sync(hdev, handle);
}
complete:
+ if (conn)
+ hci_conn_put(conn);
+
mgmt_cmd_complete(cmd->sk, cmd->hdev->id, cmd->opcode, status, &rp,
sizeof(rp));
memset(&hci_cp, 0, sizeof(hci_cp));
hci_read_clock_sync(hdev, &hci_cp);
+ hci_dev_lock(hdev);
+
/* Make sure connection still exists */
conn = hci_conn_hash_lookup_ba(hdev, ACL_LINK, &cp->addr.bdaddr);
- if (!conn || conn->state != BT_CONNECTED)
+ if (!conn || conn->state != BT_CONNECTED) {
+ hci_dev_unlock(hdev);
return MGMT_STATUS_NOT_CONNECTED;
+ }
- cmd->user_data = conn;
+ cmd->user_data = hci_conn_get(conn);
hci_cp.handle = cpu_to_le16(conn->handle);
hci_cp.which = 0x01; /* Piconet clock */
+ hci_dev_unlock(hdev);
+
return hci_read_clock_sync(hdev, &hci_cp);
}
if (!err) {
struct hci_conn_params *params;
+ hci_dev_lock(hdev);
+
params = hci_conn_params_lookup(hdev, &cp->addr.bdaddr,
le_addr_type(cp->addr.type));
device_flags_changed(NULL, hdev, &cp->addr.bdaddr,
cp->addr.type, hdev->conn_flags,
params ? params->flags : 0);
+ hci_dev_unlock(hdev);
}
mgmt_cmd_complete(cmd->sk, hdev->id, MGMT_OP_ADD_DEVICE,
static int conn_update_sync(struct hci_dev *hdev, void *data)
{
- struct hci_conn_params *params = data;
- struct hci_conn *conn;
+ struct hci_conn *conn = data;
+ struct hci_conn_params *params;
+ struct hci_conn_params local = {};
- conn = hci_conn_hash_lookup_le(hdev, ¶ms->addr, params->addr_type);
- if (!conn)
- return -ECANCELED;
+ hci_dev_lock(hdev);
+
+ if (!hci_conn_valid(hdev, conn) || conn->role != HCI_ROLE_MASTER)
+ goto cancel;
+
+ params = hci_conn_params_lookup(hdev, &conn->dst, conn->dst_type);
+ if (!params)
+ goto cancel;
+
+ local.conn_min_interval = params->conn_min_interval;
+ local.conn_max_interval = params->conn_max_interval;
+ local.conn_latency = params->conn_latency;
+ local.supervision_timeout = params->supervision_timeout;
- return hci_le_conn_update_sync(hdev, conn, params);
+ hci_dev_unlock(hdev);
+
+ return hci_le_conn_update_sync(hdev, conn, &local);
+
+cancel:
+ hci_dev_unlock(hdev);
+ return -ECANCELED;
+}
+
+static void conn_update_sync_destroy(struct hci_dev *hdev, void *data, int err)
+{
+ hci_conn_put(data);
}
static int load_conn_param(struct sock *sk, struct hci_dev *hdev, void *data,
(conn->le_conn_min_interval != min ||
conn->le_conn_max_interval != max ||
conn->le_conn_latency != latency ||
- conn->le_supv_timeout != timeout))
- hci_cmd_sync_queue(hdev, conn_update_sync,
- hci_param, NULL);
+ conn->le_supv_timeout != timeout)) {
+ hci_conn_get(conn);
+ if (hci_cmd_sync_queue(hdev, conn_update_sync,
+ conn,
+ conn_update_sync_destroy) < 0)
+ hci_conn_put(conn);
+ }
}
}
return false;
}
+u8 hci_to_mgmt_reason(u8 err)
+{
+ switch (err) {
+ case HCI_ERROR_CONNECTION_TIMEOUT:
+ return MGMT_DEV_DISCONN_TIMEOUT;
+ case HCI_ERROR_REMOTE_USER_TERM:
+ case HCI_ERROR_REMOTE_LOW_RESOURCES:
+ case HCI_ERROR_REMOTE_POWER_OFF:
+ return MGMT_DEV_DISCONN_REMOTE;
+ case HCI_ERROR_LOCAL_HOST_TERM:
+ return MGMT_DEV_DISCONN_LOCAL_HOST;
+ default:
+ return MGMT_DEV_DISCONN_UNKNOWN;
+ }
+}
+
void mgmt_device_disconnected(struct hci_dev *hdev, bdaddr_t *bdaddr,
u8 link_type, u8 addr_type, u8 reason,
bool mgmt_connected)
if (test_and_clear_bit(HCI_CONN_MGMT_CONNECTED, &conn->flags)) {
mgmt_device_disconnected(hdev, &conn->dst, conn->type,
- conn->dst_type, status, true);
+ conn->dst_type,
+ hci_to_mgmt_reason(status), true);
return;
}
monitor->state = ADV_MONITOR_STATE_OFFLOADED;
unlock:
- if (status)
+ if (status && msft->resuming)
hci_free_adv_monitor(hdev, monitor);
hci_dev_unlock(hdev);
/* Must be called on unlocked socket. */
static void sco_sock_close(struct sock *sk)
{
+ struct sco_conn *conn;
+
+ lock_sock(sk);
+ conn = sco_pi(sk)->conn;
+ if (conn)
+ sco_conn_hold(conn);
+ release_sock(sk);
+
+ if (conn)
+ disable_delayed_work_sync(&conn->timeout_work);
+
lock_sock(sk);
- sco_sock_clear_timer(sk);
__sco_sock_close(sk);
release_sock(sk);
+
+ if (conn)
+ sco_conn_put(conn);
}
static void sco_sock_init(struct sock *sk, struct sock *parent)
.get_sndtimeo = l2cap_chan_no_get_sndtimeo,
};
-static inline struct l2cap_chan *smp_new_conn_cb(struct l2cap_chan *pchan)
+static inline int smp_new_conn_cb(struct l2cap_chan *chan,
+ struct l2cap_chan *new_chan)
{
- struct l2cap_chan *chan;
-
- BT_DBG("pchan %p", pchan);
-
- chan = l2cap_chan_create();
- if (!chan)
- return NULL;
-
- chan->chan_type = pchan->chan_type;
- chan->ops = &smp_chan_ops;
- chan->scid = pchan->scid;
- chan->dcid = chan->scid;
- chan->imtu = pchan->imtu;
- chan->omtu = pchan->omtu;
- chan->mode = pchan->mode;
+ new_chan->ops = &smp_chan_ops;
/* Other L2CAP channels may request SMP routines in order to
* change the security level. This means that the SMP channel
* lock must be considered in its own category to avoid lockdep
* warnings.
*/
- atomic_set(&chan->nesting, L2CAP_NESTING_SMP);
-
- BT_DBG("created chan %p", chan);
+ atomic_set(&new_chan->nesting, L2CAP_NESTING_SMP);
- return chan;
+ return 0;
}
static const struct l2cap_ops smp_root_chan_ops = {
l2cap_add_scid(chan, cid);
- l2cap_chan_set_defaults(chan);
+ l2cap_chan_set_defaults(chan, NULL);
if (cid == L2CAP_CID_SMP) {
u8 bdaddr_type;
left - sizeof(struct ebt_entry_match) < m->match_size)
return -EINVAL;
+ if (strnlen(m->u.name, XT_EXTENSION_MAXNAMELEN) == XT_EXTENSION_MAXNAMELEN)
+ return -EINVAL;
+
match = xt_find_match(NFPROTO_BRIDGE, m->u.name, m->u.revision);
if (IS_ERR(match) || match->family != NFPROTO_BRIDGE) {
if (!IS_ERR(match))
* if an error occurs
*/
newinfo->chainstack =
- vmalloc_array(nr_cpu_ids,
- sizeof(*(newinfo->chainstack)));
+ vcalloc(nr_cpu_ids, sizeof(*(newinfo->chainstack)));
if (!newinfo->chainstack)
return -ENOMEM;
for_each_possible_cpu(i) {
if (copy_from_sockptr(&hlp, arg, sizeof(hlp)))
return -EFAULT;
+ hlp.name[sizeof(hlp.name) - 1] = '\0';
+
if (len != sizeof(hlp) + hlp.num_counters * sizeof(struct ebt_counter))
return -EINVAL;
memcpy(repl, &tmp, offsetof(struct ebt_replace, hook_entry));
+ repl->name[sizeof(repl->name) - 1] = '\0';
+
/* starting with hook_entry, 32 vs. 64 bit structures are different */
for (i = 0; i < NF_BR_NUMHOOKS; i++)
repl->hook_entry[i] = compat_ptr(tmp.hook_entry[i]);
if (copy_from_sockptr(&hlp, arg, sizeof(hlp)))
return -EFAULT;
+ hlp.name[sizeof(hlp.name) - 1] = '\0';
+
/* try real handler in case userland supplied needed padding */
if (len != sizeof(hlp) + hlp.num_counters * sizeof(struct ebt_counter))
return update_counters(net, arg, len);
#include <linux/can/skb.h>
#include <linux/can/bcm.h>
#include <linux/slab.h>
+#include <linux/workqueue.h>
#include <linux/spinlock.h>
#include <net/can.h>
#include <net/sock.h>
#define BCM_MIN_NAMELEN CAN_REQUIRED_SIZE(struct sockaddr_can, can_ifindex)
+static struct workqueue_struct *bcm_wq;
+
/*
* easy access to the first 64 bit of can(fd)_frame payload. cp->data is
* 64 bit aligned so the offset has to be multiples of 8 which is ensured
struct bcm_op {
struct list_head list;
struct rcu_head rcu;
+ struct work_struct work;
int ifindex;
canid_t can_id;
u32 flags;
- unsigned long frames_abs, frames_filtered;
+ atomic_long_t frames_abs, frames_filtered;
struct bcm_timeval ival1, ival2;
struct hrtimer timer, thrtimer;
ktime_t rx_stamp, kt_ival1, kt_ival2, kt_lastmsg;
int rx_ifindex;
+ int if_detected; /* first received ifindex in ANYDEV rx_op mode */
int cfsiz;
u32 count;
u32 nframes;
struct canfd_frame last_sframe;
struct sock *sk;
struct net_device *rx_reg_dev;
- spinlock_t bcm_tx_lock; /* protect currframe/count in runtime updates */
+ netdevice_tracker rx_reg_dev_tracker;
+ spinlock_t bcm_tx_lock; /* protect tx data and timer updates */
+ spinlock_t bcm_rx_update_lock; /* protect filter/timer data updates */
};
struct bcm_sock {
list_for_each_entry_rcu(op, &bo->rx_ops, list) {
- unsigned long reduction;
+ long reduction, frames_filtered, frames_abs;
+
+ frames_filtered = atomic_long_read(&op->frames_filtered);
+ frames_abs = atomic_long_read(&op->frames_abs);
/* print only active entries & prevent division by zero */
- if (!op->frames_abs)
+ if (!frames_abs)
continue;
seq_printf(m, "rx_op: %03X %-5s ", op->can_id,
(long long)ktime_to_us(op->kt_ival2));
seq_printf(m, "# recv %ld (%ld) => reduction: ",
- op->frames_filtered, op->frames_abs);
+ frames_filtered, frames_abs);
- reduction = 100 - (op->frames_filtered * 100) / op->frames_abs;
+ reduction = 100 - (frames_filtered * 100) / frames_abs;
seq_printf(m, "%s%ld%%\n",
(reduction == 100) ? "near " : "", reduction);
}
- list_for_each_entry(op, &bo->tx_ops, list) {
+ list_for_each_entry_rcu(op, &bo->tx_ops, list) {
seq_printf(m, "tx_op: %03X %s ", op->can_id,
bcm_proc_getifname(net, ifname, op->ifindex));
seq_printf(m, "t2=%lld ",
(long long)ktime_to_us(op->kt_ival2));
- seq_printf(m, "# sent %ld\n", op->frames_abs);
+ seq_printf(m, "# sent %ld\n",
+ atomic_long_read(&op->frames_abs));
}
seq_putc(m, '\n');
}
#endif /* CONFIG_PROC_FS */
+static void bcm_update_rx_stats(struct bcm_op *op)
+{
+ /* prevent overflow of the reduction% calculation in bcm_proc_show() */
+ if (atomic_long_inc_return(&op->frames_abs) > LONG_MAX / 100) {
+ atomic_long_set(&op->frames_filtered, 0);
+ atomic_long_set(&op->frames_abs, 0);
+ }
+}
+
+static void bcm_update_tx_stats(struct bcm_op *op)
+{
+ /* tx_op has no reduction% calculation - use the full range and
+ * just keep the displayed counter non-negative on overflow
+ */
+ if (atomic_long_inc_return(&op->frames_abs) == LONG_MAX)
+ atomic_long_set(&op->frames_abs, 0);
+}
+
/*
* bcm_can_tx - send the (next) CAN frame to the appropriate CAN interface
* of the given bcm tx op
*/
-static void bcm_can_tx(struct bcm_op *op)
+static void bcm_can_tx(struct bcm_op *op, struct canfd_frame *cf)
{
struct sk_buff *skb;
struct can_skb_ext *csx;
struct net_device *dev;
- struct canfd_frame *cf;
+ struct canfd_frame cframe;
+ bool cyclic = !cf;
+ unsigned int idx = 0;
int err;
/* no target device? => exit */
if (!op->ifindex)
return;
- /* read currframe under lock protection */
- spin_lock_bh(&op->bcm_tx_lock);
- cf = op->frames + op->cfsiz * op->currframe;
- spin_unlock_bh(&op->bcm_tx_lock);
+ if (cyclic) {
+ /* read currframe under lock protection */
+ spin_lock_bh(&op->bcm_tx_lock);
+ idx = op->currframe;
+ memcpy(&cframe, op->frames + op->cfsiz * idx, op->cfsiz);
+ cf = &cframe;
+ spin_unlock_bh(&op->bcm_tx_lock);
+ }
dev = dev_get_by_index(sock_net(op->sk), op->ifindex);
if (!dev) {
spin_lock_bh(&op->bcm_tx_lock);
if (!err)
- op->frames_abs++;
+ bcm_update_tx_stats(op);
- op->currframe++;
+ /* only advance the cyclic sequence if nothing reset currframe while
+ * we were sending - a concurrent TX_RESET_MULTI_IDX means this
+ * frame's bookkeeping belongs to a sequence that no longer exists
+ */
+ if (!cyclic || op->currframe == idx) {
+ op->currframe++;
- /* reached last frame? */
- if (op->currframe >= op->nframes)
- op->currframe = 0;
+ /* reached last frame? */
+ if (op->currframe >= op->nframes)
+ op->currframe = 0;
- if (op->count > 0)
- op->count--;
+ if (op->count > 0)
+ op->count--;
+ }
spin_unlock_bh(&op->bcm_tx_lock);
out:
{
ktime_t ival;
+ spin_lock_bh(&op->bcm_tx_lock);
+
if (op->kt_ival1 && op->count)
ival = op->kt_ival1;
- else if (op->kt_ival2)
+ else if (op->kt_ival2) {
ival = op->kt_ival2;
- else
+ } else {
+ spin_unlock_bh(&op->bcm_tx_lock);
return false;
+ }
+
+ spin_unlock_bh(&op->bcm_tx_lock);
hrtimer_set_expires(hrt, ktime_add(ktime_get(), ival));
return true;
{
struct bcm_op *op = container_of(hrtimer, struct bcm_op, timer);
struct bcm_msg_head msg_head;
+ bool tx_ival1, tx_ival2;
+
+ /* snapshot kt_ival1/kt_ival2/count under lock to avoid torn
+ * ktime_t reads racing with concurrent bcm_tx_setup() updates
+ */
+ spin_lock_bh(&op->bcm_tx_lock);
+ tx_ival1 = op->kt_ival1 && (op->count > 0);
+ tx_ival2 = !!op->kt_ival2;
+ spin_unlock_bh(&op->bcm_tx_lock);
- if (op->kt_ival1 && (op->count > 0)) {
- bcm_can_tx(op);
- if (!op->count && (op->flags & TX_COUNTEVT)) {
+ if (tx_ival1) {
+ u32 flags, count;
+ struct bcm_timeval ival1, ival2;
+ bcm_can_tx(op, NULL);
+
+ /* snapshot variables under lock to avoid torn reads racing
+ * with concurrent bcm_tx_setup() updates
+ */
+ spin_lock_bh(&op->bcm_tx_lock);
+ flags = op->flags;
+ count = op->count;
+ ival1 = op->ival1;
+ ival2 = op->ival2;
+ spin_unlock_bh(&op->bcm_tx_lock);
+
+ if (!count && (flags & TX_COUNTEVT)) {
/* create notification to user */
memset(&msg_head, 0, sizeof(msg_head));
msg_head.opcode = TX_EXPIRED;
- msg_head.flags = op->flags;
- msg_head.count = op->count;
- msg_head.ival1 = op->ival1;
- msg_head.ival2 = op->ival2;
+ msg_head.flags = flags;
+ msg_head.count = count;
+ msg_head.ival1 = ival1;
+ msg_head.ival2 = ival2;
msg_head.can_id = op->can_id;
msg_head.nframes = 0;
bcm_send_to_user(op, &msg_head, NULL, 0);
}
- } else if (op->kt_ival2) {
- bcm_can_tx(op);
+ } else if (tx_ival2) {
+ bcm_can_tx(op, NULL);
}
return bcm_tx_set_expiry(op, &op->timer) ?
{
struct bcm_msg_head head;
- /* update statistics */
- op->frames_filtered++;
-
- /* prevent statistics overflow */
- if (op->frames_filtered > ULONG_MAX/100)
- op->frames_filtered = op->frames_abs = 0;
+ /* update statistics (frames_filtered <= frames_abs) */
+ if (atomic_long_read(&op->frames_abs))
+ atomic_long_inc(&op->frames_filtered);
/* this element is not throttled anymore */
data->flags &= ~RX_THR;
struct bcm_op *op = container_of(hrtimer, struct bcm_op, timer);
struct bcm_msg_head msg_head;
+ spin_lock_bh(&op->bcm_rx_update_lock);
+
/* if user wants to be informed, when cyclic CAN-Messages come back */
if ((op->flags & RX_ANNOUNCE_RESUME) && op->last_frames) {
/* clear received CAN frames to indicate 'nothing received' */
msg_head.can_id = op->can_id;
msg_head.nframes = 0;
+ spin_unlock_bh(&op->bcm_rx_update_lock);
+
bcm_send_to_user(op, &msg_head, NULL, 0);
return HRTIMER_NORESTART;
static enum hrtimer_restart bcm_rx_thr_handler(struct hrtimer *hrtimer)
{
struct bcm_op *op = container_of(hrtimer, struct bcm_op, thrtimer);
+ enum hrtimer_restart ret;
+
+ spin_lock_bh(&op->bcm_rx_update_lock);
- if (bcm_rx_thr_flush(op)) {
+ /* kt_ival2 may have been concurrently cleared by bcm_rx_setup()
+ * before it cancels this timer - never forward with a zero
+ * interval in that case.
+ */
+ if (bcm_rx_thr_flush(op) && op->kt_ival2) {
hrtimer_forward_now(hrtimer, op->kt_ival2);
- return HRTIMER_RESTART;
+ ret = HRTIMER_RESTART;
} else {
/* rearm throttle handling */
op->kt_lastmsg = 0;
- return HRTIMER_NORESTART;
+ ret = HRTIMER_NORESTART;
}
+
+ spin_unlock_bh(&op->bcm_rx_update_lock);
+
+ return ret;
}
/*
{
struct bcm_op *op = (struct bcm_op *)data;
const struct canfd_frame *rxframe = (struct canfd_frame *)skb->data;
+ struct canfd_frame rtrframe;
unsigned int i;
unsigned char traffic_flags;
+ bool rtr_frame;
if (op->can_id != rxframe->can_id)
return;
return;
}
+ /* An ANYDEV op with an active RX timeout and/or throttle timer
+ * tracks a single source interface: claim the first interface that
+ * delivers a matching frame and reject frames from any other one,
+ * before hrtimer_cancel() below can touch op->timer - this avoids
+ * racing bcm_rx_timeout_handler() across concurrent interfaces.
+ * RX_RTR_FRAME ops are excluded, as kt_ival1/kt_ival2 may briefly
+ * hold a stale value from an earlier non-RTR configuration.
+ */
+ if (!op->ifindex) {
+ spin_lock_bh(&op->bcm_rx_update_lock);
+
+ if (!(op->flags & RX_RTR_FRAME) &&
+ (op->kt_ival1 || op->kt_ival2)) {
+ /* don't claim to vanishing interface */
+ if (!op->if_detected &&
+ READ_ONCE(skb->dev->reg_state) == NETREG_REGISTERED)
+ op->if_detected = skb->dev->ifindex;
+
+ if (op->if_detected != skb->dev->ifindex) {
+ spin_unlock_bh(&op->bcm_rx_update_lock);
+ return;
+ }
+ }
+
+ spin_unlock_bh(&op->bcm_rx_update_lock);
+ }
+
/* disable timeout */
hrtimer_cancel(&op->timer);
- /* save rx timestamp */
- op->rx_stamp = skb->tstamp;
- /* save originator for recvfrom() */
- op->rx_ifindex = skb->dev->ifindex;
- /* update statistics */
- op->frames_abs++;
+ /* op->flags/op->frames may be updated concurrently by bcm_rx_setup() */
+ spin_lock_bh(&op->bcm_rx_update_lock);
+
+ rtr_frame = op->flags & RX_RTR_FRAME;
+ if (rtr_frame) {
+ bcm_update_rx_stats(op);
+ /* snapshot RTR content under lock */
+ memcpy(&rtrframe, op->frames, op->cfsiz);
+ spin_unlock_bh(&op->bcm_rx_update_lock);
- if (op->flags & RX_RTR_FRAME) {
/* send reply for RTR-request (placed in op->frames[0]) */
- bcm_can_tx(op);
+ bcm_can_tx(op, &rtrframe);
return;
}
+ /* update statistics in the same critical section as bcm_rx_changed()
+ * below: frames_filtered must never be checked/incremented against a
+ * frames_abs snapshot from a concurrent bcm_rx_handler() call on
+ * another CPU for the same (wildcard) op, or frames_filtered can end
+ * up larger than frames_abs.
+ */
+ bcm_update_rx_stats(op);
+
/* compute flags to distinguish between own/local/remote CAN traffic */
traffic_flags = 0;
if (skb->sk) {
traffic_flags |= RX_OWN;
}
+ /* save rx timestamp and originator for recvfrom() under lock: an
+ * ANYDEV op without an active timer can still run concurrently on
+ * different CPUs, so content and meta data must be bundled here.
+ */
+ op->rx_stamp = skb->tstamp;
+ op->rx_ifindex = skb->dev->ifindex;
+
if (op->flags & RX_FILTER_ID) {
/* the easiest case */
bcm_rx_update_and_send(op, op->last_frames, rxframe,
rx_starttimer:
bcm_rx_starttimer(op);
+
+ spin_unlock_bh(&op->bcm_rx_update_lock);
}
/*
return NULL;
}
-static void bcm_free_op_rcu(struct rcu_head *rcu_head)
+static void bcm_free_op_work(struct work_struct *work)
{
- struct bcm_op *op = container_of(rcu_head, struct bcm_op, rcu);
+ struct bcm_op *op = container_of(work, struct bcm_op, work);
+
+ hrtimer_cancel(&op->timer);
+ hrtimer_cancel(&op->thrtimer);
if ((op->frames) && (op->frames != &op->sframe))
kfree(op->frames);
if ((op->last_frames) && (op->last_frames != &op->last_sframe))
kfree(op->last_frames);
+ /* the last possible access to op->timer/op->thrtimer has now
+ * happened above via hrtimer_cancel() - op->sk is no longer
+ * needed by any pending timer callback, so drop our reference
+ */
+ sock_put(op->sk);
+
kfree(op);
}
+static void bcm_free_op_rcu(struct rcu_head *rcu_head)
+{
+ struct bcm_op *op = container_of(rcu_head, struct bcm_op, rcu);
+
+ INIT_WORK(&op->work, bcm_free_op_work);
+ queue_work(bcm_wq, &op->work);
+}
+
static void bcm_remove_op(struct bcm_op *op)
{
hrtimer_cancel(&op->timer);
/* mark as removed subscription */
op->rx_reg_dev = NULL;
+ netdev_put(dev, &op->rx_reg_dev_tracker);
} else
printk(KERN_ERR "can-bcm: bcm_rx_unreg: registered device "
"mismatch %p %p\n", op->rx_reg_dev, dev);
* Only remove subscriptions that had not
* been removed due to NETDEV_UNREGISTER
* in bcm_notifier()
+ *
+ * op->rx_reg_dev is a tracked reference taken
+ * when the subscription was registered, so it
+ * stays valid here even if a concurrent
+ * NETDEV_UNREGISTER already unlisted the dev.
*/
- if (op->rx_reg_dev) {
- struct net_device *dev;
-
- dev = dev_get_by_index(sock_net(op->sk),
- op->ifindex);
- if (dev) {
- bcm_rx_unreg(dev, op);
- dev_put(dev);
- }
- }
+ if (op->rx_reg_dev)
+ bcm_rx_unreg(op->rx_reg_dev, op);
} else
can_rx_unregister(sock_net(op->sk), NULL,
op->can_id,
struct bcm_sock *bo = bcm_sk(sk);
struct bcm_op *op;
struct canfd_frame *cf;
+ bool add_op_to_list = false;
unsigned int i;
int err;
/* check the given can_id */
op = bcm_find_op(&bo->tx_ops, msg_head, ifindex);
if (op) {
+ void *new_frames;
+
/* update existing BCM operation */
/*
if (msg_head->nframes > op->nframes)
return -E2BIG;
- /* update CAN frames content */
+ /* get new CAN frames content into a staging buffer before
+ * locking: validate and normalize the frames there so that
+ * bcm_can_tx() / bcm_tx_timeout_handler() never observe a
+ * partially updated or unvalidated frame in op->frames
+ */
+ new_frames = kmalloc(msg_head->nframes * op->cfsiz, GFP_KERNEL);
+ if (!new_frames)
+ return -ENOMEM;
+
for (i = 0; i < msg_head->nframes; i++) {
- cf = op->frames + op->cfsiz * i;
+ cf = new_frames + op->cfsiz * i;
err = memcpy_from_msg((u8 *)cf, msg, op->cfsiz);
+ if (err < 0) {
+ kfree(new_frames);
+ return err;
+ }
if (op->flags & CAN_FD_FRAME) {
if (cf->len > 64)
err = -EINVAL;
}
- if (err < 0)
+ if (err < 0) {
+ kfree(new_frames);
return err;
+ }
if (msg_head->flags & TX_CP_CAN_ID) {
/* copy can_id into frame */
cf->can_id = msg_head->can_id;
}
}
+
+ spin_lock_bh(&op->bcm_tx_lock);
+
+ /* update CAN frames content */
+ memcpy(op->frames, new_frames, msg_head->nframes * op->cfsiz);
+
op->flags = msg_head->flags;
- /* only lock for unlikely count/nframes/currframe changes */
if (op->nframes != msg_head->nframes ||
- op->flags & TX_RESET_MULTI_IDX ||
- op->flags & SETTIMER) {
-
- spin_lock_bh(&op->bcm_tx_lock);
+ op->flags & TX_RESET_MULTI_IDX) {
+ /* potentially update changed nframes */
+ op->nframes = msg_head->nframes;
+ /* restart multiple frame transmission */
+ op->currframe = 0;
+ }
- if (op->nframes != msg_head->nframes ||
- op->flags & TX_RESET_MULTI_IDX) {
- /* potentially update changed nframes */
- op->nframes = msg_head->nframes;
- /* restart multiple frame transmission */
- op->currframe = 0;
- }
+ if (op->flags & SETTIMER)
+ op->count = msg_head->count;
- if (op->flags & SETTIMER)
- op->count = msg_head->count;
+ spin_unlock_bh(&op->bcm_tx_lock);
- spin_unlock_bh(&op->bcm_tx_lock);
- }
+ kfree(new_frames);
} else {
/* insert new BCM operation for the given can_id */
/* bcm_can_tx / bcm_tx_timeout_handler needs this */
op->sk = sk;
+ sock_hold(sk);
op->ifindex = ifindex;
/* initialize uninitialized (kzalloc) structure */
hrtimer_setup(&op->thrtimer, hrtimer_dummy_timeout, CLOCK_MONOTONIC,
HRTIMER_MODE_REL_SOFT);
- /* add this bcm_op to the list of the tx_ops */
- list_add(&op->list, &bo->tx_ops);
+ add_op_to_list = true;
} /* if ((op = bcm_find_op(&bo->tx_ops, msg_head->can_id, ifindex))) */
if (op->flags & SETTIMER) {
/* set timer values */
+ spin_lock_bh(&op->bcm_tx_lock);
op->ival1 = msg_head->ival1;
op->ival2 = msg_head->ival2;
op->kt_ival1 = bcm_timeval_to_ktime(msg_head->ival1);
op->kt_ival2 = bcm_timeval_to_ktime(msg_head->ival2);
+ spin_unlock_bh(&op->bcm_tx_lock);
/* disable an active timer due to zero values? */
if (!op->kt_ival1 && !op->kt_ival2)
op->flags |= TX_ANNOUNCE;
}
+ /* add this bcm_op to the list of the tx_ops? */
+ if (add_op_to_list)
+ list_add_rcu(&op->list, &bo->tx_ops);
+
if (op->flags & TX_ANNOUNCE)
- bcm_can_tx(op);
+ bcm_can_tx(op, NULL);
if (op->flags & STARTTIMER)
bcm_tx_start_timer(op);
return err;
}
+static int bcm_rx_setup_rtr_check(struct bcm_msg_head *msg_head,
+ struct bcm_op *op, void *new_frames)
+{
+ struct canfd_frame *frame0 = new_frames;
+
+ if (!(msg_head->flags & RX_RTR_FRAME))
+ return 0;
+
+ /* this frame is sent out as-is by bcm_can_tx() whenever a matching
+ * remote request is received, so validate its length the same way
+ * bcm_tx_setup() validates TX_SETUP frames before installing it
+ */
+ if (msg_head->flags & CAN_FD_FRAME) {
+ if (frame0->len > 64)
+ return -EINVAL;
+ } else {
+ if (frame0->len > 8)
+ return -EINVAL;
+ }
+
+ /* funny feature in RX(!)_SETUP only for RTR-mode:
+ * copy can_id into frame BUT without RTR-flag to
+ * prevent a full-load-loopback-test ... ;-]
+ * normalize this on the staged buffer, before it is
+ * ever installed into op->frames.
+ */
+ if ((msg_head->flags & TX_CP_CAN_ID) ||
+ frame0->can_id == op->can_id)
+ frame0->can_id = op->can_id & ~CAN_RTR_FLAG;
+
+ return 0;
+}
+
/*
* bcm_rx_setup - create or update a bcm rx op (for bcm_sendmsg)
*/
struct bcm_sock *bo = bcm_sk(sk);
struct bcm_op *op;
int do_rx_register;
+ int new_op = 0;
int err = 0;
if ((msg_head->flags & RX_FILTER_ID) || (!(msg_head->nframes))) {
/* check the given can_id */
op = bcm_find_op(&bo->rx_ops, msg_head, ifindex);
if (op) {
+ void *new_frames = NULL;
+
/* update existing BCM operation */
/*
return -E2BIG;
if (msg_head->nframes) {
- /* update CAN frames content */
- err = memcpy_from_msg(op->frames, msg,
+ /* get new CAN frames content before locking */
+ new_frames = kmalloc(msg_head->nframes * op->cfsiz,
+ GFP_KERNEL);
+ if (!new_frames)
+ return -ENOMEM;
+
+ err = memcpy_from_msg(new_frames, msg,
msg_head->nframes * op->cfsiz);
- if (err < 0)
+ if (err < 0) {
+ kfree(new_frames);
return err;
+ }
- /* clear last_frames to indicate 'nothing received' */
- memset(op->last_frames, 0, msg_head->nframes * op->cfsiz);
+ err = bcm_rx_setup_rtr_check(msg_head, op, new_frames);
+ if (err < 0) {
+ kfree(new_frames);
+ return err;
+ }
}
+ spin_lock_bh(&op->bcm_rx_update_lock);
op->nframes = msg_head->nframes;
op->flags = msg_head->flags;
- /* Only an update -> do not call can_rx_register() */
- do_rx_register = 0;
+ if (msg_head->nframes) {
+ /* update CAN frames content */
+ memcpy(op->frames, new_frames,
+ msg_head->nframes * op->cfsiz);
+
+ /* clear last_frames to indicate 'nothing received' */
+ memset(op->last_frames, 0,
+ msg_head->nframes * op->cfsiz);
+ }
+
+ if (msg_head->flags & SETTIMER) {
+ op->ival1 = msg_head->ival1;
+ op->ival2 = msg_head->ival2;
+ op->kt_ival1 = bcm_timeval_to_ktime(msg_head->ival1);
+ op->kt_ival2 = bcm_timeval_to_ktime(msg_head->ival2);
+ op->kt_lastmsg = 0;
+ op->if_detected = 0; /* reclaim ifindex in ANYDEV mode */
+ }
+ spin_unlock_bh(&op->bcm_rx_update_lock);
+
+ /* free temporary frames / kfree(NULL) is safe */
+ kfree(new_frames);
+
+ /* Don't register a new CAN filter for the rx_op update unless
+ * a concurrent NETDEV_UNREGISTER notifier already tore down
+ * the previous registration. In this case the receiver needs
+ * to be re-registered here so that this update doesn't
+ * silently stop delivering frames for the given ifindex.
+ * Ops with ifindex = 0 (all CAN interfaces) never carry a
+ * tracked rx_reg_dev and stay registered as-is.
+ */
+ do_rx_register = (ifindex && !op->rx_reg_dev) ? 1 : 0;
} else {
/* insert new BCM operation for the given can_id */
return -ENOMEM;
spin_lock_init(&op->bcm_tx_lock);
+ spin_lock_init(&op->bcm_rx_update_lock);
op->can_id = msg_head->can_id;
op->nframes = msg_head->nframes;
op->cfsiz = CFSIZ(msg_head->flags);
if (msg_head->nframes) {
err = memcpy_from_msg(op->frames, msg,
msg_head->nframes * op->cfsiz);
- if (err < 0) {
- if (op->frames != &op->sframe)
- kfree(op->frames);
- if (op->last_frames != &op->last_sframe)
- kfree(op->last_frames);
- kfree(op);
- return err;
- }
+ if (err < 0)
+ goto free_op;
+
+ err = bcm_rx_setup_rtr_check(msg_head, op, op->frames);
+ if (err < 0)
+ goto free_op;
}
/* bcm_can_tx / bcm_tx_timeout_handler needs this */
op->sk = sk;
+ sock_hold(sk);
op->ifindex = ifindex;
/* ifindex for timeout events w/o previous frame reception */
hrtimer_setup(&op->thrtimer, bcm_rx_thr_handler, CLOCK_MONOTONIC,
HRTIMER_MODE_REL_SOFT);
- /* add this bcm_op to the list of the rx_ops */
- list_add(&op->list, &bo->rx_ops);
-
/* call can_rx_register() */
do_rx_register = 1;
+ new_op = 1;
} /* if ((op = bcm_find_op(&bo->rx_ops, msg_head->can_id, ifindex))) */
/* check flags */
if (op->flags & RX_RTR_FRAME) {
- struct canfd_frame *frame0 = op->frames;
-
/* no timers in RTR-mode */
hrtimer_cancel(&op->thrtimer);
hrtimer_cancel(&op->timer);
-
- /*
- * funny feature in RX(!)_SETUP only for RTR-mode:
- * copy can_id into frame BUT without RTR-flag to
- * prevent a full-load-loopback-test ... ;-]
- */
- if ((op->flags & TX_CP_CAN_ID) ||
- (frame0->can_id == op->can_id))
- frame0->can_id = op->can_id & ~CAN_RTR_FLAG;
-
} else {
if (op->flags & SETTIMER) {
- /* set timer value */
- op->ival1 = msg_head->ival1;
- op->ival2 = msg_head->ival2;
- op->kt_ival1 = bcm_timeval_to_ktime(msg_head->ival1);
- op->kt_ival2 = bcm_timeval_to_ktime(msg_head->ival2);
+ /* set timers (locked) for newly created op */
+ if (new_op) {
+ spin_lock_bh(&op->bcm_rx_update_lock);
+ op->ival1 = msg_head->ival1;
+ op->ival2 = msg_head->ival2;
+ op->kt_ival1 = bcm_timeval_to_ktime(msg_head->ival1);
+ op->kt_ival2 = bcm_timeval_to_ktime(msg_head->ival2);
+ op->kt_lastmsg = 0;
+ spin_unlock_bh(&op->bcm_rx_update_lock);
+ }
/* disable an active timer due to zero value? */
if (!op->kt_ival1)
* In any case cancel the throttle timer, flush
* potentially blocked msgs and reset throttle handling
*/
- op->kt_lastmsg = 0;
hrtimer_cancel(&op->thrtimer);
+
+ spin_lock_bh(&op->bcm_rx_update_lock);
bcm_rx_thr_flush(op);
+ spin_unlock_bh(&op->bcm_rx_update_lock);
}
if ((op->flags & STARTTIMER) && op->kt_ival1)
HRTIMER_MODE_REL_SOFT);
}
- /* now we can register for can_ids, if we added a new bcm_op */
+ /* now we can register for can_ids, if we added a new bcm_op
+ * or need to re-register after a NETDEV_UNREGISTER tore down
+ * the previous registration of an existing op
+ */
if (do_rx_register) {
if (ifindex) {
struct net_device *dev;
bcm_rx_handler, op,
"bcm", sk);
- op->rx_reg_dev = dev;
+ /* keep a tracked reference so that a later
+ * unregister can safely reach the device even
+ * if a concurrent NETDEV_UNREGISTER has
+ * already unlisted it by ifindex
+ */
+ if (!err) {
+ op->rx_reg_dev = dev;
+ netdev_hold(dev,
+ &op->rx_reg_dev_tracker,
+ GFP_KERNEL);
+ }
dev_put(dev);
+ } else {
+ /* the requested device is gone - do not
+ * silently succeed without registering
+ */
+ err = -ENODEV;
}
- } else
+ } else {
err = can_rx_register(sock_net(sk), NULL, op->can_id,
REGMASK(op->can_id),
bcm_rx_handler, op, "bcm", sk);
+ }
+
if (err) {
- /* this bcm rx op is broken -> remove it */
- list_del_rcu(&op->list);
- bcm_remove_op(op);
+ /* newly created bcm rx op is broken -> remove it */
+ if (new_op) {
+ bcm_remove_op(op);
+ return err;
+ }
+
+ /* an existing op just stays unregistered.
+ * Cancel op->timer and (defensively) op->thrtimer.
+ * Other settings can't be reached until the next
+ * successful RX_SETUP.
+ */
+ hrtimer_cancel(&op->timer);
+ hrtimer_cancel(&op->thrtimer);
return err;
}
+
+ /* add a new bcm_op to the list of the rx_ops */
+ if (new_op)
+ list_add_rcu(&op->list, &bo->rx_ops);
}
return msg_head->nframes * op->cfsiz + MHSIZ;
+
+free_op:
+ if (op->frames != &op->sframe)
+ kfree(op->frames);
+ if (op->last_frames != &op->last_sframe)
+ kfree(op->last_frames);
+ kfree(op);
+ return err;
}
/*
{
struct sock *sk = sock->sk;
struct bcm_sock *bo = bcm_sk(sk);
- int ifindex = bo->ifindex; /* default ifindex for this bcm_op */
+ int ifindex;
struct bcm_msg_head msg_head;
int cfsiz;
int ret; /* read bytes or error codes as return value */
- if (!bo->bound)
+ /* Lockless fast-path check for bound socket */
+ if (!READ_ONCE(bo->bound))
return -ENOTCONN;
/* check for valid message length from userspace */
if ((size - MHSIZ) % cfsiz)
return -EINVAL;
+ lock_sock(sk);
+
+ /* Re-validate under the socket lock: a concurrent bcm_notify()
+ * may have unbound this socket (device removal) after the
+ * lockless fast-path check above. bo->ifindex is only ever
+ * mutated under lock_sock(), so reading it here - instead of
+ * before taking the lock - guarantees it can't be observed
+ * torn against bo->bound.
+ */
+ if (!bo->bound) {
+ ret = -ENOTCONN;
+ goto out_release;
+ }
+
+ /* default ifindex for this bcm_op */
+ ifindex = bo->ifindex;
+
/* check for alternative ifindex for this bcm_op */
if (!ifindex && msg->msg_name) {
/* no bound device as default => check msg_name */
DECLARE_SOCKADDR(struct sockaddr_can *, addr, msg->msg_name);
- if (msg->msg_namelen < BCM_MIN_NAMELEN)
- return -EINVAL;
+ if (msg->msg_namelen < BCM_MIN_NAMELEN) {
+ ret = -EINVAL;
+ goto out_release;
+ }
- if (addr->can_family != AF_CAN)
- return -EINVAL;
+ if (addr->can_family != AF_CAN) {
+ ret = -EINVAL;
+ goto out_release;
+ }
/* ifindex from sendto() */
ifindex = addr->can_ifindex;
struct net_device *dev;
dev = dev_get_by_index(sock_net(sk), ifindex);
- if (!dev)
- return -ENODEV;
+ if (!dev) {
+ ret = -ENODEV;
+ goto out_release;
+ }
if (dev->type != ARPHRD_CAN) {
dev_put(dev);
- return -ENODEV;
+ ret = -ENODEV;
+ goto out_release;
}
dev_put(dev);
}
}
- lock_sock(sk);
-
switch (msg_head.opcode) {
case TX_SETUP:
break;
}
+out_release:
release_sock(sk);
return ret;
case NETDEV_UNREGISTER:
lock_sock(sk);
- /* remove device specific receive entries */
- list_for_each_entry(op, &bo->rx_ops, list)
+ /* rx_ops: remove device specific receive entries */
+ list_for_each_entry(op, &bo->rx_ops, list) {
if (op->rx_reg_dev == dev)
bcm_rx_unreg(dev, op);
+ /* release an ANYDEV op's claim (see bcm_rx_handler())
+ * on this now confirmed-gone interface.
+ */
+ if (!op->ifindex) {
+ spin_lock_bh(&op->bcm_rx_update_lock);
+ if (op->if_detected == dev->ifindex)
+ op->if_detected = 0;
+ spin_unlock_bh(&op->bcm_rx_update_lock);
+ }
+ }
+
+ /* tx_ops: stop device specific cyclic transmissions on the
+ * vanishing ifindex. Cancelling the timer is enough to stop
+ * cyclic bcm_can_tx() calls as there is no re-arming.
+ */
+ list_for_each_entry(op, &bo->tx_ops, list)
+ if (op->ifindex == dev->ifindex)
+ hrtimer_cancel(&op->timer);
+
/* remove device reference, if this is our bound device */
if (bo->bound && bo->ifindex == dev->ifindex) {
#if IS_ENABLED(CONFIG_PROC_FS)
bo->bcm_proc_read = NULL;
}
#endif
- bo->bound = 0;
+ /* Paired with the lockless fast-path check in
+ * bcm_sendmsg(); bo->ifindex itself is only ever
+ * accessed under lock_sock() so it needs no
+ * annotation.
+ */
+ WRITE_ONCE(bo->bound, 0);
bo->ifindex = 0;
notify_enodev = 1;
}
remove_proc_entry(bo->procname, net->can.bcmproc_dir);
#endif /* CONFIG_PROC_FS */
- list_for_each_entry_safe(op, next, &bo->tx_ops, list)
+ list_for_each_entry_safe(op, next, &bo->tx_ops, list) {
+ list_del_rcu(&op->list);
bcm_remove_op(op);
+ }
list_for_each_entry_safe(op, next, &bo->rx_ops, list) {
/*
* Only remove subscriptions that had not
* been removed due to NETDEV_UNREGISTER
* in bcm_notifier()
+ *
+ * op->rx_reg_dev is a tracked reference taken
+ * when the subscription was registered, so it
+ * stays valid here even if a concurrent
+ * NETDEV_UNREGISTER already unlisted the device.
*/
- if (op->rx_reg_dev) {
- struct net_device *dev;
-
- dev = dev_get_by_index(net, op->ifindex);
- if (dev) {
- bcm_rx_unreg(dev, op);
- dev_put(dev);
- }
- }
+ if (op->rx_reg_dev)
+ bcm_rx_unreg(op->rx_reg_dev, op);
} else
can_rx_unregister(net, NULL, op->can_id,
REGMASK(op->can_id),
synchronize_rcu();
- list_for_each_entry_safe(op, next, &bo->rx_ops, list)
+ list_for_each_entry_safe(op, next, &bo->rx_ops, list) {
+ list_del_rcu(&op->list);
bcm_remove_op(op);
+ }
/* remove device reference */
if (bo->bound) {
- bo->bound = 0;
+ WRITE_ONCE(bo->bound, 0);
bo->ifindex = 0;
}
}
#endif /* CONFIG_PROC_FS */
- bo->bound = 1;
+ /* bo->ifindex above is fully assigned before this point; pairs
+ * with the lockless fast-path check in bcm_sendmsg()
+ */
+ WRITE_ONCE(bo->bound, 1);
fail:
release_sock(sk);
{
int err;
+ bcm_wq = alloc_workqueue("can-bcm-wq", WQ_UNBOUND, 0);
+ if (!bcm_wq)
+ return -ENOMEM;
+
pr_info("can: broadcast manager protocol\n");
err = register_pernet_subsys(&canbcm_pernet_ops);
if (err)
- return err;
+ goto register_pernet_failed;
err = register_netdevice_notifier(&canbcm_notifier);
if (err)
unregister_netdevice_notifier(&canbcm_notifier);
register_notifier_failed:
unregister_pernet_subsys(&canbcm_pernet_ops);
+register_pernet_failed:
+ destroy_workqueue(bcm_wq);
return err;
}
can_proto_unregister(&bcm_can_proto);
unregister_netdevice_notifier(&canbcm_notifier);
unregister_pernet_subsys(&canbcm_pernet_ops);
+ rcu_barrier();
+ destroy_workqueue(bcm_wq);
}
module_init(bcm_module_init);
struct sock sk;
int bound;
int ifindex;
+ struct net_device *dev;
+ netdevice_tracker dev_tracker;
canid_t txid;
canid_t rxid;
ktime_t tx_gap;
ktime_t lastrxcf_tstamp;
- struct hrtimer rxtimer, txtimer, txfrtimer;
+ struct hrtimer rxtimer, txtimer, txfrtimer, echotimer;
struct can_isotp_options opt;
struct can_isotp_fc_options rxfc, txfc;
struct can_isotp_ll_options ll;
u32 force_tx_stmin;
u32 force_rx_stmin;
u32 cfecho; /* consecutive frame echo tag */
+ u32 tx_gen; /* generation, bumped per new tx transfer */
struct tpcon rx, tx;
struct list_head notifier;
wait_queue_head_t wait;
hrtimer_cancel(&so->txtimer);
+ /* isotp_tx_timeout() may have given up on this job while
+ * hrtimer_cancel() above waited for it to finish; so->rx_lock
+ * (held by our caller isotp_rcv()) rules out a concurrent claim,
+ * so a plain recheck is enough here.
+ */
+ if (so->tx.state != ISOTP_WAIT_FC &&
+ so->tx.state != ISOTP_WAIT_FIRST_FC)
+ return 1;
+
if ((cf->len < ae + FC_CONTENT_SZ) ||
((so->opt.flags & ISOTP_CHECK_PADDING) &&
check_pad(so, cf, ae + FC_CONTENT_SZ, so->opt.rxpad_content))) {
so->tx.bs = 0;
so->tx.state = ISOTP_SENDING;
/* send CF frame and enable echo timeout handling */
- hrtimer_start(&so->txtimer, ktime_set(ISOTP_ECHO_TIMEOUT, 0),
+ hrtimer_start(&so->echotimer, ktime_set(ISOTP_ECHO_TIMEOUT, 0),
HRTIMER_MODE_REL_SOFT);
isotp_send_cframe(so);
break;
hrtimer_cancel(&so->rxtimer);
+ /* isotp_rx_timer_handler() may have raced us for so->rx.state
+ * while hrtimer_cancel() above waited for it to finish, already
+ * reporting ETIMEDOUT and resetting the reception; don't process
+ * this CF into a reassembly that has already been given up on.
+ */
+ if (so->rx.state != ISOTP_WAIT_DATA)
+ return 1;
+
/* CFs are never longer than the FF */
if (cf->len > so->rx.ll_dl)
return 1;
struct canfd_frame *cf = (struct canfd_frame *)skb->data;
/* only handle my own local echo CF/SF skb's (no FF!) */
- if (skb->sk != sk || so->cfecho != *(u32 *)cf->data)
+ if (skb->sk != sk)
return;
+ /* unlike isotp_rcv_fc()/isotp_rcv_cf(), not already under so->rx_lock
+ * (no isotp_rcv() caller here), so take it ourselves
+ */
+ spin_lock(&so->rx_lock);
+
+ /* so->cfecho may since belong to a new transfer; recheck under lock */
+ if (so->cfecho != *(u32 *)cf->data)
+ goto out_unlock;
+
/* cancel local echo timeout */
- hrtimer_cancel(&so->txtimer);
+ hrtimer_cancel(&so->echotimer);
/* local echo skb with consecutive frame has been consumed */
so->cfecho = 0;
+ /* claiming a transfer also takes so->rx_lock, so a plain recheck
+ * is enough: so->tx.state can't have flipped to ISOTP_SENDING for
+ * a new claim while we're still in here
+ */
+ if (so->tx.state != ISOTP_SENDING)
+ goto out_unlock;
+
if (so->tx.idx >= so->tx.len) {
/* we are done */
so->tx.state = ISOTP_IDLE;
wake_up_interruptible(&so->wait);
- return;
+ goto out_unlock;
}
if (so->txfc.bs && so->tx.bs >= so->txfc.bs) {
so->tx.state = ISOTP_WAIT_FC;
hrtimer_start(&so->txtimer, ktime_set(ISOTP_FC_TIMEOUT, 0),
HRTIMER_MODE_REL_SOFT);
- return;
+ goto out_unlock;
}
/* no gap between data frames needed => use burst mode */
if (!so->tx_gap) {
/* enable echo timeout handling */
- hrtimer_start(&so->txtimer, ktime_set(ISOTP_ECHO_TIMEOUT, 0),
+ hrtimer_start(&so->echotimer, ktime_set(ISOTP_ECHO_TIMEOUT, 0),
HRTIMER_MODE_REL_SOFT);
isotp_send_cframe(so);
- return;
+ goto out_unlock;
}
/* start timer to send next consecutive frame with correct delay */
hrtimer_start(&so->txfrtimer, so->tx_gap, HRTIMER_MODE_REL_SOFT);
+
+out_unlock:
+ spin_unlock(&so->rx_lock);
}
-static enum hrtimer_restart isotp_tx_timer_handler(struct hrtimer *hrtimer)
+/* shared by so->txtimer's and so->echotimer's callbacks. Both timers get
+ * cancelled under so->rx_lock elsewhere, so this must stay lock-free to
+ * avoid deadlocking with that; uses so->tx_gen instead to avoid tainting
+ * a new transfer with an error from the one that just timed out.
+ */
+static enum hrtimer_restart isotp_tx_timeout(struct isotp_sock *so)
{
- struct isotp_sock *so = container_of(hrtimer, struct isotp_sock,
- txtimer);
struct sock *sk = &so->sk;
+ u32 gen = READ_ONCE(so->tx_gen);
+ u32 old_state = READ_ONCE(so->tx.state);
/* don't handle timeouts in IDLE or SHUTDOWN state */
- if (so->tx.state == ISOTP_IDLE || so->tx.state == ISOTP_SHUTDOWN)
+ if (old_state == ISOTP_IDLE || old_state == ISOTP_SHUTDOWN)
+ return HRTIMER_NORESTART;
+
+ /* only claim the timeout if the state is still unchanged */
+ if (cmpxchg(&so->tx.state, old_state, ISOTP_IDLE) != old_state)
return HRTIMER_NORESTART;
/* we did not get any flow control or echo frame in time */
- /* report 'communication error on send' */
- sk->sk_err = ECOMM;
- if (!sock_flag(sk, SOCK_DEAD))
- sk_error_report(sk);
+ if (READ_ONCE(so->tx_gen) == gen) {
+ /* report 'communication error on send' */
+ sk->sk_err = ECOMM;
+ if (!sock_flag(sk, SOCK_DEAD))
+ sk_error_report(sk);
+ }
- /* reset tx state */
- so->tx.state = ISOTP_IDLE;
wake_up_interruptible(&so->wait);
return HRTIMER_NORESTART;
}
+/* so->txtimer: fires when a Flow Control frame does not arrive in time */
+static enum hrtimer_restart isotp_tx_timer_handler(struct hrtimer *hrtimer)
+{
+ struct isotp_sock *so = container_of(hrtimer, struct isotp_sock,
+ txtimer);
+
+ return isotp_tx_timeout(so);
+}
+
+/* so->echotimer: fires when a sent CF/SF's local echo does not arrive */
+static enum hrtimer_restart isotp_echo_timer_handler(struct hrtimer *hrtimer)
+{
+ struct isotp_sock *so = container_of(hrtimer, struct isotp_sock,
+ echotimer);
+
+ return isotp_tx_timeout(so);
+}
+
static enum hrtimer_restart isotp_txfr_timer_handler(struct hrtimer *hrtimer)
{
struct isotp_sock *so = container_of(hrtimer, struct isotp_sock,
txfrtimer);
/* start echo timeout handling and cover below protocol error */
- hrtimer_start(&so->txtimer, ktime_set(ISOTP_ECHO_TIMEOUT, 0),
+ hrtimer_start(&so->echotimer, ktime_set(ISOTP_ECHO_TIMEOUT, 0),
HRTIMER_MODE_REL_SOFT);
/* cfecho should be consumed by isotp_rcv_echo() here */
int ae = (so->opt.flags & CAN_ISOTP_EXTEND_ADDR) ? 1 : 0;
int wait_tx_done = (so->opt.flags & CAN_ISOTP_WAIT_TX_DONE) ? 1 : 0;
s64 hrtimer_sec = ISOTP_ECHO_TIMEOUT;
+ struct hrtimer *tx_hrt = &so->echotimer;
+ u32 new_state = ISOTP_SENDING;
int off;
int err;
if (!so->bound || so->tx.state == ISOTP_SHUTDOWN)
return -EADDRNOTAVAIL;
- while (cmpxchg(&so->tx.state, ISOTP_IDLE, ISOTP_SENDING) != ISOTP_IDLE) {
+ /* claim the socket under so->rx_lock: this serializes the claim
+ * with the RX path and with sendmsg()'s own error paths below, so
+ * none of them can ever see a transfer mid-claim
+ */
+ for (;;) {
+ spin_lock_bh(&so->rx_lock);
+ if (READ_ONCE(so->tx.state) == ISOTP_IDLE)
+ break;
+ spin_unlock_bh(&so->rx_lock);
+
/* we do not support multiple buffers - for now */
if (msg->msg_flags & MSG_DONTWAIT)
return -EAGAIN;
return -EADDRNOTAVAIL;
/* wait for complete transmission of current pdu */
- err = wait_event_interruptible(so->wait, so->tx.state == ISOTP_IDLE);
+ err = wait_event_interruptible(so->wait,
+ so->tx.state == ISOTP_IDLE);
if (err)
- goto err_event_drop;
+ return err;
+ }
+
+ /* new transfer: bump so->tx_gen and drain the old one's timers,
+ * still under the so->rx_lock we just claimed the socket with
+ */
+ WRITE_ONCE(so->tx.state, ISOTP_SENDING);
+ WRITE_ONCE(so->tx_gen, READ_ONCE(so->tx_gen) + 1);
+ hrtimer_cancel(&so->txtimer);
+ hrtimer_cancel(&so->echotimer);
+ hrtimer_cancel(&so->txfrtimer);
+ so->cfecho = 0;
+ spin_unlock_bh(&so->rx_lock);
+
+ /* so->bound is only checked once above - a wakeup may have
+ * unbound/rebound the socket meanwhile, so re-validate it
+ */
+ if (!so->bound) {
+ err = -EADDRNOTAVAIL;
+ goto err_out_drop;
}
/* PDU size > default => try max_pdu_size */
so->cfecho = *(u32 *)cf->data;
} else {
/* standard flow control check */
- so->tx.state = ISOTP_WAIT_FIRST_FC;
+ new_state = ISOTP_WAIT_FIRST_FC;
/* start timeout for FC */
hrtimer_sec = ISOTP_FC_TIMEOUT;
+ tx_hrt = &so->txtimer;
/* no CF echo tag for isotp_rcv_echo() (FF-mode) */
so->cfecho = 0;
}
}
- hrtimer_start(&so->txtimer, ktime_set(hrtimer_sec, 0),
+ spin_lock_bh(&so->rx_lock);
+ if (so->tx.state == ISOTP_SHUTDOWN) {
+ /* isotp_release() has since taken over and already drained
+ * our timers - don't send into a socket that's going away
+ */
+ spin_unlock_bh(&so->rx_lock);
+ kfree_skb(skb);
+ dev_put(dev);
+ wake_up_interruptible(&so->wait);
+ return -EADDRNOTAVAIL;
+ }
+ /* WAIT_FIRST_FC for standard FF, else stays ISOTP_SENDING */
+ so->tx.state = new_state;
+ hrtimer_start(tx_hrt, ktime_set(hrtimer_sec, 0),
HRTIMER_MODE_REL_SOFT);
+ spin_unlock_bh(&so->rx_lock);
/* send the first or only CAN frame */
cf->flags = so->ll.tx_flags;
pr_notice_once("can-isotp: %s: can_send_ret %pe\n",
__func__, ERR_PTR(err));
+ spin_lock_bh(&so->rx_lock);
/* no transmission -> no timeout monitoring */
- hrtimer_cancel(&so->txtimer);
-
- /* reset consecutive frame echo tag */
- so->cfecho = 0;
-
- goto err_out_drop;
+ hrtimer_cancel(tx_hrt);
+ goto err_out_drop_locked;
}
if (wait_tx_done) {
return size;
+err_out_drop:
+ /* claimed but nothing sent yet - no timer to cancel */
+ spin_lock_bh(&so->rx_lock);
+ goto err_out_drop_locked;
err_event_drop:
- /* got signal: force tx state machine to be idle */
- so->tx.state = ISOTP_IDLE;
+ /* interrupted waiting on our own transfer - drain its timers */
+ spin_lock_bh(&so->rx_lock);
hrtimer_cancel(&so->txfrtimer);
hrtimer_cancel(&so->txtimer);
-err_out_drop:
- /* drop this PDU and unlock a potential wait queue */
+ hrtimer_cancel(&so->echotimer);
+err_out_drop_locked:
+ /* release the claim; so->rx_lock still held from above */
+ so->cfecho = 0;
so->tx.state = ISOTP_IDLE;
+ spin_unlock_bh(&so->rx_lock);
wake_up_interruptible(&so->wait);
return err;
so = isotp_sk(sk);
net = sock_net(sk);
- /* wait for complete transmission of current pdu */
- while (wait_event_interruptible(so->wait, so->tx.state == ISOTP_IDLE) == 0 &&
- cmpxchg(&so->tx.state, ISOTP_IDLE, ISOTP_SHUTDOWN) != ISOTP_IDLE)
+ /* best-effort: wait for a running pdu to finish, but don't block on
+ * it forever - give up after the first signal
+ */
+ while (so->tx.state != ISOTP_IDLE &&
+ wait_event_interruptible(so->wait, so->tx.state == ISOTP_IDLE) == 0)
;
- /* force state machines to be idle also when a signal occurred */
+ /* claim the socket under so->rx_lock like sendmsg() does, so its
+ * claim can't race the forced ISOTP_SHUTDOWN below; force it
+ * unconditionally, even when a signal cut the wait above short
+ */
+ spin_lock_bh(&so->rx_lock);
so->tx.state = ISOTP_SHUTDOWN;
+ spin_unlock_bh(&so->rx_lock);
so->rx.state = ISOTP_IDLE;
spin_lock(&isotp_notifier_lock);
list_del(&so->notifier);
spin_unlock(&isotp_notifier_lock);
+ rtnl_lock();
lock_sock(sk);
- /* remove current filters & unregister */
- if (so->bound) {
- if (so->ifindex) {
- struct net_device *dev;
-
- dev = dev_get_by_index(net, so->ifindex);
- if (dev) {
- if (isotp_register_rxid(so))
- can_rx_unregister(net, dev, so->rxid,
- SINGLE_MASK(so->rxid),
- isotp_rcv, sk);
-
- can_rx_unregister(net, dev, so->txid,
- SINGLE_MASK(so->txid),
- isotp_rcv_echo, sk);
- dev_put(dev);
- synchronize_rcu();
- }
- }
+ /* remove current filters & unregister
+ * tracked reference so->dev is taken at bind() time with rtnl_lock
+ */
+ if (so->bound && so->dev) {
+ if (isotp_register_rxid(so))
+ can_rx_unregister(net, so->dev, so->rxid,
+ SINGLE_MASK(so->rxid),
+ isotp_rcv, sk);
+
+ can_rx_unregister(net, so->dev, so->txid,
+ SINGLE_MASK(so->txid),
+ isotp_rcv_echo, sk);
+ netdev_put(so->dev, &so->dev_tracker);
}
+ so->ifindex = 0;
+ so->bound = 0;
+ so->dev = NULL;
+
+ rtnl_unlock();
+
+ /* Always wait for a grace period before touching the timers below.
+ * A concurrent NETDEV_UNREGISTER may have already unregistered our
+ * filters and cleared so->bound in isotp_notify() without waiting
+ * for in-flight isotp_rcv() callers to finish, so this call must not
+ * be skipped just because so->bound is already 0 here.
+ */
+ synchronize_rcu();
+
hrtimer_cancel(&so->txfrtimer);
hrtimer_cancel(&so->txtimer);
+ hrtimer_cancel(&so->echotimer);
hrtimer_cancel(&so->rxtimer);
- so->ifindex = 0;
- so->bound = 0;
-
sock_orphan(sk);
sock->sk = NULL;
if (!addr->can_ifindex)
return -ENODEV;
+ rtnl_lock();
lock_sock(sk);
if (so->bound) {
goto out;
}
+ /* A transmission or reception that outlived a previous binding
+ * (unbound by NETDEV_UNREGISTER) may still be draining; the FC/echo
+ * and RX watchdog timers bound how long this takes. Checked together
+ * with so->bound in the same lock_sock() section above, so there is
+ * no window in which a concurrent isotp_notify() could be missed.
+ */
+ if (so->tx.state != ISOTP_IDLE || so->rx.state != ISOTP_IDLE) {
+ err = -EAGAIN;
+ goto out;
+ }
+
/* ensure different CAN IDs when the rx_id is to be registered */
if (isotp_register_rxid(so) && rx_id == tx_id) {
err = -EADDRNOTAVAIL;
goto out;
}
if (dev->type != ARPHRD_CAN) {
- dev_put(dev);
err = -ENODEV;
- goto out;
+ goto out_put_dev;
}
if (READ_ONCE(dev->mtu) < so->ll.mtu) {
- dev_put(dev);
err = -EINVAL;
- goto out;
+ goto out_put_dev;
}
if (!(dev->flags & IFF_UP))
notify_enetdown = 1;
can_rx_register(net, dev, tx_id, SINGLE_MASK(tx_id),
isotp_rcv_echo, sk, "isotpe", sk);
- dev_put(dev);
-
/* switch to new settings */
so->ifindex = ifindex;
so->rxid = rx_id;
so->txid = tx_id;
so->bound = 1;
+ /* bind() ok -> hold a reference for so->dev so that isotp_release()
+ * can safely reach the device later, even if a concurrent
+ * NETDEV_UNREGISTER has already unlisted it by ifindex.
+ */
+ so->dev = dev;
+ netdev_hold(so->dev, &so->dev_tracker, GFP_KERNEL);
+
+out_put_dev:
+ /* remove potential reference from dev_get_by_index() */
+ dev_put(dev);
out:
release_sock(sk);
+ rtnl_unlock();
if (notify_enetdown) {
sk->sk_err = ENETDOWN;
if (!net_eq(dev_net(dev), sock_net(sk)))
return;
- if (so->ifindex != dev->ifindex)
+ if (so->dev != dev)
return;
switch (msg) {
can_rx_unregister(dev_net(dev), dev, so->txid,
SINGLE_MASK(so->txid),
isotp_rcv_echo, sk);
+ netdev_put(so->dev, &so->dev_tracker);
}
so->ifindex = 0;
so->bound = 0;
+ so->dev = NULL;
release_sock(sk);
sk->sk_err = ENODEV;
so->ifindex = 0;
so->bound = 0;
+ so->dev = NULL;
so->opt.flags = CAN_ISOTP_DEFAULT_FLAGS;
so->opt.ext_address = CAN_ISOTP_DEFAULT_EXT_ADDRESS;
so->rx.buflen = ARRAY_SIZE(so->rx.sbuf);
so->tx.buflen = ARRAY_SIZE(so->tx.sbuf);
- hrtimer_setup(&so->rxtimer, isotp_rx_timer_handler, CLOCK_MONOTONIC, HRTIMER_MODE_REL_SOFT);
- hrtimer_setup(&so->txtimer, isotp_tx_timer_handler, CLOCK_MONOTONIC, HRTIMER_MODE_REL_SOFT);
- hrtimer_setup(&so->txfrtimer, isotp_txfr_timer_handler, CLOCK_MONOTONIC,
- HRTIMER_MODE_REL_SOFT);
+ hrtimer_setup(&so->rxtimer, isotp_rx_timer_handler,
+ CLOCK_MONOTONIC, HRTIMER_MODE_REL_SOFT);
+ hrtimer_setup(&so->txtimer, isotp_tx_timer_handler,
+ CLOCK_MONOTONIC, HRTIMER_MODE_REL_SOFT);
+ hrtimer_setup(&so->echotimer, isotp_echo_timer_handler,
+ CLOCK_MONOTONIC, HRTIMER_MODE_REL_SOFT);
+ hrtimer_setup(&so->txfrtimer, isotp_txfr_timer_handler,
+ CLOCK_MONOTONIC, HRTIMER_MODE_REL_SOFT);
init_waitqueue_head(&so->wait);
spin_lock_init(&so->rx_lock);
}
}
+static bool j1939_address_is_local(struct j1939_priv *priv, u8 addr)
+{
+ bool local = false;
+
+ read_lock_bh(&priv->lock);
+ if (j1939_address_is_unicast(addr) && priv->ents[addr].nusers)
+ local = true;
+ read_unlock_bh(&priv->lock);
+
+ return local;
+}
+
void j1939_session_skb_queue(struct j1939_session *session,
struct sk_buff *skb)
{
j1939_ac_fixup(priv, skb);
- if (j1939_address_is_unicast(skcb->addr.da) &&
- priv->ents[skcb->addr.da].nusers)
+ if (j1939_address_is_local(priv, skcb->addr.da))
skcb->flags |= J1939_ECU_LOCAL_DST;
skcb->flags |= J1939_ECU_LOCAL_SRC;
return ERR_PTR(ret);
/* fix DST flags, it may be used there soon */
- if (j1939_address_is_unicast(skcb->addr.da) &&
- priv->ents[skcb->addr.da].nusers)
+ if (j1939_address_is_local(priv, skcb->addr.da))
skcb->flags |= J1939_ECU_LOCAL_DST;
/* src is always local, I'm sending ... */
return RAW_MIN_NAMELEN;
}
-static int raw_setsockopt(struct socket *sock, int level, int optname,
- sockptr_t optval, unsigned int optlen)
+static int raw_setsockopt_locked(struct socket *sock, int optname,
+ sockptr_t optval, unsigned int optlen)
{
struct sock *sk = sock->sk;
struct raw_sock *ro = raw_sk(sk);
int flag;
int err = 0;
- if (level != SOL_CAN_RAW)
- return -EINVAL;
-
switch (optname) {
case CAN_RAW_FILTER:
if (optlen % sizeof(struct can_filter) != 0)
return -EFAULT;
}
- rtnl_lock();
- lock_sock(sk);
-
dev = ro->dev;
- if (ro->bound && dev) {
- if (dev->reg_state != NETREG_REGISTERED) {
- if (count > 1)
- kfree(filter);
- err = -ENODEV;
- goto out_fil;
- }
+ if (ro->bound && dev && dev->reg_state != NETREG_REGISTERED) {
+ if (count > 1)
+ kfree(filter);
+ return -ENODEV;
}
if (ro->bound) {
if (err) {
if (count > 1)
kfree(filter);
- goto out_fil;
+ return err;
}
/* remove old filter registrations */
}
ro->filter = filter;
ro->count = count;
-
- out_fil:
- release_sock(sk);
- rtnl_unlock();
-
break;
case CAN_RAW_ERR_FILTER:
err_mask &= CAN_ERR_MASK;
- rtnl_lock();
- lock_sock(sk);
-
dev = ro->dev;
- if (ro->bound && dev) {
- if (dev->reg_state != NETREG_REGISTERED) {
- err = -ENODEV;
- goto out_err;
- }
- }
+ if (ro->bound && dev && dev->reg_state != NETREG_REGISTERED)
+ return -ENODEV;
/* remove current error mask */
if (ro->bound) {
err_mask);
if (err)
- goto out_err;
+ return err;
/* remove old err_mask registration */
raw_disable_errfilter(sock_net(sk), dev, sk,
/* link new err_mask to the socket */
ro->err_mask = err_mask;
-
- out_err:
- release_sock(sk);
- rtnl_unlock();
-
break;
case CAN_RAW_LOOPBACK:
return err;
}
+static int raw_setsockopt(struct socket *sock, int level, int optname,
+ sockptr_t optval, unsigned int optlen)
+{
+ struct sock *sk = sock->sk;
+ int err;
+
+ if (level != SOL_CAN_RAW)
+ return -EINVAL;
+
+ rtnl_lock();
+ lock_sock(sk);
+
+ err = raw_setsockopt_locked(sock, optname, optval, optlen);
+
+ release_sock(sk);
+ rtnl_unlock();
+
+ return err;
+}
+
static int raw_getsockopt(struct socket *sock, int level, int optname,
sockopt_t *opt)
{
struct bpf_local_storage_elem *selem;
int ret = 0;
- RCU_INIT_POINTER(newsk->sk_bpf_storage, NULL);
-
rcu_read_lock_dont_migrate();
sk_storage = rcu_dereference(sk->sk_bpf_storage);
return NULL;
}
+/* Returns the skb on success, NULL if dropped, or ERR_PTR(-EINPROGRESS)
+ * if stolen by async xfrm crypto (delivered via xfrm_dev_resume()).
+ */
static struct sk_buff *validate_xmit_skb(struct sk_buff *skb, struct net_device *dev, bool *again)
{
netdev_features_t features;
skb->prev = skb;
skb = validate_xmit_skb(skb, dev, again);
- if (!skb)
+ if (IS_ERR_OR_NULL(skb))
continue;
if (!head)
goto recursion_alert;
skb = validate_xmit_skb(skb, dev, &again);
- if (!skb)
+ if (IS_ERR_OR_NULL(skb)) {
+ if (PTR_ERR(skb) == -EINPROGRESS)
+ rc = NET_XMIT_SUCCESS;
goto out;
+ }
HARD_TX_LOCK(dev, txq, cpu);
sock_copy(newsk, sk);
newsk->sk_prot_creator = prot;
+#ifdef CONFIG_BPF_SYSCALL
+ RCU_INIT_POINTER(newsk->sk_bpf_storage, NULL);
+#endif
/* SANITY */
if (likely(newsk->sk_net_refcnt)) {
{
if (sk_is_tcp(sk))
return (1 << sk->sk_state) & (TCPF_ESTABLISHED | TCPF_LISTEN);
+ if (sk_is_udp(sk))
+ return sk_hashed(sk);
if (sk_is_stream_unix(sk))
return (1 << READ_ONCE(sk->sk_state)) & TCPF_ESTABLISHED;
if (sk_is_vsock(sk) &&
*mod = true;
}
+/**
+ * ethnl_update_u8_u32() - update u8 value from an NLA_U32 attribute
+ * @dst: value to update
+ * @attr: netlink attribute with new value or null
+ * @mod: pointer to bool for modification tracking
+ *
+ * Some attributes are NLA_U32 on the wire but are stored in a u8. Read the
+ * full 32-bit value from NLA_U32 netlink attribute @attr and narrow it into
+ * the u8 pointed to by @dst; do nothing if @attr is null.
+ * Bool pointed to by @mod is set to true if this function changed the value
+ * of *dst, otherwise it is left as is.
+ */
+static inline void ethnl_update_u8_u32(u8 *dst, const struct nlattr *attr,
+ bool *mod)
+{
+ u32 val;
+
+ if (!attr)
+ return;
+ val = nla_get_u32(attr);
+ DEBUG_NET_WARN_ON_ONCE(val > U8_MAX);
+ if (*dst == val)
+ return;
+
+ *dst = val;
+ *mod = true;
+}
+
/**
* ethnl_update_bool32() - update u32 used as bool from NLA_U8 attribute
* @dst: value to update
const struct nla_policy ethnl_rss_set_policy[ETHTOOL_A_RSS_FLOW_HASH + 1] = {
[ETHTOOL_A_RSS_HEADER] = NLA_POLICY_NESTED(ethnl_header_policy),
[ETHTOOL_A_RSS_CONTEXT] = { .type = NLA_U32, },
- [ETHTOOL_A_RSS_HFUNC] = NLA_POLICY_MIN(NLA_U32, 1),
+ [ETHTOOL_A_RSS_HFUNC] = NLA_POLICY_RANGE(NLA_U32, 1, U8_MAX),
[ETHTOOL_A_RSS_INDIR] = { .type = NLA_BINARY, },
[ETHTOOL_A_RSS_HKEY] = NLA_POLICY_MIN(NLA_BINARY, 1),
[ETHTOOL_A_RSS_INPUT_XFRM] =
indir_mod = !!tb[ETHTOOL_A_RSS_INDIR];
rxfh.hfunc = data.hfunc;
- ethnl_update_u8(&rxfh.hfunc, tb[ETHTOOL_A_RSS_HFUNC], &mod);
+ ethnl_update_u8_u32(&rxfh.hfunc, tb[ETHTOOL_A_RSS_HFUNC], &mod);
if (rxfh.hfunc == data.hfunc)
rxfh.hfunc = ETH_RSS_HASH_NO_CHANGE;
goto exit_free_indir;
rxfh.input_xfrm = data.input_xfrm;
- ethnl_update_u8(&rxfh.input_xfrm, tb[ETHTOOL_A_RSS_INPUT_XFRM], &mod);
+ ethnl_update_u8_u32(&rxfh.input_xfrm, tb[ETHTOOL_A_RSS_INPUT_XFRM],
+ &mod);
xfrm_sym = rxfh.input_xfrm || data.input_xfrm;
if (rxfh.input_xfrm == data.input_xfrm)
rxfh.input_xfrm = RXH_XFRM_NO_CHANGE;
const struct nla_policy ethnl_rss_create_policy[ETHTOOL_A_RSS_INPUT_XFRM + 1] = {
[ETHTOOL_A_RSS_HEADER] = NLA_POLICY_NESTED(ethnl_header_policy),
[ETHTOOL_A_RSS_CONTEXT] = NLA_POLICY_MIN(NLA_U32, 1),
- [ETHTOOL_A_RSS_HFUNC] = NLA_POLICY_MIN(NLA_U32, 1),
+ [ETHTOOL_A_RSS_HFUNC] = NLA_POLICY_RANGE(NLA_U32, 1, U8_MAX),
[ETHTOOL_A_RSS_INDIR] = NLA_POLICY_MIN(NLA_BINARY, 1),
[ETHTOOL_A_RSS_HKEY] = NLA_POLICY_MIN(NLA_BINARY, 1),
[ETHTOOL_A_RSS_INPUT_XFRM] =
goto exit_clean_data;
indir_user_size = ret;
- ethnl_update_u8(&rxfh.hfunc, tb[ETHTOOL_A_RSS_HFUNC], &mod);
+ ethnl_update_u8_u32(&rxfh.hfunc, tb[ETHTOOL_A_RSS_HFUNC], &mod);
ret = rss_set_prep_hkey(dev, info, &data, &rxfh, &mod);
if (ret)
goto exit_free_indir;
rxfh.input_xfrm = RXH_XFRM_NO_CHANGE;
- ethnl_update_u8(&rxfh.input_xfrm, tb[ETHTOOL_A_RSS_INPUT_XFRM], &mod);
+ ethnl_update_u8_u32(&rxfh.input_xfrm, tb[ETHTOOL_A_RSS_INPUT_XFRM],
+ &mod);
ctx = ethtool_rxfh_ctx_alloc(ops, data.indir_size, data.hkey_size);
if (!ctx) {
out_remove_new_fa:
fib_remove_alias(t, tp, l, new_fa);
out_free_new_fa:
- kmem_cache_free(fn_alias_kmem, new_fa);
+ alias_free_mem_rcu(new_fa);
out:
fib_release_info(fi);
err:
static void igmp_stop_timer(struct ip_mc_list *im)
{
+ bool put = false;
+
spin_lock_bh(&im->lock);
if (timer_delete(&im->timer))
- refcount_dec(&im->refcnt);
+ put = true;
WRITE_ONCE(im->tm_running, 0);
WRITE_ONCE(im->reporter, 0);
im->unsolicit_count = 0;
spin_unlock_bh(&im->lock);
+
+ if (put)
+ ip_ma_put(im);
}
/* It must be called with locked im->lock */
return;
in_dev->mr_gq_running = 1;
- if (!mod_timer(&in_dev->mr_gq_timer, exp))
- in_dev_hold(in_dev);
+ if (in_dev_hold_safe(in_dev)) {
+ if (mod_timer(&in_dev->mr_gq_timer, exp))
+ in_dev_put(in_dev);
+ }
}
static void igmp_ifc_start_timer(struct in_device *in_dev, int delay)
{
- int tv = get_random_u32_below(delay);
+ if (in_dev_hold_safe(in_dev)) {
+ int tv = get_random_u32_below(delay);
- if (!mod_timer(&in_dev->mr_ifc_timer, jiffies+tv+2))
- in_dev_hold(in_dev);
+ if (mod_timer(&in_dev->mr_ifc_timer, jiffies + tv + 2))
+ in_dev_put(in_dev);
+ }
}
static void igmp_mod_timer(struct ip_mc_list *im, int max_delay)
{
+ bool put = false;
+
spin_lock_bh(&im->lock);
im->unsolicit_count = 0;
if (timer_delete(&im->timer)) {
spin_unlock_bh(&im->lock);
return;
}
- refcount_dec(&im->refcnt);
+ put = true;
}
igmp_start_timer(im, max_delay);
spin_unlock_bh(&im->lock);
+
+ if (put)
+ ip_ma_put(im);
}
#endif
while ((i = rtnl_dereference(in_dev->mc_list)) != NULL) {
+ ip_mc_hash_remove(in_dev, i);
in_dev->mc_list = i->next_rcu;
WRITE_ONCE(in_dev->mc_count, in_dev->mc_count - 1);
ip_mc_clear_src(i);
static int tcp_bpf_ioctl(struct sock *sk, int cmd, int *karg)
{
+ struct sk_psock *psock;
bool slow;
if (cmd != SIOCINQ)
return -EINVAL;
slow = lock_sock_fast(sk);
- *karg = sk_psock_msg_inq(sk);
+ psock = sk_psock_get(sk);
+ if (unlikely(!psock)) {
+ unlock_sock_fast(sk, slow);
+ return tcp_ioctl(sk, cmd, karg);
+ }
+ *karg = sk_psock_get_msg_len_nolock(psock);
+ /* Without a verdict program, ingress data is never diverted to
+ * ingress_msg: it stays in sk_receive_queue and is read through
+ * the fallback to tcp_recvmsg(), so account for it like
+ * tcp_ioctl() does.
+ */
+ if (!READ_ONCE(psock->progs.stream_verdict) &&
+ !READ_ONCE(psock->progs.skb_verdict))
+ *karg += tcp_inq(sk);
+ sk_psock_put(sk, psock);
unlock_sock_fast(sk, slow);
return 0;
}
drop_reason = psp_twsk_rx_policy_check(inet_twsk(sk), skb);
- if (drop_reason)
- break;
+ if (drop_reason) {
+ inet_twsk_put(inet_twsk(sk));
+ goto discard_it;
+ }
}
/* to ACK */
fallthrough;
mc_assert_locked(idev);
idev->mc_gq_running = 1;
- if (!mod_delayed_work(mld_wq, &idev->mc_gq_work, tv + 2))
- in6_dev_hold(idev);
+ if (in6_dev_hold_safe(idev)) {
+ if (mod_delayed_work(mld_wq, &idev->mc_gq_work, tv + 2))
+ in6_dev_put(idev);
+ }
}
static void mld_gq_stop_work(struct inet6_dev *idev)
mc_assert_locked(idev);
- if (!mod_delayed_work(mld_wq, &idev->mc_ifc_work, tv + 2))
- in6_dev_hold(idev);
+ if (in6_dev_hold_safe(idev)) {
+ if (mod_delayed_work(mld_wq, &idev->mc_ifc_work, tv + 2))
+ in6_dev_put(idev);
+ }
}
static void mld_ifc_stop_work(struct inet6_dev *idev)
mc_assert_locked(idev);
- if (!mod_delayed_work(mld_wq, &idev->mc_dad_work, tv + 2))
- in6_dev_hold(idev);
+ if (in6_dev_hold_safe(idev)) {
+ if (mod_delayed_work(mld_wq, &idev->mc_dad_work, tv + 2))
+ in6_dev_put(idev);
+ }
}
static void mld_dad_stop_work(struct inet6_dev *idev)
void igmp6_event_query(struct sk_buff *skb)
{
struct inet6_dev *idev = __in6_dev_get(skb->dev);
+ bool put = false;
if (!idev || idev->dead)
goto out;
spin_lock_bh(&idev->mc_query_lock);
- if (skb_queue_len(&idev->mc_query_queue) < MLD_MAX_SKBS) {
+ if (skb_queue_len(&idev->mc_query_queue) < MLD_MAX_SKBS &&
+ in6_dev_hold_safe(idev)) {
__skb_queue_tail(&idev->mc_query_queue, skb);
- if (!mod_delayed_work(mld_wq, &idev->mc_query_work, 0))
- in6_dev_hold(idev);
+ if (mod_delayed_work(mld_wq, &idev->mc_query_work, 0))
+ put = true;
skb = NULL;
}
spin_unlock_bh(&idev->mc_query_lock);
+
+ if (put)
+ in6_dev_put(idev);
out:
kfree_skb(skb);
}
void igmp6_event_report(struct sk_buff *skb)
{
struct inet6_dev *idev = __in6_dev_get(skb->dev);
+ bool put = false;
if (!idev || idev->dead)
goto out;
spin_lock_bh(&idev->mc_report_lock);
- if (skb_queue_len(&idev->mc_report_queue) < MLD_MAX_SKBS) {
+ if (skb_queue_len(&idev->mc_report_queue) < MLD_MAX_SKBS &&
+ in6_dev_hold_safe(idev)) {
__skb_queue_tail(&idev->mc_report_queue, skb);
- if (!mod_delayed_work(mld_wq, &idev->mc_report_work, 0))
- in6_dev_hold(idev);
+ if (mod_delayed_work(mld_wq, &idev->mc_report_work, 0))
+ put = true;
skb = NULL;
}
spin_unlock_bh(&idev->mc_report_lock);
+
+ if (put)
+ in6_dev_put(idev);
out:
kfree_skb(skb);
}
ktime_t tstamp = skb->tstamp;
struct ip6_frag_state state;
u8 *prevhdr, nexthdr = 0;
- unsigned int mtu, hlen;
+ unsigned int mtu, hlen, nexthdr_offset;
int hroom, err = 0;
__be32 frag_id;
goto blackhole;
hlen = err;
nexthdr = *prevhdr;
+ nexthdr_offset = prevhdr - skb_network_header(skb);
mtu = skb->dev->mtu;
if (frag_max_size > mtu ||
(err = skb_checksum_help(skb)))
goto blackhole;
+ prevhdr = skb_network_header(skb) + nexthdr_offset;
hroom = LL_RESERVED_SPACE(skb->dev);
if (skb_has_frag_list(skb)) {
unsigned int first_len = skb_pagelen(skb);
}
hdrlen = ipv6_authlen(ah);
+ if (skb->len - ptr < hdrlen) {
+ /* Packet smaller than its length field */
+ par->hotdrop = true;
+ return false;
+ }
pr_debug("IPv6 AH LEN %u %u ", hdrlen, ah->hdrlen);
pr_debug("RES %04X ", ah->reserved);
hdrlen = ipv6_optlen(oh);
if (skb->len - ptr < hdrlen) {
/* Packet smaller than it's length field */
+ par->hotdrop = true;
return false;
}
hdrlen = ipv6_optlen(rh);
if (skb->len - ptr < hdrlen) {
- /* Pcket smaller than its length field */
+ /* Packet smaller than its length field */
+ par->hotdrop = true;
return false;
}
skb_network_header(skb)[fq->nhoffset] = skb_transport_header(skb)[0];
memmove(skb->head + sizeof(struct frag_hdr), skb->head,
(skb->data - skb->head) - sizeof(struct frag_hdr));
- skb->mac_header += sizeof(struct frag_hdr);
+ if (skb_mac_header_was_set(skb))
+ skb->mac_header += sizeof(struct frag_hdr);
skb->network_header += sizeof(struct frag_hdr);
skb_reset_transport_header(skb);
return -1;
}
if (skb_copy_bits(skb, start, &hdr, sizeof(hdr)))
- BUG();
+ return -1;
if (nexthdr == NEXTHDR_AUTH)
hdrlen = ipv6_authlen(&hdr);
else
}
drop_reason = psp_twsk_rx_policy_check(inet_twsk(sk), skb);
- if (drop_reason)
- break;
+ if (drop_reason) {
+ inet_twsk_put(inet_twsk(sk));
+ goto discard_it;
+ }
}
/* to ACK */
fallthrough;
xdst->u.rt6.rt6i_idev = in6_dev_get(dev);
if (!xdst->u.rt6.rt6i_idev) {
netdev_put(dev, &xdst->u.dst.dev_tracker);
+ xdst->u.dst.dev = NULL;
return -ENODEV;
}
}
}
}
+ if (sk)
+ sock_hold(sk);
read_unlock(&iucv_sk_list.lock);
if (!iucv)
sk = NULL;
kfree_skb(skb);
}
+ if (sk)
+ sock_put(sk);
return err;
}
/* assign new connection to its SAP */
llc_sap_add_socket(sap, sk);
sock_reset_flag(sk, SOCK_ZAPPED);
+ llc_sap_put(sap);
rc = 0;
out:
dev_put(dev);
newllc->dev = dev;
dev_hold(dev);
llc_sap_add_socket(llc->sap, newsk);
- llc_sap_hold(llc->sap);
out:
return newsk;
}
fd->max_interval = params->max_interval;
old = sdata_dereference(link->u.ap.fils_discovery, sdata);
- if (old)
- kfree_rcu(old, rcu_head);
-
if (params->tmpl && params->tmpl_len) {
new = kzalloc(sizeof(*new) + params->tmpl_len, GFP_KERNEL);
if (!new)
RCU_INIT_POINTER(link->u.ap.fils_discovery, NULL);
}
+ if (old)
+ kfree_rcu(old, rcu_head);
+
*changed |= BSS_CHANGED_FILS_DISCOVERY;
return 0;
}
link_conf->unsol_bcast_probe_resp_interval = params->interval;
old = sdata_dereference(link->u.ap.unsol_bcast_probe_resp, sdata);
- if (old)
- kfree_rcu(old, rcu_head);
if (params->tmpl && params->tmpl_len) {
new = kzalloc(sizeof(*new) + params->tmpl_len, GFP_KERNEL);
RCU_INIT_POINTER(link->u.ap.unsol_bcast_probe_resp, NULL);
}
+ if (old)
+ kfree_rcu(old, rcu_head);
+
*changed |= BSS_CHANGED_UNSOL_BCAST_PROBE_RESP;
return 0;
}
ifibss->state = IEEE80211_IBSS_MLME_SEARCH;
- sta_info_flush(sdata, -1);
+ netif_carrier_off(sdata->dev);
+ if (!sta_info_flush(sdata, -1))
+ synchronize_net();
spin_lock_bh(&ifibss->incomplete_lock);
while (!list_empty(&ifibss->incomplete_stations)) {
}
spin_unlock_bh(&ifibss->incomplete_lock);
- netif_carrier_off(sdata->dev);
-
sdata->vif.cfg.ibss_joined = false;
sdata->vif.cfg.ibss_creator = false;
sdata->vif.bss_conf.enable_beacon = false;
u.ibss.csa_connection_drop_work);
ieee80211_ibss_disconnect(sdata);
- synchronize_rcu();
skb_queue_purge(&sdata->skb_queue);
/* trigger a scan to find another IBSS network to join */
u32 changed = IEEE80211_RC_SUPP_RATES_CHANGED;
u8 rx_nss = sta->sta.deflink.rx_nss;
- /* Force rx_nss recalculation */
- sta->sta.deflink.rx_nss = 0;
+ ieee80211_sta_init_nss_bw_capa(&sta->deflink,
+ &sdata->deflink.conf->chanreq.oper);
rate_control_rate_init(&sta->deflink);
if (sta->sta.deflink.rx_nss != rx_nss)
changed |= IEEE80211_RC_NSS_CHANGED;
memset(&ifibss->ht_capa, 0, sizeof(ifibss->ht_capa));
memset(&ifibss->ht_capa_mask, 0, sizeof(ifibss->ht_capa_mask));
- synchronize_rcu();
-
skb_queue_purge(&sdata->skb_queue);
timer_delete_sync(&sdata->u.ibss.timer);
WARN_ON(!list_empty(&sdata->u.ap.vlans));
} else if (sdata->vif.type == NL80211_IFTYPE_AP_VLAN) {
/* remove all packets in parent bc_buf pointing to this dev */
+ __skb_queue_head_init(&freeq);
ps = &sdata->bss->ps;
spin_lock_irqsave(&ps->bc_buf.lock, flags);
if (skb->dev == sdata->dev) {
__skb_unlink(skb, &ps->bc_buf);
local->total_ps_buffered--;
- ieee80211_free_txskb(&local->hw, skb);
+ __skb_queue_tail(&freeq, skb);
}
}
spin_unlock_irqrestore(&ps->bc_buf.lock, flags);
+
+ skb_queue_walk_safe(&freeq, skb, tmp) {
+ __skb_unlink(skb, &freeq);
+ ieee80211_free_txskb(&local->hw, skb);
+ }
}
if (going_down)
sband = kmemdup(sband, sizeof(*sband), GFP_KERNEL);
if (!sband) {
result = -ENOMEM;
- goto fail_rate;
+ goto fail_band;
}
wiphy_dbg(hw->wiphy, "copying sband (band %d) due to VHT EXT NSS BW flag\n",
#endif
wiphy_unregister(local->hw.wiphy);
fail_wiphy_register:
+ fail_band:
rtnl_lock();
rate_control_deinitialize(local);
ieee80211_remove_interfaces(local);
struct ieee80211_mgmt *mgmt, size_t len)
{
struct ieee80211_if_managed *ifmgd = &sdata->u.mgd;
- u16 reason_code = le16_to_cpu(mgmt->u.deauth.reason_code);
+ u16 reason_code;
lockdep_assert_wiphy(sdata->local->hw.wiphy);
- if (len < 24 + 2)
+ if (len < offsetofend(struct ieee80211_mgmt, u.deauth.reason_code))
return;
+ reason_code = le16_to_cpu(mgmt->u.deauth.reason_code);
+
if (!ether_addr_equal(mgmt->bssid, mgmt->sa)) {
ieee80211_tdls_handle_disconnect(sdata, mgmt->sa, reason_code);
return;
{
struct ieee80211_if_managed *ifmgd = &sdata->u.mgd;
struct ieee80211_mgd_assoc_data *assoc_data = ifmgd->assoc_data;
- u16 capab_info, status_code, aid;
+ u16 capab_info, status_code, aid = 0;
struct ieee80211_elems_parse_params parse_params = {
.bss = NULL,
.link_id = -1,
if (elems->aid_resp)
aid = le16_to_cpu(elems->aid_resp->aid);
- else
+ else if (!assoc_data->s1g)
aid = le16_to_cpu(mgmt->u.assoc_resp.aid);
+ else if (status_code == WLAN_STATUS_SUCCESS)
+ goto abandon_assoc;
/*
* The 5 MSB of the AID field are reserved for a non-S1G STA. For
{
struct ieee80211_nan_channel *sched_idx_to_chan[IEEE80211_NAN_MAX_CHANNELS] = {};
struct ieee80211_nan_sched_cfg *sched_cfg = &sdata->vif.cfg.nan_sched;
- struct ieee80211_nan_sched_cfg backup_sched;
+ struct ieee80211_nan_sched_cfg *backup_sched __free(kfree) = kmalloc_obj(*backup_sched);
int ret;
+ if (!backup_sched)
+ return -ENOMEM;
+
if (sched->n_channels > IEEE80211_NAN_MAX_CHANNELS)
return -EOPNOTSUPP;
bitmap_zero(sdata->u.nan.removed_channels, IEEE80211_NAN_MAX_CHANNELS);
- memcpy(backup_sched.schedule, sched_cfg->schedule,
- sizeof(backup_sched.schedule));
- memcpy(backup_sched.channels, sched_cfg->channels,
- sizeof(backup_sched.channels));
- memcpy(backup_sched.avail_blob, sched_cfg->avail_blob,
- sizeof(backup_sched.avail_blob));
- backup_sched.avail_blob_len = sched_cfg->avail_blob_len;
+ memcpy(backup_sched->schedule, sched_cfg->schedule,
+ sizeof(backup_sched->schedule));
+ memcpy(backup_sched->channels, sched_cfg->channels,
+ sizeof(backup_sched->channels));
+ memcpy(backup_sched->avail_blob, sched_cfg->avail_blob,
+ sizeof(backup_sched->avail_blob));
+ backup_sched->avail_blob_len = sched_cfg->avail_blob_len;
memcpy(sched_cfg->avail_blob, sched->nan_avail_blob,
sched->nan_avail_blob_len);
if (!chan_def->chan)
continue;
- if (!cfg80211_chandef_identical(&backup_sched.channels[i].chanreq.oper,
+ if (!cfg80211_chandef_identical(&backup_sched->channels[i].chanreq.oper,
chan_def))
ieee80211_nan_remove_channel(sdata,
&sched_cfg->channels[i]);
}
/* Re-add all backed up channels */
- for (int i = 0; i < ARRAY_SIZE(backup_sched.channels); i++) {
+ for (int i = 0; i < ARRAY_SIZE(backup_sched->channels); i++) {
struct ieee80211_nan_channel *chan = &sched_cfg->channels[i];
- *chan = backup_sched.channels[i];
+ *chan = backup_sched->channels[i];
/*
* For deferred update, no channels were removed and the channel
}
}
- memcpy(sched_cfg->schedule, backup_sched.schedule,
- sizeof(backup_sched.schedule));
- memcpy(sched_cfg->avail_blob, backup_sched.avail_blob,
- sizeof(backup_sched.avail_blob));
- sched_cfg->avail_blob_len = backup_sched.avail_blob_len;
+ memcpy(sched_cfg->schedule, backup_sched->schedule,
+ sizeof(backup_sched->schedule));
+ memcpy(sched_cfg->avail_blob, backup_sched->avail_blob,
+ sizeof(backup_sched->avail_blob));
+ sched_cfg->avail_blob_len = backup_sched->avail_blob_len;
sched_cfg->deferred = false;
bitmap_zero(sdata->u.nan.removed_channels, IEEE80211_NAN_MAX_CHANNELS);
if (status->flag & RX_FLAG_DUP_VALIDATED)
return RX_CONTINUE;
+ if (ieee80211_is_ext(hdr->frame_control))
+ return RX_CONTINUE;
+
/*
* Drop duplicate 802.11 retransmissions
* (IEEE 802.11-2012: 9.3.2.10 "Duplicate detection and recovery")
struct ieee80211_hdr *hdr = (void *)skb->data;
struct ieee80211_rx_status *status = IEEE80211_SKB_RXCB(skb);
u8 *bssid = ieee80211_get_bssid(hdr, skb->len, sdata->vif.type);
- bool multicast = is_multicast_ether_addr(hdr->addr1) ||
- ieee80211_is_s1g_beacon(hdr->frame_control);
+ bool multicast;
static const u8 nan_network_id[ETH_ALEN] __aligned(2) = {
0x51, 0x6F, 0x9A, 0x01, 0x00, 0x00
};
+ if (ieee80211_is_s1g_beacon(hdr->frame_control))
+ return sdata->vif.type == NL80211_IFTYPE_STATION && bssid;
+
+ multicast = is_multicast_ether_addr(hdr->addr1);
+
switch (sdata->vif.type) {
case NL80211_IFTYPE_STATION:
if (!bssid && !sdata->u.mgd.use_4addr)
hdr = (struct ieee80211_hdr *)rx->skb->data;
}
+ if (ieee80211_is_s1g_beacon(hdr->frame_control)) {
+ ieee80211_invoke_rx_handlers(rx);
+ return true;
+ }
+
/* Store a copy of the pre-translated link addresses for SW crypto */
if (unlikely(is_unicast_ether_addr(hdr->addr1) &&
!ieee80211_is_data(hdr->frame_control)))
struct sta_info *sta;
int link_id = -1;
+ if (ieee80211_is_s1g_beacon(hdr->frame_control)) {
+ if (!ieee80211_rx_data_set_sta(rx, NULL, -1))
+ return false;
+
+ return ieee80211_prepare_and_rx_handle(rx, skb, consume);
+ }
+
/*
* Look up link station first, in case there's a
* chance that they might have a link address that
err = -ENOBUFS;
else
err = skb_linearize(skb);
+ } else if (ieee80211_is_s1g_beacon(fc)) {
+ size_t s1g_hdr_len = offsetof(struct ieee80211_ext,
+ u.s1g_beacon.variable) +
+ ieee80211_s1g_optional_len(fc);
+
+ if (skb->len < s1g_hdr_len)
+ err = -ENOBUFS;
+ else
+ err = skb_linearize(skb);
+ } else if (ieee80211_is_ext(fc)) {
+ err = -EINVAL;
} else {
err = !pskb_may_pull(skb, ieee80211_hdrlen(fc));
}
free_percpu(link_sta->pcpu_rx_stats);
}
+static void sta_link_free_rcu(struct rcu_head *head)
+{
+ struct sta_link_alloc *alloc =
+ container_of(head, struct sta_link_alloc, rcu_head);
+
+ sta_info_free_link(&alloc->info);
+ kfree(alloc);
+}
+
static void sta_accumulate_removed_link_stats(struct sta_info *sta, int link_id)
{
struct link_sta_info *link_sta = wiphy_dereference(sta->local->hw.wiphy,
RCU_INIT_POINTER(sta->link[link_id], NULL);
RCU_INIT_POINTER(sta->sta.link[link_id], NULL);
- if (alloc) {
- sta_info_free_link(&alloc->info);
- kfree_rcu(alloc, rcu_head);
- }
+ if (alloc)
+ call_rcu(&alloc->rcu_head, sta_link_free_rcu);
ieee80211_sta_recalc_aggregates(&sta->sta);
}
return info_id;
}
+static void ieee80211_remove_ack_skb(struct ieee80211_local *local, u16 info_id)
+{
+ struct sk_buff *ack_skb;
+ unsigned long flags;
+
+ spin_lock_irqsave(&local->ack_status_lock, flags);
+ ack_skb = idr_remove(&local->ack_status_frames, info_id);
+ spin_unlock_irqrestore(&local->ack_status_lock, flags);
+
+ kfree_skb(ack_skb);
+}
+
/**
* ieee80211_build_hdr - build 802.11 header in the given frame
* @sdata: virtual interface to build the header for
if (ieee80211_skb_resize(sdata, skb, head_need, ENCRYPT_DATA)) {
ieee80211_free_txskb(&local->hw, skb);
skb = NULL;
- return ERR_PTR(-ENOMEM);
+ ret = -ENOMEM;
+ goto free;
}
}
return skb;
free:
+ if (info_id)
+ ieee80211_remove_ack_skb(local, info_id);
kfree_skb(skb);
return ERR_PTR(ret);
}
if (ieee80211_is_s1g_beacon(fc)) {
struct ieee80211_ext *ext = (void *) hdr;
+ if (len < offsetofend(struct ieee80211_ext, u.s1g_beacon.sa))
+ return NULL;
+
return ext->u.s1g_beacon.sa;
}
mutex_lock(&local->iflist_mtx);
list_for_each_entry_safe(sdata, tmp, &local->interfaces, list) {
- list_del(&sdata->list);
+ list_del_rcu(&sdata->list);
unregister_netdevice(sdata->dev);
}
int ifindex;
if (i == RTA_OIF) {
+ if (!tb[i])
+ continue;
+
ifindex = nla_get_u32(tb[i]);
filter->dev = dev_get_by_index_rcu(net, ifindex);
if (!filter->dev)
struct htable_gc {
struct delayed_work dwork;
struct ip_set *set; /* Set the gc belongs to */
+ spinlock_t lock; /* Lock to exclude gc and resize */
u32 region; /* Last gc run position */
};
/* The hash table: the table size stored here in order to make resizing easy */
struct htable {
- atomic_t ref; /* References for resizing */
+ bool resizing; /* Mark ongoing resize */
atomic_t uref; /* References for dumping and gc */
u8 htable_bits; /* size of hash table == 2^htable_bits */
u32 maxelem; /* Maxelem per region */
+ struct list_head ad; /* Resize add|del backlist */
struct ip_set_region *hregion; /* Region locks and ext sizes */
struct hbucket __rcu *bucket[]; /* hashtable buckets */
};
u8 netmask; /* netmask value for subnets to store */
union nf_inet_addr bitmask; /* stores bitmask */
#endif
- struct list_head ad; /* Resize add|del backlist */
- struct mtype_elem next; /* temporary storage for uadd */
#ifdef IP_SET_HASH_WITH_NETS
struct net_prefixes nets[NLEN]; /* book-keeping of prefixes */
#endif
+ /* Because 'next' is IPv4/IPv6 dependent, no elements of this
+ * structure and referred in create() may come after 'next'.
+ */
+ struct mtype_elem next; /* temporary storage for uadd */
};
/* ADD|DEL entries saved during resize */
mtype_destroy(struct ip_set *set)
{
struct htype *h = set->data;
+ struct htable *t = (__force struct htable *)h->table;
struct list_head *l, *lt;
- mtype_ahash_destroy(set, (__force struct htable *)h->table, true);
- list_for_each_safe(l, lt, &h->ad) {
+ list_for_each_safe(l, lt, &t->ad) {
list_del(l);
kfree(l);
}
+ mtype_ahash_destroy(set, t, true);
kfree(h);
set->data = NULL;
set = gc->set;
h = set->data;
- spin_lock_bh(&set->lock);
- t = ipset_dereference_set(h->table, set);
+ rcu_read_lock_bh();
+ t = rcu_dereference_bh(h->table);
atomic_inc(&t->uref);
+ rcu_read_unlock_bh();
numof_locks = ahash_numof_locks(t->htable_bits);
r = gc->region++;
if (r >= numof_locks) {
next_run = (IPSET_GC_PERIOD(set->timeout) * HZ) / numof_locks;
if (next_run < HZ/10)
next_run = HZ/10;
- spin_unlock_bh(&set->lock);
- mtype_gc_do(set, h, t, r);
+ spin_lock_bh(&gc->lock);
+ if (!t->resizing)
+ mtype_gc_do(set, h, t, r);
+ spin_unlock_bh(&gc->lock);
- if (atomic_dec_and_test(&t->uref) && atomic_read(&t->ref)) {
+ if (atomic_dec_and_test(&t->uref) && t->resizing) {
pr_debug("Table destroy after resize by expire: %p\n", t);
mtype_ahash_destroy(set, t, false);
}
}
t->htable_bits = htable_bits;
t->maxelem = h->maxelem / ahash_numof_locks(htable_bits);
+ INIT_LIST_HEAD(&t->ad);
for (i = 0; i < ahash_numof_locks(htable_bits); i++)
spin_lock_init(&t->hregion[i].lock);
/* There can't be another parallel resizing,
- * but dumping, gc, kernel side add/del are possible
+ * but dumping and kernel side add/del are possible
*/
orig = ipset_dereference_bh_nfnl(h->table);
- atomic_set(&orig->ref, 1);
atomic_inc(&orig->uref);
+ spin_lock_bh(&h->gc.lock);
+ orig->resizing = true;
+ spin_unlock_bh(&h->gc.lock);
pr_debug("attempt to resize set %s from %u to %u, t %p\n",
set->name, orig->htable_bits, htable_bits, orig);
for (r = 0; r < ahash_numof_locks(orig->htable_bits); r++) {
* Kernel-side add cannot trigger a resize and userspace actions
* are serialized by the mutex.
*/
- list_for_each_safe(l, lt, &h->ad) {
+ list_for_each_safe(l, lt, &orig->ad) {
x = list_entry(l, struct mtype_resize_ad, list);
if (x->ad == IPSET_ADD) {
mtype_add(set, &x->d, &x->ext, &x->mext, x->flags);
cleanup:
rcu_read_unlock_bh();
- atomic_set(&orig->ref, 0);
+ spin_lock_bh(&h->gc.lock);
+ orig->resizing = false;
+ spin_unlock_bh(&h->gc.lock);
+ /* Make sure parallel readers see that orig->resizing is false
+ * before we decrement uref */
+ synchronize_rcu();
atomic_dec(&orig->uref);
mtype_ahash_destroy(set, t, false);
if (ret == -EAGAIN)
goto retry;
+
+ /* Cleanup the backlog of ADD/DEL elements */
+ spin_lock_bh(&set->lock);
+ list_for_each_safe(l, lt, &orig->ad) {
+ list_del(l);
+ kfree(l);
+ }
+ spin_unlock_bh(&set->lock);
goto out;
hbwarn:
key = HKEY(value, h->initval, t->htable_bits);
r = ahash_region(key);
atomic_inc(&t->uref);
+ rcu_read_unlock_bh();
elements = t->hregion[r].elements;
maxelem = t->maxelem;
if (elements >= maxelem) {
u32 e;
- if (SET_WITH_TIMEOUT(set)) {
- rcu_read_unlock_bh();
+ if (SET_WITH_TIMEOUT(set))
mtype_gc_do(set, h, t, r);
- rcu_read_lock_bh();
- }
maxelem = h->maxelem;
elements = 0;
for (e = 0; e < ahash_numof_locks(t->htable_bits); e++)
if (elements >= maxelem && SET_WITH_FORCEADD(set))
forceadd = true;
}
- rcu_read_unlock_bh();
spin_lock_bh(&t->hregion[r].lock);
n = rcu_dereference_bh(hbucket(t, key));
ret = 0;
resize:
spin_unlock_bh(&t->hregion[r].lock);
- if (atomic_read(&t->ref) && ext->target) {
+ if (t->resizing && ext && ext->target) {
/* Resize is in process and kernel side add, save values */
struct mtype_resize_ad *x;
memcpy(&x->mext, mext, sizeof(struct ip_set_ext));
x->flags = flags;
spin_lock_bh(&set->lock);
- list_add_tail(&x->list, &h->ad);
+ list_add_tail(&x->list, &t->ad);
spin_unlock_bh(&set->lock);
}
goto out;
unlock:
spin_unlock_bh(&t->hregion[r].lock);
out:
- if (atomic_dec_and_test(&t->uref) && atomic_read(&t->ref)) {
+ if (atomic_dec_and_test(&t->uref) && t->resizing) {
pr_debug("Table destroy after resize by add: %p\n", t);
mtype_ahash_destroy(set, t, false);
}
#endif
ip_set_ext_destroy(set, data);
- if (atomic_read(&t->ref) && ext->target) {
+ if (t->resizing && ext && ext->target) {
/* Resize is in process and kernel side del,
* save values
*/
spin_unlock_bh(&t->hregion[r].lock);
if (x) {
spin_lock_bh(&set->lock);
- list_add(&x->list, &h->ad);
+ list_add(&x->list, &t->ad);
spin_unlock_bh(&set->lock);
}
- if (atomic_dec_and_test(&t->uref) && atomic_read(&t->ref)) {
+ if (atomic_dec_and_test(&t->uref) && t->resizing) {
pr_debug("Table destroy after resize by del: %p\n", t);
mtype_ahash_destroy(set, t, false);
}
rcu_read_unlock_bh();
} else if (cb->args[IPSET_CB_PRIVATE]) {
t = (struct htable *)cb->args[IPSET_CB_PRIVATE];
- if (atomic_dec_and_test(&t->uref) && atomic_read(&t->ref)) {
+ if (atomic_dec_and_test(&t->uref) && t->resizing) {
pr_debug("Table destroy after resize "
" by dump: %p\n", t);
mtype_ahash_destroy(set, t, false);
if (tb[IPSET_ATTR_MAXELEM])
maxelem = ip_set_get_h32(tb[IPSET_ATTR_MAXELEM]);
- hsize = sizeof(*h);
+#ifdef IP_SET_PROTO_UNDEF
+ hsize = sizeof(struct htype);
+#else
+ hsize = set->family == NFPROTO_IPV6 ?
+ sizeof(struct IPSET_TOKEN(HTYPE, 6)) :
+ sizeof(struct IPSET_TOKEN(HTYPE, 4));
+#endif
h = kzalloc(hsize, GFP_KERNEL);
if (!h)
return -ENOMEM;
return -ENOMEM;
}
h->gc.set = set;
+ spin_lock_init(&h->gc.lock);
for (i = 0; i < ahash_numof_locks(hbits); i++)
spin_lock_init(&t->hregion[i].lock);
h->maxelem = maxelem;
}
t->htable_bits = hbits;
t->maxelem = h->maxelem / ahash_numof_locks(hbits);
+ INIT_LIST_HEAD(&t->ad);
RCU_INIT_POINTER(h->table, t);
-
- INIT_LIST_HEAD(&h->ad);
set->data = h;
#ifndef IP_SET_PROTO_UNDEF
if (set->family == NFPROTO_IPV4) {
struct ip_vs_iphdr *ipvsh)
{
int diff;
- const unsigned int tcp_offset = ip_hdrlen(skb);
struct tcphdr *th;
__u32 seq;
- if (skb_ensure_writable(skb, tcp_offset + sizeof(*th)))
+ if (skb_ensure_writable(skb, ipvsh->len + sizeof(*th)))
return 0;
- th = (struct tcphdr *)(skb_network_header(skb) + tcp_offset);
+ th = (struct tcphdr *)(skb_network_header(skb) + ipvsh->len);
/*
* Remember seq number in case this pkt gets resized
struct ip_vs_iphdr *ipvsh)
{
int diff;
- const unsigned int tcp_offset = ip_hdrlen(skb);
struct tcphdr *th;
__u32 seq;
- if (skb_ensure_writable(skb, tcp_offset + sizeof(*th)))
+ if (skb_ensure_writable(skb, ipvsh->len + sizeof(*th)))
return 0;
- th = (struct tcphdr *)(skb_network_header(skb) + tcp_offset);
+ th = (struct tcphdr *)(skb_network_header(skb) + ipvsh->len);
/*
* Remember seq number in case this pkt gets resized
cp->app = NULL;
cp->app_data = NULL;
/* reset struct ip_vs_seq */
- cp->in_seq.delta = 0;
- cp->out_seq.delta = 0;
+ memset(&cp->in_seq, 0, sizeof(cp->in_seq));
+ memset(&cp->out_seq, 0, sizeof(cp->out_seq));
if (unlikely(flags & IP_VS_CONN_F_NO_CPORT)) {
int af_id = ip_vs_af_index(cp->af);
static inline void
ip_vs_set_state(struct ip_vs_conn *cp, int direction,
const struct sk_buff *skb,
- struct ip_vs_proto_data *pd)
+ struct ip_vs_proto_data *pd, unsigned int iph_len)
{
if (likely(pd->pp->state_transition))
- pd->pp->state_transition(cp, direction, skb, pd);
+ pd->pp->state_transition(cp, direction, skb, pd, iph_len);
}
static inline int
ip_vs_in_stats(cp, skb);
/* set state */
- ip_vs_set_state(cp, IP_VS_DIR_INPUT, skb, pd);
+ ip_vs_set_state(cp, IP_VS_DIR_INPUT, skb, pd, iph->len);
/* transmit the first SYN packet */
ret = cp->packet_xmit(skb, cp, pd->pp, iph);
snet.in6 = ciph.saddr.in6;
offset = ciph.len;
return handle_response_icmp(AF_INET6, skb, &snet, ciph.protocol, cp,
- pp, offset, sizeof(struct ipv6hdr),
- hooknum);
+ pp, offset, ipvsh->len, hooknum);
}
#endif
after_nat:
ip_vs_out_stats(cp, skb);
- ip_vs_set_state(cp, IP_VS_DIR_OUTPUT, skb, pd);
+ ip_vs_set_state(cp, IP_VS_DIR_OUTPUT, skb, pd, iph->len);
skb->ipvs_property = 1;
if (!(cp->flags & IP_VS_CONN_F_NFCT))
ip_vs_notrack(skb);
bool tunnel, new_cp = false;
union nf_inet_addr *raddr;
char *outer_proto = "IPIP";
+ unsigned int hlen_ipip;
+ int ulen = 0;
*related = 1;
/* Now find the contained IP header */
offset += sizeof(_icmph);
cih = skb_header_pointer(skb, offset, sizeof(_ciph), &_ciph);
- if (cih == NULL)
+ if (!(cih && cih->version == 4 && cih->ihl >= 5))
return NF_ACCEPT; /* The packet looks wrong, ignore */
raddr = (union nf_inet_addr *)&cih->daddr;
+ hlen_ipip = cih->ihl * 4;
/* Special case for errors for IPIP/UDP/GRE tunnel packets */
tunnel = false;
/* Only for known tunnel */
if (!dest || dest->tun_type != IP_VS_CONN_F_TUNNEL_TYPE_IPIP)
return NF_ACCEPT;
- offset += cih->ihl * 4;
+ offset += hlen_ipip;
cih = skb_header_pointer(skb, offset, sizeof(_ciph), &_ciph);
- if (cih == NULL)
+ if (!(cih && cih->version == 4 && cih->ihl >= 5))
return NF_ACCEPT; /* The packet looks wrong, ignore */
tunnel = true;
} else if ((cih->protocol == IPPROTO_UDP || /* Can be UDP encap */
/* Error for our tunnel must arrive at LOCAL_IN */
(skb_rtable(skb)->rt_flags & RTCF_LOCAL)) {
__u8 iproto;
- int ulen;
/* Non-first fragment has no UDP/GRE header */
if (unlikely(cih->frag_off & htons(IP_OFFSET)))
return NF_ACCEPT;
- offset2 = offset + cih->ihl * 4;
+ offset2 = offset + hlen_ipip;
if (cih->protocol == IPPROTO_UDP) {
ulen = ipvs_udp_decap(ipvs, skb, offset2, AF_INET,
raddr, &iproto);
}
if (tunnel) {
+ unsigned int hlen_orig = cih->ihl * 4;
__be32 info = ic->un.gateway;
__u8 type = ic->type;
__u8 code = ic->code;
goto ignore_tunnel;
offset2 -= ihl + sizeof(_icmph);
skb_reset_network_header(skb);
+ /* Ensure the IP header is present in headroom */
+ if (!pskb_may_pull(skb, hlen_ipip))
+ goto ignore_tunnel;
IP_VS_DBG(12, "ICMP for %s %pI4->%pI4: mtu=%u\n",
outer_proto, &ip_hdr(skb)->saddr,
&ip_hdr(skb)->daddr, mtu);
if (dest_dst)
mtu = dst_mtu(dest_dst->dst_cache);
}
- if (mtu > 68 + sizeof(struct iphdr))
- mtu -= sizeof(struct iphdr);
+ if (mtu > 68 + hlen_ipip + ulen)
+ mtu -= hlen_ipip + ulen;
info = htonl(mtu);
}
/* Strip outer IP, ICMP and IPIP/UDP/GRE, go to IP header of
if (pskb_pull(skb, offset2) == NULL)
goto ignore_tunnel;
skb_reset_network_header(skb);
+ /* Ensure the IP header is present in headroom */
+ if (!pskb_may_pull(skb, hlen_orig))
+ goto ignore_tunnel;
IP_VS_DBG(12, "Sending ICMP for %pI4->%pI4: t=%u, c=%u, i=%u\n",
&ip_hdr(skb)->saddr, &ip_hdr(skb)->daddr,
type, code, ntohl(info));
IP_VS_DBG_PKT(11, af, pp, skb, iph.off, "Incoming packet");
ip_vs_in_stats(cp, skb);
- ip_vs_set_state(cp, IP_VS_DIR_INPUT, skb, pd);
+ ip_vs_set_state(cp, IP_VS_DIR_INPUT, skb, pd, iph.len);
if (cp->packet_xmit)
ret = cp->packet_xmit(skb, cp, pp, &iph);
/* do not touch skb anymore */
static inline void
set_sctp_state(struct ip_vs_proto_data *pd, struct ip_vs_conn *cp,
- int direction, const struct sk_buff *skb)
+ int direction, const struct sk_buff *skb,
+ unsigned int iph_len)
{
struct sctp_chunkhdr _sctpch, *sch;
unsigned char chunk_type;
int event, next_state;
- int ihl, cofs;
+ int cofs;
-#ifdef CONFIG_IP_VS_IPV6
- ihl = cp->af == AF_INET ? ip_hdrlen(skb) : sizeof(struct ipv6hdr);
-#else
- ihl = ip_hdrlen(skb);
-#endif
-
- cofs = ihl + sizeof(struct sctphdr);
+ cofs = iph_len + sizeof(struct sctphdr);
sch = skb_header_pointer(skb, cofs, sizeof(_sctpch), &_sctpch);
if (sch == NULL)
return;
static void
sctp_state_transition(struct ip_vs_conn *cp, int direction,
- const struct sk_buff *skb, struct ip_vs_proto_data *pd)
+ const struct sk_buff *skb, struct ip_vs_proto_data *pd,
+ unsigned int iph_len)
{
spin_lock_bh(&cp->lock);
- set_sctp_state(pd, cp, direction, skb);
+ set_sctp_state(pd, cp, direction, skb, iph_len);
spin_unlock_bh(&cp->lock);
}
static void
tcp_state_transition(struct ip_vs_conn *cp, int direction,
const struct sk_buff *skb,
- struct ip_vs_proto_data *pd)
+ struct ip_vs_proto_data *pd,
+ unsigned int iph_len)
{
struct tcphdr _tcph, *th;
-#ifdef CONFIG_IP_VS_IPV6
- int ihl = cp->af == AF_INET ? ip_hdrlen(skb) : sizeof(struct ipv6hdr);
-#else
- int ihl = ip_hdrlen(skb);
-#endif
-
- th = skb_header_pointer(skb, ihl, sizeof(_tcph), &_tcph);
+ th = skb_header_pointer(skb, iph_len, sizeof(_tcph), &_tcph);
if (th == NULL)
return;
static void
udp_state_transition(struct ip_vs_conn *cp, int direction,
const struct sk_buff *skb,
- struct ip_vs_proto_data *pd)
+ struct ip_vs_proto_data *pd,
+ unsigned int iph_len)
{
if (unlikely(!pd)) {
pr_err("UDP no ns data\n");
ip_vs_bypass_xmit(struct sk_buff *skb, struct ip_vs_conn *cp,
struct ip_vs_protocol *pp, struct ip_vs_iphdr *ipvsh)
{
- struct iphdr *iph = ip_hdr(skb);
-
- if (__ip_vs_get_out_rt(cp->ipvs, cp->af, skb, NULL, iph->daddr,
+ if (__ip_vs_get_out_rt(cp->ipvs, cp->af, skb, NULL, ip_hdr(skb)->daddr,
IP_VS_RT_MODE_NON_LOCAL, NULL, ipvsh) < 0)
goto tx_error;
- ip_send_check(iph);
+ ip_send_check(ip_hdr(skb));
/* Another hack: avoid icmp_send in ip_fragment */
skb->ignore_df = 1;
/* Not found, but might be about to be confirmed */
if (PTR_ERR(found) == -EAGAIN) {
if (nf_ct_tuple_equal(&conn->tuple, &tuple) &&
- nf_ct_zone_id(&conn->zone, conn->zone.dir) ==
- nf_ct_zone_id(zone, zone->dir))
+ nf_ct_zone_id(&conn->zone, IP_CT_DIR_ORIGINAL) ==
+ nf_ct_zone_id(zone, IP_CT_DIR_ORIGINAL))
goto out_put; /* already exists */
} else {
collect++;
found_ct = nf_ct_tuplehash_to_ctrack(found);
if (nf_ct_tuple_equal(&conn->tuple, &tuple) &&
- nf_ct_zone_equal(found_ct, zone, zone->dir)) {
+ nf_ct_zone_equal(found_ct, zone, IP_CT_DIR_ORIGINAL)) {
/*
* We should not see tuples twice unless someone hooks
* this into a table without "-p tcp --syn".
hlist_nulls_del_rcu(&ct->tuplehash[IP_CT_DIR_ORIGINAL].hnnode);
hlist_nulls_add_head(&ct->tuplehash[IP_CT_DIR_REPLY].hnnode, &evicted_list);
- if (time_after(stop, jiffies)) {
+ if (time_after(jiffies, stop)) {
ret = STATE_RESTART;
break;
}
switch (route->tuple[dir].xmit_type) {
case FLOW_OFFLOAD_XMIT_DIRECT:
+ if (route->tuple[!dir].in.num_tuns) {
+ flow_tuple->dst_cache = dst;
+ flow_tuple->dst_cookie =
+ flow_offload_dst_cookie(flow_tuple);
+ } else {
+ dst_release(dst);
+ }
memcpy(flow_tuple->out.h_dest, route->tuple[dir].out.h_dest,
ETH_ALEN);
memcpy(flow_tuple->out.h_source, route->tuple[dir].out.h_source,
ETH_ALEN);
flow_tuple->out.ifidx = route->tuple[dir].out.ifindex;
- dst_release(dst);
break;
case FLOW_OFFLOAD_XMIT_XFRM:
case FLOW_OFFLOAD_XMIT_NEIGH:
static void nft_flow_dst_release(struct flow_offload *flow,
enum flow_offload_tuple_dir dir)
{
- if (flow->tuplehash[dir].tuple.xmit_type == FLOW_OFFLOAD_XMIT_NEIGH ||
- flow->tuplehash[dir].tuple.xmit_type == FLOW_OFFLOAD_XMIT_XFRM)
- dst_release(flow->tuplehash[dir].tuple.dst_cache);
+ dst_release(flow->tuplehash[dir].tuple.dst_cache);
}
void flow_offload_route_init(struct flow_offload *flow,
nf_ct_refresh(flow->ct, NF_CT_DAY);
- if (nf_flowtable_hw_offload(flow_table)) {
- __set_bit(NF_FLOW_HW, &flow->flags);
+ if (nf_flowtable_hw_offload(flow_table))
nf_flow_offload_add(flow_table, flow);
- }
return 0;
}
test_bit(NF_FLOW_CLOSING, &flow->flags))
return;
- nf_flow_offload_add(flow_table, flow);
+ if (test_bit(NF_FLOW_HW, &flow->flags))
+ nf_flow_offload_refresh(flow_table, flow);
}
EXPORT_SYMBOL_GPL(flow_offload_refresh);
static inline bool nf_flow_dst_check(struct flow_offload_tuple *tuple)
{
- if (tuple->xmit_type != FLOW_OFFLOAD_XMIT_NEIGH &&
- tuple->xmit_type != FLOW_OFFLOAD_XMIT_XFRM)
+ if (!tuple->dst_cache)
return true;
return dst_check(tuple->dst_cache, tuple->dst_cookie);
static int nf_flow_tunnel_ipip_push(struct net *net, struct sk_buff *skb,
struct flow_offload_tuple *tuple,
- __be32 *ip_daddr)
+ struct dst_entry *dst, __be32 *ip_daddr)
{
struct iphdr *iph = (struct iphdr *)skb_network_header(skb);
- struct rtable *rt = dst_rtable(tuple->dst_cache);
+ struct rtable *rt = dst_rtable(dst);
u8 tos = iph->tos, ttl = iph->ttl;
__be16 frag_off = iph->frag_off;
u32 headroom = sizeof(*iph);
static int nf_flow_tunnel_v4_push(struct net *net, struct sk_buff *skb,
struct flow_offload_tuple *tuple,
- __be32 *ip_daddr)
+ struct dst_entry *dst, __be32 *ip_daddr)
{
if (tuple->tun_num)
- return nf_flow_tunnel_ipip_push(net, skb, tuple, ip_daddr);
+ return nf_flow_tunnel_ipip_push(net, skb, tuple, dst, ip_daddr);
return 0;
}
static int nf_flow_tunnel_ip6ip6_push(struct net *net, struct sk_buff *skb,
struct flow_offload_tuple *tuple,
+ struct dst_entry *dst,
struct in6_addr **ip6_daddr)
{
struct ipv6hdr *ip6h = (struct ipv6hdr *)skb_network_header(skb);
- struct rtable *rt = dst_rtable(tuple->dst_cache);
__u8 dsfield = ipv6_get_dsfield(ip6h);
+ struct rtable *rt = dst_rtable(dst);
struct flowi6 fl6 = {
.daddr = tuple->tun.src_v6,
.saddr = tuple->tun.dst_v6,
static int nf_flow_tunnel_v6_push(struct net *net, struct sk_buff *skb,
struct flow_offload_tuple *tuple,
+ struct dst_entry *dst,
struct in6_addr **ip6_daddr)
{
if (tuple->tun_num)
- return nf_flow_tunnel_ip6ip6_push(net, skb, tuple, ip6_daddr);
+ return nf_flow_tunnel_ip6ip6_push(net, skb, tuple, dst, ip6_daddr);
return 0;
}
other_tuple = &flow->tuplehash[!dir].tuple;
ip_daddr = other_tuple->src_v4.s_addr;
- if (nf_flow_tunnel_v4_push(state->net, skb, other_tuple, &ip_daddr) < 0)
+ if (nf_flow_tunnel_v4_push(state->net, skb, other_tuple,
+ tuplehash->tuple.dst_cache, &ip_daddr) < 0)
return NF_DROP;
switch (tuplehash->tuple.xmit_type) {
ip6_daddr = &other_tuple->src_v6;
if (nf_flow_tunnel_v6_push(state->net, skb, other_tuple,
+ tuplehash->tuple.dst_cache,
&ip6_daddr) < 0)
return NF_DROP;
return offload;
}
+static bool nf_flow_offload_unsupported(struct flow_offload *flow)
+{
+ if (flow->tuplehash[FLOW_OFFLOAD_DIR_ORIGINAL].tuple.tun_num ||
+ flow->tuplehash[FLOW_OFFLOAD_DIR_REPLY].tuple.tun_num)
+ return true;
-void nf_flow_offload_add(struct nf_flowtable *flowtable,
- struct flow_offload *flow)
+ return false;
+}
+
+void nf_flow_offload_refresh(struct nf_flowtable *flowtable,
+ struct flow_offload *flow)
{
struct flow_offload_work *offload;
flow_offload_queue_work(offload);
}
+void nf_flow_offload_add(struct nf_flowtable *flowtable,
+ struct flow_offload *flow)
+{
+ if (nf_flow_offload_unsupported(flow))
+ return;
+
+ set_bit(NF_FLOW_HW, &flow->flags);
+ nf_flow_offload_refresh(flowtable, flow);
+}
+
void nf_flow_offload_del(struct nf_flowtable *flowtable,
struct flow_offload *flow)
{
/* Mangle destination port for Cisco phones, then fix up checksums */
if (dir == IP_CT_DIR_REPLY && ct_sip_info->forced_dport) {
+ int doff = *dptr - (const char *)skb->data;
struct udphdr *uh;
+ if (doff <= 0) {
+ DEBUG_NET_WARN_ON_ONCE(1);
+ return NF_DROP;
+ }
+
+ /* ct_sip_info->forced_dport only expected with UDP */
+ if (nf_ct_protonum(ct) != IPPROTO_UDP)
+ return NF_DROP;
+
if (skb_ensure_writable(skb, skb->len)) {
nf_ct_helper_log(skb, ct, "cannot mangle packet");
return NF_DROP;
}
+ *dptr = skb->data + doff;
uh = (void *)skb->data + protoff;
uh->dest = ct_sip_info->forced_dport;
if (err < 0)
return err;
+ if (!elem.priv)
+ return 0;
+
err = -ENOMEM;
skb = nlmsg_new(NLMSG_GOODSIZE, GFP_ATOMIC);
if (skb == NULL)
new_policy->max_expected =
ntohl(nla_get_be32(tb[NFCTH_POLICY_EXPECT_MAX]));
+ if (!new_policy->max_expected)
+ new_policy->max_expected = NF_CT_EXPECT_MAX_CNT;
if (new_policy->max_expected > NF_CT_EXPECT_MAX_CNT)
return -EINVAL;
goto nla_put_failure;
if (skb_copy_bits(skb, 0, nla_data(nla), data_len))
- BUG();
+ goto nla_put_failure;
}
nlh->nlmsg_len = inst->skb->tail - old_tail;
},
};
+static unsigned int nfulnl_get_copy_len(const struct nf_loginfo *li,
+ const struct sk_buff *skb,
+ unsigned int copy_len)
+{
+ unsigned int len = skb->len;
+
+ if ((li->u.ulog.flags & NF_LOG_F_COPY_LEN) &&
+ li->u.ulog.copy_len < copy_len)
+ copy_len = li->u.ulog.copy_len;
+ if (!skb_frags_readable(skb))
+ len = skb_headlen(skb);
+
+ return min(len, copy_len);
+}
+
/* log handler for internal netfilter logging api */
static void
nfulnl_log_packet(struct net *net,
break;
case NFULNL_COPY_PACKET:
- data_len = inst->copy_range;
- if ((li->u.ulog.flags & NF_LOG_F_COPY_LEN) &&
- (li->u.ulog.copy_len < data_len))
- data_len = li->u.ulog.copy_len;
-
- if (data_len > skb->len)
- data_len = skb->len;
-
+ data_len = nfulnl_get_copy_len(li, skb, inst->copy_range);
size += nla_total_size(data_len);
break;
}
#endif
+static unsigned int nfqnl_get_data_len(const struct sk_buff *entskb,
+ unsigned int copy_range)
+{
+ unsigned int data_len = entskb->len;
+
+ if (!skb_frags_readable(entskb))
+ data_len = skb_headlen(entskb);
+
+ return min(data_len, copy_range);
+}
+
static struct sk_buff *
nfqnl_build_packet_message(struct net *net, struct nfqnl_instance *queue,
struct nf_queue_entry *entry,
nf_queue_checksum_help(entskb))
return NULL;
- data_len = READ_ONCE(queue->copy_range);
- if (data_len > entskb->len)
- data_len = entskb->len;
-
+ data_len = nfqnl_get_data_len(entskb, READ_ONCE(queue->copy_range));
hlen = skb_zerocopy_headlen(entskb);
hlen = min_t(unsigned int, hlen, data_len);
size += sizeof(struct nlattr) + hlen;
bool found;
ext = nft_set_do_lookup(net, set, ®s->data[priv->sreg]);
+ if (!ext)
+ ext = nft_set_catchall_lookup(net, set);
+
found = !!ext ^ priv->invert;
if (!found) {
- ext = nft_set_catchall_lookup(net, set);
- if (!ext) {
- regs->verdict.code = NFT_BREAK;
- return;
- }
+ regs->verdict.code = NFT_BREAK;
+ return;
}
if (ext) {
if (!interval || nft_set_elem_expired(interval->from))
return ERR_PTR(-ENOENT);
- if (flags & NFT_SET_ELEM_INTERVAL_END)
+ if (flags & NFT_SET_ELEM_INTERVAL_END) {
+ if (!interval->to)
+ return NULL;
+
rbe = container_of(interval->to, struct nft_rbtree_elem, ext);
- else
+ } else {
rbe = container_of(interval->from, struct nft_rbtree_elem, ext);
+ }
return &rbe->priv;
}
return ret;
}
+static int connmark_tg_check_v2(const struct xt_tgchk_param *par)
+{
+ const struct xt_connmark_tginfo2 *info = par->targinfo;
+
+ if (info->shift_dir > D_SHIFT_RIGHT || info->shift_bits >= 32)
+ return -EINVAL;
+
+ return connmark_tg_check(par);
+}
+
static void connmark_tg_destroy(const struct xt_tgdtor_param *par)
{
nf_ct_netns_put(par->net, par->family);
.name = "CONNMARK",
.revision = 2,
.family = NFPROTO_IPV4,
- .checkentry = connmark_tg_check,
+ .checkentry = connmark_tg_check_v2,
.target = connmark_tg_v2,
.targetsize = sizeof(struct xt_connmark_tginfo2),
.destroy = connmark_tg_destroy,
.name = "CONNMARK",
.revision = 2,
.family = NFPROTO_IPV6,
- .checkentry = connmark_tg_check,
+ .checkentry = connmark_tg_check_v2,
.target = connmark_tg_v2,
.targetsize = sizeof(struct xt_connmark_tginfo2),
.destroy = connmark_tg_destroy,
static int xt_nat_checkentry(const struct xt_tgchk_param *par)
{
+ switch (par->family) {
+ case NFPROTO_IPV4:
+ case NFPROTO_IPV6:
+ case NFPROTO_INET:
+ break;
+ default:
+ return -EINVAL;
+ }
+
return nf_ct_netns_get(par->net, par->family);
}
if (X(physoutdev))
return -ENAMETOOLONG;
}
-
- if (X(in_mask))
- return -ENAMETOOLONG;
- if (X(out_mask))
- return -ENAMETOOLONG;
#undef X
if (!brnf_probed) {
{
const struct xt_rateest_match_info *info = par->matchinfo;
struct gnet_stats_rate_est64 sample = {0};
- u_int32_t bps1, bps2, pps1, pps2;
+ u64 bps1, bps2, pps1, pps2;
bool ret = true;
gen_estimator_read(&info->est1->rate_est, &sample);
#include <linux/netfilter/x_tables.h>
#include <linux/netfilter/xt_u32.h>
-static bool u32_match_it(const struct xt_u32 *data,
- const struct sk_buff *skb)
+static int u32_match_it(const struct xt_u32 *data,
+ const struct sk_buff *skb)
{
const struct xt_u32_test *ct;
unsigned int testind;
return false;
if (skb_copy_bits(skb, pos, &n, sizeof(n)) < 0)
- BUG();
+ return -1;
+
val = ntohl(n);
nnums = ct->nnums;
if (skb_copy_bits(skb, at + pos, &n,
sizeof(n)) < 0)
- BUG();
+ return -1;
val = ntohl(n);
break;
}
static bool u32_mt(const struct sk_buff *skb, struct xt_action_param *par)
{
const struct xt_u32 *data = par->matchinfo;
- bool ret;
+ int ret;
ret = u32_match_it(data, skb);
+ if (ret < 0) {
+ par->hotdrop = true;
+ return false;
+ }
+
return ret ^ data->invert;
}
{
const struct xt_u32 *data = par->matchinfo;
const struct xt_u32_test *ct;
- unsigned int i;
+ unsigned int i, j;
if (data->ntests > ARRAY_SIZE(data->tests))
return -EINVAL;
if (ct->nnums > ARRAY_SIZE(ct->location) ||
ct->nvalues > ARRAY_SIZE(ct->value))
return -EINVAL;
+
+ for (j = 1; j < ct->nnums; ++j) {
+ switch (ct->location[j].nextop) {
+ case XT_U32_LEFTSH:
+ case XT_U32_RIGHTSH:
+ if (ct->location[j].number >= 32)
+ return -EINVAL;
+ break;
+ }
+ }
}
return 0;
return used;
}
-static inline void add_nested_action_end(struct sw_flow_actions *sfa,
- int st_offset)
+static inline int add_nested_action_end(struct sw_flow_actions *sfa,
+ int st_offset)
{
- struct nlattr *a = (struct nlattr *) ((unsigned char *)sfa->actions +
- st_offset);
+ struct nlattr *a;
+ u32 attr_len;
+
+ if (WARN_ON_ONCE(st_offset < 0 ||
+ (u32)st_offset > sfa->actions_len))
+ return -EINVAL;
+
+ attr_len = sfa->actions_len - (u32)st_offset;
+ if (WARN_ON_ONCE(attr_len < NLA_HDRLEN))
+ return -EINVAL;
- a->nla_len = sfa->actions_len - st_offset;
+ if (attr_len > U16_MAX)
+ return -EMSGSIZE;
+
+ a = (struct nlattr *)((u8 *)sfa->actions + st_offset);
+ a->nla_len = attr_len;
+ return 0;
+}
+
+/* Free the generated action-list tail at @start and truncate it.
+ * If @nested, @start points to its containing nlattr header.
+ */
+static void ovs_nla_trim(struct sw_flow_actions *sfa, int start, bool nested)
+{
+ const struct nlattr *actions;
+ u32 len;
+
+ if (start < 0)
+ return;
+
+ if (WARN_ON_ONCE((u32)start > sfa->actions_len))
+ return;
+
+ actions = (const struct nlattr *)((u8 *)sfa->actions + start);
+ len = sfa->actions_len - (u32)start;
+
+ if (nested) {
+ if (len < NLA_HDRLEN)
+ goto out;
+
+ actions = (const struct nlattr *)((u8 *)actions + NLA_HDRLEN);
+ len -= NLA_HDRLEN;
+ }
+
+ ovs_nla_free_nested_actions(actions, len);
+out:
+ sfa->actions_len = start;
}
static int __ovs_nla_copy_actions(struct net *net, const struct nlattr *attr,
const struct nlattr *attrs[OVS_SAMPLE_ATTR_MAX + 1];
const struct nlattr *probability, *actions;
const struct nlattr *a;
+ int actions_start;
int rem, start, err;
struct sample_arg arg;
err = ovs_nla_add_action(sfa, OVS_SAMPLE_ATTR_ARG, &arg, sizeof(arg),
log);
if (err)
- return err;
+ goto err;
+ actions_start = (*sfa)->actions_len;
err = __ovs_nla_copy_actions(net, actions, key, sfa,
eth_type, vlan_tci, mpls_label_count, log,
depth + 1);
if (err)
- return err;
+ goto err_free;
- add_nested_action_end(*sfa, start);
+ err = add_nested_action_end(*sfa, start);
+ if (err)
+ goto err_free;
return 0;
+
+err_free:
+ ovs_nla_trim(*sfa, actions_start, false);
+err:
+ (*sfa)->actions_len = start;
+ return err;
}
static int validate_and_copy_dec_ttl(struct net *net,
return start;
action_start = add_nested_action_start(sfa, OVS_DEC_TTL_ATTR_ACTION, log);
- if (action_start < 0)
- return action_start;
+ if (action_start < 0) {
+ err = action_start;
+ goto err;
+ }
err = __ovs_nla_copy_actions(net, actions, key, sfa, eth_type,
vlan_tci, mpls_label_count, log,
depth + 1);
if (err)
- return err;
+ goto err_free;
+
+ err = add_nested_action_end(*sfa, action_start);
+ if (err)
+ goto err_free;
- add_nested_action_end(*sfa, action_start);
- add_nested_action_end(*sfa, start);
+ err = add_nested_action_end(*sfa, start);
+ if (err)
+ goto err_free;
return 0;
+
+err_free:
+ ovs_nla_trim(*sfa, action_start, true);
+err:
+ (*sfa)->actions_len = start;
+ return err;
}
static int validate_and_copy_clone(struct net *net,
u32 mpls_label_count, bool log, bool last,
u32 depth)
{
+ int actions_start;
int start, err;
u32 exec;
err = ovs_nla_add_action(sfa, OVS_CLONE_ATTR_EXEC, &exec,
sizeof(exec), log);
if (err)
- return err;
+ goto err;
+ actions_start = (*sfa)->actions_len;
err = __ovs_nla_copy_actions(net, attr, key, sfa,
eth_type, vlan_tci, mpls_label_count, log,
depth + 1);
if (err)
- return err;
+ goto err_free;
- add_nested_action_end(*sfa, start);
+ err = add_nested_action_end(*sfa, start);
+ if (err)
+ goto err_free;
return 0;
+
+err_free:
+ ovs_nla_trim(*sfa, actions_start, false);
+err:
+ (*sfa)->actions_len = start;
+ return err;
}
void ovs_match_init(struct sw_flow_match *match,
tun_dst = metadata_dst_alloc(key.tun_opts_len, METADATA_IP_TUNNEL,
GFP_KERNEL);
- if (!tun_dst)
- return -ENOMEM;
+ if (!tun_dst) {
+ err = -ENOMEM;
+ goto err;
+ }
err = dst_cache_init(&tun_dst->u.tun_info.dst_cache, GFP_KERNEL);
- if (err) {
- dst_release((struct dst_entry *)tun_dst);
- return err;
- }
+ if (err)
+ goto err_free_tun_dst;
a = __add_action(sfa, OVS_KEY_ATTR_TUNNEL_INFO, NULL,
sizeof(*ovs_tun), log);
if (IS_ERR(a)) {
- dst_release((struct dst_entry *)tun_dst);
- return PTR_ERR(a);
+ err = PTR_ERR(a);
+ goto err_free_tun_dst;
}
ovs_tun = nla_data(a);
ip_tunnel_info_opts_set(tun_info,
TUN_METADATA_OPTS(&key, key.tun_opts_len),
key.tun_opts_len, dst_opt_type);
- add_nested_action_end(*sfa, start);
+ err = add_nested_action_end(*sfa, start);
+ if (WARN_ON_ONCE(err))
+ goto err_free_tun_dst;
+
+ return 0;
+err_free_tun_dst:
+ dst_release((struct dst_entry *)tun_dst);
+err:
+ (*sfa)->actions_len = start;
return err;
}
/* Convert non-masked non-tunnel set actions to masked set actions. */
if (!masked && key_type != OVS_KEY_ATTR_TUNNEL) {
- int start, len = key_len * 2;
+ int err, start, len = key_len * 2;
struct nlattr *at;
*skip_copy = true;
return start;
at = __add_action(sfa, key_type, NULL, len, log);
- if (IS_ERR(at))
- return PTR_ERR(at);
+ if (IS_ERR(at)) {
+ err = PTR_ERR(at);
+ (*sfa)->actions_len = start;
+ return err;
+ }
memcpy(nla_data(at), nla_data(ovs_key), key_len); /* Key. */
memset(nla_data(at) + key_len, 0xff, key_len); /* Mask. */
mask->ipv6_label &= htonl(0x000FFFFF);
}
- add_nested_action_end(*sfa, start);
+ err = add_nested_action_end(*sfa, start);
+ if (WARN_ON_ONCE(err)) {
+ (*sfa)->actions_len = start;
+ return err;
+ }
}
return 0;
const struct nlattr *acts_if_greater, *acts_if_lesser_eq;
struct nlattr *a[OVS_CHECK_PKT_LEN_ATTR_MAX + 1];
struct check_pkt_len_arg arg;
- int nested_acts_start;
+ int greater_acts_start = -1;
+ int lesser_acts_start = -1;
int start, err;
err = nla_parse_deprecated_strict(a, OVS_CHECK_PKT_LEN_ATTR_MAX,
err = ovs_nla_add_action(sfa, OVS_CHECK_PKT_LEN_ATTR_ARG, &arg,
sizeof(arg), log);
if (err)
- return err;
+ goto err_free;
- nested_acts_start = add_nested_action_start(sfa,
- OVS_CHECK_PKT_LEN_ATTR_ACTIONS_IF_LESS_EQUAL, log);
- if (nested_acts_start < 0)
- return nested_acts_start;
+ lesser_acts_start =
+ add_nested_action_start(sfa,
+ OVS_CHECK_PKT_LEN_ATTR_ACTIONS_IF_LESS_EQUAL,
+ log);
+ if (lesser_acts_start < 0) {
+ err = lesser_acts_start;
+ goto err_free;
+ }
err = __ovs_nla_copy_actions(net, acts_if_lesser_eq, key, sfa,
eth_type, vlan_tci, mpls_label_count, log,
depth + 1);
if (err)
- return err;
+ goto err_free;
- add_nested_action_end(*sfa, nested_acts_start);
+ err = add_nested_action_end(*sfa, lesser_acts_start);
+ if (err)
+ goto err_free;
- nested_acts_start = add_nested_action_start(sfa,
- OVS_CHECK_PKT_LEN_ATTR_ACTIONS_IF_GREATER, log);
- if (nested_acts_start < 0)
- return nested_acts_start;
+ greater_acts_start =
+ add_nested_action_start(sfa,
+ OVS_CHECK_PKT_LEN_ATTR_ACTIONS_IF_GREATER,
+ log);
+ if (greater_acts_start < 0) {
+ err = greater_acts_start;
+ goto err_free;
+ }
err = __ovs_nla_copy_actions(net, acts_if_greater, key, sfa,
eth_type, vlan_tci, mpls_label_count, log,
depth + 1);
if (err)
- return err;
+ goto err_free;
+
+ err = add_nested_action_end(*sfa, greater_acts_start);
+ if (err)
+ goto err_free;
- add_nested_action_end(*sfa, nested_acts_start);
- add_nested_action_end(*sfa, start);
+ err = add_nested_action_end(*sfa, start);
+ if (err)
+ goto err_free;
return 0;
+
+err_free:
+ ovs_nla_trim(*sfa, greater_acts_start, true);
+ ovs_nla_trim(*sfa, lesser_acts_start, true);
+ ovs_nla_trim(*sfa, start, false);
+ return err;
}
static int validate_psample(const struct nlattr *attr)
static unsigned int tcf_offload_act_num_actions_single(struct tc_action *act)
{
- if (is_tcf_pedit(act))
- return tcf_pedit_nkeys(act);
- else
- return 1;
+ unsigned int count;
+
+ if (is_tcf_pedit(act)) {
+ spin_lock_bh(&act->tcfa_lock);
+ count = tcf_pedit_nkeys_locked(act);
+ spin_unlock_bh(&act->tcfa_lock);
+ return count;
+ }
+ return 1;
}
static bool tc_act_skip_hw(u32 flags)
{
if (bind) {
struct flow_action_entry *entry = entry_data;
+ int nkeys = tcf_pedit_nkeys_locked(act);
int k;
- for (k = 0; k < tcf_pedit_nkeys(act); k++) {
+ /* If the required keys exceed the remaining capacity return
+ * -ENOSPC to abort the offload and fallback to software.
+ */
+ if (nkeys > *index_inc) {
+ NL_SET_ERR_MSG_MOD(extack, "Not enough space to offload all pedit keys");
+ return -ENOSPC;
+ }
+
+ for (k = 0; k < nkeys; k++) {
switch (tcf_pedit_cmd(act, k)) {
case TCA_PEDIT_KEY_EX_CMD_SET:
entry->id = FLOW_ACTION_MANGLE;
return -EOPNOTSUPP;
}
- for (k = 1; k < tcf_pedit_nkeys(act); k++) {
+ for (k = 1; k < tcf_pedit_nkeys_locked(act); k++) {
if (cmd != tcf_pedit_cmd(act, k)) {
NL_SET_ERR_MSG_MOD(extack, "Unsupported pedit command offload");
return -EOPNOTSUPP;
[TCA_TUNNEL_KEY_ENC_TTL] = { .type = NLA_U8 },
};
-static void tunnel_key_release_params(struct tcf_tunnel_key_params *p)
+static void tunnel_key_release_params_rcu(struct rcu_head *head)
{
- if (!p)
- return;
+ struct tcf_tunnel_key_params *p = container_of(head, typeof(*p), rcu);
+
if (p->tcft_action == TCA_TUNNEL_KEY_ACT_SET)
dst_release(&p->tcft_enc_metadata->dst);
+ kfree(p);
+}
- kfree_rcu(p, rcu);
+static void tunnel_key_release_params(struct tcf_tunnel_key_params *p)
+{
+ if (!p)
+ return;
+ call_rcu(&p->rcu, tunnel_key_release_params_rcu);
}
static int tunnel_key_init(struct net *net, struct nlattr *nla,
entry = &flow_action->entries[j];
spin_lock_bh(&act->tcfa_lock);
+
+ /* Abort the offload if we have exhausted the allocated capacity */
+ if (j >= flow_action->num_entries) {
+ NL_SET_ERR_MSG_MOD(extack, "Flow action buffer overflow");
+ err = -ENOSPC;
+ goto err_out_locked;
+ }
+
err = tcf_act_get_user_cookie(entry, act);
if (err)
goto err_out_locked;
- index = 0;
- err = tc_setup_offload_act(act, entry, &index, extack);
+ index = flow_action->num_entries - j;
+ err = tc_setup_offload_act(act, entry, &index,
+ extack);
if (err)
goto err_out_locked;
int i;
tcf_exts_for_each_action(i, act, exts) {
- if (is_tcf_pedit(act))
- num_acts += tcf_pedit_nkeys(act);
- else
+ if (is_tcf_pedit(act)) {
+ spin_lock_bh(&act->tcfa_lock);
+ num_acts += tcf_pedit_nkeys_locked(act);
+ spin_unlock_bh(&act->tcfa_lock);
+ } else {
num_acts++;
+ }
}
return num_acts;
}
if (qd->min_netlen > len)
WRITE_ONCE(qd->min_netlen, len);
- len += q->rate_overhead;
-
- if (len < q->rate_mpu)
- len = q->rate_mpu;
+ len = max((s32)len + q->rate_overhead, (s32)q->rate_mpu);
if (q->atm_mode == CAKE_ATM_ATM) {
len += 47;
int subq = skb_get_queue_mapping(skb);
struct sk_buff *skb_res = NULL;
- rcu_read_lock_bh();
-
- start = rcu_dereference_bh(master->slaves);
-
restart:
nores = 0;
busy = 0;
+ rcu_read_lock();
+
+ start = rcu_dereference(master->slaves);
+
q = start;
if (!q)
goto drop;
netdev_start_xmit(skb, slave, slave_txq, false) ==
NETDEV_TX_OK) {
__netif_tx_unlock(slave_txq);
- spin_lock_bh(&master->slaves_lock);
+ spin_lock(&master->slaves_lock);
if (rcu_dereference_protected(master->slaves,
lockdep_is_held(&master->slaves_lock)) == q)
rcu_assign_pointer(master->slaves,
rcu_dereference_protected(NEXT_SLAVE(q),
lockdep_is_held(&master->slaves_lock)));
- spin_unlock_bh(&master->slaves_lock);
+ spin_unlock(&master->slaves_lock);
netif_wake_queue(dev);
master->tx_packets++;
master->tx_bytes += length;
- rcu_read_unlock_bh();
+ rcu_read_unlock();
return NETDEV_TX_OK;
}
__netif_tx_unlock(slave_txq);
busy = 1;
break;
case 1:
- spin_lock_bh(&master->slaves_lock);
+ spin_lock(&master->slaves_lock);
if (rcu_dereference_protected(master->slaves,
lockdep_is_held(&master->slaves_lock)) == q)
rcu_assign_pointer(master->slaves,
rcu_dereference_protected(NEXT_SLAVE(q),
lockdep_is_held(&master->slaves_lock)));
- spin_unlock_bh(&master->slaves_lock);
- rcu_read_unlock_bh();
+ spin_unlock(&master->slaves_lock);
+ rcu_read_unlock();
return NETDEV_TX_OK;
default:
nores = 1;
break;
}
__skb_pull(skb, skb_network_offset(skb));
- } while ((q = rcu_dereference_bh(NEXT_SLAVE(q))) != start);
+ } while ((q = rcu_dereference(NEXT_SLAVE(q))) != start);
if (nores && skb_res == NULL) {
skb_res = skb;
+ rcu_read_unlock();
goto restart;
}
if (busy) {
netif_stop_queue(dev);
- rcu_read_unlock_bh();
+ rcu_read_unlock();
return NETDEV_TX_BUSY;
}
master->tx_errors++;
drop:
master->tx_dropped++;
- rcu_read_unlock_bh();
+ rcu_read_unlock();
dev_kfree_skb(skb);
return NETDEV_TX_OK;
}
const struct sctp_association *asoc,
const union sctp_subtype type,
void *arg,
- struct sctp_cmd_seq *commands);
+ struct sctp_cmd_seq *commands,
+ struct sctp_errhdr *err);
static enum sctp_disposition sctp_sf_shut_8_4_5(
struct net *net,
const struct sctp_endpoint *ep,
* errors.
*/
sctp_walk_errors(err, chunk->chunk_hdr) {
- if (SCTP_ERROR_STALE_COOKIE == err->cause)
- return sctp_sf_do_5_2_6_stale(net, ep, asoc, type,
- arg, commands);
+ if (err->cause != SCTP_ERROR_STALE_COOKIE)
+ continue;
+ /* The staleness is only meaningful if the cause is long
+ * enough to hold it; a shorter one is malformed.
+ */
+ if (ntohs(err->length) < sizeof(*err) + sizeof(__be32))
+ break;
+ return sctp_sf_do_5_2_6_stale(net, ep, asoc, type,
+ arg, commands, err);
}
/* It is possible to have malformed error causes, and that
const struct sctp_association *asoc,
const union sctp_subtype type,
void *arg,
- struct sctp_cmd_seq *commands)
+ struct sctp_cmd_seq *commands,
+ struct sctp_errhdr *err)
{
int attempts = asoc->init_err_counter + 1;
- struct sctp_chunk *chunk = arg, *reply;
struct sctp_cookie_preserve_param bht;
struct sctp_bind_addr *bp;
- struct sctp_errhdr *err;
+ struct sctp_chunk *reply;
u32 stale;
if (attempts > asoc->max_init_attempts) {
return SCTP_DISPOSITION_DELETE_TCB;
}
- err = (struct sctp_errhdr *)(chunk->skb->data);
-
/* When calculating the time extension, an implementation
* SHOULD use the RTT information measured based on the
* previous COOKIE ECHO / ERROR exchange, and should add no
{
struct smc_link *link = (struct smc_link *)wc->qp->qp_context;
struct smc_cdc_msg *cdc = buf;
+ struct smc_sock *smc = NULL;
struct smc_connection *conn;
struct smc_link_group *lgr;
- struct smc_sock *smc;
if (wc->byte_len < offsetof(struct smc_cdc_msg, reserved))
return; /* short message */
lgr = smc_get_lgr(link);
read_lock_bh(&lgr->conns_lock);
conn = smc_lgr_find_conn(ntohl(cdc->token), lgr);
- read_unlock_bh(&lgr->conns_lock);
- if (!conn || conn->out_of_sync)
+ if (!conn || conn->out_of_sync) {
+ read_unlock_bh(&lgr->conns_lock);
return;
+ }
smc = container_of(conn, struct smc_sock, conn);
+ sock_hold(&smc->sk);
+ read_unlock_bh(&lgr->conns_lock);
if (cdc->prod_flags.failover_validation) {
smc_cdc_msg_validate(smc, cdc, link);
- return;
+ goto out;
}
if (smc_cdc_before(ntohs(cdc->seqno),
conn->local_rx_ctrl.seqno))
/* received seqno is old */
- return;
+ goto out;
smc_cdc_msg_recv(smc, cdc);
+out:
+ sock_put(&smc->sk);
}
static struct smc_wr_rx_handler smc_cdc_rx_handlers[] = {
return NULL;
}
-/*
- * Release reference to the RPC client
+/**
+ * rpc_hold_client - acquire a reference on an rpc_clnt
+ * @clnt: rpc_clnt to pin
+ *
+ * Pairs with rpc_release_client().
+ */
+void rpc_hold_client(struct rpc_clnt *clnt)
+{
+ refcount_inc(&clnt->cl_count);
+}
+
+/**
+ * rpc_release_client - release a reference on an rpc_clnt
+ * @clnt: rpc_clnt to release
+ *
+ * Pairs with rpc_hold_client(). The rpc_clnt's resources are
+ * freed once its reference count drops to zero.
*/
void
rpc_release_client(struct rpc_clnt *clnt)
lower_xprt = rcu_dereference(lower_clnt->cl_xprt);
rcu_read_unlock();
- if (wait_on_bit_lock(&lower_xprt->state, XPRT_LOCKED, TASK_KILLABLE))
+ if (wait_on_bit_lock(&lower_xprt->state, XPRT_LOCKED, TASK_KILLABLE)) {
+ /* XPRT_LOCKED was never acquired. */
+ rpc_shutdown_client(lower_clnt);
goto out_unlock;
+ }
status = xs_tls_handshake_sync(lower_xprt, &upper_xprt->xprtsec);
if (status) {
out_unlock:
current_restore_flags(pflags, PF_MEMALLOC);
upper_transport->clnt = NULL;
+ rpc_release_client(upper_clnt);
xprt_unlock_connect(upper_xprt, upper_transport);
return;
} else
dprintk("RPC: xs_connect scheduled xprt %p\n", xprt);
- transport->clnt = task->tk_client;
+ /*
+ * Only the TLS connect_worker reads transport->clnt; pinning
+ * the upper rpc_clnt unconditionally would form a cycle with
+ * cl_xprt and prevent xprt destruction.
+ */
+ if (xprt->xprtsec.policy != RPC_XPRTSEC_NONE) {
+ rpc_hold_client(task->tk_client);
+ transport->clnt = task->tk_client;
+ }
queue_delayed_work(xprtiod_workqueue,
&transport->connect_worker,
delay);
goto read_sock_requeue;
}
+ /* An empty data record (legal in TLS 1.3) gives a zero
+ * read_actor return, indistinguishable from the consumer
+ * stalling; the used <= 0 path would requeue it at the
+ * head of rx_list and block all later records. Consume it
+ * here instead.
+ */
+ if (rxm->full_len == 0) {
+ consume_skb(skb);
+ continue;
+ }
+
used = read_actor(desc, skb, rxm->offset, rxm->full_len);
if (used <= 0) {
if (!copied)
/* this has nothing to do now but make sure it's gone */
cancel_work_sync(&rdev->wiphy_work);
+ cancel_work_sync(&rdev->sched_scan_res_wk);
cancel_work_sync(&rdev->rfkill_block);
cancel_work_sync(&rdev->conn_work);
flush_work(&rdev->event_work);
list_del_rcu(&wdev->list);
synchronize_net();
rdev->devlist_generation++;
+ wiphy_work_cancel(wdev->wiphy, &wdev->disconnect_wk);
cfg80211_mlme_purge_registrations(wdev);
INIT_LIST_HEAD(&wdev->mgmt_registrations);
INIT_LIST_HEAD(&wdev->pmsr_list);
spin_lock_init(&wdev->pmsr_lock);
- INIT_WORK(&wdev->pmsr_free_wk, cfg80211_pmsr_free_wk);
+ wiphy_work_init(&wdev->pmsr_free_wk, cfg80211_pmsr_free_wk);
#ifdef CONFIG_CFG80211_WEXT
wdev->wext.default_key = -1;
wdev->iftype == NL80211_IFTYPE_ADHOC) && !wdev->use_4addr)
wdev->netdev->priv_flags |= IFF_DONT_BRIDGE;
- INIT_WORK(&wdev->disconnect_wk, cfg80211_autodisconnect_wk);
+ wiphy_work_init(&wdev->disconnect_wk, cfg80211_autodisconnect_wk);
}
void cfg80211_register_wdev(struct cfg80211_registered_device *rdev,
break;
case NETDEV_GOING_DOWN:
cfg80211_leave(rdev, wdev, -1);
- scoped_guard(wiphy, &rdev->wiphy)
+ scoped_guard(wiphy, &rdev->wiphy) {
cfg80211_remove_links(wdev);
- /* since we just did cfg80211_leave() nothing to do there */
- cancel_work_sync(&wdev->disconnect_wk);
- cancel_work_sync(&wdev->pmsr_free_wk);
+ /* since we just did cfg80211_leave() nothing to do there */
+ wiphy_work_cancel(wdev->wiphy, &wdev->disconnect_wk);
+ }
break;
case NETDEV_DOWN:
wiphy_lock(&rdev->wiphy);
const u8 *td_bitmap, u8 td_bitmap_len);
int cfg80211_mgd_wext_connect(struct cfg80211_registered_device *rdev,
struct wireless_dev *wdev);
-void cfg80211_autodisconnect_wk(struct work_struct *work);
+void cfg80211_autodisconnect_wk(struct wiphy *wiphy, struct wiphy_work *work);
/* SME implementation */
void cfg80211_conn_work(struct work_struct *work);
void cfg80211_release_pmsr(struct wireless_dev *wdev, u32 portid);
void cfg80211_pmsr_wdev_down(struct wireless_dev *wdev);
-void cfg80211_pmsr_free_wk(struct work_struct *work);
+void cfg80211_pmsr_free_wk(struct wiphy *wiphy, struct wiphy_work *work);
void cfg80211_remove_link(struct wireless_dev *wdev, unsigned int link_id);
void cfg80211_remove_links(struct wireless_dev *wdev);
.timeout_reason = NL80211_TIMEOUT_UNSPECIFIED,
.req_ie = data->req_ies,
.req_ie_len = data->req_ies_len,
- .resp_ie = mgmt->u.assoc_resp.variable,
- .resp_ie_len = data->len -
- offsetof(struct ieee80211_mgmt,
- u.assoc_resp.variable),
- .status = le16_to_cpu(mgmt->u.assoc_resp.status_code),
.ap_mld_addr = data->ap_mld_addr,
.assoc_encrypted = data->assoc_encrypted,
};
unsigned int link_id;
+ bool is_s1g = false;
for (link_id = 0; link_id < ARRAY_SIZE(data->links); link_id++) {
cr.links[link_id].status = data->links[link_id].status;
if (cr.links[link_id].bss->channel->band == NL80211_BAND_S1GHZ) {
WARN_ON(link_id);
- cr.resp_ie = (u8 *)&mgmt->u.s1g_assoc_resp.variable;
- cr.resp_ie_len = data->len -
- offsetof(struct ieee80211_mgmt,
- u.s1g_assoc_resp.variable);
+ is_s1g = true;
}
if (cr.ap_mld_addr)
cr.valid_links |= BIT(link_id);
}
+ if (is_s1g) {
+ if (data->len < offsetof(struct ieee80211_mgmt,
+ u.s1g_assoc_resp.variable))
+ goto free_bss;
+ cr.resp_ie = (u8 *)&mgmt->u.s1g_assoc_resp.variable;
+ cr.resp_ie_len = data->len -
+ offsetof(struct ieee80211_mgmt,
+ u.s1g_assoc_resp.variable);
+ } else {
+ if (data->len < offsetof(struct ieee80211_mgmt,
+ u.assoc_resp.variable))
+ goto free_bss;
+ cr.resp_ie = mgmt->u.assoc_resp.variable;
+ cr.resp_ie_len = data->len -
+ offsetof(struct ieee80211_mgmt,
+ u.assoc_resp.variable);
+ }
+ cr.status = le16_to_cpu(mgmt->u.assoc_resp.status_code);
+
trace_cfg80211_send_rx_assoc(dev, data);
/*
* and got a reject -- we only try again with an assoc
* frame instead of reassoc.
*/
- if (cfg80211_sme_rx_assoc_resp(wdev, cr.status)) {
- for (link_id = 0; link_id < ARRAY_SIZE(data->links); link_id++) {
- struct cfg80211_bss *bss = data->links[link_id].bss;
-
- if (!bss)
- continue;
-
- cfg80211_unhold_bss(bss_from_pub(bss));
- cfg80211_put_bss(wiphy, bss);
- }
- return;
- }
+ if (cfg80211_sme_rx_assoc_resp(wdev, cr.status))
+ goto free_bss;
nl80211_send_rx_assoc(rdev, dev, data);
/* update current_bss etc., consumes the bss reference */
__cfg80211_connect_result(dev, &cr, cr.status == WLAN_STATUS_SUCCESS);
+ return;
+
+free_bss:
+ for (link_id = 0; link_id < ARRAY_SIZE(data->links); link_id++) {
+ struct cfg80211_bss *bss = data->links[link_id].bss;
+
+ if (!bss)
+ continue;
+
+ cfg80211_unhold_bss(bss_from_pub(bss));
+ cfg80211_put_bss(wiphy, bss);
+ }
}
EXPORT_SYMBOL(cfg80211_rx_assoc_resp);
{
struct wireless_dev *wdev = dev->ieee80211_ptr;
struct ieee80211_mgmt *mgmt = (void *)buf;
+ __le16 fc;
lockdep_assert_wiphy(wdev->wiphy);
- trace_cfg80211_rx_mlme_mgmt(dev, buf, len);
+ if (len < sizeof(fc))
+ return;
+
+ fc = mgmt->frame_control;
- if (WARN_ON(len < 2))
+ if (ieee80211_is_auth(fc)) {
+ if (len < offsetofend(struct ieee80211_mgmt, u.auth.status_code))
+ return;
+ } else if (ieee80211_is_deauth(fc)) {
+ if (len < offsetofend(struct ieee80211_mgmt, u.deauth.reason_code))
+ return;
+ } else if (ieee80211_is_disassoc(fc)) {
+ if (len < offsetofend(struct ieee80211_mgmt, u.disassoc.reason_code))
+ return;
+ } else {
return;
+ }
+
+ trace_cfg80211_rx_mlme_mgmt(dev, buf, len);
- if (ieee80211_is_auth(mgmt->frame_control))
+ if (ieee80211_is_auth(fc))
cfg80211_process_auth(wdev, buf, len);
- else if (ieee80211_is_deauth(mgmt->frame_control))
+ else if (ieee80211_is_deauth(fc))
cfg80211_process_deauth(wdev, buf, len, false);
- else if (ieee80211_is_disassoc(mgmt->frame_control))
+ else
cfg80211_process_disassoc(wdev, buf, len, false);
}
EXPORT_SYMBOL(cfg80211_rx_mlme_mgmt);
{
struct wireless_dev *wdev = dev->ieee80211_ptr;
struct ieee80211_mgmt *mgmt = (void *)buf;
+ __le16 fc;
lockdep_assert_wiphy(wdev->wiphy);
- trace_cfg80211_tx_mlme_mgmt(dev, buf, len, reconnect);
+ if (len < sizeof(fc))
+ return;
- if (WARN_ON(len < 2))
+ fc = mgmt->frame_control;
+
+ if (ieee80211_is_deauth(fc)) {
+ if (len < offsetofend(struct ieee80211_mgmt, u.deauth.reason_code))
+ return;
+ } else if (ieee80211_is_disassoc(fc)) {
+ if (len < offsetofend(struct ieee80211_mgmt, u.disassoc.reason_code))
+ return;
+ } else {
return;
+ }
+
+ trace_cfg80211_tx_mlme_mgmt(dev, buf, len, reconnect);
- if (ieee80211_is_deauth(mgmt->frame_control))
+ if (ieee80211_is_deauth(fc))
cfg80211_process_deauth(wdev, buf, len, reconnect);
else
cfg80211_process_disassoc(wdev, buf, len, reconnect);
static const struct nla_policy
nl80211_pmsr_ftm_req_attr_policy[NL80211_PMSR_FTM_REQ_ATTR_MAX + 1] = {
[NL80211_PMSR_FTM_REQ_ATTR_ASAP] = { .type = NLA_FLAG },
- [NL80211_PMSR_FTM_REQ_ATTR_PREAMBLE] = { .type = NLA_U32 },
+ [NL80211_PMSR_FTM_REQ_ATTR_PREAMBLE] =
+ NLA_POLICY_RANGE(NLA_U32, NL80211_PREAMBLE_LEGACY,
+ NL80211_PREAMBLE_HE),
[NL80211_PMSR_FTM_REQ_ATTR_NUM_BURSTS_EXP] =
NLA_POLICY_MAX(NLA_U8, 15),
[NL80211_PMSR_FTM_REQ_ATTR_BURST_PERIOD] = { .type = NLA_U16 },
[NL80211_MBSSID_CONFIG_ATTR_TX_IFINDEX] = { .type = NLA_U32 },
[NL80211_MBSSID_CONFIG_ATTR_EMA] = { .type = NLA_FLAG },
[NL80211_MBSSID_CONFIG_ATTR_TX_LINK_ID] =
- NLA_POLICY_MAX(NLA_U8, IEEE80211_MLD_MAX_NUM_LINKS),
+ NLA_POLICY_RANGE(NLA_U8, 0, IEEE80211_MLD_MAX_NUM_LINKS - 1),
};
static const struct nla_policy
}
static struct cfg80211_mbssid_elems *
-nl80211_parse_mbssid_elems(struct wiphy *wiphy, struct nlattr *attrs)
+nl80211_parse_mbssid_elems(struct wiphy *wiphy, struct nlattr *attrs,
+ struct netlink_ext_ack *extack)
{
struct nlattr *nl_elems;
struct cfg80211_mbssid_elems *elems;
return ERR_PTR(-EINVAL);
nla_for_each_nested(nl_elems, attrs, rem_elems) {
+ int ret;
+
+ ret = validate_ie_attr(nl_elems, extack);
+ if (ret)
+ return ERR_PTR(ret);
+
if (num_elems >= 255)
return ERR_PTR(-EINVAL);
num_elems++;
if (attrs[NL80211_ATTR_MBSSID_ELEMS]) {
struct cfg80211_mbssid_elems *mbssid =
nl80211_parse_mbssid_elems(&rdev->wiphy,
- attrs[NL80211_ATTR_MBSSID_ELEMS]);
+ attrs[NL80211_ATTR_MBSSID_ELEMS],
+ extack);
if (IS_ERR(mbssid))
return PTR_ERR(mbssid);
if (IS_ERR(rnr))
return PTR_ERR(rnr);
- if (rnr && rnr->cnt < bcn->mbssid_ies->cnt)
+ if (rnr && rnr->cnt < bcn->mbssid_ies->cnt) {
+ kfree(rnr);
return -EINVAL;
+ }
bcn->rnr_ies = rnr;
}
wdev->nl_owner_dead = true;
schedule_work(&rdev->destroy_work);
} else if (wdev->conn_owner_nlportid == notify->portid) {
- schedule_work(&wdev->disconnect_wk);
+ wiphy_work_queue(wdev->wiphy,
+ &wdev->disconnect_wk);
}
cfg80211_release_pmsr(wdev, notify->portid);
NL_SET_ERR_MSG_ATTR(info->extack,
tb[NL80211_PMSR_FTM_REQ_ATTR_REQUEST_LCI],
"FTM: LCI request not supported");
+ return -EOPNOTSUPP;
}
out->ftm.request_civicloc =
NL_SET_ERR_MSG_ATTR(info->extack,
tb[NL80211_PMSR_FTM_REQ_ATTR_REQUEST_CIVICLOC],
"FTM: civic location request not supported");
+ return -EOPNOTSUPP;
}
out->ftm.trigger_based =
{
struct nlattr *tb[NL80211_PMSR_PEER_ATTR_MAX + 1];
struct nlattr *req[NL80211_PMSR_REQ_ATTR_MAX + 1];
+ bool have_measurement_type = false;
struct nlattr *treq;
int err, rem;
}
nla_for_each_nested(treq, req[NL80211_PMSR_REQ_ATTR_DATA], rem) {
+ if (have_measurement_type) {
+ NL_SET_ERR_MSG_ATTR(info->extack, treq,
+ "multiple measurement types in request data");
+ return -EINVAL;
+ }
+
+ have_measurement_type = true;
+
switch (nla_type(treq)) {
case NL80211_PMSR_TYPE_FTM:
err = pmsr_parse_ftm(rdev, treq, out, info);
"unsupported measurement type");
err = -EINVAL;
}
+ if (err)
+ return err;
}
- if (err)
- return err;
+ if (!have_measurement_type) {
+ NL_SET_ERR_MSG_ATTR(info->extack,
+ req[NL80211_PMSR_REQ_ATTR_DATA],
+ "missing measurement type in request data");
+ return -EINVAL;
+ }
return 0;
}
}
}
+ if (!count) {
+ NL_SET_ERR_MSG_ATTR(info->extack, peers, "No peers specified");
+ return -EINVAL;
+ }
+
req = kzalloc_flex(*req, peers, count);
if (!req)
return -ENOMEM;
}
}
-void cfg80211_pmsr_free_wk(struct work_struct *work)
+void cfg80211_pmsr_free_wk(struct wiphy *wiphy, struct wiphy_work *work)
{
struct wireless_dev *wdev = container_of(work, struct wireless_dev,
pmsr_free_wk);
- guard(wiphy)(wdev->wiphy);
-
cfg80211_pmsr_process_abort(wdev);
}
}
spin_unlock_bh(&wdev->pmsr_lock);
- cancel_work_sync(&wdev->pmsr_free_wk);
+ wiphy_work_cancel(wdev->wiphy, &wdev->pmsr_free_wk);
if (found)
cfg80211_pmsr_process_abort(wdev);
list_for_each_entry(req, &wdev->pmsr_list, list) {
if (req->nl_portid == portid) {
req->nl_portid = 0;
- schedule_work(&wdev->pmsr_free_wk);
+ wiphy_work_queue(wdev->wiphy, &wdev->pmsr_free_wk);
}
}
spin_unlock_bh(&wdev->pmsr_lock);
return true;
if (elem->id == WLAN_EID_EXTENSION) {
- if (!ext_id_len)
+ if (!ext_id_len || !elem->datalen)
return true;
loop_len = ext_id_len;
list = &non_inherit_elem->data[3 + id_len];
/* For ML probe response, match the MLE in the frame body with
* MLD id being 'bssid_index'
*/
- if (parent->id == WLAN_EID_EXTENSION && parent->datalen > 1 &&
+ if (parent->id == WLAN_EID_EXTENSION &&
parent->data[0] == WLAN_EID_EXT_EHT_MULTI_LINK &&
+ ieee80211_mle_type_ok(parent->data + 1,
+ IEEE80211_ML_CONTROL_TYPE_BASIC,
+ parent->datalen - 1) &&
bssid_index == ieee80211_mle_get_mld_id(parent->data + 1)) {
if (!cfg80211_copy_elem_with_frags(parent,
ie, ielen,
bssid = ext->u.s1g_beacon.sa;
capability = le16_to_cpu(compat->compat_info);
beacon_interval = le16_to_cpu(compat->beacon_int);
+ tsf = le32_to_cpu(ext->u.s1g_beacon.timestamp);
+ tsf |= (u64)le32_to_cpu(compat->tsf_completion) << 32;
} else {
bssid = mgmt->bssid;
beacon_interval = le16_to_cpu(mgmt->u.probe_resp.beacon_int);
capability = le16_to_cpu(mgmt->u.probe_resp.capab_info);
+ tsf = le64_to_cpu(mgmt->u.probe_resp.timestamp);
}
- tsf = le64_to_cpu(mgmt->u.probe_resp.timestamp);
-
if (ieee80211_is_probe_resp(mgmt->frame_control))
ftype = CFG80211_BSS_FTYPE_PRESP;
else if (ext)
/* translate "Scan for SSID" request */
if (wreq) {
if (wrqu->data.flags & IW_SCAN_THIS_ESSID) {
- if (wreq->essid_len > IEEE80211_MAX_SSID_LEN)
- return -EINVAL;
+ if (wreq->essid_len > IEEE80211_MAX_SSID_LEN) {
+ err = -EINVAL;
+ goto out;
+ }
memcpy(creq->req.ssids[0].ssid, wreq->essid,
wreq->essid_len);
creq->req.ssids[0].ssid_len = wreq->essid_len;
* Used to clean up after the connection / connection attempt owner socket
* disconnects
*/
-void cfg80211_autodisconnect_wk(struct work_struct *work)
+void cfg80211_autodisconnect_wk(struct wiphy *wiphy, struct wiphy_work *work)
{
struct wireless_dev *wdev =
container_of(work, struct wireless_dev, disconnect_wk);
- struct cfg80211_registered_device *rdev = wiphy_to_rdev(wdev->wiphy);
-
- guard(wiphy)(wdev->wiphy);
+ struct cfg80211_registered_device *rdev = wiphy_to_rdev(wiphy);
if (wdev->conn_owner_nlportid) {
switch (wdev->iftype) {
err = x->type_offload->xmit(x, skb, esp_features);
if (err) {
if (err == -EINPROGRESS)
- return NULL;
+ return ERR_PTR(-EINPROGRESS);
XFRM_INC_STATS(xs_net(x), LINUX_MIB_XFRMOUTSTATEPROTOERROR);
kfree_skb(skb);
pskb = skb2;
}
- return skb;
+ /* skb_gso_segment() set skb->prev to the last segment, but async
+ * crypto may have stolen it above without updating ->prev. Repoint
+ * it at the last retained segment so validate_xmit_skb_list() does
+ * not chain onto a segment now owned by the crypto engine.
+ */
+ if (skb)
+ skb->prev = pskb;
+
+ return skb ? skb : ERR_PTR(-EINPROGRESS);
}
EXPORT_SYMBOL_GPL(validate_xmit_xfrm);
}
xso->dev = dev;
+ xso->ifindex = dev->ifindex;
netdev_tracker_alloc(dev, &xso->dev_tracker, GFP_ATOMIC);
if (xuo->flags & XFRM_OFFLOAD_INBOUND)
}
__skb_frag_ref(tofrag);
shinfo->nr_frags++;
+ shinfo->flags |= SKBFL_SHARED_FRAG;
/* see if we are done */
fraglen = tofrag->len;
ka->encap_sport, sock_net_uid(net, NULL));
rt = ip_route_output_key(net, &fl4);
- if (IS_ERR(rt))
+ if (IS_ERR(rt)) {
+ kfree_skb(skb);
return PTR_ERR(rt);
+ }
skb_dst_set(skb, &rt->dst);
dst = ip6_dst_lookup_flow(net, sk, &fl6, NULL);
if (IS_ERR(dst)) {
local_unlock_nested_bh(&nat_keepalive_sk_ipv6.bh_lock);
+ kfree_skb(skb);
return PTR_ERR(dst);
}
sizeof(struct ipv6hdr)) +
sizeof(struct udphdr);
const u8 nat_ka_payload = 0xFF;
- int err = -EAFNOSUPPORT;
struct sk_buff *skb;
struct udphdr *uh;
switch (ka->family) {
case AF_INET:
- err = nat_keepalive_send_ipv4(skb, ka);
+ nat_keepalive_send_ipv4(skb, ka);
break;
#if IS_ENABLED(CONFIG_IPV6)
case AF_INET6:
- err = nat_keepalive_send_ipv6(skb, ka, uh);
+ nat_keepalive_send_ipv6(skb, ka, uh);
break;
#endif
- }
- if (err)
+ default:
kfree_skb(skb);
+ break;
+ }
}
struct nat_keepalive_work_ctx {
}
}
- if (policy->selector.prefixlen_d < dbits ||
- policy->selector.prefixlen_s < sbits)
+ if (policy->selector.prefixlen_d >= dbits &&
+ policy->selector.prefixlen_s >= sbits)
continue;
bin = xfrm_policy_inexact_alloc_bin(policy, dir);
xso->type = XFRM_DEV_OFFLOAD_PACKET;
xso->dir = xdo->dir;
xso->dev = dev;
+ xso->ifindex = dev->ifindex;
xso->flags = XFRM_DEV_OFFLOAD_FLAG_ACQ;
netdev_hold(dev, &xso->dev_tracker, GFP_ATOMIC);
error = dev->xfrmdev_ops->xdo_dev_state_add(dev, x,
x->mode_cbs = orig->mode_cbs;
if (x->mode_cbs && x->mode_cbs->clone_state) {
- if (x->mode_cbs->clone_state(x, orig))
+ if (x->mode_cbs->clone_state(x, orig)) {
+ if (!x->mode_data)
+ x->mode_cbs = NULL;
goto error;
+ }
}
x->props.reqid = m->new_reqid;
if (sockptr_is_null(optval) && !optlen) {
xfrm_sk_policy_insert(sk, XFRM_POLICY_IN, NULL);
xfrm_sk_policy_insert(sk, XFRM_POLICY_OUT, NULL);
- __sk_dst_reset(sk);
+ sk_dst_reset(sk);
return 0;
}
if (err >= 0) {
xfrm_sk_policy_insert(sk, err, pol);
xfrm_pol_put(pol);
- __sk_dst_reset(sk);
+ sk_dst_reset(sk);
err = 0;
}
if (x->mode_cbs->init_state)
err = x->mode_cbs->init_state(x);
module_put(x->mode_cbs->owner);
+ if (err && !x->mode_data)
+ x->mode_cbs = NULL;
}
error:
return err;
return 0;
}
-static void xso_to_xuo(const struct xfrm_dev_offload *xso,
- struct xfrm_user_offload *xuo)
+static void xso_to_xuo_ifindex(const struct xfrm_dev_offload *xso, int ifindex,
+ struct xfrm_user_offload *xuo)
{
- xuo->ifindex = xso->dev->ifindex;
+ xuo->ifindex = ifindex;
if (xso->dir == XFRM_DEV_OFFLOAD_IN)
xuo->flags = XFRM_OFFLOAD_INBOUND;
if (xso->type == XFRM_DEV_OFFLOAD_PACKET)
xuo->flags |= XFRM_OFFLOAD_PACKET;
}
-static int copy_user_offload(struct xfrm_dev_offload *xso, struct sk_buff *skb)
+#ifdef CONFIG_XFRM_MIGRATE
+static void xso_to_xuo(const struct xfrm_dev_offload *xso,
+ struct xfrm_user_offload *xuo)
+{
+ xso_to_xuo_ifindex(xso, xso->dev->ifindex, xuo);
+}
+#endif
+
+static int copy_user_offload_ifindex(const struct xfrm_dev_offload *xso,
+ int ifindex, struct sk_buff *skb)
{
struct xfrm_user_offload *xuo;
struct nlattr *attr;
xuo = nla_data(attr);
memset(xuo, 0, sizeof(*xuo));
- xso_to_xuo(xso, xuo);
+ xso_to_xuo_ifindex(xso, ifindex, xuo);
return 0;
}
+static int copy_user_offload(struct xfrm_dev_offload *xso, struct sk_buff *skb)
+{
+ return copy_user_offload_ifindex(xso, xso->dev->ifindex, skb);
+}
+
+static int copy_user_state_offload(const struct xfrm_dev_offload *xso,
+ struct sk_buff *skb)
+{
+ return copy_user_offload_ifindex(xso, READ_ONCE(xso->ifindex), skb);
+}
+
static bool xfrm_redact(void)
{
return IS_ENABLED(CONFIG_SECURITY) &&
&x->replay);
if (ret)
goto out;
- if(x->xso.dev)
- ret = copy_user_offload(&x->xso, skb);
+ if (READ_ONCE(x->xso.dev))
+ ret = copy_user_state_offload(&x->xso, skb);
if (ret)
goto out;
if (x->if_id) {
switch (ut[i].mode) {
case XFRM_MODE_TUNNEL:
case XFRM_MODE_BEET:
+ case XFRM_MODE_IPTFS:
if (ut[i].optional && dir == XFRM_POLICY_OUT) {
NL_SET_ERR_MSG(extack, "Mode in optional template not allowed in outbound policy");
return -EINVAL;
}
break;
- case XFRM_MODE_IPTFS:
- break;
default:
if (ut[i].family != prev_family) {
NL_SET_ERR_MSG(extack, "Mode in template doesn't support a family change");
l += nla_total_size(sizeof(*x->coaddr));
if (x->props.extra_flags)
l += nla_total_size(sizeof(x->props.extra_flags));
- if (x->xso.dev)
- l += nla_total_size(sizeof(struct xfrm_user_offload));
+ if (READ_ONCE(x->xso.dev))
+ l += nla_total_size(sizeof(struct xfrm_user_offload));
if (x->props.smark.v | x->props.smark.m) {
l += nla_total_size(sizeof(x->props.smark.v));
l += nla_total_size(sizeof(x->props.smark.m));
addr.end = promote ? node1_end_addr : node0_end_addr;
}
+ if (addr.start >= addr.end)
+ goto free_out;
+
range.start = addr.start;
range.end = addr.end;
ops_destroy(ops_irrelevant, nr_ops_irrelevant);
/*
- * The benchmark completed sucessfully, but there's no reason to keep
- * the module around. Return an error do the user doesn't have to
+ * The benchmark completed successfully, but there's no reason to keep
+ * the module around. Return an error so the user doesn't have to
* manually unload the module.
*/
return -EINVAL;
access_mask_t access_request;
int ret = 0;
+ if ((msg->msg_flags & MSG_FASTOPEN) && address && sk_is_tcp(sock->sk)) {
+ ret = current_check_access_socket(
+ sock, address, addrlen, LANDLOCK_ACCESS_NET_CONNECT_TCP,
+ true);
+ if (ret != 0)
+ return ret;
+ }
+
if (sk_is_udp(sock->sk))
access_request = LANDLOCK_ACCESS_NET_CONNECT_SEND_UDP;
else
*/
struct {
/**
- * @quiet: Suppresses denial logs for the object covered by this
- * rule in this domain. For filesystem rules, this inherits
+ * @flags.quiet: Suppresses denial logs for the object covered by
+ * this rule in this domain. For filesystem rules, this inherits
* down the file hierarchy.
*/
u8 quiet : 1;
if (!parent_subject)
return 0;
- scoped_guard(rcu)
- {
+ scoped_guard(rcu) {
const struct landlock_ruleset *const child_dom =
landlock_get_task_domain(child);
err = domain_ptrace(parent_subject->domain, child_dom);
if (!subject)
return 0;
- scoped_guard(rcu)
- {
+ scoped_guard(rcu) {
is_scoped = domain_is_scoped(subject->domain,
landlock_get_task_domain(p),
signal_scope.scope);
if (task_tgid(tsk) == landlock_file(fown->file)->fown_tg)
return 0;
- scoped_guard(rcu)
- {
+ scoped_guard(rcu) {
is_scoped = domain_is_scoped(subject->domain,
landlock_get_task_domain(tsk),
signal_scope.scope);
static int default_noexec __ro_after_init;
-static int __file_map_prot_check(const struct cred *cred,
- const struct file *file, unsigned long prot,
- bool shared, bool bf_user_file)
+static int __file_map_prot_check(const struct file *file, unsigned long prot,
+ bool shared, bool mounter_check,
+ bool bf_user_file)
{
struct inode *inode = NULL;
bool prot_exec = prot & PROT_EXEC;
inode = file_inode(file);
}
- if (default_noexec && prot_exec &&
+ if (!mounter_check && default_noexec && prot_exec &&
(!file || IS_PRIVATE(inode) || (!shared && prot_write))) {
int rc;
- u32 sid = cred_sid(cred);
+ u32 sid = current_sid();
/*
* We are making executable an anonymous mapping or a private
}
if (file) {
+ const struct cred *cred = mounter_check ?
+ file->f_cred : current_cred();
/* "read" always possible, "write" only if shared */
u32 av = FILE__READ;
if (shared && prot_write)
return 0;
}
-static inline int file_map_prot_check(const struct cred *cred,
- const struct file *file,
- unsigned long prot, bool shared)
+static inline int file_map_prot_check(const struct file *file,
+ unsigned long prot, bool shared,
+ bool mounter_check)
{
- return __file_map_prot_check(cred, file, prot, shared, false);
+ return __file_map_prot_check(file, prot, shared, mounter_check, false);
}
static int selinux_mmap_addr(unsigned long addr)
return rc;
}
-static int selinux_mmap_file_common(const struct cred *cred, struct file *file,
- unsigned long prot, bool shared)
+static int selinux_mmap_file_common(struct file *file, unsigned long prot,
+ bool shared, bool mounter_check)
{
if (file) {
int rc;
struct common_audit_data ad;
+ const struct cred *cred = mounter_check ?
+ file->f_cred : current_cred();
ad.type = LSM_AUDIT_DATA_FILE;
ad.u.file = file;
return rc;
}
- return file_map_prot_check(cred, file, prot, shared);
+ return file_map_prot_check(file, prot, shared, mounter_check);
}
static int selinux_mmap_file(struct file *file,
unsigned long reqprot __always_unused,
unsigned long prot, unsigned long flags)
{
- return selinux_mmap_file_common(current_cred(), file, prot,
- (flags & MAP_TYPE) == MAP_SHARED);
+ return selinux_mmap_file_common(file, prot,
+ (flags & MAP_TYPE) == MAP_SHARED,
+ false);
}
/**
if (vma->vm_flags & VM_EXEC)
prot |= PROT_EXEC;
- return selinux_mmap_file_common(backing_file->f_cred, backing_file,
- prot, vma->vm_flags & VM_SHARED);
+ return selinux_mmap_file_common(backing_file, prot,
+ vma->vm_flags & VM_SHARED,
+ true);
}
static int selinux_file_mprotect(struct vm_area_struct *vma,
}
}
- rc = __file_map_prot_check(cred, file, prot, shared, backing_file);
+ rc = __file_map_prot_check(file, prot, shared, false, backing_file);
if (rc)
return rc;
if (backing_file) {
- rc = file_map_prot_check(file->f_cred, file, prot, shared);
+ rc = file_map_prot_check(file, prot, shared, true);
if (rc)
return rc;
}
Need to determine whether we should perform a name_bind
permission check between the socket and the port number. */
-static int selinux_socket_bind(struct socket *sock, struct sockaddr *address, int addrlen)
+static int __selinux_socket_bind(struct sock *sk, struct sockaddr *address, int addrlen)
{
- struct sock *sk = sock->sk;
struct sk_security_struct *sksec = selinux_sock(sk);
u16 family;
int err;
return -EAFNOSUPPORT;
}
+static int selinux_socket_bind(struct socket *sock, struct sockaddr *address, int addrlen)
+{
+ return __selinux_socket_bind(sock->sk, address, addrlen);
+}
+
/* This supports connect(2) and SCTP connect services such as sctp_connectx(3)
* and sctp_sendmsg(3) as described in Documentation/security/SCTP.rst
*/
-static int selinux_socket_connect_helper(struct socket *sock,
+static int selinux_socket_connect_helper(struct sock *sk,
struct sockaddr *address, int addrlen)
{
- struct sock *sk = sock->sk;
struct sk_security_struct *sksec = selinux_sock(sk);
int err;
int err;
struct sock *sk = sock->sk;
- err = selinux_socket_connect_helper(sock, address, addrlen);
+ err = selinux_socket_connect_helper(sk, address, addrlen);
if (err)
return err;
static int selinux_socket_sendmsg(struct socket *sock, struct msghdr *msg,
int size)
{
- return sock_has_perm(sock->sk, SOCKET__WRITE);
+ int rc;
+ struct sockaddr *const addr = msg->msg_name;
+ const int addrlen = msg->msg_namelen;
+
+ rc = sock_has_perm(sock->sk, SOCKET__WRITE);
+ if (rc)
+ return rc;
+
+ if (addr && (msg->msg_flags & MSG_FASTOPEN) &&
+ (sk_is_tcp(sock->sk) ||
+ (sk_is_inet(sock->sk) && sock->sk->sk_type == SOCK_STREAM &&
+ sock->sk->sk_protocol == IPPROTO_MPTCP))) {
+ rc = selinux_socket_connect(sock, addr, addrlen);
+ if (rc)
+ return rc;
+ }
+
+ return 0;
}
static int selinux_socket_recvmsg(struct socket *sock, struct msghdr *msg,
int len, err = 0, walk_size = 0;
void *addr_buf;
struct sockaddr *addr;
- struct socket *sock;
if (!selinux_policycap_extsockclass())
return 0;
/* Process one or more addresses that may be IPv4 or IPv6 */
- sock = sk->sk_socket;
addr_buf = address;
while (walk_size < addrlen) {
case SCTP_PRIMARY_ADDR:
case SCTP_SET_PEER_PRIMARY_ADDR:
case SCTP_SOCKOPT_BINDX_ADD:
- err = selinux_socket_bind(sock, addr, len);
+ err = __selinux_socket_bind(sk, addr, len);
break;
/* Connect checks */
case SCTP_SOCKOPT_CONNECTX:
case SCTP_PARAM_SET_PRIMARY:
case SCTP_PARAM_ADD_IP:
case SCTP_SENDMSG_CONNECT:
- err = selinux_socket_connect_helper(sock, addr, len);
+ err = selinux_socket_connect_helper(sk, addr, len);
if (err)
return err;
#include <linux/firewire.h>
#include <linux/firewire-constants.h>
#include <linux/module.h>
-#include <linux/mod_devicetable.h>
#include <linux/delay.h>
#include <linux/slab.h>
#include <linux/sched/signal.h>
#include <linux/firewire-constants.h>
#include <linux/jiffies.h>
#include <linux/module.h>
-#include <linux/mod_devicetable.h>
#include <linux/mutex.h>
#include <linux/slab.h>
#include <linux/spinlock.h>
#include <linux/device.h>
#include <linux/firewire.h>
#include <linux/module.h>
-#include <linux/mod_devicetable.h>
#include <linux/delay.h>
#include <linux/slab.h>
#include <linux/sched/signal.h>
#include <linux/firewire.h>
#include <linux/firewire-constants.h>
#include <linux/module.h>
-#include <linux/mod_devicetable.h>
#include <linux/mutex.h>
#include <linux/slab.h>
#include <linux/compat.h>
#include <linux/firewire.h>
#include <linux/firewire-constants.h>
#include <linux/module.h>
-#include <linux/mod_devicetable.h>
#include <linux/delay.h>
#include <linux/slab.h>
#include <linux/sched/signal.h>
#include <linux/firewire.h>
#include <linux/firewire-constants.h>
#include <linux/module.h>
-#include <linux/mod_devicetable.h>
#include <linux/mutex.h>
#include <linux/slab.h>
#include <linux/compat.h>
#include <linux/firewire.h>
#include <linux/firewire-constants.h>
#include <linux/module.h>
-#include <linux/mod_devicetable.h>
#include <linux/mutex.h>
#include <linux/slab.h>
#include <linux/compat.h>
#include <linux/firewire.h>
#include <linux/firewire-constants.h>
#include <linux/module.h>
-#include <linux/mod_devicetable.h>
#include <linux/mutex.h>
#include <linux/slab.h>
#include <linux/compat.h>
{
unsigned int mic_present;
- /* fix some headset type recognize fail issue, such as EDIFIER headset */
- /* set micbias output current comparator threshold from 66% to 55%. */
- snd_hda_codec_write(codec, 0x1c, 0, 0x320, 0x010);
/* set OFF voltage for DFET from -1.2V to -0.8V, set headset micbias register
* value adjustment trim from 2.2K ohms to 2.0K ohms.
*/
ALC275_FIXUP_DELL_XPS,
ALC293_FIXUP_LENOVO_SPK_NOISE,
ALC233_FIXUP_LENOVO_LINE2_MIC_HOTKEY,
+ ALC233_FIXUP_WUJIE_SPEAKERS,
ALC233_FIXUP_LENOVO_L2MH_LOW_ENLED,
ALC255_FIXUP_DELL_SPK_NOISE,
ALC225_FIXUP_DISABLE_MIC_VREF,
ALC236_FIXUP_DELL_DUAL_CODECS,
ALC287_FIXUP_CS35L41_I2C_2_THINKPAD_ACPI,
ALC287_FIXUP_TAS2781_I2C,
+ ALC287_FIXUP_ASUS_ALLY_X,
+ ALC287_FIXUP_ASUS_ALLY_X_SPEAKER,
+ ALC287_FIXUP_ASUS_ALLY_X_I2C,
ALC295_FIXUP_DELL_TAS2781_I2C,
ALC245_FIXUP_TAS2781_SPI_2,
ALC287_FIXUP_TXNW2781_I2C,
ALC256_FIXUP_HONOR_MRB_XXX_M1020_AUDIO,
ALC245_FIXUP_HP_ENVY_X360_15_FH0XXX,
ALC287_FIXUP_ACER_MICMUTE_LED,
+ ALC236_FIXUP_DELL_HP_POP_NOISE,
};
/* A special fixup for Lenovo C940 and Yoga Duet 7;
}
static const struct hda_fixup alc269_fixups[] = {
+ [ALC233_FIXUP_WUJIE_SPEAKERS] = {
+ .type = HDA_FIXUP_PINS,
+ .v.pins = (const struct hda_pintbl[]) {
+ { 0x1b, 0x90170150 }, /* internal speaker */
+ { }
+ },
+ },
[ALC269_FIXUP_GPIO2] = {
.type = HDA_FIXUP_FUNC,
.v.func = alc_fixup_gpio2,
.chained = true,
.chain_id = ALC285_FIXUP_THINKPAD_HEADSET_JACK,
},
+ [ALC287_FIXUP_ASUS_ALLY_X] = {
+ .type = HDA_FIXUP_PINS,
+ .v.pins = (const struct hda_pintbl[]) {
+ { 0x19, 0x03a11050 }, /* headset mic */
+ { }
+ },
+ .chained = true,
+ .chain_id = ALC287_FIXUP_ASUS_ALLY_X_SPEAKER,
+ },
+ [ALC287_FIXUP_ASUS_ALLY_X_SPEAKER] = {
+ .type = HDA_FIXUP_FUNC,
+ .v.func = alc285_fixup_speaker2_to_dac1,
+ .chained = true,
+ .chain_id = ALC287_FIXUP_ASUS_ALLY_X_I2C,
+ },
+ [ALC287_FIXUP_ASUS_ALLY_X_I2C] = {
+ .type = HDA_FIXUP_FUNC,
+ .v.func = tas2781_fixup_tias_i2c,
+ .chained = true,
+ .chain_id = ALC225_FIXUP_HEADSET_JACK,
+ },
[ALC245_FIXUP_TAS2781_SPI_2] = {
.type = HDA_FIXUP_FUNC,
.v.func = tas2781_fixup_spi,
.chained = true,
.chain_id = ALC2XX_FIXUP_HEADSET_MIC,
},
+ [ALC236_FIXUP_DELL_HP_POP_NOISE] = {
+ .type = HDA_FIXUP_FUNC,
+ .v.func = alc285_fixup_invalidate_dacs,
+ },
};
static const struct hda_quirk alc269_fixup_tbl[] = {
SND_PCI_QUIRK(0x1028, 0x0cc3, "Dell Oasis 14 Low Weight MTL-U", ALC289_FIXUP_DELL_CS35L41_SPI_2),
SND_PCI_QUIRK(0x1028, 0x0cc4, "Dell Oasis 16 MTL-H/U", ALC289_FIXUP_DELL_CS35L41_SPI_2),
SND_PCI_QUIRK(0x1028, 0x0cc5, "Dell Oasis 14", ALC289_FIXUP_RTK_AMP_DUAL_SPK),
+ SND_PCI_QUIRK(0x1028, 0x0e6b, "Dell Pro QCM1255", ALC236_FIXUP_DELL_HP_POP_NOISE),
+ SND_PCI_QUIRK(0x1028, 0x0e6d, "Dell Pro Micro QCM1255", ALC236_FIXUP_DELL_HP_POP_NOISE),
SND_PCI_QUIRK(0x1028, 0x164a, "Dell", ALC293_FIXUP_DELL1_MIC_NO_PRESENCE),
SND_PCI_QUIRK(0x1028, 0x164b, "Dell", ALC293_FIXUP_DELL1_MIC_NO_PRESENCE),
SND_PCI_QUIRK(0x103c, 0x1586, "HP", ALC269_FIXUP_HP_MUTE_LED_MIC2),
SND_PCI_QUIRK(0x103c, 0x856a, "HP Pavilion 15-cs1xxx", ALC295_FIXUP_HP_PAVILION_MUTE_LED_1B),
SND_PCI_QUIRK(0x103c, 0x85c6, "HP Pavilion x360 Convertible 14-dy1xxx", ALC295_FIXUP_HP_MUTE_LED_COEFBIT11),
SND_PCI_QUIRK(0x103c, 0x85de, "HP Envy x360 13-ar0xxx", ALC285_FIXUP_HP_ENVY_X360),
+ SND_PCI_QUIRK(0x103c, 0x85f0, "HP Laptop 15-dw0xxx", ALC236_FIXUP_HP_MUTE_LED_COEFBIT2),
SND_PCI_QUIRK(0x103c, 0x8603, "HP Omen 17-cb0xxx", ALC285_FIXUP_HP_MUTE_LED),
SND_PCI_QUIRK(0x103c, 0x860c, "HP ZBook 17 G6", ALC285_FIXUP_HP_GPIO_AMP_INIT),
SND_PCI_QUIRK(0x103c, 0x860f, "HP ZBook 15 G6", ALC285_FIXUP_HP_GPIO_AMP_INIT),
SND_PCI_QUIRK(0x103c, 0x8a36, "HP Pavilion Plus 14-eh0xxx", ALC245_FIXUP_HP_MUTE_LED_COEFBIT),
SND_PCI_QUIRK(0x103c, 0x8a3d, "HP Victus 15-fb0xxx (MB 8A3D)", ALC245_FIXUP_HP_MUTE_LED_V2_COEFBIT),
SND_PCI_QUIRK(0x103c, 0x8a4f, "HP Victus 15-fa0xxx (MB 8A4F)", ALC245_FIXUP_HP_MUTE_LED_COEFBIT),
+ SND_PCI_QUIRK(0x103c, 0x8a50, "HP Victus 15-fa0xxx (MB 8A50)", ALC245_FIXUP_HP_MUTE_LED_COEFBIT),
SND_PCI_QUIRK(0x103c, 0x8a6e, "HP EDNA 360", ALC287_FIXUP_CS35L41_I2C_4),
SND_PCI_QUIRK(0x103c, 0x8a74, "HP ProBook 440 G8 Notebook PC", ALC236_FIXUP_HP_GPIO_LED),
SND_PCI_QUIRK(0x103c, 0x8a75, "HP ProBook 450 G8 Notebook PC", ALC236_FIXUP_HP_GPIO_LED),
SND_PCI_QUIRK(0x103c, 0x8da8, "HP 16 Piston OmniBook X", ALC245_FIXUP_HP_ENVY_X360_15_FH0XXX),
SND_PCI_QUIRK(0x103c, 0x8dc9, "HP Laptop 15-fc0xxx", ALC236_FIXUP_HP_DMIC),
SND_PCI_QUIRK(0x103c, 0x8dd4, "HP EliteStudio 8 AIO", ALC274_FIXUP_HP_AIO_BIND_DACS),
- SND_PCI_QUIRK(0x103c, 0x8dd7, "HP Laptop 15-fd0xxx", ALC236_FIXUP_HP_MUTE_LED_COEFBIT2),
+ SND_PCI_QUIRK(0x103c, 0x8dd7, "HP Laptop 15-fd0xxx", ALC236_FIXUP_HP_MUTE_LED_MICMUTE_GPIO),
SND_PCI_QUIRK(0x103c, 0x8de8, "HP Gemtree", ALC245_FIXUP_TAS2781_SPI_2),
SND_PCI_QUIRK(0x103c, 0x8de9, "HP Gemtree", ALC245_FIXUP_TAS2781_SPI_2),
SND_PCI_QUIRK(0x103c, 0x8dec, "HP EliteBook 640 G12", ALC236_FIXUP_HP_GPIO_LED),
SND_PCI_QUIRK(0x1043, 0x1e83, "ASUS GA605W", ALC285_FIXUP_ASUS_GU605_SPI_SPEAKER2_TO_DAC1),
SND_PCI_QUIRK(0x1043, 0x1e8e, "ASUS Zephyrus G15", ALC289_FIXUP_ASUS_GA401),
SND_PCI_QUIRK(0x1043, 0x1e93, "ASUS ExpertBook B9403CVAR", ALC294_FIXUP_ASUS_HPE),
- SND_PCI_QUIRK(0x1043, 0x1eb3, "ASUS Ally RCLA72", ALC287_FIXUP_TAS2781_I2C),
+ SND_PCI_QUIRK(0x1043, 0x1eb3, "ASUS Ally RC72LA", ALC287_FIXUP_ASUS_ALLY_X),
SND_PCI_QUIRK(0x1043, 0x1ed3, "ASUS HN7306W", ALC287_FIXUP_CS35L41_I2C_2),
HDA_CODEC_QUIRK(0x1043, 0x1ee2, "ASUS UM6702RA/RC", ALC285_FIXUP_ASUS_I2C_SPEAKER2_TO_DAC1),
SND_PCI_QUIRK(0x1043, 0x1ee2, "ASUS UM6702RA/RC", ALC287_FIXUP_CS35L41_I2C_2),
HDA_CODEC_QUIRK(0x17aa, 0x386e, "Legion Y9000X 2022 IAH7", ALC287_FIXUP_CS35L41_I2C_2),
SND_PCI_QUIRK(0x17aa, 0x386e, "Yoga Pro 7 14ARP8", ALC285_FIXUP_SPEAKER2_TO_DAC1),
HDA_CODEC_QUIRK(0x17aa, 0x38a8, "Legion Pro 7 16ARX8H", ALC287_FIXUP_TAS2781_I2C), /* this must match before PCI SSID 17aa:386f below */
+ HDA_CODEC_QUIRK(0x17aa, 0x38a7, "Legion Pro 7 16ARX8H", ALC287_FIXUP_TAS2781_I2C), /* this must match before PCI SSID 17aa:386f below */
SND_PCI_QUIRK(0x17aa, 0x386f, "Legion Pro 7i 16IAX7", ALC287_FIXUP_CS35L41_I2C_2),
SND_PCI_QUIRK(0x17aa, 0x3870, "Lenovo Yoga 7 14ARB7", ALC287_FIXUP_YOGA7_14ARB7_I2C),
SND_PCI_QUIRK(0x17aa, 0x3874, "Legion 7i 16IAX7", ALC287_FIXUP_CS35L41_I2C_2),
SND_PCI_QUIRK(0x1d05, 0x300f, "TongFang X6AR5xxY", ALC2XX_FIXUP_HEADSET_MIC),
SND_PCI_QUIRK(0x1d05, 0x3019, "TongFang X6FR5xxY", ALC2XX_FIXUP_HEADSET_MIC),
SND_PCI_QUIRK(0x1d05, 0x3031, "TongFang X6AR55xU", ALC2XX_FIXUP_HEADSET_MIC),
+ SND_PCI_QUIRK(0x1d05, 0x3034, "TongFang X6xx45xU", ALC2XX_FIXUP_HEADSET_MIC),
SND_PCI_QUIRK(0x1d17, 0x3288, "Haier Boyue G42", ALC269VC_FIXUP_ACER_VCOPPERBOX_PINS),
SND_PCI_QUIRK(0x1d72, 0x1602, "RedmiBook", ALC255_FIXUP_XIAOMI_HEADSET_MIC),
SND_PCI_QUIRK(0x1d72, 0x1701, "XiaomiNotebook Pro", ALC298_FIXUP_DELL1_MIC_NO_PRESENCE),
SND_PCI_QUIRK(0x8086, 0x2080, "Intel NUC 8 Rugged", ALC256_FIXUP_INTEL_NUC8_RUGGED),
SND_PCI_QUIRK(0x8086, 0x2081, "Intel NUC 10", ALC256_FIXUP_INTEL_NUC10),
SND_PCI_QUIRK(0x8086, 0x3038, "Intel NUC 13", ALC295_FIXUP_CHROME_BOOK),
+ SND_PCI_QUIRK(0xc011, 0x1d05, "MECHREVO WUJIE Series", ALC233_FIXUP_WUJIE_SPEAKERS),
SND_PCI_QUIRK(0xf111, 0x0001, "Framework Laptop", ALC295_FIXUP_FRAMEWORK_LAPTOP_MIC_NO_PRESENCE),
SND_PCI_QUIRK(0xf111, 0x0006, "Framework Laptop", ALC295_FIXUP_FRAMEWORK_LAPTOP_MIC_NO_PRESENCE),
SND_PCI_QUIRK(0xf111, 0x0009, "Framework Laptop", ALC295_FIXUP_FRAMEWORK_LAPTOP_MIC_NO_PRESENCE),
SND_PCI_QUIRK(0xf111, 0x000c, "Framework Laptop", ALC295_FIXUP_FRAMEWORK_LAPTOP_MIC_NO_PRESENCE),
SND_PCI_QUIRK(0xf111, 0x000f, "Framework Laptop 13 Pro PTL", ALC295_FIXUP_FRAMEWORK_LAPTOP_LIMIT_INT_MIC_BOOST),
SND_PCI_QUIRK(0xf111, 0x010f, "Framework Laptop 13 PTL", ALC295_FIXUP_FRAMEWORK_LAPTOP_LIMIT_INT_MIC_BOOST),
+ SND_PCI_QUIRK(0xf111, 0x0010, "Framework Laptop", ALC295_FIXUP_FRAMEWORK_LAPTOP_LIMIT_INT_MIC_BOOST),
#if 0
/* Below is a quirk table taken from the old code.
# SPDX-License-Identifier: GPL-2.0-only
config SND_HDA_CIRRUS_SCODEC
- tristate
+ tristate "Cirrus side-codec library" if KUNIT
config SND_HDA_CIRRUS_SCODEC_KUNIT_TEST
tristate "KUnit test for Cirrus side-codec library" if !KUNIT_ALL_TESTS
NULL, "bin");
return;
}
-
- /*
- * Check for system-specific bin files without wmfw before
- * falling back to generic firmware
- */
- if (amp_name)
- cs35l56_hda_request_firmware_file(cs35l56, coeff_firmware, coeff_filename,
- base_name, system_name, amp_name, "bin");
- if (!*coeff_firmware)
- cs35l56_hda_request_firmware_file(cs35l56, coeff_firmware, coeff_filename,
- base_name, system_name, NULL, "bin");
-
- if (*coeff_firmware)
- return;
}
ret = cs35l56_hda_request_firmware_file(cs35l56, wmfw_firmware, wmfw_filename,
&wmfw_firmware, &wmfw_filename,
&coeff_firmware, &coeff_filename);
- /*
- * If the BIOS didn't patch the firmware a bin file is mandatory to
- * enable the ASP·
- */
- if (!coeff_firmware && firmware_missing) {
- dev_err(cs35l56->base.dev, ".bin file required but not found\n");
- goto err_fw_release;
+ /* If the BIOS didn't patch the firmware a wmfw and bin file are mandatory */
+ if (firmware_missing) {
+ if (!wmfw_firmware) {
+ dev_err(cs35l56->base.dev, ".%s file required but not found\n", "wmfw");
+ goto err_fw_release;
+ } else if (!coeff_firmware) {
+ dev_err(cs35l56->base.dev, ".%s file required but not found\n", "bin");
+ goto err_fw_release;
+ }
}
mutex_lock(&cs35l56->base.irq_lock);
err = snd_hdac_exec_verb(codec, verb, 0, NULL);
if (err < 0)
return err;
- verb = (reg & ~0xfffff) | (AC_VERB_GET_COEF_INDEX << 8);
+ verb = (reg & ~0xfffff) | (AC_VERB_GET_PROC_COEF << 8);
return snd_hdac_exec_verb(codec, verb, 0, val);
}
err = snd_hdac_exec_verb(codec, verb, 0, NULL);
if (err < 0)
return err;
- verb = (reg & ~0xfffff) | (AC_VERB_GET_COEF_INDEX << 8) |
+ verb = (reg & ~0xfffff) | (AC_VERB_SET_PROC_COEF << 8) |
(val & 0xffff);
return snd_hdac_exec_verb(codec, verb, 0, NULL);
}
config SND_SOC_ACPI_AMD_MATCH
tristate
- select SND_SOC_ACPI_AMD_SDCA_QUIRKS if SND_SOC_SDCA
select SND_SOC_ACPI if ACPI
config SND_SOC_ACPI_AMD_SDCA_QUIRKS
- tristate
- depends on ACPI
+ bool "AMD ACPI SDCA quirks"
+ depends on SND_SOC_ACPI_AMD_MATCH
depends on SND_SOC_SDCA
+ depends on SND_SOC_ACPI_AMD_MATCH = m || SND_SOC_SDCA = y
+ default y
+ help
+ Enable SDCA quirk support for AMD ACPI match tables.
+ This is compiled into the snd-soc-acpi-amd-match module.
if SND_SOC_AMD_ACP_COMMON
snd-acp-legacy-mach-y := acp-legacy-mach.o acp3x-es83xx/acp3x-es83xx.o
snd-acp-sof-mach-y := acp-sof-mach.o
snd-soc-acpi-amd-match-y := amd-acp63-acpi-match.o amd-acp70-acpi-match.o
+snd-soc-acpi-amd-match-$(CONFIG_SND_SOC_ACPI_AMD_SDCA_QUIRKS) += soc-acpi-amd-sdca-quirks.o
snd-acp-sdw-mach-y := acp-sdw-mach-common.o
snd-acp-sdw-sof-mach-y += acp-sdw-sof-mach.o
snd-acp-sdw-legacy-mach-y += acp-sdw-legacy-mach.o
-snd-soc-acpi-amd-sdca-quirks-y += soc-acpi-amd-sdca-quirks.o
obj-$(CONFIG_SND_SOC_AMD_ACP_PCM) += snd-acp-pcm.o
obj-$(CONFIG_SND_SOC_AMD_ACP_I2S) += snd-acp-i2s.o
obj-$(CONFIG_SND_AMD_ASOC_ACP63) += snd-acp63.o
obj-$(CONFIG_SND_AMD_ASOC_ACP70) += snd-acp70.o
-obj-$(CONFIG_SND_SOC_ACPI_AMD_SDCA_QUIRKS) += snd-soc-acpi-amd-sdca-quirks.o
obj-$(CONFIG_SND_AMD_SOUNDWIRE_ACPI) += snd-amd-sdw-acpi.o
obj-$(CONFIG_SND_SOC_AMD_MACH_COMMON) += snd-acp-mach.o
obj-$(CONFIG_SND_SOC_AMD_LEGACY_MACH) += snd-acp-legacy-mach.o
{}
};
+static const struct snd_soc_acpi_adr_device tas2783_2_adr[] = {
+ {
+ /* left */
+ .adr = 0x00003c0102000001ull,
+ .num_endpoints = 1,
+ .endpoints = &spk_l_endpoint,
+ .name_prefix = "tas2783-1",
+ },
+ {
+ /* right */
+ .adr = 0x00003d0102000001ull,
+ .num_endpoints = 1,
+ .endpoints = &spk_l_endpoint,
+ .name_prefix = "tas2783-2",
+ },
+ {
+ /* left */
+ .adr = 0x0000390102000001ull,
+ .num_endpoints = 1,
+ .endpoints = &spk_r_endpoint,
+ .name_prefix = "tas2783-3",
+ },
+ {
+ /* right */
+ .adr = 0x00003a0102000001ull,
+ .num_endpoints = 1,
+ .endpoints = &spk_r_endpoint,
+ .name_prefix = "tas2783-4",
+ },
+};
+
+static const struct snd_soc_acpi_link_adr acp70_tas2783_2[] = {
+ {
+ .mask = BIT(0),
+ .num_adr = ARRAY_SIZE(tas2783_2_adr),
+ .adr_d = tas2783_2_adr,
+ },
+ {}
+};
+
static const struct snd_soc_acpi_adr_device rt1320_0_single_adr[] = {
{
.adr = 0x000030025D132001ull,
};
struct snd_soc_acpi_mach snd_soc_acpi_amd_acp70_sdw_machines[] = {
+ {
+ .link_mask = BIT(0),
+ .links = acp70_tas2783_2,
+ .drv_name = "amd_sdw",
+ },
{
.link_mask = BIT(0) | BIT(1),
.links = acp70_rt1320_l0_rt722_l1,
MODULE_LICENSE("GPL");
MODULE_AUTHOR("Vijendar.Mukunda@amd.com");
#if IS_ENABLED(CONFIG_SND_SOC_ACPI_AMD_SDCA_QUIRKS)
-MODULE_IMPORT_NS("SND_SOC_ACPI_AMD_SDCA_QUIRKS");
+MODULE_IMPORT_NS("SND_SOC_SDCA");
#endif
return false;
}
-EXPORT_SYMBOL_NS(snd_soc_acpi_amd_sdca_is_device_rt712_vb, "SND_SOC_ACPI_AMD_SDCA_QUIRKS");
-MODULE_DESCRIPTION("ASoC ACPI AMD SDCA quirks");
-MODULE_LICENSE("GPL");
-MODULE_IMPORT_NS("SND_SOC_SDCA");
if (sdw_dma_irq_flag)
return IRQ_WAKE_THREAD;
- if (irq_flag | wake_irq_flag)
+ if (irq_flag || wake_irq_flag)
return IRQ_HANDLED;
else
return IRQ_NONE;
return -ENODEV;
}
- ret = pci_request_regions(pci, "AMD ACP6.2 audio");
+ ret = pci_request_regions(pci, "AMD ACP6.3 audio");
if (ret < 0) {
dev_err(&pci->dev, "pci_request_regions failed\n");
goto disable_pci;
return acp_hw_runtime_resume(dev);
}
+static void acp_disable_msi_on_resume(struct pci_dev *pdev)
+{
+ u16 control;
+
+ if (!pdev->msi_cap)
+ return;
+
+ pci_read_config_word(pdev, pdev->msi_cap + PCI_MSI_FLAGS, &control);
+ if (control & PCI_MSI_FLAGS_ENABLE) {
+ dev_warn(&pdev->dev,
+ "ACP: MSI unexpectedly enabled after resume (flags=0x%04x), disabling\n",
+ control);
+ control &= ~PCI_MSI_FLAGS_ENABLE;
+ pci_write_config_word(pdev, pdev->msi_cap + PCI_MSI_FLAGS, control);
+ }
+}
+
static int snd_acp_resume(struct device *dev)
{
+ struct pci_dev *pdev = to_pci_dev(dev);
+
+ /*
+ * BIOS/firmware may re-enable MSI in PCI config space during
+ * system resume even though this driver only uses legacy INTx
+ * interrupts. If MSI is left enabled with stale address/data
+ * registers, the device will write interrupts to a bogus address
+ * causing IOMMU IO_PAGE_FAULT and interrupt delivery failure.
+ * Explicitly clear the MSI Enable bit before reinitializing
+ * the ACP hardware.
+ */
+ acp_disable_msi_on_resume(pdev);
return acp_hw_resume(dev);
}
#define CS42L43_JACK_ABSENT 0x0
#define CS42L43_JACK_OPTICAL (SND_JACK_MECHANICAL | SND_JACK_AVOUT)
+#define CS42L43_JACK_MICROPHONE (SND_JACK_MECHANICAL | SND_JACK_MICROPHONE)
#define CS42L43_JACK_HEADPHONE (SND_JACK_MECHANICAL | SND_JACK_HEADPHONE)
#define CS42L43_JACK_HEADSET (SND_JACK_MECHANICAL | SND_JACK_HEADSET)
#define CS42L43_JACK_LINEOUT (SND_JACK_MECHANICAL | SND_JACK_LINEOUT)
.hsdet_mode = CS42L43_JACK_3_POLE_SWITCHES,
.mic_ctrl = (0x3 << CS42L43_JACK_STEREO_CONFIG_SHIFT) |
CS42L43_HS1_BIAS_EN_MASK | CS42L43_HS2_BIAS_EN_MASK,
- .report = CS42L43_JACK_LINEIN,
+ .report = CS42L43_JACK_MICROPHONE,
},
[CS42L43_JACK_RAW_OPTICAL] = {
.hsdet_mode = CS42L43_JACK_3_POLE_SWITCHES,
}
}
+static void rt712_sdca_reset(struct rt712_sdca_priv *rt712)
+{
+ rt712_sdca_index_update_bits(rt712, RT712_VENDOR_REG,
+ RT712_PARA_VERB_CTL, RT712_HIDDEN_REG_SW_RESET,
+ RT712_HIDDEN_REG_SW_RESET);
+ rt712_sdca_index_update_bits(rt712, RT712_VENDOR_HDA_CTL,
+ RT712_HDA_LEGACY_RESET_CTL, 0x1, 0x1);
+}
+
int rt712_sdca_io_init(struct device *dev, struct sdw_slave *slave)
{
struct rt712_sdca_priv *rt712 = dev_get_drvdata(dev);
pm_runtime_get_noresume(&slave->dev);
+ rt712_sdca_reset(rt712);
+
rt712_sdca_index_read(rt712, RT712_VENDOR_REG, RT712_JD_PRODUCT_NUM, &val);
rt712->hw_id = (val & 0xf000) >> 12;
rt712->version_id = (val & 0x0f00) >> 8;
if (tas2562->sdz_gpio == NULL) {
tas2562->sdz_gpio = devm_gpiod_get_optional(dev, "shut-down",
GPIOD_OUT_HIGH);
- if (IS_ERR(tas2562->sdz_gpio))
+ if (IS_ERR(tas2562->sdz_gpio)) {
if (PTR_ERR(tas2562->sdz_gpio) == -EPROBE_DEFER)
return -EPROBE_DEFER;
- tas2562->sdz_gpio = NULL;
+ tas2562->sdz_gpio = NULL;
+ }
}
if (tas2562->model_id == TAS2110)
SND_PCI_QUIRK(0x17aa, 0x2348, "Lenovo P16", SOC_SDW_CODEC_MIC),
SND_PCI_QUIRK(0x17aa, 0x2349, "Lenovo P1", SOC_SDW_CODEC_MIC),
SND_PCI_QUIRK(0x17aa, 0x3821, "Lenovo 0x3821", SOC_SDW_SIDECAR_AMPS),
+ SND_PCI_QUIRK(0x17aa, 0x383c, "Lenovo 0x383c", SOC_SDW_SIDECAR_AMPS),
{}
};
#define AIU_MEM_IEC958_CONTROL_MODE_16BIT BIT(7)
#define AIU_MEM_IEC958_CONTROL_MODE_LINEAR BIT(8)
#define AIU_MEM_IEC958_BUF_CNTL_INIT BIT(0)
+#define AIU_RST_SOFT_958_FAST BIT(2)
#define AIU_FIFO_SPDIF_BLOCK 8
case SNDRV_PCM_TRIGGER_START:
case SNDRV_PCM_TRIGGER_RESUME:
case SNDRV_PCM_TRIGGER_PAUSE_RELEASE:
+ snd_soc_component_write(component, AIU_RST_SOFT,
+ AIU_RST_SOFT_958_FAST);
fifo_spdif_dcu_enable(component, true);
break;
case SNDRV_PCM_TRIGGER_SUSPEND:
case SNDRV_PCM_TRIGGER_PAUSE_PUSH:
case SNDRV_PCM_TRIGGER_STOP:
+ snd_soc_component_write(component, AIU_RST_SOFT,
+ AIU_RST_SOFT_958_FAST);
fifo_spdif_dcu_enable(component, false);
break;
default:
}
static const struct of_device_id snd_sc8280xp_dt_match[] = {
+ {.compatible = "qcom,eliza-sndcard", "eliza"},
{.compatible = "qcom,kaanapali-sndcard", "kaanapali"},
{.compatible = "qcom,qcm6490-idp-sndcard", "qcm6490"},
{.compatible = "qcom,qcs615-sndcard", "qcs615"},
struct device *dev = &urb->dev->dev;
struct snd_usb_caiaqdev *cdev = urb->context;
unsigned char *buf = urb->transfer_buffer;
+ unsigned int payload_len;
+ unsigned int copy_len;
if (urb->status || !cdev) {
dev_warn(dev, "received EP1 urb->status = %i\n", urb->status);
return;
}
+ if (urb->actual_length < 1)
+ return;
+
+ payload_len = urb->actual_length - 1;
switch(buf[0]) {
case EP1_CMD_GET_DEVICE_INFO:
+ if (payload_len < sizeof(struct caiaq_device_spec))
+ break;
memcpy(&cdev->spec, buf+1, sizeof(struct caiaq_device_spec));
cdev->spec.fw_version = le16_to_cpu(cdev->spec.fw_version);
dev_dbg(dev, "device spec (firmware %d): audio: %d in, %d out, "
wake_up(&cdev->ep1_wait_queue);
break;
case EP1_CMD_AUDIO_PARAMS:
+ if (payload_len < 1)
+ break;
cdev->audio_parm_answer = buf[1];
wake_up(&cdev->ep1_wait_queue);
break;
case EP1_CMD_MIDI_READ:
+ if (urb->actual_length < 3 || urb->actual_length - 3 < buf[2])
+ break;
snd_usb_caiaq_midi_handle_input(cdev, buf[1], buf + 3, buf[2]);
break;
case EP1_CMD_READ_IO:
if (cdev->chip.usb_id ==
USB_ID(USB_VID_NATIVEINSTRUMENTS, USB_PID_AUDIO8DJ)) {
- if (urb->actual_length > sizeof(cdev->control_state))
- urb->actual_length = sizeof(cdev->control_state);
- memcpy(cdev->control_state, buf + 1, urb->actual_length);
+ copy_len = min_t(unsigned int, payload_len, sizeof(cdev->control_state));
+ memcpy(cdev->control_state, buf + 1, copy_len);
wake_up(&cdev->ep1_wait_queue);
break;
}
switch (cdev->chip.usb_id) {
case USB_ID(USB_VID_NATIVEINSTRUMENTS, USB_PID_RIGKONTROL2):
+ if (len < 6)
+ return;
snd_caiaq_input_report_abs(cdev, ABS_X, buf, 2);
snd_caiaq_input_report_abs(cdev, ABS_Y, buf, 0);
snd_caiaq_input_report_abs(cdev, ABS_Z, buf, 1);
case USB_ID(USB_VID_NATIVEINSTRUMENTS, USB_PID_RIGKONTROL3):
case USB_ID(USB_VID_NATIVEINSTRUMENTS, USB_PID_KORECONTROLLER):
case USB_ID(USB_VID_NATIVEINSTRUMENTS, USB_PID_KORECONTROLLER2):
+ if (len < 6)
+ return;
snd_caiaq_input_report_abs(cdev, ABS_X, buf, 0);
snd_caiaq_input_report_abs(cdev, ABS_Y, buf, 1);
snd_caiaq_input_report_abs(cdev, ABS_Z, buf, 2);
break;
case USB_ID(USB_VID_NATIVEINSTRUMENTS, USB_PID_TRAKTORKONTROLX1):
+ if (len < 16)
+ return;
snd_caiaq_input_report_abs(cdev, ABS_HAT0X, buf, 4);
snd_caiaq_input_report_abs(cdev, ABS_HAT0Y, buf, 2);
snd_caiaq_input_report_abs(cdev, ABS_HAT1X, buf, 6);
if (!cval->cmask) {
snd_usb_set_cur_mix_value(cval, 0, 0, cval->max);
} else {
+ idx = 0;
for (i = 0; i < MAX_CHANNELS; i++) {
- idx = 0;
if (cval->cmask & BIT(i)) {
snd_usb_set_cur_mix_value(cval, i + 1, idx, cval->max);
idx++;
QUIRK_FLAG_FORCE_IFACE_RESET | QUIRK_FLAG_IFACE_DELAY),
DEVICE_FLG(0x03f0, 0x654a, /* HP 320 FHD Webcam */
QUIRK_FLAG_GET_SAMPLE_RATE | QUIRK_FLAG_MIC_RES_16),
+ DEVICE_FLG(0x040b, 0x0897, /* Weltrend Semiconductor, sold as Redragon H510-PRO Wireless headset */
+ QUIRK_FLAG_MIXER_GET_CUR_BROKEN),
DEVICE_FLG(0x041e, 0x3000, /* Creative SB Extigy */
QUIRK_FLAG_IGNORE_CTL_ERROR),
DEVICE_FLG(0x041e, 0x4080, /* Creative Live Cam VF0610 */
QUIRK_FLAG_DSD_RAW),
DEVICE_FLG(0x2708, 0x0002, /* Audient iD14 */
QUIRK_FLAG_IGNORE_CTL_ERROR),
+ DEVICE_FLG(0x2772, 0x0502, /* Musical Fidelity M6s DAC */
+ 0), /* for avoiding QUIRK_FLAG_DSD_RAW with vendor match */
DEVICE_FLG(0x2912, 0x30c8, /* Audioengine D1 */
QUIRK_FLAG_GET_SAMPLE_RATE),
DEVICE_FLG(0x2a70, 0x1881, /* OnePlus Technology (Shenzhen) Co., Ltd. BE02T */
QUIRK_FLAG_MIXER_GET_CUR_BROKEN),
DEVICE_FLG(0x2fc6, 0xf06b, /* MOONDROP Moonriver2 Ti */
QUIRK_FLAG_CTL_MSG_DELAY),
+ DEVICE_FLG(0x2fc6, 0xf0b5, /* iBasso DC-Elite */
+ QUIRK_FLAG_CTL_MSG_DELAY_1M),
DEVICE_FLG(0x2fc6, 0xf0b7, /* iBasso DC07 Pro */
QUIRK_FLAG_CTL_MSG_DELAY_1M),
DEVICE_FLG(0x30be, 0x0101, /* Schiit Hel */
/* SPDX-License-Identifier: GPL-2.0 OR MIT */
+#include <stdint.h>
#ifndef __LINUX_OVERFLOW_H
#define __LINUX_OVERFLOW_H
if (a->base != b->base || a->nr_cids != b->nr_cids)
return false;
- nr_words = CMASK_NR_WORDS(a->nr_cids);
+ if (a->nr_cids == 0)
+ return true;
+ nr_words = (a->base + a->nr_cids - 1) / 64 - a->base / 64 + 1;
bpf_for(i, 0, CMASK_MAX_WORDS) {
if (i >= nr_words)
return true;
}
-/*
- * True iff every bit set in @a is also set in @b over the intersection of
- * their ranges. Bits of @a outside @b's range fail the test.
- */
-static __always_inline bool cmask_subset(const struct scx_cmask __arena *a,
- const struct scx_cmask __arena *b)
-{
- u32 a_end = a->base + a->nr_cids;
- u32 b_end = b->base + b->nr_cids;
- u32 a_wbase = a->base / 64;
- u32 b_wbase = b->base / 64;
- u32 nr_words, i;
-
- /* any bit of @a outside @b's range is a subset violation */
- if (a->base < b->base || a_end > b_end)
- return false;
-
- nr_words = CMASK_NR_WORDS(a->nr_cids);
- bpf_for(i, 0, CMASK_MAX_WORDS) {
- u32 wi_b;
-
- if (i >= nr_words)
- break;
- wi_b = a_wbase + i - b_wbase;
- if (a->bits[i] & ~b->bits[wi_b])
- return false;
- }
- return true;
-}
-
/**
* cmask_next_set - find the first set bit at or after @cid
* @m: cmask to search
(cid) < (m)->base + (m)->nr_cids; \
(cid) = cmask_next_set((m), (cid) + 1))
+/*
+ * True iff every bit set in @a is also set in @b. Matches the kernel-side
+ * scx_cmask_subset(): ranges don't need to nest, and set bits of @a outside
+ * @b's range fail the test.
+ */
+static __always_inline bool cmask_subset(const struct scx_cmask __arena *a,
+ const struct scx_cmask __arena *b)
+{
+ u32 a_end = a->base + a->nr_cids;
+ u32 b_end = b->base + b->nr_cids;
+ u32 a_wbase = a->base / 64;
+ u32 b_wbase = b->base / 64;
+ u32 lo = a->base > b->base ? a->base : b->base;
+ u32 hi = a_end < b_end ? a_end : b_end;
+ u32 lo_word, hi_word, i;
+
+ /* set bits of @a outside @b's range can't be in @b */
+ if (a->base < b->base &&
+ cmask_next_set(a, a->base) < (b->base < a_end ? b->base : a_end))
+ return false;
+ if (a_end > b_end &&
+ cmask_next_set(a, a->base > b_end ? a->base : b_end) < a_end)
+ return false;
+
+ if (lo >= hi)
+ return true;
+
+ /*
+ * Walk the words the range intersection spans. Plain word tests
+ * suffice: the scans above guarantee @a has no set bit outside @b's
+ * range and padding bits are kept clear by all cmask helpers.
+ */
+ lo_word = lo / 64;
+ hi_word = (hi - 1) / 64;
+
+ bpf_for(i, 0, CMASK_MAX_WORDS) {
+ u32 w = lo_word + i;
+
+ if (w > hi_word)
+ break;
+ if (a->bits[w - a_wbase] & ~b->bits[w - b_wbase])
+ return false;
+ }
+ return true;
+}
+
/*
* Population count over [base, base + nr_cids). Padding bits in the head/tail
* words are guaranteed zero by the mutating helpers, so a flat popcount over
- * all words is correct.
+ * the words the range spans is correct.
*/
static __always_inline u32 cmask_weight(const struct scx_cmask __arena *m)
{
- u32 nr_words = CMASK_NR_WORDS(m->nr_cids), i;
+ u32 nr_words, i;
u32 count = 0;
+ if (!m->nr_cids)
+ return 0;
+ nr_words = (m->base + m->nr_cids - 1) / 64 - m->base / 64 + 1;
+
bpf_for(i, 0, CMASK_MAX_WORDS) {
if (i >= nr_words)
break;
if (err < 0) {
ksft_print_msg("Failed to get hctl for card %d: %s\n",
card, snd_strerror(err));
+ free(card_data);
goto next_card;
}
}
if (int64_val > snd_ctl_elem_info_get_max64(ctl->info)) {
- ksft_print_msg("%s.%d value %lld more than maximum %ld\n",
+ ksft_print_msg("%s.%d value %lld more than maximum %lld\n",
ctl->name, index, int64_val,
- snd_ctl_elem_info_get_max(ctl->info));
+ snd_ctl_elem_info_get_max64(ctl->info));
return false;
}
* We can access a GCS via ptrace
*
* This could usefully have a fixture but note that each test is
- * fork()ed into a new child whcih causes issues. Might be better to
+ * fork()ed into a new child which causes issues. Might be better to
* lift at least some of this out into a separate, non-harness, test
* program.
*/
/*
* fork() does not change keys. Only exec() does so call a worker program.
- * Its only job is to sign a value and report back the resutls
+ * Its only job is to sign a value and report back the results
*/
TEST(exec_changed_keys)
{
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+
+#include <test_progs.h>
+#include "kfunc_implicit_args_tracing.skel.h"
+
+void test_kfunc_implicit_args_tracing(void)
+{
+ struct kfunc_implicit_args_tracing *skel;
+ LIBBPF_OPTS(bpf_test_run_opts, topts);
+ int err, fd;
+
+ skel = kfunc_implicit_args_tracing__open_and_load();
+ if (!ASSERT_OK_PTR(skel, "open_and_load"))
+ return;
+
+ err = kfunc_implicit_args_tracing__attach(skel);
+ if (!ASSERT_OK(err, "attach"))
+ goto cleanup;
+
+ fd = bpf_program__fd(skel->progs.trigger_implicit_arg);
+ err = bpf_prog_test_run_opts(fd, &topts);
+ if (!ASSERT_OK(err, "test_run"))
+ goto cleanup;
+
+ ASSERT_EQ(topts.retval, 5, "kfunc_retval");
+ ASSERT_EQ(skel->bss->fentry_arg_cnt, 2, "fentry_arg_cnt");
+ ASSERT_NEQ(skel->bss->fentry_aux_arg, 0, "fentry_aux_arg");
+ ASSERT_EQ(skel->bss->fentry_result, 1, "fentry_result");
+ ASSERT_EQ(skel->bss->fexit_arg_cnt, 2, "fexit_arg_cnt");
+ ASSERT_NEQ(skel->bss->fexit_aux_arg, 0, "fexit_aux_arg");
+ ASSERT_EQ(skel->bss->fexit_result, 1, "fexit_result");
+
+cleanup:
+ kfunc_implicit_args_tracing__destroy(skel);
+}
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+
+#include <test_progs.h>
+#include "test_kmods/bpf_testmod.h"
+#include "bpf_util.h"
+
+static void check_attach_reject(const struct bpf_insn *program, size_t prog_len)
+{
+ LIBBPF_OPTS(bpf_prog_load_opts, opts);
+ char error[4096];
+ int bpf_fd, tp_fd;
+
+ opts.log_level = 2;
+ opts.log_buf = error;
+ opts.log_size = sizeof(error);
+
+ bpf_fd = bpf_prog_load(BPF_PROG_TYPE_RAW_TRACEPOINT_WRITABLE, NULL, "GPL v2",
+ program, prog_len, &opts);
+ if (!ASSERT_GE(bpf_fd, 0, "prog_load"))
+ return;
+
+ tp_fd = bpf_raw_tracepoint_open("bpf_testmod_test_writable_bare_tp", bpf_fd);
+ ASSERT_EQ(tp_fd, -EINVAL, "bpf_raw_tracepoint_open");
+ if (tp_fd >= 0)
+ close(tp_fd);
+
+ close(bpf_fd);
+}
+
+void test_raw_tp_writable_reject_bad_access(void)
+{
+ const struct bpf_insn program[] = {
+ /* r6 is our tp buffer */
+ BPF_LDX_MEM(BPF_DW, BPF_REG_6, BPF_REG_1, 0),
+ /* one byte beyond the end of the writable context */
+ BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_6,
+ sizeof(struct bpf_testmod_test_writable_ctx)),
+ BPF_EXIT_INSN(),
+ };
+
+ const struct bpf_insn negative_var_off_program[] = {
+ BPF_LDX_MEM(BPF_DW, BPF_REG_6, BPF_REG_1, 0),
+ /* make var_off negative, but keep the effective access offset non-negative */
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_6, -8),
+ /* one byte beyond the end of the writable context */
+ BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_6,
+ sizeof(struct bpf_testmod_test_writable_ctx) + 8),
+ BPF_EXIT_INSN(),
+ };
+
+ if (test__start_subtest("past_end"))
+ check_attach_reject(program, ARRAY_SIZE(program));
+
+ if (test__start_subtest("negative_var_off_past_end"))
+ check_attach_reject(negative_var_off_program,
+ ARRAY_SIZE(negative_var_off_program));
+}
+++ /dev/null
-// SPDX-License-Identifier: GPL-2.0
-
-#include <test_progs.h>
-#include <linux/nbd.h>
-#include "bpf_util.h"
-
-void test_raw_tp_writable_reject_nbd_invalid(void)
-{
- __u32 duration = 0;
- char error[4096];
- int bpf_fd = -1, tp_fd = -1;
-
- const struct bpf_insn program[] = {
- /* r6 is our tp buffer */
- BPF_LDX_MEM(BPF_DW, BPF_REG_6, BPF_REG_1, 0),
- /* one byte beyond the end of the nbd_request struct */
- BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_6,
- sizeof(struct nbd_request)),
- BPF_EXIT_INSN(),
- };
-
- LIBBPF_OPTS(bpf_prog_load_opts, opts,
- .log_level = 2,
- .log_buf = error,
- .log_size = sizeof(error),
- );
-
- bpf_fd = bpf_prog_load(BPF_PROG_TYPE_RAW_TRACEPOINT_WRITABLE, NULL, "GPL v2",
- program, ARRAY_SIZE(program),
- &opts);
- if (CHECK(bpf_fd < 0, "bpf_raw_tracepoint_writable load",
- "failed: %d errno %d\n", bpf_fd, errno))
- return;
-
- tp_fd = bpf_raw_tracepoint_open("nbd_send_request", bpf_fd);
- if (CHECK(tp_fd >= 0, "bpf_raw_tracepoint_writable open",
- "erroneously succeeded\n"))
- goto out_bpffd;
-
- close(tp_fd);
-out_bpffd:
- close(bpf_fd);
-}
return;
}
- udp = xsocket(AF_INET, SOCK_DGRAM | SOCK_NONBLOCK, 0);
+ udp = socket_loopback(AF_INET, SOCK_DGRAM | SOCK_NONBLOCK);
if (udp < 0) {
close(dgram);
close(tcp);
return;
}
- udp = xsocket(AF_INET, SOCK_DGRAM | SOCK_NONBLOCK, 0);
+ udp = socket_loopback(AF_INET, SOCK_DGRAM | SOCK_NONBLOCK);
if (udp < 0) {
close(dgram);
close(tcp);
return;
}
- udp = xsocket(AF_INET, SOCK_DGRAM | SOCK_NONBLOCK, 0);
+ udp = socket_loopback(AF_INET, SOCK_DGRAM | SOCK_NONBLOCK);
if (udp < 0) {
close(dgram);
close(tcp);
test_sockmap_pass_prog__destroy(skel);
}
+/* A socket in a sockmap without a verdict program keeps its ingress data
+ * in sk_receive_queue: FIONREAD must account for it.
+ */
+static void test_sockmap_no_verdict_fionread(void)
+{
+ int err, map, zero = 0, sent, avail;
+ int c0 = -1, c1 = -1, p0 = -1, p1 = -1;
+ struct test_sockmap_pass_prog *skel;
+ char buf[256] = "0123456789";
+
+ skel = test_sockmap_pass_prog__open_and_load();
+ if (!ASSERT_OK_PTR(skel, "open_and_load"))
+ return;
+ map = bpf_map__fd(skel->maps.sock_map_rx);
+
+ err = create_socket_pairs(AF_INET, SOCK_STREAM, &c0, &c1, &p0, &p1);
+ if (!ASSERT_OK(err, "create_socket_pairs()"))
+ goto out;
+
+ err = bpf_map_update_elem(map, &zero, &c1, BPF_NOEXIST);
+ if (!ASSERT_OK(err, "bpf_map_update_elem(c1)"))
+ goto out_close;
+
+ sent = xsend(p1, &buf, sizeof(buf), 0);
+ ASSERT_EQ(sent, sizeof(buf), "xsend(p1)");
+ avail = wait_for_fionread(c1, sizeof(buf), IO_TIMEOUT_SEC);
+ ASSERT_EQ(avail, sizeof(buf), "ioctl(FIONREAD)");
+
+out_close:
+ close(c0);
+ close(p0);
+ close(c1);
+ close(p1);
+out:
+ test_sockmap_pass_prog__destroy(skel);
+}
+
void test_sockmap_basic(void)
{
if (test__start_subtest("sockmap create_update_free"))
test_sockmap_skb_verdict_shutdown();
if (test__start_subtest("sockmap skb_verdict fionread"))
test_sockmap_skb_verdict_fionread(true);
+ if (test__start_subtest("sockmap no_verdict fionread"))
+ test_sockmap_no_verdict_fionread();
if (test__start_subtest("sockmap skb_verdict fionread on drop"))
test_sockmap_skb_verdict_fionread(false);
if (test__start_subtest("sockmap skb_verdict change tail"))
int family, int sotype, int mapfd)
{
u32 key = 0;
- u64 value;
int err, s;
+ u64 value;
s = xsocket(family, sotype, 0);
if (s == -1)
errno = 0;
value = s;
err = bpf_map_update_elem(mapfd, &key, &value, BPF_NOEXIST);
- if (sotype == SOCK_STREAM) {
- if (!err || errno != EOPNOTSUPP)
- FAIL_ERRNO("map_update: expected EOPNOTSUPP");
- } else if (err)
- FAIL_ERRNO("map_update: expected success");
+ ASSERT_ERR(err, "map_update");
+ ASSERT_EQ(errno, EOPNOTSUPP, "errno");
xclose(s);
}
struct sockaddr_storage addr;
socklen_t len = 0;
u32 key = 0;
- u64 value;
int err, s;
+ u64 value;
init_addr_loopback(family, &addr, &len);
errno = 0;
value = s;
err = bpf_map_update_elem(mapfd, &key, &value, BPF_NOEXIST);
- if (!err || errno != EOPNOTSUPP)
- FAIL_ERRNO("map_update: expected EOPNOTSUPP");
+ if (sotype == SOCK_STREAM) {
+ ASSERT_ERR(err, "map_update");
+ ASSERT_EQ(errno, EOPNOTSUPP, "errno");
+ } else {
+ ASSERT_OK(err, "map_update");
+ }
close:
xclose(s);
}
/* insert */
TEST(test_insert_invalid),
TEST(test_insert_opened),
- TEST(test_insert_bound, SOCK_STREAM),
+ TEST(test_insert_bound),
TEST(test_insert),
/* delete */
TEST(test_delete_after_insert),
#include "verifier_precision.skel.h"
#include "verifier_prevent_map_lookup.skel.h"
#include "verifier_private_stack.skel.h"
+#include "verifier_ptr_to_buf.skel.h"
#include "verifier_raw_stack.skel.h"
#include "verifier_raw_tp_writable.skel.h"
#include "verifier_reg_equal.skel.h"
void test_verifier_precision(void) { RUN(verifier_precision); }
void test_verifier_prevent_map_lookup(void) { RUN(verifier_prevent_map_lookup); }
void test_verifier_private_stack(void) { RUN(verifier_private_stack); }
+void test_verifier_ptr_to_buf(void) { RUN(verifier_ptr_to_buf); }
void test_verifier_raw_stack(void) { RUN(verifier_raw_stack); }
void test_verifier_raw_tp_writable(void) { RUN(verifier_raw_tp_writable); }
void test_verifier_reg_equal(void) { RUN(verifier_reg_equal); }
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_tracing.h>
+#include <errno.h>
+
+extern int bpf_kfunc_implicit_arg(int a) __weak __ksym;
+
+char _license[] SEC("license") = "GPL";
+
+/* Shared arg checks; reports arg count and aux, returns 1 on success. */
+static __always_inline __u64
+check_implicit_args(void *ctx, __u64 *arg_cnt, __u64 *aux_arg)
+{
+ __u64 a = 0, aux = 0, z = 0;
+ __u64 result;
+ __s64 err;
+
+ *arg_cnt = bpf_get_func_arg_cnt(ctx);
+ result = *arg_cnt == 2;
+
+ err = bpf_get_func_arg(ctx, 0, &a);
+ result &= err == 0 && (int)a == 5;
+
+ err = bpf_get_func_arg(ctx, 1, &aux);
+ *aux_arg = aux;
+ result &= err == 0 && aux != 0;
+
+ err = bpf_get_func_arg(ctx, 2, &z);
+ result &= err == -EINVAL;
+
+ return result;
+}
+
+__u64 fentry_result;
+__u64 fentry_arg_cnt;
+__u64 fentry_aux_arg;
+
+SEC("fentry/bpf_kfunc_implicit_arg")
+int BPF_PROG(trace_implicit_arg_fentry)
+{
+ __u64 ret = 0;
+ __s64 err;
+
+ fentry_result = check_implicit_args(ctx, &fentry_arg_cnt, &fentry_aux_arg);
+
+ err = bpf_get_func_ret(ctx, &ret);
+ fentry_result &= err == -EOPNOTSUPP;
+
+ return 0;
+}
+
+__u64 fexit_result;
+__u64 fexit_arg_cnt;
+__u64 fexit_aux_arg;
+
+SEC("fexit/bpf_kfunc_implicit_arg")
+int BPF_PROG(trace_implicit_arg_fexit)
+{
+ __u64 ret = 0;
+ __s64 err;
+
+ fexit_result = check_implicit_args(ctx, &fexit_arg_cnt, &fexit_aux_arg);
+
+ err = bpf_get_func_ret(ctx, &ret);
+ fexit_result &= err == 0 && ret == 5;
+
+ return 0;
+}
+
+SEC("syscall")
+int trigger_implicit_arg(void *ctx)
+{
+ return bpf_kfunc_implicit_arg(5);
+}
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+
+SEC("iter/bpf_map_elem")
+__description("PTR_TO_BUF: reject negative const offset")
+__failure
+__msg("invalid negative rdwr buffer offset")
+__naked void ptr_to_buf_reject_negative_const_offset(void)
+{
+ asm volatile ("r0 = 0; \
+ r2 = *(u64 *)(r1 + %[value_off]); \
+ if r2 == 0 goto l0_%=; \
+ r2 += -8; \
+ r0 = *(u64 *)(r2 + 0); \
+l0_%=: \
+ exit; \
+ "
+ :
+ : __imm_const(value_off,
+ offsetof(struct bpf_iter__bpf_map_elem, value))
+ : __clobber_all);
+}
+
+char _license[] SEC("license") = "GPL";
: __clobber_all);
}
+SEC("raw_tracepoint.w")
+__description("raw_tracepoint_writable: reject negative const offset")
+__failure
+__msg("invalid negative tracepoint buffer offset")
+__naked void tracepoint_writable_reject_negative_const_offset(void)
+{
+ asm volatile (" \
+ r6 = *(u64 *)(r1 + 0); \
+ r6 += -8; \
+ r0 = *(u64 *)(r6 + 0); \
+ exit; \
+" :
+ :
+ : __clobber_all);
+}
+
char _license[] SEC("license") = "GPL";
goto out_sockmap;
}
- /* Test update with unsupported UDP socket */
+ /* Test update with unsupported unbound UDP socket */
udp = socket(AF_INET, SOCK_DGRAM, 0);
- i = 0;
- err = bpf_map_update_elem(fd, &i, &udp, BPF_ANY);
- if (err) {
- printf("Failed socket update SOCK_DGRAM '%i:%i'\n",
- i, udp);
+ CHECK(udp < 0, "socket(AF_INET, SOCK_DGRAM)", "errno:%d\n", errno);
+ err = bpf_map_update_elem(fd, &(int){0}, &udp, BPF_ANY);
+ close(udp);
+ if (!err) {
+ printf("Unexpectedly succeeded unbound UDP update '0:%i'\n", udp);
goto out_sockmap;
}
- close(udp);
/* Test update without programs */
for (i = 0; i < 6; i++) {
# Restore MACs
ip netns exec "${NAMESPACE}" ip link set "${DSTIF}" \
address "${SAVED_DSTMAC}"
- if [ "${BINDMODE}" == "mac" ]; then
+ if [ "${BINDMODE}" == "mac" ] &&
+ [ "$(mac_get "${SRCIF}")" != "${SAVED_SRCMAC}" ]; then
ip link set dev "${SRCIF}" down
ip link set dev "${SRCIF}" address "${SAVED_SRCMAC}"
# Rename device in order to trigger target resume, as initial
FUNC1='foo_bar*'
FUNC2='vfs_read'
+:;: "Save enabled functions count" ;:
+ecount=`cat enabled_functions | wc -l`
+
+count_enabled_functions() {
+ count=`cat enabled_functions | wc -l`
+ count=$(($count-$ecount))
+ echo $count
+}
+
:;: "Add an event on the test module" ;:
echo "f:test1 $FUNC1" >> dynamic_events
echo 1 > events/fprobes/test1/enable
:;: "Ensure it is enabled" ;:
-funcs=`cat enabled_functions | wc -l`
+funcs=`count_enabled_functions`
test $funcs -ne 0
:;: "Check the enabled_functions is cleared on unloading" ;:
rmmod trace-events-sample
-funcs=`cat enabled_functions | wc -l`
+funcs=`count_enabled_functions`
test $funcs -eq 0
:;: "Check it is kept clean" ;:
modprobe trace-events-sample
echo 1 > events/fprobes/test1/enable || echo "OK"
-funcs=`cat enabled_functions | wc -l`
+funcs=`count_enabled_functions`
test $funcs -eq 0
:;: "Add another event not on the test module" ;:
echo 1 > events/fprobes/test2/enable
:;: "Ensure it is enabled" ;:
-ofuncs=`cat enabled_functions | wc -l`
+ofuncs=`count_enabled_functions`
test $ofuncs -ne 0
:;: "Disable and remove the first event"
echo 0 > events/fprobes/test1/enable
echo "-:fprobes/test1" >> dynamic_events
-funcs=`cat enabled_functions | wc -l`
+funcs=`count_enabled_functions`
test $ofuncs -eq $funcs
:;: "Disable and remove other events" ;:
echo 0 > events/fprobes/enable
echo > dynamic_events
-funcs=`cat enabled_functions | wc -l`
+funcs=`count_enabled_functions`
test $funcs -eq 0
rmmod trace-events-sample
echo 1 > events/fprobes/test1/enable
echo "f:test2 $FUNC2" >> dynamic_events
echo 1 > events/fprobes/test2/enable
-ofuncs=`cat enabled_functions | wc -l`
+ofuncs=`count_enabled_functions`
test $ofuncs -ne 0
:;: "Unload module (ftrace entry should be removed)" ;:
rmmod trace-events-sample
-funcs=`cat enabled_functions | wc -l`
+funcs=`count_enabled_functions`
test $funcs -ne 0
test $ofuncs -ne $funcs
echo "-:fprobes/test2" >> dynamic_events
:;: "Ensure ftrace is disabled." ;:
-funcs=`cat enabled_functions | wc -l`
+funcs=`count_enabled_functions`
test $funcs -eq 0
echo 0 > events/fprobes/enable
#!/bin/sh
# SPDX-License-Identifier: GPL-2.0
# description: event trigger - test poll wait on histogram
-# requires: set_event events/sched/sched_process_free/trigger events/sched/sched_process_free/hist
+# requires: set_event events/sched/sched_process_exit/trigger events/sched/sched_process_exit/hist
# flags: instance
POLL=${FTRACETEST_ROOT}/poll
exit_unresolved
fi
-EVENT=events/sched/sched_process_free/
+EVENT=events/sched/sched_process_exit/
# Check poll ops is supported. Before implementing poll on hist file, it
# returns soon with POLLIN | POLLOUT, but not POLLPRI.
gpio-mockup-cdev
gpio-chip-info
gpio-line-name
+gpio-cdev-uaf
HOSTPKG_CONFIG := pkg-config
-CFLAGS += -g -O0 -rdynamic -Wall -Werror -I$(OUTPUT)
+CFLAGS += -g -O0 -rdynamic -Wall -Werror -I$(OUTPUT) $(KHDR_INCLUDES)
CFLAGS += -I$(OUTPUT)/tools/include
LDLIBS += -lelf -lz -lrt -lpthread
self->skel = hid__open();
ASSERT_OK_PTR(self->skel) TEARDOWN_LOG("Error while calling hid__open");
+ /*
+ * Disable all struct_ops maps by default so libbpf does not autoload
+ * programs referenced by maps that are unrelated to the current test.
+ */
+ bpf_object__for_each_map(iter_map, *self->skel->skeleton->obj) {
+ if (bpf_map__type(iter_map) == BPF_MAP_TYPE_STRUCT_OPS) {
+ err = bpf_map__set_autocreate(iter_map, false);
+ ASSERT_OK(err) TH_LOG("can not disable struct_ops map '%s'",
+ bpf_map__name(iter_map));
+ }
+
+ bpf_map__set_autoattach(iter_map, false);
+ }
+
for (int i = 0; i < progs_count; i++) {
struct bpf_program *prog;
struct bpf_map *map;
ASSERT_OK_PTR(map) TH_LOG("can not find struct_ops by name '%s'",
programs[i].name + 4);
+ err = bpf_map__set_autocreate(map, true);
+ ASSERT_OK(err) TH_LOG("can not enable struct_ops map '%s'",
+ programs[i].name + 4);
+
/* hid_id is the first field of struct hid_bpf_ops */
ops_hid_id = bpf_map__initial_value(map, NULL);
ASSERT_OK_PTR(ops_hid_id) TH_LOG("unable to retrieve struct_ops data");
*ops_hid_id = self->hid.hid_id;
}
- /* we disable the auto-attach feature of all maps because we
- * only want the tested one to be manually attached in the next
- * call to bpf_map__attach_struct_ops()
- */
- bpf_object__for_each_map(iter_map, *self->skel->skeleton->obj)
- bpf_map__set_autoattach(iter_map, false);
-
err = hid__load(self->skel);
ASSERT_OK(err) TH_LOG("hid_skel_load failed: %d", err);
ASSERT_EQ(rpt_desc.value[4], 0x42);
}
+TEST_F(hid_bpf, test_rdesc_fixup_get_data_overflow)
+{
+ const struct test_program progs[] = {
+ { .name = "hid_rdesc_fixup_get_data_overflow" },
+ };
+
+ LOAD_PROGRAMS(progs);
+
+ ASSERT_EQ(self->skel->bss->get_data_overflow_check, 1);
+}
+
static int libbpf_print_fn(enum libbpf_print_level level,
const char *format, va_list args)
{
__u64 callback_check = 52;
__u64 callback2_check = 52;
+__u64 get_data_overflow_check;
SEC("?struct_ops/hid_device_event")
int BPF_PROG(hid_first_event, struct hid_bpf_ctx *hid_ctx, enum hid_report_type type)
.hid_rdesc_fixup = (void *)hid_rdesc_fixup,
};
+SEC("?struct_ops.s/hid_rdesc_fixup")
+int BPF_PROG(hid_rdesc_fixup_get_data_overflow, struct hid_bpf_ctx *hid_ctx)
+{
+ if (!hid_bpf_get_data(hid_ctx, 2 /* offset */, ~0ULL /* size */))
+ get_data_overflow_check = 1;
+
+ return 0;
+}
+
+SEC(".struct_ops.link")
+struct hid_bpf_ops rdesc_fixup_get_data_overflow = {
+ .hid_rdesc_fixup = (void *)hid_rdesc_fixup_get_data_overflow,
+};
+
SEC("?struct_ops/hid_device_event")
int BPF_PROG(hid_test_insert1, struct hid_bpf_ctx *hid_ctx, enum hid_report_type type)
{
return absinfo is not None and absinfo.resolution == 3
+class MinWin8TSParallelBigContactMax(Digitizer):
+ """A parallel Win8 touchscreen that advertises a ContactCountMaximum much
+ larger than the number of contacts it actually reports.
+
+ Such firmware makes the driver allocate that many input slots (up to 255)
+ while the input report only carries a few contacts. This is what used to
+ drive the per-slot bit operations on mt_io_flags out of bounds. The number
+ of contacts a HID report can describe is limited by the descriptor size,
+ so a large ContactCountMaximum can only be expressed this way, decoupled
+ from the number of finger collections."""
+
+ def __init__(self, n_fingers=5, contact_max=250):
+ self.phys_max = 120, 90
+ rdesc_finger_str = f"""
+ Usage Page (Digitizers)
+ Usage (Finger)
+ Collection (Logical)
+ Report Size (1)
+ Report Count (1)
+ Logical Minimum (0)
+ Logical Maximum (1)
+ Usage (Tip Switch)
+ Input (Data,Var,Abs)
+ Report Size (7)
+ Logical Maximum (127)
+ Input (Cnst,Var,Abs)
+ Report Size (8)
+ Logical Maximum (255)
+ Usage (Contact Id)
+ Input (Data,Var,Abs)
+ Report Size (16)
+ Unit Exponent (-1)
+ Unit (SILinear: cm)
+ Logical Maximum (4095)
+ Physical Minimum (0)
+ Physical Maximum ({self.phys_max[0]})
+ Usage Page (Generic Desktop)
+ Usage (X)
+ Input (Data,Var,Abs)
+ Physical Maximum ({self.phys_max[1]})
+ Usage (Y)
+ Input (Data,Var,Abs)
+ End Collection
+"""
+ rdesc_str = f"""
+ Usage Page (Digitizers)
+ Usage (Touch Screen)
+ Collection (Application)
+ Report ID (1)
+ {rdesc_finger_str * n_fingers}
+ Unit Exponent (-4)
+ Unit (SILinear: s)
+ Logical Maximum (65535)
+ Physical Maximum (65535)
+ Usage Page (Digitizers)
+ Usage (Scan Time)
+ Input (Data,Var,Abs)
+ Report Size (8)
+ Logical Maximum (255)
+ Usage (Contact Count)
+ Input (Data,Var,Abs)
+ Report ID (2)
+ Logical Maximum ({contact_max})
+ Usage (Contact Max)
+ Feature (Data,Var,Abs)
+ End Collection
+ {Digitizer.msCertificationBlob(68)}
+"""
+ super().__init__(
+ f"uhid test parallel big contact max {contact_max}", rdesc_str
+ )
+
+
class BaseTest:
class TestMultitouch(base.BaseTestCase.TestUhid):
kernel_modules = [KERNEL_MODULE]
return MinWin8TSParallel(10)
+class TestMinWin8TSParallelBigContactMax(base.BaseTestCase.TestUhid):
+ """Regression test for the out-of-bounds bit operations on
+ struct mt_device.mt_io_flags.
+
+ A Win8 touchscreen may advertise a ContactCountMaximum much larger than
+ the number of contacts it reports. The driver used to keep the per-slot
+ active state in the bits of a single unsigned long while indexing
+ set_bit()/clear_bit() by the slot number, so such a device drove those bit
+ operations out of bounds. The sticky-fingers release timer made it fatal:
+ mt_release_contacts() cleared one bit per slot, overwrote the adjacent
+ struct mt_device members and panicked the kernel.
+
+ Send a single contact, let the 100ms sticky-fingers timer release it, and
+ check that the kernel reports the release cleanly instead of crashing."""
+
+ kernel_modules = [KERNEL_MODULE]
+
+ def create_device(self):
+ return MinWin8TSParallelBigContactMax()
+
+ def test_sticky_fingers_release_big_contact_max(self):
+ uhdev = self.uhdev
+ evdev = uhdev.get_evdev()
+
+ assert evdev.num_slots == uhdev.max_contacts
+
+ t0 = Touch(1, 5, 10)
+ r = uhdev.event([t0])
+ events = uhdev.next_sync_events()
+ self.debug_reports(r, uhdev, events)
+ assert evdev.slots[0][libevdev.EV_ABS.ABS_MT_TRACKING_ID] == 0
+
+ # do not release the contact; the sticky-fingers timer must do it
+ # after 100ms, which is where the out-of-bounds release used to hit
+ time.sleep(0.2)
+ events = uhdev.next_sync_events()
+ self.debug_reports(r, uhdev, events)
+ assert libevdev.InputEvent(libevdev.EV_KEY.BTN_TOUCH, 0) in events
+ assert evdev.slots[0][libevdev.EV_ABS.ABS_MT_TRACKING_ID] == -1
+
+
class TestMinWin8TSHybrid(BaseTest.TestWin8Multitouch):
def create_device(self):
return MinWin8TSHybrid()
TEST_GEN_PROGS_arm64 += arm64/host_sve
TEST_GEN_PROGS_arm64 += arm64/hypercalls
TEST_GEN_PROGS_arm64 += arm64/external_aborts
+TEST_GEN_PROGS_arm64 += arm64/mmio_sign_ext
TEST_GEN_PROGS_arm64 += arm64/page_fault_test
TEST_GEN_PROGS_arm64 += arm64/psci_test
TEST_GEN_PROGS_arm64 += arm64/sea_to_user
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * mmio_sign_ext - Test sign-extending MMIO load emulation (LDRSB/LDRSH/LDRSW)
+ *
+ * Copyright (c) 2026 Google LLC
+ * Author: Fuad Tabba <fuad.tabba@linux.dev>
+ */
+
+#include <asm/ptrace.h>
+
+#include "processor.h"
+#include "test_util.h"
+
+#define MMIO_ADDR 0x8000000ULL
+
+/* AP[1]: allow unprivileged (EL0) access to a mapping. */
+#define PTE_USER BIT(6)
+
+/* SPSR for ERET to EL0t with DAIF masked. */
+#define SPSR_EL0 (PSR_MODE_EL0t | PSR_D_BIT | PSR_A_BIT | PSR_I_BIT | PSR_F_BIT)
+
+struct mmio_test {
+ const char *name;
+ uint64_t data; /* access-width value, host byte order */
+ uint8_t len;
+ uint64_t expected; /* sign-extended result; same for LE and BE */
+};
+
+/* Paired 1:1, in order, with the loads in guest_loads_le() and el0_be_loads. */
+static const struct mmio_test tests[] = {
+ /* LDRSB Xt: byte sign-extended to 64 bits */
+ { "LDRSB Xt 0xFF", 0xFF, 1, 0xFFFFFFFFFFFFFFFFULL },
+ { "LDRSB Xt 0x7F", 0x7F, 1, 0x7FULL },
+
+ /* LDRSB Wt: byte sign-extended to 32 bits, upper 32 bits zeroed */
+ { "LDRSB Wt 0xFF", 0xFF, 1, 0xFFFFFFFFULL },
+ { "LDRSB Wt 0x7F", 0x7F, 1, 0x7FULL },
+
+ /* LDRSH Xt: halfword sign-extended to 64 bits */
+ { "LDRSH Xt 0x8001", 0x8001, 2, 0xFFFFFFFFFFFF8001ULL },
+ { "LDRSH Xt 0x7FFF", 0x7FFF, 2, 0x7FFFULL },
+
+ /* LDRSH Wt: halfword sign-extended to 32 bits, upper 32 bits zeroed */
+ { "LDRSH Wt 0x8001", 0x8001, 2, 0xFFFF8001ULL },
+ { "LDRSH Wt 0x7FFF", 0x7FFF, 2, 0x7FFFULL },
+
+ /* LDRSW Xt: word sign-extended to 64 bits (no Wt form) */
+ { "LDRSW Xt 0x80000001", 0x80000001, 4, 0xFFFFFFFF80000001ULL },
+ { "LDRSW Xt 0x7FFFFFFF", 0x7FFFFFFF, 4, 0x7FFFFFFFULL },
+};
+
+/* Issue one sign-extending load from MMIO and report the result. */
+#define GUEST_LDRS(load) do { \
+ uint64_t val; \
+ \
+ asm volatile(load : "=r"(val) : "r"(MMIO_ADDR) : "memory"); \
+ GUEST_SYNC(val); \
+} while (0)
+
+/* Little-endian pass: loads issued at EL1. */
+static void guest_loads_le(void)
+{
+ GUEST_LDRS("ldrsb %0, [%1]");
+ GUEST_LDRS("ldrsb %0, [%1]");
+ GUEST_LDRS("ldrsb %w0, [%1]");
+ GUEST_LDRS("ldrsb %w0, [%1]");
+ GUEST_LDRS("ldrsh %0, [%1]");
+ GUEST_LDRS("ldrsh %0, [%1]");
+ GUEST_LDRS("ldrsh %w0, [%1]");
+ GUEST_LDRS("ldrsh %w0, [%1]");
+ GUEST_LDRS("ldrsw %0, [%1]");
+ GUEST_LDRS("ldrsw %0, [%1]");
+}
+
+/*
+ * Run the big-endian loads at EL0, where SCTLR_EL1.E0E flips only the data
+ * endianness; at EL1, SCTLR_EL1.EE would also flip the page-table walk and
+ * fault on the little-endian tables. x0 holds MMIO_ADDR; results return in
+ * x19..x28 (tests[] order) via a single SVC.
+ */
+extern char el0_be_loads[];
+asm(
+" .pushsection .text, \"ax\"\n"
+" .global el0_be_loads\n"
+"el0_be_loads:\n"
+" ldrsb x19, [x0]\n"
+" ldrsb x20, [x0]\n"
+" ldrsb w21, [x0]\n"
+" ldrsb w22, [x0]\n"
+" ldrsh x23, [x0]\n"
+" ldrsh x24, [x0]\n"
+" ldrsh w25, [x0]\n"
+" ldrsh w26, [x0]\n"
+" ldrsw x27, [x0]\n"
+" ldrsw x28, [x0]\n"
+" svc #0\n"
+" .popsection\n"
+);
+
+/* EL1 handler for the EL0 SVC: report the results, then finish. */
+static void el0_svc_handler(struct ex_regs *regs)
+{
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(tests); i++)
+ GUEST_SYNC(regs->regs[19 + i]);
+
+ GUEST_DONE();
+}
+
+static bool guest_mixed_endian_el0(void)
+{
+ uint64_t mmfr0 = read_sysreg(id_aa64mmfr0_el1);
+
+ return SYS_FIELD_GET(ID_AA64MMFR0_EL1, BIGEND, mmfr0) ||
+ SYS_FIELD_GET(ID_AA64MMFR0_EL1, BIGENDEL0, mmfr0);
+}
+
+static void guest_code(void)
+{
+ guest_loads_le();
+
+ if (guest_mixed_endian_el0()) {
+ write_sysreg(read_sysreg(sctlr_el1) | SCTLR_EL1_E0E, sctlr_el1);
+ isb();
+
+ asm volatile(
+ " msr elr_el1, %[pc]\n"
+ " msr spsr_el1, %[spsr]\n"
+ " mov x0, %[mmio]\n"
+ " isb\n"
+ " eret\n"
+ :
+ : [pc] "r"(el0_be_loads),
+ [spsr] "r"((uint64_t)SPSR_EL0),
+ [mmio] "r"(MMIO_ADDR)
+ : "x0", "memory");
+ __builtin_unreachable(); /* el0_svc_handler ends the test */
+ }
+
+ GUEST_DONE();
+}
+
+static void handle_mmio(struct kvm_run *run, const struct mmio_test *t, bool be)
+{
+ int i;
+
+ TEST_ASSERT_EQ(run->mmio.phys_addr, MMIO_ADDR);
+ TEST_ASSERT(!run->mmio.is_write, "Expected MMIO read for %s", t->name);
+ TEST_ASSERT_EQ(run->mmio.len, t->len);
+
+ memset(run->mmio.data, 0, sizeof(run->mmio.data));
+ if (be) {
+ /* The guest reads the device bytes most-significant first. */
+ for (i = 0; i < t->len; i++)
+ run->mmio.data[i] = t->data >> (8 * (t->len - 1 - i));
+ } else {
+ /* Works because arm64 KVM hosts are always little-endian. */
+ memcpy(run->mmio.data, &t->data, t->len);
+ }
+}
+
+static void expect_sync(struct kvm_vcpu *vcpu, struct ucall *uc,
+ const struct mmio_test *t)
+{
+ switch (get_ucall(vcpu, uc)) {
+ case UCALL_SYNC:
+ TEST_ASSERT(uc->args[1] == t->expected,
+ "%s: got %#lx, want %#lx", t->name,
+ (unsigned long)uc->args[1], (unsigned long)t->expected);
+ break;
+ case UCALL_ABORT:
+ REPORT_GUEST_ASSERT(*uc);
+ break;
+ default:
+ TEST_FAIL("Unexpected ucall for %s", t->name);
+ }
+}
+
+/* OR PTE_USER into the leaf descriptors covering [gva, gva + len). */
+static void make_el0_accessible(struct kvm_vm *vm, uint64_t gva, uint64_t len)
+{
+ uint64_t addr;
+
+ for (addr = gva & ~((uint64_t)vm->page_size - 1); addr < gva + len;
+ addr += vm->page_size)
+ *virt_get_pte_hva(vm, addr) |= PTE_USER;
+}
+
+static bool vcpu_mixed_endian_el0(struct kvm_vcpu *vcpu)
+{
+ uint64_t mmfr0 = vcpu_get_reg(vcpu, KVM_ARM64_SYS_REG(SYS_ID_AA64MMFR0_EL1));
+
+ return SYS_FIELD_GET(ID_AA64MMFR0_EL1, BIGEND, mmfr0) ||
+ SYS_FIELD_GET(ID_AA64MMFR0_EL1, BIGENDEL0, mmfr0);
+}
+
+int main(void)
+{
+ struct kvm_vcpu *vcpu;
+ struct kvm_vm *vm;
+ struct ucall uc;
+ unsigned int i;
+ bool be;
+
+ vm = vm_create_with_one_vcpu(&vcpu, guest_code);
+ virt_map(vm, MMIO_ADDR, MMIO_ADDR, 1);
+
+ vm_init_descriptor_tables(vm);
+ vcpu_init_descriptor_tables(vcpu);
+ vm_install_sync_handler(vm, VECTOR_SYNC_LOWER_64, ESR_ELx_EC_SVC64,
+ el0_svc_handler);
+
+ be = vcpu_mixed_endian_el0(vcpu);
+ if (be)
+ make_el0_accessible(vm, MMIO_ADDR, vm->page_size);
+
+ ksft_print_header();
+ ksft_set_plan(ARRAY_SIZE(tests) * (be ? 2 : 1));
+
+ /* Little-endian pass: one load and one result per iteration. */
+ for (i = 0; i < ARRAY_SIZE(tests); i++) {
+ const struct mmio_test *t = &tests[i];
+
+ vcpu_run(vcpu);
+ TEST_ASSERT_KVM_EXIT_REASON(vcpu, KVM_EXIT_MMIO);
+ handle_mmio(vcpu->run, t, false);
+
+ vcpu_run(vcpu);
+ expect_sync(vcpu, &uc, t);
+
+ ksft_test_result_pass("%s\n", t->name);
+ }
+
+ if (be) {
+ /* The EL0 stub issues all the loads, then reports the results. */
+ for (i = 0; i < ARRAY_SIZE(tests); i++) {
+ vcpu_run(vcpu);
+ TEST_ASSERT_KVM_EXIT_REASON(vcpu, KVM_EXIT_MMIO);
+ handle_mmio(vcpu->run, &tests[i], true);
+ }
+ for (i = 0; i < ARRAY_SIZE(tests); i++) {
+ vcpu_run(vcpu);
+ expect_sync(vcpu, &uc, &tests[i]);
+ ksft_test_result_pass("BE %s\n", tests[i].name);
+ }
+ }
+
+ vcpu_run(vcpu);
+ TEST_ASSERT(get_ucall(vcpu, &uc) == UCALL_DONE, "Expected UCALL_DONE");
+
+ kvm_vm_free(vm);
+
+ ksft_finished();
+}
KVM_X86_SEV_VMSA_FEATURES,
&supported_vmsa_features);
- have_sev = kvm_cpu_has(X86_FEATURE_SEV);
- TEST_ASSERT(have_sev == !!(kvm_check_cap(KVM_CAP_VM_TYPES) & BIT(KVM_X86_SEV_VM)),
- "sev: KVM_CAP_VM_TYPES (%x) does not match cpuid (checking %x)",
- kvm_check_cap(KVM_CAP_VM_TYPES), 1 << KVM_X86_SEV_VM);
+ /*
+ * Whether a VM type is available depends on KVM, not just CPUID: e.g.
+ * when all SEV ASIDs are assigned to SEV-SNP, KVM does not offer the
+ * SEV VM type even though X86_FEATURE_SEV is set. Derive availability
+ * from KVM_CAP_VM_TYPES and only assert the one-way implication that a
+ * type offered by KVM must also be reported in CPUID.
+ */
+ have_sev = kvm_check_cap(KVM_CAP_VM_TYPES) & BIT(KVM_X86_SEV_VM);
+ TEST_ASSERT(!have_sev || kvm_cpu_has(X86_FEATURE_SEV),
+ "sev: SEV_VM supported without SEV in CPUID");
- TEST_REQUIRE(kvm_check_cap(KVM_CAP_VM_TYPES) & BIT(KVM_X86_SEV_VM));
+ TEST_REQUIRE(have_sev);
have_sev_es = kvm_check_cap(KVM_CAP_VM_TYPES) & BIT(KVM_X86_SEV_ES_VM);
TEST_ASSERT(!have_sev_es || kvm_cpu_has(X86_FEATURE_SEV_ES),
kvm_vm_free(vm_no_vcpu);
}
+static void test_sev_snp_migrate_reject(void)
+{
+ struct kvm_vm *src_vm, *dst_vm;
+ int ret;
+
+ src_vm = vm_create_barebones_type(KVM_X86_SNP_VM);
+ snp_vm_init(src_vm);
+ __vm_vcpu_add(src_vm, 0);
+ vm_sev_launch(src_vm, snp_default_policy(), NULL);
+
+ dst_vm = vm_create_barebones_type(KVM_X86_SNP_VM);
+ __vm_vcpu_add(dst_vm, 0);
+
+ ret = __sev_migrate_from(dst_vm, src_vm);
+ TEST_ASSERT(ret == -1 && errno == EINVAL,
+ "SNP VM migration should be rejected. ret: %d, errno: %d",
+ ret, errno);
+
+ kvm_vm_free(src_vm);
+ kvm_vm_free(dst_vm);
+}
+
+static void test_sev_snp_mirror_reject(void)
+{
+ struct kvm_vm *src_vm, *dst_vm;
+ int ret;
+
+ src_vm = vm_create_barebones_type(KVM_X86_SNP_VM);
+ snp_vm_init(src_vm);
+ __vm_vcpu_add(src_vm, 0);
+ vm_sev_launch(src_vm, snp_default_policy(), NULL);
+
+ dst_vm = aux_vm_create(false);
+
+ ret = __sev_mirror_create(dst_vm, src_vm);
+ TEST_ASSERT(ret == -1 && errno == EINVAL,
+ "SNP VM mirroring should be rejected. ret: %d, errno: %d",
+ ret, errno);
+
+ kvm_vm_free(src_vm);
+ kvm_vm_free(dst_vm);
+}
+
static void test_sev_move_copy(void)
{
struct kvm_vm *dst_vm, *dst2_vm, *dst3_vm, *sev_vm, *mirror_vm,
test_sev_migrate_parameters();
if (kvm_has_cap(KVM_CAP_VM_COPY_ENC_CONTEXT_FROM))
test_sev_move_copy();
+ if (kvm_cpu_has(X86_FEATURE_SEV_SNP))
+ test_sev_snp_migrate_reject();
}
if (kvm_has_cap(KVM_CAP_VM_COPY_ENC_CONTEXT_FROM)) {
test_sev_mirror(/* es= */ false);
if (have_sev_es)
test_sev_mirror(/* es= */ true);
test_sev_mirror_parameters();
+ if (kvm_cpu_has(X86_FEATURE_SEV_SNP))
+ test_sev_snp_mirror_reject();
}
return 0;
}
{
TEST_REQUIRE(kvm_cpu_has(X86_FEATURE_SEV));
- test_sev_smoke(guest_sev_code, KVM_X86_SEV_VM, 0);
+ /*
+ * Only exercise VM types the host actually offers. CPUID reporting
+ * SEV does not guarantee KVM offers the SEV VM type: when all SEV
+ * ASIDs are assigned to SEV-SNP, KVM_X86_SEV_VM is unavailable even
+ * though X86_FEATURE_SEV is set. Gate every type on KVM_CAP_VM_TYPES.
+ */
+ if (kvm_check_cap(KVM_CAP_VM_TYPES) & BIT(KVM_X86_SEV_VM))
+ test_sev_smoke(guest_sev_code, KVM_X86_SEV_VM, 0);
if (kvm_check_cap(KVM_CAP_VM_TYPES) & BIT(KVM_X86_SEV_ES_VM))
test_sev_smoke(guest_sev_es_code, KVM_X86_SEV_ES_VM, SEV_POLICY_ES);
EXPECT_EQ(0, close(bind_fd));
}
+TEST_F(protocol, tcp_fastopen)
+{
+ const bool restricted = variant->sandbox == TCP_SANDBOX &&
+ variant->prot.type == SOCK_STREAM &&
+ (variant->prot.protocol == IPPROTO_TCP ||
+ variant->prot.protocol == IPPROTO_IP) &&
+ (variant->prot.domain == AF_INET ||
+ variant->prot.domain == AF_INET6);
+ const struct landlock_ruleset_attr ruleset_attr = {
+ .handled_access_net = LANDLOCK_ACCESS_NET_CONNECT_TCP,
+ };
+ int bind_fd, client_fd, status;
+ char buf;
+ pid_t child;
+
+ bind_fd = socket_variant(&self->srv0);
+ ASSERT_LE(0, bind_fd);
+ EXPECT_EQ(0, bind_variant(bind_fd, &self->srv0));
+ if (self->srv0.protocol.type == SOCK_STREAM)
+ EXPECT_EQ(0, listen(bind_fd, backlog));
+
+ child = fork();
+ ASSERT_LE(0, child);
+ if (child == 0) {
+ int connect_fd, ret;
+
+ /* Closes listening socket for the child. */
+ EXPECT_EQ(0, close(bind_fd));
+
+ connect_fd = socket_variant(&self->srv0);
+ ASSERT_LE(0, connect_fd);
+
+ if (variant->sandbox == TCP_SANDBOX) {
+ const int ruleset_fd = landlock_create_ruleset(
+ &ruleset_attr, sizeof(ruleset_attr), 0);
+ ASSERT_LE(0, ruleset_fd);
+
+ enforce_ruleset(_metadata, ruleset_fd);
+ EXPECT_EQ(0, close(ruleset_fd));
+ }
+
+ /* Fast Open with no address. */
+ ret = sendto_variant(connect_fd, NULL, NULL, 0, MSG_FASTOPEN);
+ if (self->srv0.protocol.domain == AF_UNIX) {
+ EXPECT_EQ(-ENOTCONN, ret);
+ } else if (self->srv0.protocol.type == SOCK_DGRAM) {
+ EXPECT_EQ(-EDESTADDRREQ, ret);
+ } else {
+ EXPECT_EQ(-EINVAL, ret);
+ }
+
+ /* Fast Open to a denied address. */
+ ret = sendto_variant(connect_fd, &self->srv0, "A", 1,
+ MSG_FASTOPEN);
+ if (restricted) {
+ EXPECT_EQ(-EACCES, ret);
+ } else if (self->srv0.protocol.domain == AF_UNIX &&
+ self->srv0.protocol.type == SOCK_STREAM) {
+ EXPECT_EQ(-EOPNOTSUPP, ret);
+ } else {
+ EXPECT_EQ(0, ret);
+ }
+
+ EXPECT_EQ(0, close(connect_fd));
+ _exit(_metadata->exit_code);
+ return;
+ }
+
+ client_fd = bind_fd;
+ if (!restricted && self->srv0.protocol.type == SOCK_STREAM &&
+ self->srv0.protocol.domain != AF_UNIX) {
+ client_fd = accept(bind_fd, NULL, 0);
+ ASSERT_LE(0, client_fd);
+ }
+
+ if (restricted) {
+ EXPECT_EQ(-1, read(client_fd, &buf, 1));
+ EXPECT_EQ(ENOTCONN, errno);
+ } else if (self->srv0.protocol.domain == AF_UNIX &&
+ self->srv0.protocol.type == SOCK_STREAM) {
+ EXPECT_EQ(-1, read(client_fd, &buf, 1));
+ EXPECT_EQ(EINVAL, errno);
+ } else {
+ EXPECT_EQ(1, read(client_fd, &buf, 1));
+ EXPECT_EQ('A', buf);
+ }
+
+ EXPECT_EQ(child, waitpid(child, &status, 0));
+ EXPECT_EQ(1, WIFEXITED(status));
+ EXPECT_EQ(EXIT_SUCCESS, WEXITSTATUS(status));
+
+ if (client_fd != bind_fd)
+ EXPECT_LE(0, close(client_fd));
+
+ EXPECT_EQ(0, close(bind_fd));
+}
+
TEST_F(protocol, sendmsg_stream)
{
int srv0_fd, tmp_fd, client_fd, res;
_metadata->exit_code = KSFT_FAIL;
}
-enum thread_return {
- THREAD_INVALID = 0,
- THREAD_SUCCESS = 1,
- THREAD_ERROR = 2,
- THREAD_TEST_FAILED = 3,
-};
+/* clang-format off */
+#define THREAD_INVALID ((void *)0)
+#define THREAD_SUCCESS ((void *)1)
+#define THREAD_ERROR ((void *)2)
+#define THREAD_TEST_FAILED ((void *)3)
+/* clang-format on */
static void *thread_sync(void *arg)
{
char buf;
if (read(pipe_read, &buf, 1) != 1)
- return (void *)THREAD_ERROR;
+ return THREAD_ERROR;
- return (void *)THREAD_SUCCESS;
+ return THREAD_SUCCESS;
}
TEST(signal_scoping_thread_before)
{
pthread_t no_sandbox_thread;
- enum thread_return ret = THREAD_INVALID;
+ void *ret = THREAD_INVALID;
int thread_pipe[2];
drop_caps(_metadata);
EXPECT_EQ(0, pthread_kill(no_sandbox_thread, 0));
EXPECT_EQ(1, write(thread_pipe[1], ".", 1));
- EXPECT_EQ(0, pthread_join(no_sandbox_thread, (void **)&ret));
+ EXPECT_EQ(0, pthread_join(no_sandbox_thread, &ret));
EXPECT_EQ(THREAD_SUCCESS, ret);
EXPECT_EQ(0, close(thread_pipe[0]));
TEST(signal_scoping_thread_after)
{
pthread_t scoped_thread;
- enum thread_return ret = THREAD_INVALID;
+ void *ret = THREAD_INVALID;
int thread_pipe[2];
drop_caps(_metadata);
EXPECT_EQ(0, pthread_kill(scoped_thread, 0));
EXPECT_EQ(1, write(thread_pipe[1], ".", 1));
- EXPECT_EQ(0, pthread_join(scoped_thread, (void **)&ret));
+ EXPECT_EQ(0, pthread_join(scoped_thread, &ret));
EXPECT_EQ(THREAD_SUCCESS, ret);
EXPECT_EQ(0, close(thread_pipe[0]));
char buf;
if (read(arg->pipe_read, &buf, 1) != 1)
- return (void *)THREAD_ERROR;
+ return THREAD_ERROR;
/* libc's setuid() should update all thread's credentials. */
if (getuid() != arg->new_uid)
- return (void *)THREAD_TEST_FAILED;
+ return THREAD_TEST_FAILED;
- return (void *)THREAD_SUCCESS;
+ return THREAD_SUCCESS;
}
TEST(signal_scoping_thread_setuid)
{
struct thread_setuid_args arg;
pthread_t no_sandbox_thread;
- enum thread_return ret = THREAD_INVALID;
+ void *ret = THREAD_INVALID;
int pipe_parent[2];
int prev_uid;
EXPECT_EQ(arg.new_uid, getuid());
EXPECT_EQ(1, write(pipe_parent[1], ".", 1));
- EXPECT_EQ(0, pthread_join(no_sandbox_thread, (void **)&ret));
+ EXPECT_EQ(0, pthread_join(no_sandbox_thread, &ret));
EXPECT_EQ(THREAD_SUCCESS, ret);
clear_cap(_metadata, CAP_SETUID);
return sigaction(SIGURG, &sa, NULL);
}
+/*
+ * MSG_OOB might be disabled in the kernel via the CONFIG_AF_UNIX_OOB
+ * switch, so this function can be used for probing for its availability.
+ */
+static bool has_af_unix_oob(void)
+{
+ bool available = false;
+ int sp[2];
+
+ if (socketpair(AF_UNIX, SOCK_STREAM, 0, sp) == 0) {
+ available = (send(sp[0], ".", 1, MSG_OOB) == 1);
+ close(sp[0]);
+ close(sp[1]);
+ }
+
+ return available;
+}
+
/* clang-format off */
FIXTURE(fown) {};
/* clang-format on */
int pipe_parent[2], pipe_child[2];
pid_t child;
+ if (!has_af_unix_oob())
+ SKIP(return, "CONFIG_AF_UNIX_OOB / MSG_OOB not available");
+
memset(&server_address, 0, sizeof(server_address));
set_unix_address(&server_address, 0);
ruleset_fd =
landlock_create_ruleset(&ruleset_attr, sizeof(ruleset_attr), 0);
if (ruleset_fd < 0)
- return (void *)THREAD_ERROR;
+ return THREAD_ERROR;
if (prctl(PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0) ||
landlock_restrict_self(ruleset_fd, 0)) {
close(ruleset_fd);
- return (void *)THREAD_ERROR;
+ return THREAD_ERROR;
}
close(ruleset_fd);
/* Makes this process group own the SIGIO source. */
if (fcntl(fd, F_SETSIG, SIGURG) || fcntl(fd, F_SETOWN, -getpgrp()) ||
fcntl(fd, F_SETFL, O_ASYNC))
- return (void *)THREAD_ERROR;
+ return THREAD_ERROR;
- return (void *)THREAD_SUCCESS;
+ return THREAD_SUCCESS;
}
/*
{
int trigger[2];
pthread_t thread;
- enum thread_return ret = THREAD_INVALID;
+ void *ret = THREAD_INVALID;
int i;
drop_caps(_metadata);
*/
ASSERT_EQ(0, pthread_create(&thread, NULL, thread_setown_scoped,
&trigger[0]));
- ASSERT_EQ(0, pthread_join(thread, (void **)&ret));
+ ASSERT_EQ(0, pthread_join(thread, &ret));
ASSERT_EQ(THREAD_SUCCESS, ret);
/* Fans SIGURG out to the process group. */
#include <time.h>
#include <pthread.h>
#include <limits.h>
+#include <linux/mman.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <sys/mman.h>
#!/bin/sh -e
# SPDX-License-Identifier: GPL-2.0
-./run_vmtests.sh -t mmap
+./run_vmtests.sh -t process_madv
ksft_exit_fail_msg("pthread_barrier_wait\n");
for (i = 0; i < access_per_thread; ++i)
- __atomic_add_fetch(m + i * (0x1000 / sizeof(*m)), 1, __ATOMIC_SEQ_CST);
+ __atomic_add_fetch(m + i * (page_size / sizeof(*m)), 1, __ATOMIC_SEQ_CST);
ret = pthread_barrier_wait(&end_barrier);
if (ret && ret != PTHREAD_BARRIER_SERIAL_THREAD)
if (pthread_barrier_init(&end_barrier, NULL, nthreads + 1))
ksft_exit_fail_msg("pthread_barrier_init\n");
- mem = mmap(NULL, 0x1000 * nthreads * pages_per_thread, PROT_READ | PROT_WRITE,
+ mem = mmap(NULL, page_size * nthreads * pages_per_thread, PROT_READ | PROT_WRITE,
MAP_ANONYMOUS | MAP_PRIVATE, -1, 0);
if (mem == MAP_FAILED)
ksft_exit_fail_msg("Error mmap %s.\n", strerror(errno));
- wp_init(mem, 0x1000 * nthreads * pages_per_thread);
- wp_addr_range(mem, 0x1000 * nthreads * pages_per_thread);
+ wp_init(mem, page_size * nthreads * pages_per_thread);
+ wp_addr_range(mem, page_size * nthreads * pages_per_thread);
- memset(mem, 0, 0x1000 * nthreads * pages_per_thread);
+ memset(mem, 0, page_size * nthreads * pages_per_thread);
count = get_dirty_pages_reset(mem, nthreads * pages_per_thread, 1, page_size);
ksft_test_result(count > 0, "%s count %u\n", __func__, count);
finish = 0;
for (i = 0; i < nthreads; ++i)
- pthread_create(&th, NULL, thread_proc, mem + 0x1000 * i * pages_per_thread);
+ pthread_create(&th, NULL, thread_proc, mem + page_size * i * pages_per_thread);
extra_pages = 0;
for (i = 0; i < iter_count; ++i) {
$ksft_xfail $ksft_pass $ksft_skip $ksft_fail
}
+timestamp_ms()
+{
+ local now
+ local seconds
+ local nanoseconds
+
+ now=$(date -u +%s:%N) || return
+ seconds=${now%:*}
+ nanoseconds=${now#*:}
+
+ if [[ $nanoseconds =~ ^[0-9]+$ ]]; then
+ nanoseconds=${nanoseconds:0:9}
+ else
+ nanoseconds=0
+ fi
+
+ echo $((seconds * 1000 + 10#$nanoseconds / 1000000))
+}
+
loopy_wait()
{
local sleep_cmd=$1; shift
local timeout_ms=$1; shift
+ local start_time
+ local current_time
- local start_time="$(date -u +%s%3N)"
+ start_time=$(timestamp_ms) || return
while true
do
local out
return 0
fi
- local current_time="$(date -u +%s%3N)"
+ current_time=$(timestamp_ms) || return
if ((current_time - start_time > timeout_ms)); then
echo -n "$out"
return 1
ret=1
fi
+if ip -net "$nsr1" link show tun0 > /dev/null 2>&1 &&
+ ip -net "$nsr2" link show tun0 > /dev/null 2>&1; then
+ ip -net "$nsr1" route change default via 192.168.100.2
+ ip -net "$nsr2" route change default via 192.168.100.1
+ ip -6 -net "$nsr1" route delete default
+ ip -6 -net "$nsr1" route add default via fee1:3::2
+ ip -6 -net "$nsr2" route delete default
+ ip -6 -net "$nsr2" route add default via fee1:3::1
+ ip -net "$ns2" route add default via 10.0.2.1
+ ip -6 -net "$ns2" route add default via dead:2::1
+
+ ip netns exec "$nsr1" nft -a insert rule inet filter forward \
+ 'meta oif "tun0" tcp dport 12345 ct mark set 1 flow add @f1 counter name routed_orig accept'
+ ip netns exec "$nsr1" nft -a insert rule inet filter forward \
+ 'meta oif "tun6" tcp dport 12345 ct mark set 1 flow add @f1 counter name routed_orig accept'
+ ip netns exec "$nsr1" nft -a insert rule inet filter forward \
+ 'meta oif "veth0" tcp sport 12345 ct mark set 1 flow add @f1 counter name routed_repl accept'
+ ip netns exec "$nsr1" nft -a insert rule inet filter forward \
+ 'meta oif "br0" tcp sport 12345 ct mark set 1 flow add @f1 counter name routed_repl accept'
+ ip netns exec "$nsr1" nft -a insert rule inet filter forward \
+ 'meta oif "tun0" accept'
+ ip netns exec "$nsr1" nft -a insert rule inet filter forward \
+ 'meta oif "tun6" accept'
+
+ ip netns exec "$nsr1" nft reset counters table inet filter >/dev/null
+
+ if test_tcp_forwarding "$ns1" "$ns2" 1 4 10.0.2.99 12345; then
+ check_counters "bridge + IPIP tunnel"
+ else
+ echo "FAIL: flow offload for ns1/ns2 with bridge + IPIP tunnel" 1>&2
+ ip netns exec "$nsr1" nft list ruleset
+ ret=1
+ fi
+
+ if test_tcp_forwarding "$ns1" "$ns2" 1 6 "[dead:2::99]" 12345; then
+ check_counters "bridge + IP6IP6 tunnel"
+ else
+ echo "FAIL: flow offload for ns1/ns2 with bridge + IP6IP6 tunnel" 1>&2
+ ip netns exec "$nsr1" nft list ruleset
+ ret=1
+ fi
+
+ ip -net "$nsr1" route change default via 192.168.10.2
+ ip -net "$nsr2" route change default via 192.168.10.1
+ ip -net "$ns2" route del default via 10.0.2.1
+ ip -6 -net "$nsr1" route delete default
+ ip -6 -net "$nsr1" route add default via fee1:2::2
+ ip -6 -net "$nsr2" route delete default
+ ip -6 -net "$nsr2" route add default via fee1:2::1
+ ip -6 -net "$ns2" route del default via dead:2::1
+else
+ echo "SKIP: bridge + tunnel flowtable regression (tun0 missing)"
+ [ "$ret" -eq 0 ] && ret=$ksft_skip
+fi
+
# Another test:
# Add bridge interface br0 to Router1, with NAT and VLAN.
tcp_info_get_rcv_mss(fd));
}
error:
+ if (ctx)
+ EVP_MD_CTX_free(ctx);
munmap(buffer, buffer_sz);
close(fd);
if (zflg)
EVP_DigestFinal_ex(ctx, digest, &digest_len);
send(fd, digest, (size_t)SHA256_DIGEST_LENGTH, 0);
}
+ if (ctx)
+ EVP_MD_CTX_free(ctx);
close(fd);
munmap(buffer, buffer_sz);
return 0;
// SPDX-License-Identifier: GPL-2.0-only
#include <sys/ptrace.h>
+#include <sys/syscall.h>
#include <sys/types.h>
#include <sys/wait.h>
#include <sys/uio.h>
SKIP(return, "Vector not supported");
chld_lock = 1;
- pid = fork();
+ pid = (pid_t)syscall(SYS_clone, SIGCHLD, 0, NULL, 0, NULL);
ASSERT_LE(0, pid)
- TH_LOG("fork: %m");
+ TH_LOG("clone: %m");
if (pid == 0) {
while (chld_lock == 1)
ASSERT_EQ(-1, ret);
/* cleanup */
-
+ free(regset_data);
ASSERT_EQ(0, kill(pid, SIGKILL));
}
}
EXPECT_EQ(vl_csr, regset_data->vl);
/* cleanup */
-
+ free(regset_data);
ASSERT_EQ(0, kill(pid, SIGKILL));
}
}
EXPECT_EQ(0UL, regset_data->vl);
/* cleanup */
-
+ free(regset_data);
ASSERT_EQ(0, kill(pid, SIGKILL));
}
}
ASSERT_EQ(ret, -1);
/* cleanup */
-
+ free(regset_data);
ASSERT_EQ(0, kill(pid, SIGKILL));
}
}
EXPECT_EQ(regset_data->vlenb, vlenb);
/* cleanup */
-
+ free(regset_data);
ASSERT_EQ(0, kill(pid, SIGKILL));
}
}
endif
top_srcdir = ../../../..
+include $(top_srcdir)/scripts/subarch.include
+ARCH ?= $(SUBARCH)
+LINUX_TOOL_ARCH_INCLUDE = $(top_srcdir)/tools/arch/$(ARCH)/include
CFLAGS += -O2 -Wall -g -I./ $(KHDR_INCLUDES) -L$(OUTPUT) -Wl,-rpath=./ \
- $(CLANG_FLAGS) -I$(top_srcdir)/tools/include
+ $(CLANG_FLAGS) -I$(top_srcdir)/tools/include \
+ -I$(LINUX_TOOL_ARCH_INCLUDE)
LDLIBS += -lpthread -ldl
# Own dependencies because we only want to build against 1st prerequisite, but
maybe_null \
minimal \
non_scx_kfunc_deny \
+ nohz_tick \
numa \
allowed_cpus \
peek_dsq \
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES
+ *
+ * Exercise tick dependency transitions between infinite and finite slices.
+ */
+#include <scx/common.bpf.h>
+
+char _license[] SEC("license") = "GPL";
+
+const volatile s32 test_cpu;
+bool finite_phase;
+u64 nr_inf_running;
+u64 nr_finite_running;
+u64 nr_finite_ticks;
+
+UEI_DEFINE(uei);
+
+s32 BPF_STRUCT_OPS(nohz_tick_select_cpu, struct task_struct *p, s32 prev_cpu,
+ u64 wake_flags)
+{
+ return prev_cpu;
+}
+
+void BPF_STRUCT_OPS(nohz_tick_enqueue, struct task_struct *p, u64 enq_flags)
+{
+ u64 slice = finite_phase ? 1000000ULL : SCX_SLICE_INF;
+
+ scx_bpf_dsq_insert(p, SCX_DSQ_GLOBAL, slice, enq_flags);
+ if (enq_flags & SCX_ENQ_LAST)
+ scx_bpf_kick_cpu(test_cpu, SCX_KICK_IDLE);
+}
+
+void BPF_STRUCT_OPS(nohz_tick_running, struct task_struct *p)
+{
+ if (bpf_get_smp_processor_id() != test_cpu)
+ return;
+
+ if (finite_phase)
+ __sync_fetch_and_add(&nr_finite_running, 1);
+ else
+ __sync_fetch_and_add(&nr_inf_running, 1);
+}
+
+void BPF_STRUCT_OPS(nohz_tick_tick, struct task_struct *p)
+{
+ if (bpf_get_smp_processor_id() == test_cpu && finite_phase)
+ __sync_fetch_and_add(&nr_finite_ticks, 1);
+}
+
+void BPF_STRUCT_OPS(nohz_tick_exit, struct scx_exit_info *ei)
+{
+ UEI_RECORD(uei, ei);
+}
+
+SEC(".struct_ops.link")
+struct sched_ext_ops nohz_tick_ops = {
+ .select_cpu = (void *)nohz_tick_select_cpu,
+ .enqueue = (void *)nohz_tick_enqueue,
+ .running = (void *)nohz_tick_running,
+ .tick = (void *)nohz_tick_tick,
+ .exit = (void *)nohz_tick_exit,
+ .name = "nohz_tick",
+ .timeout_ms = 1000U,
+};
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES
+ *
+ * Validate that a finite-slice EXT task restarts the scheduler tick when it
+ * follows an infinite-slice EXT task and an idle interval on a NOHZ_FULL CPU.
+ */
+#define _GNU_SOURCE
+
+#include <bpf/bpf.h>
+#include <errno.h>
+#include <sched.h>
+#include <signal.h>
+#include <stdbool.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <sys/prctl.h>
+#include <sys/wait.h>
+#include <unistd.h>
+
+#include <scx/common.h>
+
+#include "nohz_tick.bpf.skel.h"
+#include "scx_test.h"
+
+#ifndef SCHED_EXT
+#define SCHED_EXT 7
+#endif
+
+#define MIN_FINITE_TICKS 3
+#define PHASE_TIMEOUT_MS 1000
+
+struct nohz_tick_ctx {
+ struct nohz_tick *skel;
+ cpu_set_t original_mask;
+ int test_cpu;
+};
+
+static int first_allowed_cpu(const cpu_set_t *mask, int first, int last)
+{
+ int cpu;
+
+ for (cpu = first; cpu <= last && cpu < CPU_SETSIZE; cpu++)
+ if (CPU_ISSET(cpu, mask))
+ return cpu;
+
+ return -1;
+}
+
+static int find_nohz_full_cpu(const cpu_set_t *allowed)
+{
+ char buf[4096], *cur, *end;
+ FILE *file;
+
+ file = fopen("/sys/devices/system/cpu/nohz_full", "r");
+ if (!file)
+ return -1;
+ if (!fgets(buf, sizeof(buf), file)) {
+ fclose(file);
+ return -1;
+ }
+ fclose(file);
+
+ cur = buf;
+ while (*cur) {
+ long first, last;
+ int cpu;
+
+ while (*cur == ' ' || *cur == '\t' || *cur == ',')
+ cur++;
+ if (*cur < '0' || *cur > '9')
+ break;
+
+ errno = 0;
+ first = strtol(cur, &end, 10);
+ if (errno || end == cur || first < 0 || first >= CPU_SETSIZE)
+ return -1;
+ cur = end;
+ last = first;
+ if (*cur == '-') {
+ cur++;
+ errno = 0;
+ last = strtol(cur, &end, 10);
+ if (errno || end == cur || last < first)
+ return -1;
+ cur = end;
+ }
+
+ cpu = first_allowed_cpu(allowed, first, last);
+ if (cpu >= 0)
+ return cpu;
+ }
+
+ return -1;
+}
+
+static pid_t start_worker(int cpu)
+{
+ struct sched_param param = {};
+ cpu_set_t mask;
+ pid_t parent;
+ pid_t pid;
+
+ parent = getpid();
+ pid = fork();
+ if (pid != 0)
+ return pid;
+ if (prctl(PR_SET_PDEATHSIG, SIGKILL) || getppid() != parent)
+ _exit(1);
+
+ /*
+ * Become EXT before touching the target so it stays idle until wakeup.
+ */
+ if (sched_setscheduler(0, SCHED_EXT, ¶m))
+ _exit(1);
+
+ CPU_ZERO(&mask);
+ CPU_SET(cpu, &mask);
+ if (sched_setaffinity(0, sizeof(mask), &mask))
+ _exit(1);
+
+ for (;;)
+ asm volatile("" ::: "memory");
+}
+
+static void stop_worker(pid_t pid)
+{
+ if (pid <= 0)
+ return;
+
+ kill(pid, SIGKILL);
+ waitpid(pid, NULL, 0);
+}
+
+static int pause_worker(pid_t pid)
+{
+ int status;
+
+ if (kill(pid, SIGSTOP))
+ return -errno;
+ if (waitpid(pid, &status, WUNTRACED) != pid)
+ return -errno;
+ if (!WIFSTOPPED(status))
+ return -ECHILD;
+
+ return 0;
+}
+
+static bool wait_for_counter(const u64 *counter, u64 value, int timeout_ms)
+{
+ int elapsed;
+
+ for (elapsed = 0; elapsed < timeout_ms; elapsed++) {
+ if (__atomic_load_n(counter, __ATOMIC_RELAXED) >= value)
+ return true;
+ usleep(1000);
+ }
+
+ return false;
+}
+
+static enum scx_test_status setup(void **ctx_ptr)
+{
+ struct nohz_tick_ctx *ctx;
+ cpu_set_t controller_mask;
+ int cpu;
+
+ ctx = calloc(1, sizeof(*ctx));
+ SCX_FAIL_IF(!ctx, "Failed to allocate context");
+ if (sched_getaffinity(0, sizeof(ctx->original_mask),
+ &ctx->original_mask)) {
+ free(ctx);
+ SCX_FAIL("Failed to get affinity (%d)", errno);
+ }
+
+ cpu = find_nohz_full_cpu(&ctx->original_mask);
+ if (cpu < 0) {
+ fprintf(stderr, "SKIP: no allowed NOHZ_FULL CPU\n");
+ free(ctx);
+ return SCX_TEST_SKIP;
+ }
+
+ controller_mask = ctx->original_mask;
+ CPU_CLR(cpu, &controller_mask);
+ if (CPU_COUNT(&controller_mask) == 0) {
+ fprintf(stderr, "SKIP: no housekeeping CPU available\n");
+ free(ctx);
+ return SCX_TEST_SKIP;
+ }
+
+ ctx->test_cpu = cpu;
+ ctx->skel = nohz_tick__open();
+ if (!ctx->skel) {
+ free(ctx);
+ SCX_FAIL("Failed to open skeleton");
+ }
+
+ SCX_ENUM_INIT(ctx->skel);
+ ctx->skel->rodata->test_cpu = cpu;
+ ctx->skel->struct_ops.nohz_tick_ops->flags |= SCX_OPS_SWITCH_PARTIAL |
+ SCX_OPS_ENQ_LAST;
+ if (nohz_tick__load(ctx->skel)) {
+ nohz_tick__destroy(ctx->skel);
+ free(ctx);
+ SCX_FAIL("Failed to load skeleton");
+ }
+
+ if (sched_setaffinity(0, sizeof(controller_mask), &controller_mask)) {
+ nohz_tick__destroy(ctx->skel);
+ free(ctx);
+ SCX_FAIL("Failed to move controller off CPU %d (%d)", cpu, errno);
+ }
+
+ *ctx_ptr = ctx;
+ return SCX_TEST_PASS;
+}
+
+static enum scx_test_status run(void *ctx_ptr)
+{
+ struct nohz_tick_ctx *ctx = ctx_ptr;
+ struct nohz_tick *skel = ctx->skel;
+ struct bpf_link *link = NULL;
+ enum scx_test_status status = SCX_TEST_FAIL;
+ pid_t finite_worker = -1;
+ pid_t inf_worker = -1;
+ u64 finite_running;
+ u64 finite_ticks;
+ int ret;
+
+ link = bpf_map__attach_struct_ops(skel->maps.nohz_tick_ops);
+ if (!link) {
+ SCX_ERR("Failed to attach scheduler");
+ goto out;
+ }
+
+ /*
+ * Establish SCX_RQ_CAN_STOP_TICK with an infinite-slice task.
+ */
+ inf_worker = start_worker(ctx->test_cpu);
+ if (inf_worker < 0) {
+ SCX_ERR("Failed to start infinite-slice worker (%d)", errno);
+ goto out;
+ }
+ if (!wait_for_counter(&skel->bss->nr_inf_running, 1,
+ PHASE_TIMEOUT_MS)) {
+ SCX_ERR("Infinite-slice worker was not scheduled");
+ goto out;
+ }
+
+ /* Block without exiting so the rq retains the infinite-slice state. */
+ ret = pause_worker(inf_worker);
+ if (ret) {
+ SCX_ERR("Failed to stop infinite-slice worker (%d)", ret);
+ goto out;
+ }
+
+ /* Let the target enter idle with its tick stopped. */
+ usleep(100000);
+
+ /*
+ * The next EXT task receives a finite slice and must restart the tick.
+ */
+ __atomic_store_n(&skel->bss->finite_phase, true, __ATOMIC_RELEASE);
+ finite_worker = start_worker(ctx->test_cpu);
+ if (finite_worker < 0) {
+ SCX_ERR("Failed to start finite-slice worker (%d)", errno);
+ goto out;
+ }
+ if (!wait_for_counter(&skel->bss->nr_finite_running, 1,
+ PHASE_TIMEOUT_MS)) {
+ SCX_ERR("Finite-slice worker was not scheduled");
+ goto out;
+ }
+ if (!wait_for_counter(&skel->bss->nr_finite_ticks, MIN_FINITE_TICKS,
+ PHASE_TIMEOUT_MS)) {
+ SCX_ERR("Finite-slice worker received only %llu scheduler ticks",
+ (unsigned long long)skel->bss->nr_finite_ticks);
+ goto out;
+ }
+ stop_worker(finite_worker);
+ finite_worker = -1;
+
+ /*
+ * Leave the CPU idle after a finite-slice task. The next finite-slice
+ * task must restart the tick even though the slice type is unchanged.
+ */
+ usleep(100000);
+ finite_running = __atomic_load_n(&skel->bss->nr_finite_running,
+ __ATOMIC_RELAXED);
+ finite_ticks = __atomic_load_n(&skel->bss->nr_finite_ticks,
+ __ATOMIC_RELAXED);
+
+ finite_worker = start_worker(ctx->test_cpu);
+ if (finite_worker < 0) {
+ SCX_ERR("Failed to start second finite-slice worker (%d)", errno);
+ goto out;
+ }
+ if (!wait_for_counter(&skel->bss->nr_finite_running,
+ finite_running + 1, PHASE_TIMEOUT_MS)) {
+ SCX_ERR("Second finite-slice worker was not scheduled");
+ goto out;
+ }
+ if (!wait_for_counter(&skel->bss->nr_finite_ticks,
+ finite_ticks + MIN_FINITE_TICKS,
+ PHASE_TIMEOUT_MS)) {
+ SCX_ERR("Second finite-slice worker received only %llu scheduler ticks",
+ (unsigned long long)(skel->bss->nr_finite_ticks -
+ finite_ticks));
+ goto out;
+ }
+
+ if (skel->data->uei.kind != EXIT_KIND(SCX_EXIT_NONE)) {
+ SCX_ERR("Scheduler exited unexpectedly (kind=%llu code=%lld)",
+ (unsigned long long)skel->data->uei.kind,
+ (long long)skel->data->uei.exit_code);
+ goto out;
+ }
+
+ fprintf(stderr, "CPU %d received %llu finite-slice ticks\n",
+ ctx->test_cpu,
+ (unsigned long long)skel->bss->nr_finite_ticks);
+ status = SCX_TEST_PASS;
+out:
+ stop_worker(finite_worker);
+ stop_worker(inf_worker);
+ if (link)
+ bpf_link__destroy(link);
+ return status;
+}
+
+static void cleanup(void *ctx_ptr)
+{
+ struct nohz_tick_ctx *ctx = ctx_ptr;
+
+ sched_setaffinity(0, sizeof(ctx->original_mask), &ctx->original_mask);
+ nohz_tick__destroy(ctx->skel);
+ free(ctx);
+}
+
+struct scx_test nohz_tick = {
+ .name = "nohz_tick",
+ .description = "Verify finite EXT slices restart the NOHZ_FULL tick",
+ .setup = setup,
+ .run = run,
+ .cleanup = cleanup,
+};
+REGISTER_SCX_TEST(&nohz_tick)
return ret;
}
+/*
+ * Deleting an event drops its last reference, but an unregister may defer
+ * that put (and the freeing of the associated enabler) past an RCU grace
+ * period. The delete can therefore transiently fail with -EBUSY while the
+ * previous reference is still being dropped. Retry only on that transient
+ * failure; treat an already-deleted event (-ENOENT) as success and return
+ * any other error immediately rather than spinning for the full timeout.
+ */
+static int wait_for_event_delete(void)
+{
+ int i, ret;
+
+ for (i = 0; i < 10000; ++i) {
+ ret = event_delete();
+
+ if (ret == 0 || errno == ENOENT)
+ return 0;
+
+ if (errno != EBUSY)
+ return ret;
+
+ usleep(1000);
+ }
+
+ return ret;
+}
+
static int reg_enable_multi(void *enable, int size, int bit, int flags,
char *args)
{
ASSERT_TRUE(event_exists());
/* Ensure we can delete it */
- ASSERT_EQ(0, event_delete());
+ ASSERT_EQ(0, wait_for_event_delete());
/* USER_EVENT_REG_MAX or above is not allowed */
ASSERT_EQ(-1, reg_enable_flags(&self->check, sizeof(int), 0,
static int clear(int *check)
{
struct user_unreg unreg = {0};
+ int i, ret;
unreg.size = sizeof(unreg);
unreg.disable_bit = 31;
if (errno != ENOENT)
return -1;
- if (ioctl(fd, DIAG_IOCSDEL, "__test_event") == -1)
- if (errno != ENOENT)
+ /*
+ * Deleting the event drops its last reference, but the unregister
+ * above defers that put (and the freeing of the enabler) past an RCU
+ * grace period. The delete can therefore transiently fail with -EBUSY
+ * until that reference is dropped. Retry for up to ~10 seconds so the
+ * event is actually gone before the next test registers the same name.
+ */
+ for (i = 0; i < 10000; ++i) {
+ ret = ioctl(fd, DIAG_IOCSDEL, "__test_event");
+
+ if (ret == 0 || errno == ENOENT) {
+ ret = 0;
+ break;
+ }
+
+ if (errno != EBUSY) {
+ close(fd);
return -1;
+ }
+
+ usleep(1000);
+ }
close(fd);
- return 0;
+ return ret;
}
FIXTURE(user) {
LIB_OUTPUT = $(CURDIR)/lib
endif
+LIB_CTYPE = $(LIB_OUTPUT)/ctype.o
+LIB_CTYPE_SRC = $(srctree)/tools/lib/ctype.c
+
LIB_STRING = $(LIB_OUTPUT)/string.o
LIB_STRING_SRC = $(srctree)/tools/lib/string.c
$(Q)echo "BPF skeleton support is disabled, skipping tests/bpf/bpf_action_map.o"
endif
-$(RTLA): $(RTLA_IN) $(LIBSUBCMD) $(LIB_STRING) $(LIB_STR_ERROR_R)
- $(QUIET_LINK)$(CC) $(LDFLAGS) -o $(RTLA) $(RTLA_IN) $(LIBSUBCMD) $(LIB_STRING) $(LIB_STR_ERROR_R) $(EXTLIBS)
+$(RTLA): $(RTLA_IN) $(LIBSUBCMD) $(LIB_CTYPE) $(LIB_STRING) $(LIB_STR_ERROR_R)
+ $(QUIET_LINK)$(CC) $(LDFLAGS) -o $(RTLA) $(RTLA_IN) $(LIBSUBCMD) $(LIB_CTYPE) $(LIB_STRING) $(LIB_STR_ERROR_R) $(EXTLIBS)
-static: $(RTLA_IN) $(LIBSUBCMD) $(LIB_STRING) $(LIB_STR_ERROR_R)
+static: $(RTLA_IN) $(LIBSUBCMD) $(LIB_CTYPE) $(LIB_STRING) $(LIB_STR_ERROR_R)
$(eval LDFLAGS += -static)
- $(QUIET_LINK)$(CC) -static $(LDFLAGS) -o $(RTLA)-static $(RTLA_IN) $(LIBSUBCMD) $(LIB_STRING) $(LIB_STR_ERROR_R) $(EXTLIBS)
+ $(QUIET_LINK)$(CC) -static $(LDFLAGS) -o $(RTLA)-static $(RTLA_IN) $(LIBSUBCMD) $(LIB_CTYPE) $(LIB_STRING) $(LIB_STR_ERROR_R) $(EXTLIBS)
rtla.%: fixdep FORCE
make -f $(srctree)/tools/build/Makefile.build dir=. $@
$(LIB_STRING): $(LIB_STRING_SRC) | $(LIB_OUTPUT)
$(QUIET_CC)$(CC) $(CFLAGS) -c -o $@ $<
+$(LIB_CTYPE): $(LIB_CTYPE_SRC) | $(LIB_OUTPUT)
+ $(QUIET_CC)$(CC) $(CFLAGS) -c -o $@ $<
+
libsubcmd-clean:
$(call QUIET_CLEAN, libsubcmd)
$(Q)$(RM) -r -- $(LIBSUBCMD_OUTPUT)
#include <signal.h>
#include <stdlib.h>
#include <string.h>
+#include <unistd.h>
#include <sys/sysinfo.h>
#include "common.h"
{ USB_SPEED_HIGH, "480", "High Speed(480Mbps)" },
{ USB_SPEED_WIRELESS, "53.3-480", "Wireless"},
{ USB_SPEED_SUPER, "5000", "Super Speed(5000Mbps)" },
+ { USB_SPEED_SUPER_PLUS, "10000", "Super Speed Plus(10000Mbps)" },
+ { USB_SPEED_SUPER_PLUS, "20000", "Super Speed Plus(20000Mbps)" },
{ 0, NULL, NULL }
};
.speed = USB_SPEED_SUPER,
.name = "super-speed",
},
+ {
+ .speed = USB_SPEED_SUPER_PLUS,
+ .name = "super-speed-plus",
+ },
};
static
switch (speed) {
case USB_SPEED_SUPER:
+ case USB_SPEED_SUPER_PLUS:
if (vhci_driver->idev[i].hub != HUB_SPEED_SUPER)
continue;
break;
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_PERCPU_TYPES_H
+#define _ASM_PERCPU_TYPES_H
+
+#define __percpu_qual
+
+#endif /* _ASM_PERCPU_TYPES_H */
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_COMPLETION_H
+#define _LINUX_COMPLETION_H
+
+struct completion {
+ unsigned int done;
+};
+
+#endif /* _LINUX_COMPLETION_H */
#ifndef LINUX_DEVICE_H
+#define LINUX_DEVICE_H
struct device {
void *parent;
#define DMA_MAPPING_ERROR (~(dma_addr_t)0)
#define DMA_ATTR_CPU_CACHE_CLEAN (1UL << 11)
+#define DMA_ATTR_DEBUGGING_IGNORE_CACHELINES 0
#endif
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_MOD_DEVICETABLE_H
+#define _LINUX_MOD_DEVICETABLE_H
+
+#include <linux/types.h>
+
+struct virtio_device_id {
+ __u32 device;
+ __u32 vendor;
+};
+
+#define VIRTIO_DEV_ANY_ID 0xffffffff
+
+#endif /* _LINUX_MOD_DEVICETABLE_H */
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_VIRTIO_FEATURES_H
+#define _LINUX_VIRTIO_FEATURES_H
+
+#include <linux/bug.h>
+#include <linux/string.h>
+#include <linux/types.h>
+
+#define VIRTIO_FEATURES_U64S 2
+#define VIRTIO_FEATURES_BITS (VIRTIO_FEATURES_U64S * 64)
+
+#define VIRTIO_BIT(b) (1ULL << ((b) & 0x3f))
+#define VIRTIO_U64(b) ((b) >> 6)
+
+#define VIRTIO_DECLARE_FEATURES(name) \
+ union { \
+ u64 name; \
+ u64 name##_array[VIRTIO_FEATURES_U64S];\
+ }
+
+static inline bool virtio_features_chk_bit(unsigned int bit)
+{
+ return bit < VIRTIO_FEATURES_BITS;
+}
+
+static inline bool virtio_features_test_bit(const u64 *features,
+ unsigned int bit)
+{
+ return virtio_features_chk_bit(bit) &&
+ !!(features[VIRTIO_U64(bit)] & VIRTIO_BIT(bit));
+}
+
+static inline void virtio_features_set_bit(u64 *features, unsigned int bit)
+{
+ if (virtio_features_chk_bit(bit))
+ features[VIRTIO_U64(bit)] |= VIRTIO_BIT(bit);
+}
+
+static inline void virtio_features_clear_bit(u64 *features, unsigned int bit)
+{
+ if (virtio_features_chk_bit(bit))
+ features[VIRTIO_U64(bit)] &= ~VIRTIO_BIT(bit);
+}
+
+static inline void virtio_features_zero(u64 *features)
+{
+ memset(features, 0, sizeof(features[0]) * VIRTIO_FEATURES_U64S);
+}
+
+static inline void virtio_features_from_u64(u64 *features, u64 from)
+{
+ virtio_features_zero(features);
+ features[0] = from;
+}
+
+static inline bool virtio_features_equal(const u64 *f1, const u64 *f2)
+{
+ int i;
+
+ for (i = 0; i < VIRTIO_FEATURES_U64S; ++i)
+ if (f1[i] != f2[i])
+ return false;
+ return true;
+}
+
+static inline void virtio_features_copy(u64 *to, const u64 *from)
+{
+ memcpy(to, from, sizeof(to[0]) * VIRTIO_FEATURES_U64S);
+}
+
+static inline void virtio_features_andnot(u64 *to, const u64 *f1, const u64 *f2)
+{
+ int i;
+
+ for (i = 0; i < VIRTIO_FEATURES_U64S; i++)
+ to[i] = f1[i] & ~f2[i];
+}
+
+#endif /* _LINUX_VIRTIO_FEATURES_H */
gpa_t addr)
{
struct kvm_io_bus *bus;
- int dev_idx, srcu_idx;
- struct kvm_io_device *iodev = NULL;
+ int dev_idx;
- srcu_idx = srcu_read_lock(&kvm->srcu);
+ lockdep_assert_held(&kvm->srcu);
bus = kvm_get_bus_srcu(kvm, bus_idx);
if (!bus)
- goto out_unlock;
+ return NULL;
dev_idx = kvm_io_bus_get_first_dev(bus, addr, 1);
if (dev_idx < 0)
- goto out_unlock;
-
- iodev = bus->range[dev_idx].dev;
-
-out_unlock:
- srcu_read_unlock(&kvm->srcu, srcu_idx);
+ return NULL;
- return iodev;
+ return bus->range[dev_idx].dev;
}
EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_io_bus_get_dev);
projects / linux.git / commitdiff