Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 1 | |
Grant Grundler | 74da15e | 2006-12-25 01:06:35 -0700 | [diff] [blame] | 2 | How To Write Linux PCI Drivers |
| 3 | |
| 4 | by Martin Mares <mj@ucw.cz> on 07-Feb-2000 |
| 5 | updated by Grant Grundler <grundler@parisc-linux.org> on 23-Dec-2006 |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 6 | |
| 7 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
Grant Grundler | 74da15e | 2006-12-25 01:06:35 -0700 | [diff] [blame] | 8 | The world of PCI is vast and full of (mostly unpleasant) surprises. |
| 9 | Since each CPU architecture implements different chip-sets and PCI devices |
| 10 | have different requirements (erm, "features"), the result is the PCI support |
| 11 | in the Linux kernel is not as trivial as one would wish. This short paper |
| 12 | tries to introduce all potential driver authors to Linux APIs for |
| 13 | PCI device drivers. |
| 14 | |
| 15 | A more complete resource is the third edition of "Linux Device Drivers" |
| 16 | by Jonathan Corbet, Alessandro Rubini, and Greg Kroah-Hartman. |
| 17 | LDD3 is available for free (under Creative Commons License) from: |
| 18 | |
| 19 | http://lwn.net/Kernel/LDD3/ |
| 20 | |
| 21 | However, keep in mind that all documents are subject to "bit rot". |
| 22 | Refer to the source code if things are not working as described here. |
| 23 | |
| 24 | Please send questions/comments/patches about Linux PCI API to the |
| 25 | "Linux PCI" <linux-pci@atrey.karlin.mff.cuni.cz> mailing list. |
| 26 | |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 27 | |
| 28 | |
| 29 | 0. Structure of PCI drivers |
| 30 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
Grant Grundler | 74da15e | 2006-12-25 01:06:35 -0700 | [diff] [blame] | 31 | PCI drivers "discover" PCI devices in a system via pci_register_driver(). |
| 32 | Actually, it's the other way around. When the PCI generic code discovers |
| 33 | a new device, the driver with a matching "description" will be notified. |
| 34 | Details on this below. |
| 35 | |
| 36 | pci_register_driver() leaves most of the probing for devices to |
| 37 | the PCI layer and supports online insertion/removal of devices [thus |
| 38 | supporting hot-pluggable PCI, CardBus, and Express-Card in a single driver]. |
| 39 | pci_register_driver() call requires passing in a table of function |
| 40 | pointers and thus dictates the high level structure of a driver. |
| 41 | |
| 42 | Once the driver knows about a PCI device and takes ownership, the |
| 43 | driver generally needs to perform the following initialization: |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 44 | |
| 45 | Enable the device |
Grant Grundler | 74da15e | 2006-12-25 01:06:35 -0700 | [diff] [blame] | 46 | Request MMIO/IOP resources |
| 47 | Set the DMA mask size (for both coherent and streaming DMA) |
| 48 | Allocate and initialize shared control data (pci_allocate_coherent()) |
| 49 | Access device configuration space (if needed) |
| 50 | Register IRQ handler (request_irq()) |
| 51 | Initialize non-PCI (i.e. LAN/SCSI/etc parts of the chip) |
| 52 | Enable DMA/processing engines |
| 53 | |
| 54 | When done using the device, and perhaps the module needs to be unloaded, |
| 55 | the driver needs to take the follow steps: |
| 56 | Disable the device from generating IRQs |
| 57 | Release the IRQ (free_irq()) |
| 58 | Stop all DMA activity |
| 59 | Release DMA buffers (both streaming and coherent) |
| 60 | Unregister from other subsystems (e.g. scsi or netdev) |
| 61 | Release MMIO/IOP resources |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 62 | Disable the device |
| 63 | |
Grant Grundler | 74da15e | 2006-12-25 01:06:35 -0700 | [diff] [blame] | 64 | Most of these topics are covered in the following sections. |
| 65 | For the rest look at LDD3 or <linux/pci.h> . |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 66 | |
| 67 | If the PCI subsystem is not configured (CONFIG_PCI is not set), most of |
Grant Grundler | 74da15e | 2006-12-25 01:06:35 -0700 | [diff] [blame] | 68 | the PCI functions described below are defined as inline functions either |
| 69 | completely empty or just returning an appropriate error codes to avoid |
| 70 | lots of ifdefs in the drivers. |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 71 | |
| 72 | |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 73 | |
Grant Grundler | 74da15e | 2006-12-25 01:06:35 -0700 | [diff] [blame] | 74 | 1. pci_register_driver() call |
| 75 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 76 | |
| 77 | PCI device drivers call pci_register_driver() during their |
| 78 | initialization with a pointer to a structure describing the driver |
| 79 | (struct pci_driver): |
| 80 | |
| 81 | field name Description |
| 82 | ---------- ------------------------------------------------------ |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 83 | id_table Pointer to table of device ID's the driver is |
| 84 | interested in. Most drivers should export this |
| 85 | table using MODULE_DEVICE_TABLE(pci,...). |
Grant Grundler | 74da15e | 2006-12-25 01:06:35 -0700 | [diff] [blame] | 86 | |
| 87 | probe This probing function gets called (during execution |
| 88 | of pci_register_driver() for already existing |
| 89 | devices or later if a new device gets inserted) for |
| 90 | all PCI devices which match the ID table and are not |
| 91 | "owned" by the other drivers yet. This function gets |
| 92 | passed a "struct pci_dev *" for each device whose |
| 93 | entry in the ID table matches the device. The probe |
| 94 | function returns zero when the driver chooses to |
| 95 | take "ownership" of the device or an error code |
| 96 | (negative number) otherwise. |
| 97 | The probe function always gets called from process |
| 98 | context, so it can sleep. |
| 99 | |
| 100 | remove The remove() function gets called whenever a device |
| 101 | being handled by this driver is removed (either during |
| 102 | deregistration of the driver or when it's manually |
| 103 | pulled out of a hot-pluggable slot). |
| 104 | The remove function always gets called from process |
| 105 | context, so it can sleep. |
| 106 | |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 107 | suspend Put device into low power state. |
Grant Grundler | 74da15e | 2006-12-25 01:06:35 -0700 | [diff] [blame] | 108 | suspend_late Put device into low power state. |
| 109 | |
| 110 | resume_early Wake device from low power state. |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 111 | resume Wake device from low power state. |
Grant Grundler | 74da15e | 2006-12-25 01:06:35 -0700 | [diff] [blame] | 112 | |
| 113 | (Please see Documentation/power/pci.txt for descriptions |
| 114 | of PCI Power Management and the related functions.) |
| 115 | |
Grant Grundler | 74da15e | 2006-12-25 01:06:35 -0700 | [diff] [blame] | 116 | shutdown Hook into reboot_notifier_list (kernel/sys.c). |
| 117 | Intended to stop any idling DMA operations. |
| 118 | Useful for enabling wake-on-lan (NIC) or changing |
| 119 | the power state of a device before reboot. |
| 120 | e.g. drivers/net/e100.c. |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 121 | |
Grant Grundler | 74da15e | 2006-12-25 01:06:35 -0700 | [diff] [blame] | 122 | err_handler See Documentation/pci-error-recovery.txt |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 123 | |
Grant Grundler | 74da15e | 2006-12-25 01:06:35 -0700 | [diff] [blame] | 124 | |
| 125 | The ID table is an array of struct pci_device_id entries ending with an |
Andrew Morton | 9f9351b | 2008-03-10 11:43:34 -0700 | [diff] [blame] | 126 | all-zero entry; use of the macro DEFINE_PCI_DEVICE_TABLE is the preferred |
Jonas Bonn | 90a1ba0 | 2008-02-22 11:02:21 +0100 | [diff] [blame] | 127 | method of declaring the table. Each entry consists of: |
Grant Grundler | 74da15e | 2006-12-25 01:06:35 -0700 | [diff] [blame] | 128 | |
| 129 | vendor,device Vendor and device ID to match (or PCI_ANY_ID) |
| 130 | |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 131 | subvendor, Subsystem vendor and device ID to match (or PCI_ANY_ID) |
Grant Grundler | 74da15e | 2006-12-25 01:06:35 -0700 | [diff] [blame] | 132 | subdevice, |
| 133 | |
| 134 | class Device class, subclass, and "interface" to match. |
| 135 | See Appendix D of the PCI Local Bus Spec or |
| 136 | include/linux/pci_ids.h for a full list of classes. |
| 137 | Most drivers do not need to specify class/class_mask |
| 138 | as vendor/device is normally sufficient. |
| 139 | |
| 140 | class_mask limit which sub-fields of the class field are compared. |
| 141 | See drivers/scsi/sym53c8xx_2/ for example of usage. |
| 142 | |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 143 | driver_data Data private to the driver. |
Grant Grundler | 74da15e | 2006-12-25 01:06:35 -0700 | [diff] [blame] | 144 | Most drivers don't need to use driver_data field. |
| 145 | Best practice is to use driver_data as an index |
| 146 | into a static list of equivalent device types, |
| 147 | instead of using it as a pointer. |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 148 | |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 149 | |
Grant Grundler | 74da15e | 2006-12-25 01:06:35 -0700 | [diff] [blame] | 150 | Most drivers only need PCI_DEVICE() or PCI_DEVICE_CLASS() to set up |
| 151 | a pci_device_id table. |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 152 | |
Grant Grundler | 74da15e | 2006-12-25 01:06:35 -0700 | [diff] [blame] | 153 | New PCI IDs may be added to a device driver pci_ids table at runtime |
| 154 | as shown below: |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 155 | |
| 156 | echo "vendor device subvendor subdevice class class_mask driver_data" > \ |
Grant Grundler | 74da15e | 2006-12-25 01:06:35 -0700 | [diff] [blame] | 157 | /sys/bus/pci/drivers/{driver}/new_id |
| 158 | |
| 159 | All fields are passed in as hexadecimal values (no leading 0x). |
Jean Delvare | 6ba1863 | 2007-04-07 17:21:28 +0200 | [diff] [blame] | 160 | The vendor and device fields are mandatory, the others are optional. Users |
| 161 | need pass only as many optional fields as necessary: |
| 162 | o subvendor and subdevice fields default to PCI_ANY_ID (FFFFFFFF) |
Grant Grundler | 74da15e | 2006-12-25 01:06:35 -0700 | [diff] [blame] | 163 | o class and classmask fields default to 0 |
| 164 | o driver_data defaults to 0UL. |
| 165 | |
| 166 | Once added, the driver probe routine will be invoked for any unclaimed |
| 167 | PCI devices listed in its (newly updated) pci_ids list. |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 168 | |
| 169 | When the driver exits, it just calls pci_unregister_driver() and the PCI layer |
| 170 | automatically calls the remove hook for all devices handled by the driver. |
| 171 | |
Grant Grundler | 74da15e | 2006-12-25 01:06:35 -0700 | [diff] [blame] | 172 | |
| 173 | 1.1 "Attributes" for driver functions/data |
| 174 | |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 175 | Please mark the initialization and cleanup functions where appropriate |
| 176 | (the corresponding macros are defined in <linux/init.h>): |
| 177 | |
| 178 | __init Initialization code. Thrown away after the driver |
| 179 | initializes. |
| 180 | __exit Exit code. Ignored for non-modular drivers. |
Grant Grundler | 74da15e | 2006-12-25 01:06:35 -0700 | [diff] [blame] | 181 | |
| 182 | |
| 183 | __devinit Device initialization code. |
| 184 | Identical to __init if the kernel is not compiled |
| 185 | with CONFIG_HOTPLUG, normal function otherwise. |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 186 | __devexit The same for __exit. |
| 187 | |
Grant Grundler | 74da15e | 2006-12-25 01:06:35 -0700 | [diff] [blame] | 188 | Tips on when/where to use the above attributes: |
| 189 | o The module_init()/module_exit() functions (and all |
| 190 | initialization functions called _only_ from these) |
| 191 | should be marked __init/__exit. |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 192 | |
Grant Grundler | 74da15e | 2006-12-25 01:06:35 -0700 | [diff] [blame] | 193 | o Do not mark the struct pci_driver. |
| 194 | |
Jonas Bonn | 90a1ba0 | 2008-02-22 11:02:21 +0100 | [diff] [blame] | 195 | o The ID table array should be marked __devinitconst; this is done |
Andrew Morton | 9f9351b | 2008-03-10 11:43:34 -0700 | [diff] [blame] | 196 | automatically if the table is declared with DEFINE_PCI_DEVICE_TABLE(). |
Grant Grundler | 74da15e | 2006-12-25 01:06:35 -0700 | [diff] [blame] | 197 | |
| 198 | o The probe() and remove() functions should be marked __devinit |
| 199 | and __devexit respectively. All initialization functions |
| 200 | exclusively called by the probe() routine, can be marked __devinit. |
| 201 | Ditto for remove() and __devexit. |
| 202 | |
Grant Grundler | 26ba05e | 2007-02-11 00:04:04 -0700 | [diff] [blame] | 203 | o If mydriver_remove() is marked with __devexit(), then all address |
| 204 | references to mydriver_remove must use __devexit_p(mydriver_remove) |
Grant Grundler | 74da15e | 2006-12-25 01:06:35 -0700 | [diff] [blame] | 205 | (in the struct pci_driver declaration for example). |
| 206 | __devexit_p() will generate the function name _or_ NULL if the |
| 207 | function will be discarded. For an example, see drivers/net/tg3.c. |
| 208 | |
| 209 | o Do NOT mark a function if you are not sure which mark to use. |
| 210 | Better to not mark the function than mark the function wrong. |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 211 | |
| 212 | |
Grant Grundler | 74da15e | 2006-12-25 01:06:35 -0700 | [diff] [blame] | 213 | |
| 214 | 2. How to find PCI devices manually |
| 215 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 216 | |
| 217 | PCI drivers should have a really good reason for not using the |
| 218 | pci_register_driver() interface to search for PCI devices. |
| 219 | The main reason PCI devices are controlled by multiple drivers |
| 220 | is because one PCI device implements several different HW services. |
| 221 | E.g. combined serial/parallel port/floppy controller. |
| 222 | |
| 223 | A manual search may be performed using the following constructs: |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 224 | |
| 225 | Searching by vendor and device ID: |
| 226 | |
| 227 | struct pci_dev *dev = NULL; |
| 228 | while (dev = pci_get_device(VENDOR_ID, DEVICE_ID, dev)) |
| 229 | configure_device(dev); |
| 230 | |
| 231 | Searching by class ID (iterate in a similar way): |
| 232 | |
| 233 | pci_get_class(CLASS_ID, dev) |
| 234 | |
| 235 | Searching by both vendor/device and subsystem vendor/device ID: |
| 236 | |
Grant Grundler | 74da15e | 2006-12-25 01:06:35 -0700 | [diff] [blame] | 237 | pci_get_subsys(VENDOR_ID,DEVICE_ID, SUBSYS_VENDOR_ID, SUBSYS_DEVICE_ID, dev). |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 238 | |
Grant Grundler | 74da15e | 2006-12-25 01:06:35 -0700 | [diff] [blame] | 239 | You can use the constant PCI_ANY_ID as a wildcard replacement for |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 240 | VENDOR_ID or DEVICE_ID. This allows searching for any device from a |
| 241 | specific vendor, for example. |
| 242 | |
Grant Grundler | 74da15e | 2006-12-25 01:06:35 -0700 | [diff] [blame] | 243 | These functions are hotplug-safe. They increment the reference count on |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 244 | the pci_dev that they return. You must eventually (possibly at module unload) |
| 245 | decrement the reference count on these devices by calling pci_dev_put(). |
| 246 | |
| 247 | |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 248 | |
Grant Grundler | 74da15e | 2006-12-25 01:06:35 -0700 | [diff] [blame] | 249 | 3. Device Initialization Steps |
| 250 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 251 | |
Grant Grundler | 74da15e | 2006-12-25 01:06:35 -0700 | [diff] [blame] | 252 | As noted in the introduction, most PCI drivers need the following steps |
| 253 | for device initialization: |
| 254 | |
| 255 | Enable the device |
| 256 | Request MMIO/IOP resources |
| 257 | Set the DMA mask size (for both coherent and streaming DMA) |
| 258 | Allocate and initialize shared control data (pci_allocate_coherent()) |
| 259 | Access device configuration space (if needed) |
| 260 | Register IRQ handler (request_irq()) |
| 261 | Initialize non-PCI (i.e. LAN/SCSI/etc parts of the chip) |
| 262 | Enable DMA/processing engines. |
| 263 | |
| 264 | The driver can access PCI config space registers at any time. |
| 265 | (Well, almost. When running BIST, config space can go away...but |
| 266 | that will just result in a PCI Bus Master Abort and config reads |
| 267 | will return garbage). |
| 268 | |
| 269 | |
| 270 | 3.1 Enable the PCI device |
| 271 | ~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 272 | Before touching any device registers, the driver needs to enable |
| 273 | the PCI device by calling pci_enable_device(). This will: |
| 274 | o wake up the device if it was in suspended state, |
| 275 | o allocate I/O and memory regions of the device (if BIOS did not), |
| 276 | o allocate an IRQ (if BIOS did not). |
| 277 | |
| 278 | NOTE: pci_enable_device() can fail! Check the return value. |
Grant Grundler | 74da15e | 2006-12-25 01:06:35 -0700 | [diff] [blame] | 279 | |
| 280 | [ OS BUG: we don't check resource allocations before enabling those |
| 281 | resources. The sequence would make more sense if we called |
| 282 | pci_request_resources() before calling pci_enable_device(). |
| 283 | Currently, the device drivers can't detect the bug when when two |
| 284 | devices have been allocated the same range. This is not a common |
| 285 | problem and unlikely to get fixed soon. |
| 286 | |
| 287 | This has been discussed before but not changed as of 2.6.19: |
| 288 | http://lkml.org/lkml/2006/3/2/194 |
| 289 | ] |
| 290 | |
| 291 | pci_set_master() will enable DMA by setting the bus master bit |
| 292 | in the PCI_COMMAND register. It also fixes the latency timer value if |
| 293 | it's set to something bogus by the BIOS. |
| 294 | |
| 295 | If the PCI device can use the PCI Memory-Write-Invalidate transaction, |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 296 | call pci_set_mwi(). This enables the PCI_COMMAND bit for Mem-Wr-Inval |
| 297 | and also ensures that the cache line size register is set correctly. |
Grant Grundler | 74da15e | 2006-12-25 01:06:35 -0700 | [diff] [blame] | 298 | Check the return value of pci_set_mwi() as not all architectures |
Randy Dunlap | 694625c | 2007-07-09 11:55:54 -0700 | [diff] [blame] | 299 | or chip-sets may support Memory-Write-Invalidate. Alternatively, |
| 300 | if Mem-Wr-Inval would be nice to have but is not required, call |
| 301 | pci_try_set_mwi() to have the system do its best effort at enabling |
| 302 | Mem-Wr-Inval. |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 303 | |
Grant Grundler | 74da15e | 2006-12-25 01:06:35 -0700 | [diff] [blame] | 304 | |
| 305 | 3.2 Request MMIO/IOP resources |
| 306 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 307 | Memory (MMIO), and I/O port addresses should NOT be read directly |
| 308 | from the PCI device config space. Use the values in the pci_dev structure |
| 309 | as the PCI "bus address" might have been remapped to a "host physical" |
| 310 | address by the arch/chip-set specific kernel support. |
| 311 | |
| 312 | See Documentation/IO-mapping.txt for how to access device registers |
| 313 | or device memory. |
| 314 | |
| 315 | The device driver needs to call pci_request_region() to verify |
| 316 | no other device is already using the same address resource. |
| 317 | Conversely, drivers should call pci_release_region() AFTER |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 318 | calling pci_disable_device(). |
Grant Grundler | 74da15e | 2006-12-25 01:06:35 -0700 | [diff] [blame] | 319 | The idea is to prevent two devices colliding on the same address range. |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 320 | |
Grant Grundler | 74da15e | 2006-12-25 01:06:35 -0700 | [diff] [blame] | 321 | [ See OS BUG comment above. Currently (2.6.19), The driver can only |
| 322 | determine MMIO and IO Port resource availability _after_ calling |
| 323 | pci_enable_device(). ] |
| 324 | |
| 325 | Generic flavors of pci_request_region() are request_mem_region() |
| 326 | (for MMIO ranges) and request_region() (for IO Port ranges). |
| 327 | Use these for address resources that are not described by "normal" PCI |
| 328 | BARs. |
| 329 | |
| 330 | Also see pci_request_selected_regions() below. |
| 331 | |
| 332 | |
| 333 | 3.3 Set the DMA mask size |
| 334 | ~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 335 | [ If anything below doesn't make sense, please refer to |
| 336 | Documentation/DMA-API.txt. This section is just a reminder that |
| 337 | drivers need to indicate DMA capabilities of the device and is not |
| 338 | an authoritative source for DMA interfaces. ] |
| 339 | |
| 340 | While all drivers should explicitly indicate the DMA capability |
| 341 | (e.g. 32 or 64 bit) of the PCI bus master, devices with more than |
| 342 | 32-bit bus master capability for streaming data need the driver |
| 343 | to "register" this capability by calling pci_set_dma_mask() with |
| 344 | appropriate parameters. In general this allows more efficient DMA |
| 345 | on systems where System RAM exists above 4G _physical_ address. |
| 346 | |
| 347 | Drivers for all PCI-X and PCIe compliant devices must call |
| 348 | pci_set_dma_mask() as they are 64-bit DMA devices. |
| 349 | |
| 350 | Similarly, drivers must also "register" this capability if the device |
| 351 | can directly address "consistent memory" in System RAM above 4G physical |
| 352 | address by calling pci_set_consistent_dma_mask(). |
| 353 | Again, this includes drivers for all PCI-X and PCIe compliant devices. |
| 354 | Many 64-bit "PCI" devices (before PCI-X) and some PCI-X devices are |
| 355 | 64-bit DMA capable for payload ("streaming") data but not control |
| 356 | ("consistent") data. |
| 357 | |
| 358 | |
| 359 | 3.4 Setup shared control data |
| 360 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 361 | Once the DMA masks are set, the driver can allocate "consistent" (a.k.a. shared) |
| 362 | memory. See Documentation/DMA-API.txt for a full description of |
| 363 | the DMA APIs. This section is just a reminder that it needs to be done |
| 364 | before enabling DMA on the device. |
| 365 | |
| 366 | |
| 367 | 3.5 Initialize device registers |
| 368 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 369 | Some drivers will need specific "capability" fields programmed |
| 370 | or other "vendor specific" register initialized or reset. |
| 371 | E.g. clearing pending interrupts. |
| 372 | |
| 373 | |
| 374 | 3.6 Register IRQ handler |
| 375 | ~~~~~~~~~~~~~~~~~~~~~~~~ |
Michael Opdenacker | 59c5159 | 2007-05-09 08:57:56 +0200 | [diff] [blame] | 376 | While calling request_irq() is the last step described here, |
Grant Grundler | 74da15e | 2006-12-25 01:06:35 -0700 | [diff] [blame] | 377 | this is often just another intermediate step to initialize a device. |
| 378 | This step can often be deferred until the device is opened for use. |
| 379 | |
| 380 | All interrupt handlers for IRQ lines should be registered with IRQF_SHARED |
| 381 | and use the devid to map IRQs to devices (remember that all PCI IRQ lines |
| 382 | can be shared). |
| 383 | |
| 384 | request_irq() will associate an interrupt handler and device handle |
| 385 | with an interrupt number. Historically interrupt numbers represent |
| 386 | IRQ lines which run from the PCI device to the Interrupt controller. |
| 387 | With MSI and MSI-X (more below) the interrupt number is a CPU "vector". |
| 388 | |
| 389 | request_irq() also enables the interrupt. Make sure the device is |
| 390 | quiesced and does not have any interrupts pending before registering |
| 391 | the interrupt handler. |
| 392 | |
| 393 | MSI and MSI-X are PCI capabilities. Both are "Message Signaled Interrupts" |
| 394 | which deliver interrupts to the CPU via a DMA write to a Local APIC. |
| 395 | The fundamental difference between MSI and MSI-X is how multiple |
| 396 | "vectors" get allocated. MSI requires contiguous blocks of vectors |
| 397 | while MSI-X can allocate several individual ones. |
| 398 | |
| 399 | MSI capability can be enabled by calling pci_enable_msi() or |
| 400 | pci_enable_msix() before calling request_irq(). This causes |
| 401 | the PCI support to program CPU vector data into the PCI device |
| 402 | capability registers. |
| 403 | |
| 404 | If your PCI device supports both, try to enable MSI-X first. |
| 405 | Only one can be enabled at a time. Many architectures, chip-sets, |
| 406 | or BIOSes do NOT support MSI or MSI-X and the call to pci_enable_msi/msix |
| 407 | will fail. This is important to note since many drivers have |
| 408 | two (or more) interrupt handlers: one for MSI/MSI-X and another for IRQs. |
| 409 | They choose which handler to register with request_irq() based on the |
| 410 | return value from pci_enable_msi/msix(). |
| 411 | |
| 412 | There are (at least) two really good reasons for using MSI: |
| 413 | 1) MSI is an exclusive interrupt vector by definition. |
| 414 | This means the interrupt handler doesn't have to verify |
| 415 | its device caused the interrupt. |
| 416 | |
| 417 | 2) MSI avoids DMA/IRQ race conditions. DMA to host memory is guaranteed |
| 418 | to be visible to the host CPU(s) when the MSI is delivered. This |
| 419 | is important for both data coherency and avoiding stale control data. |
| 420 | This guarantee allows the driver to omit MMIO reads to flush |
| 421 | the DMA stream. |
| 422 | |
| 423 | See drivers/infiniband/hw/mthca/ or drivers/net/tg3.c for examples |
| 424 | of MSI/MSI-X usage. |
| 425 | |
| 426 | |
| 427 | |
| 428 | 4. PCI device shutdown |
| 429 | ~~~~~~~~~~~~~~~~~~~~~~~ |
| 430 | |
| 431 | When a PCI device driver is being unloaded, most of the following |
| 432 | steps need to be performed: |
| 433 | |
| 434 | Disable the device from generating IRQs |
| 435 | Release the IRQ (free_irq()) |
| 436 | Stop all DMA activity |
| 437 | Release DMA buffers (both streaming and consistent) |
| 438 | Unregister from other subsystems (e.g. scsi or netdev) |
| 439 | Disable device from responding to MMIO/IO Port addresses |
| 440 | Release MMIO/IO Port resource(s) |
| 441 | |
| 442 | |
| 443 | 4.1 Stop IRQs on the device |
| 444 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 445 | How to do this is chip/device specific. If it's not done, it opens |
| 446 | the possibility of a "screaming interrupt" if (and only if) |
| 447 | the IRQ is shared with another device. |
| 448 | |
| 449 | When the shared IRQ handler is "unhooked", the remaining devices |
| 450 | using the same IRQ line will still need the IRQ enabled. Thus if the |
| 451 | "unhooked" device asserts IRQ line, the system will respond assuming |
| 452 | it was one of the remaining devices asserted the IRQ line. Since none |
| 453 | of the other devices will handle the IRQ, the system will "hang" until |
| 454 | it decides the IRQ isn't going to get handled and masks the IRQ (100,000 |
| 455 | iterations later). Once the shared IRQ is masked, the remaining devices |
| 456 | will stop functioning properly. Not a nice situation. |
| 457 | |
| 458 | This is another reason to use MSI or MSI-X if it's available. |
| 459 | MSI and MSI-X are defined to be exclusive interrupts and thus |
| 460 | are not susceptible to the "screaming interrupt" problem. |
| 461 | |
| 462 | |
| 463 | 4.2 Release the IRQ |
| 464 | ~~~~~~~~~~~~~~~~~~~ |
| 465 | Once the device is quiesced (no more IRQs), one can call free_irq(). |
| 466 | This function will return control once any pending IRQs are handled, |
| 467 | "unhook" the drivers IRQ handler from that IRQ, and finally release |
| 468 | the IRQ if no one else is using it. |
| 469 | |
| 470 | |
| 471 | 4.3 Stop all DMA activity |
| 472 | ~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 473 | It's extremely important to stop all DMA operations BEFORE attempting |
| 474 | to deallocate DMA control data. Failure to do so can result in memory |
| 475 | corruption, hangs, and on some chip-sets a hard crash. |
| 476 | |
| 477 | Stopping DMA after stopping the IRQs can avoid races where the |
| 478 | IRQ handler might restart DMA engines. |
| 479 | |
| 480 | While this step sounds obvious and trivial, several "mature" drivers |
| 481 | didn't get this step right in the past. |
| 482 | |
| 483 | |
| 484 | 4.4 Release DMA buffers |
| 485 | ~~~~~~~~~~~~~~~~~~~~~~~ |
| 486 | Once DMA is stopped, clean up streaming DMA first. |
| 487 | I.e. unmap data buffers and return buffers to "upstream" |
| 488 | owners if there is one. |
| 489 | |
| 490 | Then clean up "consistent" buffers which contain the control data. |
| 491 | |
| 492 | See Documentation/DMA-API.txt for details on unmapping interfaces. |
| 493 | |
| 494 | |
| 495 | 4.5 Unregister from other subsystems |
| 496 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 497 | Most low level PCI device drivers support some other subsystem |
| 498 | like USB, ALSA, SCSI, NetDev, Infiniband, etc. Make sure your |
| 499 | driver isn't losing resources from that other subsystem. |
| 500 | If this happens, typically the symptom is an Oops (panic) when |
| 501 | the subsystem attempts to call into a driver that has been unloaded. |
| 502 | |
| 503 | |
| 504 | 4.6 Disable Device from responding to MMIO/IO Port addresses |
| 505 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 506 | io_unmap() MMIO or IO Port resources and then call pci_disable_device(). |
| 507 | This is the symmetric opposite of pci_enable_device(). |
| 508 | Do not access device registers after calling pci_disable_device(). |
| 509 | |
| 510 | |
| 511 | 4.7 Release MMIO/IO Port Resource(s) |
| 512 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 513 | Call pci_release_region() to mark the MMIO or IO Port range as available. |
| 514 | Failure to do so usually results in the inability to reload the driver. |
| 515 | |
| 516 | |
| 517 | |
| 518 | 5. How to access PCI config space |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 519 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
Grant Grundler | 74da15e | 2006-12-25 01:06:35 -0700 | [diff] [blame] | 520 | |
| 521 | You can use pci_(read|write)_config_(byte|word|dword) to access the config |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 522 | space of a device represented by struct pci_dev *. All these functions return 0 |
| 523 | when successful or an error code (PCIBIOS_...) which can be translated to a text |
| 524 | string by pcibios_strerror. Most drivers expect that accesses to valid PCI |
| 525 | devices don't fail. |
| 526 | |
Grant Grundler | 74da15e | 2006-12-25 01:06:35 -0700 | [diff] [blame] | 527 | If you don't have a struct pci_dev available, you can call |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 528 | pci_bus_(read|write)_config_(byte|word|dword) to access a given device |
| 529 | and function on that bus. |
| 530 | |
Grant Grundler | 74da15e | 2006-12-25 01:06:35 -0700 | [diff] [blame] | 531 | If you access fields in the standard portion of the config header, please |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 532 | use symbolic names of locations and bits declared in <linux/pci.h>. |
| 533 | |
Grant Grundler | 74da15e | 2006-12-25 01:06:35 -0700 | [diff] [blame] | 534 | If you need to access Extended PCI Capability registers, just call |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 535 | pci_find_capability() for the particular capability and it will find the |
| 536 | corresponding register block for you. |
| 537 | |
| 538 | |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 539 | |
| 540 | 6. Other interesting functions |
| 541 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
Grant Grundler | 74da15e | 2006-12-25 01:06:35 -0700 | [diff] [blame] | 542 | |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 543 | pci_find_slot() Find pci_dev corresponding to given bus and |
| 544 | slot numbers. |
| 545 | pci_set_power_state() Set PCI Power Management state (0=D0 ... 3=D3) |
| 546 | pci_find_capability() Find specified capability in device's capability |
| 547 | list. |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 548 | pci_resource_start() Returns bus start address for a given PCI region |
| 549 | pci_resource_end() Returns bus end address for a given PCI region |
| 550 | pci_resource_len() Returns the byte length of a PCI region |
| 551 | pci_set_drvdata() Set private driver data pointer for a pci_dev |
| 552 | pci_get_drvdata() Return private driver data pointer for a pci_dev |
| 553 | pci_set_mwi() Enable Memory-Write-Invalidate transactions. |
| 554 | pci_clear_mwi() Disable Memory-Write-Invalidate transactions. |
| 555 | |
| 556 | |
Grant Grundler | 74da15e | 2006-12-25 01:06:35 -0700 | [diff] [blame] | 557 | |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 558 | 7. Miscellaneous hints |
| 559 | ~~~~~~~~~~~~~~~~~~~~~~ |
Grant Grundler | 74da15e | 2006-12-25 01:06:35 -0700 | [diff] [blame] | 560 | |
| 561 | When displaying PCI device names to the user (for example when a driver wants |
| 562 | to tell the user what card has it found), please use pci_name(pci_dev). |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 563 | |
| 564 | Always refer to the PCI devices by a pointer to the pci_dev structure. |
| 565 | All PCI layer functions use this identification and it's the only |
| 566 | reasonable one. Don't use bus/slot/function numbers except for very |
| 567 | special purposes -- on systems with multiple primary buses their semantics |
| 568 | can be pretty complex. |
| 569 | |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 570 | Don't try to turn on Fast Back to Back writes in your driver. All devices |
| 571 | on the bus need to be capable of doing it, so this is something which needs |
| 572 | to be handled by platform and generic code, not individual drivers. |
| 573 | |
| 574 | |
Grant Grundler | 74da15e | 2006-12-25 01:06:35 -0700 | [diff] [blame] | 575 | |
Ingo Oeser | 9b860b8 | 2006-04-18 11:20:55 +0200 | [diff] [blame] | 576 | 8. Vendor and device identifications |
| 577 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
Ingo Oeser | 9b860b8 | 2006-04-18 11:20:55 +0200 | [diff] [blame] | 578 | |
Grant Grundler | 74da15e | 2006-12-25 01:06:35 -0700 | [diff] [blame] | 579 | One is not not required to add new device ids to include/linux/pci_ids.h. |
| 580 | Please add PCI_VENDOR_ID_xxx for vendors and a hex constant for device ids. |
Ingo Oeser | 9b860b8 | 2006-04-18 11:20:55 +0200 | [diff] [blame] | 581 | |
Grant Grundler | 74da15e | 2006-12-25 01:06:35 -0700 | [diff] [blame] | 582 | PCI_VENDOR_ID_xxx constants are re-used. The device ids are arbitrary |
| 583 | hex numbers (vendor controlled) and normally used only in a single |
| 584 | location, the pci_device_id table. |
| 585 | |
| 586 | Please DO submit new vendor/device ids to pciids.sourceforge.net project. |
| 587 | |
| 588 | |
Ingo Oeser | 9b860b8 | 2006-04-18 11:20:55 +0200 | [diff] [blame] | 589 | |
| 590 | 9. Obsolete functions |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 591 | ~~~~~~~~~~~~~~~~~~~~~ |
Grant Grundler | 74da15e | 2006-12-25 01:06:35 -0700 | [diff] [blame] | 592 | |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 593 | There are several functions which you might come across when trying to |
| 594 | port an old driver to the new PCI interface. They are no longer present |
| 595 | in the kernel as they aren't compatible with hotplug or PCI domains or |
| 596 | having sane locking. |
| 597 | |
Grant Grundler | 74da15e | 2006-12-25 01:06:35 -0700 | [diff] [blame] | 598 | pci_find_device() Superseded by pci_get_device() |
| 599 | pci_find_subsys() Superseded by pci_get_subsys() |
| 600 | pci_find_slot() Superseded by pci_get_slot() |
| 601 | |
| 602 | |
| 603 | The alternative is the traditional PCI device driver that walks PCI |
| 604 | device lists. This is still possible but discouraged. |
| 605 | |
| 606 | |
| 607 | |
Grant Grundler | d48b5d3 | 2007-12-24 00:08:51 -0700 | [diff] [blame] | 608 | 10. MMIO Space and "Write Posting" |
Grant Grundler | 74da15e | 2006-12-25 01:06:35 -0700 | [diff] [blame] | 609 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 610 | |
| 611 | Converting a driver from using I/O Port space to using MMIO space |
| 612 | often requires some additional changes. Specifically, "write posting" |
| 613 | needs to be handled. Many drivers (e.g. tg3, acenic, sym53c8xx_2) |
| 614 | already do this. I/O Port space guarantees write transactions reach the PCI |
| 615 | device before the CPU can continue. Writes to MMIO space allow the CPU |
| 616 | to continue before the transaction reaches the PCI device. HW weenies |
| 617 | call this "Write Posting" because the write completion is "posted" to |
| 618 | the CPU before the transaction has reached its destination. |
| 619 | |
| 620 | Thus, timing sensitive code should add readl() where the CPU is |
| 621 | expected to wait before doing other work. The classic "bit banging" |
| 622 | sequence works fine for I/O Port space: |
| 623 | |
| 624 | for (i = 8; --i; val >>= 1) { |
| 625 | outb(val & 1, ioport_reg); /* write bit */ |
| 626 | udelay(10); |
| 627 | } |
| 628 | |
| 629 | The same sequence for MMIO space should be: |
| 630 | |
| 631 | for (i = 8; --i; val >>= 1) { |
| 632 | writeb(val & 1, mmio_reg); /* write bit */ |
| 633 | readb(safe_mmio_reg); /* flush posted write */ |
| 634 | udelay(10); |
| 635 | } |
| 636 | |
| 637 | It is important that "safe_mmio_reg" not have any side effects that |
| 638 | interferes with the correct operation of the device. |
| 639 | |
| 640 | Another case to watch out for is when resetting a PCI device. Use PCI |
| 641 | Configuration space reads to flush the writel(). This will gracefully |
| 642 | handle the PCI master abort on all platforms if the PCI device is |
| 643 | expected to not respond to a readl(). Most x86 platforms will allow |
| 644 | MMIO reads to master abort (a.k.a. "Soft Fail") and return garbage |
| 645 | (e.g. ~0). But many RISC platforms will crash (a.k.a."Hard Fail"). |
| 646 | |