|
|
This topic is intended to aid you in porting and multithreading HBA drivers for systems supporting multiprocessing. First discussed is the uniprocessor and multiprocessor design of the SDI HBA drivers. In subsequent topics, guidelines and hints for writing and porting HBA drivers are provided.
The Device Driver Interface (DDI) defines a standard applying to both uniprocessor environments, referred to as UP, where there is just a single processor executing code, and multiprocessor environments, referred to as MP, where there may be one or more CPUs operating in the same system simultaneously.
Kernels can be built as either UP or MP, depending on the level of support necessary for the hardware platform. Similarly, SDI HBA drivers can run as either UP or MP. SDI HBA drivers in a UP kernel run in UP mode only. SDI HBA drivers in an MP kernel can run as UP or MP mode, depending on the hardware platform and level of locking support provided by the driver. HBA drivers that have been multithreaded, with internal locks that protect critical data structures and sensitive code regions, can run as MP. HBA drivers that have no internal locks, but depend on external protection, are run in UP mode, no matter what the hardware platform.
Performance improvements over UP drivers will vary depending on the granularity of locking implemented and the number of instances of the driver.
Before an HBA driver can run in multiprocessor mode, it must declare that it is ``MP safe''. For DDI 7 and earlier versions, this is done by setting the HBA_MP bit in its devflag(D1) and referencing that value from its hba_info(D4sdi) declaration. For HBA driver xxx, this would be accomplished as follows:
static int xxx_devflag = HBA_MP;This flag should be set only in drivers that have been multithreaded. Its purpose is to prevent the kernel from running a driver that is not ``MP safe'' in MP mode. Such drivers must run in UP mode or risk data corruption, resulting from the unprotected entry of critical sections of code.HBA_INFO(xxx, &xxx_devflag, max_xfer);
When run in UP mode, HBA drivers are bound to a specific CPU. This binding occurs just before any of the HBA's entry points are called from SDI. In addition, interrupts for devices supported by the driver are routed to the specified CPU.
By default, HBA drivers running in UP mode are bound to CPU 0. This default can be overridden by specifying the CPU number in the optional cpu field of the HBA driver's System(DSP/4dsp) file.
The rules for binding are as follows:
Following are guidelines for multithreading HBA drivers. It might not cover all aspects or details of locking, since not all HBA drivers are similar. However, the general concepts should be found helpful. For additional information, see ``Critical code section'' and ``Multithreaded drivers'' for additional information.
Many multiprocessing hardware platforms provide symmetric I/O and distributed interrupt capability. To make the most advantage of such hardware, the HBA driver locking strategy should provide as fine a level of locking as possible. Since HBA drivers are performing actions directed to a specific device, a reasonable locking approach is based on a per-device locking strategy.
The data structure associated with a device usually consists of a logical unit queue where jobs associated with the device are queued before being sent. A lock-per-device queue provides protection for the device queues and allows parallel operation among devices. The lock need only be held during the critical operation of adding and removing jobs from the queue. At other times, the job is known only locally and needs no protection. To make this lock optimal, the lock might be acquired by one routine, and freed by another. This typically is not a recommended style of locking; however, in certain cases such as this, it is necessary.
The second major critical region of the HBA driver is the interface to the controller. Only one CPU engine should be sending or checking for pending interrupts at one time. A per-controller lock is required to protect this region.
The device queue lock and controller lock are independent of one another and need not be held at the same time, so the lock hierarchy level should be the same for these locks. Some HBA drivers send the next job out in the interrupt routine after having processed the interrupt and completing a job. If this is done, the HBA driver interrupt routine may have to be modified slightly to avoid acquiring the device queue lock while the controller lock is held. This is accomplished by looping for pending interrupts, keeping a local list of devices that need to be serviced. A second loop traverses the local list of devices, dequeuing the next job for the device and sending it out.
The HBA driver may have other resources needing protection, such as controller control blocks. If a free list of the resource is maintained, a global lock is needed in the allocation and deallocation routines that manage the free list.
Following are several conventions and recommendations when multithreading SDI drivers.
Multithreaded HBA should use the sb(D4sdi) structure to store request sense data when a fault occurs. Previously, HBAs would cache the data on their local device queue and wait for a request from sd01 for the data. Passing the request sense data with the sb structure on the callback to sd01 ensures that the data will not be overwritten by a subsequent fault condition.
This strategy is not required for single-threaded drivers or for multithreaded drivers running in bound mode, but it is essential for all multithreaded drivers running in MP mode.
#define TARGET_HIER_BASE 5 #define SDI_HIER_BASE 15 #define HBA_HIER_BASE 20