atcp_kapi_lock

ArrayTCP and KernelAPI Locking Strategy
The goal of this locking design is to allow any one ArrayTCP thread to suspend all other ArrayTCP threads in the event a KernelAPI operation (configuration change) is received. Stopping the other threads is necessary because configuration changes often invalidate the running state. The primary consideration of the design is how to maintain a high level of performance, as a poorly designed locking strategy can impose significant penalties. The initial solution used a single sx lock as the synchronization object. Each ArrayTCP thread would acquire a shared lock on entrance, and release it on exit. If a KernelAPI packet was received, that thread would release the shared lock and acquire an exclusive lock. As other ArrayTCP threads finished, they would release their shared locks and be blocked by the exclusive lock. When the last thread released its shared lock, the thread waiting on the exclusive lock would resume and process the KernelAPI traffic. After it finished, this thread would downgrade to and then release a shared lock, allowing the blocked threads to resume. Although effective, this approach resulted in a 40% hit to performance due to the constant locking and unlocking of the sx lock.
For the new design, four global variables are used as synchronization objects: ∙ A shared/exclusive lock config_sx held by ArrayTCP threads.
∙An integer value kAPI_flag used as a checking flag.
∙ A conditional variable config_cv used as a signaler.
∙ A mutex lock cv_mtx used to protect the config_cv.
The key realization leading to the new design is that the penalty lies in constantly locking and unlocking the sx lock, not in actually having the lock itself. Therefore the new design holds a read lock on the sx lock in the common case, rather than not holding it. Only when a KernelAPI operation is detected do the threads release then reacquire their locks. This flip in logic results in the common case not having any locking operations, and is identical to the case where no locks are used at all. However, it does add the requirement for another method of synchronizing the threads, resulting in the three added variables. The usage of these variables and the synchronization details are shown below.
State transition diagram for ArrayTCP/KernelAPI locking.。