@ -19,11 +19,11 @@ The idea of making it easier to interact with low-level hardware from a \gls{PC}
\caption{\label{fig:rpi}Raspberry Pi minicomputers}
\end{figure}
The Raspberry Pi's \gls{GPIO} header, a row of pins which can be directly controlled by user applications running on the minicomputer, was one of the inspirations behind GEX. It can be controlled using C and Python (among others) and offers \gls{GPIO}, \gls{SPI}, \gls{I2C}, \gls{UART} and \gls{PWM}, with other protocols and functions easy to emulate thanks to the high speed of the system processor.
The Raspberry Pi's \gls{GPIO} header, a row of pins which can be directly controlled by user applications running on the minicomputer, was one of the inspirations behind GEX. It can be controlled using C and Python (among others) and offers \gls{GPIO}, \gls{SPI}, \gls{I2C}, \gls{UART}, and \gls{PWM}, with other protocols and functions easy to emulate thanks to the high speed of the system processor.
The Raspberry Pi is used in schools as a low-cost PC alternative that encourage students' interest in \gls{STEM}. The board is often built into more permanent projects that make use of its powerful processor, such as wildlife camera traps, fish feeders etc.
The Raspberry Pi could be used for the same quick evaluations or experiments we want to perform with GEX, however they would either have to be performed directly on the minicomputer, with an attached monitor and a keyboard, or use some form of remote access (e.g., \gls{SSH}, screen sharing).
The Raspberry Pi could be used for the same quick evaluations or experiments we want to perform with GEX, however they would either have to be performed directly on the minicomputer, with an attached monitor and a keyboard, or use some form of remote access (e.g., \gls{SSH}, or screen sharing).
\section{Bus Pirate}
@ -36,7 +36,7 @@ The Raspberry Pi could be used for the same quick evaluations or experiments we
%http://dangerousprototypes.com/blog/about/
% Dangerous Prototypes and manufactured by Seeed Studio\todo{link}
Bus Pirate, a project by Ian Lesnet, is a USB-attached device providing access to hardware interfaces like \gls{SPI}, \gls{I2C}, \gls{USART} and 1-Wire, as well as frequency measurement and direct pin access. The board aims to make it easy for users to familiarize themselves with new chips and modules; it also provides a range of programming interfaces for flashing microcontroller firmwares and memories. It communicates with the \gls{PC} using a FTDI USB-serial bridge.
Bus Pirate, a project by Ian Lesnet, is a USB-attached device providing access to hardware interfaces like \gls{SPI}, \gls{I2C}, \gls{USART}, and 1-Wire, as well as frequency measurement and direct pin access. The board aims to make it easy for users to familiarize themselves with new chips and modules; it also provides a range of programming interfaces for flashing microcontroller firmwares and memories. It communicates with the \gls{PC} using a FTDI USB-serial bridge.
Bus Pirate is open source and is, in its scope, similar to GEX. It can be scripted and controlled from the PC, connects to USB and provides a wide selection of hardware interfaces.
@ -46,7 +46,7 @@ The board is based on a PIC16 microcontroller running at 32\,MHz. Its \gls{ADC}
Various professional tools that would fulfill our needs exist on the market, but their high price makes them inaccessible for users with a limited budget, such as hobbyists or students who would like to keep such a device for personal use. An example is the National Instruments \IIC/SPI Interface Device which also includes several \gls{GPIO} lines, their USB DAQ module, or some of the Total Phase \IIC/SPI gadgets (\cref{fig:profidaq}).
The performance GEX can provide may not always match that of those professional tools, but in many cases it'll be a sufficient substitute at a fraction of the cost.
The performance GEX can provide may not always match that of those professional tools, but in many cases it will be a sufficient substitute at a fraction of the cost.
FreeRTOS is a free, open-source realtime operating system kernel targeted at embedded systems and ported to many different microcontroller architectures~\cite{freertos-ports-list}. FreeRTOS provides the task scheduler, forming the central part of the RTOS, and implements queues, semaphores, and mutexes for message passing and synchronization between concurrent tasks (summarily called synchronization objects\footnote{In this chapter sometimes simply called ``objects''.}). FreeRTOS is compact and designed to be easy to understand; it is written in C, with the exception of some architecture-specific routines which use assembly. A complete overview of the system is available in the FreeRTOS reference manual~\cite{freertos-rm} and its guide book~\cite{freertos-book}.
FreeRTOS is a free, open-source real-time operating system kernel targeted at embedded systems;it has been ported to many different microcontroller architectures~\cite{freertos-ports-list} and it is the de-facto industry standard. The system is compact and designed to be easy to understand; it is written in C, with the exception of some architecture-specific routines which use assembly. A complete overview of its \gls{API} is available in the FreeRTOS reference manual~\cite{freertos-rm} and its guide book~\cite{freertos-book}.
FreeRTOS is used in GEX for its synchronization objects that make it easy to safely pass messages between interrupts and working threads, without deadlocks or race conditions. The built-in stack overflow protection helps with a more efficient memory allocation, and the heap allocator provided by FreeRTOS enables a thread-safe dynamic allocation with a shared heap.
FreeRTOS provides a task scheduler, forming the central part of the system, and implements queues, semaphores, and mutexes for message passing and the synchronization of concurrent tasks. Those features are summarily called \textit{synchronization objects}, or simply \textit{objects}.
The system is used in GEX for its synchronization objects that allow us to safely pass messages between interrupts and working threads, without deadlocks or race conditions; the particular usage of FreeRTOS features will be explained in \cref{sec:rtos-in-gex}. The built-in stack overflow protection helps us optimize task memory allocation\footnote{The stack monitor reports how much stack space was really used, so we can expand or shrink it as needed to make the application work reliably, without wasting memory that would never be used.}, and the heap allocator provided by FreeRTOS enables thread-safe dynamic allocation with a shared heap.
\section{Basic FreeRTOS Concepts and Functions}
\subsection{Tasks}
Threads in FreeRTOS are called \textit{tasks}. Each task is assigned a memory area to use as its stack space, and a holding structure with its name, saved \textit{context}, and other metadata used by the kernel. A task context includes the program counter, stack pointer and other register values. Task switching is done by saving and restoring this context by manipulating the values on the stack before leaving and interrupt. The FreeRTOS website provides an example with the AVR port~\cite{freertos-task-switching} demonstrating how its internal functionality is implemented, including the context switch.
Threads in FreeRTOS are called \textit{tasks}. Each task is assigned a memory area to use as its stack space, and a holding structure with its name, saved \textit{context}, and other metadata used by the kernel. A task context includes the program counter, stack pointer and other register values. Task switching is done by saving and restoring this context by manipulating the values on the stack before leaving an\gls{ISR}. The FreeRTOS website provides an example with the AVR port~\cite{freertos-task-switching} demonstrating how its internal functionality is implemented, including the context switch.
At start-up the firmware initializes the kernel, registers tasks to run and starts the scheduler. From this point onward the scheduler is in control and runs the tasks using a round robin scheme, always giving a task one tick of run time (usually 1\,ms) before interrupting it. Which task should run is primarily determined by their priority numbers, but there are other factors.
At start-up the firmware initializes the kernel, registers tasks to run, and starts the scheduler. From this point onward the scheduler is in control and runs the tasks using a round robin scheme, always giving a task one tick of run time (usually 1\,ms) before interrupting it. Which task should run is determined primarily by their priority numbers, but there are other factors, as will be shown in \cref{sec:task-switching}.
\subsubsection{Task Run States}
Tasks can be in one of four states: Suspended, Ready, Blocked, Running. The Suspended state does not normally occur in a task's life cycle, it is entered and left using API calls from the application. A task is in the Ready state when it can run, but is currently paused because a higher priority task is running. It enters the Running state when the scheduler switches to it. A Running task can wait for a synchronization object (e.g., a mutex) to be available; at this point it enters a Blocked state and the scheduler runs the next Ready task. When no tasks can run, the Idle Task takes control; it can either enter a sleep state to save power, or wait in a loop until another task is available. The Idle task is always either Ready or Running and has the lowest priority of all tasks.
Tasks can be in one of four states: Suspended, Ready, Blocked, and Running. The Suspended state does not normally occur in a task's life cycle, it is entered and left using API calls from the application. A task is in the Ready state when it can run, but is currently paused because a higher priority task is running. It enters the Running state when the scheduler switches to it. A Running task can wait for a synchronization object (e.g., a mutex) to be available; at this point it enters a Blocked state and the scheduler runs the next Ready task. When no tasks can run, the Idle Task takes control; it can either enter a sleep state to save power, or wait in a loop until another task is available. The Idle task is always either Ready or Running and has the lowest priority of all tasks.
\subsubsection{Task Switching and Interrupts}
\subsubsection{Task Switching and Interrupts}\label{sec:task-switching}
Task switching occurs periodically in a timer interrupt, usually every 1\,ms; in \armcm chips this is typically the SysTick interrupt, a timer designed for this purpose that is included in the core itself and thus available on all derived platforms.
After one tick of run time, the Running task is paused (enters Ready state), or continues to run if no higher priority task is available. If a higher priority task waits for an object and this is made available in an interrupt, the previously running task is paused and the waiting task is resumed immediately (enters the Running state). FreeRTOS defines an interrupt-friendly variant of some of the \gls{API} functions intended for this purpose.
Only a subset of the FreeRTOS \gls{API} can be accessed from interrupt routines this way, for example it is not possible to use the delay function or wait for an object with a timeout, because the SysTick interrupt, which increments the tick counter, has the lowest priority and could not run. This is by design, to prevent unexpected context switching in application interrupts.
After one tick of run time, the Running task is paused and becomes Ready, or continues to run if no higher-priority task is available. If a higher-priority task waits for an object and this is made available in an \gls{ISR}, the running lower-priority task is paused and the waiting task resumes immediately. FreeRTOS defines interrupt-friendly variants of some of the \gls{API} functions intended for this purpose; however, only a subset of the \gls{API} is available in an \gls{ISR}, for example, it is not possible to use the delay function or wait for an object with a timeout, as the SysTick interrupt, incrementing the tick counter, has the lowest priority and could not run. This is by design, intended to prevent unexpected context switching in application interrupts.
FreeRTOS uses a \textit{priority inheritance} mechanism to prevent situations where a high priority task waits for an object held by a lower priority task (called \textit{priority inversion}). The blocking task's priority is temporarily raised to the level of the blocked high priority task so it can finish faster and release the held object. Its priority is then degraded back to the original value. When the lowerpriority task itself is blocked, the same process can be repeated.
FreeRTOS uses a \textit{priority inheritance} mechanism to prevent situations where a high-priority task waits for an object held by a lower-priority task (called \textit{priority inversion}). The blocking task's priority is temporarily raised to the level of the blocked high-priority task so it can finish earlier and release the held object. Its priority is then degraded back to the original value. When the lower-priority task itself is blocked, the same process can be repeated.
\subsection{Synchronization Objects}
FreeRTOS provides binary and counting semaphores, mutexes and queues. Each of those objects will now be briefly introduced.
FreeRTOS provides binary and counting semaphores, mutexes, and queues, which will now be briefly explained; a more in-depth description can be found in the guide book~\cite{freertos-book}.
\begin{itemize}
\item\textbf{Binary semaphores}can be used for task notifications, e.g., when a task waits for a semaphore to be set by an \gls{ISR}. This makes the task Ready and if it has a higher priority than the task previously running, it is immediately resumed to process the event.
\item\textbf{Binary semaphores}are used for task notifications, e.g., when a task waits for a semaphore to be set by an \gls{ISR}. This makes the task Ready and if it has a higher priority than the task previously running, it is immediately resumed to process the event.
\item\textbf{Counting semaphores}are used to represent available resources. A pool of software or hardware resources is accompanied by a counting semaphore, so that tasks can wait for a resource to become available in the pool and then subtract the semaphore value. After finishing with a resource, the semaphore is incremented again and another task can use it.
\item\textbf{Counting semaphores}represent available resources in a resource pool, a set of software or hardware resources used by tasks. The pool is accompanied by a counting semaphore on which tasks wait for a resource to become available, and then subtract the semaphore value. After a resource is no longer needed by the task, the semaphore is incremented again and another task can use it.
\item\textbf{Mutexes} (locks), unlike semaphores, must be taken and released in the same thread (task). They're used to guard exclusive access to a resource, such as writing to a serial port, or accessing a shared memory. When a mutex is taken, a different task which needs to use it enters the Blocked state and is resumed once the mutex becomes available (at which point the task is resumed and simultaneously takes it).
\item\textbf{Mutexes} (locks) are similar to semaphores, but they must be taken and released in the same task. We use them to
guard an exclusive access to a resource, typically a hardware peripheral or a shared memory area. When a mutex is taken, any other tasks trying to take it too enter become Blocked. A Blocked task waiting for a mutex is resumed once this becomes available, at which point the task becomes its owner and is resumed.
\item\textbf{Queues} are used for passing messages between tasks, or from interrupts to tasks. Both sending and receiving of queue messages can block the task until the operation becomes possible. A queue handing task is often simply a loop which tries to read from the queue with an infinite timeout and processes the received data once the reading succeeds.
\end{itemize}
@ -44,7 +45,7 @@ It must be noted that synchronization objects like mutexes and semaphores can he
\section{Stack Overflow Protection}
Each task in FreeRTOS is assigned a block of \gls{RAM} to use as its stack when it runs. This is where the stack pointer is restored to in the context switch. It can happen that a insufficient stack is exceeded and the stack pointer moves outside the designated area. Without countermeasures this would mean that we are overwriting bytes in some unrelated memory structure, perhaps another task's control block or a stack.
Each task in FreeRTOS is assigned a block of \gls{RAM} to use as its stack when it runs. This is where the stack pointer is restored to in the context switch. The stack pointer could move outside the designated area if the allocated space is insufficient; without countermeasures, this would mean that we are overwriting bytes in some unrelated memory structure, perhaps another task's stack memory.
A stack overflow protection can be enabled by a flag in the FreeRTOS configuration file (more details in~\cite{freertos-stackov}). This function works in two ways: the more obvious is a simple check that the stack pointer remains in the designated area; however, as the check may be performed only in the scheduler interrupt, it can happen that the stack pointer exceeds the bounds only temporarily and returns back before the stack can be checked for overflow. The stack is thus filled with a known filler value before starting the task, and the last few bytes are then tested to match this value. Not only can we detect a stack overflow more reliably, this feature also makes it possible to estimate the peak stack usage by counting the remaining filler values in the stack area. Naturally, we cannot distinguish between the original filler values and the same value stored on the stack; still, this method proves remarkably successful in the detection of misconfigured stack size.
A stack overflow protection can be enabled by a flag in the FreeRTOS configuration file. This function works in two ways: the more obvious is a simple check that the stack pointer remains in the designated area; however, as the check may be performed only in the scheduler interrupt, it is possible that the stack pointer exceeds the bounds only temporarily and returns back before the check can run. A more advanced solution, used by FreeRTOS, fills the stack memory with a known filler value before starting the task; the last few bytes are then tested to match this value. Not only can we detect a stack overflow more reliably, this feature also makes it possible to estimate the peak stack usage by counting the remaining filler bytes. We cannot distinguish between the original values and the same data stored on the stack by the program, but the possibility of this happening is sufficiently low and this method proves remarkably successful at detecting misconfigured stack sizes.
The firmware is built on FreeRTOS (\cref{sec:freertos}) and a number of its synchronization objects and patterns are used to make its operation more robust.
@ -22,14 +22,14 @@ Communication between the host and a function is organized into virtual channels
\caption{\label{fig:usb-logical}The logical structure of USB}
\end{figure}
Endpoints can be either unidirectional or bidirectional; the direction from the host to a function is called OUT, the other direction (function the host) is called IN. A bidirectional endpoint is technically composed of a IN and OUT endpoint with the same number. All transactions (both IN and OUT) are initiated by the host; functions have to wait for their turn. Endpoint 0 is bidirectional, always enabled, and serves as a \textit{control endpoint}. The host uses the control endpoint to read information about the device and configure it as needed.
Endpoints can be either unidirectional or bidirectional; the direction from the host to a function is called OUT, the other direction (function to host) is called IN. A bidirectional endpoint is technically composed of IN and OUT endpoints with the same number. All transactions (both IN and OUT) are initiated by the host; functions have to wait for their turn. Endpoint 0 is bidirectional, always enabled, and serves as a \textit{control endpoint}. The host uses the control endpoint to read information about the device and configure it as needed.
\subsection{Transfer Types}
There are four types of data transfers defined in \gls{USB}: control, bulk, isochronous, and interrupt. Each endpoint is configured for a fixed transfer type:
\begin{itemize}
\item\textit{Control} -- initial configuration after device plug-in; also used for other aplication-specific control messages that can affect other pipes.
\item\textit{Control} -- initial configuration after device plug-in; also used for other application-specific control messages that can affect other pipes.
\item\textit{Bulk} -- used for burst transfers of large messages
\item\textit{Isochronous} -- streaming with guaranteed low latency; designed for audio or video streams where some data loss is preferred over stuttering
\item\textit{Interrupt} -- low latency short messages, used for human interface devices like mice and keyboards
@ -58,7 +58,7 @@ The descriptor table used by GEX is captured in \cref{fig:gex-descriptors} for i
\section{USB Physical Layer}
\gls{USB} uses differential signaling with \gls{NRZI} encoding and bit stuffing (the insertion of dummy bits to prevent long intervals in the same \gls{DC} level). The encoding, together with frame formatting, checksum verification, retransmission, and other low-level aspects of the \gls{USB} connection are entirely handled by the \gls{USB} physical interface block in the microcontroller's silicon. Normally we do not need to worry about those details; nonetheless, a curious reader may find more information in chapters 7 and 8 of~\cite{usbif-spec}. What needs our attention are the electrical characteristics of the physical connection, which need to be understood correctly for a successful schematic and \gls{PCB} design.
\gls{USB} uses differential signaling with \gls{NRZI} encoding and bit stuffing (the insertion of dummy bits to prevent long intervals in the same \gls{DC} level). The encoding, together with frame formatting, checksum verification, retransmission, and other low-level aspects of the \gls{USB} connection are entirely handled by the \gls{USB} physical interface block in the microcontroller's silicon. Normally we do not need to worry about those details; nonetheless, a curious reader may find more information in chapters 7 and 8 of~\cite{usbif-spec}. The electrical characteristics of the physical connection deserve more attention, as they need to be understood correctly for a successful schematic and \gls{PCB} design.
The \gls{USB} cable contains 4 conductors: V$_\mathrm{BUS}$ (+5\,V), D+, D--, and \gls{GND}. The data lines, D+ and D--, are also commonly labeled DP and DM. This differential pair should be routed in parallel on the \gls{PCB} and kept at the same length.
@ -82,7 +82,7 @@ This section explains the classes used in the GEX firmware. A list of all standa
\subsection{Mass Storage Class}\label{sec:msc}
The \gls{MSC} is supported by all modern operating systems (MS Windows, MacOS, GNU/Linux, FreeBSD etc.) to support thumb drives, external disks, memory card readers and other storage devices.
The \gls{MSC} is supported by all modern \gls{PC} operating systems to support \gls{USB} thumb drives, external disks, memory card readers, and other storage devices.
@ -94,7 +94,7 @@ For the mass storage device to be recognized by the host operating system, it mu
Unfortunately, the \gls{SCSI} Transparent command set appears to have been deliberately left unspecified for license or copyright reasons (see discussion in~\cite{usb-tscsi-wtf} and the surrounding thread) and the protocol now used under this name is an industry standard without a clear definition. Some pointers may be found in~\cite{usb-tscsi} and by examining the source code of the USB Device driver library provided by ST Microelectronics.
This command set lets the host read information about the attached storage, such as its capacity, and check for media presence and readiness to write or detach. This is used, e.g., for the ``Safely Remove'' function, which ensures that all internal buffers have been written to Flash.
This command set lets the host read information about the attached storage device, such as its capacity, and check for media presence and readiness to write or detach. This is used, e.g., for the ``Safely Remove'' function, which ensures that all internal buffers have been written to the flash memory.
In order to emulate a mass storage device without having a physical storage medium, we need to generate and parse the file system on-the-fly as the host \gls{OS} tries to access it. This will be discussed in \cref{sec:fat16}.
@ -106,7 +106,7 @@ Historically meant for modem communication, \gls{CDCACM} is now the de facto sta
The interrupt endpoint is used for control commands, such as toggling the auxiliary lines of RS-232 or setting the baud rate. Since GEX does not translate the data communication to any physical UART, those commands are not applicable and can be silently ignored.
An interesting property of the \gls{CDC} class is that the bulk endpoints transport raw data without any wrapping frames. By changing the interface's class in the descriptor table to 255 (\textit{Vendor Specific Class}), we can retain the messaging functionality of the designated endpoints, while accessing the endpoints device directly using, e.g., libUSB, without any interference from the \gls{OS}. This approach is also used to hide the \gls{MSC} interface when its not needed.
An interesting property of the \gls{CDC} class is that the bulk endpoints transport raw data without any wrapping frames. By changing the interface's class in the descriptor table to 255 (\textit{Vendor Specific Class}), we can retain the messaging functionality of the designated endpoints, while accessing the endpoints device directly using, e.g., libUSB, without any interference from the \gls{OS}. This approach is also used to hide the \gls{MSC} interface when it is not needed.
@ -114,5 +114,5 @@ The original \gls{USB} specification expected that each function will have only
The \gls{IAD} is an entry in the descriptor table that defines which interfaces belong together and should be handled by the same software driver. To use the \gls{IAD}, the function's class must be set to 0xEF, subclass 0x02, and protocol 0x01 in the top level descriptor, so that the \gls{OS} knows to look for this descriptor before binding drivers to any interfaces.
In GEX, the \gls{IAD} is used to tie together the \gls{CDC} and \gls{ACM} interfaces while leaving out the \gls{MSC} interface which should be handled by a different driver. To make this work, a new \textit{composite class}had to be created as a wrapper for the library-provided \gls{MSC} and \gls{CDCACM} implementation.
In GEX, the \gls{IAD} is used to tie together the \gls{CDC} and \gls{ACM} interfaces while leaving out the \gls{MSC} interface which should be handled by a different driver. To make this work, a new \textit{composite class}was created as a wrapper for the library-provided \gls{MSC} and \gls{CDCACM} implementation.
