Add loop syntactic sugar; errors now use print instead of debug; add check for duplicate labels.

README.md

@@ -187,7 +187,8 @@ The different ways to specify a value can be grouped as "reads" and "writes":
 - `RW` - intersection of the two sets, capable of reading and writing: `REG`, `SYM`, `@REG`, `@SYM`
 
 Objects (`@reg`, `@sym`) can be read or written as if they were a register, but only if the referenced object supports it.
-Other objects may produce a runtime fault or set the INVALID flag.
+Other objects may produce a runtime fault or set the INVALID flag. 
+The object syntax is also used to read values yielded by a generator-like coroutine.
 
 In the instruction lists below, I will use the symbols `Rd` for reads, `Wr` for writes, `RW` for read-writes, and `@Obj` for object handles,
 with optional description after an apostrophe, such as: `(add Wr'dst Rd'a Rd'b)`. 
@@ -286,7 +287,105 @@ It can also be written like this:
     ...
 )
 ```
-  
+
+## Coroutines
+
+Croissant implements something that could be called "pre-emptive coroutines". They do not provide any performance gain, 
+but add asynchronicity to the program, and can work as generators!
+
+There is no true parallelism, it is difficult to implement safely and efficiently with a global state.
+
+### Spawning
+
+*Any procedure can be used as a coroutine.*
+
+A coroutine is created using the `spawn` instruction, which produces an object handle.
+
+```
+(spawn r0 do_stuff 1 2 3)
+```
+
+At this point, the program is evenly divided between the original and the coroutine "thread". 
+
+The spawned coroutine is scheduled to run immediately after being spawned.
+
+### Task switching
+
+Coroutines take turns to execute the program. The scheduling interval can be configured.
+
+Control can be given up using the `yield` instruction; for example, when waiting for a mutex. This happens automatically when 
+a `sleep` instruction is invoked.
+
+### Race conditions
+
+Take care when working with objects, resources and global registers: you can get race conditions 
+with coroutines. Use atomic instructions (`cas`, `casXX`, `bfcas`…) to implement synchronization.
+
+The `casXX` instruction is very powerful: you can use one bit of a register as a mutex and the rest of it to store some useful data. 
+You can also use one register for up to 64 mutexes.
+
+Remember to only use global registers (or buffer items) as mutexes: `g0`-`g15`. Each coroutine has its own set of *regular* registers.
+
+Another way to avoid race conditions is to use **critical sections**. 
+Context switch (switching between active coroutines) is forbidden in a critical section. Try to keep critical sections as short as possible,
+since they can distort sleep times and cause other similar problems.
+
+```
+(crit-begin)
+...
+(crit-end)
+```
+
+A safer way is to use the "critical block", which expands to the same, but also detects some common bugs at compile time,
+like trying to jump out of the critical section.
+
+```
+(crit
+    ...
+)
+```
+
+Critical section nesting is allowed, but probably a bug.
+
+**Beware deadlocks!**
+
+### Using coroutines as generators
+
+A coroutine can "yield a value" by invoking the `yield` instruction with an operand. This can be done any number of times.
+
+```
+(yield r0)
+```
+
+The coroutine is blocked until the value is consumed by someone. To consume a yielded value, read the coroutine object handle:
+
+```
+(spawn r5 foo)
+
+(ld r0 @r5) ; read a yielded value
+```
+
+**Caution!!!** Due to the way this is implemented, the instruction that tries to use a yielded value might partially execute
+before detecting that it is blocked, and will be retried when the thread is next scheduled to run. This has the consequence
+that if something like a object handle is read by the same instruction, it may be read multiple times while waiting for the
+yielded value. It is recommended to *never combine reads of a yielded value with reads of other object handles*.
+
+### Joining a coroutine
+
+Use the `join` instruction with a coroutine object handle to wait for its completion.
+
+A coroutine completes by calling `ret` at its top level. This naturally means that a coroutine can return values!
+
+The returned values are placed in the result registers, just like with the `call` instruction.
+
+```
+(spawn r5 foo)
+; ...
+
+(join @r5)
+; res0-res15 now contain return values
+```
+
 # Instruction Set
 
 Crsn instruction set is composed of extensions.
@@ -309,6 +408,36 @@ to a varying number of skips and conditional instructions by the assembler. Only
 
 Jumping to a label is always safer than a manual skip.
 
+### Loop syntactic sugar
+
+Infinite loops are a very common construct, so there is special syntax added for them.
+
+These are converted by the assembler to an anonymous label and a jump to it.
+
+```
+(loop
+    ...ops
+)
+```
+
+If you want to have a convenient jump target, give the loop a name. This lets you easily "break" and "continue" 
+by jumping to the labels.
+
+```
+(loop :label
+    ...ops
+)
+```
+
+becomes
+
+```
+(:label)
+...ops
+(j :label)
+(:label-end)
+```
+
 ## Built-in Instructions
 
 ...and pseudo-instructions
@@ -377,6 +506,18 @@ Jumping to a label is always safer than a manual skip.
 ; Offsets can be specified to work with arbitrary bit slices
 (xchXX RW:offset RW:offset)
 
+; Compare and swap; atomic instruction for coroutine synchronization
+; - The "Equal" flag is set on success, use it with conditional branches or conditional execution.
+; - The new value is not read until the comparison passes and it is needed. 
+;   This behavior may matter for side effects when used with object handles.
+(cas RW Rd'expected Rd'new)
+
+; Compare and swap a bit slice; atomic instruction for coroutine synchronization
+; See (cas) above for more info.
+;
+; Offsets can be specified to work with arbitrary bit slices
+(casXX RW:offset Rd:offset'expected Rd:offset'new)
+
 ; Store status flags to a register
 (stf Wr)
 
@@ -391,9 +532,34 @@ Jumping to a label is always safer than a manual skip.
 ; The arguments are passed as argX. Return values are stored in resX registers.
 (call PROC Rd...)
 
-; Exit the current routine with return values
+; Spawn a coroutine. The handle is stored in the output register.
+(spawn Wr'handle PROC Rd...)
+
+; Exit the current routine (or coroutine) with return values
 (ret Rd...)
 
+; Yield control from a coroutine (use when waiting for a mutex to give control early to 
+; other coroutines that might be holding it)
+(yield)
+
+; Yield a value generated by a coroutine. Gives up control and blocks until the value is consumed.
+; Conversely, an instruction that tries to read a yielded value using the object handle is blocked
+; until such value becomes available.
+(yield Rd'value)
+
+; Wait for a coroutine to complete, read its return values and delete it.
+(join @Obj)
+
+; Begin a critical section (no context switch allowed)
+(crit-begin)
+; End a critical section
+(crit-end)
+
+; Shortcut to define a critical section
+(crit
+    ...ops
+)
+
 ; Generate a run-time fault with a debugger message
 (fault)
 (fault message)
@@ -409,6 +575,9 @@ Jumping to a label is always safer than a manual skip.
 ; The label can be a numeric or string label, its sole purpose is tying the two together. They must be unique in the program.
 (barrier-open LABEL)
 (barrier-close LABEL)
+
+; Set coroutine scheduler timeslice (in microseconds). Set to zero to disable preemption.
+(rt-opt RT_TIMESLICE Rd'usec)
 ```
 
 ## Arithmetic Module
@@ -735,6 +904,17 @@ Primitive buffer ops (position is always 0-based)
 
 ; Remove item at a position, shifting the rest to the left to fill the empty space
 (bfrm Wr @Obj Rd:index)
+
+; Buffer value compare and swap; atomic instruction for coroutine synchronization.
+; - The "Equal" flag is set on success, use it with conditional branches or conditional execution.
+; - The new value is not read until the comparison passes and it is needed. 
+;   This behavior may matter for side effects when used with object handles.
+;
+; This instruction is useful when more than one lock is needed and they are stored in a buffer at well known positions.
+; Naturally, the buffer must not be mutated in other ways that would undermine the locking.
+;
+; If an index just outside the buffer is used, the value is read as zero the position is created (if zero was expected).
+(bfcas @Obj Rd:index Rd'expected Rd'new)
 ```
 
 Whole buffer manipulation:

crsn/src/asm/error.rs

@@ -12,13 +12,13 @@ use crate::asm::instr::Cond;
 /// csn_asm unified error type
 #[derive(Error, Debug)]
 pub enum CrsnError {
-    #[error("S-expression parsing error: {0:?}")]
+    #[error("S-expression parsing error: {0}")]
     Sexp(#[from] Box<sexp::Error>),
-    #[error("Parse error: {0:?} at {1:?}")]
+    #[error("Parse error: {0} at {1}")]
     Parse(Cow<'static, str>, SourcePosition),
-    #[error("Parse error: {0:?} at {1:?}")]
+    #[error("Parse error: {0} at {1}")]
     ParseOther(Box<dyn Error + Send + Sync>, SourcePosition),
-    #[error("Assembler error: {0:?} at {1:?}")]
+    #[error("Assembler error: {0} at {1}")]
     Asm(AsmError, SourcePosition),
 }
 
@@ -33,8 +33,10 @@ pub enum AsmError {
     DiscardAsValue,
     #[error("Conditional branch already defined for \"{0}\"")]
     ConditionalAlreadyUsed(Cond),
-    #[error("Label \"{0:?}\" not defined")]
+    #[error("Label \"{0}\" not defined")]
     LabelNotDefined(Label),
+    #[error("Label \"{0}\" already defined in this scope")]
+    LabelDuplicate(Label),
     #[error("Bad register type: {0}")]
     BadRegisterType(Register),
 }

crsn/src/asm/instr/flatten.rs

@@ -29,6 +29,16 @@ impl Flatten for () {
     }
 }
 
+impl Flatten for Op {
+    fn flatten(self: Box<Self>, _label_num: &AtomicU32) -> Result<Vec<Op>, CrsnError> {
+        Ok(vec![*self])
+    }
+
+    fn pos(&self) -> SourcePosition {
+        self.pos.clone()
+    }
+}
+
 impl Flatten for InstrWithBranches {
     fn pos(&self) -> SourcePosition {
         self.pos.clone()
@@ -111,6 +121,23 @@ impl Flatten for Vec<Box<dyn Flatten>> {
     }
 }
 
+impl Flatten for Vec<Op> {
+    fn pos(&self) -> SourcePosition {
+        match self.first() {
+            None => {
+                Default::default()
+            }
+            Some(f) => {
+                f.pos()
+            }
+        }
+    }
+
+    fn flatten(self: Box<Self>, _label_num: &AtomicU32) -> Result<Vec<Op>, CrsnError> {
+        Ok(*self)
+    }
+}
+
 impl Flatten for Routine {
     fn pos(&self) -> SourcePosition {
         self.pos.clone()
@@ -138,15 +165,19 @@ impl Flatten for Routine {
             }.into_op(self_pos)
         );
 
-        labels_to_skips(ops)
+        jumps_to_skips(ops)
     }
 }
 
 /// Convert jumps to relative skips
-pub fn labels_to_skips(ops: Vec<Op>) -> Result<Vec<Op>, CrsnError> {
+pub fn jumps_to_skips(ops: Vec<Op>) -> Result<Vec<Op>, CrsnError> {
     let mut label_positions = HashMap::<Label, usize>::new();
     for (n, op) in ops.iter().enumerate() {
         if let OpKind::BuiltIn(BuiltinOp::Label(name)) = &op.kind {
+            if label_positions.contains_key(name) {
+                return Err(CrsnError::Asm(AsmError::LabelDuplicate(name.clone()), op.pos.clone()));
+            }
+
             label_positions.insert(name.clone(), n - label_positions.len());
         }
     }

crsn/src/asm/instr/op.rs

@@ -4,7 +4,7 @@ use sexp::{Atom, Sexp, SourcePosition};
 
 use crate::asm::instr::Cond;
 use crate::builtin::defs::BuiltinOp;
-use crate::module::{EvalRes, OpTrait};
+use crate::module::{EvalRes, OpTrait, SchedSignal};
 use crate::runtime::fault::Fault;
 use crate::runtime::run_thread::{info::ThreadInfo, state::RunState};
 
@@ -30,6 +30,7 @@ impl OpTrait for Op {
                 return Ok(EvalRes {
                     cycles: 0,
                     advance: 1,
+                    sched: SchedSignal::Normal
                 });
             }
         }

crsn/src/asm/mod.rs

@@ -3,13 +3,14 @@ use std::sync::Arc;
 
 use sexp::SourcePosition;
 
-use crate::asm::instr::flatten::labels_to_skips;
+use crate::asm::instr::flatten::jumps_to_skips;
 use crate::asm::parse::{ParserContext, ParserState};
 use crate::module::{CrsnExtension, CrsnUniq};
 use crate::runtime::program::Program;
 use crate::builtin::BuiltinOps;
 use crate::runtime::run_thread::{RunState, ThreadInfo, ThreadToken};
 use crate::runtime::frame::REG_COUNT;
+use std::sync::atomic::AtomicU32;
 
 pub mod data;
 pub mod error;
@@ -42,9 +43,14 @@ pub fn assemble(source: &str, uniq : &CrsnUniq, mut parsers: Vec<Box<dyn CrsnExt
         uniq: Default::default(),
         program: Program::new(vec![], parsers_arc.clone()).unwrap(),
         cycle_time: Default::default(),
+        scheduler_interval: Default::default(),
         extensions: parsers_arc.clone(),
     });
 
+    /* numbered labels start with a weird high number
+    to avoid conflicts with user-defined numbered labels */
+    let label_num = Arc::new(AtomicU32::new(0x7890_0000));
+
     let pcx = ParserContext {
         parsers: &parsers_arc,
         state: RefCell::new(ParserState {
@@ -57,18 +63,21 @@ pub fn assemble(source: &str, uniq : &CrsnUniq, mut parsers: Vec<Box<dyn CrsnExt
             // This allows to evaluate nearly all instructions at compile time.
             const_eval: RunState {
                 thread_info: ti.clone(),
-                frame: Default::default(),
-                call_stack: vec![],
+                cr: Default::default(),
+                parked: Default::default(),
                 global_regs: [0; REG_COUNT],
-                ext_data: Default::default()
+                ext_data: Default::default(),
+                cr_deadline: None,
+                critical_section: 0,
             },
             const_eval_ti: ti,
-            parsing_expr: false
+            parsing_expr: false,
+            label_num: label_num.clone()
         }),
     };
 
     let ops = parse::parse(source, &SourcePosition::default(), &pcx)?;
-    let ops = labels_to_skips(ops)?;
+    let ops = jumps_to_skips(ops)?;
 
     Ok(Program::new(ops, parsers_arc)?)
 }

crsn/src/asm/parse/arg_parser.rs

@@ -109,7 +109,7 @@ impl<'a> TokenParser<'a> {
     }
 
     /// Look at the next entry
-    pub fn peek(&mut self) -> Option<&Sexp> {
+    pub fn peek(&self) -> Option<&Sexp> {
         self.args.last()
     }
 
@@ -291,6 +291,36 @@ impl<'a> TokenParser<'a> {
         }
     }
 
+    /// Parse (RdWr, Rd, Rd) with an optional mask
+    pub fn parse_masked_rdwr_rd_rd(&mut self, keyword: &str, prefix: &str) -> Result<Option<(RdWr, Rd, Rd, BitMask)>, CrsnError> {
+        if let Some(s) = keyword.strip_prefix(prefix) {
+            let width = if s.is_empty() {
+                (std::mem::size_of::<Value>() as u32) * 8
+            } else {
+                s.parse().err_pos(self.start_pos)?
+            };
+            if self.len() == 3 {
+                let (rw, dst_pos) = self.next_rdwr_offset()?;
+                let (r1, src_pos) = self.next_rd_offset()?;
+                let (r2, src2_pos) = self.next_rd_offset()?;
+
+                let mask = BitMask {
+                    width,
+                    dst_pos,
+                    src_pos,
+                    src2_pos,
+                };
+                mask.validate(self.start_pos)?;
+
+                Ok(Some((rw, r1, r2, mask)))
+            } else {
+                Err(CrsnError::Parse("Instruction needs 3 (RW,Rd,Rd) arguments!".into(), self.start_pos.clone()))
+            }
+        } else {
+            Ok(None)
+        }
+    }
+
     /// Parse combining binary instruction operands (i.e. add) without masks
     /// Accepts (Wr, Rd, Rd) and (RdWr, Rd)
     pub fn parse_wr_rd_rd(&mut self) -> Result<(Wr, Rd, Rd), CrsnError> {

crsn/src/asm/parse/mod.rs

@@ -51,14 +51,25 @@ pub struct ParserState {
 
     /// True if we are in an expression parser context
     pub parsing_expr : bool,
+
+    /// Label numberer
+    pub label_num : Arc<AtomicU32>,
+}
+
+impl ParserState {
+    pub fn reset(&mut self) {
+        self.global_reg_aliases.clear();
+        self.reg_aliases.clear();
+        self.reg_alias_stack.clear();
+        self.constants.clear();
+        self.parsing_expr = false;
+        self.const_eval.reset();
+    }
 }
 
 pub fn parse(source: &str, pos: &SourcePosition, parsers: &ParserContext) -> Result<Vec<Op>, CrsnError> {
     let (items, _pos) = expect_list(sexp::parse(source)?, true)?;
 
-    /* numbered labels start with a weird high number
-    to avoid conflicts with user-defined numbered labels */
-    let label_num = AtomicU32::new(0x7890_0000);
     parse_instructions(items.into_iter(), pos, parsers)?
-        .flatten(&label_num)
+        .flatten(&parsers.state.borrow().label_num)
 }

crsn/src/asm/parse/parse_data.rs

@@ -124,7 +124,7 @@ pub fn parse_data_disp(tok: Sexp, pcx: &ParserContext) -> Result<DataDisp, CrsnE
                 }
 
                 let mut state_mut = pcx.state.borrow_mut();
-                state_mut.const_eval.clear_all();
+                state_mut.const_eval.reset();
 
                 let ticl = state_mut.const_eval.thread_info.clone();
                 op.execute(&ticl, &mut state_mut.const_eval)
@@ -139,7 +139,7 @@ pub fn parse_data_disp(tok: Sexp, pcx: &ParserContext) -> Result<DataDisp, CrsnE
                     state_mut.parsing_expr = false;
                 }
 
-                return Ok(DataDisp::Immediate(state_mut.const_eval.frame.gen[0]));
+                return Ok(DataDisp::Immediate(state_mut.const_eval.cr.frame.gen[0]));
             }
 
             Err(CrsnError::Parse("List not expected here".into(), pos))

crsn/src/asm/parse/parse_instr.rs

@@ -14,7 +14,7 @@ use super::parse_op::parse_op;
 
 pub fn parse_instructions(items: impl Iterator<Item=Sexp>, pos: &SourcePosition, pcx: &ParserContext) -> Result<Box<dyn Flatten>, CrsnError> {
     let mut parsed = vec![];
-    for expr in items {
+    'exprs: for expr in items {
         let (tokens, listpos) = expect_list(expr, false)?;
 
         let mut toki = tokens.into_iter();
@@ -40,8 +40,13 @@ pub fn parse_instructions(items: impl Iterator<Item=Sexp>, pos: &SourcePosition,
         let mut token_parser = TokenParser::new(toki.collect(), &listpos, pcx);
         for p in pcx.parsers {
             token_parser = match p.parse_syntax(pos, &name, token_parser) {
-                Ok(ParseRes::Parsed(op)) => return Ok(op),
-                Ok(ParseRes::ParsedNone) => return Ok(Box::new(())),
+                Ok(ParseRes::Parsed(op)) => {
+                    parsed.push(op);
+                    continue 'exprs;
+                },
+                Ok(ParseRes::ParsedNone) => {
+                    continue 'exprs;
+                },
                 Ok(ParseRes::Unknown(to_reuse)) => {
                     if to_reuse.parsing_started() {
                         panic!("Module \"{}\" started parsing syntax, but returned Unknown!", p.name());

crsn/src/builtin/defs.rs

@@ -136,7 +136,9 @@ pub enum BuiltinOp {
     FarJump(Label),
     /// Call a routine with arguments.
     /// The arguments are passed as argX. Return values are stored in resX registers.
-    Call(RoutineName, Vec<Rd>),
+    Call { proc: RoutineName, args: Vec<Rd> },
+    /// Spawn a coroutine. The invocation is similar to (call).
+    Spawn { handle: Wr, proc: RoutineName, args: Vec<Rd> },
     /// Exit the current routine with return values
     Ret(Vec<Rd>),
     /// Mark a routine entry point (call target).
@@ -144,6 +146,17 @@ pub enum BuiltinOp {
     Routine(RoutineName),
     /// Skip backward or forward. The skip count can be defined by an argument.
     Skip(Rd),
+    /// Join a coroutine
+    Join(RdObj),
+    /// Yield control, optionally yielding a value that must be consumed (by reading the task handle)
+    /// before execution can resume
+    Yield { value: Option<Rd> },
+    /// Set runtime option
+    RuntimeOpt { opt: Rd, value: Rd },
+    /// Begin critical section
+    CriticalBegin,
+    /// End critical section
+    CriticalEnd,
     /// Deny jumps, skips and run across this address, producing a run-time fault.
     Barrier {
         kind: Barrier,
@@ -164,6 +177,8 @@ pub enum BuiltinOp {
     LoadSequence { dst: Wr, value: LdsValue },
     /// Swap two registers
     Exchange { a: RdWr, b: RdWr, mask: BitMask },
+    /// Swap if equal to a pattern
+    CompareSwap { dst: RdWr, expected: Rd, src: Rd, mask: BitMask },
     /// Store runtime status to a register
     StoreFlags { dst: Wr },
     /// Load runtime status from a register

crsn/src/builtin/exec.rs

@@ -4,12 +4,14 @@ use sexp::Sexp;
 
 use crate::asm::data::literal::{Addr, Value};
 use crate::builtin::defs::{Barrier, BuiltinOp, LdsValue};
-use crate::module::{EvalRes, OpTrait};
+use crate::module::{EvalRes, OpTrait, SchedSignal};
 use crate::runtime::fault::Fault;
 use crate::runtime::frame::StackFrame;
 use crate::runtime::run_thread::{state::RunState, ThreadInfo};
 use crate::asm::instr::cond::Flag;
 
+use super::RT_OPT_TIMESLICE;
+
 impl OpTrait for BuiltinOp {
     fn execute(&self, info: &ThreadInfo, state: &mut RunState) -> Result<EvalRes, Fault> {
         let program = &info.program;
@@ -63,7 +65,7 @@ impl OpTrait for BuiltinOp {
             BuiltinOp::Jump(_name) => {
                 panic!("jump not translated to skip by assembler!");
             }
-            BuiltinOp::Call(name, args) => {
+            BuiltinOp::Call { proc: name, args } => {
                 match program.find_routine(&name) {
                     Ok(pos) => {
                         let mut values = Vec::with_capacity(args.len());
@@ -72,8 +74,8 @@ impl OpTrait for BuiltinOp {
                         }
 
                         let mut frame2 = StackFrame::new(pos, &values);
-                        std::mem::swap(&mut state.frame, &mut frame2);
-                        state.call_stack.push(frame2);
+                        std::mem::swap(&mut state.cr.frame, &mut frame2);
+                        state.cr.call_stack.push(frame2);
                         res.advance = 0;
                     }
                     Err(e) => {
@@ -81,18 +83,40 @@ impl OpTrait for BuiltinOp {
                     }
                 }
             }
-            BuiltinOp::Ret(retvals) => {
-                match state.call_stack.pop() {
-                    Some(previous) => {
-                        let mut values = Vec::with_capacity(retvals.len());
-                        for arg in retvals {
+            BuiltinOp::Spawn { handle, proc, args } => {
+                match program.find_routine(&proc) {
+                    Ok(pos) => {
+                        let mut values = Vec::with_capacity(args.len());
+                        for arg in args {
                             values.push(state.read(arg)?);
                         }
-                        state.frame = previous;
-                        state.frame.set_retvals(&values);
+
+                        let frame2 = StackFrame::new(pos, &values);
+                        let handle_val = state.add_coroutine(frame2);
+                        state.write(handle, handle_val)?;
+
+                        // Yield execution so the new thread can start running
+                        res.sched = SchedSignal::Yield(None);
+                    }
+                    Err(e) => {
+                        return Err(e);
+                    }
+                }
+            }
+            BuiltinOp::Ret(retvals) => {
+                let mut values = Vec::with_capacity(retvals.len());
+                for arg in retvals {
+                    values.push(state.read(arg)?);
+                }
+
+                match state.cr.call_stack.pop() {
+                    Some(previous) => {
+                        state.cr.frame = previous;
+                        state.cr.frame.set_retvals(&values);
                     }
                     None => {
-                        return Err(Fault::CallStackUnderflow);
+                        // Stack underflow - if this is a coroutine, it's finished
+                        res.sched = SchedSignal::Ret(values);
                     }
                 }
             }
@@ -173,16 +197,38 @@ impl OpTrait for BuiltinOp {
                     state.write(b, bb2)?;
                 }
             }
+            BuiltinOp::CompareSwap { dst, expected, src, mask: slice } => {
+                state.clear_status();
+                let old = state.read(dst)?;
+                let exp = state.read(expected)?;
+
+                if slice.is_full() {
+                    if old == exp {
+                        let new = state.read(src)?;
+                        state.write(dst, new)?;
+                        state.set_flag(Flag::Equal, true);
+                    }
+                } else {
+                    let ones: u64 = (1 << slice.width) - 1;
+
+                    if (old & (ones << slice.dst_pos)) == (exp & (ones << slice.src_pos)) {
+                        let new = state.read(src)?;
+                        let replacement = (old & !(ones << slice.dst_pos)) | (((new & (ones << slice.src2_pos)) >> slice.src2_pos) << slice.dst_pos);
+                        state.write(dst, replacement)?;
+                        state.set_flag(Flag::Equal, true);
+                    }
+                }
+            }
             BuiltinOp::StoreFlags { dst } => {
-                let packed = state.frame.status.store();
+                let packed = state.cr.frame.status.store();
                 state.write(dst, packed)?;
             }
             BuiltinOp::LoadFlags { src } => {
                 let x = state.read(src)?;
-                state.frame.status.load(x);
+                state.cr.frame.status.load(x);
             }
             BuiltinOp::Sleep { count: micros, unit_us } => {
-                std::thread::sleep(Duration::from_micros(state.read(micros)? * unit_us.micros()))
+                res.sched = SchedSignal::Sleep(Duration::from_micros(state.read(micros)? * unit_us.micros()));
             }
             BuiltinOp::Delete(obj) => {
                 let hv = obj.read(state)?;
@@ -199,6 +245,44 @@ impl OpTrait for BuiltinOp {
                     state.set_flag(Flag::Invalid, true);
                 }
             }
+            BuiltinOp::Yield { value } => {
+                res.sched = SchedSignal::Yield(match value {
+                    None => None,
+                    Some(rd) => Some(state.read(rd)?)
+                });
+            }
+            BuiltinOp::Join(obj) => {
+                res.sched = SchedSignal::Join(obj.read(state)?);
+            }
+            BuiltinOp::RuntimeOpt { opt, value } => {
+                let opt = state.read(opt)?;
+                let val = state.read(value)?;
+
+                match opt {
+                    RT_OPT_TIMESLICE => {
+                        *state.thread_info.scheduler_interval.write() = Duration::from_micros(val);
+                    }
+                    _ => {
+                        warn!("Invalid rt-opt {}", opt);
+                        state.set_flag(Flag::Invalid, true);
+                    }
+                }
+
+            }
+            BuiltinOp::CriticalBegin => {
+                if state.critical_section == Value::MAX {
+                    return Err(Fault::UnbalancedCriticalSection);
+                }
+                state.critical_section += 1;
+                res.cycles = 0;
+            }
+            BuiltinOp::CriticalEnd => {
+                if state.critical_section == 0 {
+                    return Err(Fault::UnbalancedCriticalSection);
+                }
+                state.critical_section -= 1;
+                res.cycles = 0;
+            }
         }
 
         Ok(res)

crsn/src/builtin/mod.rs

@@ -1,14 +1,23 @@
-use sexp::SourcePosition;
+use sexp::{SourcePosition, Sexp, Atom};
 
 use crate::asm::error::CrsnError;
 use crate::asm::instr::op::OpKind;
 use crate::asm::parse::arg_parser::TokenParser;
 use crate::module::{CrsnExtension, ParseRes};
+use crate::asm::data::literal::{Value, Label};
+use crate::asm::instr::{Flatten, Op};
+use crate::asm::parse::parse_instructions;
+use crate::builtin::defs::BuiltinOp;
+use crate::asm::instr::flatten::jumps_to_skips;
+use crate::asm::data::{Rd, RdData};
+use crate::asm::parse::parse_data::parse_label;
 
 pub mod defs;
 pub mod exec;
 pub mod parse;
 
+pub(crate) const RT_OPT_TIMESLICE : u64 = 1;
+
 #[derive(Debug, Clone)]
 pub struct BuiltinOps;
 
@@ -26,4 +35,116 @@ impl CrsnExtension for BuiltinOps {
     fn parse_op<'a>(&self, pos: &SourcePosition, keyword: &str, args: TokenParser<'a>) -> Result<ParseRes<'a, OpKind>, CrsnError> {
         parse::parse_op(pos, keyword, args)
     }
+
+    /// Get value of an extension-provided constant.
+    /// This constant may be an object handle, or a constant value used as argument in some other instruction.
+    fn get_constant_value<'a>(&self, name: &str) -> Option<Value> {
+        match name {
+            "RT_TIMESLICE" => Some(RT_OPT_TIMESLICE),
+            _ => None,
+        }
+    }
+
+    /// Parse a generic S-expression (non-op) that started with the given keyword
+    ///
+    /// pcx is available on the arg_tokens parser
+    fn parse_syntax<'a>(&self, pos: &SourcePosition, keyword: &str, mut tokens: TokenParser<'a>)
+                        -> Result<ParseRes<'a, Box<dyn Flatten>>, CrsnError>
+    {
+        if keyword == "crit" || keyword == "critical" {
+            let pcx = tokens.pcx;
+            let opts = parse_instructions(tokens.into_iter(), pos, pcx)?;
+
+            let flattened = jumps_to_skips(opts.flatten(&pcx.state.borrow_mut().label_num)?)?;
+
+            let len = flattened.len();
+
+            for (n, op) in flattened.iter().enumerate() {
+                match op.kind {
+                    OpKind::BuiltIn(BuiltinOp::Skip(Rd(RdData::Immediate(skip)))) => {
+                        let signed = i64::from_ne_bytes(skip.to_ne_bytes());
+                        let target = n as i64 + signed;
+                        if target < 0 || target > len as i64 {
+                            return Err(CrsnError::Parse("Cannot jump out of a critical section!".into(), op.pos()));
+                        }
+                    }
+                    /* Non-constant skip cannot be validated */
+                    OpKind::BuiltIn(BuiltinOp::Skip(_)) => {
+                        return Err(CrsnError::Parse("Variable skips are not allowed in a critical section".into(), op.pos()));
+                    }
+                    /* Yield in critical makes zero sense */
+                    OpKind::BuiltIn(BuiltinOp::Yield { .. }) => {
+                        return Err(CrsnError::Parse("Yield in a critical section!".into(), op.pos()));
+                    }
+                    /* This is likely a bug */
+                    OpKind::BuiltIn(BuiltinOp::Ret(_)) => {
+                        return Err(CrsnError::Parse("Ret in a critical section!".into(), op.pos()));
+                    }
+                    /* Probably also a bug. If someone really wants this, they can start and end the critical section manually. */
+                    OpKind::BuiltIn(BuiltinOp::FarJump(_)) => {
+                        return Err(CrsnError::Parse("Far jump a critical section!".into(), op.pos()));
+                    }
+                    _ => {}
+                }
+            }
+
+            let vec : Vec<Box<dyn Flatten>> = vec![
+                Box::new(Op {
+                    kind: OpKind::BuiltIn(BuiltinOp::CriticalBegin),
+                    cond: None,
+                    pos: pos.clone(),
+                }),
+                Box::new(flattened),
+                Box::new(Op {
+                    kind: OpKind::BuiltIn(BuiltinOp::CriticalEnd),
+                    cond: None,
+                    pos: pos.clone(),
+                }),
+            ];
+
+            return Ok(ParseRes::Parsed(Box::new(vec)));
+        }
+
+        if keyword == "loop" {
+            let mut end_label = None;
+            let label = if let Some(Sexp::Atom(Atom::S(_), _)) = tokens.peek() {
+                let label = parse_label(tokens.next().unwrap())?;
+                if let Label::Named(n) = &label {
+                    end_label = Some(Label::Named(format!("{}-end", n)));
+                }
+                label
+            } else {
+                Label::unique(&tokens.pcx.state.borrow().label_num)
+            };
+
+            let pcx = tokens.pcx;
+            let inner = parse_instructions(tokens.into_iter(), pos, pcx)?;
+
+            let mut vec : Vec<Box<dyn Flatten>> = vec![
+                Box::new(Op {
+                    kind: OpKind::BuiltIn(BuiltinOp::Label(label.clone())),
+                    cond: None,
+                    pos: pos.clone(),
+                }),
+                inner,
+                Box::new(Op {
+                    kind: OpKind::BuiltIn(BuiltinOp::Jump(label)),
+                    cond: None,
+                    pos: pos.clone(),
+                }),
+            ];
+
+            if let Some(el) = end_label {
+                vec.push(Box::new(Op {
+                    kind: OpKind::BuiltIn(BuiltinOp::Label(el)),
+                    cond: None,
+                    pos: pos.clone(),
+                }));
+            }
+
+            return Ok(ParseRes::Parsed(Box::new(vec)))
+        }
+
+        Ok(ParseRes::Unknown(tokens))
+    }
 }

crsn/src/builtin/parse.rs

@@ -25,21 +25,21 @@ pub(crate) fn parse_op<'a>(op_pos: &SourcePosition, keyword: &str, mut args: Tok
             BuiltinOp::Halt
         }
 
-        "uslp" => {
+        "uslp" | "usleep" => {
             BuiltinOp::Sleep {
                 count: args.next_rd()?,
                 unit_us: SleepUnit::Usec,
             }
         }
 
-        "mslp" => {
+        "mslp" | "msleep" => {
             BuiltinOp::Sleep {
                 count: args.next_rd()?,
                 unit_us: SleepUnit::Msec,
             }
         }
 
-        "sslp" => {
+        "sslp" | "ssleep" => {
             BuiltinOp::Sleep {
                 count: args.next_rd()?,
                 unit_us: SleepUnit::Sec,
@@ -132,7 +132,7 @@ pub(crate) fn parse_op<'a>(op_pos: &SourcePosition, keyword: &str, mut args: Tok
             for t in args {
                 call_args.push(parse_rd(t, pcx)?);
             }
-            BuiltinOp::Call(dest, call_args)
+            BuiltinOp::Call { proc: dest, args: call_args }
         }
 
         "ret" => {
@@ -183,6 +183,21 @@ pub(crate) fn parse_op<'a>(op_pos: &SourcePosition, keyword: &str, mut args: Tok
             })
         }
 
+        "crit-begin" => {
+            BuiltinOp::CriticalBegin
+        }
+
+        "crit-end" => {
+            BuiltinOp::CriticalEnd
+        }
+
+        "rt-opt" => {
+            BuiltinOp::RuntimeOpt {
+                opt: args.next_rd()?,
+                value: args.next_rd()?,
+            }
+        }
+
         "ld" => {
             BuiltinOp::Load {
                 dst: args.next_wr()?,
@@ -258,6 +273,38 @@ pub(crate) fn parse_op<'a>(op_pos: &SourcePosition, keyword: &str, mut args: Tok
             }
         }
 
+        "spawn" => {
+            let handle = args.next_wr()?;
+            let dest = RoutineName { name: args.next_string()?.0, arity: args.len() as u8 };
+
+            let mut call_args = vec![];
+            for t in args {
+                call_args.push(parse_rd(t, pcx)?);
+            }
+
+            BuiltinOp::Spawn {
+                handle,
+                proc: dest,
+                args: call_args
+            }
+        }
+
+        "join" => {
+            BuiltinOp::Join(args.next_rdobj()?)
+        }
+
+        "yield" => {
+            if args.have_more() {
+                BuiltinOp::Yield {
+                    value: Some(args.next_rd()?),
+                }
+            } else {
+                BuiltinOp::Yield {
+                    value: None,
+                }
+            }
+        }
+
         "del" => {
             BuiltinOp::Delete(args.next_rdobj()?)
         }
@@ -289,6 +336,10 @@ pub(crate) fn parse_op<'a>(op_pos: &SourcePosition, keyword: &str, mut args: Tok
                 return Ok(ParseRes::builtin(BuiltinOp::Exchange { a, b, mask }));
             }
 
+            if let Some((dst, expected, src, mask)) = args.parse_masked_rdwr_rd_rd(other, "cas")? {
+                return Ok(ParseRes::builtin(BuiltinOp::CompareSwap { dst, expected, src, mask }));
+            }
+
             if other.starts_with(':') {
                 BuiltinOp::Label(parse_label_str(other, &op_pos)?)
             } else {
@@ -313,6 +364,9 @@ pub(crate) fn parse_routine_name(name: String, pos: &SourcePosition) -> Result<R
 
 pub(crate) fn to_sexp(op: &BuiltinOp) -> Sexp {
     match op {
+        BuiltinOp::CriticalBegin => sexp::list(&[A("crit-begin")]),
+        BuiltinOp::CriticalEnd => sexp::list(&[A("crit-end")]),
+        BuiltinOp::RuntimeOpt { opt, value } => sexp::list(&[A("rt-opt"), A(opt), A(value)]),
         BuiltinOp::Nop => sexp::list(&[A("nop")]),
         BuiltinOp::Halt => sexp::list(&[A("halt")]),
         BuiltinOp::Sleep { count: micros, unit_us: SleepUnit::Sec } => sexp::list(&[A("sslp"), A(micros)]),
@@ -322,7 +376,7 @@ pub(crate) fn to_sexp(op: &BuiltinOp) -> Sexp {
         BuiltinOp::Jump(label) => sexp::list(&[A("j"), A(label)]),
         BuiltinOp::FarLabel(label) => sexp::list(&[A("far"), A(label)]),
         BuiltinOp::FarJump(label) => sexp::list(&[A("fj"), A(label)]),
-        BuiltinOp::Call(name, args) => {
+        BuiltinOp::Call { proc: name, args } => {
             if args.is_empty() {
                 sexp::list(&[A("call"), A(&name.name)])
             } else {
@@ -389,6 +443,13 @@ pub(crate) fn to_sexp(op: &BuiltinOp) -> Sexp {
                 sexp::list(&[A(format!("xch{}", mask.width)), AM(a, mask.dst_pos), AM(b, mask.src_pos)])
             }
         },
+        BuiltinOp::CompareSwap { dst, expected, src, mask } => {
+            if mask.is_full() {
+                sexp::list(&[A("cas"), A(dst), A(expected), A(src)])
+            } else {
+                sexp::list(&[A(format!("cas{}", mask.width)), AM(dst, mask.dst_pos), AM(expected, mask.src_pos), AM(src, mask.src2_pos)])
+            }
+        },
         BuiltinOp::StoreFlags { dst } => sexp::list(&[A("stf"), A(dst)]),
         BuiltinOp::LoadFlags { src } => sexp::list(&[A("ldf"), A(src)]),
         BuiltinOp::LoadSequence { dst, value } => {
@@ -405,6 +466,22 @@ pub(crate) fn to_sexp(op: &BuiltinOp) -> Sexp {
                 }
             }
         }
+        BuiltinOp::Spawn { handle, proc, args } => {
+            if args.is_empty() {
+                sexp::list(&[A("spawn"), A(handle), A(proc)])
+            } else {
+                let mut v = vec![A("spawn"), A(handle), A(&proc.name)];
+                v.extend(args.iter().map(|r| A(r)));
+                sexp::list(&v)
+            }
+        }
+        BuiltinOp::Join(handle) => sexp::list(&[A("join"), A(handle)]),
+        BuiltinOp::Yield { value } => {
+            match value {
+                None => sexp::list(&[A("yield")]),
+                Some(rd) => sexp::list(&[A("yield"), A(rd)]),
+            }
+        }
     }
 }
 
@@ -414,10 +491,15 @@ mod test {
 
     use sexp::SourcePosition;
 
-    use crate::asm::parse::{parse_instructions, ParserContext};
+    use crate::asm::parse::{parse_instructions, ParserContext, ParserState};
     use crate::asm::parse::sexp_expect::expect_list;
     use crate::builtin::BuiltinOps;
     use crate::module::OpTrait;
+    use std::cell::RefCell;
+    use std::sync::Arc;
+    use crate::runtime::run_thread::{ThreadInfo, ThreadToken, RunState};
+    use crate::runtime::program::Program;
+    use crate::runtime::frame::REG_COUNT;
 
     #[test]
     fn roundtrip() {
@@ -474,6 +556,8 @@ mod test {
             ("(xch r0 r1)", "(xch r0 r1)"),
             ("(xch32 r0 r1)", "(xch32 r0 r1)"),
             ("(xch32 r0:8 r1:16)", "(xch32 r0:8 r1:16)"),
+            ("(cas r0 r1 r2)", "(cas r0 r1 r2)"),
+            ("(cas8 r0:8 r1:16 r2:24)", "(cas8 r0:8 r1:16 r2:24)"),
             ("(ld r0 r0)", "(ld r0 r0)"),
             ("(ld8 r0 r1)", "(ld8 r0 r1)"),
             ("(ld16 r0 r1)", "(ld16 r0 r1)"),
@@ -492,17 +576,53 @@ mod test {
             ("(far :label)", "(far :label)"),
             ("(del @r5)", "(del @r5)"),
             ("(sym cat r0)(del @cat)", "(del @r0)"),
+            ("(spawn r0 foo 1 2 3)", "(spawn r0 foo 1 2 3)"),
+            ("(yield)", "(yield)"),
+            ("(yield -1)", "(yield -1)"),
+            ("(yield r5)", "(yield r5)"),
+            ("(join @r5)", "(join @r5)"),
         ];
 
         let parser = BuiltinOps::new();
 
-        let parsers = &[parser];
+        let parsers = Arc::new(vec![parser]);
+
+        let ti = Arc::new(ThreadInfo {
+            id: ThreadToken(0),
+            uniq: Default::default(),
+            program: Program::new(vec![], parsers.clone()).unwrap(),
+            cycle_time: Default::default(),
+            scheduler_interval: Default::default(),
+            extensions: parsers.clone(),
+        });
+
+        let pcx = ParserContext {
+            parsers: &parsers,
+            state: RefCell::new(ParserState {
+                reg_aliases: Default::default(),
+                reg_alias_stack: vec![],
+                global_reg_aliases: Default::default(),
+                constants: Default::default(),
+
+                // This is a fake thread to pass to constant expressions when evaluating them.
+                // This allows to evaluate nearly all instructions at compile time.
+                const_eval: RunState {
+                    thread_info: ti.clone(),
+                    cr: Default::default(),
+                    parked: Default::default(),
+                    global_regs: [0; REG_COUNT],
+                    ext_data: Default::default(),
+                    cr_deadline: None,
+                    critical_section: 0,
+                },
+                const_eval_ti: ti.clone(),
+                parsing_expr: false,
+                label_num: Default::default()
+            }),
+        };
 
         for (sample, expected) in samples {
-            let pcx = ParserContext {
-                parsers,
-                state: Default::default(),
-            };
+            pcx.state.borrow_mut().reset();
 
             println!("Parse: {}", sample);
 
@@ -521,10 +641,7 @@ mod test {
             assert_eq!(expected, exported);
 
             println!(" - 2nd cycle");
-            let pcx = ParserContext {
-                parsers,
-                state: Default::default(),
-            };
+            pcx.state.borrow_mut().reset();
 
             /* second cycle, nothing should change */
             let s = sexp::parse(&format!("({})", exported))

crsn/src/module/eval_res.rs

@@ -1,3 +1,6 @@
+use crate::asm::data::literal::Value;
+use std::time::Duration;
+
 /// Cycles spent executing an instruction
 pub type CyclesSpent = usize;
 
@@ -6,6 +9,7 @@ pub type CyclesSpent = usize;
 pub struct EvalRes {
     pub cycles: CyclesSpent,
     pub advance: i64,
+    pub sched: SchedSignal,
 }
 
 impl Default for EvalRes {
@@ -13,6 +17,24 @@ impl Default for EvalRes {
         Self {
             cycles: 1,
             advance: 1,
+            sched: SchedSignal::Normal,
         }
     }
 }
+
+/// Signal to the scheduler
+#[derive(Debug)]
+pub enum SchedSignal {
+    /// No signal, execution went normally.
+    Normal,
+    /// Yield control, optionally with a value.
+    /// If a value is yielded, it must be consumed through the handle before execution can resume.
+    Yield(Option<Value>),
+    /// The routine requests a delay in execution. The actual sleep time can be longer due to task
+    /// switching overhead.
+    Sleep(Duration),
+    /// Return from a coroutine - the thread ends and should be joined.
+    Ret(Vec<Value>),
+    /// Request to join a coroutine/thread
+    Join(Value),
+}

crsn/src/module/mod.rs

@@ -2,7 +2,7 @@
 
 use std::fmt::Debug;
 
-pub use eval_res::EvalRes;
+pub use eval_res::*;
 use sexp::{Sexp, SourcePosition};
 
 use crate::asm::data::literal::Value;

crsn/src/runtime/exec.rs

@@ -8,9 +8,9 @@ impl RunThread {
         let state = &mut self.state;
         let info = &self.info;
 
-        let op = info.program.fetch_instr(state.frame.pc);
+        let op = info.program.fetch_instr(state.cr.frame.pc);
 
-        trace!("### {:04} : {:?}", state.frame.pc.0, op);
+        trace!("### {:04} : {:?}", state.cr.frame.pc.0, op);
 
         op.execute(info, state)
     }

crsn/src/runtime/fault.rs

@@ -26,6 +26,9 @@ pub enum Fault {
     #[error("Program ended.")]
     Halt,
 
+    #[error("Instruction did not finish in time for context switch, retry later")]
+    Blocked,
+
     #[error("Operation not allowed: {0}")]
     NotAllowed(Cow<'static, str>),
 
@@ -74,6 +77,18 @@ pub enum Fault {
 
     #[error("Attempt to read undefined extension data store")]
     ExtDataNotDefined,
+
+    #[error("Unbalanced critical sections")]
+    UnbalancedCriticalSection,
+
+    #[error("Deadlock detected")]
+    Deadlock,
+
+    #[error("Root routine returned or yielded a value")]
+    RootReturned,
+
+    #[error("System IO error: {0}")]
+    IOError(#[from] std::io::Error),
 }
 
 #[derive(Error, Debug)]

crsn/src/runtime/frame.rs

@@ -33,10 +33,14 @@ impl StackFrame {
         sf
     }
 
-    #[inline(always)]
     pub fn set_retvals(&mut self, vals: &[Value]) {
         for n in 0..(vals.len().min(REG_COUNT)) {
             self.res[n] = vals[n];
         }
+
+        // Zero the rest
+        for n in vals.len()..REG_COUNT {
+            self.res[n] = 0;
+        }
     }
 }

crsn/src/runtime/run_thread.rs

@@ -1,16 +1,20 @@
 use std::sync::Arc;
 
-use std::thread::JoinHandle;
-use std::time::{Duration};
+use std::time::{Duration, Instant};
 
 pub use info::ThreadInfo;
 pub use state::RunState;
 
 use crate::asm::data::literal::Addr;
-use crate::module::{EvalRes, CrsnUniq};
+use crate::module::{EvalRes, CrsnUniq, SchedSignal};
 use crate::runtime::fault::Fault;
 use crate::runtime::frame::{StackFrame, REG_COUNT};
 use crate::runtime::program::Program;
+use crate::runtime::run_thread::state::{CoroutineContext, CoroutineState};
+use std::{mem, thread};
+use crate::asm::instr::cond::Flag;
+use parking_lot::RwLock;
+use crate::asm::instr::Flatten;
 
 #[derive(Clone, Copy, Eq, PartialEq, Debug, Ord, PartialOrd)]
 pub struct ThreadToken(pub u32);
@@ -29,6 +33,7 @@ pub struct ThreadParams<'a> {
     pub program: Arc<Program>,
     pub pc: Addr,
     pub cycle_time: Duration,
+    pub scheduler_interval: Duration,
     pub args: &'a [u64],
 }
 
@@ -36,20 +41,30 @@ impl RunThread {
     pub fn new(params: ThreadParams) -> Self {
         let extensions = params.program.extensions.clone();
 
+        // TODO investigate if this division to 2 structs is still needed
+
         let ti = Arc::new(ThreadInfo {
             id: params.id,
             uniq: params.uniq,
             program: params.program,
             cycle_time: params.cycle_time,
+            scheduler_interval: RwLock::new(params.scheduler_interval),
             extensions,
         });
 
         let rs = RunState {
             thread_info: ti.clone(),
-            frame: StackFrame::new(params.pc, params.args),
-            call_stack: vec![],
+            cr: CoroutineContext {
+                handle: 0, // this is the root
+                frame: StackFrame::new(params.pc, params.args),
+                call_stack: vec![],
+                cr_state: Default::default(),
+            },
+            parked: Default::default(),
             global_regs: [0; REG_COUNT],
             ext_data: Default::default(),
+            cr_deadline: None,
+            critical_section: 0,
         };
 
         Self {
@@ -58,41 +73,219 @@ impl RunThread {
         }
     }
 
-    /// Spawn as a thread
-    pub fn spawn(self) -> JoinHandle<()> {
-        std::thread::spawn(move || {
-            self.run();
-        })
-    }
-
     /// Start synchronously
     pub fn run(mut self) {
         let mut loop_helper = spin_sleep::LoopHelper::builder()
             .build_with_target_rate(1.0/self.info.cycle_time.as_secs_f64());
 
         loop_helper.loop_start();
-        'run: loop {
+        let mut orig_pc;
+
+        let fault = 'run: loop {
+            let mut want_switch = false;
+            orig_pc = self.state.cr.frame.pc;
             match self.eval_op() {
-                Ok(EvalRes { cycles, advance }) => {
+                Ok(EvalRes { cycles, advance, sched }) => {
                     for _ in 0..cycles {
                         loop_helper.loop_sleep();
                         loop_helper.loop_start();
                     }
-                    trace!("Step {}; Status = {}", advance, self.state.frame.status);
-                    self.state.frame.pc.advance(advance);
+                    trace!("Step {}; Status = {}", advance, self.state.cr.frame.status);
+                    self.state.cr.frame.pc.advance(advance);
+
+                    if let Some(dl) = self.state.cr_deadline {
+                        want_switch = dl <= Instant::now();
+                    }
+
+                    match sched {
+                        SchedSignal::Normal => {}
+                        SchedSignal::Yield(None) => {
+                            trace!("yield");
+                            self.state.cr.cr_state = CoroutineState::Ready;
+                            want_switch = true;
+                        }
+                        SchedSignal::Yield(Some(value)) => {
+                            trace!("yield {}", value);
+                            self.state.cr.cr_state = CoroutineState::YieldedValue(value);
+                            want_switch = true;
+                        }
+                        SchedSignal::Sleep(time) => {
+                            trace!("sleep {:?}", time);
+                            self.state.cr.cr_state = CoroutineState::Sleep { due: Instant::now() + time };
+                            want_switch = true;
+                        }
+                        SchedSignal::Ret(results) => {
+                            self.state.cr.cr_state = CoroutineState::Finished(results);
+                            want_switch = true;
+                            // TODO prioritize a thread that is waiting for this return value
+                        }
+                        SchedSignal::Join(handle) => {
+                            trace!("Join cr {:#}", handle);
+                            let mut found = false;
+                            'find: for (n, th) in self.state.parked.iter_mut().enumerate() {
+                                if th.handle == handle {
+                                    if let CoroutineState::Finished(_) = &th.cr_state {
+                                        let crs = mem::replace(&mut th.cr_state, CoroutineState::Ready);
+                                        self.state.parked.remove(n); // delete it
+                                        found = true;
+
+                                        if let CoroutineState::Finished(vals) = crs {
+                                            self.state.cr.frame.set_retvals(&vals);
+                                            break 'find;
+                                        } else {
+                                            unreachable!();
+                                        }
+                                    } else {
+                                        self.state.cr.frame.pc = orig_pc; // Retry
+                                        self.state.cr.cr_state = CoroutineState::Ready;
+                                        want_switch = true;
+                                        found = true;
+                                    }
+                                }
+                            }
+
+                            if !found {
+                                self.state.set_flag(Flag::Invalid, true);
+                                warn!("Join with invalid thread handle {:#}!", handle);
+                                self.state.cr.frame.set_retvals(&[]);
+                            }
+                        }
+                    }
                 }
-                Err(Fault::Halt) => {
-                    // TODO implement coordinated shutdown when more threads are running!
-                    break 'run;
+                Err(Fault::Blocked) => {
+                    trace!("Thread reports being blocked!");
+                    self.state.cr.frame.pc = orig_pc; // Retry
+                    self.state.cr.cr_state = CoroutineState::Ready;
+                    want_switch = true;
                 }
                 Err(e) => {
-                    error!("Fault: {:?}", e);
-                    error!("Core dump: {:?}", self.state);
-                    break 'run;
+                    break 'run e;
                 }
             }
-        }
 
-        debug!("Thread ended.");
+            // Resolve the next coroutine to run, or wait a bit...
+            'next: loop {
+                if want_switch {
+                    let now = Instant::now();
+
+                    if self.state.critical_section > 0 {
+                        match self.state.cr.cr_state {
+                            CoroutineState::Ready => {}
+                            CoroutineState::Sleep { due } => {
+                                let time = due.saturating_duration_since(now);
+                                trace!("Sleep in critical: {:?}", time);
+                                thread::sleep(time);
+                                continue 'run;
+                            }
+                            CoroutineState::Finished(_) | CoroutineState::YieldedValue(_) => {
+                                // This is not good
+                                break 'run Fault::Deadlock;
+                            }
+                        }
+
+                        // This is either a bug, or waiting for IO.
+                        // If it is the latter, try to give the OS a bit of breathing room..
+                        std::thread::yield_now();
+                        continue 'run;
+                    }
+
+                    trace!("Switch requested");
+
+                    let mut candidate = None;
+                    let mut closest_due = None;
+                    'findbest: for _ in 0..self.state.parked.len() {
+                        if let Some(mut rt) = self.state.parked.pop_front() {
+                            match rt.cr_state {
+                                CoroutineState::Ready => {
+                                    trace!("Found READY thread to run next");
+                                    candidate = Some(rt);
+                                    break 'findbest;
+                                }
+                                CoroutineState::Sleep { due } => {
+                                    if due <= now {
+                                        trace!("Found DUE sleeping thread to run next");
+                                        rt.cr_state = CoroutineState::Ready;
+                                        candidate = Some(rt);
+                                        break 'findbest;
+                                    } else {
+                                        match closest_due {
+                                            Some(d) => {
+                                                if d > due {
+                                                    closest_due = Some(due);
+                                                }
+                                            },
+                                            None => {
+                                                closest_due = Some(due);
+                                            }
+                                        }
+                                        self.state.parked.push_back(rt);
+                                    }
+                                }
+                                _ => {
+                                    self.state.parked.push_back(rt);
+                                }
+                            }
+                        }
+                    }
+
+                    if let Some(cr) = candidate {
+                        trace!("Context switch to {:?}", cr);
+
+                        // Do switch
+                        let old = mem::replace(&mut self.state.cr, cr);
+                        self.state.parked.push_back(old);
+
+                        self.state.start_task_switching();
+
+                    } else if let Some(due) = closest_due {
+                        if self.state.cr.cr_state == CoroutineState::Ready {
+                            trace!("No candidate found to run, retry same thread.");
+                            // Let it run another quantum, maybe it was just blocked
+                            continue 'run;
+                        }
+
+                        let time = due.saturating_duration_since(now);
+                        trace!("No thread to switch to, sleep {:?}", time);
+                        thread::sleep(time);
+                        continue 'next;
+                    } else {
+                        // Nothing to run?
+                        thread::yield_now();
+                        trace!("No thread to switch to!");
+                    }
+
+                    let n_alive = self.state.parked.iter()
+                        .filter(|p| p.cr_state.is_alive()).count();
+
+                    if n_alive == 0 {
+                        trace!("Stop task switching, no parked threads are alive");
+                        self.state.stop_task_switching(); // This should improve performance in single-threaded mode
+                    }
+
+                    match self.state.cr.cr_state {
+                        CoroutineState::Finished(_) | CoroutineState::YieldedValue(_) => {
+                            break 'run Fault::RootReturned;
+                        }
+                        _ => {}
+                    }
+                }
+
+                break 'next;
+            }
+        };
+
+        match fault {
+            Fault::Halt => {
+                debug!("Thread ended.");
+            }
+            e => {
+                if let Some(instr) = self.info.program.ops.get(orig_pc.0 as usize) {
+                    error!("Fault at {}: {}", instr.pos(), e);
+                } else {
+                    error!("Fault at PC {}: {}", orig_pc, e);
+                }
+                warn!("Core dump: {:?}", self.state);
+            }
+        }
     }
 }

crsn/src/runtime/run_thread/info.rs

@@ -6,6 +6,7 @@ use crate::asm::data::literal::Value;
 use crate::runtime::program::Program;
 use crate::runtime::run_thread::ThreadToken;
 use crate::module::{CrsnExtension, CrsnUniq};
+use parking_lot::RwLock;
 
 #[derive(Debug)]
 pub struct ThreadInfo {
@@ -17,6 +18,8 @@ pub struct ThreadInfo {
     pub program: Arc<Program>,
     /// Program to run
     pub(crate) cycle_time: Duration,
+    /// Interval one thread/coroutine is let to run before the context switches
+    pub(crate) scheduler_interval: RwLock<Duration>,
     /// Extensions
     pub extensions: Arc<Vec<Box<dyn CrsnExtension>>>,
 }

crsn/src/runtime/run_thread/state.rs

@@ -1,5 +1,5 @@
 use std::any::{Any, TypeId};
-use std::collections::HashMap;
+use std::collections::{HashMap, VecDeque};
 
 use crate::asm::data::{Rd, RdData, RdObj, Register, Wr, WrData};
 use crate::asm::data::literal::{Addr, Value};
@@ -12,34 +12,114 @@ use nudge::{likely};
 use crate::asm::instr::cond::Flag;
 use std::fmt::{Debug, Formatter};
 use std::fmt;
+use std::time::{Instant, Duration};
 
 pub struct RunState {
     pub thread_info: Arc<ThreadInfo>,
-    /// Active stack frame
-    pub frame: StackFrame,
-    /// Call stack
-    pub call_stack: CallStack,
+    /// The active coroutine
+    pub cr: CoroutineContext,
+    /// Parked coroutines
+    pub parked: VecDeque<CoroutineContext>,
     /// General purpose registers that stay valid for the entire run-time of the thread
     pub global_regs: [Value; REG_COUNT],
     /// Extension data
     pub ext_data: ExtensionDataStore,
+    /// Execution deadline, if multi-tasked
+    pub cr_deadline: Option<Instant>,
+    /// Nonzero if inside a critical section
+    pub critical_section: Value,
+}
+
+#[derive(Debug,Default)]
+pub struct CoroutineContext {
+    pub handle: Value,
+    /// Active stack frame
+    pub frame: StackFrame,
+    /// Call stack
+    pub call_stack: CallStack,
+    /// Coroutine run state
+    pub cr_state: CoroutineState,
+}
+
+/// Execution state of an inactive coroutine
+#[derive(Debug, Eq, PartialEq)]
+pub enum CoroutineState {
+    /// Ready to run, just started, or interrupted by the scheduler
+    Ready,
+    /// Delay in progress
+    Sleep { due: Instant },
+    /// The task finished
+    Finished(Vec<Value>),
+    /// The task yielded a value that must be consumed before it can resume.
+    /// State switches to Ready when the value is read.
+    YieldedValue(Value),
+}
+
+impl CoroutineState {
+    pub fn is_alive(&self) -> bool {
+        match self {
+            CoroutineState::Ready => true,
+            CoroutineState::Sleep { .. } => true,
+            CoroutineState::Finished(_) => false,
+            CoroutineState::YieldedValue(_) => true,
+        }
+    }
+}
+
+impl Default for CoroutineState {
+    fn default() -> Self {
+        Self::Ready
+    }
 }
 
 impl RunState {
     /// Clear everything. Caution: when done at runtime, this effectively reboots the thread
-    pub fn clear_all(&mut self) {
-        self.frame = Default::default();
-        self.call_stack = Default::default();
+    pub fn reset(&mut self) {
+        self.cr = Default::default();
+        self.parked = Default::default();
         self.global_regs = Default::default();
         self.ext_data = Default::default();
     }
+
+    /// Add a coroutine, marked as Ready, to run next
+    pub fn add_coroutine(&mut self, frame : StackFrame) -> Value {
+        let handle = self.thread_info.unique_handle();
+        trace!("Spawn cr {:#}", handle);
+
+        // front - so it runs ASAP
+        self.parked.push_front(CoroutineContext {
+            handle,
+            frame,
+            call_stack: vec![],
+            cr_state: Default::default()
+        });
+        if self.cr_deadline.is_none() {
+            // start context switching
+            self.start_task_switching();
+        }
+        handle
+    }
+
+    pub(crate) fn start_task_switching(&mut self) {
+        let ival = *self.thread_info.scheduler_interval.read();
+        if ival > Duration::default() {
+            self.cr_deadline = Some(Instant::now() + ival);
+        } else {
+            // Disabled
+            self.cr_deadline = None;
+        }
+    }
+
+    pub(crate) fn stop_task_switching(&mut self) {
+        self.cr_deadline = None;
+    }
 }
 
 impl Debug for RunState {
     fn fmt(&self, f: &mut Formatter<'_>) -> fmt::Result {
         f.debug_struct("RunState")
-            .field("frame", &self.frame)
-            .field("call_stack", &self.call_stack)
+            .field("cr", &self.cr)
+            .field("parked", &self.parked)
             .field("global_regs", &self.global_regs)
             //.field("ext_data", &self.ext_data)
             .finish()
@@ -79,44 +159,44 @@ impl RunState {
     #[inline(always)]
     pub fn set_flag(&mut self, flag: Flag, set: bool) {
         trace!("Flag {} = {:?}", flag, set);
-        self.frame.status.set(flag, set);
+        self.cr.frame.status.set(flag, set);
     }
 
     /// Check status flags for a condition
     #[inline(always)]
     pub fn test_cond(&self, cond: Cond) -> bool {
-        self.frame.status.test(cond)
+        self.cr.frame.status.test(cond)
     }
 
     /// Set program counter - address of the next instruction to run
     pub fn set_pc(&mut self, pc: Addr) {
         trace!("PC := {}", pc);
-        self.frame.pc = pc;
+        self.cr.frame.pc = pc;
     }
 
     /// Get program counter - address of the next instruction to run
     pub fn get_pc(&self) -> Addr {
-        self.frame.pc
+        self.cr.frame.pc
     }
 
     /// Clear status flags
     #[inline(always)]
     pub fn clear_status(&mut self) {
-        self.frame.status.clear();
+        self.cr.frame.status.clear();
     }
 
     /// Update status flags using a variable.
     /// The update is additive - call `clear_status()` first if desired!
     #[inline(always)]
     pub fn update_status(&mut self, val: Value) {
-        self.frame.status.update(val);
+        self.cr.frame.status.update(val);
     }
 
     /// Update status flags using a variable.
     /// The update is additive - call `clear_status()` first if desired!
     #[inline(always)]
     pub fn update_status_float(&mut self, val: f64) {
-        self.frame.status.update_float(val);
+        self.cr.frame.status.update_float(val);
     }
 
     /// Read object handle value
@@ -133,8 +213,8 @@ impl RunState {
             RdData::Immediate(v) => Ok(v),
             RdData::Register(Register::Gen(rn)) => {
                 if likely(rn < REG_COUNT as u8) {
-                    trace!("Rd {:?} = {}", rd, self.frame.gen[rn as usize]);
-                    Ok(self.frame.gen[rn as usize])
+                    trace!("Rd {:?} = {}", rd, self.cr.frame.gen[rn as usize]);
+                    Ok(self.cr.frame.gen[rn as usize])
                 } else {
                     Err(Fault::RegisterNotExist { reg: Register::Gen(rn) })
                 }
@@ -149,16 +229,16 @@ impl RunState {
             }
             RdData::Register(Register::Arg(rn)) => {
                 if likely(rn < REG_COUNT as u8) {
-                    trace!("Rd {:?} = {}", rd, self.frame.arg[rn as usize]);
-                    Ok(self.frame.arg[rn as usize])
+                    trace!("Rd {:?} = {}", rd, self.cr.frame.arg[rn as usize]);
+                    Ok(self.cr.frame.arg[rn as usize])
                 } else {
                     Err(Fault::RegisterNotExist { reg: Register::Arg(rn) })
                 }
             }
             RdData::Register(Register::Res(rn)) => {
                 if likely(rn < REG_COUNT as u8) {
-                    trace!("Rd {:?} = {}", rd, self.frame.res[rn as usize]);
-                    Ok(self.frame.res[rn as usize])
+                    trace!("Rd {:?} = {}", rd, self.cr.frame.res[rn as usize]);
+                    Ok(self.cr.frame.res[rn as usize])
                 } else {
                     Err(Fault::RegisterNotExist { reg: Register::Res(rn) }) // TODO use match after @ when stabilized https://github.com/rust-lang/rust/issues/65490
                 }
@@ -174,6 +254,26 @@ impl RunState {
     }
 
     fn read_object(&mut self, reference: Value) -> Result<Value, Fault> {
+        // values yielded from coroutines
+        for cr in &mut self.parked {
+            if cr.handle == reference {
+                match cr.cr_state {
+                    CoroutineState::Ready | CoroutineState::Sleep { .. } => {
+                        return Err(Fault::Blocked);
+                    }
+                    CoroutineState::Finished(_) => {
+                        self.set_flag(Flag::Eof, true);
+                        warn!("Attempt to read yielded value of a finished coroutine");
+                        return Ok(0);
+                    }
+                    CoroutineState::YieldedValue(v) => {
+                        cr.cr_state = CoroutineState::Ready;
+                        return Ok(v);
+                    }
+                }
+            }
+        }
+
         // This is a shitty dirty hack to allow iterating over the extensions while passing a mutable reference
         // to self to the reading methods. Since the extensions array is in an Arc, it can't be mutated internally
         // anyway, and we know it will still live after the method returns - unless someone does something incredibly stupid.
@@ -213,7 +313,7 @@ impl RunState {
         match wr.0 {
             WrData::Register(Register::Gen(rn)) => {
                 if likely(rn < REG_COUNT as u8) {
-                    self.frame.gen[rn as usize] = val;
+                    self.cr.frame.gen[rn as usize] = val;
                     Ok(())
                 } else {
                     Err(Fault::RegisterNotExist { reg: Register::Gen(rn) })

crsn_buf/src/defs.rs

@@ -45,6 +45,7 @@ pub enum BufOps {
     Write { obj: RdObj, idx: Rd, value: Rd },
     Insert { obj: RdObj, idx: Rd, value: Rd },
     Remove { dst: Wr, obj: RdObj, idx: Rd },
+    CompareSwap { obj: RdObj, idx: Rd, expected: Rd, src: Rd },
 
     // Whole buffer ops
     Resize { obj: RdObj, len: Rd },

crsn_buf/src/exec.rs

@@ -138,6 +138,33 @@ impl OpTrait for BufOps {
                 }
             }
 
+            BufOps::CompareSwap { obj, idx, expected, src } => {
+                state.clear_status();
+                let handle = state.read_obj(obj)?;
+                let idx = state.read(idx)? as usize;
+                let expected = state.read(expected)?;
+                let store: &mut ExtData = state.ext_mut();
+                let buf = store.buffers.get_mut(&handle).ok_or_else(|| Fault::ObjectNotExist(handle))?;
+                if idx < buf.data.len() {
+                    if buf.data[idx] == expected {
+                        // This looks stupid and it is stupid - the dance is needed to satisfy the borrow checker.
+                        let val = state.read(src)?;
+                        let store: &mut ExtData = state.ext_mut();
+                        let buf = store.buffers.get_mut(&handle).unwrap();
+                        buf.data[idx] = val;
+                        state.set_flag(Flag::Equal, true);
+                    }
+                } else if idx == buf.data.len() && expected == 0 {
+                    let val = state.read(src)?;
+                    let store: &mut ExtData = state.ext_mut();
+                    let buf = store.buffers.get_mut(&handle).unwrap();
+                    buf.data.push_back(val);
+                    state.set_flag(Flag::Equal, true);
+                } else {
+                    state.set_flag(Flag::Overflow, true);
+                }
+            }
+
             BufOps::Insert { obj, idx, value } => {
                 state.clear_status();
                 let handle = state.read_obj(obj)?;
@@ -255,6 +282,10 @@ impl OpTrait for BufOps {
                 sexp::list(&[A("bfwr"), A(obj), A(idx), A(value)])
             }
 
+            BufOps::CompareSwap { obj, idx, expected , src } => {
+                sexp::list(&[A("bfcas"), A(obj), A(idx), A(expected), A(src)])
+            }
+
             BufOps::Insert { obj, idx, value } => {
                 sexp::list(&[A("bfins"), A(obj), A(idx), A(value)])
             }

crsn_buf/src/parse.rs

@@ -75,6 +75,14 @@ pub(crate) fn parse<'a>(_pos: &SourcePosition, keyword: &str, mut args: TokenPar
             BufOps::Write { obj, idx, value }
         }
 
+        "bfcas" => {
+            let obj = args.next_rdobj()?;
+            let idx = args.next_rd()?;
+            let expected = args.next_rd()?;
+            let src = args.next_rd()?;
+            BufOps::CompareSwap { obj, idx, expected, src }
+        }
+
         "bfsz" => {
             let dst = args.next_wr()?;
             let obj = args.next_rdobj()?;

crsn_stdio/src/lib.rs

@@ -21,13 +21,29 @@ mod console {
 
     use std::ffi::c_void;
     use std::mem::{self, MaybeUninit};
+    use crsn::runtime::fault::Fault;
+    use std::time::{Instant};
+
+    struct ReadCharState {
+        bytes: [u8; 4],
+        cursor: usize,
+        len: usize,
+        last_timeout : u8,
+    }
+
+    static mut READ_CHAR_STATE: ReadCharState = ReadCharState {
+        bytes: [0; 4],
+        cursor: 0,
+        len: 0,
+        last_timeout : 0,
+    };
 
-    fn setup_fd(fd: RawFd) -> io::Result<libc::termios> {
+    fn setup_fd(fd: RawFd) -> Result<libc::termios, Fault> {
         use libc::*;
 
         let mut tio = MaybeUninit::uninit();
         if 0 != unsafe { tcgetattr(fd, tio.as_mut_ptr()) } {
-            return Err(io::Error::last_os_error());
+            return Err(Fault::IOError(io::Error::last_os_error()));
         }
         let mut tio = unsafe { MaybeUninit::assume_init(tio) };
         let old_tio : termios = unsafe { mem::transmute_copy(&tio) };
@@ -39,36 +55,76 @@ mod console {
         tio.c_cc[VMIN] = 1;
         tio.c_cc[VTIME] = 0;
         if 0 != unsafe { tcsetattr(fd, TCSANOW, &tio) } {
-            return Err(io::Error::last_os_error());
+            return Err(Fault::IOError(io::Error::last_os_error()));
         }
 
         Ok(old_tio)
     }
 
-    pub fn init_io() -> io::Result<libc::termios> {
+    pub fn init_io() -> Result<libc::termios, Fault> {
         setup_fd(libc::STDIN_FILENO)
     }
 
-    pub fn read_byte() -> io::Result<u8> {
+    pub fn read_byte(deadline : Option<Instant>) -> Result<u8, Fault> {
+        // Set TIO timeout
+        let state = unsafe { &mut READ_CHAR_STATE };
+
+        if (state.last_timeout == 0) && deadline.is_none() {
+            // Keep it like that
+        }  else {
+            let vtime = if let Some(dl) = deadline {
+                let timeout = dl.saturating_duration_since(Instant::now());
+                ((timeout.as_secs_f32() * 10.0).round() as u32).min(255).max(1) as u8
+            } else {
+                0
+            };
+
+            if state.last_timeout != vtime {
+                // vtime changes
+                state.last_timeout = vtime;
+
+                let mut tio = MaybeUninit::uninit();
+                if 0 != unsafe { libc::tcgetattr(libc::STDIN_FILENO, tio.as_mut_ptr()) } {
+                    return Err(Fault::IOError(io::Error::last_os_error()));
+                }
+                let mut tio = unsafe { MaybeUninit::assume_init(tio) };
+
+                if vtime > 0 {
+                    tio.c_cc[libc::VTIME] = vtime; /* unit = 0.1 */
+                    tio.c_cc[libc::VMIN] = 0;
+                } else {
+                    tio.c_cc[libc::VTIME] = 0; // no counting
+                    tio.c_cc[libc::VMIN] = 1; // want at least one character
+                }
+
+                unsafe { libc::tcsetattr(libc::STDIN_FILENO, libc::TCSANOW, &tio) };
+            }
+        }
+
         let mut buf = 0u8;
         let len = unsafe { libc::read(libc::STDIN_FILENO, &mut buf as *mut u8 as *mut c_void, 1) };
+
+        if len == 0 && state.last_timeout != 0 {
+            return Err(Fault::IOError(io::Error::new(std::io::ErrorKind::TimedOut, "")));
+        }
+
         if len <= 0 {
-            Err(io::Error::last_os_error())
+            Err(Fault::IOError(io::Error::last_os_error()))
         } else {
             Ok(buf as u8)
         }
     }
 
-    pub fn write_byte(b : u8) -> io::Result<()> {
+    pub fn write_byte(b : u8) -> Result<(), Fault> {
         let len = unsafe { libc::write(libc::STDOUT_FILENO, &b as *const u8 as *const c_void, 1) };
         if len <= 0 {
-            Err(io::Error::last_os_error())
+            Err(Fault::IOError(io::Error::last_os_error()))
         } else {
             Ok(())
         }
     }
 
-    pub fn write_char(c : char) -> io::Result<()> {
+    pub fn write_char(c : char) -> Result<(), Fault> {
         let mut buf = [0u8; 4];
         for b in c.encode_utf8(&mut buf).as_bytes() {
             write_byte(*b)?;
@@ -76,30 +132,43 @@ mod console {
         Ok(())
     }
 
-    pub fn read_char() -> io::Result<char> {
-        let first = read_byte()?;
+    pub fn read_char(deadline : Option<Instant>) -> Result<char, Fault> {
+        let state = unsafe { &mut READ_CHAR_STATE };
 
-        if first & 0x80 == 0 {
-            return Ok(first as char);
-        }
+        if state.cursor == 0 {
+            let first = read_byte(deadline)?;
 
-        let mut bytes = [first, 0, 0, 0];
+            if first & 0x80 == 0 {
+                return Ok(first as char);
+            }
 
-        let remain = if first & 0b1110_0000 == 0b1100_0000 {
-            1
-        } else if first & 0b1111_0000 == 0b1110_0000 {
-            2
-        } else /*if first & 0b1111_1000 == 0b1111_0000*/ {
-            3
-        };
+            state.bytes[0] = first;
+            state.cursor = 1;
+            state.len = if first & 0b1110_0000 == 0b1100_0000 {
+                2
+            } else if first & 0b1111_0000 == 0b1110_0000 {
+                3
+            } else /*if first & 0b1111_1000 == 0b1111_0000*/ {
+                4
+            };
+        }
 
-        for n in 1..=remain {
-            bytes[n] = read_byte()?;
+        let len = state.len;
+
+        while state.cursor < len {
+            let b = read_byte(deadline)?;
+            state.bytes[state.cursor] = b;
+            state.cursor += 1;
         }
 
-        std::str::from_utf8(&bytes[..=remain])
+        let rv = std::str::from_utf8(&state.bytes[..=len])
             .map(|s| s.chars().nth(0).unwrap())
-            .map_err(|e| io::Error::new(io::ErrorKind::InvalidData, e))
+            .map_err(|e| Fault::IOError(io::Error::new(io::ErrorKind::InvalidData, e)));
+
+        state.cursor = 0;
+        state.len = 0;
+
+        rv
     }
 }
 
@@ -183,35 +252,47 @@ impl CrsnExtension for StdioOps {
     fn read_obj(&self, state: &mut RunState, handle: Value)
                 -> Result<Option<Value>, Fault>
     {
+        let deadline = state.cr_deadline;
+
         if handle == self.hdl_stdin {
-            match console::read_char() {
+            match console::read_char(deadline) {
                 Ok(c) => {
                     return Ok(Some(c as Value));
                 }
-                Err(e) => {
+                Err(Fault::IOError(e)) => {
+                    if e.kind() == io::ErrorKind::TimedOut {
+                        return Err(Fault::Blocked);
+                    }
+
                     state.set_flag(Flag::Invalid, true);
                     if e.kind() != io::ErrorKind::InvalidData {
                         state.set_flag(Flag::Eof, true);
                     }
                     return Ok(Some(0));
                 }
+                Err(other) => {
+                    return Err(other);
+                }
             }
-        }
-
-        if handle == self.hdl_stdin_raw {
-            match console::read_byte() {
+        } else if handle == self.hdl_stdin_raw {
+            match console::read_byte(deadline) {
                 Ok(b) => {
                     return Ok(Some(b as Value));
                 }
-                Err(_e) => {
+                Err(Fault::IOError(e)) => {
+                    if e.kind() == io::ErrorKind::TimedOut {
+                        return Err(Fault::Blocked);
+                    }
+
                     state.set_flag(Flag::Invalid, true);
                     state.set_flag(Flag::Eof, true);
                     return Ok(Some(0));
                 }
+                Err(other) => {
+                    return Err(other);
+                }
             }
-        }
-
-        if handle == self.hdl_stdout || handle == self.hdl_stdout_raw {
+        } else if handle == self.hdl_stdout || handle == self.hdl_stdout_raw {
             state.set_flag(Flag::Invalid, true);
             return Ok(Some(0));
         }
@@ -254,13 +335,15 @@ impl CrsnExtension for StdioOps {
     fn read_obj_all(&self, state: &mut RunState, whandle: Wr, rhandle: Value)
                     -> Result<Option<()>, Fault>
     {
+        // XXX This is blocking, there is no sensible way to split it up.
+
         if rhandle == self.hdl_stdin {
             loop {
-                match console::read_char() {
+                match console::read_char(None) {
                     Ok(c) => {
                         state.write(whandle, c as Value)?;
                     }
-                    Err(e) => {
+                    Err(Fault::IOError(e)) => {
                         if e.kind() != io::ErrorKind::InvalidData {
                             state.set_flag(Flag::Eof, true);
                         } else {
@@ -268,17 +351,20 @@ impl CrsnExtension for StdioOps {
                         }
                         return Ok(Some(()));
                     }
+                    Err(other) => {
+                        return Err(other);
+                    }
                 }
             }
         }
 
         if rhandle == self.hdl_stdin_raw {
             loop {
-                match console::read_byte() {
+                match console::read_byte(None) {
                     Ok(c) => {
                         state.write(whandle, c as Value)?;
                     }
-                    Err(e) => {
+                    Err(Fault::IOError(e)) => {
                         if e.kind() != io::ErrorKind::InvalidData {
                             state.set_flag(Flag::Eof, true);
                         } else {
@@ -286,6 +372,9 @@ impl CrsnExtension for StdioOps {
                         }
                         return Ok(Some(()));
                     }
+                    Err(other) => {
+                        return Err(other);
+                    }
                 }
             }
         }

examples/coroutines1.csn

@@ -0,0 +1,38 @@
+(
+    (lds @cout "main\n")
+    (msleep 500)
+    (lds @cout "main\n")
+    (msleep 500)
+
+    (lds @cout "Spawnign bg\n")
+
+    (spawn r15 bg 10)
+
+    (msleep 500)
+    (lds @cout "FG\n")
+    (msleep 500)
+    (lds @cout "FG\n")
+    (msleep 500)
+    (lds @cout "FG\n")
+
+    (msleep 1000)
+
+    (lds @cout "Wait for BG\n")
+    (join @r15)
+    (lds @cout "Joined\n")
+    (msleep 500)
+    (lds @cout "main\n")
+    (msleep 500)
+    (lds @cout "main\n")
+    (msleep 500)
+
+    (proc bg times
+        (ld r0 times)
+        (:x)
+        (msleep 500)
+        (lds @cout "***BG\n")
+        (dec r0 (nz? (j :x)))
+        (lds @cout "***BG done.\n")
+        (ret)
+    )
+)

examples/coroutines2-gen.csn

@@ -0,0 +1,44 @@
+(
+    ; using a coroutine as a generator
+
+    (spawn r15 fibo)
+
+    (ld r0 20)
+    (:next)
+
+    (ld r1 @r15) ; Read a yielded value
+    (call printnum r1)
+    (lds @cout ", ")
+
+    (dec r0)
+    (j.nz :next)
+    (ld @cout '\n')
+    (halt)
+
+    (proc fibo
+        (ld r0 0)
+        (ld r1 1)
+        (:x)
+        (yield r1)
+        (add r2 r0 r1)
+        (ld r0 r1)
+        (ld r1 r2)
+        (j :x)
+    )
+
+    (proc printnum num
+        (mkbf r15)
+        (ld r1 num)
+        (tst r1 (<0? (mul r1 -1)))
+        (:next)
+            (mod r0 r1 10)
+            (add r0 '0')
+            (bfrpush @r15 r0)
+            (div r1 10 (z?
+                (tst num (<0? (bfrpush @r15 '-')))
+                (lds @cout @r15)
+                (del @r15)
+                (ret)))
+            (j :next)
+    )
+)

examples/coroutines3-crit.csn

@@ -0,0 +1,44 @@
+(
+    ; This example shows the use of critical sections.
+
+    ; Set short timeslice (50us) to make the effect more pronounced
+    (rt-opt RT_TIMESLICE 50)
+
+    (spawn _ unsafe 'A' 'Z')
+    (spawn _ unsafe 'a' 'z')
+    (spawn _ safe '0' '9') ; Notice that the sequence 0-9 is always printed in its entirety - because it is in a critical section
+    (msleep 200)
+    (halt)
+
+    (proc unsafe start end
+        ; This can be interrupted any time
+        (:x)
+        (yield)
+        (ld r0 start)
+        (:l)
+            (ld @cout r0)
+            (cmp r0 end)
+            (j.eq :x)
+            (inc r0)
+            (j :l)
+    )
+
+    (proc safe start end
+        (:again)
+        (ld r0 start)
+        ; The sequence will always be complete
+        (crit
+            (ld @cout ' ') ; space to make it easier to read
+            (:l)
+                (ld @cout r0)
+                (cmp r0 end)
+                (j.eq :x)
+                (inc r0)
+                (j :l)
+            (:x)
+            (ld @cout ' ') ; space to make it easier to read
+        )
+        (yield)
+        (j :again)
+    )
+)

examples/coroutines4-io.csn

@@ -0,0 +1,17 @@
+(
+    ; This demo shows that stdin is nearly non-blocking
+
+    (spawn _ ReadAndEcho)
+
+    (:a)
+    (mslp 2500)
+    (lds @cout "Tick\n")
+    (j :a)
+
+    (proc ReadAndEcho
+        (:a)
+        (ld @cout '?')
+        (ld @cout @cin)
+        (j :a)
+    )
+)

examples/itoa.csn

@@ -1,24 +1,41 @@
 (
     ; This is an example implementation of itoa using a buffer of characters
     (ld r0 -123456789)
-    
+
     (mkbf r1)
     (call itoa r1 r0)
-    
+
     ; print it
     (:pn) (bfrpop @cout @r1 (em? (ld @cout '\n') (halt))) (j :pn)
     (halt)
-    
-    (proc itoa buf num    
+
+    (proc itoa buf num
         (ld r1 num)
         (tst r1 (<0? (mul r1 -1)))
         (:next)
             (mod r0 r1 10)
             (add r0 '0')
             (bfrpush @buf r0)
-            (div r1 10 (z? 
+            (div r1 10 (z?
                 (tst num (<0? (bfrpush @buf '-')))
                 (ret)))
             (j :next)
     )
+
+    ; other version that prints it
+    (proc printnum num
+        (mkbf r15)
+        (ld r1 num)
+        (tst r1 (<0? (mul r1 -1)))
+        (:next)
+            (mod r0 r1 10)
+            (add r0 '0')
+            (bfrpush @r15 r0)
+            (div r1 10 (z?
+                (tst num (<0? (bfrpush @r15 '-')))
+                (lds @cout @r15)
+                (del @r15)
+                (ret)))
+            (j :next)
+    )
 )

examples/loop.csn

@@ -0,0 +1,43 @@
+(
+    ; Unlabeled loop. Can be exited by jumping out
+
+    (loop
+        (nop)
+        (nop)
+        (nop)
+    )
+
+    (barrier)
+
+    ; The above is equivalent to:
+
+    (:label)
+    (nop)
+    (nop)
+    (nop)
+    (j :label)
+
+    (barrier)
+
+    ; The label name can be specified.
+    ; This adds a start label and ":label-end" at the end of the loop:
+
+    (loop :repeat
+        (nop)
+        (nop)
+        (j :repeat-end)
+        (nop)
+    )
+
+    (barrier)
+
+    ; The above is equivalent to: (labels changed to avoid a compile error)
+
+    (:repeat2)
+    (nop)
+    (nop)
+    (j :repeat2-end)
+    (nop)
+    (j :repeat2)
+    (:repeat2-end)
+)

launcher/src/main.rs

@@ -38,6 +38,8 @@ struct Config {
     assemble_debug: bool,
     #[serde(with = "serde_duration_millis")]
     cycle_time: Duration,
+    #[serde(with = "serde_duration_millis")]
+    switch_time: Duration,
 }
 
 impl Default for Config {
@@ -51,6 +53,7 @@ impl Default for Config {
             assemble_only: false,
             assemble_debug: false,
             cycle_time: Duration::default(),
+            switch_time: Duration::from_millis(10),
         }
     }
 }
@@ -105,6 +108,14 @@ impl AppConfig for Config {
                     .help("Cycle time (us, append \"s\" or \"ms\" for coarser time)")
                     .takes_value(true),
             )
+            .arg(
+                clap::Arg::with_name("sched")
+                    .long("sched")
+                    .short("S")
+                    .value_name("MILLIS")
+                    .help("Scheduler switch time (ms, append \"u\", \"s\" or \"ms\" for other unit)")
+                    .takes_value(true),
+            )
     }
 
     fn configure(mut self, clap: &ArgMatches) -> anyhow::Result<Self> {
@@ -123,11 +134,28 @@ impl AppConfig for Config {
             } else if t.ends_with("s") {
                 (&t[..(t.len()-1)], 1000_000)
             } else {
-                (t, 1)
+                (t, 1) // us default
             };
 
             self.cycle_time = Duration::from_micros(t.parse::<u64>().expect("parse -C value") * mul);
-            println!("ct = {:?}", self.cycle_time);
+        }
+
+        if let Some(t) = clap.value_of("sched") {
+            let (t, mul) = if t.ends_with("us") {
+                (&t[..(t.len()-2)], 1)
+            } else if t.ends_with("ms") {
+                (&t[..(t.len()-2)], 1000)
+            } else if t.ends_with("m") {
+                (&t[..(t.len()-1)], 1000)
+            } else if t.ends_with("u") {
+                (&t[..(t.len()-1)], 1)
+            } else if t.ends_with("s") {
+                (&t[..(t.len()-1)], 1000_000)
+            } else {
+                (t, 1000) // ms default
+            };
+
+            self.switch_time = Duration::from_micros(t.parse::<u64>().expect("parse -S value") * mul);
         }
         Ok(self)
     }
@@ -177,6 +205,7 @@ fn main() -> anyhow::Result<()> {
         program: parsed,
         pc: Addr(0),
         cycle_time: config.cycle_time,
+        scheduler_interval: config.switch_time,
         args
     });