mittos65/documentation/4 - musl-libc.md

Musl libc in the Mittos kernel

The Mittos kernel is linked with Musl libc which provides functionality such as fine-grained memory management and string formating.

The musl library is a tiny c standard library impelentation which is easy to use. It is however highly linux-specific and therefore relies of the availability of a large number of linux syscalls.

In building musl for the kernel, I patch away this reliance entirely and replace syscalls with calls to internal functions that are prototyped and can then be provided by the kernel or a separate library when needed.

By syncing a patch directory into the musl source with rsync before compilation:

> rsync -a src/patch-musl/ external/musl/

I can provide ovrrides for specific parts of the musl code such as the following which replaces all syscall instructions with calls to external functions:

src/patch-musl/arch/x86_64/syscall_arch.h

#define __SYSCALL_LL_E(x) (x)
#define __SYSCALL_LL_O(x) (x)

extern long syscall0(long);
extern long syscall1(long, long);
...
extern long syscall6(long, long, long, long, long, long, long);

static __inline long __syscall0(long n) {
    return syscall0(n);
}
...
static __inline long __syscall6(long n, long a1, long a2, long a3, long a4, long a5, long a6) {
    return syscall6(n, a1, a2, a3, a4, a5, a6);
}

#define VDSO_USEFUL
#define VDSO_CGT_SYM "__vdso_clock_gettime"
#define VDSO_CGT_VER "LINUX_2.6"
#define VDSO_GETCPU_SYM "__vdso_getcpu"
#define VDSO_GETCPU_VER "LINUX_2.6"

#define IPC_64 0

The kernel is then linked with -L/opt/sysroot/usr/lib -lc, but must provide the functions syscall0(long) through syscall6(long, long, long, long, long, long, long) itself.

At first, those can be very simple:

long syscall0(long num) {
    PANIC("Unknown syscall: %d()\n", num);
}

This will clearly tell me when I try to use a c function that requires support for a syscall, such as malloc which will result in the panic message "Unknown syscall: 12(0)".

Looking into the musl source code, the file arch/x86_64/bits/syscall.h.in tells me that syscall 12 is brk, so that's a good thing to start implementing.

Once that's done, I put a reference to my k_brk function into an array and update the syscallN functions to call my implementations if available:

long syscall0(long num) {
    long retval = 0;
    if(syscall_handlers[num])
        retval = syscall_handlers[num](0, 0, 0, 0, 0, 0);
    else
        PANIC("Unknown syscall: %d()\n", num);
    return retval;
}

The k_brk implementation is very simple too. Just allocate pages until the brk requirement is fulfilled:

static long _brk = KERNEL_BRK0;
long k_brk(long brk, long, long, long, long, long) {
    if(brk) {
        while(_brk < brk) {
            vmm_set_page(kernel_P4, _brk, pmm_alloc(), PAGE_GLOBAL | PAGE_WRITE | PAGE_PRESENT);
            _brk += PAGE_SIZE;
        }
    }
    return _brk;
}

And now I can use standard malloc and calloc in my kernel. After implementing mmap as well, I also have realloc. Nice!

printf

In order to enable printf for debug output, I only need two "syscalls" mocked - ioctl and writev.

When writing to stdout, musl will call ioctl with command 0x5413 to get the terminal size:

long k_ioctl(long fd, long cmd, long arg3, long, long, long) {
    if(fd == 1 && cmd == 0x5413) {
        struct {
            unsigned short ws_row;
            unsigned short ws_col;
            unsigned short ws_xp;
            unsigned short ws_yp;
        } *wsz = (void *)arg3;
        wsz->ws_row = 24;
        wsz->ws_col = 80;
        return 0;
    }
    long retval = -1;
    PANIC("Unknown IOCTL request: fd: %d, command: %x\n", fd, cmd);
    return retval;
}

And it will then call writev with the vectors of data to write:

#include <sys/uio.h> // for struct iovec
long k_writev(long fd, long iov, long iovcnt, long, long, long) {
    if(fd == 1)
    {
        size_t len = 0;
        struct iovec *v = (void *) iov;
        for(int i = 0; i < iovcnt; i++) {
            debug_putsn(v[i].iov_base, v[i].iov_len);
            len += v[i].iov_len;
        }
        return len;
    }
    long retval = 0;
    PANIC("Unknown writev request: fd: %d\n", fd);
    return retval;
}

I then have a fully featured printf implementation to help me in "printf debugging".

Relevant files

• toolchain/build-musl.sh
• src/patch-musl/arch/x86_64
• src/kernel/lib/musl-glue.c
• src/kernel/include/musl-glue.c
• src/kernel/memory/kbrk.c