Vojta's blog

Telneting to HelenOS

2012-01-13T19:30:00.000+01:00

Not that long ago TCP rewrite was merged into HelenOS mainline branch. As a proof of its functionality, webserver was improved recently. That encouraged me to implement "remote console" offering posibility to run applications on HelenOS remotely over the network...

The screenshot below displays HelenOS running in Qemu with GNOME terminal in front with Telnet session to it.

If you want to try it, grab the branch lp:~vojtech-horky/helenos/remote-console and compile it. Then start HelenOS and run remcons from the console. If the networking is okay, the program shall display HelenOS remote console service and wait for incoming connection. The program listens on port 2223 and you can connect to it with following command:

telnet hostname 2223

Do not forget to add port redirection when running in Qemu:

qemu ... -redir tcp:2223::2223

A few notes to the implementation. The program behaves as a console-protocol service and on accepting new network connection, it creates a virtual terminal and runs getterm on it (with bdsh). It then translates the incoming traffic to console events and writing to console to outgoing packets.

Despite the title of this post, the server is far from being a standard-compliant Telnet server. It only sends Telnet commands for switching into character mode (for better interactivity) and then it sends only printable characters. It is also able to skip incoming Telnet commands but that is about all related with the protocol.

The service is not able to handle special keys (the most special keys it handles more or less okay are backspace and tab) and also has no support for drawing. So, no playing Tetris remotely. And national characters are passed through in a very ostrich way ;-).

The biggest flaw of the server is caused by limitations of current TCP implementation in HelenOS. The problem is that when a fibril waits for recv(), no other operation on any socket can occur because all network-related functions hold one global lock. That effectively means that you cannot have two remote sessions.

Apart from the fun of it, this service can be useful for simpler copying of textual data between HelenOS (running in some simulator) and host system. It seems to me easier than mounting virtual harddisks and copying the data off-line.

And that's all. I hope you like this new HelenOS toy ;-).

Two days with HelenOS USB

2011-05-11T21:54:00.000+02:00

We decided to have a coding weekend (the last one) due to approaching deadline for our USB project and I can (again) announce some good news.

First of all, JV fixed the problem of accessing memory mapped registers in IRQ pseudo-code by using a simple hack. During the registration, address space of the caller is remembered and in the handler the address space is switched back. Of course, it is not optimal and I am not sure how this would behave on other architectures, but on Intel it works just fine.

JV also fixed some initialization problems with EHCI. Actually, we do not support EHCI (that is host controller for USB 2.0) but we need to turn it off to allow UHCI and OHCI drivers take control. As a result, all machines we have access to are able to boot and work with our USB branch.

LS a MK again engaged in a fight with USB HID, discovered several problems and fixed a lot of bugs. The support for multimedia keys is almost complete, pressing them shall write their names to the console. As a bonus, LS wrote a very small subdriver that converts mouse wheel to arrows so now you can scroll in edit.

MD was fixing some bugs concerning device initialization and proper device discovery of peripherals plugged at boot time. Unless I am mistaken, we have currently only 2 devices that refuses to get initialized when plugged directly into root hub. Both Microsoft-made mouses. Both not respecting standard Microsoft helped to write. Oh, well...

I did some refactoring of our virtual USB and revived the virtual HID. I hope that the current state would allow to use it as a tester for the HID driver, especially for the HID parser. The HID specification is definitely becoming the most cursed part of our project - it allows almost any sequence of bytes to be a valid descriptor.

I also added a very simple lsusb for listing attached USB devices. For more detailed information, we have usbinfo. Today I fixed a bug in mass storage stub. Tested three flash disks + the emulated in QEMU and all are able to report themselves correctly.

And together we discussed how our presentation shall look like and how we would ship our project to the committee. The deadline is dangerously close.

And that's all for tonight. Stay tuned.

OHCI, HID parser and more

2011-05-02T19:00:00.000+02:00

It has been a while since my last post about USB support for HelenOS. But we have made some progress that I am proud to announce.

First of all, we have functional OHCI driver. OHCI is a host controller supporting USB 1.1 and, unlike UHCI, is not a PC-platform only product (OHCI is used in Macs as well). However, we do our development on PC and we have not tried our driver on anything else. The driver is complete (except for support for isochronous transfers) and was tested on real hardware. It also work in VirtualBox and so now you are not limited to QEMU only for trying HelenOS with USB support.

Since the beginning, our flagship was the USB HID driver and a noticable progress was made there as well. First of all, the base skeleton is finished and I dare say that HelenOS can be fully controlled by USB keyboard without any limitations. Obviously, this does not include the kernel console but that is to be expected. Huge progress was done with the HID parser that is almost completed now.

The HID parser reminds me of regular expressions in Unix. There is a problem inside another one that is of the same size as the outer one. The concept and internals of the parser were already changed several times because the specification does not give enough practical (I really mean practical, not the kind reader will...) examples to create good design from it alone. More than once, we found a keyboard (or mouse) that used settings (i.e. the hardware configuration wired in the physical device) not described well by the specification that forced us to rethink the design. I am not well versed with the parser to elaborate more on its design but it seems to me that HID specification was done purely by hardware vendors whose main concern is to save a few bytes. The rest can be done in software by some poor devils ;-).

To demonstrate the capabilites of the HID driver we decided to add a sample support for multimedia keys. Probably, they will not do anything useful but print some message somewhere but that shall be sufficient as a proof of concept. The support for such special HID devices is created using something we call a subdriver. Such subdriver is a set of callback functions that are registered in the generic HID driver and are called upon data retrieval. Their thask is extract the special keys and control some specific DDF interface to provide these events to client applications. In the future, these so called subdrivers might become fully independent drivers spawned by the generic HID driver. Definitely, that would be task for someone else.

Next we created a simple stub for mass storage driver to test that bulk transfers work. Because HelenOS lacks support for SCSI devices, the stub merely tries to identify the device by sending INQUIRY command. For some devices it works, some stall the communication and some even do not attempt to answer. Probably there is some magic in the initial handshake. However, mass storage support is not our goal and we will probably leave the driver alone for now. Someone else may carry on...

Last thing worth reporting is that JV found some awful bug in QEMU that causes that transfers ending with error are never reported as completed by the simulated UHCI chip. Bad. Although patch was provided, no action was taken yet.

As our deadline is approaching, we need to focus on fixing bugs that we know about, and writing the documentation - both Doxygen one and developers' one. Documentation can be found on-line here. Also, we need to prepare some distribution CD that we can ship to the project committee. We also need to make a presentation and I hope that we would be able to put something together before the next HelenOS Saturday meeting to present our project there. And maybe some refactoring and code beautification would be needed as well.

And that is all. We are planning to end in June so we have less than month for finalization. May the force, debugger and standard compliant devices be with us ;-).

Adding HelenOS to existing GRUB

2011-02-28T11:58:00.001+01:00

Last week we started testing our USB drivers on real hardware. That involves a lot of quick fixes, followed by burning the CD image (or creating bootable USB disk) and rebooting the computer. Not very comfortable. So I decided to polish my script for adding HelenOS to existing GRUB and make it available for everyone.

The script is intended for PC platform and all it does is that it copies HelenOS kernel image to directory where existing GRUB can find it (i.e. the GRUB that is used to boot on real hardware) and adds a menu entry to GRUB.

The usual disclaimer - the script is provided as is. I have used it on my PC and it worked. However, it modifies boot loader files and double checking them before rebooting is more than recommended. Otherwise, you may end up with computer that will not boot at all.

You can download the script here. I placed it in /usr/local/sbin and created sudo entry for it (it needs root privilege to modify /boot directory where GRUB usually resides).

The script can modify GRUB menu in two ways. First, it can append the HelenOS entry to it or replace existing HelenOS entry. I prefer the latter because I do not need to modify the menu file manually at all. In order to use the replacing version, you need to add following two lines to your GRUB menu (usually /boot/grub/menu.lst or /boot/grub/grub.conf). Configuration needed for booting HelenOS will be placed between them.

### HelenOS automatic install start
### HelenOS automatic install end

Once you have done that, you can install HelenOS to it. Build HelenOS and run the script with -h parameter. It will display quite verbose help together with guessed values of several important parameters. If the guesses looks okay, you can proceed to actual installation. I use following command:

sudo /usr/local/sbin/helenos_add_to_grub \
    -c -r -t "HelenOS (USB support)"

This replaces existing entry in GRUB (-r), removes files from previous install (-c) and sets a nice title. The script produces following output informing about performed actions:

==> Cleaning previous install ...
  --> rm -rf /boot/helenos
==> Creating GRUB HelenOS directory ...
  --> mkdir -p /boot/helenos
==> Installing HelenOS kernel and modules ...
  --> cp -R REPO-PATH/boot/distroot/boot /boot/helenos
==> Updating HelenOS GRUB menu ...
==> Backing-up current GRUB menu ...
  --> cp /boot/grub/menu.lst /boot/grub/menu.lst.bak
==> Replacing HelenOS entry in GRUB menu ...
==> Adding HelenOS to GRUB completed.

If you are not using standard locations of GRUB, you can use following parameters to change them:

-g: Path to GRUB menu configuration file
-d: GRUB device name where boot partition is (something like (hdn,k))
-o: Directory where to copy HelenOS files (directory /boot/helenos is used by default)
-O: GRUB path to -o directory (depending on your partition settings, it might be the same as -o or without the /boot prefix)
-b: Where is the boot dir of HelenOS build (REPO_DIR/boot/distroot/boot)

Ideas for improvements are welcomed. Patches with these improvements even more so.

UPDATE: New version supporting GRUB2 can be downloaded here. It also fixes bad usage of getopt. Using it with GRUB2 is quite easy, difference is that as a GRUB configuration file is taken HelenOS file in /etc/grub.d/.

I tested the updated script on Ubuntu in VirtualBox and adding following lines to Makefile in root directory of HelenOS (USB branch checkout) allowed me smooth install:

install-to-grub:
    [ -e "image.iso" ]
    sudo /usr/local/sbin/helenos_add_to_grub \
        --grub2 --title "HelenOS (USB)" \
        -g /etc/grub.d/35_helenos_usb \
        -o /boot/helenosusb -O /boot/helenosusb
    sudo update-grub

Advantage of having GRUB2 is that you can have more entries and removal is pretty simple (just remove entry in /etc/grub.d and run update-grub).

Still, be careful. Your GRUB installation may use slightly different settings. Verify that the /etc/grub.d/35_helenos file looks okay, try to run it to see that it produces reasonable output before running update-grub. Before rebooting double check /boot/grub/grub.cfg.

Hello from USB keyboard to HelenOS

2011-02-04T23:09:00.000+01:00

I am proud to announce that today we were able to control HelenOS with plugged in USB keyboard. Although the abilities of the keyboard and host controller drivers are very limited, it is a huge step forward in our project.

First, a bit about the limitations.

We have run it only in QEMU so far because the host controller driver is not yet in state where we can deploy it onto real hardware.
The keyboard driver accepts only the boot protocol so far (the report protocols are really awful things to parse and process) and is not able to processes composed key sequences (e.g. Shift+<key>) and has no knowledge of automatic key repeat etc.

And before I forgot. We have problems with testing hub driver - we were not able to bridge real USB hubs to QEMU.

Today, you can buy only hubs for USB 2.0 (or better) and although the hubs are backwards compatible (i.e. they support USB 1.1), QEMU refuses to connect them to its emulated UHCI because they are USB 2.0 hubs. And we do not know how to downgrade it. Thus we are looking for some older USB hub that supports only USB 1.1 (or some wizard who knows how to force QEMU to bridge host USB 2.0 hub through its UHCI controller). Doesn't anybody have such hub? Or the knowledge (we have quite a wide range of USB hubs to test with but they are 2.0)?

Now onto more positive stuff: if you want to try it out, get our branch

bzr branch bzr://helenos-usb.bzr.sourceforge.net/bzrroot/helenos-usb/development/ helenos-usb

and compile it for amd64 or ia32.

Running it is a bit trickier. Although one would expect that newer versions of some software are better than the old ones, this is quite not true with the QEMU emulator. Experimentally, we found out several differences between versions 0.13 and 0.11 and the version 0.11 came out better (the simulated keyboard is closer to the real hardware). So, if you want to try it with the emulated keyboard in QEMU, I recommend QEMU 0.11.1. If you trust us and want to try with real keyboard, it worked even with QEMU 0.13 (although I cannot confirm it now as I do not have a USB keyboard at home).

So, let's run it.

qemu-system-x86_64 -m 256 \
    -usb -usbdevice keyboard \
    -cdrom image.iso

If you run it with QEMU 0.11.1, the AT keyboard is not disabled immediately but only after the USB one is properly initialized, so you can switch to virtual console 8 and watch what is going on. In QEMU 0.13, the AT keyboard is disabled once you leave GRUB, but you can switch the consoles with mouse. However, I recommend QEMU 0.11 (I experienced some very weird hung-ups with newer versions of QEMU).

There is a lot of messages, but they should stop and you shall see image similar to the following:

If you see this, everything went fine and HelenOS is now controlled by the USB keyboard. Try pressing F1 to get into some free console and type ls<Enter>.

You might encounter some problems described above, but on a very rough level, HelenOS is controllable by USB keyboard. Isn't that wonderful? Of course it isn't, I hear you there is still a lot of things to be done but, please, do not spoil this very special moment ;-).

Stay tunned for updates.

IPC cheatsheet for HelenOS

2011-01-06T01:04:00.000+01:00

Sometimes as a last resort when debugging HelenOS you need to use the kernel console. If the problem has something to do with IPC, you will end deciphering the IPC communication with ipc command. If you also do not remember that 1031 means request to VFS for file closing, you might find the following cheat sheet useful.

In HelenOS, IPC methods are numbered and enums are used for these. That means that you cannot search the source code for particular number to find the method name. Thus, you would spent a lot of time counting what is the nth item to find out the name (to find out the corresponding handler). I had enough of that very soon. Thus I wanted to get a pencil and rewrite the names on a piece of paper. Well... I stopped even before writing down VFS methods because there are about 30 of them. So I decided to write a script for it...

The PDF I created can be downloaded here. The list is definitely not complete but contains the methods I was dealing with.

If you think the cheat sheet might be useful to you to and you want to extend it, the sources are available as well. I had them placed in /contrib/tools but there is not strict rule about it. You only need to set the PATH2ROOT variable in the Makefile when they are placed elsewhere.

USB hub driver working

2010-12-13T02:35:00.000+01:00

I would love to sing some ode right now because I just found a bug and get a working hub driver, but I am afraid that I would be dressed down by my father because it is half past 1am. So, hence this post to let the euphoria out ;-).

Today (well, yesterday) we had a "coding session" and our whole USB team worked on HelenOS/USB. As a result of this and previous meeting, we can announce some real progress.

First, we tried to tame the HID reports and tried to come up with compromise of simplicity vs. extensibility vs. finishability. Because of the complexity of HID reports in general, we ended (so far) with something very simple but it shall be sufficient for most USB keyboards (fingers crossed). We decided to simply search for the proper usage and usage page and ignore all the rest (see the incomprehensible USB HID specification for more details if you want to know what usage and usage page means). This shall work for keyboard but I am afraid that for rats this would be way to restrictive. We will see.

Next, actual work was started on the keyboard driver itself. So far, there was no use writing "blind" code when there was not possibility to connect the driver to some device. Writing the driver would also include hacking into console to enable more keyboards (simply, we can't allow to cut-off the standard keyboard). At first sight, it looks quite easy... Hopefully, the second sight would confirm that ;-).

And finally, we did quite a lot of work on the hub driver. That included hunting some bugs in the virtual host controller and studying USB specification (chapter 11) again and again to find out what is the correct behavior. Both virtual USB host controller and the hub driver needs some refactoring but they are able to communicate with each other, at least...

And if you plug in a virtual USB keyboard, the hub driver is actually able to detect the change on the port, enable the port and launch the proper driver. Applause expected ;-). All right, I know that after several months that is not that much but in the last weeks we are really accelerating and producing something that even works :-).

If you want to see it by yourself, you can try it. First, get sources from our development branch

bzr://helenos-usb.bzr.sourceforge.net/bzrroot/helenos-usb/development

and compile it:

make HANDS_OFF=y PROFILE=ia32

And then launch it in QEMU. And now be quick :-). Switch to console eight (F8) and you will see devman starting. You shall see that a driver called vhc (for virtual host controller) is started. This driver immediately announces its child device, a root hub and assigns USB address 1 to it. And this root hub is driven by the hub driver.

You will see that the hub driver first asks for hub configuration descriptors and then enables power on all ports (for testing, we reduced the number of ports to only 2). The VHC prints which USB transfers it processes and also status of the ports. You shall see that they start in - (meaning not configured) and enters a disconnected state X. The hub driver then periodically checks for the their change. It is time to plug in the virtual USB keyboard.

Switch to console one and type ls /dev/devices to see also following lines, informing about existing virtual USB devices.

\virt
\virt\usbhc
\virt\usbhc\hub

And now you can start the keyboard. Simply type vuk (stands for virtual USB keyboard) and quickly get back into devman console. You shall see that the port status of port 1 (yeah, ports are numbered starting at 1) changed to disabled (D), quickly went through the resetting state (R) and ended in enabled one (E). Once it entered the enabled state, a new address was assigned to the device and device descriptors were queried to determine the device kind. It all ends with messages by devman, announcing that new device was detected and that it searches for a proper driver.

And driver is actually found and started. If you switch back to the console with vuk, you will see that it reports some pressed keys and that the driver polls the device for its status. Isn't that great? And, of course, listing /dev/devices reveals that a new device was added. The name is awful (something like \virt\usbhc\hub\usb0x00017a00) but using device structure address was the simplest way to avoid name clashes...

And that's all. Just do not look at the code ;-). It needs polishing. It needs a lot of polishing. Once we have done that I will definitely inform you about it and invite you to go through it ;-).

I hope the text makes at least some sense and that it is not full of grammatical and spelling mistakes. And I also hope that I have not offended anyone when writing we...

USB for HelenOS

2010-11-29T01:04:00.000+01:00

Because I am also member of the team that wants to add support for USB to HelenOS and I have enough of programming, I decided to write few paragraphs about HelenOS/USB project...

Disclaimer: text below is not an opinion of the whole team and shall not be taken literally. I am trying to make this post at least a bit fun to read ;-).

We are adding support for USB devices to HelenOS as a part of Software project subject and we officially started this month.

Because we just started, there is not much progress to report. Actually, we had first meeting several months ago, but it takes some time from first thoughts to written specification approved by the Software project committee.

However, we weren't refining the specification all the time. We were studying ahead the USB specifications to which our "product" must comply. Have you ever seen a USB specification? If no, then believe me that you have missed nothing. USB specifications consists of thousands of pages written in technical English by someone (actually, these specifications are always work of a group because solitary person would go crazy from writing it) with apparently no sense of humor. Not talking about different formatting conventions used in individual specifications.

Because of the huge scope of USB, our project goal is limited to USB 1.1 only and we are supposed to create a functional driver for USB keyboard (the official specification talks about HID keyboard but we all hope that it does not mean that we need to implement the whole HID specification which is very long and includes support for such items as gloves for virtual reality software). Also, priority is a static configuration (i.e. no hot unplug).

So, to sum it up, at the end we want to present HelenOS as an OS that is usable with keyboard attached by USB wire. This sounds that it is not much but because HelenOS has no support for USB at all, we are actually starting at zero and want to end with something that both developer and ordinary user can appreciate (well, for HelenOS, these are identical but that is not the point).

But back to our progress I promised to inform you about. In three words, "we are fighting". With what? Luckily, so far, not among ourselves. But there are couple of things we need to get familiar with.

First, we are fighting with Bazaar: it is something different that Git or Mercurial and the worst thing is different terminology and different command naming.

Next, we are fighting with HelenOS itself. It is simply different system than Unix or Windows and getting used to it needs some time. The difference includes the microkernel design in general, asynchronous IPC or fibrils. Plus limited functionality of libraries.

And, of course, we need to unify our coding styles and start writing according to the conventions used in HelenOS. Currently, we are in the phase so common among programming enthusiasts (after all, why we have HelenOS ;-)?): "I do not want to read this (ugly) code, I can do it better by myself" and "my coding style supersedes everything written so far in this file". In this, I am the "enforcer" because I am already past that phase. I had to fight with that months ago when I started working on NUMA ;-).

We decided to use the device driver framework and currently we are fighting with it as well. I believe that using the DDF is a good step in general although right now we are a bit confused. I guess we all expected that it would do more (and that we would not need to touch it at all). We already encountered some problems that might be bugs in DDF. Or maybe we only botched-up something on our side. As soon as we find out where the problem actually is and if it is really in DDF, we will report to the mailing list.

Okay, I hear you ;-). You want to know whether we actually wrote some code or we are only grunting and swearing. So here is our "coding" progress.

We have a very simple mini-framework for creating virtual USB devices to allow testing USB drivers. Currently, it pretends work of host controller with single hub and plugged-in keyboard.

We are now working on generic framework (built above DDF) for creating host controller and hub drivers that will first operate above the virtual host controller. Other efforts, running in parallel, are implementation of UHCI driver and keyboard driver with fixed (i.e. built into the source code, not read dynamically) descriptors. If you have no idea what descriptors I mean, lucky you. Descriptors are way how USB device for human input informs the host about the items it reports (e.g. whether it reports pressed keys or joystick button status or finger bend) and are very generic and the specification is simply awful (I guess it is in English by used prepositions but I am not 100% sure).

And last success is that we are getting used to having "programming sessions" each week where we meet and code together (and we are even having fun there).

And that's all. I hope I haven't omitted something and haven't insulted anyone. I will try to inform about our progress regularly. If this source is not enough (or you simply don't like my style of writing or whatever), you can follow our activities at our Trac...

Support for NUMA in HelenOS

2010-11-23T10:52:00.018+01:00

Because it will be almost 6 months since I started flirting with the idea to base my thesis on HelenOS somehow, it is time to report some progress.

Actually, there are two reasons for this post. First, I think that something has been implemented and it is time to tell the community about it. And also I feel guilty after JS and JJ complained that they feel lack of communication from students participating on HelenOS. So here you have it ;-).

First about intended goal and target platform: my work is aimed at supporting ccNUMA (cache coherent NUMA) hardware on platform amd64 (this is represented e.g. by servers with Opteron processors on NUMA capable motherboard). More precisely, my work is targeted at such (hardware) NUMA configurations where HelenOS is able to run without any NUMA support at all. This means that the address space is continuous across individual nodes and the node boundary is transparent to the OS (i.e. information about memory distribution on the nodes is obtained through other channels and is not necessary for correct memory usage and management), the same for CPUs and also that the hardware does not require any special instructions to ensure cache coherency. And, of course, the main goal is to defend my diploma thesis about making HelenOS NUMA-aware. My supervisor for the thesis is Mr. Děcký.

Now, a few words about simulators I use for development (so far I have not tried to run HelenOS on physical NUMA hardware). Most of the time, I use QEMU which is able to simulate NUMA machine, using the -numa switch. Because it took me some time to understand the syntax of this switch, I post here example command that launches machine with 8 CPUs and 512M of memory, both CPU and memory distributed evenly on 4 nodes:

qemu-system-x86_64 -m 512 -smp 8 \
        -numa node,mem=128,cpus=0-1 \
        -numa node,mem=128,cpus=2-3 \
        -numa node,mem=128,cpus=4-5 \
        -numa node,mem=128,cpus=6-7 \
        -boot d -cdrom ./image.iso

Second simulator I sometimes use is SimNow created by AMD which allows you to build interactively your own machine (including configuration of low level stuff such as northbridges and various other controllers) and run it. The downside is that the simulator is very precise (in the detail of the simulation) and thus extremely slow.

So, this was a bit of the background of the work and now about what was actually added to the source codes of HelenOS. My branch is registered at Launchpad, under HelenOS team as numa branch:

bzr branch lp:~vojtech-horky/helenos/numa

If you want to try it, go ahead: check out the branch and in the configure dialog select amd64 from the preconfigured defaults and do not forget to check the NUMA support option (I added several other options which are useful for debugging, I will remove them later). However, do not be disappointed after the machine boots up. There is nothing shiny and new to be seen - all changes are quite low level and currently the only way to see them is to try the numa application in /app/ directory.

That was the theoretical background. Now a brief summary about what is actually implemented.

First of all, the hardware detection uses ACPI tables (SRAT and SLIT) and assigns to each CPU node id (node index). With memory, it is a bit more complicated. I decided not to add any other layer above existing zones (because that would mean changing a lot of code and, furthermore, each node is usually formed by single zone) and only split zones that run across node boundaries. On AMD64, this typically means that only on the first node is memory split into several zones (some of them are reserved) and all other nodes have only single zone. And that is actually all. There were several problems when to read the topology information from ACPI (I might write about it later), but it works now pretty well.

Next, I made some changes to kernel slab allocator. Here, I decided to add extra layer to the allocator (if you want to know the details of the implementation of the allocator, the one in HelenOS is very similar to the one introduced by Jeff Bonwick), that groups the magazines of CPUs of the same node. This is, of course, enabled only in NUMA build (#ifdefs are used). So far, I have not done any benchmarking so I cannot report whether the change was worth the effort.

To allow more precise thread placement on nodes, I added support for thread affinity. It is nothing extra: each thread has bitmap of CPUs where setting/clearing bit allows the thread to run on respective CPU. The load balancer respects this setting and when the scheduler finds out that it is supposed to schedule thread that cannot run on current CPU, it migrates it away. By default, the thread can run on all CPUs.

For address space areas, I added similar thing plus something that could be called allocation policy. The affinity for AS areas determines from which nodes could be allocated frames that back the area. And the policy specifies how the allocation shall be done. So far, there is first touch (default, allocate from node where is currently running the thread that caused the fault-in) and interleave (allocate from all nodes in the affinity mask evenly, with precise ordering).

Last thing in kernel is automatic page migration (by the way, this is something I am really looking forward to measure). What is it about in a few words? When a thread is migrated on UMA system, its distance to the memory is not changed. However, when it is migrated to different node on NUMA system, it suddenly has the memory much farer. So, obvious remedy is to migrate the memory as well. However, copying huge amount of memory takes some time and it also may not be the best solution for multi threaded programs. Thus, I implemented something that could be called lazy migration: the pages are simply marked as not present and when they are accessed, the are migrated if necessary. As I stated at the beginning of this paragraph, I am very itchy about this and I will definitely write about this more.

And that is all for the kernel side. Still is missing some better load balancing that respects the fact that it is more expensive to migrate thread to different node. And, of course, I need to do some benchmarking. That will be real fun because I do not have many ideas how to create fair benchmarks that would measure performance gains and losses when the difference between accessing local and remote memory is almost imperceptible.

For userspace, there is almost complete port of libnuma API (the one from Linux). The implementation was written from scratch, but the API shall be compatible with the Linux version. There are, of course, some changes due to fact that Linux expects process tree (i.e. the relation parent/child process is well defined and widely used) but the core works. If you want to see it, the already mentioned numa application has several tests and samples to show what is ready. What is still missing is port of the numactl(1) that is used to set policies and affinities of new tasks. The problem is again that it expects fork()/exec() style of launching new tasks. This will be probably solved by adding some optional methods to the loader.

Uff, that is all for now. I will try to write some kind of "progress report" on regular basis.