December 12th, 2021 | Categories: illumos, Networking, Technology

I’m running a HA ldap setup however I keep hitting illumos issue #13326, hoping to mitigate this a tiny bit by using a VIP instead of multiple A records behind a host name. (Sadly that doesn’t help as the connection state itself ends up in the bad state, but it did send me down the VRRP on illumos rabbit hole.)

Solaris 10 and therefore also the illumos decedents support VRRP for both IPv4 and IPv6, documentation on this is a bit sparse so hopefully this note dump is useful to someone, most likely myself in a few years. On OmniOS it’s pretty easy to get the required utilities.

root@jupiter:~# pkg install system/network/routing/vrrp

If you plan on using the VIP only in a (l)ipkg zone, you don’t even need to install this in the global zone! So just install it where you will be using the VIP. In my case this is the ldap1 and ldap2, these zones cannot be on the same host!

Sadly there is no support in the brand scripts yet, so we will need to create the special VNIC in the global zone manually:

dladm create-vnic -m vrrp -l ixgbe0 -V 98 -A inet -v 110 ldap1vip0
dladm create-vnic -m vrrp -l ixgbe0 -V 98 -A inet6 -v 110 ldap1vip1

Notice that I create two special VNICs, one for IPv4 and one for IPv6. There are some things to keep in mind there, the VNICs needs to be created on top of the same link and with the same vlan-id as the regular vnic!

root@jupiter:~# dladm show-vnic
ldap10       ixgbe0     10000 0:22:6:fd:d4:db   fixed          110  ldap1
ldap1vip0    ixgbe0     10000 0:0:5e:0:1:62     vrrp, 98/inet  110 --
ldap1vip1    ixgbe0     10000 0:0:5e:0:2:62     vrrp, 98/inet6 110 --

In my case the ldap10 VNIC is created automatically by the brand scripts, so my final net config for the ldap1 zone looks like this:

root@jupiter:~# zadm show ldap1 | jq .net
    "global-nic": "ixgbe0",
    "mac-addr": "00:22:06:fd:d4:db",
    "physical": "ldap10",
    "vlan-id": "110"
    "physical": "ldap1vip0"
    "physical": "ldap1vip1"

Inside the ldap1 zone we now need to configure the VIP and also create the vrrp router.

root@ldap1:~# ipadm create-addr -T static -d -a ldap1vip0/v4
root@ldap1:~# vrrpadm create-router -V 98 -l ldap20 -A inet ldap1vip0

It is important to use the same VRID (98) and address type we use in the global zone.

root@ldap1:~# vrrpadm show-router
ldap1vip0 98 ldap20  IPv4 255  1000     eopa- MASTER ldap1vip0
ldap1vip1 98 ldap20  IPv6 255  1000     eopa- INIT  ldap1vip1
root@ldap1:~# vrrpadm show-router -x
ldap1vip0 MASTER INIT    15.246s ldap1vip0
ldap1vip1 INIT NONE       9.371s ldap1vip1 --                --

Of course you need to repeat these steps on the 2nd host + zone.

Comments Off on Using VRRP inside a lipkg zone on OmniOS
August 18th, 2018 | Categories: FreeBSD, Hardware, Technology, Uncategorized

If you are reading this because you, like me were lost and frustrated… I feel your pain. But let me try and help!

A few weeks ago I added a POE HAT to a Raspberry Pi 3B+ and installed FreeBSD to so I could build a remote access server. I also bough another Raspberry Pi 3B+ to replace by OrangePi Zero + USB uBlox 5 NTP server.

Unlike Raspbian it is surprisingly hard to stop FreeBSD’s kernel from grabbing uart0 for it’s console. Lets start with why you are probably here… how do I stop the kernel from doing this?

echo 'hw.fdt.console="NO"' >> /boot/loader.conf.local

Easy right? It took me more than 24 hours to find this. Most of which was one long Saturday. I could not find much (any) documentation on this parameter. So how did I find it?

One of the embedded device devs from FreeBSD recommended to remove /chosen/stdout-path from the fdt tree. This did not work. After a few hours of messing around with building kernels without baked in uart support and other dead-ends I went back to looking at the fdt stuff.


These dtsi and dts files are what eventually end up in the fdt… bcm2837-rpi-3-b-plus.dts holds the /chosen/stdout-path for the rPI 3B+. 

        chosen {
                /* 8250 auxiliary UART instead of pl011 */
                stdout-path = "serial1:115200n8";

Older version had this set to serial0 but I switched to FreeBSD 12-APHA2 before my long day of experimentation. This one lists serial1 as the default path… which is odd given bluetooth is still disabled so that the hardware UART is attached to the TX and RX GPIO pins. (That is serial0)

So with the default or with a empty (or missing stdout-path) we still somehow end up using serial0. After about an hour of grokking the source I came to these files.


Both files deal with setting up the uart to have the kernel’s console attached. The function uart_cpu_fdt_probe is the interesting one here.

        /* Has the user forced a specific device node? */
        cp = kern_getenv("hw.fdt.console");
        if (cp == NULL) {
                 * Retrieve /chosen/std{in,out}.
                node = -1;
                if ((chosen = OF_finddevice("/chosen")) != -1) {
                        for (name = propnames; *name != NULL; name++) {
                                if (phandle_chosen_propdev(chosen, *name,
                                    &node) == 0)
                if (chosen == -1 || *name == NULL)
                        node = OF_finddevice("serial0"); /* Last ditch */
        } else {
                node = OF_finddevice(cp);

        if (node == -1)
                return (ENXIO);

Above I have highlighted the most interesting bit.

  1. hw.fdt.console checked
  2. if hw.fdt.console is not set, /chosen/std{in,out,out-path} are checked
  3. if the config in /chosen/std* was not valid a last ditch effort is done to configure serial0

If hw.fdt.console is set to an invalid path e.g. NO no console will be configured! So this is the solution to disabling the kernel from grabbing uart0! We did it reddit

Now that I knew what to do I managed to find this change D3559. A handy note was left from the developer:

The user could disable the fdt console with this by setting it to an invalid node, e.g. hw.fdt.console="none"

I could not find any other documentation about this parameter so it may stop working in the future. Time will tell.

EDIT: It looks like the internal UART can only run at 115200… I am looking into this as 9600 is what I need for the GPS HAT. I did have the RTC and PPS working without anything fancy aside from adding the dtoverlays and loading the drivers. So that is a small win.

I have also attached a u-boot binary with silent=1 and nulldev configured for people who do not want to recompile u-boot themselves.

UPDATE: the UART can indeed only run at 115200 and FreeBSD’s PL011 driver does not support changing the speed unless running on x86! However u-boot can! Adding the following line to config.txt pins the UART’s baudrate at 9600

Comments Off on FreeBSD, UART and Raspberry Pi 3 B+
August 4th, 2018 | Categories: FreeBSD, Hardware, Personal, Technology


I have a small homelab and I was looking into some sort of KVM console to be able to access them both locally and remotely instead of moving around VGA cables/serial cables.

Those are rather expensive so I opted to build my own remote access system. Since all my servers and switch(es) have a serial console that should not be to hard.

I ended up using:

For the software I ended up using conserver. Below is a very brief tutorial on how to set everything up. I assume you have basic unix skills.


Preparing the microSD

At the time of writing there were no Raspberry Pi 3+ images…

  1. download and extract the Raspberry Pi 3 image from
  2. copy the image to a microSD card using either dd or pv.
  3. grab bcm2710-rpi-3-b-plus.dtb, bootcode.bin, fixup.dat, fixup_cd.dat, fixup_cd.dat fixup_cd.dat, start.elf, start_cd.elf, start_db.elf, start_x.elf, and u-boot.bin from
  4. mount the msdos partition from the microSD card and copy over the new dtb, bin, and elf files.

Preparing the hardware

If you have a head-sink kit only install the larger one that goes on the SoC, the smaller one for the chip near the USB ports will prevent the micro POE HAT from being seated properly.

  1. install the micro POE HAT
  2. (Optionally) install a heat-sink kit, I ended up adding the smaller one to the chip on the micro POE HAT as it get rather hot.
  3. place the Raspberry Pi + HAT inside the case and assemble it

Configuring FreeBSD

Because I did not want to mess with HDMI and a keyboard I used a USB-to-TTL adaptor to configure the rPI. Both a monitor and keyboard should work equally well.

  1. set a root password
  2. create one or more accounts (and add ssh public keys to them)
  3. configure your network and firewall
  4. install conserver-com there is another one with -com and that is the wrong one for this tutorial.
  5. install additional software e.g. ipmitool, minicom, …
  6. update /boot/loader.conf to load the required drivers
## Raspberry Pi 3+ USB Ethernet
## Load USB Serial support
## Load FTDI driver
  1. enable both sshd and conserver
sysrc sshd_enable="YES"
sysrc conserver_enable="YES"
  1. create /usr/local/etc/conserver.passwd so that our users we created earlier can use conserver
  1. create /usr/local/etc/ (use the correct cuaU and ipmitool settings)
### access
config * {
	defaultaccess allowed;
access localhost {
	admin usera;
	trusted localhost;

### consoles
## defaults
default * {
	# master server
	master localhost;
	# timestamps
        # NOTE:  every hour with activity and break logging
	timestamp 1hab;
	# enable writing
	rw *;
        # enable logging
        logfile /var/log/console_&.log;
## switch
console switch {
	motd "Ubiquiti ES-48-500W (serial)";
	type device;
	device /dev/cuaU0; parity none; baud 115200;
## computenode 1
console cn1 {
	motd "SmartOS Node 1 (serial)";
	type device;
	device /dev/cuaU1; parity none; baud 115200;
console cn1_ipmi {
	motd "Compute Node 1 (IPMI)";
	logfile "";
	type exec;
	exec /usr/local/bin/ipmitool -I lanplus -C 3 -L OPERATOR -H -U consoleuser -P consolepassword shell;
## computenode 2
console cn2 {
	motd "SmartOS Node 2 (serial)";
	type device;
	device /dev/cuaU2; parity none; baud 115200;
console cn2_ipmi {
	motd "Compute Node 2 (IPMI)";
	logfile "";
	type exec;
	exec /usr/local/bin/ipmitool -I lanplus -C 3 -L OPERATOR -H -U consoleuser -P consolepassword shell;
  1. drop in a newsyslog configuration file to rotate the logs console session logs
# logfilename           [owner:group]    mode count size when   flags [/pid_file] [sig_num]
/var/log/console_*.log                      644  4     *    $W0D23 GZ    /var/run/
  1. restart conserver and test if consoles work
usera@hydra:~ # console -u
usera@hydra:~ # console c
May 26th, 2016 | Categories: illumos, Personal, Technology

I was in the market for a configuration management and remote execution tool. After looking around I didn’t really find any I liked and also supported SmartOS (and illumos).

It quickly became clear that I would need to do some legwork. I’m by no means a great programmer and my language selection is mostly limited to Turbo Pascal, C#, and Java supplemented with shell script (~bash really), perl (long ago), and Python.

My eye fell on SaltStack as it uses mostly Python, the community looked welcoming. I looked at some PR’s and the feedback happening on them was constructive. As a plus, Nahum Shalman already did some legwork to get SaltStack into the SmartOS global zone!

I spend a lot of time getting SmartOS support for managing zones/VMs usable.

  • imgadm execution module got cleaned up and reworked
  • vmadm execution module was written as a replacement for the existing ‘virt’ compatible module
  • smartos state module was written that uses the imgadm and vmadm execution modules, this allows for management of zones/vms, images, and configuration.

I’ve been running these at home for about 3 months. They are pretty solid, I have a few things I want to improve in them but they are certainly usable. If you use them and find any issues, please open an issue on the saltstack github and tag me using @sjorge. I will try and fix them as soon as I can.

After this I started to manage my zones themselves hitting a few roadblocks along the way. Most of these changes also apply to illumos in general. (Sometimes to *BSD’s too)

  • zpool execution module got reworked and now supports most actions the underlying cli command does.
  • zfs execution module also got reworked
  • zfs state module was writting that uses the rewritten zpool and zfs execution modules.
    • zfs.scheduled_snapshot written to replace my usage of zsnapper in the global zone. It’s not perfect but gets the job done. (Use the schedule state to execute this state often.)
  • mdata execution module to retrieve and set SmartOS metadata.
  • fmadm execution module written for Solarish platforms.
  • system execution module written for Solarish platforms.
  • cron execution module fixed for Solarish platforms when setting jobs for a different user.
  • shadow execution module improved with gen_password and del_password for Solarish platforms.
  • timezone execution module fixed for Solarish platforms
  • locale execution module fixed for Solarish platforms
  • pkgin execution module now loads as default on SmartOS
  • grains now show more info on SmartOS, including mdata
  • grains now show correct info when running on SmartOS under LX branded emulation.

Some other small fixes here and there.

Overall SaltStack 2016.3 is working great for me on SmartOS and OmniOS. If you use these platforms, give it a spin!

How do I get this awesomeness? A pip install should work fine inside a zone. For the global zone I have a esky package hosted here.

Comments Off on SmartOS, would you like some Salt with that?
July 25th, 2015 | Categories: illumos, Personal, Programming, Technology

I spent the last 3 days yak-shaving. I had the intent of looking into deploying SaltStack to monitor my zones at home. Instead I give you asmd AKA “Advanced SmartOS Management Daemon”. Yes I should really get better at naming things.

So what is this asmd you speak of, you may ask? It is my one pythonbash script to replace them all. SmartOS wise anyway. As you know I run a few SmartOS nodes at home, you may also know I really try to push IPv6 everywhere. SmartOS does not support this. After a few months I had collected a few hacks to make my life easier, each had their own smf service… You could see my learning progression in them as the quality improved as I got more familiar with the SmartOS internals.

I wrote a simple “framework” to plug in my little hacks and scripts, this framework is asmd. Once I had the base line I rewrote most of my hacks to leverage this framework. I originally wrote this in python that I pushed to the SmartOS nodes as an esky package. This was powerful but also complete, on day two I rewrote everything in bash.

So how does it work? Each ‘service’ lives in $PREFIX/services/name.service and sets a few variables to describe it self and implements 3 functions:

  • asmd_service_config :: execute on asmd-setup, this can be used to prepare some files and directories.
  • asmd_service_start :: runs when the service starts
  • asmd_service_stop :: runs when the service stops

The following variables are required

  • ASMD_SERVICE_NAME :: name of the service as it appears under smf
  • ASMD_SERVICE_DESC :: description use for the smf service
  • ASMD_SERVICE_TYPE :: for now only ‘transient’ is tested, although ‘daemon’ should also be supported.
  • ASMD_SERVICE_DEPENDENCIES :: svc identifiers for service dependencies
  • ASMD_SERVICE_DEPENDENTS :: svc identifiers for services dependent on this service

Functions marked in italic can be omitted, variables marked in italic can be empty.

When asmd-setup is ran, usually once after install or when adding/removing services a smf manifest is generated and placed in /opt/custom/smf so that SmartOS loads it at boot. Each service is a named instance under system/asmd. I decided to use /usbkey/config to store all the configuration individual services need.

Currently the following services are implented:

  • hostname :: set hostname and/or domain name at boot
  • profile :: symlinks files and folders into /root to customize the user environment (aka inject bash aliasses like vmadm wrapper that can use aliasses of zones)
  • exec :: executes each script located in /usbkey/ after network is up. Mostly used for quick and dirty hacks
  • swap :: manage additional swap devices / can also remove the default zones/swap device
  • mail :: configure forwarding for root mails / (smart)relay host
  • cron :: inject cron jobs defined in /usbkey/config
  • ipv6 :: configure ipv6 address/gateway on admin_nic.

You can check it out on github:

Feel free to play around, open issue on github if you find things that are broken.

Comments Off on Advanced SmartOS Management Daemon
May 23rd, 2015 | Categories: illumos, Personal, Technology

My new package repo is now live!

I’m only targeting the current base64-lts release.
Currently only SmartOS is supported, although it should be compatible with OmniOS using the pkgsrc bootstrap available at some recent changes (epoll) seem to have broke them as SmartOS supports it but OmniOS and Nexenta Community both seem to be lacking it.

As of now is deprecated and no further updates will happen. Somewhere later this year it will go offline, I have no date yet but it will probably happen the next time I need to do a big round of updates.

Comments Off on live, depricated
May 20th, 2015 | Categories: illumos, Technology

As previously mentioned I will be deprecating IPS package repository soonish.

Work has started on, the initial set of packages is up. But most are untested and they need some more love. Once the packages are where I want them to be. I will start rolling packages for omnios also.

Check it out if you want to poke it. Although stuff will change and change often!

Comments Off on enters alpha stages
April 18th, 2015 | Categories: illumos, Personal, Technology

Just a heads up, somewhere later this year I will turn down and will bring up There I will host a smaller selection of packages for use in SmartOS zones or on OmniOS once the illumos flavor of pkgsrc is installed.

Not sure when this will happen because I am currently changing jobs. More info will follow.

Comments Off on OmniOS package repo eventual deprecation
March 25th, 2015 | Categories: illumos, Technology

I wrote a wiki page on how to run a ntp server on SmartOS. Check it out!

Comments Off on Running an NTP server under SmartOS
March 22nd, 2015 | Categories: illumos, Personal, Technology

As it comes as no surprise I migrated my hosting from ESXi Free to SmartOS a year or so ago.
However every major update I spend an hour or so building a new zone from scratch to then switch over.

This is tiresome so now I’m working on a bootstrap script that will rebuild the zone for me from the data on a delegated dataset.
So far 2/4 zones are done and this is AMAZING, a simple vmadm reprovision and all is set.

Anyway, back to the other 2 zones that need some love! I hope to start update more again soon.
I will be switching jobs again and the new one will give me some more free time to write!

Comments Off on SmartOS and updates