DIY Banana Pi 2-WAN gateway: Getting started with a BPI-R2

During our last Sandstorm Retreat there was not a single day without a remote participant. "But how fast and reliable is the internet connection really, in a location we never visited before?", we asked ourselves and decided to pack our Starlink receiver and, just like in the office, implement a network at the location with two independent WAN-connections. To manage  these WAN-connections, we packed our office gateway. Thus we had a redundant internet connection at the location and a non-redundant one at our main office… we know our priorities. :-)

As removing the gateway from the networking rack and re-configuring it was quite some effort, we decided to go a different way in the future:

We are building our own 2 WAN gateway on a much smaller, more light-weight and more mobile Banana Pi R2 (BPI-R2 for short). In this blog post I share my experience during the installation of the operating system up to the first successful connection via SSH.

Note that I work on OS X on my MacBook. All shell commands in this blog post are for Macs. They are slightly (Linux) or very (Windows) different on other operating systems.

After checking the Wiki page I started to try out some operating systems. Below I share my experience with

  1. Opening a TTY console (no HDMI, no SSH)
  2. Creating a bootable SD card
  3. How to back up a bootable SD card
  4. Powering on the BPI-R2 (not as simple as it sounds)
  5. Bananian Linux
  6. OpenWrt
  7. Alpine Router Platform
  8. Debian 10
  9. Ubuntu 16.04.
  10. Open work (more things to explore)
  11. Helpful links

Note: I did not go into full-detail for each and every OS. Maybe I'll come back to an open topic later. Then I'll update this blog post.

Starlink Antenna

Opening a TTY console (no HDMI, no SSH)

First things first: How to debug a device without an HDMI driver when it fails booting? You connect via a TTY console. This is a fully operational CLI terminal. To open it, you need to

  1. fetch your TTY-USB-adpater
  2. find the TTY pins on the BPI-R2 main board between the SATA ports and the WAN port
  3. connect the pins to your adapter as follows
    • GND - GND
    • TXD - RX
    • RXD - TX
  4. connect the USB-adapter to your MacBook and wait for mount
  5. run screen /dev/$(ls /dev | grep cu.usbserial) 115200

For more details and operating systems see BananaPi R2 - Debug-UART and Mac's and serial TTY's.

TTY USB Adapter
TTY pins on board

Creating a bootable SD card

How you write an image on the SD card, depends on the image. In the easiest case you can use dd but sometimes you have to adjust details of the command. Sometimes it does not work and it is rather slow. An alternative is an app called balenaEtcher. I started using the app instead of dd.

# insert SD Card # ignore that it is unreadable # !!! check the disk number !!! sudo su diskutil unmountDisk disk99 dd if=the-image.img of=/dev/rdisk99 diskutil eject disk99

How to back up a bootable SD card

To be fair: so far I did not need to recover an SD card from a backup. It should work in theory. 

sudo su # zipped and with progress (my card has 32GB) dd bs=4m if=/dev/rdisk2 | pv --size 32000000000 | gzip | dd bs=4m of=my-pi.img.zip 5.71GiB 0:04:23 [15.4MiB/s] [=====> ] 19% ETA 0:18:29
Banana Pi board

Powering on the BPI-R2 (not as simple as it sounds)

Ironically I had some trouble turning on the BPI-R2. Execute the following steps and it might work. I explain after the list what can go wrong in which step.

  1. insert the SD card
  2. move little switch over the SD card slot to "SD"
  3. insert power supply
  4. press the PWR button

Sounds simple enough, but you should know the following.

  • (step 2) the switch has no effect in some cases, it also depends on the (pre?) bootloader on the SD card
  • (step 3) make sure that the plug sits firmly and that the inner diameter is not too wide (yes, this took me some time)
  • (step 3) do not touch the board, you might shortcut something and leave the board in an error-state
  • (step 4) hold the PWR button for 5 to 10 seconds or the BPI shuts down
  • (step 4) it always shuts down when there are issues with the SD card
  • (step 4) … and you might see nothing on the TTY terminal

There is a dedicated forum thread for switching on the BPI-R2.

EMMC SD switch on board

Bananian Linux

On bananian.org/download I found the following statement dated April 2017.

Bananian Linux is no longer under active development.

The image actually did not start to boot. Maybe I made a mistake writing the image to the SD card with dd. Maybe it would have worked using Etcher.

OpenWrt

Since I want to configure a router, OpenWrt looks like a perfect match. There is a nice documentation and a multi-WAN extension. It has a downside though: no support for HDMI and WiFi. HDMI is not supported in most operating systems for the BPI-R2 and luckily we do not need it. WiFi on the other hand… We want to use the self-built router in a remote location. Usually we bring our own access points, but if one WAN-Uplink is only available via WiFi, it would be nice to use the WiFi as a WAN port.

I gave it a try. Two versions seemed to support the BPI-R2. I calculated the SHA-256 hashes myself after the download, so no guarantee on that.

  • 19.07.7 (SHA256 59097804f1543087466ce2540e656bf639899ddb84171a923a9e60bb954888a3)
  • 19.07.8 (SHA-256 8ae12c5593faffb076db2ccb6309f6ffc1ba973ff7edb424dd5025562d1f6979)

Version 19.07.7 started to boot but produced errors visible in the TTY console. Version 19.07.8 did not boot. Again: I did not (yet) try to write the 19.07.8 image to the SD card using Etcher instead of dd. Here are the two links which helped me the most:

Alpine Router Platform

The latest version with a ready-to-use SD card image for the BPI-R2 I found was from 2018-06-16. This one is 8GB in size. There are smaller images available. According to the documentation you can login via SSH as root:pantacor and have to find the IP address in the DHCP lease of the BPI on your DHCP server.

However this operating system seems tightly linked to the use of the Pantacor platform. This is fine but does not match our use-case. There is a forum thread you can read for more details.

Debian 10

I found an SD card image in a public Google Drive (thanks a million) as advertised on the BPI-R2 wiki page. The MD5 hash is in the drive, I calculated the SHA256 as 251b3ac7da22c20bc8aa4fd16d38c08e4984d009b8b328af08e25e625caf5595. According to the Wiki it supports Wifi and SATA (I did not test either one yet). The TTY console and SSH work out-of-the-box. You can login as root/bananapi at 192.168.0.11 on LAN port 0 (the one next to the WAN port). HDMI is not supported.

Most likely we stay with this operating system for our project.

Most likely we stay with Debian 10 for our project.

Ubuntu 16.04.

The BPI-R2 wiki page promotes Ubuntu 16.04. but there is no SD card image available for download. You have to create it yourself. This took me the most time by far (due to lack of experience with manually creating bootable devices). Hence this part is rather long and verbose – written from a beginners perspective.

Eventually I created a booting SD card. There have been kernel modules which did not load (got errors in the TTY console) but still… it did boot. I did not spend time fixing those errors since I just wanted to create the SD card. Fixing a broken operating system was out-of-scope for me. Nonetheless here come the steps I took:

  1. create a VM with Ubuntu 16.04.
  2. checkout and compile BPI-SINOVOIP / BPI-R2-bsp-4.14 from Github
  3. create an SD card image
  4. copy the image to the SD card
  5. remaining problems

Create a VM with Ubuntu 16.04.

On my local machine I use VirtualBox whenever I need a VM. You will find a lot of tutorials online explaining how to install Ubuntu in VirtualBox. I like to use a bridged network for the VM and connect via SSH. Just as if I would connect to a remote server.

Compile BPI-R2-bsp-4.14

Since the advertised download page no longer exists, you need to create your own SD card image. The first step is to build the project. Once you have a working terminal open, type in the following commands.

# on Ubuntu 16.04 VM # install required packages sudo apt update && apt install -y \ git \ make \ gcc \ bc \ gcc-arm-linux-gnueabihf \ u-boot-tools # download sources git clone https://github.com/BPI-SINOVOIP/BPI-R2-bsp-4.14 # enter directory cd BPI-R2-bsp-4.14 # run the build script and select option 1 ./build.sh 1

Create an SD card image

The documentation of BPI-R2-bsp-4.14 does not cover the creation of an SD card image. You cannot just dd one of the many build artifacts onto an SD card though. I found this very helpful forum post: How to build an Ubuntu/Debian SD image from scratch. I'll got through the commands here and add some more explanation.

# first we create a large empty file # does becomes our SD card image later dd if=/dev/zero bs=1M count=7296 | pv | dd of=BPI-R2-Ubuntu-4.14.img

Next we want to install an operating system and a boot sector into this empty file. First mount the empty file as "a disk". This makes working with the file easier. Especially since we need to create partitions.

# now we create a loop device # in other words: we mount our empty file as disk sudo su losetup --all losetup /dev/loop8 BPI-R2-Ubuntu-4.14.img losetup --all # prints /dev/loop8: [2049]:298277 (/home/christoph/src/BPI-R2-Ubuntu-4.14.img)

Now we can treat /dev/loop8 as a disk and create disk partitions with the required file systems.

# label disk # (-s prevents prompts) parted -s /dev/loop8 mklabel msdos # create a fat32 partition # starting at 100MiB # ending at 356MiB parted -s /dev/loop8 unit MiB mkpart primary fat32 -- 100MiB 356MiB # create an ext2 partition # starting at 356MiB # ending at end of disk # (which is 8GB in size) parted -s /dev/loop8 unit MiB mkpart primary ext2 -- 356MiB 7295MiB # let's look at the partitions partprobe -s /dev/loop8 # -> /dev/loop8: msdos partitions 1 2 # reading: the disk /dev/loop8 # with label msdos # has the partitions 1 and 2 # now let's format the partition 1 mkfs.vfat /dev/loop8p1 -I -n BPI-BOOT # … and the partition 2 mkfs.ext4 \ -O ^has_journal -E stride=2,stripe-width=1024 \ -b 4096 /dev/loop8p2 \ -L BPI-ROOT # flush all caches to disks sync # again let's inspect the disk parted -s /dev/loop8 print # -> Model: Loopback device (loopback) # -> Disk /dev/loop8: 7650MB # -> Sector size (logical/physical): 512B/512B # -> Partition Table: msdos # -> Disk Flags: # -> # -> Number Start End Size Type File system Flags # -> 1 105MB 373MB 268MB primary fat16 lba # -> 2 373MB 7649MB 7276MB primary ext2

Now that we formatted the partitions, we want to write to them. So let's mount the partitions.

mkdir /mnt/rootfs mount /dev/loop8p2 /mnt/rootfs mkdir /mnt/rootfs/boot mount /dev/loop8p1 /mnt/rootfs/boot

Now everything is set up and we can install a minimal Debian system on the new partition.

debootstrap --arch=armhf --foreign xenial /mnt/rootfs cp /usr/bin/qemu-arm-static /mnt/rootfs/usr/bin/

The next step excited me the most. We switch the operating system from our current VM to the SD card. This is what I usually do to recover a damaged operating system after booting from a CD or a USB stick.

chroot /mnt/rootfs

To be clear: 

  • We are on your working machine inside a VM
  • with a file mounted as a disk with a partition on it
  • which we now use as our new root directory. 

Now we can complete the Debian setup and configure our system.

# finish installation export LANG=C /debootstrap/debootstrap --second-stage # add package repositories cat >> /etc/apt/sources.list <<EOF deb http://ports.ubuntu.com/ubuntu-ports/ xenial main restricted deb-src http://ports.ubuntu.com/ubuntu-ports/ xenial main restricted deb http://ports.ubuntu.com/ubuntu-ports/ xenial universe deb-src http://ports.ubuntu.com/ubuntu-ports/ xenial universe deb http://ports.ubuntu.com/ubuntu-ports/ xenial multiverse deb-src http://ports.ubuntu.com/ubuntu-ports/ xenial multiverse deb http://ports.ubuntu.com/ubuntu-ports/ xenial-updates main restricted deb-src http://ports.ubuntu.com/ubuntu-ports/ xenial-updates main restricted deb http://ports.ubuntu.com/ubuntu-ports/ xenial-updates universe deb-src http://ports.ubuntu.com/ubuntu-ports/ xenial-updates universe deb http://ports.ubuntu.com/ubuntu-ports/ xenial-security main restricted deb-src http://ports.ubuntu.com/ubuntu-ports/ xenial-security main restricted deb http://ports.ubuntu.com/ubuntu-ports/ xenial-security multiverse deb-src http://ports.ubuntu.com/ubuntu-ports/ xenial-security multiverse EOF # install updates and missing packages apt update apt upgrade -y apt dist-upgrade -y apt install -y sudo openssh-server locales ntpdate htop pv # set hostname to bpi-r2 echo "bpi-r2" > /etc/hostname vi /etc/host # (add 127.0.0.1 bpi-r2) # set root password passwd # add user bpi useradd -m -G users,sudo,ssh -s /bin/bash bpi passwd bpi # set locales locale-gen de_DE locale-gen de_DE.UTF-8 dpkg-reconfigure locales dpkg-reconfigure tzdata update-locale LANG=de_DE.UTF-8 # auto-mount our two partitions cat >> /etc/fstab <<EOF proc /proc proc defaults 0 0 LABEL=BPI-BOOT /boot vfat errors=remount-ro 0 1 LABEL=BPI-ROOT / ext4 defaults,noatime 0 0 EOF # configure the network cat >> /etc/network/interfaces <<EOF auto lo iface lo inet loopback auto eth0 iface eth0 inet manual auto wan iface wan inet dhcp auto lan0 iface lan0 inet static address 192.168.1.1 netmask 255.255.255.0 #gateway 192.168.0.1 auto lan1 iface lan1 inet static address 192.168.1.2 netmask 255.255.255.0 #gateway 192.168.0.1 auto lan3 iface lan3 inet static address 192.168.1.3 netmask 255.255.255.0 #gateway 192.168.0.1 auto lan4 iface lan4 inet static address 192.168.1.4 netmask 255.255.255.0 #gateway 192.168.0.1 EOF # add SSH key for root login # (should be changed later) mkdir /root/.ssh chmod 700 /root/.ssh/ vi /root/.ssh/authorized_keys

We leave our soon-to-be SD card operating system and switch back to the VM Ubuntu 16.04 installation.

exit

With the base operating system in place, we still have to do something with our self-compiled artifacts. Let's copy some files to the still mounted partitions before ejecting them.

# remove qemu-arm-static again rm /mnt/rootfs/usr/bin/qemu-arm-static # extract first build artifacts tar -xvf BPI-R2-bsp-4.14/SD/BPI-BOOT-bpi-r2.tgz --keep-directory-symlink \ -C /mnt/rootfs/boot tar -xvf BPI-R2-bsp-4.14/SD/4.14.34-BPI-R2-Kernel.tgz --keep-directory-symlink \ -C /mnt/rootfs tar -xvf BPI-R2-bsp-4.14/SD/4.14.34-BPI-R2-Kernel-net.tgz --keep-directory-symlink \ -C /mnt/rootfs tar -xvf BPI-R2-bsp-4.14/SD/BOOTLOADER-bpi-r2.tgz --keep-directory-symlink \ -C /mnt/rootfs # add some more configuration echo "blacklist mtk_pmic_keys" > /mnt/rootfs/etc/modules-load.d/mtk_pmic_keys.conf # eject the partitions umount /mnt/rootfs/boot umount /mnt/rootfs

No, not done yet.

We cannot directly boot our final operating system. Some steps must happen before, to enable more features, e.g. access to more board components or access to a fat16 file-system… features we need to even start booting Ubuntu.

Remember that we left the first part of the card image empty? The first partition starts at 100MiB. Now we write into this empty space. We use dd to transfer bytes directly into our image file, which is still mounted as a disk.

# download more artifacts wget https://github.com/BPI-SINOVOIP/BPI-files/raw/master/SD/100MB/BPI-R2-HEAD440-0k.img.gz wget https://github.com/BPI-SINOVOIP/BPI-files/raw/master/SD/100MB/BPI-R2-HEAD1-512b.img.gz # and write those to the image gunzip -c BPI-R2-HEAD440-0k.img.gz | dd of=/dev/loop8 bs=1024 seek=0 gunzip -c BPI-R2-HEAD1-512b.img.gz | dd of=/dev/loop8 bs=512 seek=1 # write self-compiled artifacts to the image as well dd if=BPI-R2-bsp-4.14/mt-pack/mtk/bpi-r2/bin/preloader_iotg7623Np1_sd_1600M.bin \ of=/dev/loop8 bs=1024 seek=2 dd if=BPI-R2-bsp-4.14/u-boot-mt/u-boot.bin \ of=/dev/loop8 bs=1024 seek=320

Note that the exact position within the image file is of uttermost importance. Sometimes the offset is part of the filename. For more information I recommend this Wiki page and this Forum thread.

Finally – the image is ready. Time to fetch it from the VM and write it to an actual SD card.

# flush caches to disk sync # eject our image file losetup -d /dev/loop8

Since I have been too lazy to make the physical SD card accessible in my Ubuntu VM, I have to copy the image file to my MacBook.

# first shrink the image zip BPI-R2-Ubuntu-4.14.img.zip BPI-R2-Ubuntu-4.14.img # -> adding: BPI-R2-Ubuntu-4.14.img (deflated 95%) # the resulting file size is much smaller # since our image file contains mostly zeros # on a mostly empty partition 2 ls -lisah BPI-R2-Ubuntu-4.14.img.zip # -> … 330M Oct 16 15:49 BPI-R2-Ubuntu-4.14.img.zip # now leave the VM exit # and download the image file scp 192.168.12.34:BPI-R2-Ubuntu-4.14.img.zip .

You can write the zipped image file to the SD card using Etcher now.

Remaining Problems

As mentioned, the resulting Ubuntu SD card boots with some errors. TTY works but network is down. I did not find time yet to analyze or fix the error. I can access the system via the TTY console and get the following error message:

[FAILED] Failed to start Load Kernel Modules. See 'systemctl status systemd-modules-load.service' for details. … [FAILED] Failed to start Raise network interfaces. See 'systemctl status networking.service' for details. … root@bpi-r2:~# systemctl status systemd-modules-load.service ● systemd-modules-load.service - Load Kernel Modules Loaded: loaded (/lib/systemd/system/systemd-modules-load.service; static; ven Active: failed (Result: exit-code) since Fr 2021-04-02 04:51:59 CEST; 7min ag Docs: man:systemd-modules-load.service(8) man:modules-load.d(5) Process: 126 ExecStart=/lib/systemd/systemd-modules-load (code=exited, status= Main PID: 126 (code=exited, status=1/FAILURE) Apr 02 04:51:59 bpi-r2 systemd[1]: systemd-modules-load.service: Main process ex Apr 02 04:51:59 bpi-r2 systemd[1]: Failed to start Load Kernel Modules. Apr 02 04:51:59 bpi-r2 systemd[1]: systemd-modules-load.service: Unit entered fa Apr 02 04:51:59 bpi-r2 systemd[1]: systemd-modules-load.service: Failed with res Warning: Journal has been rotated since unit was started. Log output is incomple root@bpi-r2:~# systemctl status networking.service ● networking.service - Raise network interfaces Loaded: loaded (/lib/systemd/system/networking.service; enabled; vendor prese Drop-In: /run/systemd/generator/networking.service.d └─50-insserv.conf-$network.conf Active: failed (Result: timeout) since Fr 2021-04-02 04:57:02 CEST; 1min 41s Docs: man:interfaces(5) Process: 380 ExecStart=/sbin/ifup -a --read-environment (code=killed, signal=T Process: 295 ExecStartPre=/bin/sh -c [ "$CONFIGURE_INTERFACES" != "no" ] && [ Main PID: 380 (code=killed, signal=TERM) Apr 02 04:52:01 bpi-r2 systemd[1]: Starting Raise network interfaces... Apr 02 04:52:02 bpi-r2 ifup[380]: /sbin/ifup: waiting for lock on /run/network/i Apr 02 04:57:02 bpi-r2 systemd[1]: networking.service: Start operation timed out Apr 02 04:57:02 bpi-r2 systemd[1]: Failed to start Raise network interfaces. Apr 02 04:57:02 bpi-r2 systemd[1]: networking.service: Unit entered failed state Apr 02 04:57:02 bpi-r2 systemd[1]: networking.service: Failed with result `timeo

Open work (more things to explore)

First: configure the installed Debian 10 to use two ports as WAN ports.

Further things I would like to look into:

  • it would be nice to fix the Ubuntu image
  • some SD card did not boot after writing the image with dd
  • test these images again with Etcher

Helpful links

I added all links which I considered helpful into the post above. Of course, there is much, much more to explore. Here are some additional related links I found while searching the web. A huge thanks to everyone who shares his/her experience!!!

Thanks for reading

Thank you a lot for reading this huge post. As always, feedback is very welcome. Please get in touch.

BPI-R2 in case