During our last Sandstorm Retreat there was not a single day without a remote participant. "But how fast and reliable is the internet connection really, in a location we never visited before?", we asked ourselves and decided to pack our Starlink receiver and, just like in the office, implement a network at the location with two independent WAN-connections. To manage these WAN-connections, we packed our office gateway. Thus we had a redundant internet connection at the location and a non-redundant one at our main office… we know our priorities. :-)
As removing the gateway from the networking rack and re-configuring it was quite some effort, we decided to go a different way in the future:
We are building our own 2 WAN gateway on a much smaller, more light-weight and more mobile Banana Pi R2 (BPI-R2 for short). In this blog post I share my experience during the installation of the operating system up to the first successful connection via SSH.
Note that I work on OS X on my MacBook. All shell commands in this blog post are for Macs. They are slightly (Linux) or very (Windows) different on other operating systems.
After checking the Wiki page I started to try out some operating systems. Below I share my experience with
Note: I did not go into full-detail for each and every OS. Maybe I'll come back to an open topic later. Then I'll update this blog post.
Opening a TTY console (no HDMI, no SSH)
First things first: How to debug a device without an HDMI driver when it fails booting? You connect via a TTY console. This is a fully operational CLI terminal. To open it, you need to
fetch your TTY-USB-adpater
find the TTY pins on the BPI-R2 main board between the SATA ports and the WAN port
connect the pins to your adapter as follows
GND - GND
TXD - RX
RXD - TX
connect the USB-adapter to your MacBook and wait for mount
run screen /dev/$(ls /dev | grep cu.usbserial) 115200
How you write an image on the SD card, depends on the image. In the easiest case you can use dd but sometimes you have to adjust details of the command. Sometimes it does not work and it is rather slow. An alternative is an app called balenaEtcher. I started using the app instead of dd.
# insert SD Card
# ignore that it is unreadable
# !!! check the disk number !!!
sudo su
diskutil unmountDisk disk99
dd if=the-image.img of=/dev/rdisk99
diskutil eject disk99
How to back up a bootable SD card
To be fair: so far I did not need to recover an SD card from a backup. It should work in theory.
sudo su
# zipped and with progress (my card has 32GB)
dd bs=4m if=/dev/rdisk2 | pv --size 32000000000 | gzip | dd bs=4m of=my-pi.img.zip
5.71GiB 0:04:23 [15.4MiB/s] [=====> ] 19% ETA 0:18:29
Powering on the BPI-R2 (not as simple as it sounds)
Ironically I had some trouble turning on the BPI-R2. Execute the following steps and it might work. I explain after the list what can go wrong in which step.
insert the SD card
move little switch over the SD card slot to "SD"
insert power supply
press the PWR button
Sounds simple enough, but you should know the following.
(step 2) the switch has no effect in some cases, it also depends on the (pre?) bootloader on the SD card
(step 3) make sure that the plug sits firmly and that the inner diameter is not too wide (yes, this took me some time)
(step 3) do not touch the board, you might shortcut something and leave the board in an error-state
(step 4) hold the PWR button for 5 to 10 seconds or the BPI shuts down
(step 4) it always shuts down when there are issues with the SD card
(step 4) … and you might see nothing on the TTY terminal
Bananian Linux is no longer under active development.
The image actually did not start to boot. Maybe I made a mistake writing the image to the SD card with dd. Maybe it would have worked using Etcher.
OpenWrt
Since I want to configure a router, OpenWrt looks like a perfect match. There is a nice documentation and a multi-WAN extension. It has a downside though: no support for HDMI and WiFi. HDMI is not supported in most operating systems for the BPI-R2 and luckily we do not need it. WiFi on the other hand… We want to use the self-built router in a remote location. Usually we bring our own access points, but if one WAN-Uplink is only available via WiFi, it would be nice to use the WiFi as a WAN port.
I gave it a try. Two versions seemed to support the BPI-R2. I calculated the SHA-256 hashes myself after the download, so no guarantee on that.
Version 19.07.7 started to boot but produced errors visible in the TTY console. Version 19.07.8 did not boot. Again: I did not (yet) try to write the 19.07.8 image to the SD card using Etcher instead of dd. Here are the two links which helped me the most:
The latest version with a ready-to-use SD card image for the BPI-R2 I found was from 2018-06-16. This one is 8GB in size. There are smaller images available. According to the documentation you can login via SSH as root:pantacor and have to find the IP address in the DHCP lease of the BPI on your DHCP server.
However this operating system seems tightly linked to the use of the Pantacor platform. This is fine but does not match our use-case. There is a forum thread you can read for more details.
Debian 10
I found an SD card image in a public Google Drive (thanks a million) as advertised on the BPI-R2 wiki page. The MD5 hash is in the drive, I calculated the SHA256 as 251b3ac7da22c20bc8aa4fd16d38c08e4984d009b8b328af08e25e625caf5595. According to the Wiki it supports Wifi and SATA (I did not test either one yet). The TTY console and SSH work out-of-the-box. You can login as root/bananapi at 192.168.0.11 on LAN port 0 (the one next to the WAN port). HDMI is not supported.
Most likely we stay with this operating system for our project.
Most likely we stay with Debian 10 for our project.
Ubuntu 16.04.
The BPI-R2 wiki page promotes Ubuntu 16.04. but there is no SD card image available for download. You have to create it yourself. This took me the most time by far (due to lack of experience with manually creating bootable devices). Hence this part is rather long and verbose – written from a beginners perspective.
Eventually I created a booting SD card. There have been kernel modules which did not load (got errors in the TTY console) but still… it did boot. I did not spend time fixing those errors since I just wanted to create the SD card. Fixing a broken operating system was out-of-scope for me. Nonetheless here come the steps I took:
On my local machine I use VirtualBox whenever I need a VM. You will find a lot of tutorials online explaining how to install Ubuntu in VirtualBox. I like to use a bridged network for the VM and connect via SSH. Just as if I would connect to a remote server.
Compile BPI-R2-bsp-4.14
Since the advertised download page no longer exists, you need to create your own SD card image. The first step is to build the project. Once you have a working terminal open, type in the following commands.
# on Ubuntu 16.04 VM
# install required packages
sudo apt update && apt install -y \
git \
make \
gcc \
bc \
gcc-arm-linux-gnueabihf \
u-boot-tools
# download sources
git clone https://github.com/BPI-SINOVOIP/BPI-R2-bsp-4.14
# enter directory
cd BPI-R2-bsp-4.14
# run the build script and select option 1
./build.sh
1
Create an SD card image
The documentation of BPI-R2-bsp-4.14 does not cover the creation of an SD card image. You cannot just dd one of the many build artifacts onto an SD card though. I found this very helpful forum post: How to build an Ubuntu/Debian SD image from scratch. I'll got through the commands here and add some more explanation.
# first we create a large empty file
# does becomes our SD card image later
dd if=/dev/zero bs=1M count=7296 | pv | dd of=BPI-R2-Ubuntu-4.14.img
Next we want to install an operating system and a boot sector into this empty file. First mount the empty file as "a disk". This makes working with the file easier. Especially since we need to create partitions.
# now we create a loop device
# in other words: we mount our empty file as disk
sudo su
losetup --all
losetup /dev/loop8 BPI-R2-Ubuntu-4.14.img
losetup --all
# prints /dev/loop8: [2049]:298277 (/home/christoph/src/BPI-R2-Ubuntu-4.14.img)
Now we can treat /dev/loop8 as a disk and create disk partitions with the required file systems.
# label disk
# (-s prevents prompts)
parted -s /dev/loop8 mklabel msdos
# create a fat32 partition
# starting at 100MiB
# ending at 356MiB
parted -s /dev/loop8 unit MiB mkpart primary fat32 -- 100MiB 356MiB
# create an ext2 partition
# starting at 356MiB
# ending at end of disk
# (which is 8GB in size)
parted -s /dev/loop8 unit MiB mkpart primary ext2 -- 356MiB 7295MiB
# let's look at the partitions
partprobe -s /dev/loop8
# -> /dev/loop8: msdos partitions 1 2
# reading: the disk /dev/loop8
# with label msdos
# has the partitions 1 and 2
# now let's format the partition 1
mkfs.vfat /dev/loop8p1 -I -n BPI-BOOT
# … and the partition 2
mkfs.ext4 \
-O ^has_journal -E stride=2,stripe-width=1024 \
-b 4096 /dev/loop8p2 \
-L BPI-ROOT
# flush all caches to disks
sync
# again let's inspect the disk
parted -s /dev/loop8 print
# -> Model: Loopback device (loopback)
# -> Disk /dev/loop8: 7650MB
# -> Sector size (logical/physical): 512B/512B
# -> Partition Table: msdos
# -> Disk Flags:
# ->
# -> Number Start End Size Type File system Flags
# -> 1 105MB 373MB 268MB primary fat16 lba
# -> 2 373MB 7649MB 7276MB primary ext2
Now that we formatted the partitions, we want to write to them. So let's mount the partitions.
mkdir /mnt/rootfs
mount /dev/loop8p2 /mnt/rootfs
mkdir /mnt/rootfs/boot
mount /dev/loop8p1 /mnt/rootfs/boot
Now everything is set up and we can install a minimal Debian system on the new partition.
The next step excited me the most. We switch the operating system from our current VM to the SD card. This is what I usually do to recover a damaged operating system after booting from a CD or a USB stick.
chroot /mnt/rootfs
To be clear:
We are on your working machine inside a VM
with a file mounted as a disk with a partition on it
which we now use as our new root directory.
Now we can complete the Debian setup and configure our system.
# finish installation
export LANG=C
/debootstrap/debootstrap --second-stage
# add package repositories
cat >> /etc/apt/sources.list <<EOF
deb http://ports.ubuntu.com/ubuntu-ports/ xenial main restricted
deb-src http://ports.ubuntu.com/ubuntu-ports/ xenial main restricted
deb http://ports.ubuntu.com/ubuntu-ports/ xenial universe
deb-src http://ports.ubuntu.com/ubuntu-ports/ xenial universe
deb http://ports.ubuntu.com/ubuntu-ports/ xenial multiverse
deb-src http://ports.ubuntu.com/ubuntu-ports/ xenial multiverse
deb http://ports.ubuntu.com/ubuntu-ports/ xenial-updates main restricted
deb-src http://ports.ubuntu.com/ubuntu-ports/ xenial-updates main restricted
deb http://ports.ubuntu.com/ubuntu-ports/ xenial-updates universe
deb-src http://ports.ubuntu.com/ubuntu-ports/ xenial-updates universe
deb http://ports.ubuntu.com/ubuntu-ports/ xenial-security main restricted
deb-src http://ports.ubuntu.com/ubuntu-ports/ xenial-security main restricted
deb http://ports.ubuntu.com/ubuntu-ports/ xenial-security multiverse
deb-src http://ports.ubuntu.com/ubuntu-ports/ xenial-security multiverse
EOF
# install updates and missing packages
apt update
apt upgrade -y
apt dist-upgrade -y
apt install -y sudo openssh-server locales ntpdate htop pv
# set hostname to bpi-r2
echo "bpi-r2" > /etc/hostname
vi /etc/host
# (add 127.0.0.1 bpi-r2)
# set root password
passwd
# add user bpi
useradd -m -G users,sudo,ssh -s /bin/bash bpi
passwd bpi
# set locales
locale-gen de_DE
locale-gen de_DE.UTF-8
dpkg-reconfigure locales
dpkg-reconfigure tzdata
update-locale LANG=de_DE.UTF-8
# auto-mount our two partitions
cat >> /etc/fstab <<EOF
proc /proc proc defaults 0 0
LABEL=BPI-BOOT /boot vfat errors=remount-ro 0 1
LABEL=BPI-ROOT / ext4 defaults,noatime 0 0
EOF
# configure the network
cat >> /etc/network/interfaces <<EOF
auto lo
iface lo inet loopback
auto eth0
iface eth0 inet manual
auto wan
iface wan inet dhcp
auto lan0
iface lan0 inet static
address 192.168.1.1
netmask 255.255.255.0
#gateway 192.168.0.1
auto lan1
iface lan1 inet static
address 192.168.1.2
netmask 255.255.255.0
#gateway 192.168.0.1
auto lan3
iface lan3 inet static
address 192.168.1.3
netmask 255.255.255.0
#gateway 192.168.0.1
auto lan4
iface lan4 inet static
address 192.168.1.4
netmask 255.255.255.0
#gateway 192.168.0.1
EOF
# add SSH key for root login
# (should be changed later)
mkdir /root/.ssh
chmod 700 /root/.ssh/
vi /root/.ssh/authorized_keys
We leave our soon-to-be SD card operating system and switch back to the VM Ubuntu 16.04 installation.
exit
With the base operating system in place, we still have to do something with our self-compiled artifacts. Let's copy some files to the still mounted partitions before ejecting them.
# remove qemu-arm-static again
rm /mnt/rootfs/usr/bin/qemu-arm-static
# extract first build artifacts
tar -xvf BPI-R2-bsp-4.14/SD/BPI-BOOT-bpi-r2.tgz --keep-directory-symlink \
-C /mnt/rootfs/boot
tar -xvf BPI-R2-bsp-4.14/SD/4.14.34-BPI-R2-Kernel.tgz --keep-directory-symlink \
-C /mnt/rootfs
tar -xvf BPI-R2-bsp-4.14/SD/4.14.34-BPI-R2-Kernel-net.tgz --keep-directory-symlink \
-C /mnt/rootfs
tar -xvf BPI-R2-bsp-4.14/SD/BOOTLOADER-bpi-r2.tgz --keep-directory-symlink \
-C /mnt/rootfs
# add some more configuration
echo "blacklist mtk_pmic_keys" > /mnt/rootfs/etc/modules-load.d/mtk_pmic_keys.conf
# eject the partitions
umount /mnt/rootfs/boot
umount /mnt/rootfs
No, not done yet.
We cannot directly boot our final operating system. Some steps must happen before, to enable more features, e.g. access to more board components or access to a fat16 file-system… features we need to even start booting Ubuntu.
Remember that we left the first part of the card image empty? The first partition starts at 100MiB. Now we write into this empty space. We use dd to transfer bytes directly into our image file, which is still mounted as a disk.
# download more artifacts
wget https://github.com/BPI-SINOVOIP/BPI-files/raw/master/SD/100MB/BPI-R2-HEAD440-0k.img.gz
wget https://github.com/BPI-SINOVOIP/BPI-files/raw/master/SD/100MB/BPI-R2-HEAD1-512b.img.gz
# and write those to the image
gunzip -c BPI-R2-HEAD440-0k.img.gz | dd of=/dev/loop8 bs=1024 seek=0
gunzip -c BPI-R2-HEAD1-512b.img.gz | dd of=/dev/loop8 bs=512 seek=1
# write self-compiled artifacts to the image as well
dd if=BPI-R2-bsp-4.14/mt-pack/mtk/bpi-r2/bin/preloader_iotg7623Np1_sd_1600M.bin \
of=/dev/loop8 bs=1024 seek=2
dd if=BPI-R2-bsp-4.14/u-boot-mt/u-boot.bin \
of=/dev/loop8 bs=1024 seek=320
Note that the exact position within the image file is of uttermost importance. Sometimes the offset is part of the filename. For more information I recommend this Wiki page and this Forum thread.
Finally – the image is ready. Time to fetch it from the VM and write it to an actual SD card.
# flush caches to disk
sync
# eject our image file
losetup -d /dev/loop8
Since I have been too lazy to make the physical SD card accessible in my Ubuntu VM, I have to copy the image file to my MacBook.
# first shrink the image
zip BPI-R2-Ubuntu-4.14.img.zip BPI-R2-Ubuntu-4.14.img
# -> adding: BPI-R2-Ubuntu-4.14.img (deflated 95%)
# the resulting file size is much smaller
# since our image file contains mostly zeros
# on a mostly empty partition 2
ls -lisah BPI-R2-Ubuntu-4.14.img.zip
# -> … 330M Oct 16 15:49 BPI-R2-Ubuntu-4.14.img.zip
# now leave the VM
exit
# and download the image file
scp 192.168.12.34:BPI-R2-Ubuntu-4.14.img.zip .
You can write the zipped image file to the SD card using Etcher now.
Remaining Problems
As mentioned, the resulting Ubuntu SD card boots with some errors. TTY works but network is down. I did not find time yet to analyze or fix the error. I can access the system via the TTY console and get the following error message:
[FAILED] Failed to start Load Kernel Modules.
See 'systemctl status systemd-modules-load.service' for details.
…
[FAILED] Failed to start Raise network interfaces.
See 'systemctl status networking.service' for details.
…
root@bpi-r2:~# systemctl status systemd-modules-load.service
● systemd-modules-load.service - Load Kernel Modules
Loaded: loaded (/lib/systemd/system/systemd-modules-load.service; static; ven
Active: failed (Result: exit-code) since Fr 2021-04-02 04:51:59 CEST; 7min ag
Docs: man:systemd-modules-load.service(8)
man:modules-load.d(5)
Process: 126 ExecStart=/lib/systemd/systemd-modules-load (code=exited, status=
Main PID: 126 (code=exited, status=1/FAILURE)
Apr 02 04:51:59 bpi-r2 systemd[1]: systemd-modules-load.service: Main process ex
Apr 02 04:51:59 bpi-r2 systemd[1]: Failed to start Load Kernel Modules.
Apr 02 04:51:59 bpi-r2 systemd[1]: systemd-modules-load.service: Unit entered fa
Apr 02 04:51:59 bpi-r2 systemd[1]: systemd-modules-load.service: Failed with res
Warning: Journal has been rotated since unit was started. Log output is incomple
root@bpi-r2:~# systemctl status networking.service
● networking.service - Raise network interfaces
Loaded: loaded (/lib/systemd/system/networking.service; enabled; vendor prese
Drop-In: /run/systemd/generator/networking.service.d
└─50-insserv.conf-$network.conf
Active: failed (Result: timeout) since Fr 2021-04-02 04:57:02 CEST; 1min 41s
Docs: man:interfaces(5)
Process: 380 ExecStart=/sbin/ifup -a --read-environment (code=killed, signal=T
Process: 295 ExecStartPre=/bin/sh -c [ "$CONFIGURE_INTERFACES" != "no" ] && [
Main PID: 380 (code=killed, signal=TERM)
Apr 02 04:52:01 bpi-r2 systemd[1]: Starting Raise network interfaces...
Apr 02 04:52:02 bpi-r2 ifup[380]: /sbin/ifup: waiting for lock on /run/network/i
Apr 02 04:57:02 bpi-r2 systemd[1]: networking.service: Start operation timed out
Apr 02 04:57:02 bpi-r2 systemd[1]: Failed to start Raise network interfaces.
Apr 02 04:57:02 bpi-r2 systemd[1]: networking.service: Unit entered failed state
Apr 02 04:57:02 bpi-r2 systemd[1]: networking.service: Failed with result `timeo
Open work (more things to explore)
First: configure the installed Debian 10 to use two ports as WAN ports.
Further things I would like to look into:
it would be nice to fix the Ubuntu image
some SD card did not boot after writing the image with dd
test these images again with Etcher
Helpful links
I added all links which I considered helpful into the post above. Of course, there is much, much more to explore. Here are some additional related links I found while searching the web. A huge thanks to everyone who shares his/her experience!!!
