Again, something that isn’t directly ESP8266/32 connected, but interesting non the less. This little beastie popped up on CNX-Software yesterday (and if CNX isn’t in your daily reading list, you should add it!). It’s another one of the crop of tiny, cheap, Linux-capable microprocessor boards that have recently hit the market. This is the LicheeRV-Nano board from Sipeed, using the SG2002 SoC from Sophgo.
There are a couple of things which cause this little board to stand out from the crowd, though. First, there are four different models available (in order of increasing cost):-
The Nano-B “basic” board (no networking options).
The Nano-E board with an added RJ45 100Mbs Ethernet socket.
The Nano-W board with an added dual-band WiFi6 and BLE 5.4 module.
The Nano-WE board with both Ethernet and WiFi/BLE options.
It also has four (count ’em!) CPU cores on-board:-
1 x 1GHz C906 Risc-V core
1 x 700MHz C906 Risc-V core
1 x 1GHz ARM A53 core
1 x 25~300MHz 8051 (low-power) core.
Now here’s the really weird part …The 1GHz C906 core and the 1GHz ARM A53 core are jumper selectable before power-on. That is, your main core can be either the 1GHz C906 Risc-V core or the Arm A53 core, but not both concurrently (and they cannot be swapped when the device is running). So effectively, you only have three available cores. That second, slower, 700MHz Risc-V core is also labelled as being for use by FreeRTOS, so no parallel, multi-core Linux processing (at this point, anyway).
To balance that though, there is an additional INT8 TPU, running at 0.5T to 1.0T (and you can argue among yourselves whether you think that now counts as a three, four or five core device).
Note that the diagram above shows the SG2002 SoC, not the LicheeRV board itself (obviously not all of the SoC pins are brought out to pads on the board).
The SG2002 SoC sits in the middle of the PCB, to the right of the TF/SD card. If you look carefully in the gap between the SD-card holder and the SG2002 (and squint your eyes just right) you’ll see a white silkscreened label with “A53/C906”, with a loop extending around a couple of resistor pads at the A53 end. That is the CPU core selector jumper. The other weird thing about this arrangement is that Sipeed have advised using a rather unorthodox method for shorting jumpers in their Wiki; quote “..or use a 2B pencil to create a short circuit in the designated area“. Mmm, can’t say I’ve used that method since I was about 10 years old (and it didn’t work worth a crap even back then). Although that specific text comes from the WiFi/BLE section of the Wiki, the same method was suggested for the CPU jumpers in the comments section of the original CNX Software article (and didn’t seem to be tongue-in-cheek).
2B pencil leads and RJ45 support legs taking up GPIO pads notwithstanding (take another, closer look at that photo), the Sipeed LicheeRV-nano does seem to be a very interesting little board. It has MIPI support for both touchscreens and cameras, an on-board 1W audio amp and provision for microphone input, as well as video input and output capabilities (those being specific to processing camera image data, rather than TV-Box type applications). The USB port can support an RNDIS network connection under Linux (so, if you plan to tether the board to an existing laptop/PC for image processing, you can buy the cheapest, basic model and use USB to provide both power and networking to the LicheeRV — not particularly secure, but proven to work quite well with other, similar boards, like the MilkV-Duo).
The SoC is very well documented, with a 696-page (Chinese) manual laying out the registers in excruciating detail. You won’t find pinouts for the more exotic interfaces (Wiegand anyone?!?) on the LicheeRV, though (just not enough space).
Finally, although these boards are cheap, you should know before you drop your hard-earned cash that there are some teething problems with the SG2002 SoC which have already been identified (and noted at the bottom of the LicheeRV Nano “Introduction” page in Sipeed’s Wiki). The two main issues are the TPU currently only working reliably at 0.5T, not at the 1.0T target speed and the WiFi currently appears to have throughput issues.
Other than that, you’re good to go. Gentlemen, sharpen your pencils!
Note to self:- Must get into the habit of re-installing the bootstrap loader when doing upgrades (will be happier in the end).
I thought I’d worked out a fairly safe path with my FreeBSD 14.0 upgrades. First of all, I installed a VM and explored the new release. Nothing exploded when I touched it. It looked like FreeBSD, quacked like FreeBSD and felt like FreeBSD (and it also seemed a little snappier than the 13.x releases, which is always a good thing). So following that, I went ahead and upgraded a couple of my existing VM systems (running actual application loads) and again had an almost painless journey (it seems as though the structure of one of the compiler packages has changed, so there is a slight hiccough where a file already exists where the new package expects a directory, but it doesn’t prevent the upgrade process from completing). The merge process (manual update of files) was definitely longer than with 13.x minor upgrades, but nothing too stressful.
After a few successful VM upgrades, I turned my attention to the smallest (and least important) of my servers and, once again, the upgrade process ran through with nothing I hadn’t already seen in the previous, VM upgrades. I rebooted it during the upgrade process and again a couple of times afterwards, just to make sure that everything was stable. Then I just left it alone for a few days while I got on with other things. The server was fine and the VMs running on it were also fine (both the upgraded, 14.0 ones and the older 13.1 and 13.2, yet-to-be-upgraded ones).
Emboldened by my success, I turned my attention to one of the busier servers on our network (another machine where I’d already upgraded a couple of VMs, without incident). Again, the upgrade went without major incident. I got through the reboots and everything was looked fine.
At this point, I unwittingly loaded and tamped the old flintlock (you know …the one I use to shoot myself in the foot every now and then). I remembered that this particular server is a little bit “iffy” and, if you actually touch it (as in plugging in the video cable so that you can see what’s going on), it tends to spit out a couple of disk errors. Years of painful experience have shown that this isn’t actually a problem, unless you try to fix it. So, taking a deep breath, I run “zpool status” and, as expected, see a couple of errors showing on one of the disks in the root pool. Knowing better than to start messing with the disk (or its cables), I run “zpool clear” and note that, as expected, the errors have gone and the resilver shows a measly 1.62MB updated. But wait! What’s this? There’s a nice little note tucked in there letting me know that the ZFS feature set has been updated, but I won’t be able to use those shiny, new features until I’ve run “zpool upgrade”. Okay, been there, done that, didn’t buy the crappy tee-shirt. I balk at doing all of the pools and choose a victim which won’t cause too much trouble if it falls off the edge of the world:- “zpool upgrade zstore02“.
The machine didn’t burst into flames, the dog didn’t throw up on the carpet (that’s unusual …I should have realized right then that something was up) and zstore02 still responded to gentle prodding. It also let me know that these new features had been added:-
edonr
zilsaxattr
head_errlog
blake3
block_cloning
vdev_zaps_v2
Being intrigued by the names, I actually took the time to go to “man zpool-features” to try and work out what they all did. No chance! (nice names, though …who wouldn’t want a zilsaxattr or a vdev_zaps_v2 on a wet Saturday afternoon?).
Having noted absolutely nothing untoward, I worked my way through the other pools on the machine and never even noticed the little extra message at the end of the output when I updated the root pool:-
Pool ‘zroot’ has the bootfs property set, you might need to update the boot code. See gptzfsboot(8) and loader.efi(8) for details.
[Don’t be fooled by the “might“] I didn’t even notice the click as I (unknowingly) cocked the old flintlock.
Warmed by the knowledge that yet another upgrade was “in the bag”, I rebooted the system.
And you don’t need a commercial break, or even an over-long pause, to know that it didn’t. Instead, it threw out a goodly amount of console messages letting me know that it had no clue as to what the feature “com.klarasystems:vdev_zaps_v2” was (or any of the others in the list) and it most certainly wasn’t going to boot from the primary, secondary or tertiary disks, so there!
Futtocks!!* What have you done to my system, Allan?!?!
Never a bootable floppy in sight when you need one. Let me just shout at you for a second, just in case you’ve missed the moral of this story so far:-
DON’T ASSUME THAT ALL ZFS UPDATES ARE BENIGN, ESPECIALLY WHEN THEY TELL YOU IN ADVANCE THAT IT MIGHT HURT!
In other words, always re-install the EFI/GPT/BIOS (or whatever else you might be using) on at least one of your designated boot disks after doing a “zpool upgrade“.
Okay, so apart from panicking, what did I do to get out of this self-inflicted mess? Luckily, since I know that all of my machines use UEFI, there wasn’t too much mucking around to get things going. I knew that I needed a bootable USB stick before anything else, so off I went to find a working machine and downloaded the FreeBSD 14.0 “bootonly” image (just because it’s smaller and quicker to pull over the skinny wires I have here). Booting that was easy (as far as the system was concerned, it was the only bootable media in the whole machine). Hit return to get a shell and away we go…
mount_msdosfs /dev/da5p1 /mnt
cp /boot/loader.efi /mnt/efi/boot/BOOTX64.EFI
Two things which you need to take notice of here. The da5p1 disk will need to be changed for your specific installation (assuming you didn’t take any notice of my shouting and got yourself into this mess, too), but do note that, because we booted from a USB device, all of the other USB drive numbers have been shuffled up by one. My USB thumb-drive is now da0 and the drive which I want to install the EFI bootstrap code on is normally drive da4, but (for this boot only) has become da5. I am (in this case) deliberately not changing either the primary or secondary boot disks, just in case I mess up again (they will be done later, once this change is proven good).
The second thing is that the EFI partition on disk is always quite small, so it is generally better to overwrite the existing file, rather than write a new file with a different name (running out of space when copying to the MSDOS UEFI partition will generate a general “I/O Error” message, which can be a bit confusing). Re-using the same name also ensures that the bootable drive description created by the efibootmgr command remains the same and doesn’t require updating.
That was it. I synced and shut down the system and it rebooted successfully from the tertiary boot disk.
So, did my foot hurt? Nope! The old flintlock misfired and the ball went sailing off across the room. It would probably have killed the dog, but she was hiding behind the sofa …puking on the carpet. All is back to normal (for the time being, anyway) in Schloss Puce.
*Futtocks – Sounds like a good, old-fashioned Anglo-Saxon swear word …but isn’t. Your homework for today is to find a dictionary and look it up.
In the first part of this series, we looked at accessing our LuckFox Pico Pro using an RDNIS network connection over the USB port to connect with the minimal “busybox” Linux shipped from the factory. In this second part, we’ll explore replacing that minimal system with a much more capable release of Alpine Linux to access the full potential of this tiny board.
Before starting on our upgrade, we do need to make one small change to the hardware on the board. This is the addition of header pins for TX, RX and GND connections to UART2, to allow us to connect to the serial console of the device.
LuckFox Pico Pro with SD-Card, RTC-Battery, Console Adapter & Ethernet
While the bulk of the GPIO pins will probably be soldered to the bottom of the board (to allow mounting on a larger carrier board and thus access to the GPIOs), it makes sense to mount these three pins to the top of the board, for ease of connection of a serial-to-USB module (note that this board has 3v3 I/O pins, so you’ll need to use an appropriately rated serial-to-USB module, not a 5v one).
Flashing Alpine Linux
There is already an Alpine Linux image available for this board, linked to from the LuckFox Wiki page. User “walker2048” contributed two very good articles on the Chinese “EEWorld” forum giving a very good step-by-step description of building, installing and using an Alpine Linux image (these are the posts available in English on the LuckFox Wiki), but unfortunately the articles are Windows centric and assume that you’re going to be using the SocToolKit GUI to load the image to the board. The linked Alpine Linux image is a zip file and not the “.img” format that we’d normally “dd” to an SD-card or memory under Linux. Although there is a command-line “upgrade_tool” available for Mac and Linux users, frustratingly it also assumes a target XXXX.img file (the same as the Windows GUI SocToolKit version), but as just mentioned, that’s not the format that the Alpine Linux zip file contains. What to do?!?
There are some hints in the Alpine Linux zip package. There are a couple of text files contained within it:- sd_update.txt & tftp_update.txt.
Unfortunately, neither of them is the hoped-for “how to” text for users, but they do turn out to be how to texts for some sort of assembler or builder program. That started me off on a search for the method used to build a modified distribution from the SDK and, lo and behold, sitting right there in the luckfox-pico SDK bundle is a /tools/linux/Linux_Pack_Firmware directory. Yes!! There are both mk-update_pack.sh and mk-update_unpack.sh scripts in that directory. A quick perusal of the pack script shows that it only needs to be passed a couple of options, which identify the target chip type and the input source location (also where the “update.img” output file will be written). I quickly un-zipped the Alpine-Linux.zip package into a new “alpine_test” directory and ran:-
TOOL_PATH=luckfox-pico-SDK/tools/linux/Linux_Pack_Firmware Android Firmware Package Tool v2.2 ------ PACKAGE ------ Add file: ./alpine_test/env.img env,Add file: ./alpine_test/env.img done,offset=0x800,size=0x40000,userspace=0x80,flash_address=0x00000000 ... ... package-file,Add file: ./alpine_test/package-file done,offset=0x1365800,size=0x83,userspace=0x1 Add file: ./alpine_test/download.bin bootloader,Add file: ./alpine_test/download.bin done,offset=0x1366000,size=0x3e9c0,userspace=0x7e Add CRC... Make firmware OK! ------ OK ------ ********rkImageMaker ver 2.2******** Generating new image, please wait... Writing head info... Writing boot file... Writing firmware... Generating MD5 data... MD5 data generated successfully! New image generated successfully! [mk-update_pack.sh:info] Making -RK1106 update.img OK.
…and was rewarded with a nice, shiny new alpine_test/update.img file. Okay, now we’re more than halfway to our objective. The next part is a little bit scary, in that we have to overwrite the existing “busybox” image in NAND flash with our untested Alpine image, which could leave us with a semi-bricked Pico Pro. The saving grace here is that, should this happen and the Pico cannot boot from flash, it should automatically try to boot from the SD, so we do have a plan B.
Loading the image to NAND flash is now fairly simple, as we just need to follow the Wiki instructions for Linux Burning for Linux/MacOS (before doing this, you should already have added the UART2 header pins as outlined above and connected your serial-USB adapter). The basic procedure is to grab the command-line upgrade_tool for Linux from the link on the Wiki page, hold down the “BOOT” button on the Pico Pro and then press and release the “RESET” button (to force the board into download mode) and then use the command:-
upgrade_tool uf upgrade.img
…to start the actual update process.
Running Alpine Linux on the LuckFox Pico Pro
Probably the first thing to note about running Alpine Linux is that there is no RNDIS connection started on the USB port, so you must use the serial console to configure the board from this point, until you enable ssh logins over ethernet.
This Alpine 3.18 build from “walker2048” uses a default ethernet address of 192.168.50.59, so in all probability, you’ll want to log in on the serial console to modify /etc/network/interfaces to match your own LAN configuration.
You don’t have to remember any pesky user-ID/password combinations, as “walker2048” has also configured an auto-login for “root” on the console. You might want to change this configuration by editing the /etc/inittab file to comment-out this line (just a quick word of caution here …before you begin this change, you need to ensure that you have a valid password set and tested for the root account):-
The two things to notice with this replacement are the obvious, missing “--autologin” and the not so obvious change from “agetty” to just plain old “getty“ (“agetty” is “advanced getty” and has the magic sauce included to create automatic logins, which plain old-fashioned getty doesn’t).
If everything went according to plan, you should now have a very usable Alpine distribution on your tiny LuckFox board and should be able to log in over the network and use the apk package manager to search for and install standard packages.
I’ve been diving down multiple rabbit holes recently with Linux on really tiny SBCs. Nope, not the Raspberry Pi, but things that are roughly a quarter of the size and just one-quarter of the price, too. One of these is the LuckFox Pico Pro (an ARM7 board with a whopping 128MB of memory). See links at the bottom of the page for more info on the LuckFox series of boards.
The most obvious feature of the Pico Pro is the RJ45 network connector, taking up roughly a quarter of the total board space on what is rather a small PCB. At roughly 75mm x 25mm (3″ X 1″), that board packs quite a few surprises into a very limited space. The first is that there are actually two network ports available for use on the board, as shipped. While most of us would classify using the USB port as “cheating”, it is actually the enabler which makes this board so accessible straight out of the box (no soldering header pins or fiddling around with serial adapters required). You can literally take this board straight out of it’s anti-static shipping bag, plug in a USB cable for power and start talking to it using RNDIS.
I have to admit that I had to ask Gewgul what the heck RNDIS was when I first encountered it with one of these tiny boards. I was not encouraged by the initial description of “The Remote Network Driver Interface Specification (RNDIS) is a Microsoft proprietary protocol” and it only got worse from there on, with the phrase “is inherently and uncorrectably insecure” appearing just a couple of paragraphs further down. Having said all of that though, the practical application of RNDIS in this instance is that we get to run an “Ethernet” connection over the USB cable between our Pico board and a Linux machine. This means that you can access the cut-down version of Linux running on the Pico without having to (physically) do anything other than plug in a USB cable.
This is great because, although the LuckFox Wiki is really very good (excellent, when compared with some of the competitors), it can be rather overwhelming on your first encounter and it’s not immediately clear how to access your new Pico Pro board from Linux (hence this post).
So, to begin with, the Pico Pro is shipped with a basic “busybox” version of Linux, which will boot-up as soon as the board gets power from the USB port. This very limited implementation will automatically start an RNDIS process (on network device “usb0”) to allow access without having to solder headers, or change the hardware in any way.
How to use ADB to access the Pico via USB
I’m going to start off with a warning. The code burned into the LuckFox Pico as it comes will start NSDIS on the USB port with an address of 172.32.0.93. This address is not in the RFC1918 “private” address range (The top address in the class-B 172 range is 172.31.255.255). According to the whois database, the 172.32 range is owned and assigned to T-Mobile USA. This means that you must ensure that packets transitioning the RNDIS link to your Linux machine are NATed, as typical home/small-office firewalls will be blocking “private” ranges from leaving your network, but not the 172.32.0.0 range. While in 99% of cases your router or your ISP (or both) will be NATing your connection anyway, there will be cases where these packets might leak out.
I’ve raised an issue on the LuckFox Pico github, but even if they fix it immediately, all of the current stock already out there will suffer from this problem. See below for an example script to ensure that your Linux system NATs the packets flowing over the usb0 connection.
Okay, so having noted that RNDIS is inherently insecure and that the burnt-in address is not in the private range and needs to be NATed, here’s how to get started as quickly as possible.
You don’t really have to do very much on any Debian-based distribution to get up and running. The kernel will recognize an RNDIS connection and automatically initialize your machine’s end of the USB connection with an IPV4 connection on your “usb0″. What it won’t do is assign an IP address, so you should use the NetworkManager application (or edit the newly created /etc/NetworkManager/system-connections/”Wired connection XXXX” file) to assign a reasonable address (I used 172.32.0.100). Watch out for the netmask; 255.255.0.0 will work (255.255.255.0 will not).
So let’s go get that script to clean-up the “usb0” connection. Note that this assumes that your connection is on “usb0” and that you’re using the address 172.32.0.100; you’ll need to edit it if not. It also adds ip-forwarding on your local machine, so you should be prepared to turn that off (or reboot the system) if this is a security issue for you (use the command sysctl net.ipv4.ip_forward=0 to turn off ip-forwarding without rebooting the machine).
Now comes the interesting bit. The minimal Linux on the LuckFox Pico does not have sshd running, so although we have a basic network connection, we still can’t directly ssh to the device (“It’s Linux Jim, but not as we know it.”).
What we need to do is run the Android debug tool, “adb”. This is available in the “adb” package (so, apt install adb prefaced with “sudo” if you prefer to work that way). The adb command provides a command-line interface. First we need to use it to connect to the adb daemon running on port 5555 of the LuckFox Pro:-
adbconnect172.32.0.93:5555
Next we use adb to spawn a root shell:-
adb -s 172.32.0.93:5555 shell
With that, you should be dropped to a “#” prompt on the LuckFox board. There is a fairly limited command set available to you, but you can find your way around and get a feel for the filesystem configuration and memory capacity of this little machine. It really is surprisingly powerful for such a tiny bit of hardware (thus speaks Ye Olde GreyBeard who used to work on production systems with just 4K of memory).
Coming up next, in Part-II, we’ll delve into the absolute minimum hardware requirements to install Alpine Linux on the LuckFox and get the on-board ethernet port working (on your LAN, with a sensible IP address).
My search for a “daily steal” this week has been influenced by the thought that I’d quite like to upgrade the backbone network in the old, ancestral home to 2.5GbE. I don’t particularly want to go all the way to 10Gb fibre, basically because it’s too much of a hassle to pull new fibre between the dungeon and the castle, but 2.5GbE seems to be a reasonable compromise between speed, pragmatism and budget. At any rate, I thought I’d take a look and see what was out there that might work as a 2.5GbE firewall, without breaking the bank.
It not only has 8GB of RAM and dual 2.5GbE ports, but a USB C port and two USB-3 ports, too. In addition to all of that, it comes with Linux pre-installed. I just couldn’t believe it; I thought I’d died and gone to (cheap techie) heaven!
Of course, the initial midge in the ointment was that the CPU is a fairly lacklustre, 2016 vintage, dual-core AMD A9-9400, but what the heck, for a small firewall/router box for a home network, it should pass muster.
Then I scrolled down to the “product information” section. The good thing is that, unlike some makers, they do actually have some useful information there …unfortunately, this is also the point that the midge transmogrifies into Jeff Goldblum standing waist-deep in your Vaseline. In the Q&A section it explicitly states “A: Both ethernet ports are Intel i225-V”. That is not good news and might explain why these machines are selling so cheaply. Intel’s i225-V has been plagued by instability problems and was quickly replaced with upgraded hardware in the i226 version by them. The fact that this board has the i225s makes it a non-starter for me; the last thing I need to buy for a firewall project is known flaky hardware. I’m not going to take a chance on this one; you might feel differently if you have a project where the network is a little less critical, though.
Anyway, I am (unsurprisingly) not the first person to find this particular bargain and this morning I came across “TechBecause”‘s video of the unboxing and de-bagging of his machine. So far he’s reasonably impressed and he’s also verified that the ports do indeed come up at 2.5Gb when plugged into a switch. You can also see that there’s a SATA cable taped down to the motherboard in one of his shots. As the video is only five days old, we’re going to have to wait for longer term usage results, though.
I came across today’s choice while idly meandering the less-travelled passageways of Mr Baba’s emporium looking for an ESP32-S3 board on which to run Linux, but got side-tracked when I noticed that the older B-Max N3350 (Apollo Lake) machines were now down to less than $70 (nope, I don’t recommend them, given what else is now available). That, in turn, lead to today’s screamin’ steal.
Now I should just warn you in advance that this machine is stupidly named and absolutely butt-ugly, but as long as you have a dark shelf where you can hide it, this is still a steal.
All in all, the quad-core N5105 processor will provide roughly twice the performance of the N3350 mentioned above. For just $10 more than the N5105 model, you can instead get the same, basic machine equipped with a newer, Alder Lake N95 (quad-core), which will take you to (again, very roughly) four time the performance of the N3350 (and now you can probably see why I don’t recommend that older, dual-core machine, despite the low, low price).
In conclusion, the N5105 8G/256G model is currently $98.60 with free shipping. The N95 8G/256G model comes in at $108.10, also with free shipping. Personally, I’d be quite happy with the N5105 model, but if you’re planning to go with virtual machines or multiple containerized apps, the N95 may be worth the extra $10. As I write this (Aug 3rd 2023), there are 59 reviews for this item; 54 of those are 5-star, 4 are 4-star and the single remaining 2-star review is someone complaining that they had to pay import duty on the package. Eleven of the reviews include photos of the unit (including one showing the 2.5″ bay and cables), so you can get a pretty good impression of what you’re getting.
Have you ever wondered what that column header, “EXPANDSZ”, in the zpool list command actually refers to, or why it is always empty? Below is a short, real-world example of how it can help you (and did help me).
TL/DR; The manual page for “zpoolprops” tells us that EXPANDSZ is the “Amount of uninitialized space within the pool or device that can be used to increase the total capacity of the pool”. In other words, space that is not already being used by ZFS that might possibly be used to expand our pool into. The manual page also mentions “resizing in place”, which implies that we can expand the size of a ZFS pool on a live, running machine. This sounds, and is, incredibly useful, but there are caveats. For instance, if your pool is a mirror, then all of the sub-mirrors must have free space available. More on this below, where I’ll demonstrate that yes, we certainly can resize a filesystem on a live, running machine (in certain circumstances).
Back story — I have several low-cost, mini-pc systems which work together to provide network services on our LAN. Each of these machines came with some eMMC memory built in (the amount varies, depending mainly on how long ago the machine was purchased). Those machines were, in some cases, installed with the root partition on the internal eMMC and then later updated with external hard disks, using ZFS to mirror all partitions (including the original root partition). This was convenient at the time and also meant that it was easy to reuse the machine without having to do a full install.
Moving forward over the years, a couple of those machines have been upgraded through several versions of FreeBSD and, among other things, the lax management on my part of snapshots and boot-environments has resulted in a squeeze for space on the original root partitions along with (horror of horrors!) an exploding fragmentation level. As I’ve mentioned in these pages before, anything north of 80% fragmentation on a ZFS filesystem is very bad news and needs remedial work. I was hoping to hang on and re-install with a nice, squeaky-clean 14.0 release on the worst affected machine, but the fragmentation issue has forced my hand. So here’s what the fairly critical state of affairs on that machine looked like, before remediation:-
As you can see, we have 2.11GB of “free” space, but that 91% fragmentation figure means that the system is probably going to have to work exceptionally hard to squeeze any sizeable file in there. At this point, the machine becomes almost unusable.
There are a few things worth noting here. The first thing is that the space on the original eMMC partition, /dev/mmcsd0p3, is almost exhausted, but the other partitions in this multi-way mirror all have partition sizes almost double that, at 100GB. The second is that those three partitions are all on identical disks, with identical block counts (you can still use the expansion method described here on mirrors where the sizes are not exactly equal, but in that case you absolutely must use the smallest partition as the resize target, as you cannot attach a smaller partition to a larger mirror). Lastly, despite the obviously available space on those three partitions, the EXPANDSZ column does not show any space being available for expansion. Why not?
Well, despite there obviously being space available, ZFS doesn’t consider it to be really available until it can actually implement a resize operation to claim that space. In the specific case of the multi-way mirror shown above, ZFS can’t resize it until it is reduced to a single physical device. This sounds scary, but in practice simply means that you need to detach physical devices from the mirror until you have a single device, resize the filesystem and then reattach the other devices again. All of this can be done with the system up and running normally, with no reboots required (but you do need to take into account that the resilvering of reattached devices can load your system down more than usual).
So let’s start by detaching the eMMC partition, /dev/mmcsd0p3:-
Okay, so now the eMMC partition has gone, but there are no other changes to free space or EXPANDSZ.
Next we detach two of the remaining three physical partitions:-
The detach of da4p3 again doesn’t produce any noticeable change in free space or EXPANDSZ, but when da1p3 is detached, we suddenly see 42.5GB appear in the EXPANDSZ column.
Okay, now we can go ahead and run the resize operation. To do this we use the “online -e” command on our remaining partition, da0p3:-
The “online -e” command takes just a couple of seconds to complete, after which we can see that EXPANDSZ has gone back to displaying nothing available, while our free space has jumped to 44.6GB and the fragmentation has gone down from 91% to 52%. At this point, all that is left to do is to reattach da1p3 and da4p3 to the mirror again:-
And that’s it! We’ve removed the original, cramped root device, /dev/mmcsd0p3, from the root mirror and resized the mirror to use the additional 42.5GB which was available on those external disks, all without a single reboot and without any interruption to the other services running on the machine. All done with the magic of ZFS.
Update Aug 20th 2023 — This board, as well as the additional Ethernet RJ45 PCB, is now available from the SpotPear Electronics shop on Aliexpress (as usual, this is for your information only, not an endorsement of the Milk-V Duo or of SpotPear).
News today of a very low cost, RISC-V based, ESP (and possibly RPi) killer coming to a store not-near-you sometime real soon. The board in question is the Milk-V Duo “embedded development platform”, with a dual-core, CV1800B SOC. The advertised price (in big letters at the top of the product page) is $9.
Unfortunately, there are some minor drawbacks:-
Currently (early June 2023) it is only available within the Middle Kingdom1.
It is a dual-core processor, but currently only a single core is supported under Linux.
It doesn’t have a complete, physical network interface (you need to add magnetics and an RJ45 for ethernet)2.
It only has 64MB on-board3.
The supplied Linux image apparently requires RNDIS4.
“Single sided PCB”. I think they mean components on one side only (okay, not really a drawback).
USB2 (or not). They mention a pin-header for a USB2 port, which doesn’t seem to exist.
On the face of it, this still looks like a very attractive board for a small, low-power system.
It has 0.1″ pin spacing (Yay!).
Lots of GPIOs (including multiple UARTs).
A MIPI camera connector.
An un-populated microphone connector.
It is really cheap.
It has an available Linux image and is also touted to run RTOS.
2 – There is no on-board WiFi. The product page notes that a separate RJ45 board for the ethernet connection will be available later, but note that RJ45s with the magnetics already built in are widely (and cheaply) available …and you’re going to need a soldering iron whichever route you choose.
3 – The unpopulated chip outline on the L/H side of the board (photo above) is an “SD NAND solder pad”, but anything soldered there will obviously block access to the SD card slot.
4 – RNDIS — I didn’t know either …had to look it up. Some random Windows protocol to enable a virtual ethernet link over USB.
This is just a brain-dump of the information I’ve gleaned so far while trying to load Linux onto one of these boards.
Update 9th Dec 2023 — Just a note here that [ValdikSS], who has contributed useful hints and pointers here in the past, has recently released UoWPrint, a product based on the 3G-IoT-A and Debian, to allow you to convert older, USB-only printers to being WiFi accessible. [ValdikSS] has pledged that for each device sold, he will donate $2 each to the CUPS and AirSane projects. Over at CNX-Software, Jean-Luc has covered the project in a little more depth, with photos of the finished product.
Update 23rd May 2023 — A slightly longer update today. First of all, I was getting a little bit tired of troubleshooting the Debian/Ubuntu installations on my original board (with the broken console) by pulling the power, moving the SD-card to my laptop and inspecting the contents of /var/log. It did work and I did make a few, faltering, forward steps, but progress was deathly slow (even by my standards). So, a few weeks back, I ordered another two boards from the same vendor (and again, they arrived very quickly) and I’m happy to report that both of the new boards work perfectly.
I’ve been helped very much along the way by reader [titus m] (and I recommend that you check out all of his very helpful comments below) and his updated scatter file (linked here and below) has been a great boon in easily loading both of the available Orange-pi supplied Linux image files. His latest post is a brief comparison of performance between the 3G-IoT-A and his Nano-Pi-1.
Reader [Fueryon] also posted some useful information (see the comments section, below) and it’s worth repeating here (for anyone considering purchasing one of these cheap boards) that both the UART pins and the GPIO pins are 1.8v logic and are not 5v or 3v3 tolerant (I use a “DSD-Tech” branded TTL-to-USB adapter (model number:- SH-U07A) which works reliably and is available from Amazon). The GPIO pins could be a problem if you intend to interface to just about any commonly available SPI/I2C modules.
One important point here is that there appears to be an incompatibility between the mkfs/fsck binaries and other filesystem tools (such as “mount”) under Linux, which cause the ext4 filesystems on external drives connected via USB to fail with superblock errors, even though fsck runs successfully. Using a (non-ext4) swap partition appears to work. Using a different type of filesystem (ie:- vfat) does work. On a related note, NFS doesn’t appear to be baked into the kernel, either. Taken together, these latest findings make these boards significantly less attractive (I have always tried to steer clear of running from an SD-card by using it as read-only media where the hardware will only boot from SD and then remounting the filesystems read-write on HDD/SSD once the kernel has loaded).
Update 27th April 2023 — A major update from [titus m] today. He has confirmed that both Ubuntu and Debian are booting on his board (he recommends Debian as having better all round performance). The key to this is his updated “scatter” file, which allows you to install the Linux distributions using the original, Android partitioning scheme (you just need to use “Download” in SP_Flash_Tool, -not- “Format and Download”). I can also confirm that my board will download the Linux Debian image using this scatter file and the v5-2032 version of SP_Flash_Tool for Linux on an Ubuntu 18.04-6 system (just ignore the initial start up errors, select [titus m]‘s scatter file and click “Download”).
Initial Check
The 3G-IoT-A doesn’t have an HDMI connector, so ascertaining whether the board is functional or not can be a bit tricky.
The easy way is just to plug earphones into the 3.5mm audio jack and then power the board using the micro-USB connector.
The red LED should light when the power is applied. The green LED will light shortly after that, then after a few seconds, will go back out again.
At this point you can press the “power” button for a couple of seconds to have the pre-loaded Android start up. After about 45~60 seconds you should hear a few bars of drum and guitar music play. This indicates that the board is operational and has booted into Android.
USB info
J1 is the full-size, type-A USB connector.
J12 is the micro USB connector
CON1 is the USB data select jumper group. The jumper configuration shown here will work with “mtkclient”.
CON1, the group of jumpers situated behind the type-A connector select whether the type-A or the micro-USB is connected to the USB data lines to the (single) USB controller on the chip. You can only use one or other of the connectors, not both at once.
J12, the micro-USB connector, has the legend “USB DC JACK” on the schematic (page-8) and uses the data lines “DL_USB_DP” and “DL_USB_DM”. The flashing software (ie:- SP_Flash_Tool, mtkclient) will use the data lines on this connector to communicate with the board, so the jumper group CON1 must be set correctly for this to work. NOTE — The Orange-Pi User Manual shows an incorrect configuration for CON1 in the Linux flashing instructions, the photo in the Android instruction section (immediately above) is correct for both Android and Linux.
The tiny HUB/USB dots-in-squares silkscreened on the board (see photo above) doesn’t appear to convey any useful information at all. Here’s an additional ASCII diagram to help you visualize the jumper settings correctly.
ASCII-art. If the pictures of connectors and jumpers just look
like a jumble of vertical lines all squashed
together, try selecting your display font to be "Courier 10"
(or some other, monospaced or non-proportionally spaced font).
JUMPERS
-------
*** IMPORTANT ***
Note that the jumpers on CON1 (the "six-pack" jumper block just behind
and slightly to one side of the A-type USB connector) are -not- placed
side-to-side across the jumpers in the normal fashion, but up and down
between the centre pins and either bottom or top pins, like this:-
o -------------
CON2 | | AUDIO |
o -------------
o o o o
| |
EITHER o o CON1 OR o o -------------/
====== == | | | ====|
o o o o | USB |
| A-TYPE |
| ====|
-------------\
The middle two pins connect to the USB data lines on the CPU.
The top and bottom pins connect to the USB A-type (L/H example)
and the micro-USB port (R/H example), respectively.
The jumpers should therefore be in the bottom position when trying
to flash the device via the micro-USB port, or in the top position
when connecting an external device (ie:- a portable, USB HDD) to the
full-size, A-type port.
You can only ever use one port at a time.
-----
JUMPER SETTINGS FOR FLASHING/PROGRAMMING THE BOARD
==================================================
CON2, the single jumper behind the audio jack, needs to be removed
to enable flashing.
The CON1 jumpers need to be set to be on the bottom two pins (with
the audio jack towards the top of the board) to enable the USB-micro
port.
o -------------
CON2 | AUDIO |
o -------------
o o
CON1 o o -------------/
| | | ====|
o o | USB |
| A-TYPE |
| ====|
-------------\
-----
JUMPER SETTINGS FOR NORMAL OPERATION OF THE BOARD
=================================================
For normal operation, the CON2 jumper should be in place.
NOTE -- Using the current (as of May 2023), Orange-Pi
supplied Debian and Ubuntu images, there appears
to be an inconsistency between the versions of mkfs/fsck and other
filesystem related utilities such as mount, which causes actual
filesystem operations to fail with "superblock" errors. In other
words, although external drives are recognized and partitions can be
created, they can -not- be used (swap partitions can also be created
and do, in my limited experimentation so far, appear to work).
...Other filesystems (ie:- vfat) -will- work on external drives.
USB-Micro Port Enabled
----------------------
If the CON1 connectors are on the bottom pins, the USB-micro
port is enabled.
o -------------
CON2 | | AUDIO |
o -------------
o o
CON1 o o -------------/
| | | ====|
o o | USB |
| A-TYPE |
| ====|
-------------\
USB-A (Full-Size) Port Enabled
------------------------------
If the CON1 connectors are on the top pins, the larger, USB
A-type port is enabled.
o -------------
CON2 | | AUDIO |
o -------------
o o
| |
CON1 o o -------------/
| ====|
o o | USB |
| A-TYPE |
| ====|
-------------\
J1, the type-A connector, has the legend “ON THE GO” on the schematic and uses the data lines “HB_USB_DP” and “HB_USB_DM”. The 5v output on pin-1 of this connector (to power external devices) is controlled by U15, an SY6280 chip.
Reader [Vinicius] has added more information in the comments below about his efforts in trying to get the Android screen-mirroring function to work. He notes that is the CON2 jumper is removed, Android will go into “adb” mode. From my Linux system, this means that some pseudo disk devices and other “android”-labelled devices suddenly become available in /dev. The 3G-IoT-A board itself also appears in the output from “lsusb -v” (it is interesting to note that it identifies itself as an “Android Phone [Fairphone First Edition (FP1)]”). Before you get too excited, I have to report that trying to access any of the Android disk devices results in a “no media present” error …but!, “adb” mode actually means that the board is in flash-programming mode and can now be updated from Linux.
Progress — See the “adb” section, under Linux, below.
40-Pin Header
Holding the board with the two USB ports to the left and the “L”-shaped, metal RF shield to the right, pin-1 of the forty pin connector is at the right-hand side of the board (next to the mounting hole and slightly above and in from the power switch) on the top row of pins (the top row is the odd numbered pins and the bottom is the even numbered).
Note that these pins are 1v8 logic and are not 3v3 or 5v tolerant.
So, counting in from the right, the 5th, 6th and 7th pins on the top row are the GND, RX and TX pins for UART1. Using the 40-pin connector numbering, pin-9 (the 5th pin from the R/H side) is GND, pin-11 (the 6th pin from the R/H side) is RX and pin-13 is TX.
My original board was faulty, so there wasn’t any output from UART1 during normal Android boot-up. However, 921600 is the baud rate to use for your terminal and is now confirmed to work with two, new boards (from the same vendor).
Reader [titus m] notes that his board will –not– boot Android unless the UART connection is first disconnected (ie:- disconnect the serial adapter, plug in the micro-USB power cable and press the power button to start the boot sequence and only then reconnect the serial adapter).
Chips
U201 is the MT6572 CPU [page-2].
U301 is the MT6323 PMU chip [page-3].
U401 is the KTN0403CS-TCR1 combined memory chip [page-4].
U6 is the 2617 5V regulator [page-8]
U15 is the SY6280 power sw/curr protect control for the 5v to J1 USB [p-8].
Linux
It is fairly important to set up the user groups and the /etc/udev/rules.d correctly. Use the instructions in the “mtkclient” README to get this correct.
In addition, it seems that removing the modemmanager package can help, too.
Once you’ve set everything up as per the “mtkclient” recommendations, you should be able to work with the board as a normal user, without needing “root” permissions.
ADB
NOTE — Unless you’re an Android geek and want to explore more, you probably don’t need the information here (this turned out to be a not too productive diversion in our quest to get Linux running on this board — see the update at the very top of the page relating to the “scatter” file and the info on setting CON1 and CON2 jumpers correctly).
—
Removing the CON2 jumper and then booting Android (as suggested by [Vinicius], above) will create some new device files in your /dev directory:-
As per Andy’s article, you first need to download the Linux version of the platform-tools zip from dl.google.com. Once you have that unzipped, simply cd platform-tools and execute the adb file with ./adb shell . Your prompt should change to something like:- root@hexing72_cwet_lca:/ # to indicate that you now have root access on Android running on your 3G board.
From this point, you should be able to use the adb “install” or “sideload” commands to install new packages on your under Android. One step closer to having a fully functional device.
NOTE:- This is a work-in-progress, live document and will be updated at irregular intervals as I discover more about the 3G-IoT-A and the tools we need to work with it
The really good news is that it shipped out promptly and was delivered very quickly. It arrived on my doorstep just six days after I ordered it. This mirrors a couple of other purchases made from the Middle Kingdom recently; nowadays we have to pay shipping, but they do seem to arrive in a much more reasonable timeframe.
The very good news is that the board is actually functional (more on this below).
The good news is that I can talk to it and get what look like reasonable responses.
The not so good news is that it appears to be (at best) somewhat complicated and perhaps (at worst) impossible to get Linux onto the board using only Linux resources.
The very, very bad news is that my board seems to have a faulty UART1, which means I can’t get a console via the GPIO pins (okay, maybe I could if I was able to swap he console onto UART2, but I can’t see an easy way of doing that at the moment). So, even though reader [titus m] has made huge strides in massaging the scatter file and also made some very helpful suggestions on the UART issue, I can’t easily get this board running (I thought that with a working WiFi config on the SD card I might be able to get it to come up “blind” and log in over the network, but so far that hasn’t worked either).
For those interested,, the URXD1 and UTXD1 signals go to the 40-pin connector and two test points (on the back of the board). Unfortunately, the URXD1 pin on my board measures only 12~13 Ohms to ground (no matter which polarity the meter is connected). That is much too low for a pull-down resistor (and is probably the reason that the CPU chip is much warmer than I expected it to get). The URXD2 pin shows no such oddness.
Functionality & Linux
Preface — For anyone interested in using the “mtkcllient” tool to prove the functionality of their board (it works via the micro-USB port, by the way, not the UART), please read on. Otherwise I would suggest that you jump straight to the “Info” post (the next in this series), which is more up to date.
—-
Let me start with basic functionality. It appears that these boards ship from Orange-Pi with some version of Android installed. Don’t ask me what, because I don’t have a display which uses the specific ribbon-cable used by this board and it doesn’t have HDMI output.
A (Very) Simple Functionality Test
So, how do you tell if the board is at all functional? That’s actually pretty easy. There is a 3.5mm audio jack on the board, so just plug in a pair of earbuds or headphones or whatever you have to hand (if it’s an amp, turn the volume down!), connect power to the board via the micro-USB and verify that the red LED lights. After about three seconds, the green LED will light and then go off again. Press the “power” button and note that the green LED comes back on (Android is booting up at this point). After quite a while (more than 30-seconds, but less than two minutes) you should hear a jazzy, drum and guitar tune play. Ta-dah! You have a functioning board. If you don’t get music, don’t panic; it is possible that you might have received a board flashed with something completely different. There is still hope — see “mtkclient”, below.
SP Flash Tool
This is where it started to go off the rails for me. There are quite a few articles and videos on the ‘net which describe how to flash your MT6572 based device using “SP_Flash_Tool”. 95% of them assume that you are using Windows. I don’t have a Windows machine in the house (or anywhere else), so that option was a non-starter for me. Well there must be something available for Linux users nowadays, surely? And yes there is, but…
It turns out (and none of the how-to articles I found mentioned this) there are two very distinct versions of SP_Flash_Tool available. The current 6-series version, which uses an XML formatted installation configuration and the previous 5-series versions, which use something called a “scatter” file. This is a pretty important point with this board, as several Linux versions of the flash tool are based on v6 code and willNOTwork with the older, scatter-file based Debian or Ubuntu downloads available from the Orange-Pi site. It took me much longer that I care to admit to realize this.
Once you come to this realization, you rapidly hit the second branch in the twisty-little-maze of the SP-Flash rabbit hole. There are many, many sites offering downloads of the flash-tool for MTK-based phones (and, as the Orange-Pi 3G-IoT-A uses the MediaTek MT6572, which is basically an ARM A7 variant designed for the smart-phone market, this is what we’re looking for). Many of those sites seem to carry pretty much identical content and have very similar URLs (ie:- spflashtools.com, spflashtool.com). As there’s no mention of flashing the MT6572 even in the “Legacy” section of the MediaTek web site, there’s no obvious way to know which, if any, of these other sites might be an official repository (so, even after checking the SSL certificate details, I can’t recommend any of them as being a safe, virus-free provider; so no links here).
The next twist in the road is when you cross your fingers and, hoping for the best, download one of the multitudinous SP_Flash_Tool packages and immediately get an exception error because “libpng12.so.0” is just -so- last century.
More searching …”what you really need is this specific revision of SP_Flash_Tool running on this specific revision of Ubuntu”. Ah, okay, well now we know. And the result? “This version of SP_Flash_Tool requires kernel version XXXX. Bye!”. Grrrrr!
This was followed by an interminable cycle of Gewgull search. System build. SP_Flash_Tool download. …SP_Flash_Tool: “Floating point exception: core dump”.
Finally, after several days of intense frustration, pulled hair and chewed tablecloths, I came to the pinnacle of SP_Flashdom …revision 2032 of the Linux version aligned perfectly with Ubuntu 18.04-6 and my machine could finally talk to the 3G-IoT-A board.
And of course the re-flash just quietly failed. Bug writ!!
MtkClient
If nothing else, all of those Giggle searches at least provided one single gem in an ocean of dross. I was directed to Bjoern Kerler’s excellent “mtkclient” package as having some relation to the general subject of flashing MediaTek products and indeed, Bjoern’s Python script turned out to be the first (and so far only) package that actually produced any useful, readable output from the board. This occurred somewhere close to the start of the SP-Flash cycle and gave me some hope that all was not lost, with confirmation that the 3G-IoT-A board was communicating and wasn’t just a very bad, single tune, MP3 player.
The “mtk” program will allow you to read back (and save) various parts of the 3G’s flash memory and promises to write them, too (untested, so far). So, for instance, we can ask mtk to display what it knows about the 3G board with:-
I’ve saved the boot-ROM and SRAM contents so far, but have once again hit a bit of a roadblock. The fly in this particular ointment is that mtk doesn’t like the preloader on the 3G board, so I still haven’t managed to get Linux loaded. At this stage it is mainly because I don’t have any clear idea of what the preloader actually does or whether I can just go right ahead and use mtk to install the preloader file supplied with the Orange-Pi supplied Ubuntu distribution.
Update 24th April 2023 — Through a combined effort, we now have ADB running on the 3G-IoT-A board. See the “Linux” section of the “Information” post.
Update 26the April 2023 — Reader [titus m] has posted that his updates to the “scatter” file have created a bootable Debian on the original partition map (meaning that the initial Android install should be fully recoverable and that the all important [and mysterious!] “calibration” data should remain inatact). See his comment in the info post.
In the event that I can muster just a little enthusiasm following this tale of woe, I might try manually scraping the addresses and filenames from that “scatter” file I mentioned earlier and use the info with mtkclient to load each section, one at a time. The worst that can happen is that I lose the ability to play a few bars of music (watch this space, but don’t hold your breath!).
“Bug writ” — Middle English [meaning]: Written by a (software) bug.
esp8266hints 