Category Archives: English

A deep-dive into UEFI Booting

[According to my own standards, this post about “UEFI booting” was only like 70% “ready” — I had it pending in “draft” state for many months, because I was lacking the time to finish it… I now decided to release it in its current state, simply because I believe it will still be very useful to many people interested in the topic…]

Before we actually dive deep into how UEFI booting works, a short and simple introduction is due.

Introduction

What is UEFI, anyway?

UEFI could be called the successor of the old BIOS concept. It is a unified version and successor of “EFI”, which was an architecture for a platform firmware used to boot operating systems (in the following abbreviated as “OS”), and the corresponding interface to interact with the firmware and the operating system.

The advantages of UEFI over the traditional BIOS are, among others, the following:

  • Boot disks with large partitions (over 2 TB), using GUID partitioning (GPT),
  • network capabilities already in pre-OS phase, and
  • modular design.

Boot Mechanism

So, how does booting with UEFI work?

When you enter your UEFI, you will find a user interface that shows all devices that have been detected, and that support booting. Usually that includes all your hard drives.

You can freely define a desired boot order (regardless of hardware paths, i.e. the way your drives are connected, be it via an SATA port, be it via an NVME slot), i. e. the primary OS that should be booted, if that fails the next OS that should be tried, etc. That’s called your “boot configuration”, and it resides in your motherboard’s NVRAM. (a concept that’s basically the successor of what used to be called “CMOS” in the old days of the BIOS). More specifically, we speak about “UEFI Variables“, which allow the OS and the firmware to interact.

When UEFI’s Firmware Boot Manager wants to boot an OS, it first needs to load something called the OS “Boot Manager.” Common OS boot managers are:

  • BOOTMGFW.EFI used to load Windows, or
  • SHIMX64.EFI used to load Linux

The OS boot manager is located on the “EFI system partition” (ESP). This is a small partition (usually only a few 100 M) at the start of your hard drive, formatted with basically a FAT filesystem. FAT is a very simple filesystem, so that the code to parse it and load files from it can be reasonably small and fit into a boot firmware.

A typical disk layout for a Windows installation may look as follows:

UEFI /EFI System Partition as seen under Windows

The first partition is the ESP, then comes the Windows boot and system drive (with a drive letter of C:), and then comes the recovery partition.

Apart from boot loaders, the ESP can contain kernel images or (device) drivers, e. g. to support hardware that must be initialized prior to the start of the OS, or to give access to a complex filesystem that holds the actual OS to be booted.

Depending on which OS you want to boot, the OS boot manager then loads

  • in the case of Linux: the OS kernel, and the kernel in turn loads the OS, or
  • in the case of Windows: the Windows Boot Loader (\Windows\system32\winload.efi)

Boot Configuration Details

Now that we got a good overview of the mechanism as a whole, let’s dive into the details. Let’s look at the boot configuration of my machine. To do so, invoke the below command (I did it under Ubuntu 23.04, but it should work the same under any reasonably current Linux distro where the tool is installed):

# efibootmgr -v
BootCurrent: 0003
Timeout: 1 seconds
BootOrder: 0000,0002,0003
Boot0000* Windows Boot Manager    HD(1,GPT,a39d5736-7aaf-4be0-b6b6-0852ba2f7803,0x800,0x32000)/File(\EFI\MICROSOFT\BOOT\BOOTMGFW.EFI)WINDOWS………x…B.C.D.O.B.J.E.C.T.=.{.9.d.e.a.8.6.2.c.-.5.c.d.d.-.4.e.7.0.-.a.c.c.1.-.f.3.2.b.3.4.4.d.4.7.9.5.}…R…………….
Boot0002* ubuntu    HD(1,GPT,280ea55d-c182-5242-bb52-a2b40812190c,0x800,0x219800)/File(\EFI\UBUNTU\SHIMX64.EFI)..BO
Boot0003* ubuntu    HD(1,GPT,91b56f9f-3526-404b-b681-1c684551ec4f,0x800,0xd87be)/File(\EFI\UBUNTU\SHIMX64.EFI)..BO

So, what does the above tell us? First, we see that “(Boot)0003” is the boot entry used to start the currently running system. Secondly, the order in which boot is tried is 0000, 0002, and then 0003. So by default, Windows (by the “Windows Boot Manager”) will be booted. Then we see the three boot entries. The star/asterisk (*) after the boot entry shows that all these entries are “active.”

What about the remaining info in the above command output? Immediately after the boot entry, we see the names that are also displayed on screen by the Firmware Boot Manager (“Windows Boot Manager” and two times “ubuntu”). We then see references to the ESPs used to boot these OS.

HD obviously means “hard drive”, then we see a 1 which refers to the first partition on the respective drive, then we see GPT which refers to the partitioning table format, and then we see a UUID. To find the respective partitions, we can use the below command:

# blkid --match-token TYPE=vfat
/dev/nvme0n1p1: LABEL="UBUNTU_TEMP" UUID="EC76-8E7F" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="280ea55d-c182-5242-bb52-a2b40812190c"
/dev/nvme1n1p1: SEC_TYPE="msdos" LABEL_FATBOOT="UBUNTU_MAIN" LABEL="UBUNTU_MAIN" UUID="8140-E4C0" BLOCK_SIZE="512" TYPE="vfat" PARTLABEL="EFI" PARTUUID="91b56f9f-3526-404b-b681-1c684551ec4f"
/dev/sda1: UUID="40AD-1127" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="a39d5736-7aaf-4be0-b6b6-0852ba2f7803"

The PARTUUID values in the above output match the UUIDs in the boot configuration as shown by efibootmgr. So, the ESPs are located by searching for the partitions’ UUIDs. That means that you can replug your drives to different ports, or even copy partitions to different drives, and the UEFI boot mechanism will still find them. That’s a nice and very stable design.

UEFI User Interface

Now, let’s enter the UEFI and look at some of the details there. My PC’s motherboard is an MSI, and to enter the UEFI I need to press “F2” after the beep when powering on the PC (from “off” state, not when suspended to RAM, i.e. “sleeping”!) or restarting it.

# lspci
[...]
21:00.0 Non-Volatile memory controller: Sandisk Corp WD Blue SN570 NVMe SSD 1TB
2b:00.0 Non-Volatile memory controller: Sandisk Corp WD Blue SN570 NVMe SSD 1TB

“Excire Search 1.4” Review

This is a quick-and-dirty review of the Excire Search 1.4.1 plugin for Adobe Lightroom. I got this plugin for free in an online “advent calendar”. As it was free, I can’t expect anything, obviously. But still I can tell you what I think about the tool. 😉

Let me mention that version 1.4 is not the latest version as of today. So the more recent 2.0 version may be performing much better than the one I tested, but I have no way of verifying this.

So, what is this tool about? Excire Search is a photo organizing software that sifts thru your photos in order to organize them. The company claims Excire Search is “The best software for organizing your photo treasures”, using AI it analyzes and tags photos automatically so that “image management [is] fun again.”

Ok, let’s see how helpful the tool actually is. One category that Excire creates in your keyword hierarchy is “Image Color.” So if you are searching for images of a certain color, this should be helpful, right? Well, have a look yourself. This is the hierarchy it created for some 15K photos I have in this Lightroom catalog:

“Image Color” keywords

Let’s have a look at some of the images Excire classifies as “black”:

Continue reading “Excire Search 1.4” Review

How to manually rebuild your Windows BCD Store

I recently had a mishap where on a system that I had just upgraded from Windows 10 to Windows 11 the Windows BCD Store (Boot Configuration Data Store) was suddenly messed up, presumably by a patch update I had taken earlier:

Blue screen showing “Wiederherstellung: Der PC muss repariert werden.” (German for: “Recovery: The PC must be repaired.”), hinting to a problem with the Windows BCD Store.

Many people will recommend in that situation to boot to a Windows Recovery Environment (RE) and execute the following:

>bootrec /rebuildbcd

But what if the scan for all Windows installation turns out not to identify yours?! Like in the below photo?

Photo of a Windows RE session, failing to automatically rebuild the Windows BCD Store

Are you SOL then?

Don’t despair.

The solution is actually quite simple. Other than the above which is supposed to be fully automatic, it involves some manual tasks, but it should be quite easy if you are at least a bit tech-savvy.

Continue reading How to manually rebuild your Windows BCD Store

Flashing Android Firmware Builds using Android Flash Tool

As a long-time Android user (since the days of Android 2.0, if I remember correctly), I have seen many things already, especially as I was involved in the development of mobile phones, and also fiddled a lot with CyanogenMod, LineageOS, and other custom ROMs. Still, I recently discovered something new for myself, and this is something I really find spectacular. It’s a new mechanism to flash Android firmware builds, and it’s called “Android Flash Tool” (AFT).

Let me elaborate…

Usually, to reflash a device you need to install a program on your workstation, like a vendor flashing tool, or the Android Debug Bridge (adb) command-line tool. With the Android Flash Tool, this is no longer required. The only thing you need is your WebUSB-capable (browser). 🙂 Currently, only Chrome or Edge support it. (Firefox currently doesn’t support it due to security concerns.)

Also, the number of devices which support the AFT is very, very limited. Basically, it’s just the Google Pixel series of phones, plus a few development boards.

But if you have one of those devices, it’s pretty cool.

Continue reading Flashing Android Firmware Builds using Android Flash Tool

Netgear GS108Ev3 – Huge packet loss when pinging management IP

I have a quite complex infrastructure in my house, and I monitor everything.  Among others, I have SmokePing probes towards multiple destinations in my LAN, and also to my internet provider’s infrastructure, and even external hosts.

When I added a graph for the management IP of my Netgear GS108Ev3 8 Port Gigabit Ethernet Smart Managed Plus Switch, which is my “core” switch, I noticed that there is a huge packet loss, in the range of 4% average, about 40% max.

The strange thing is that other destinations in my LAN which can be reached via this switch, have absolutely zero packet loss. That means that the cabling itself is definitely ok, and also that the packet switching functionality of the switch is ok.

So I opened a ticket towards NetGear, and the response I got was quite surprising (but still somehow makes sense):

This is a known issue, and it is by design. The put the highest priority on the packets flowing thru the switch, while the management traffic has a lower priority, so that ICMP packets to the switch’s management IP may get lost, and in extreme cases the web UI may not be reachable at all.

It’s not a big issue for me, because in the end I wanted to create graphs that exhibit whether I have any internal packet loss. Whether the switch’s management interface has packet loss or not is not really important to me. As long as all my internal devices that I actively use, like my NAS devices or streaming boxes, are properly connected with no packet loss, all is well. 🙂

Google Fi Activation fails with Error Code B050

If you get an error code of “B050” when trying to activate your “Google Fi” service, it is because you are identified as not being in the US right now. This was confirmed to my by Google Support on the phone a few minutes ago.

Whether this is being identified by your IP address or your phone’s location, I can’t tell (yet). Could be that I bring up a VPN to check… 😉

2020-02-19 Update: I was just too curious, so only a day later I quickly wrapped up a VPN with a source IP from the US (thanks, AWS! :-)). And, presto, I could activate Google Fi. So it is crystal clear that they identify you by your IP address.

I could make calls via VoWiFi, but when I disabled WiFi I couldn’t attach to the mobile network. This is probably because I’ve never joined my “home (mobile) network” yet. I’ll be in Florida soon, then I will join the mobile network there. And I’m sure when I return to Germany I’ll be able to join German mobile networks as well…

Use SmartThings Multi-Purpose Sensor without Magnet

While browsing around in the SmartThings IDE I stumbled across a setting for the multi-purpose sensor called garageSensor.

Googling it turned out that you can manually put the sensor into a mode where it detects it has been tilted, and then reports this state as “open.” This can be used for sectional garage doors, in which case you don’t have to use the magnet, but just the main “sensor” part.

Mount the garage door sensor vertically so that when the garage door is vertical, it will read “closed.” When horizontal, it will read “open.”

To put the sensor into the “garage sensor” mode, open the device in the IDE, and then click on the “edit” link for the device preferences:

This could possibly also be used for windows which can only be tilted — it wouldn’t make much sense to use this setup for windows which can also be “turned” open (like most European-style windows).

This helped? Then please let me know… 🙂

Huawei P8 (GRA-L09) modding

This post about “modding” of a Huawei P8 aims to summarize most of the standard procedures you need when you want to modify the device software (also called “firmware”), because you are a developer or want to be more flexible in how you use your device.

Normally, in order to gain this knowledge, you have to read a lot of “geeky” posts on sites like xda developers, which is very time consuming, and sometimes even leads to complete failure (you end up “bricking” your device), because many people there don’t bother to post clear and comprehensive instructions, but simply assume people have some pre-knowledge and know how to do things.

I don’t yet consider this post “finalized” yet (it needs some “polishing”), but I would like to make it available already now to make sure it won’t be forgotten… 😉

Fastboot Mode

“Fastboot” mode is a special mode your device can be put into in order to perform certain maintenance operations, such as (un-) locking your boot loader, flashing partition images, etc. The tool you use on your PC or Mac to communicate with the phone while it is in “fastboot” mode is also called fastboot. It is part of the Android platform tools which you can download here directly from Google.

Continue reading Huawei P8 (GRA-L09) modding

ntp running in chroot stopped working after Debian Stretch upgrade

Today I upgraded my root server from Jessie to Stretch, and suddenly ntp stopped working.

I found errors like follows in the log, which were obviously due to failures in name resolution:

2018-05-31T07:44:48.900756+00:00 myhost ntpd[22855]: giving up resolving host 1.debian.pool.ntp.org: Servname not supported for ai_socktype (-8)

The solution to make this work was to bind-mount some files and directories essential for name resolution into the chroot jail.

First create some directories and some dummy files:

# mkdir /var/lib/ntp/etc /var/lib/ntp/lib /var/lib/ntp/proc
# mkdir /var/lib/ntp/usr /var/lib/ntp/usr/lib
# touch /var/lib/ntp/etc/resolv.conf /var/lib/ntp/etc/services

Then update your /etc/fstab as follows:

...
#ntpd chroot mounts
/etc/resolv.conf  /var/lib/ntp/etc/resolv.conf none bind 0 0
/etc/services	  /var/lib/ntp/etc/services none bind 0 0
/lib		  /var/lib/ntp/lib none bind 0 0
/usr/lib	  /var/lib/ntp/usr/lib none bind 0 0
/proc		  /var/lib/ntp/proc none bind 0 0

Finally mount all these binds:

# mount -a

Thanks to the ArchLinux guys where I found this.

To check whether your ntp is working again, you can use the following command which shows a list of peers known to your ntp server:

# ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 0.debian.pool.n .POOL.          16 p    -   64    0    0.000    0.000   0.000
 1.debian.pool.n .POOL.          16 p    -   64    0    0.000    0.000   0.000
 2.debian.pool.n .POOL.          16 p    -   64    0    0.000    0.000   0.000
 3.debian.pool.n .POOL.          16 p    -   64    0    0.000    0.000   0.000
*ptbtime1.ptb.de .PTB.            1 u   46   64  377   11.483   -0.411   0.201
+ptbtime2.ptb.de .PTB.            1 u   52   64  377   11.502   -0.533   1.069
+ptbtime3.ptb.de .PTB.            1 u   47   64  377   11.451   -0.510   3.866
#batleth.sapient 131.188.3.221    2 u   44   64  377    0.188    1.097   0.176
#5.199.135.170 ( 130.149.17.21    2 u   45   64  377   11.271    0.581   0.396
-mail.morbitzer. 193.175.73.151   2 u   47   64  377    2.760    0.556   0.278
#hotel.zq1.de    161.62.157.173   3 u   46   64  377    0.094    1.384   0.261
-ntp2.m-online.n 212.18.1.106     2 u   47   64  377    7.167   -0.333   0.190
#2a03:b0c0:3:d0: 81.94.123.16     3 u   49   64  377    6.288   -2.071   1.760
#touka.thehomeof 130.149.17.21    2 u   48   64  377    0.206    0.932   0.222
+maggo.info      124.216.164.14   2 u   42   64  377    0.278   -0.137   0.436
+weyoun4.cord.de 124.216.164.14   2 u   44   64  377    2.849   -0.255   0.409
+opnsense.sauff. 222.217.153.8    2 u   43   64  377    0.270   -0.617   0.167
-web1.sys.ccs-ba 192.53.103.104   2 u   35   64  377    0.173   -1.251   0.220
#uruz.org        122.227.206.195  3 u   49   64  377    0.216    1.694   0.309
#clients5.arcani 162.23.41.55     2 u   38   64  377    6.120   -1.500   0.130
+stratum2-1.NTP. 129.70.130.71    2 u   47   64  377   14.043    1.625   0.394

The following command confirms that your current time is actually correct (within certain limits, of course):

# ntpstat
synchronised to NTP server (192.53.103.108) at stratum 2
   time correct to within 15 ms
   polling server every 64 s

If this was helpful, I would be happy to hear from you.

Exim malware scanner issue after upgrade from Jessie to Stretch

Today I finally upgraded by personal root server from Debian Jessie to Stretch, thereby upgrading Exim from 4.84 to 4.89.

After the upgrade, I observed the following errors in mainlog:

2018-05-31 08:02:03 +0000 1fOIX5-0001rg-AM malware acl condition: cmdline  : scanner returned error code: 36096
2018-05-31 08:02:03 +0000 1fOIX5-0001rg-AM H=([IPv6:2a00:6020:1efc:ee20:8857:7824:6a49:8368]) [2a00:6020:1efc:ee20:8857:7824:6a49:8368]:48523 I=[2a01:4f8:141:429::2]:465 Warning: ACL "warn" statement skipped: condition test deferred
2018-05-31 08:02:04 +0000 1fOIX5-0001rg-AM malware acl condition: cmdline  : scanner returned error code: 13
2018-05-31 08:02:04 +0000 1fOIX5-0001rg-AM H=([IPv6:2a00:6020:1efc:ee20:8857:7824:6a49:8368]) [2a00:6020:1efc:ee20:8857:7824:6a49:8368]:48523 I=[2a01:4f8:141:429::2]:465 Warning: ACL "warn" statement skipped: condition test deferred
2018-05-31 08:02:05 +0000 1fOIX5-0001rg-AM malware acl condition: cmdline  : scanner returned error code: 13
2018-05-31 08:02:05 +0000 1fOIX5-0001rg-AM H=([IPv6:2a00:6020:1efc:ee20:8857:7824:6a49:8368]) [2a00:6020:1efc:ee20:8857:7824:6a49:8368]:48523 I=[2a01:4f8:141:429::2]:465 Warning: ACL "warn" statement skipped: condition test deferred

Each of the three cmdline scanners caused an error, as shown above.

It seems there was a change in Exim from upstream, as reported by another user. Somehow it seems that if you define a cmdline scanner that uses a chain of commands, when there was an error return code encountered in the middle of the chain, the whole chain is considered failed.

To “fix” this issue (or rather work-around it), I changed the three ACL clauses as follows:

   warn  message                = This message contains malware ($malware_name)
         set acl_m0      = cmdline:\
-                               /usr/lib/AntiVir/guard/avscan -s --batch --scan-mode=all %s; /bin/echo -e \N"\navira_retval $?"\N:\
+                               /usr/local/bin/avscan_wrapper %s:\
                                \N^avira_retval 1$\N:\
                                \N^.*ALERT::[ \t]+([^;]*)[ \t]+;.*$\N
         malware                = *

I created a “wrapper” that effectively hides error return codes, and forces a return code of 0. The above wrapper looks like this:

#!/bin/bash

ARG="$1"

/usr/lib/AntiVir/guard/avscan -s --batch --scan-mode=all "${ARG}"
/bin/echo -e "\navira_retval $?"

exit 0

To make sure I didn’t break the malware scanning by my changes, I downloaded the EICAR test virus and sent it to myself. Exim caught the “virus” and ditched it.