Author: Synetech

The server has been down for over a year now. What happened was that on May 25, 2021, the Orange Pi’s power-connector had a cold-joint, so the power was flaky and kept causing it to power off and back on. This caused Armbian to reboot several times in a row. This wasn’t new and it always came back up before, but this time, it wouldn’t boot anymore. This time, it said it couldn’t find a valid filesystem. After a lot of consternation and testing, I determined that the ext4 partition had become corrupted. I don’t know wth happened to the journaling function of ext4 that’s supposed to prevent this. 🤨

Anyway, after making a backup and wasting a lot of time trying to analyze and trying to do data-recovery, I fixed the partition table and superblock, and can see some of the items in the root, but there’s a lot of broken directories. Apparently a lot of inodes are somehow broken.

Doing raw searches seems to find most of the directories and of course, most of the files are there, I just need a way to rebuild the directory structure and/or inodes. That is, I need either a way to fix the corruption, or a data-recovery program that can search for directories and rebuild the tree (with user-input if needed).

I’m currently using a backup that was made two years before the crash, so it’s pretty devastating that two years of work (the most productive two years 🤦) are lost. 😕

Any ideas? Can anybody help? Is anybody out there? (Comments can be posted without an account.)

The server is up-to-date on everything. WordPress is up to date, its themes and plugins are all up to date, PHP is the latest version, Pi-hole is current, and all Linux packages are updated.

But what a pain! 😬

One of the biggest problems was updating PHP to 7.4. It’s (still? 🤨) too new, so official repositories still don’t currently support it for ARM/PIs, and even installing from an unofficial repository doesn’t help because some PHP extensions aren’t available. One extension that wasn’t available was imagick and I ended up having to compile it myself. Fortunately (or should I say, miraculously), it actually worked.

I finally took the time to fix the HTTPS redirection (I disabled Really Simple SSL altogether and manually dealt with SSL). After hours of debugging, the solution ended up being to remove the ServerName entry at the top of the Apache configuration (outside of the VirtualHost block). 🤦 I still need to clean out the configuration file some day, but it’s currently much more streamlined than before and working correctly.

Updating WordPress wasn’t hard, but the update to PHP messed some things up and the site-check/health-check function was no longer perfect. I had to jump through a few hoops to change the site and home URLs to be HTTPS, and had to compile imagick to get a perfect status again. Likewise, I had to adjust a few things to restore the security rating in the WIP Security AIO plugin after it fell a bit. I also tweaked and streamlined Autoptimize’s settings, so hopefully it’ll perform better.

Pi-hole was recently updated, but it broke the dashboard. Filing a bug-report didn’t help because they don’t support Apache at all and just dismissed the issue. Fortunately, checking the Apache error log pointed me to the solution: installing the sqlite PHP extension (which for some reason, wasn’t already installed 😕). It’s fixed now, and even better, they’re going to stop requiring nginx in a future version and serve the web-interface from Pi-hole’s own server.

The latest version of WordPress with the latest version of PHP are supposed to provide better performance, but I’m not sure if I’m seeing it. I think it might feel slightly faster, but that might be a placebo effect, or it might actually be slightly faster. 🤷 Either way, I assume the Orange Pi is the bottleneck despite it having more cores than my laptop. 🤔

It took literally all day (~13 hours), but it all worked out in the end. A few days ago, I had a similar day where I spent all day (over 14 hours) working on the server.

I got it set up to auto-update the DDNS when its IP changes. Unfortunately, there’s no way to auto-renew the DDNS; that’s a paid feature, only auto-updating the IP is possible. They won’t even increase the expiration to 31 days instead of 30 days so that it can be possible to renew on the first of the month rather than having to choose between the renewal date drifting over time or having the site down for a day every other month. I tried explaining this to them, but they just said I could pay for a subscription. 😒 I need to get an actual domain…

I also fixed AJAX queries in the web-server (the solution was to prevent caching), this was really annoying but it’s working now—I just have to remember to prevent caching whenever I use AJAX in the future. I fixed the ServerFileEditor, but it turns out this isn’t even necessary anymore because the NppFTP plugin for Notepad++ lets me edit files directly on the server like they’re local files. 👍 I fixed my Identity Survey. I also updated my sorted-table plugin that’s used in some list posts.

There have been two hyper-productive days of work on the server in the past week. It’s pretty much set and as good as I can make it. Now I just have to wait a week to see if LetsEncrypt will auto-renew the SSL certificate like it used to or if it’s still broken. 🤞

WordPress 5.3 came out a couple of months ago and PHP 7.4 came out last month, but the server won’t be getting updated to those just yet. It’s always safer to defer updating things for a while since updates often have unintended problems (Microsoft has certainly proven that with Windows 10 in the past few years). Since developers don’t seem to bother doing testing anymore, it’s better to wait until other people play guinea-pig before upgrading. (Also, PHP 7.4 doesnt’ seem to be available in the Pi/Buster repository yet and I”m not inclined to use third-party repositories anymore.)

The server is (more or less) stable right now and changing things almost always reduces that, so as they say, if it ain’t broke, don’t fix it.

The server was getting very unstable recently; it would drop the WiFi connection frequently and boot-loop and was just getting overloaded (it’s not just Windows that gets “Winrot”).

I was worried it was a hardware problem, but before getting a replacement, I held out hope that it was just software. I figured I’d “kill three birds” with one install, and both stabilize it and upgrade to Buster by starting over from scratch. That meant getting a list of installed packages, and compiling a list of configuration files and settings to be reset, which was a pain, but still easier than the daunting, Herculean task of reinstalling Windows 7 (or *bite my tongue!* upgrading to Windows 10). The third bird was to swap out the microSD card the Orange Pi was running from with a new higher-speed class A1 card which was conveniently on sale at Best Buy at the time.

Burning a copy of Armbian Buster to the card was quick (the card was rated for 130MBps read, but didn’t mention its write speed, but it must be at least decent 🤷). The Pi booted from it just fine and the WiFi connection was stable. I began installing the packages, but decided to keep it lean and install only the ones necessary to run the server; all the other stuff was ditched since I wouldn’t be using them anyway.

Unfortunately, things started going south. The WiFi connection started having problems, the packages started getting messed up, and migrating the settings wasn’t going smoothly.

I started over again. (And wasted a bunch of write-cycles on the memory-card. :-\)

I re-burned the Armbian image to the card, then took it slow. I tried to reboot after each step and leave it for a bit and check if things fail. I also trimmed down what I migrated from the backed-up installation, and minimized the changes from the base install.

It took the entire weekend (according to my daily log, about 25.5 hours!), but in the end, the server was up and running on Buster, smaller, faster, and more stable than before. 👍

Now I only need to wait for a while until I can convince myself to delete the backup and free up a few gigabytes.

The blogs have all been changed to colorlib’s clean and customizable Sparkling theme for consistency. (Almost all of) the colors are set to stay consistent with each blog’s previous identity.

The web-server is now portable. 😁 When not on the go, it can tolerate power-outages. 👍

I’ll probably make a post (on the Synetech blog) on how to make a homemade battery-backup/UPS for the Orange/Raspberry Pi.

After a long hiatus, the blogs are back thanks to an Orange Pi Lite.

The blogs have officially been migrated from Google’s Blogger service to our own WordPress installation.

Let the blogging continue…

(There are links to the various blogs on each blog’s page.)