by alex on Networking

“easy” failover home network access..

I just came back from a trip to Singapore and Malaysia. While in Malaysia, we stayed at the Legoland Resort hotel, which was actually pretty cool.

What wasn’t cool, was that its provided wifi blocks outbound TCP except to a couple of well known protocols (http, https, pop3, pop3s, imap, imaps, smtp), and well, my wife and I were nerds and we were working on something on one of our home Linux boxes and we needed to ssh into it, over our wireguard VPN.

So I did what any reasonable person would do… got a friend to give me a temporary account on their box which runs proxied sshd on the same port as https, and then used that setup a reverse tunnel and SSH into our home network.

.. where I then setup an OpenVPN server using a shared port with our home server’s https connection, which is forwarded through the NAT router, and connected to that to finish off what we were doing.

I then wanted to solve another problem – we have failover 4G, for the rare times when the main NBN connection is down. Except, my NBN provider gives me a real IP, and my 4G provider uses CGNAT, so I can’t use the existing home VPN server when the main NBN link is down – not that this has ever come about, but “be prepared”, as the boy scounts say.

I didn’t want an always on connection from my VPS server in California to my home network, so I setup sshd on a second port on my VPS server, and used autossh to establish a tunnel from the VPN server on my home network on that port.. and then used ufw on the VPS to block my main ISP, so it can only connect if the connection fails over to the 4G link.

So for home access – normally I can use OpenVPN or Wireguard (I have both now) directly to my home static IP, or if the NBN is down, the home will tunnel out and expose OpenVPN via my VPS. The functionality is good, it can be explained easily, but the behind the scenes implementation is a little complex, though I think pretty cool.

by alex on Storage

Dead men don’t tell tales, and neither do crushed hard drives

I’ve had this conversation more than a few times on reddit and with people who don’t work in the storage industry. “Why are they destroying perfectly good hard drives/iPads/etc”

  • I suppose standard disclaimer should apply here – I work for a company which sells storage systems (HDD+SSD), and this is all my own opinion and does not reflect the opinions of my company. One might think that because a company like the one I work for sells hard drives, they would want to remove secondhand ones from the market in order to sell more. This naïve view doesn’t account that the financial model enterprises storage systems are sold under is weighted more towards licensing than hardware, I also don’t get paid based on sales, so I assure you, there isn’t any bias here.

Anyway.. the most recent discussion on this topic from reddit brought this article from IEEE Computer to my attention. This article correctly states that there are benefits to enabling a circular economy for storage devices, and that many storage devices can be cleared, or at the very least purged, of confidential data, to enable secure reuse.

  • I’ll note that one of the contributing authors is part of the Chia project, which uses the blockchain and huge amounts of storage to enable.. I dunno, commerce or something. Chia is not without criticism, that while it claims it’s greener than bitcoin, it still involves spinning rust, which is not super cheap. I don’t actually take a position on the Chia project, other than to say that encouraging re-use and circular economy for data storage isn’t actually a bad thing, but that I think if I were part of the project and writing an article about why people spend the time and effort to make cheap drives available, I might consider declaring a conflict of interest.

So here’s the thing – you can absolutely clear data from drives, in some cases, very quickly, in other cases, more slowly (“purge” from the article). But you can make it gone, no doubt about it.

And then what happens? You put a sticker on it, and put it in a pile. How many drives did you have attached to your wiping station at the time? Did you put a sticker on the right drive though? Drive sits around for a while and maybe the sticker falls off. You have a blank drive sitting around, did it get cleared? Well I’ll just wander over and check the device serial number against my log of cleared/purged drives. 5 minutes later, you’ve found it, and the drive’s probably good to go. Then you find another one.. did they use bad adhesive on this batch of stickers? or did it get too hot or too cold. Oh well, it’s probably fine, no one misread your sign and put an uncleared drive in the cleared pile. And no one grabbed a drive from that pile for another project and put it back in there again after they were done, Right?

Or maybe you’re lucky, maybe you have some secure cages and you can make sure that no one other than you moves drives in or out of the cage. But IT is usually kept in the basement for a reason – space is at a premium, and no one wants you to have more than you absolutely need.

Off they go to the drive reseller who’s paid you $20-50 each for them. It’s taken a couple of hours to unrack, attach, clear/purge the drives, label them, arrange the sale, which has taken you away from your core job tasks. “Earned $2000 by selling hard drives” is your end of year achievement.

But.. oops, no, can’t put that down. You missed a step and someone found an uncleared drive that they could recover data from. Well oops. Oops again. Our bad.

Suddenly your organisation is on the hook for regulatory fines, or legal settlements, gets bad press, your auditors are breathing down your neck asking “why did you think this would be ok?!?” and then your $2000 for the IT slush fund doesn’t look like such a great trade.

What’s the answer then? Crush the drives. Make it instantly visible that the storage devices leaving your premises do not have data on them.

So, sorry. I take the view that while it’s technically quite possible to erase data from storage devices, that it is unfortunately still irresponsible from a risk management point of view to then allow any drives out of your control, especially since the financial benefits of selling them is so small. The bigger the organisation, the more likely procedures are to break down, and the more likely you are to have a data spill.

My commentary here is full of strawmen, I totally grant that. But to save your company from risk, you need to follow the logical data clearance 100% of the time, and it’s not simple to tell if it has been done, and that the consequences of even a single slip can be catastrophic. Crush the drives.

by alex on proxmox, Virtualization, vsphere

Creating nested ESXi inside Proxmox

Ok, sounds a little silly, but there are times when you might want to have a nested ESX VM inside your proxmox environment – I found this great page on how to do it, but I’d rather do it via CLI so it’s repeatable.

This is the CLI I found which worked for me:

qm create 703 --balloon "0" --boot "order=sata0;ide2;net0" --cores "2" --cpu "host" --ide2 "ISOs:iso/VMware-VMvisor-Installer-201912001-15160138.x86_64.iso,media=cdrom,size=343064K" --machine "q35" --memory "8192" --name "pveESX3" --net0 "vmxnet3,bridge=vmbr0" --numa "1" --onboot "1" --ostype "l26" --sata0 "nvmeLVMlocal:32,backup=0,discard=on,ssd=1"

In the lead up to that, I also found a page on stackoverflow asking if there was an easy way to take the output of “qm config” and turn it into “qm create”. It doesn’t seem like there is exactly that – “qm showcmd” as recommended on there isn’t terrible, but isn’t enough.

So this is what I came up with – unfortunately it only gets 90% of the way there – 

qm config 703 | grep -v 'vmgenid' | grep -v 'smbios1' | grep -v 'meta' | sed -e 's/vmxnet3=.*,/vmxnet3,/g' -e 's/://' | awk '{ printf("--%s \"%s\" ",$1,$2);} END {print("\n");}'

The issue is that it outputs “nvmeLVMlocal:vm-701-disk-0,backup=0,discard=on,size=32G,ssd=1”, but really it needs to be “nvmeLVMlocal:32,backup=0,discard=on,ssd=1” (replace 32 with desired size).

But it’s better than nothing.

Hope this helps someone one day!

by alex on proxmox, Storage, Uncategorized, Virtualization

Setting up a LVM LV quickly..

I’m playing with proxmox right now, and had a need to setup LVM locally on each node. I figure why not script it.. I’ve ended up with this abomination of a script. You probably want to do better, but it’s a start:

set -x;
lvmid=`hostname`_localLVM; 
pvcreate /dev/sda; 
pvs; 
vgcreate vg_$lvmid /dev/sda; 
vgs; l
vcreate -n lv_$lvmid -l 100%FREE vg_$lvmid;
lvs; 
mkdir -p /local/vg_$lvmid-lv_$lvmid; 
mkfs.ext4 /dev/mapper/vg_$lvmid-lv_$lvmid;
echo /dev/mapper/vg_$lvmid-lv_$lvmid /local/vg_$lvmid-lv_$lvmid ext4 defaults 0 0 >> /etc/fstab; 
mount /local/vg_$lvmid-lv_$lvmid; 
df -h

Or on one line..

set -x;lvmid=`hostname`_localLVM; pvcreate /dev/sda; pvs; vgcreate vg_$lvmid /dev/sda; vgs; lvcreate -n lv_$lvmid -l 100%FREE vg_$lvmid;lvs; mkdir -p /local/vg_$lvmid-lv_$lvmid; mkfs.ext4 /dev/mapper/vg_$lvmid-lv_$lvmid;echo /dev/mapper/vg_$lvmid-lv_$lvmid /local/vg_$lvmid-lv_$lvmid ext4 defaults 0 0 >> /etc/fstab; mount /local/vg_$lvmid-lv_$lvmid; df -h

Hope this helps someone in the future.. maybe me!

by alex on Hardware, Netapp

NetApp H610S Fan speed too high – controlling with IPMI

I was chatting with someone on the NetApp Discord today – they’d just bought a used NetApp H610S aka NAF-1703 aka Quanta D52B-1U, and installed Proxmox on it, but the fans were running too high. This was a WAF issue, and they were looking for a way to calm it down.

We did some digging and found this document on the IPMI commands for a similar platform – but they didn’t work on this one.

Some digging around suggested that they needed to use

ipmitool raw 0x30 0x39 0x00 0x00

instead of

ipmitool raw 0x30 0x39 0x01 0x00

What’s the difference? Who knows, but it worked.

Hope this helps someone!

by alex on Netapp, Storage

7 Mode? In this economy?

I had two discussions about Data ONTAP 7-mode last week, which was a bit of a surprise, since it’s been something NetApp has been working to help customers get away from for.. some time now. 8 years really pushing it, 10 years since NetApp started providing Clustered ONTAP as an option.

You can totally understand it – data has GRAVITY. It’s heavy and hard to move. Those moves and cutovers need to be as seamless, or quick (or ideally both) as possible. And 7DOT was a platform people had a lot of experience with and understood, and change is difficult.

I’ve been in videos and given countless presentations how to do 7toC migrations quickly and easily, and done a LOT of them, either personally, or working with customers, but the end result is that some people haven’t done it, and it’s now 2023, and the remaining 7DOT users find themselves in a tough spot.

Last November, Microsoft made some AD changes, which means that to continue using 8.2.5P5 with Active Directory, you need to re-enable RC4 encryption. RC4 is.. not terribly secure, so I wouldn’t do that.

At the beginning of Feb 2023, NetApp stopped supporting the FAS255x and FAS80x0 controllers, which are the last generation to run 8.2.5, the last release of 7DOT, which itself is now in “self support”. Self support means NetApp won’t delete the webpages which help with troubleshooting. But once they’re gone in Jan 2026 (less than three years away.. and it can’t come soon enough), you’ll be stuck with some random university in Wollongong hosting an old copy..

ONTAP does everything 7DOT did except allow direct FC connections (because it requires NPIV for FC LIF hosting) and providing an FTP server. The first is an easy fix (buy an FC switch.. if you’re still running 7DOT, you’re probably not adverse to eBay purchases of infrastructure) and the second is a matter of setting up a small Linux VM somewhere if it’s a really big concern.

The best time to migrate off 7DOT was 2016. The second best time is now.

by alex on Uncategorized

Setting up a PowerBook G4 12 inch, 1.5Ghz in 2020

Some time ago, I received a PowerBook G4 12 inch from a friend. As is healthy, the drive had been wiped, but being one to not keep too much old stuff, I didn’t have install media for Leopard (10.5). It went on the backburner for a while, but I recently received (back) some old storage devices which had been on ice in the WA Wheatbelt for about 10 years, and felt I had the right stuff together to give it a go again.

  1. Downloaded 10.5.4 Leopard Installer from Archive.org
  2. Attach 30GB PATA drive by PATA to USB2.0 dongle
  3. Using 10.15 Catalina, partition drive with Apple Partition Map to 10GB+20GB partition
  4. Mount Leopard Installer ISO previously download
  5. Use ASR to restore contents of ISO to 10GB partition
bash-3.2$ sudo asr restore --source /Volumes/Mac\ OS\ X\ Install\ DVD/ --target /Volumes/Emptied/ --erase
	Validating target...done
	Validating source...done
	Erase contents of /dev/disk3s3 (/Volumes/Emptied)? [ny]: y
	Validating sizes...done
	Restoring  ....10....20....30....40....50....60....70....80....90....100
	Verifying  ....10....20....30....40....50....60....70....80....90....100
	Restored target device is /dev/disk3s3.
	Remounting target volume...done
  1. Try to boot Powerbook using PATA drive on USB dongle, failed
  2. Move PATA drive from USB dongle to Sarotech Cutie FireWire caddy (originally purchased in Tokyo, 17 years ago), success
  3. Install Leopard
  4. Reboot, install 10.5.8 combo update
  5. Reboot
  6. All works! Yay
by alex on Virtualization, vsphere

A bit of vmware fun for a change!

I’ve got a VMware VCP – Have done for about 8 years (passed the exam 4 times now..), but most of my day these days is dealing with storage – but I’ve had a family fixit to migrate a 17 year old laptop into a VM on a more recent mac.

Nice and easy I sez! Just use the converter! Nekt minit..

vmware error.PNG

For the keyword lulz of anyone searching for this problem: “Error 1920.Service VMware vCenter Convert Standalone Server (vmware-converter-server) failed to start. Verify that you have sufficient privileges to start system services”

First I tried the obvious – checking the local user was an Admin, running it as administrator, trying vmware’s fix of creating a group called “Domain Admins”.. all with no dice.

Then I found someone suggesting how to start the agent manually, and when I did that, it complained that the certificate could not be verified.. which lead me on another path, checking Properties in Windows, which lead me to this KB entry on how to install the code signing certificate root on Windows XP.. which lead to a KB entry on another site, which led to 404, which led to web.archive.org, saving the file as a PEM, adding the Certification MMC snapin, importing it, and finally, the service started up.

Nice and simple.. only took someone with 19 years of experience with VMware and as a professional infrastructure admin an hour to fix it..

Edit: Oh ho, but there’s more! VMware Converter then was unable to send the image to the Mac – under the hood it’s still the same old converter that has been around for years, which means that it saves the image over SMB to the Mac. Except, Windows XP can’t talk to Catalina by default. Some will suggest upgrading to SP3 (a good idea, but I want to make minimal changes to this system image..), but that isn’t necessary – as outlined at this post, all you need to do is set HKLM\System\CurrentControlSet\Control\Lsa\lmcompatibility level to 3, from the default of 1 on SP3 or 0 on SP2

by alex on Uncategorized

Quickly convert HEIC to PDF for expenses submission

Recently I had to convert a large number of photographs taken on my iPhone into PDF for submission with my expense report. I took to my old faithful ImageMagick (installed via HomeBrew) and its mogrify command:

mogrify -resize 50% -format pdf -level 0%,100%,2.0 -type Grayscale -compress jpeg *.HEIC

Hope this helps!

by alex on Netapp, Storage

How to wipe a partitioned ADP NetApp system

With ONTAP 9, there is now an “option 9” in the boot menu that allows you to re-initialise a system to/from ADP, like wipeconfig.

It is a three step process to wipe an HA pair – the first one, option 9a –  removes the existing partition information. And the second, option 9b, will repartition and re-initialise the system, and then finally on the node that was halted, boot it, then wipe it (option 4) from its boot menu.

*************************************************
* Advanced Drive Partitioning Boot Menu Options *
*************************************************
(9a) Unpartition disks and remove their ownership information.
(9b) Clean configuration and initialize node with partitioned disks.
(9c) Clean configuration and initialize node with whole disks.
(9d) Reboot the node.
(9e) Return to main boot menu.

The caveat is that one node has to be halted at the LOADER> prompt while you run the first two commands. That should be it!