Dead men don’t tell tales, and neither do crushed hard drives

I’ve had this conversation more than a few times on reddit and with people who don’t work in the storage industry. “Why are they destroying perfectly good hard drives/iPads/etc”

  • I suppose standard disclaimer should apply here – I work for a company which sells storage systems (HDD+SSD), and this is all my own opinion and does not reflect the opinions of my company. One might think that because a company like the one I work for sells hard drives, they would want to remove secondhand ones from the market in order to sell more. This naïve view doesn’t account that the financial model enterprises storage systems are sold under is weighted more towards licensing than hardware, I also don’t get paid based on sales, so I assure you, there isn’t any bias here.

Anyway.. the most recent discussion on this topic from reddit brought this article from IEEE Computer to my attention. This article correctly states that there are benefits to enabling a circular economy for storage devices, and that many storage devices can be cleared, or at the very least purged, of confidential data, to enable secure reuse.

  • I’ll note that one of the contributing authors is part of the Chia project, which uses the blockchain and huge amounts of storage to enable.. I dunno, commerce or something. Chia is not without criticism, that while it claims it’s greener than bitcoin, it still involves spinning rust, which is not super cheap. I don’t actually take a position on the Chia project, other than to say that encouraging re-use and circular economy for data storage isn’t actually a bad thing, but that I think if I were part of the project and writing an article about why people spend the time and effort to make cheap drives available, I might consider declaring a conflict of interest.

So here’s the thing – you can absolutely clear data from drives, in some cases, very quickly, in other cases, more slowly (“purge” from the article). But you can make it gone, no doubt about it.

And then what happens? You put a sticker on it, and put it in a pile. How many drives did you have attached to your wiping station at the time? Did you put a sticker on the right drive though? Drive sits around for a while and maybe the sticker falls off. You have a blank drive sitting around, did it get cleared? Well I’ll just wander over and check the device serial number against my log of cleared/purged drives. 5 minutes later, you’ve found it, and the drive’s probably good to go. Then you find another one.. did they use bad adhesive on this batch of stickers? or did it get too hot or too cold. Oh well, it’s probably fine, no one misread your sign and put an uncleared drive in the cleared pile. And no one grabbed a drive from that pile for another project and put it back in there again after they were done, Right?

Or maybe you’re lucky, maybe you have some secure cages and you can make sure that no one other than you moves drives in or out of the cage. But IT is usually kept in the basement for a reason – space is at a premium, and no one wants you to have more than you absolutely need.

Off they go to the drive reseller who’s paid you $20-50 each for them. It’s taken a couple of hours to unrack, attach, clear/purge the drives, label them, arrange the sale, which has taken you away from your core job tasks. “Earned $2000 by selling hard drives” is your end of year achievement.

But.. oops, no, can’t put that down. You missed a step and someone found an uncleared drive that they could recover data from. Well oops. Oops again. Our bad.

Suddenly your organisation is on the hook for regulatory fines, or legal settlements, gets bad press, your auditors are breathing down your neck asking “why did you think this would be ok?!?” and then your $2000 for the IT slush fund doesn’t look like such a great trade.

What’s the answer then? Crush the drives. Make it instantly visible that the storage devices leaving your premises do not have data on them.

So, sorry. I take the view that while it’s technically quite possible to erase data from storage devices, that it is unfortunately still irresponsible from a risk management point of view to then allow any drives out of your control, especially since the financial benefits of selling them is so small. The bigger the organisation, the more likely procedures are to break down, and the more likely you are to have a data spill.

My commentary here is full of strawmen, I totally grant that. But to save your company from risk, you need to follow the logical data clearance 100% of the time, and it’s not simple to tell if it has been done, and that the consequences of even a single slip can be catastrophic. Crush the drives.

Creating nested ESXi inside Proxmox

Ok, sounds a little silly, but there are times when you might want to have a nested ESX VM inside your proxmox environment – I found this great page on how to do it, but I’d rather do it via CLI so it’s repeatable.

This is the CLI I found which worked for me:

qm create 703 --balloon "0" --boot "order=sata0;ide2;net0" --cores "2" --cpu "host" --ide2 "ISOs:iso/VMware-VMvisor-Installer-201912001-15160138.x86_64.iso,media=cdrom,size=343064K" --machine "q35" --memory "8192" --name "pveESX3" --net0 "vmxnet3,bridge=vmbr0" --numa "1" --onboot "1" --ostype "l26" --sata0 "nvmeLVMlocal:32,backup=0,discard=on,ssd=1"

In the lead up to that, I also found a page on stackoverflow asking if there was an easy way to take the output of “qm config” and turn it into “qm create”. It doesn’t seem like there is exactly that – “qm showcmd” as recommended on there isn’t terrible, but isn’t enough.

So this is what I came up with – unfortunately it only gets 90% of the way there –

qm config 703 | grep -v 'vmgenid' | grep -v 'smbios1' | grep -v 'meta' | sed -e 's/vmxnet3=.*,/vmxnet3,/g' -e 's/://' | awk '{ printf("--%s \"%s\" ",$1,$2);} END {print("\n");}'

The issue is that it outputs “nvmeLVMlocal:vm-701-disk-0,backup=0,discard=on,size=32G,ssd=1”, but really it needs to be “nvmeLVMlocal:32,backup=0,discard=on,ssd=1” (replace 32 with desired size).

But it’s better than nothing.

Hope this helps someone one day!

Setting up a LVM LV quickly..

I’m playing with proxmox right now, and had a need to setup LVM locally on each node. I figure why not script it.. I’ve ended up with this abomination of a script. You probably want to do better, but it’s a start:

set -x;
lvmid=`hostname`_localLVM; 
pvcreate /dev/sda; 
pvs; 
vgcreate vg_$lvmid /dev/sda; 
vgs; l
vcreate -n lv_$lvmid -l 100%FREE vg_$lvmid;
lvs; 
mkdir -p /local/vg_$lvmid-lv_$lvmid; 
mkfs.ext4 /dev/mapper/vg_$lvmid-lv_$lvmid;
echo /dev/mapper/vg_$lvmid-lv_$lvmid /local/vg_$lvmid-lv_$lvmid ext4 defaults 0 0 >> /etc/fstab; 
mount /local/vg_$lvmid-lv_$lvmid; 
df -h

Or on one line..

set -x;lvmid=`hostname`_localLVM; pvcreate /dev/sda; pvs; vgcreate vg_$lvmid /dev/sda; vgs; lvcreate -n lv_$lvmid -l 100%FREE vg_$lvmid;lvs; mkdir -p /local/vg_$lvmid-lv_$lvmid; mkfs.ext4 /dev/mapper/vg_$lvmid-lv_$lvmid;echo /dev/mapper/vg_$lvmid-lv_$lvmid /local/vg_$lvmid-lv_$lvmid ext4 defaults 0 0 >> /etc/fstab; mount /local/vg_$lvmid-lv_$lvmid; df -h

Hope this helps someone in the future.. maybe me!

NetApp H610S Fan speed too high – controlling with IPMI

I was chatting with someone on the NetApp Discord today – they’d just bought a used NetApp H610S aka NAF-1703 aka Quanta D52B-1U, and installed Proxmox on it, but the fans were running too high. This was a WAF issue, and they were looking for a way to calm it down.

We did some digging and found this document on the IPMI commands for a similar platform – but they didn’t work on this one.

Some digging around suggested that they needed to use

ipmitool raw 0x30 0x39 0x00 0x00

instead of

ipmitool raw 0x30 0x39 0x01 0x00

What’s the difference? Who knows, but it worked.

Hope this helps someone!

7 Mode? In this economy?

I had two discussions about Data ONTAP 7-mode last week, which was a bit of a surprise, since it’s been something NetApp has been working to help customers get away from for.. some time now. 8 years really pushing it, 10 years since NetApp started providing Clustered ONTAP as an option.

You can totally understand it – data has GRAVITY. It’s heavy and hard to move. Those moves and cutovers need to be as seamless, or quick (or ideally both) as possible. And 7DOT was a platform people had a lot of experience with and understood, and change is difficult.

I’ve been in videos and given countless presentations how to do 7toC migrations quickly and easily, and done a LOT of them, either personally, or working with customers, but the end result is that some people haven’t done it, and it’s now 2023, and the remaining 7DOT users find themselves in a tough spot.

Last November, Microsoft made some AD changes, which means that to continue using 8.2.5P5 with Active Directory, you need to re-enable RC4 encryption. RC4 is.. not terribly secure, so I wouldn’t do that.

At the beginning of Feb 2023, NetApp stopped supporting the FAS255x and FAS80x0 controllers, which are the last generation to run 8.2.5, the last release of 7DOT, which itself is now in “self support”. Self support means NetApp won’t delete the webpages which help with troubleshooting. But once they’re gone in Jan 2026 (less than three years away.. and it can’t come soon enough), you’ll be stuck with some random university in Wollongong hosting an old copy..

ONTAP does everything 7DOT did except allow direct FC connections (because it requires NPIV for FC LIF hosting) and providing an FTP server. The first is an easy fix (buy an FC switch.. if you’re still running 7DOT, you’re probably not adverse to eBay purchases of infrastructure) and the second is a matter of setting up a small Linux VM somewhere if it’s a really big concern.

The best time to migrate off 7DOT was 2016. The second best time is now.

Setting up a PowerBook G4 12 inch, 1.5Ghz in 2020

Some time ago, I received a PowerBook G4 12 inch from a friend. As is healthy, the drive had been wiped, but being one to not keep too much old stuff, I didn’t have install media for Leopard (10.5). It went on the backburner for a while, but I recently received (back) some old storage devices which had been on ice in the WA Wheatbelt for about 10 years, and felt I had the right stuff together to give it a go again.

  1. Downloaded 10.5.4 Leopard Installer from Archive.org
  2. Attach 30GB PATA drive by PATA to USB2.0 dongle
  3. Using 10.15 Catalina, partition drive with Apple Partition Map to 10GB+20GB partition
  4. Mount Leopard Installer ISO previously download
  5. Use ASR to restore contents of ISO to 10GB partition
bash-3.2$ sudo asr restore --source /Volumes/Mac\ OS\ X\ Install\ DVD/ --target /Volumes/Emptied/ --erase
	Validating target...done
	Validating source...done
	Erase contents of /dev/disk3s3 (/Volumes/Emptied)? [ny]: y
	Validating sizes...done
	Restoring  ....10....20....30....40....50....60....70....80....90....100
	Verifying  ....10....20....30....40....50....60....70....80....90....100
	Restored target device is /dev/disk3s3.
	Remounting target volume...done
  1. Try to boot Powerbook using PATA drive on USB dongle, failed
  2. Move PATA drive from USB dongle to Sarotech Cutie FireWire caddy (originally purchased in Tokyo, 17 years ago), success
  3. Install Leopard
  4. Reboot, install 10.5.8 combo update
  5. Reboot
  6. All works! Yay

A bit of vmware fun for a change!

I’ve got a VMware VCP – Have done for about 8 years (passed the exam 4 times now..), but most of my day these days is dealing with storage – but I’ve had a family fixit to migrate a 17 year old laptop into a VM on a more recent mac.

Nice and easy I sez! Just use the converter! Nekt minit..

vmware error.PNG

For the keyword lulz of anyone searching for this problem: “Error 1920.Service VMware vCenter Convert Standalone Server (vmware-converter-server) failed to start. Verify that you have sufficient privileges to start system services”

First I tried the obvious – checking the local user was an Admin, running it as administrator, trying vmware’s fix of creating a group called “Domain Admins”.. all with no dice.

Then I found someone suggesting how to start the agent manually, and when I did that, it complained that the certificate could not be verified.. which lead me on another path, checking Properties in Windows, which lead me to this KB entry on how to install the code signing certificate root on Windows XP.. which lead to a KB entry on another site, which led to 404, which led to web.archive.org, saving the file as a PEM, adding the Certification MMC snapin, importing it, and finally, the service started up.

Nice and simple.. only took someone with 19 years of experience with VMware and as a professional infrastructure admin an hour to fix it..

Edit: Oh ho, but there’s more! VMware Converter then was unable to send the image to the Mac – under the hood it’s still the same old converter that has been around for years, which means that it saves the image over SMB to the Mac. Except, Windows XP can’t talk to Catalina by default. Some will suggest upgrading to SP3 (a good idea, but I want to make minimal changes to this system image..), but that isn’t necessary – as outlined at this post, all you need to do is set HKLM\System\CurrentControlSet\Control\Lsa\lmcompatibility level to 3, from the default of 1 on SP3 or 0 on SP2

Quickly convert HEIC to PDF for expenses submission

Recently I had to convert a large number of photographs taken on my iPhone into PDF for submission with my expense report. I took to my old faithful ImageMagick (installed via HomeBrew) and its mogrify command:

mogrify -resize 50% -format pdf -level 0%,100%,2.0 -type Grayscale -compress jpeg *.HEIC

Hope this helps!

How to wipe a partitioned ADP NetApp system

With ONTAP 9, there is now an “option 9” in the boot menu that allows you to re-initialise a system to/from ADP, like wipeconfig.

It is a three step process to wipe an HA pair – the first one, option 9a –  removes the existing partition information. And the second, option 9b, will repartition and re-initialise the system, and then finally on the node that was halted, boot it, then wipe it (option 4) from its boot menu.

*************************************************
* Advanced Drive Partitioning Boot Menu Options *
*************************************************
(9a) Unpartition disks and remove their ownership information.
(9b) Clean configuration and initialize node with partitioned disks.
(9c) Clean configuration and initialize node with whole disks.
(9d) Reboot the node.
(9e) Return to main boot menu.

The caveat is that one node has to be halted at the LOADER> prompt while you run the first two commands. That should be it!

Moving your Windows install to an SSD, breaking it, then fixing it.

I’d been putting this off long enough, but yesterday was the day! I was going to move our Windows install to an SSD.

And I did. But you shouldn’t.

If at all possible, do a fresh re-install of Windows on your new SSD, and move your data across. Download the Windows DVD Creator (also makes USB keys), or use the “Make a recovery disk” option in Windows to blatt the installer onto an 8GB USB key, and start fresh.

So why didn’t I do that? I like a challenge, and at this point I’m just being obstinate about not re-installing.

For a short period of time in 2009, my wife worked for a company in Canada, that then got bought out by Microsoft. Like, a really short period of time – she “worked” for 3 weeks, then got 4 weeks severance… and her desk.. and her computer, which was a pretty speccy (for 2009) Dell, running Windows 7. It got case swapped, then we swapped the motherboard, then I moved it from a 750GB SATA HDD to a 2TB SATA SSHD (Hybrid HDD.. I wouldn’t recommend them frankly), and in the process moved from MBR to GPT and BIOS to uEFI, all without re-installing. At this point, we’re now in a different country, almost 8 years later, in it’s third case, second motherboard, fourth graphics card, and it’s now running Windows 10, but it hasn’t been reinstalled.

The first challenge – C:\ was a 750GB partition, and the new SSD was 500GB. Reboot into SysRescCD and use gparted to re-size – except for some reason, it couldn’t unmount it. Mess around for a few reboots, and eventually boot with the option to cache everything into memory, and we’re good – resized down to 450GB

Next challenge – the rest of the source drive isn’t empty – there’s another 750GB scratch partition, as well as two Linux partitions. This means I can’t just copy the entire disk to the new one. But I do need the GPT, EFI boot partition, and Windows partition, and they’re all in the first 500GB. Cue “dd if=/dev/sda of=/dev/sdb bs=4096 count=115500000”. Then load up “gdisk”, delete the entries for partitions that don’t exist, and away we go.

And it works.. until I plug the old drive back in (even after deleting the old C: drive and EFI drive with SysRescCD..). Then it stops booting.

At this point, I’m pretty sure the EFI partition and BCD is hosed, so eventually I find this article – http://www.hasper.info/repair-a-destroyed-windows-7-uefi-boot-sector/ – it works for Windows 10, thankfully, and now everything is back working again, and speedy and on an SSD.

Most people’s saturday night’s don’t involve rewriting partition tables and fixing EFI. Perhaps that mine does is a sign I’m in the right career right now, working for a SAN/NAS Vendor..