Autosupport stopped working

I installed a Clustered ONTAP System about 4 months ago, and I’ve been working with the customer since then on migration of their several hundred workloads onto the system in a staged approach. While doing a regular checkin, I noticed that Autosupport had stopped working on three of their 4 nodes, despite working when I finished the initial build.

Some checks of logs and within the organization showed that it had stopped working at the same time that they had changed their mail server IPs. Easy, you think. Maybe I put in the IPs into the autosupport setup? Checked that, nope, it goes to the hostname. Well, maybe I put in an /etc/hosts entry? (system services hosts show) – nope, wasn’t that. Checked autosupport’s destinations were configured the same on all four nodes – and they were. Maybe there’s a firewall issue? Ping from the node management LIF to the SMTP servers all works. Maybe it’s a specific SMTP firewall block? Used debug mode systemshell and tcp_client (note: don’t try this at home..) – that all worked. I got their firewall and exchange admins to check logs for the node management LIFs trying to make connections, and no attempts, other than my tcp_client ones. Ran pktt on all interfaces with target IPs of the mailhosts, and found no attempts to send out from e0M (home of the node-mgmt LIF), only one of the data LIFs. NetApp KB 3012724 talks about LIFs, and has this to say on the topic:

Clustered Data ONTAP 8.2.x:

  • AutoSupport is delivered from the node-mgmt LIF per node.

Looking through the autosupport history, the attempts fail, and the last error recorded is “FTP: weird server reply”. Uhh.. transport can be either http, https or smtp. Why is it mentioning FTP?

NetApp KB 201727 shows how to access debug logs for autosupport. I did that and saw the error message of “421 Service Unavailable”. Remember the FTP error? Well, that dear readers is because your NetApp, at its heart, is a big FreeBSD box, and it uses curl to send autosupport emails. And when curl gets a “421 Service Not Available” response from the mail server, that’s what it does.

Looking at the pktt logs closer, it’s because the autosupport email is going out of one of the data LIFs for an SVM on the host. Why would you suddenly decide to do that?! Well, let’s look at KB 3012724 again..

By default, routes for the node mgmt LIF have a lower (more preferred) metric than routes of data LIFs. However, the metric is used as a tie-breaker. The more-specific route to the destination will always be picked regardless of the metric.

..

Case 3 – The node-mgmt LIF and data LIFs on different subnets, destination is on the same subnet as the data LIFs. The implicit subnet route of the data LIFs (which isn’t seen in ngsh) will be the most specific route to the destination, and will therefore be the selected route. A data LIF will be used.

So, despite the earlier assurance that autosupport uses the node-mgmt LIF, the actual story is somewhat more complicated. It uses the node-mgmt LIF, unless it likes another one better. As for why only one of the 4 nodes worked? Well that node didn’t have any SVM LIFs on the same subnet as the mail servers, so it didn’t try using them to send the ASUP email.

So what do you do? You can either create individual host routes (/32) in the routing group for the node admin SVMs, or create a subnet route in there to prevent it occuring if IPs change again. I also found (as did it seems another posted on NetApp Communities), that setting the metric lower didn’t solve the problem, you had to set the metric to “1”.

Going forward, part of my system installation will always include a route for the mail server in the routing group that the node management SVM uses.

Go away and I will replace you with a very small shell script..

I did a recent migration of SAN to NAS for a client recently, and had to unmount all of their datastores.

This little one liner for the esxi shell lists all SAN volumes, then gets rid of them..

# esxcfg-scsidevs -m | sed -e ‘s/\:1 //g’ | awk ‘{ printf(“esxcli storage filesystem unmount -l %s;\nsleep 1;\nesxcli storage core device set –state=off -d %s;\n”,$4,$1);}’ >; /tmp/unmountluns.sh

Review the output for sanity, and run.

Setting clock from CLI is not allowed in this VDC

If you’re trying to set the time on a brand new out of box Cisco Nexus 5500 and you get the message “Setting clock from CLI is not allowed in this VDC.”, it’s because the clock protocol is set to ntp, even though you didn’t configure NTP. Go into config and type “clock protocol none”, and then it will let you set the time.

Then, when you’ve finished the config, set up NTP!

And while you’re at it, this page from Cisco is awesome for troubleshooting VPC

Sometimes you can’t get to here from there.

Sometimes things look impossibe. Like screwing in a screw with a handle directly above it (seriously, if I ever met the person who designed this..)

If I ever meet the person responsible for this, I'm punching them in the face

But there’s always a way around things. In this case, I used my fingers to tighten it into place.

Or this screw, which was cross threaded and wouldn’t come out. It didn’t stand up to a pair of channel locks. Sometimes you have to take the hard way.

And sometimes? What you don’t know is a blessing. I don’t have any photos of this unfortunately. But let’s say there was a two storey building, and on the second storey, was a server room, with two 50U racks inside it. You would think – ok, I need to add another one, these two obviously got in here. You enlist some burly gentlemen to help you move the rack up the stairs, and find problem 1 – it doesn’t fit through the back stairs. They take it down the back stairs, and up the front. Problem 2 – it doesn’t fit through the front door standing up, so you lay it down and move it into the corridor in front of the server room. Problem 3 – there are fire sprinklers in the middle of the ceiling, so you can’t stand it up. I scratched my head for a while, and then started removing bits of drop ceiling, until I found a section big enough to get it standing up, without any sprinkler pipes under it. I stood it up, and then went to move it into the server room. Same issue – but found another part of the drop ceiling without pipes to angle it down again to fit through the lower server room door. And it’s done and in place. I went back to the company’s office and asked them how they got the original racks in there?

The older and wiser sysadmin, who I hadn’t been working on for this project, answer sagely: “we built the room around them”. Sometimes not knowing is the solution to your problem. I doubt I would have even tried if I knew that…