NSX-T 3.0 to 3.0.1 upgrade fails on ESXi 7.0 host VIB upgrades due to "Failed to load module nsxt-vsip"

Problem: Cannot upgrade ESXi 7.0 hosts from NSX-T 3.0.0 VIBs to 3.0.1 VIBs.

Scenario: vCenter 7.0, ESXi 7.0, NSX-T 3.0.0, ESXi hosts are running N-VDS exclusively (2 pNIC, upgraded from NSX-T 2.5).

NSX-T Edge upgrade from 3.0.0 to 3.0.1 was successful, but none of the ESXi hosts in the 4-node cluster are able to have their VIBs upgraded from 3.0.0 to 3.0.1.  Error message in NSX-T Manager is:

Install of offline bundle failed on host 09e41e11-6ce5-4fd8-a4ad-3295f927e540 with error : [LiveInstallationError] Error in running ['/etc/init.d/nsx-datapath-dl', 'start', 'upgrade']: Return code: 1 Output: start upgrade begin Exception: Traceback (most recent call last): File "/etc/init.d/nsx-datapath-dl", line 963, in <module> DualLoadUpgrade() File "/etc/init.d/nsx-datapath-dl", line 835, in DualLoadUpgrade LoadKernelModules() File "/etc/init.d/nsx-datapath-dl", line 180, in LoadKernelModules nsxesxutils.loadModule(modName, modParam) File "/usr/lib/vmware/nsx-esx-datapath/lib/python3.5/nsxesxutils.py", line 360, in loadModule (moduleName, out.decode())) Exception: Failed to load module nsxt-vsip-16404614: vmkmod: VMKModLoad: VMKernel_LoadKernelModule(nsxt-vsip-16404614): Failure Cannot load module nsxt-vsip-16404614: Failure vmkmod: VMKModLoad: VMKernel_LoadKernelModule(nsxt-vsip-16404614): Failure Cannot load module nsxt-vsip-16404614: Failure It is not safe to continue. Please reboot the host immediately to discard the unfinished update. Please refer to the log file for more details..

Error in esxupdate.log:

[LiveInstallationError]

Error in running ['/etc/init.d/nsx-datapath-dl', 'start', 'upgrade']:

Return code: 1

Output: start upgrade begin

Exception:

Traceback (most recent call last):

   File "/etc/init.d/nsx-datapath-dl", line 963, in <module>

     DualLoadUpgrade()

   File "/etc/init.d/nsx-datapath-dl", line 835, in DualLoadUpgrade

     LoadKernelModules()

   File "/etc/init.d/nsx-datapath-dl", line 180, in LoadKernelModules

     nsxesxutils.loadModule(modName, modParam)

   File "/usr/lib/vmware/nsx-esx-datapath/lib/python3.5/nsxesxutils.py", line 360, in loadModule

     (moduleName, out.decode()))

Exception: Failed to load module nsxt-vsip-16404614: vmkmod: VMKModLoad: VMKernel_LoadKernelModule(nsxt-vsip-16404614): Failure

Cannot load module nsxt-vsip-16404614: Failure

vmkmod: VMKModLoad: VMKernel_LoadKernelModule(nsxt-vsip-16404614): Failure

Cannot load module nsxt-vsip-16404614: Failure

It is not safe to continue. Please reboot the host immediately to discard the unfinished update.

Please refer to the log file for more details.

Installing the VIBs manually via Lifecycle Manager (KB 78682) fails with exact same error as above in esxupdate.log.  Installing VIBs via CLI (KB 78679) results in same error.  I verified the boot banks (KB 74864) and they have plenty of free space (95% free space).  ESXi install was fresh install of ESXi 7.0 on 128 GB boot from SAN LUN (SSD).

I never had any problems with NSX-T VIB upgrades from 2.4.x to 2.5.x to 3.0.0, so I'm curious why the 3.0.0 -> 3.0.1 upgrade is so challenging.  Has anyone else run into this?

Thanks,

Bill