Jump to content
Welcome to our new Citrix community!
  • 1

2016 RDSH shows NTFS errors


Robert Raschke

Question

Hi everyone,

 

I am currently evaluating Windows 2016 RDSH with VDA 7.15.1 and PVS TargetDevice 7.15.2.11. We are planning to go to RDSH 2016 from 2008R2 in early September. Currently I am the only user in the test environment (user tests about to start in two weeks). All componentes are on LTSR 7.15.1 except the PVS-Servers which are still on 7.15. The VDA is monitored by MS OpsMgr.

 

Nearly every day I get a "NTFS reported the logical disk is in error or corrupt" event log entry. Everythings seems to work fine, though. There are no problems with any application until now, sometimes the error comes up when nobody is logged in the VDA.

 

What I have tried so far:

- re-imaged the vDisk

- re-installed the master VM and created a new VHD

- tried the same with VHDX

- ran an offline "chkdsk" with the writeable, mounted vDisk -> no errors found

- ran "chkdsk" right after imaging the vDisk -> no errors found

- checked for Scheduled Tasks running right before the error message -> no tasks found

 

So this has to happen during runtime.

 

Has anyone seen this before?

 

Regards

Robert

Link to comment

Recommended Posts

  • 0
19 minutes ago, Wolfgang Germ said:

What is the event-ID for the mentioned entry?

I have almost the same configuration and would have look, if the event is on my VDAs too ...

 

Hi Panteraa,

 

thanks for the reply.

there are two events in the system log at exactly the same time:

 

ID 98, Source 'Ntfs (Microsoft-Windows-Ntfs)'

ID 55, Source 'Ntfs (Ntfs)'

 

Robert

Link to comment
  • 0

I did check the event-log on 3 Server 2016 Hosts. (Running through Citrix PVS v-disk.)

I did find a lot of events with ID 98 - NTFS, but only as Information and not as warning or error:

 

5aaff11b692b1_EventLog1.thumb.png.3b244bb998ded2f7790a69b6c5d0731c.png

 

5aaff11f35c22_EventLog2.thumb.png.9ed0b473272d24302c095b3a519ef522.png

 

It seems to be a regular check. First message is the System-Disk (C:) and the second one is a static-disk. (D:)

Your name sounds german and therefore I think you can read the text on the screenshots. It means, that no errors were found.

I didn't find any "ID 55, Source 'Ntfs (Ntfs)".

 

I am sure I have seen NTFS-Errors in the past, but maybe on Server 2008r2? I don't know anymore ...

 

Not that helpful, I know ...

Link to comment
  • 0

We are having the same problem in two completely different environments (event 55 and 98). One is provisioned on Hyper-V and the other one is non provisioned on XenServer. It only happens on the 7.15 CU2 VDAs, and not on the infrastructure servers like Delivery Controllers or StoreFront servers.

 

What did you find out Robert?

 

Link to comment
  • 0
17 hours ago, Kenneth Mogstad said:

We are having the same problem in two completely different environments (event 55 and 98). One is provisioned on Hyper-V and the other one is non provisioned on XenServer. It only happens on the 7.15 CU2 VDAs, and not on the infrastructure servers like Delivery Controllers or StoreFront servers.

 

What did you find out Robert?

 

 

Sorry, I have not found anything useful. Until now, we are living with the events. This seems to only happen when using a specisif application and this applicatio is working normaly...

Link to comment
  • 0

Ok! We can't reboot our VMs when it happens. If we do, the VM enters a neverending chkdsk loop. Since chkdsk can't correct the disk error, we have to revert to a snapshot before the error occured to solve the problem. Then it takes anything from something like 1-3 days before the disk error occurs again. It only happens on VDAs with users, and not on VDAs without any. Our current VDAs have a lot of different applications and users, so it's difficult to pinpoint anything specific. We suspected the VDA software, but Citrix says there's no disk driver in it, and that this has to do with the underlying OS. Problem is that the VDA software seems to be the common denominator since it doesn't happen on Server 2016 VMs without it. But then again it doesn't happen on VDAs without users either, so difficult to say. Considering trying to disable Windows Defender on the VDAs next week and see how it goes...

 

What application are you suspecting by the way?

Link to comment
  • 0

We think we are closing in on what causes this now. We don't think it's IBM Notes or any other applications' direct fault. It just seems like it is because using these applications results in increased disk activity, but we think the problem is the Citrix disk filter driver intercepting disk writes (CtxMcsWbc.sys). This driver comes with the VDA installation. Can you check if you have it? Go to Device Manager > Disk drives > "OEMU HARDDISK ATA Device" or "XENSRC PVDISK SCSI Disk Device" > Properties > Driver > Driver Details. Do you see it in the list of drivers here (should be four sys files listed)? Also you can check if it's effectively enabled in registry. See you can find "PvsVmBoot" under HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\BootExecute.

Link to comment
  • 0

Yes, we got it, too.

Both disks ("VMware Virtual disk SCSI Disk Device" and "Citrix Virtual Disk SCSI Disk Device") have identical drivers including "CtxMcsWbc.sys".

 

Do you have a ticket opened with Citrix? Can you keep this thread updates? that would be great.

Edited by Robert Raschke
Link to comment
  • 0

Thank you. That's good because that means that the three environments, yours and two of ours getting disk errors all have the Citrix disk filter driver. We actually have another environment without CtxMcsWbc.sys, and this solution does not get the disk errors. Here the VDA was installed with the parameter /exclude "Machine Identity Service". That seems to be the only way to avoid it. Just selecting "Enable connections to a server machine" (and not "Create a Master Image") during VDA installation still installs it.

 

Yes, we just opened a new case with Citrix and will keep you informed.

Link to comment
  • 0

I want to bring you an update:

 

For yesterday we tried removing the Citrix disk filter driving by modifying the registry in three different places. After doing that we could not see the Citrix disk filter driver in Device Manager, but still we got the disk error on 2/2 VDAs in test. Perhaps our registry modifications were not sufficent...

 

So for today we've reinstalled the VDA software without the Citrix disk filter driver, and so far it's looking good. We do however feel it's too early to conclude yet.

 

These are the command line parameters we used for the installation:

 

XenDesktopVDASetup.exe /components VDA /controllers "controller1 controller2" /disableexperiencemetrics /enable_hdx_ports /enable_hdx_udp_ports /enable_real_time_transport /enable_remote_assistance /exclude "Personal vDisk","Machine Identity Service","Citrix Telemetry Service","Citrix Personalization for App-V - VDA" /hdxflashv2only /logpath "C:\Windows\Logs" /noreboot /quiet /virtualmachine

 

The exclusion of the Machine Identity Service above is what's doing the trick here.

 

Perhaps you could try reinstalling the VDA software in your environment without the Machine Identity Service and see if that solves the problem Robert Raschke?

Link to comment
  • 0
Quote

Perhaps you could try reinstalling the VDA software in your environment without the Machine Identity Service and see if that solves the problem Robert Raschke?

Unfortunately, we only have a small envrionment with everything running via PVS. I am not sure if I can remove the "Machine Identity Service" from a PVS provisioned VM.

Link to comment
  • 0

You can do that Robert Raschke. Just open up the image, uninstall the VDA software and install it again using the proper command line parameters. We are already running PVS provisioned VMs in another environment without the Machine Identity Service (and without the disk errors), and have been doing that for a while. PVS provisioned VMs takes care of identity through the PVS Target Device software. The Machine Identity Service is for Machine Creation Services/MCS. We feel the VDA GUI installer from Citrix handles this very poorly. For instance, it still installs the Machine Identity Service even though you select this during installation (and not the first option):

 

image.thumb.png.7c6c3e546b3d2b197fdd8cd7d1bc5c91.png

 

By the way.. if you're having trouble uninstalling/installing the VDA software, make sure you add Authenticated Users to the Remote Desktop Users group temporarily beforehand. This is another annoyance from Citrix.

Link to comment
  • 0

We have not had the disk error the last two work days since we reinstalled the VDA without the Citrix disk filter driver. Hence we're pretty much confident that our VDAs are stabilized now. We also noticed that the disk error couldn't be repaired when the Citrix disk filter driver was present. Once we uninstalled the VDA and the Citrix disk filter driver with it, the disk error could be repaired. This once again points to problems with the Machine Identity Service and it's disk filter driver. Citrix have escalated the case to engineering.

Link to comment
  • 0

Yeah! It's unfortunate.

 

I've been trying to reproduce the issue today for hours. I've been running DiskSpd from MS, full antivirusscan, chkdsk, Automatic Maintenance and 16 user logins and logoffs all at the same time with no luck. It has to be something very rare occuring that causes these disk errors making it very difficult to solve.

 

It's also unfortunate that I haven't gotten any help from Citrix so far, and that we've spent so much time and resources on the issue (not to mention all the other bugs with 7.15 LTSR, Receiver, UPM and XenServer Management Agent). Perhaps if more people starts creating support cases. I feel Citrix rather want to point their fingers at Microsoft at the moment.

Link to comment
  • 0

Are there any news on this?

We are now running into the problem in our production environment. The NTFS error comes up early in the morning on the PVS provisioned machines when the first users connect. After some time during the day, we get massive "System"-process CPU (about 25% on an 8 vCPU VM). Tracking this with Windows Performance Monitor shows, that "ntfs.sys" is driver thats using up the ressources. I suspect this to be a follow up to the early-morning NTFS errors. 

I will open up a ticket later...

Link to comment
  • 0

We could break this downdto our AntiMalware solution...

There were also quite a lot of SMB errors in the eventlogs, mainly related to IBM Notes. After tinkering with the AntiMalware configuration we could get rid of these (by adding file extensions exclusion, the process exclusions for the main IBM Notes processes did not work). With that change most of the NTFS errors were gone, too. They come up from time to time on single servers, but do not disturb the users working on those servers.

We now have an open ticket with our AntiMalware vendor as there seems to be some specific incompatibility with Server 2016 in the version we are using.

Link to comment

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...