• Dell PowerEdge R6725

• 2 x AMD 9375F

• 1536GB DDR5

• 2 x Nvidia L4 GPU

When attempting to start a VM with a GPU profile assigned we receive the following error "You attempted to run a VM on a host which doesn't have I/O virtualization (IOMMU/VT-d) enabled, which is needed by the VM". The following settings in the BIOS are enabled: Virtualization Technology, IOMMU Support, SR-IOV Global Enable.

We have installed the following Nvidia Driver versions: NVIDIA-GRID-XenServer-8-580.65.05-580.65.06-580.88 / NVIDIA-GRID-XenServer-8-570.172.07-570.172.08-573.48. We have also installed the latest firmware and BIOS versions but there is an issue with IOMMU.

The following command xe host-param-get uuid=c28a5fa7-6f50-4855-a73f-3da934cb0f28 param-name=chipset-info param-key=iommu outputs false.

The following command xl dmesg | grep -i iommu outputs nothing.

The following command xl info | grep -i virt_caps outputs virt_caps : pv hvm hap gnttab-v1 gnttab-v2.

Does anyone have any expertise on IOMMU and why it would not be working in this environment? Are there any config files that need amending? Any help would be appreciated, if we cannot get this working unfortunately we will have no other choice but to choose another hypervisor.

The Citrix Host is not licensed - see the attached screenshot. My question:

How can I assign to a VM the GPU as "Pass Through"? Which conditions must be met in order to use the GPU in a VM?

I must copy the NVIDIA Virtual GPU Manager RPM file to the Citrix Hypervisor dom0 shell. But it's my first time doing it I set up the smb iso library but it only sees iso files and I have no idea how to put rpm files easily into this domain (I accessed the console and looked into the graphical interface but dont find anything). Also, there may be better ways to keep all files for installation in XenCenter as the storage option only sees ISO files. 

Also, I wanted to use Debian 12 but looking at the  Nvidia website for XenCenter only Ubuntu 20.04 LTS is supported. But it should be supported as it's forked from Ubuntu?

We currently have an A16 card with licenses. When setting up in XenClient, I'm unsure what's the best option: 

NVIDIA A16-16Q virtual GPU (1per GPU, multiple vGPU support)
NVIDIA A16-16Q virtual GUP (1 per GPU)

NVIDIA A16-8Q virtual GPU (2 per GUP, multiple vGPU support)

NVIDIA A16-8Q virtual GPU (2 per GPU)

I'm unsure what the "multiple vGPU support" means. Thanks for any assistance or advice. 

The plan is to use an unsupported scenario with a Windows Hyper-V server as hypervisor and then use dedicated GPU pass-through for the Citrix multi-session OS.

Would a single Nvidia vPC license work for this scenario?

While reading this document https://images.nvidia.com/content/Solutions/data-center/deployment-guide-vgpu-software-citrix-apps-desktops-citrix-hypervisor.pdf we found this Note:

Quote

For NVIDIA vApps you can use Citrix Virtual Apps with a multi-session OS, and for NVIDIA vPC and RTX vWS you can use Citrix Virtual Desktop and is limited to single-session OS

Unfortunately, Citrix Virtual Desktops with a multi-session OS is not mentioned..

after upgrade the NVIDIA driver on the hypervisor host, xencentor shows Virtual GPU types only allow "Pass-though whole GPU"

I found this issue when upgrade Nvidia driver. all hypervisor hosts has identical hardware (AMD EPYC CPU and Nvidia A2 GPU) in the same pool running Citrix Hypervisor 8.2.0

upgrade from NVIDIA-GRID-CitrixHypervisor-8.2-470.63-470.63.01-471.68 to NVIDIA-GRID-CitrixHypervisor-8.2-525.85.07-525.85.05-528.24

I also tried the upgrade on a standalone host, but works fine ( I have NVIDIA A2 types options ). the hypervisor version is 8.2 CU1

how to fix this issue, is it possible to roll back the driver to the old one?

Regards,

Hong

I have local SSD disk in Dell servern, and hypervisorn and my vm´s are on these disks.

I use GPU passthrough to one vm for the first test. If i rightclick on the vm in xencenter, it is possible to assign the GPU
to my vm and it boots correct. I can then also install the host driver for my card and it will find S7150 inside device manager. So far so good.

The vm´s are regular vm´, i mean not pvs or mcs. The vm´s i have tested is:

Windows Server 2012 R2

Windows 10 1703

Windows 10 1709

Windows Server 2016

The Citrix DDC/Storefront are 7.13 and the VDA is 7.15 LTSR. When installing VDA i choose 3DHDX Pro.

When installing VDA, i choose remote pc option and not the master image option.

I have not set any policys, is there anything to set?

I have installed a benchmark program from Unigene, https://benchmark.unigine.com/valley

And when i start my benchmark and use full screen and open gl i gor poor performance.

The FPS is around 25-30 and i can see on the screen that there are no "flow" in the picture.

And also the CPU is peak on 100% och i can see in performance monitor that GPU is around 70%, so it seems to work
anyway, i mean it find the graphics card.

I have not tested mxgpu yet, i will try that soon also.

I dont have a qlue now why it is so bad. Is there anyone here that have tested this configuration with better results?

Shall i try another benchmark program?

Just for your info, i have also done a test installation on vmware and same result.

I have a hardware server that is spec'ed to accept two PCIe3 x16 cards for vGPU cards.  I would like to buy an NVidia Tesla M10 with 32GB of DDR5 as a starter card to get the setup right before buying the second one.  Then, I would like to setup multiple Win10 VMs but NOT Pass-through.  I would like to have each VM use a 1, 2, 3 or 4 GB profile.  IDK the size yet as I need to test what is minimally needed.  What I have described is the two end pieces of the setup as a whole...

[M10 hardware] - [HyperVisor with NVidia driver] - [?] - [vGPU desktop VMs that are not pass-through]

I see in the section "Prepare host for graphics" a sentence that needs clarity..."NVIDIA vGPU is available for Citrix Hypervisor Premium Edition customers, or customers who have access to Citrix Hypervisor through their Citrix Virtual Apps and Desktops entitlement."

Do I need Hypervisor Premium (~$3K) or do I need to pursue the Citrix Virtual Desktops "entitlement"?

My confusion is, what is minimally NEEDED in the middle there?  Hypervisor Premium or Virtual Desktops entitlement?  Thank you for your time, you are appreciated!

Have a great day.

Cheers,
Pete

Running into a problem setting this up and I dont know where to look next.

Environment:

Server: Cisco C240M5

GPU: Nvidia Tesla T4

Nvidia Driver: 451.82-tesla-desktop-win10-64bit-international

Hypervisor version 8.2 (thried with 8.1 just in case, same thing)

VDA version: 2006

As a PoC this is what I've done

• spooled up a vanilla Windows 10 VM
• ran windows updates
• installed Nvidia driver
• installed xen tools
• installed VDA

At this point when when runing the VM it indicates that it is using microsoft display driver. Important to note that the T4 is currently visible to the VM as a device and the driver installed succesfully, confirming that the card is present considering that it does a compatibility check. MSIAfterburner sees the card but its graphs stay bottomed out no matter what GPU intensive application is running (or trying to run as well as they can using the basic driver anyhow). 

Just in case I used the command to enable Nvidia:

NVFBCEnable.exe -enable -noreset

This had no effect

Proceeded to follow a write up that indicated the use of registry keys to influence the driver desision. 

Path: HKLM\Software\Citrix\Graphics
Key: AdapterMerits

DWORD: Nvidia, CitrixDoD, CitrixIDD (with the Nvidia DWORD having the highest value)

This had no effect either.

At this point I dont know what else to try. If im not mistaken there shouldnt be any need to install drivers on the hypervisor itself as this is on passthrough mode and not vGPU. Any secrets to get GPU passthrough to work?

Thanks,

Stephen

can someone help me to find the right intel Xeon cpu for a dell r740 Server which supports intel gvt-g. 
 

best thanks. 

The doc says the group is updated by xen automatically, but I have removed the T4,M60 and put in the RTX, but it is not updated.

[root@localhost ~]# nvidia-smi

Tue Feb 25 16:23:17 2020

+-----------------------------------------------------------------------------+

| NVIDIA-SMI 440.53 Driver Version: 440.53 CUDA Version: N/A |

|-------------------------------+----------------------+----------------------+

| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |

| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |

|===============================+======================+======================|

| 0 Quadro RTX 6000 On | 00000000:86:00.0 Off | Off |

| 33% 55C P8 33W / 260W | 176MiB / 24575MiB | 0% Default |

+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+

| Processes: GPU Memory |

| GPU PID Type Process name Usage |

|=============================================================================|

| No running processes found |

+-----------------------------------------------------------------------------+

[root@localhost ~]# xe gpu-group-list

uuid ( RO) : e062fc86-82d6-e7fc-969e-9c2837698e0f

name-label ( RW): Group of NVIDIA Corporation Device 1e30 GPUs

name-description ( RW):

uuid ( RO) : 313004be-b34a-115c-d6eb-d7d8431232ac

name-label ( RW): Group of NVIDIA Corporation TU104GL [Tesla T4] GPUs

name-description ( RW):

uuid ( RO) : 6afa24fa-4b19-4174-82e4-93b8687271d6

name-label ( RW): Group of NVIDIA Corporation GM204GL [Tesla M60] GPUs

name-description ( RW):

uuid ( RO) : 60b2c8b3-ed41-3a02-b334-133e3bd78204

name-label ( RW): Group of Matrox Electronics Systems Ltd. Device 0538 GPUs

name-description ( RW):

In my company we use quite a lot of virtualization.

We have just setup one machine with a use of Citrix Hypervisor, to be used with Premiere PRO and Adobe Media Composer.

Operating system of this VM is Windows server 2016.

The grafic card we use on the host is :nvidia tesla p40 24gb.

I attached some screenshots.

Basically the card is not use by any of the two programs.

Premiere says sth like wrong driver which is not the case.

I tried nvidia tools nvsmi to check if the card is used but it says insufficient permissions.

Has any of you ever tried this kind of setup?
Anyone has an idea why it doesn't work?

Attached some screenshots.

Artur

uuid ( RO)                        : 117e4ed9-74d1-4a9a-9ca3-1bb014bef89a
                 vendor-name ( RO): NVIDIA Corporation
                 device-name ( RO): GP107GL [Quadro P620]
                 dom0-access ( RO): enabled
    is-system-display-device ( RO): false
              gpu-group-uuid ( RW): 51ac198a-a379-d4a1-429b-e16b86f2134f
        gpu-group-name-label ( RO): Group of NVIDIA Corporation GP107GL [Quadro P620] GPUs
                   host-uuid ( RO): c41eea7d-76d3-48a6-b97a-7a8e5eb61ede
             host-name-label ( RO): xen03
                      pci-id ( RO): 0000:86:00.0
                dependencies (SRO): 0000:86:00.1
                other-config (MRW):
        supported-VGPU-types ( RO): 0b431a33-6241-46a1-acec-db3d92d44a77
          enabled-VGPU-types (SRW): 0b431a33-6241-46a1-acec-db3d92d44a77
              resident-VGPUs ( RO):
 

The driver installer appears to complete OK (passes system checks, etc.), but the driver does not start. The Microsoft Basic Display Driver is still active and the Device Manager reports that "Windows has stopped this device because it has reported problems. (Code 43)". The Events tab in Device Manager says "Device PCI\VEN_10DE&DEV_1CB6&SUBSYS_126410DE&REV_A1\3&267a616a&0&30 requires further installation."

I've used various drivers, but today tried the latest:

https://www.nvidia.com/Download/driverResults.aspx/152647/en-us

The problem is, I get an error when booting saying "The GPU group does not contain any GPUs". The VM seems to be attached to some vGPU that we previously had and no more exists. I want to know, through CLI or xencenter, how can I remove that vGPU attachment. Also, is there a way to tell xenserver to "deafult" to a non-GPU state? Following are some screenshots from xenceter showing the error and VM's properties. Thanks!

Here is the driver version installed.

My problem is, when i'm running a vm, i have the following message when i want to open the nvidia control panel :

I tried different gpu type for the vm with xen, none of them make a difference.

The current profile used is the following :

I tried different nvidia drivers, idem. Here is the current driver installed on the vm:

The os is a windows server 2012r2 but have the same issue with a windows 7.

I think it's the xenserver version. do i have to upgrade it?

thanks for your answers.

I was using GPU passthrough of Intel iGPU from Windows 7 guest, Xenserver 6.5 to 7.1, last one working was 7.4. From that version I am not able to see video output. GPU is successfully pass through, drivers installed, however no video output. I also tested Win10 and several Linux distros guests, same issue. I prefer to stay with Xenserver, as the pci passthrough of devices works really good in comparison to KVM/ESX on whiteboxes.

Anybody who can confirm same or raise it with Citrix (and Intel GVT-d) ?

Thanks,

G.

This is my environment info.

Xenserver : 7.1 CU2 Enterprise Edition (installed hotfix)

GPU : Tesla P40 * 2ea

Xendesktop : 7.15.3

Citrix License : Using 90-Trial license to test.

And also made a nvidia grid license server on linux.

<Issue>

1. Allocate Pass-through or GRID P40-24Q GPU Tpye on VM 

2. Start a VM

3. When I check a GPU allocation status on xencenter - xenserver - GPU tab, I can see GPU allocated.

4. But when I connect a VM , GPU didn’t allocated.  I can not see a “NVIDIA GPU” infomation on display-adapter.

Add, other GPU TYPE is working properly. (ex: GPU P40-12Q , 8Q, 4Q....)

Please let me know how to fix it.

Scenario:

XenServer and Xendesktop (latest versions)

Nvidia M10 (4 x GPU on the board)

VM Windows 10 or Server 2016/2019 with assigned 1 M10 GPU in Passthrough

I need to know what is the maximum number of 1080 heads display.

in the documentation I found only the reference to the maximum number of displays in the case of virtualization profiles (4), but not in the case of passthrough.

Thank you

Adriano

I have installed XenServer 7.5 with NVidia Tesla P40.

Unfortunately, I can only assign Pass-through whole GPU. When VM has assigned any other vgpu profile I'm not able to see any console output or do RDP.

Any suggestion what can be wrong?

Thank you in advance.

Diamant