Post to Twitter |
Comments (26) |
Views (10315) |
PCoIP is VMware's latest attempt at delivering a decent user experience for a virtual desktop. After failed attempts with RDP, Sun Ray, RGS and TCX, VMware View 4 is betting that a software version of the PCoIP protocol will deliver the great user experience customers demand from a VDI solution.
I've been in the virtualization business for many years. Currently I lead the HDX technology for XenDesktop. In the past I've worked on tons of projects for the ICA protocol including CGP, Secure Gateway, and Thinwire. In recent years I've led the Apollo project which has created technologies now in XenDesktop 4 like HDX MediaStream for Flash, HDX 3D Pro Graphics, HDX RealTimeand HDX Broadcast. So I've watched with amusement as VMware attempts to position PCoIP as the next great remoting protocol. The three most amusing 'marketing' tactics about PCoIP are:
PCoIP bets on UDP as the foundational transport for graphics
One of the major design flaws in PCoIP is that it relies exclusively on UDP for deliver bitmaps. UDP is valid for some narrow use cases but PCoIP relies on it entirely. When you need a reliable transport, TCP is a much better option. The fact that PCoIP has application-layer packet reliability shows you need reliable delivery for desktop graphics. If all you are doing is playing a video, fine... but that's not what a virtual desktop is all about. You may not know this but many years ago, ICA supported a datagram-based protocol with application-layer reliability just like PCoIP. Since then, we have learned that TCP is the ideal transport for delivering desktop graphics over the network. It is also friendlier to firewall and network infrastructure. And it is cheaper to deploy as customers can leverage their existing network infrastructure.
PCoIP claims bitmap remoting is the best way to deliver graphics
Another interesting aspect of PCoIP is that the protocol is based on the idea of sending bitmaps. No wonder, since their hardware solution used as input the DVI port of the graphics card. It is interesting that VMware claim that sending bitmaps is better than sending graphic primitives. This is a half truth. While sending bitmaps make sense in some scenarios, sending graphic primitives is much more efficient in other scenarios. Think of this, what is more efficient when sending a 400x300 rectangle with black borders and white background? As a bitmap or sending a RECT command with both upper left and lower right coordinates? The key is to be smart about it and know when one scenario makes more sense than the other. That's what we call SmartRendering. Getting this right is very hard and it has taken us years of fine tuning. But a half truth is convenient because sending bitmaps is the easiest thing to do, after all, that's all most graphic remoting protocols can do.
PCoIP relies primarily on the server to do all the heavy lifting
PCoIP also focuses on the use of server resources to deliver the graphics. But you soon realize that does not get you far enough. I have spoken with countless customers asking us to solve their scalability issues with playing Flash multimedia. I'm sure VMware have shown some YouTube videos to get people excited but you have to look at the CPU and bandwidth consumption. The Flash player uses up lots of CPU, so if your only available solution is server-side rendering then you are going to need a lot of servers. Customers need solutions that scale, are cost effective and leverage their computing resources in the data center and also on the user device. PCoIP fails to do this because it is an incomplete protocol.
Delivering a complete solution takes time and it's hard, very hard. I see PCoIP making some of the same mistakes we made 15 years ago. I congratulate them for trying, but they have a long way to go.
To deliver a great user experience you not only need a robust protocol, you need all the components in the delivery infrastructure working together to optimize the delivery of virtual desktops and applications. This is what we are doing with HDX at Citrix.
Follow me on Twitter
Comments (26)
Nov 12
Anonymous says:
Hi Juan, I think ICA is a great protocol, but why are you saying VMware failed ...Hi Juan,
I think ICA is a great protocol, but why are you saying VMware failed in the past with the other protocols? I don't think that's true! I'm working for a system integrator and I've seen loads of happy View users. It just depends on what you are doing. Of course there a also use cases where RDP is not the best choice. It's fun that you are mentioning HP RGS! Do you really know VMware's solution? RGS and View is only working for HP blades, not for virtual machines.
PCoIP looks great and it just works, like ICA does.
I was a bit dissapointed when I've read that HDX audio can consume up 1.4 MBit/s and also the HDX Flash is a no go for most of my customers due it's architecture.
Before you tattle do also you homework! The customer will choose the solution anyway and they are going for the product which fits best.
Regards.
Stephen Smith
Nov 12
Juan Rivera says:
Stephen, Thanks for your reply. I would like to point out that XenDesktop 4 in...Stephen,
Thanks for your reply.
I would like to point out that XenDesktop 4 introduced a new HDX audio stack that offers HD audio quality at 100Kbps and speech optimized audio at 20Kbps. The new audio stack also significantly reduces CPU usage for increase server scalability.
As far as the issues your are having with HDX Flash, I'd love to hear about them. We are continuing improving the technology and your feedback is very important to us. Feel free to contact me via email.
Thanks,
Juan
Nov 12
Anonymous says:
Don't worry Juan. This is so typical VMware people; they believe everything they...Don't worry Juan.
This is so typical VMware people; they believe everything they are told without questioning. In some countries that technique is called brainwashing.
Anyway, grate to hear all the good news about XenDesktop 4.0, I am pretty sure that VMware will compare View 4 with XenDesktop 2.0 or something, just to be able to get some benefits on the chart... and btw, I have seen PCoIP, it doesn't work as it is supposed to.
Nov 14
Anonymous says:
Yes VMWARE adopts the same brainwashing techniques at Cisco and EMC. no wonder t...Yes VMWARE adopts the same brainwashing techniques at Cisco and EMC.
no wonder the three of them have such a strong alliance
Nov 13
Anonymous says:
http://www.brianmadden.com/blogs/brianmadden/archive/2009/11/13/answers-to-this-...http://www.brianmadden.com/blogs/brianmadden/archive/2009/11/13/answers-to-this-week-s-most-common-questions-about-view-4.aspx
So actually I think yesterday's blog post by Juan Rivera(the guy at Citrix who leads HDX) is not entirely accurate. It seems he's assuming the software version of PCoIP works exactly the same as the hardware version, which is doesn't. (By the way, I think one of the reasons people assume this is because the hardware and software PCoIP clients are fully inter-operable, so there's an assumption that they all must be doing pretty much the same thing. But I think there's a capabilities exchange that takes place, so if a software PCoIP host knows it's talking to a hardware PCoIP client, then it will be sure to send data in a format that client can decode. But it's not like the protocol is identical for every connection scenario.) For example, VMware moved their multimedia redirection into PCoIP, so it's definitely NOT 100% host-side rendering like Juan said.
Nov 13
Tyrone Thomas says:
I think the HDX Flash architecture Stephen is referring to, is that the endpoint...I think the HDX Flash architecture Stephen is referring to, is that the endpoint must have direct access to the source of the Flash content - Today HDX just sends down the URI and the client does the rest.
For many environments this will not work as the content may be hosted on the internal network which the endpoint - in a remote access scenario - may not have access to - in this case ICA/HDX falls back to host side rendering.
In a funny twist - In the tech previews the content data was actually streamed from the host to the endpoints meaning that the endpoints did not have to have direct access to the content source - it was all piped and contained safely, securely and controlled within the ICA/HDX protocol - strangely this was undone for the release!
Would have been nice to have given this as a option to securely deliver content from internal sources.
Nov 13
Juan Rivera says:
Tyrone, We heard your feedback and we added this feature back in the R...Tyrone,
We heard your feedback and we added this feature back in the RTM build of XD4. I'll blog about it in more detail next week.
Thanks,
Juan
Nov 13
Timothy Bardzil says:
Another ramification of PCoIP's reliance on UDP transport is the lack of integra...Another ramification of PCoIP's reliance on UDP transport is the lack of integration with WAN accelerators. VMWare's primary WAN optimization partners -- Riverbed and Cisco -- do not optimize UDP traffic today. This means customers cannot perform network-level compression and de-duplication across multiple VDI sessions. Not good for a protocol that already consumes 10X more bandwidth than XenDestkop 4.
As Juan pointed out, to deliver a great user experience you not only need a robust protocol, you need all the components in the delivery infrastructure working together to optimize the delivery of virtual desktops. Citrix Branch Repeater with HDX IntelliCache technology works seamlessly with XenDestkop 4 to deliver all the benefits of WAN optimization that are not currently available to customers using VMWare View 4 with Riverbed or Cisco.
Nov 13
Timothy Bardzil says:
Also just learned that PCoIP natively encrypts all data traffic which is yet ano...Also just learned that PCoIP natively encrypts all data traffic which is yet another obstacle to accelerating it with most WAN optimization solutions.
Not so with Branch Repeater and XenDestkop 4. Branch Repeater can optimize natively encrypted ICA traffic delivering the maximum performance without sacrificing end-to-end security.
Nov 14
Anonymous says:
since the major use case for VDI has been remote users.. How does VMWARE plan to...since the major use case for VDI has been remote users.. How does VMWARE plan to allow access to PcoIp remotely? I'm assuming their own gateway wont handle this traffic.. Are the suggesting end users levergae an IPSEC VPN ? would this even work with a SSL VPN solution?
Nov 13
Anonymous says:
PCoIP equals no need for WAN optimizers. Secure and highly compressed. Since...PCoIP equals no need for WAN optimizers. Secure and highly compressed.
Since PCoIP is highly compressed and encrypted for security, it should eliminate the need for costly WAN Optimizers.
Nov 16
Juan Rivera says:
One of the value add of WAN optimizers like Branch Repeater is that they can cac...One of the value add of WAN optimizers like Branch Repeater is that they can cache data across multiple sessions.
If multiple users are running similar apps, or have the same desktop background, etc; Branch Repeater would be able to send the information once and multiple users benefit from it. Another interesting scenario is when a user plays a video and the video is delivered via HDX MediaStream, the entire video is cached so the next person would not have to redownload the video again. This corporate communications or training videos where it is likely that more than one person would play the same video.
To do this, you need a protocol that's friendly to WAN optimizers.
Nov 13
Anonymous says:
MYTH: PCoIP bets on UDP as the foundational transport for graphics REALITY: T...MYTH: PCoIP bets on UDP as the foundational transport for graphics
REALITY: TCP constrains Citrix ICA and Microsoft RDP
While TCP is a great reliable transport for data traffic that is not latency sensitive and does not require constant feedback, for example file transfers, HTTP web pages which send all interactive data up-front, or streaming data that is uni-directional and does not depend on latency (i.e. media streaming with large buffers). TCP is not a great reliable transport layer for streaming a PC experience which requires instantaneous interaction and constant content updates. Using TCP limits the way in which you use the reliable transport. For specific data types, you do not need reliabiility, for example if you have 3 screen changes A,B, and C, throwing B out is perfectly fine as long as C screen change overlaps what is done in B. You cannot do this if you use TCP. So for certain types of networks (high latency) TCP actually limits you. TCP may be 'necessary' for the Citrix ICA protocol as the data that they present is generated on the client and if you miss one screen or graphical command it could affect all subsequent graphic commands.
MYTH: PCoIP claims bitmap remoting is the best way to deliver graphics
REALITY: Teradici claims host rendering with intelligent display decomposition & compression is the best way to deliver desktops (Citrix & Microsoft now focusing more and more on host rendering too)
True Bitmap remoting isn't the best way to deliver graphics. This is why PCoIP does NOT do bitmap remoting. As discussed in other blogs, host rendering with a display decomposition layer takes the display input and determines which way to encode a specific region of the screen. So for the example above, the 400x300 rectangle with black borders and white background, this isn't sent as a bitmap. This is decomposed into a white background and a black border, and sent compressed accordingly. One example is under some conditions a fully text based web page encodes using PCoIP protocol into the same amount of data as it would take to send the HTML for that given web-page. So the PCoIP algorithm is indeed an intelligent rendering function wrapped into the encoding. I would agree that for client rendering it does take years of fine tuning if you need to deal with all of the different graphic primitives of OpenGL, DirectX and whatever may come i.e. (Windows WDDM). However, if you let the rendering occur on the host and then deal with the final result with intelligent decomposition, you reduce the amount of time and effort you use supporting all of the different Graphic API's and spend more time concentrating on how to properly encode the data to provide the best user experience. PCoIP has several years of fine tuning already under it's belt, and several more years to come.
MYTH: PCoIP relies primarily on the server to do all the heavy lifting
REALITY: PCoIP protocol is a comprehensive solution with options for state-less hardware zero clients
Yes, true under certain situations, you may find that the PCoIP protocol uses more CPU than other protocols, but on average the PCoIP protocol uses less CPU resources than RDP based on VMware interactive benchmark testing results. So if you are satisfied with the scalability of your VDI using the RDP protocol, you should not have any issues with PCoIP (however, PCoIP will have a better user experience than the equivalent RDP VDI solution). On another note, the Host side PCoIP software has a much smaller memory footprint than other protocols mentioned.
In terms of being an incomplete protocol, it depends on how one defines incomplete. One of the main reasons for keeping the server side rendering is to ensure that the client or portal device is robust and has very little state. This allows zero-client hardware to be used to access a given PCoIP desktop. With the Zero client (stateless), customers can now reduce the number of IT dollars used to manage the 'user device'. Consider the zero-client a network appliance that you configure and forget about. So I would argue that PCoIP is as complete as any other remoting protocol, but with additional implementation options (optional hardware PCoIP).
Paul Helter
Principal Engineer, Teradici
Nov 16
Anonymous says:
Paul, apparently you have missed the fundamental techniques between TCP and UDP!...Paul, apparently you have missed the fundamental techniques between TCP and UDP!
How on earth would you be able to accelerate UDP?
You write the following."TCP is not a great reliable transport layer for streaming a PC experience which requires instantaneous interaction and constant content updates"..- TCP is the ONLY reliable protocol between TCP and UDP, if you have missed that part than please go back and do your homework again. On a congested WAN link, do you actually believe that UDP will provide any fault tolerance? Or do you think that security guys will allow incoming UDP traffic to devices within your network? Please think again.. I am not a network expert, and not a security consultant, but there are some things you would do. Leveraging UDP for a VDI environment just proves how narrow minded VMware is as a company, when the main purpose is to centralise the user behaviour -from all type of connections and destinations. Actually this proves that VMware are only thinking LAN not WAN!
It is actually a vague attempt to distract peoples mind and knowledge, as somebody else wrote. it is called brainwashing in some countries!
Nov 17
Anonymous says:
whew, good thing no one uses things liek DNS, VoIP, On-line gaming, IPTV or anyh...whew, good thing no one uses things liek DNS, VoIP, On-line gaming, IPTV or anyhting like that on a WAN or across the internet.
Nov 17
Anonymous says:
Brother, DNS packages are small not large, that is a difference, but why do you ...Brother, DNS packages are small not large, that is a difference, but why do you think a DNS Zone Transfer runs over TCP and not UDP? Cause you want to ensure that a delivery has been made. And here is the difference, it is about Quality (HDX) not Quantity (PCoIP).
Why do you think small Kerberos packages are UDP based ?? Answer: Quick and Dirty
Why do you think large Kerberos (AD) packages are TCP base when they reach a certain size? Answer: Quality, Ensurance, Delivery Control
Nov 16
Juan Rivera says:
Citrix HDX deals with latency and bandwidth constrained networks with a TCP tran...Citrix HDX deals with latency and bandwidth constrained networks with a TCP transport just fine. Check my previous blog about one of many things we do to support limited network connections:
http://community.citrix.com/display/ocb/2009/08/12/HDX+Learning+Series+-+Queuing+and+Tossing
Regarding the second point, I would like to point out to our readers that the key here is how much overhead the PCoIP adds compared to other solutions.
Nov 18
Anonymous says:
Thanks Juan for the explanation on Queueing and Tossing. PCoIP technology does...Thanks Juan for the explanation on Queueing and Tossing. PCoIP technology does a very similar function in conjunction with the application specific reliable transport that runs on top of the UDP network layer. I agree this method of operation is how a remoting protocol must deal with long latencies and bandwidth restrictions in a WAN environment when you are using TCP type reliable transport mechanisms. In the scenario you defined in your linked blog, if you receive any packet loss for 'frame 3', you still have to wait for the TCP reliable transport to resend that packet, and under those scenarios you wait the latency back from client to host and then host to client for that 'reliable transport' to resend that specific packet and unless frame 5 is a full update and does not rely on previous frames, the user experience is completely stalled for that timeframe. The only other option is to limit the bandwidth provided to the TCP network layer such that you never get into a packet loss situation. So the network protocol (TCP) is indeed affecting the capabilities of providing the best user experience as the remoting protocol must limit the bandwidth so that it doesn't cause the networking layer to adversely affect the user experience.
This is why the PCoIP protocol chose to manage reliability within the application layer while using UDP as a transport layer instead of letting the TCP transport layer manage reliability. This provides the flexibility of adjusting the bandwidth to the fullest potential (as well as let the user limit the max/min values) and is also why most people believe that PCoIP uses more bandwidth than other protocols. Yes it does use more bandwidth - Why? Because the PCoIP protocol can actively adjust the protocol bandwidth to the network's instantaneous congestion and capabilities.
As for overhead, like all other protocols you will have overhead. I do not know the CPU overhead specifics of HDX, but I do know that the memory footprint is much smaller for PCoIP, and the CPU overhead is similar or better than other remoting protocols such as RDP.
In response to the previous 'anonymous' reply - I hope I've answered your question about reliability - PCoIP performs reliability at the application layer and not at the network transport layer. In terms of security, TCP and UDP both operate within the network security layer (IPSec), so both have the similar security capabilities. Here's a good explanation on UDP for those who would like to know more: http://en.wikipedia.org/wiki/User_Datagram_Protocol#Applications
Paul Helter
Principal Engineer
Nov 18
Anonymous says:
"if you have 3 screen changes A,B, and C, throwing B out is perfectly fine as lo..."if you have 3 screen changes A,B, and C, throwing B out is perfectly fine as long as C screen change overlaps what is done in B. You cannot do this if you use TCP"
Which part of the process would throw out screen change B?
UDP works best when each packet contains the entire set of information (i.e. the entire frame for a video), because packets can be received out-of-order, you can ignore any 'older' packets. If each packet only contains sub-sections of the frame, then you need to receive all of the packets to know that you don't need certain ones.
If I'm understanding this correctly, the only scenario where UDP beats TCP, is if the network is 100% congested, and you need to start rejecting packet delivery. But even then, which part of the delivery process can decide which packets to reject if the data is within an encrypted packet. Only things which can understand (and decrypt) the protocol can do this, which is likely to be the two ends - hence the data has to be delivered anyway.
Nov 25
Anonymous says:
As some of use explained earlier, you guys are used to work on switched LAN's an...As some of use explained earlier, you guys are used to work on switched LAN's and not on WAN or high latency networks. This is the difference between practice and theory. On paper, your solution works fine, but if it that good, can you please pin point me to 1 customer that has PCoIP running over WAN with grate experience? I know at least 2 customers, in my city where I reside that say that PCoIP doesn't deliver what VMware promise...Now, on PAPER PCoIP is probably excellent, but not on practice..
With UDP you need to have the fault tolerance on another leverl, causing even more latency... and when you say "congested" actually you know like I know that a Ethernet LAN or any type of WAN for that part, doesnt have to have that level of congestion to cause issues....
Go ahread, I challenge you., prove me wrong.
Nov 14
Anonymous says:
Must be getting cold in Canada. Explain why PCoIP suck so much bandwidth and why...Must be getting cold in Canada. Explain why PCoIP suck so much bandwidth and why it doesn't support Terminal Services?
Nov 16
Anonymous says:
I've read multiple comments mentioning the encryption as a problem because of "W...I've read multiple comments mentioning the encryption as a problem because of "WAN optimizers".
It would be irresponsible for an IT department to turn off encryption on the desktop. Network streams can potentially be intercepted at many different points in the network. Because you're displaying a full desktop, someone could potentially capture EVERYTHING being done by that user in a completely passive nature. If that traffic is unencrypted they could watch that user indefinitely, capturing usernames/passwords, patient information (HIPPA violation), Financial/Accounting information (SOX/GLB/SEC violations), customer payment information (PCI violation), Intellectual Property (trade secrets), etc.
Because end users typically don't know anything about how their desktop (or application) is presented to them, they would have no knowledge that this would even be possible. It thus falls on IT to make sure that none of those potential problems could exist in the first place. As IT professionals we would not be performing our due diligence for protecting the corporate infrastructure and corporate data if we purposely disabled such a valuable security mechanism for the potential bandwidth savings.
The encryption mechanism is essential, and the lack of a "disable" option is designed to prevent IT administrators from accidentally shooting themselves in the head.
Nov 16
Timothy Bardzil says:
Agree 100%. Encryption is essential in many IT environments. That is precisely w...Agree 100%. Encryption is essential in many IT environments. That is precisely why Branch Repeater is designed to work with the native ICA encryption. For IT departments that means maximum acceleration without compromising end-to-end security.
Most of the comments above refer to PCoIP which lacks this sort of integration with 3rd party WAN acceleators.
Nov 17
Anonymous says:
I'm confused by the discussion of WAN acceleration in the context of PCoIP. Nor...I'm confused by the discussion of WAN acceleration in the context of PCoIP. Normal WAN acceleration uses one of three basic techniques to improve performance - 1) Caching, 2) Compression and 3) TCP "optimization" like window sizing or local ACKs. It seems like #1 is going to help with anything that is repetitive (which would be useful for repeated bitmaps) but is also easy to do on the client end and less useful in the network since client traffic is usually fairly unique (I'll give you standard window and menu bitmaps however). #2 is pointless for a protocol that is already designed specifically around compression. #3 is basically designed to overcome the limitations of TCP at high latencies, something that UDP doesn't have a problem with. It seems to me like protocols like PCoIP or similar VoIP protocols are already designed with most of these features in mind (assuming you add client caching) and remove the need for the additional complexity of WAN accelerators. Am I missing something fundamental here?
Nov 17
Timothy Bardzil says:
For #1, the primary benefit is to be able to cache content across multiple users...For #1, the primary benefit is to be able to cache content across multiple users sessions. Imagine a branch office with 50 employees who are all accessing a standard corporate desktop and common set of applications. Not only is there repetitive window and bitmap data but also repetitive bulk data in the form of print jobs, client rendered videos and file transfers. I cannot speak for PCoIP but I know that with our own Branch Repeater product we see typical bandwidth savings of more than 50% and in some extreme cases, such as repetitive printing, reductions of more than 99%. This is not to say that caching on the client is not important -- it is and we have been doing this for a long time. However, for many customers with congested, low bandwidth connections, these additional bandwidth savings can save them the expense of having to upgrade their WAN circuits.
The one technique you might have left off your list 4) QoS and Traffic Prioritization. When network congestion does occur it is important to prioritize interactive traffic, such as display data over bulk flows. I don't know enough about PCoIP to know if it tunnels things like print jobs within the protocol like we do with the print virtual channel within ICA. If it does, a properly integrated WAN optimization solution can separate out the traffic streams and prioritize each appropriately.
Dec 14
Anonymous says:
http://www.uggboots-online.net http://www.uggboots-online.net ugg boots online...http://www.uggboots-online.net
http://www.uggboots-online.net ugg boots online
http://www.uggboots-online.net ugg boots
http://www.uggboots-online.net ugg online
http://www.uggboots-online.net ugg boots snow
http://www.uggboots-online.net snow ugg boots
http://www.uggboots-online.net outlets ugg
http://www.uggboots-online.net/UGG_Amelie_Sandals.html UGG Amelie Sandals
http://www.uggboots-online.net/UGG_Bailey_Button_boots.html UGG Bailey Button boots
http://www.uggboots-online.net/UGG_Classic.html UGG Classic
http://www.uggboots-online.net/UGG_Classic_Argyle_Knit.html UGG Classic Argyle Knit
http://www.uggboots-online.net/UGG_Classic_Cardy_Boots.html UGG Classic Cardy Boots
http://www.uggboots-online.net/UGG_Classic_Crochet_Boots.html UGG Classic Crochet Boots
http://www.uggboots-online.net/UGG_Classic_Mini_Boots.html UGG Classic Mini Boots
http://www.uggboots-online.net/UGG_Classic_Short_Boots.html UGG Classic Short Boots
http://www.uggboots-online.net/UGG_Classic_Tall_Boots.html UGG Classic Tall Boots
http://www.uggboots-online.net/UGG_Classic_Tall_Metallic_Boots.html UGG Classic Tall Metallic Boots
http://www.uggboots-online.net/UGG_Coquette_Sandals.html UGG Coquette Sandals
http://www.uggboots-online.net/UGG_Classic_Tall_Metallic_Boots.html UGG Classic Tall Metallic Boots
http://www.uggboots-online.net/UGG_Dakota_sandals.html UGG Dakota sandals
http://www.uggboots-online.net/UGG_Gypsy_sandals.html UGG Gypsy sandals
http://www.uggboots-online.net/UGG_Halendi_sandals.html UGG Halendi sandals
http://www.uggboots-online.net/UGG_Kids_Boots.html UGG Kids Boots
http://www.uggboots-online.net/UGG_Morocco_sandals.html UGG Morocco sandals
http://www.uggboots-online.net/UGG_Nightfall_Boots.html UGG Nightfall Boots
http://www.uggboots-online.net/UGG_Sandals.html UGG Sandals
http://www.uggboots-online.net/UGG_Sundance_boots.html UGG Sundance boots
http://www.uggboots-online.net/UGG_Sundance_Grab_bags.html UGG Sundance Grab bags
http://www.uggboots-online.net/UGG_Ultra_boots.html UGG Ultra boots
http://www.uggboots-online.net/UGG_Ultra_Short_boots.html UGG Ultra Short boots
http://www.uggboots-online.net/UGG_Ultra_Tall_boots.html UGG Ultra Tall boots
Add Comment