A Quick test of Accelerated Networking in Azure

Ben SteginkAzure, IaaS1 Comment

Intro

Recently, Microsoft announced Accelerated Networking in Azure. At a high level, they took FPGAs in the Azure Data Center, connected them to the Azure IaaS hosts over PCIe and use that for networking as opposed to the standard way of networking using Hyper-V. You can view a diagram and an article from Microsoft on how it all works here – https://docs.microsoft.com/en-us/azure/virtual-network/virtual-network-accelerated-networking-portal

The Setup

The purpose of this post was actually to do an actual comparison of these speed improvements you can by using Accelerated Networking. For this test, I spun up the exact same VMs (DS15_v2) in the same region (West Central US) – currently the only configuration supporting Accelerate Networking in the US. I also put all the files on the temp drive as this is a very fast drive and I wanted to try to eliminate any chance of the disk speed being a bottle neck to the file copy.

To do the test I spun up two servers with Accelerated Networking and two servers without as you see the most improvement using accelerated networking going server to server. I then took a 5.5GB ISO to copy between VMs as well as made a bunch of copies of that ISO to copy multiple large files, and archived them into a .zip file to create on massive file. Below are the results.

The Tests

Here are the numbers from the tests I ran. I only did a single test for each file transfer.

Accelerated Networking (max 25 Gbps)

Single 5.5 GB File Copy

This took about 4 seconds to copy between VMs using accelerated networking. You’ll see in this screen shot that at 71% Windows still hasn’t even calculated the time remaining.

44 x 5.5 GB Files (242GB total)

You can see this one was a little slower, probably due to copying several files. But still really quick with a transfer rate of about 534 MB/s and taking around 9 minutes.

1 x 121 GB File

This one speeds went right back up. The entire file copied in just a few minutes. It his speeds of just above 1.15GB/s and did slow down to around 830MB/s the last third of the file copy. When I didn’t make it up to 25 Gbps, I did get north of 10Gbps, which is still significantly faster than the “normal” network as we’ll see in the next section.

“Normal” Networking (max Extremely High*)

For “normal” networking, your speeds to vary based on the VM size/category. Since we used DS15’s to test accelerated networking, I used them to test normal networking as well. If you use a lesser VM, you’ll see your networking speeds drop off as well.

Single 5.5 GB File Copy

The initial test of the same 5.5GB file showed while the network speed is “Extremely High” we only reached about 23% of the speed we got when using accelerated networking.

44 x 5.5 GB Files (242GB total)

When copying several files, the speeds actually came a little closer together. However, “Normal” networking was still less, but did achieve a speed of about 70%. These files copied at around 375MB/s and took a little over 10 minutes. I didn’t run multiple tests with accelerated networking to see if I could get a larger differential, but it does show that just like a normal networking, there isn’t going to be a guaranteed speed for data transfer. An interesting thing to note is how close in speed the transfer rate was between the single 5.5 GB files and copying 44 x 5.5GB files.

1 x 121 GB File

The final test brought the large speed difference to light again as it did with the first file copy. Once more we see the “normal” networking only getting between 25% – 40% of the speed of accelerated networking and getting right around 3 Gbps.

* I couldn’t find any hard numbers, Microsoft simply states extremely high – https://docs.microsoft.com/en-us/azure/cloud-services/cloud-services-sizes-specs . It also remains undocumented in the network limits here – https://docs.microsoft.com/en-us/azure/azure-subscription-service-limits#networking-limits

Conclusion

In Short, accelerated networking is definitely noticeably faster when transfer large files. A good followup test would be to do something similar transfer a bunch of small files. While Microsoft also claims accelerated networking can achieve speeds of up to 25Gbps per second, I didn’t see anything near that speed. It also appears, that while no max speed is specified for the normal network, I did seem to hight a ceiling around 375 MB/s (3 Gbps). It doesn’t mean that’s the limit as I never hit the limit with Accelerated networking either, but if you need to transfer a lot of data between large VMs (for now), accelerated networking is worth looking at. We’ll have to see if this eventually comes to any of the smaller VMs, or if this remains as a premium feature on the high performance VMs.

  • Gabriel Silva

    Hey Ben – This is Gabe Silva, Sr PM with Microsoft responsible for Accelerated Networking. First off, thanks for this blog! Great read and great results! Just wanted to point out that the reason you are not seeing 25Gbps is because the storage is the bottleneck. We use a tool called ntttcp that tests only the network. You can find the tool here, https://gallery.technet.microsoft.com/NTttcp-Version-528-Now-f8b12769, if you care to try.

    Thanks,
    Gabe
    gsilva@microsoft.com