Wednesday, March 16, 2011

7000 Concurrent Connections With Asynchronous WCF

It’s rare that a web service has some intensive processor bound computation to execute. Far more common for business applications, is a web service that executes one or more IO intensive operations. Typically our web service would access a database over the network, read and write files, or maybe call another web service. If we execute these operations synchronously, the thread that processes the web service request will spend most of its time waiting on IO. By executing IO operations asynchronously we can free the thread processing the request to process other requests while waiting for the IO operation to complete.

In my experiments with a simple self-hosted WCF service, I’ve been able to demonstrate up to 7000 concurrent connections handled by just 12 threads.

Before I show you how to write an asynchronous WCF service, I want to clear up the commonly held misconception (yes, by me too until a year or so ago), that asynchronous IO operations spawn threads. Many of the APIs in the .NET BCL (Base Class Library) provide asynchronous versions of their methods. So, for example, HttpWebRequest has a BeginGetResponse / EndGetResponse method pair alongside the synchronous method GetResponse. This pattern is called the Asynchronous Programming Model (APM). When the APM supports IO operations, they are implemented using an operating system service called IO Completion Ports (IOCP). IOCP provides a queue where IO operations can be parked while the OS waits for them to complete, and provides a thread pool to handle the completed operations. This means that in-progress IO operations do not consume threads.

The WCF infrastructure allows you to define your operation contracts using APM. Here’s a contract for a GetCustomer operation:

[ServiceContract(SessionMode = SessionMode.NotAllowed)]
public interface ICustomerService
{
[OperationContract(AsyncPattern = true)]
IAsyncResult BeginGetCustomerDetails(int customerId, AsyncCallback callback, object state);
Customer EndGetCustomerDetails(IAsyncResult asyncResult);
}

Essentially ‘GetCustomerDetails’ takes a customerId and returns a Customer. In order to create an asynchronous version of the contract I’ve simply followed the APM pattern and created a BeginGetCustomerDetails and an EndGetCustomerDetails. You tell WCF that you are implementing APM by setting AsyncPattern to true on the operation contract.

The IAsyncResult that’s returned from the ‘begin’ method and passed as an argument to the ‘end’ method links the two together. Here’s a simple implementation of IAsyncResult that I’ve used for these experiments, you should be able to use it for any asynchronous WCF service:

public class SimpleAsyncResult<T> : IAsyncResult
{
private readonly object accessLock = new object();
private bool isCompleted = false;
private T result;

public SimpleAsyncResult(object asyncState)
{
AsyncState = asyncState;
}

public T Result
{
get
{
lock (accessLock)
{
return result;
}
}
set
{
lock (accessLock)
{
result = value;
}
}
}

public bool IsCompleted
{
get
{
lock (accessLock)
{
return isCompleted;
}
}
set
{
lock (accessLock)
{
isCompleted = value;
}
}
}

// WCF seems to use the async callback rather than checking the wait handle
// so we can safely return null here.
public WaitHandle AsyncWaitHandle { get { return null; } }

// We will always be doing an async operation so completed synchronously should always
// be false.
public bool CompletedSynchronously { get { return false; } }

public object AsyncState { get; private set; }
}

Now we’ve got an AsyncResult we can implement our APM based service by implementing ICustomerService:

[ServiceBehavior(
InstanceContextMode = InstanceContextMode.PerCall,
ConcurrencyMode = ConcurrencyMode.Multiple)]
public class CustomerService : ICustomerService
{
public const int DelayMilliseconds = 10000;

public IAsyncResult BeginGetCustomerDetails(int customerId, AsyncCallback callback, object state)
{
var asyncResult = new SimpleAsyncResult<Customer>(state);

// mimic a long running operation
var timer = new System.Timers.Timer(DelayMilliseconds);
timer.Elapsed += (_, args) =>
{
asyncResult.Result = GetCustomer(customerId);
asyncResult.IsCompleted = true;
callback(asyncResult);
timer.Enabled = false;
timer.Close();
};
timer.Enabled = true;
return asyncResult;
}

public Customer EndGetCustomerDetails(IAsyncResult asyncResult)
{
return ((SimpleAsyncResult<Customer>) asyncResult).Result;
}

private static Customer GetCustomer(int customerId)
{
return new Customer(customerId, "Mike_" + customerId);
}
}

We’re mimicking a long running IO operation by using a timer. I believe that the Timer also uses an IO completion port, but don’t quote me on that. When WCF invokes BeginGetCustomerDetails we first create a new SimpleAsyncResult with WCF’s state object. WCF will pass the AsyncResult to the EndGetCustomerDetails method when the timer completes, so we can use it to pass any response state. In our case this is an instance of Customer.

Next we set up a Timer and attach a closure to the Elapsed event. When the timer’s Elapsed event fires we create a customer instance, pass it to our AsyncResult and then pass the AsyncResult to WCF’s callback function.

After the BeginGetCustomerDetails method completes, WCF returns its thread back to the WCF thread pool so that it can service other requests. Ten seconds later, the operating system posts an item on to the IOCP queue, a thread from the pool picks up the item and executes the continuation. This in turn calls WCF’s callback which in turn executes EndGetCustomerDetails. WCF then packages up the Customer as a SOAP response and returns it to the client.

Only a tiny fraction of that 10 seconds has actually occupied a thread, so we should be able to make thousands of concurrent calls to this service.

In my tests, this service has been able to handle a little over 7000 concurrent connections.

The code is here: https://github.com/mikehadlow/Mike.AsyncWcf

Just follow the README instructions to run the tests and try it out for yourself.

16 comments:

Anonymous said...

Mike,
 
If you create your service proxy using the /async flag to svcutil.exe, all this is taken of for you: what are the advantages in your manual approach?
 
Regards

Mike Hadlow said...

Hi Anonymous,

This about the WCF server. svcutil creates a client proxy, so it has no effect at all on how the server is implemented.

You can write a synchronous server and then get svcutil to create you an asynchronous client. This will make your client asynchronous, but it will not help your server at all. Does this make sense?

Richard OD said...

Hi Mike,

I/O Completion Ports are the way to go for scalable server side I/O.

The Timer class does not use an I/O completion port (why would it- it's not doing I/O?) It uses a Threadpool thread that is used by all Timers. When the Timer is due this thread wakes up and calls the Threadpool's QueueUserWorkItem, which is how your callback method ends up getting called.

All of the things you mention in your post are discussed in great detail in CLR via C# 3rd Edn.

Paul Hadfield said...

Hi Mike,

Just trying to get my head around this. So if I understand your response to Anonymous correctly, when you run svcutil.exe against your service you will end up with client side async methods on both your BeginGetCustomerDetails and EndGetCustomerDetails calls?

- Paul.

Mike Hadlow said...

Hi Paul,

Nope, the WSDL looks identical irrespective of whether you implement your server synchronously or asynchronously. So the async option on svcutil is entirely independent of the server implementation.

Unknown said...

Hi Mike

I don't think the locking is necessary on SimpleAsyncResult.

The IsCompleted property is bool, and reads/write of bools are an atomic operation. Similarly if the generic Result is a reference type (or a one of the value types that are atomic) then that is also atomic.

See here:
http://msdn.microsoft.com/en-us/library/aa691278.aspx

GC said...

So I can see how this server method is async because you are using the timer. How do you make the server async for real life scenarios without spinning up another thread?

GC said...

...like reading from a data source or whatever

Mike Hadlow said...

Hi GC,

Take a look at the proxy service in the example code. I didn't mention this in the post, but you can browse it on github here:

https://github.com/mikehadlow/Mike.AsyncWcf/blob/master/Mike.AsyncWcf.Proxy/CustomerProxyService.cs

You can follow the same pattern for any IO API that exposes APM methods (BeingXXX & EndXXX). SqlCommand has BeginExecuteReader/EndExecuteReader that you can use for asynchronously reading data from a database.

As I pointed out in the post, these operation use IOCP so the WCF worker threads are free to handle other requests while the IO operation is in progress.

Anonymous said...

Why should i make the work to build my wcf Server asynchronous? How about ConcurrencyMode.Multiple? I can call as many Methods at the same time as I want. I don't see any sense..

Anonymous said...

ConcurrencyMode.Multiple allows concurrent threads against the service, but the client thread(s) that called the service are blocked waiting for a response because the service operation is still synchronous. Using the APM for service operations allows the client to not block, thus ConcurrencMode.Multiple with APM service operations allows for concurrent, asychronous requests.

Anonymous said...

in Mike.AsyncWcf.Client
var servicePoint = ServicePointManager.FindServicePoint(configuration.ServiceUri);
giving error like value can not be null
paramenter:
name = address

Anonymous said...

Hello , how to call this method with javascript with ajax or something like that ...?

Anonymous said...

how to call this method with ajax or jquery

Anonymous said...

to call this via jquery you need to add it as a service reference on your solution.

Unknown said...

What if my service uses session (SessionMode.Required),does it still can handle a large number of concurrent connections.
I mean is it gonna effect the performance?