Deadlock Again

Three years ago, our QA team was testing our latest web services (WCF). What they discovered was that after making some web service requests, the web service would no stop responding. Also, after each request, the web service memory usage would go up.  However, no one has a clue on why the web service would stop responding. So on that morning, I spent an hour to figure out what was going on.

Great, it is another deadlock!

In the world of software development, multi-threaded programming is probably one of the best strategies making your software performance scale. However, troubleshooting deadlock is far from trivial. The more complicated the software, the higher the chance you run into deadlocks if you choose to do multi-thread programming.

Chances are, even if you think you were writing a single thread program,  the frameworks or libraries that your code calls could execute their instructions on another thread.

There are many guidelines on the internet shows you how to avoid deadlocks.  Notice I said the word "guidelines," there aren't any specific pattern or algorithm that we can copy to ensure your multi-thread program never run into deadlocks. Every application is different, and you need to understand how your code is executed to troubleshoot deadlock issues.

Deadlock Scenarios

I am sure there are many ways your application get a deadlock. Here are the common ones that I often encountered.

Single Application internal deadlock

• Deadlock occurs within the application code itself. This is the classic deadlock that you probably learned from your computer science course.

Single Application Lock Combine with Table Access Deadlock

• Application Thread 1 performed Insert/Update on database TableA, but has not commit yet, because it was waiting to access a memory lock. Meanwhile, Thread 2 has acquired this memory lock, but it is waiting to on TableA to be unlocked.
• To make this scenario more complicated, any database tables that have foreign key constraints linking to TableA could potentially cause a deadlock in chained relationships.

Single Application’s multiple DB Transactional deadlock

• Application Thread 1 performed insert/update on database TableB, and then waiting for access to TableA. Meanwhile, Thread 2 has already locked on TableA, and is waiting for TableB to unlock.
• Note that this occurs only when an application uses multiple database connections or database transactions with different threads. There is no application memory lock involved either.

Multiple Applications DB Transactional deadlock

• Application1 performed insert/update on database TableA, and it tried to access TableB, but Application2 already has a lock on TableB, but it is also stuck on waiting for TableA.

Tips to Avoid Deadlocks

• Pick the Right Framework
  Try to use framework that does not require you to worry about locks that fits your application’s needs.
  For example, LMAX Disruptor uses ring buffer to avoid using locks, and yet it provides high message process throughput.
  Ref:
• Avoid Waiting on Locks Forever!
  For example, in C#, instead of using lock(), or Monitor.Enter(), try using the following locks that allow you to stop waiting:
  
  Monitor.TryEnter()
  AutoResetEvent.WaitOne(int32)
  ReaderWriterLockSlim.TryEnterReadLock(int32)
  
  
• Use the Framework in a Thread-Safe Manner.
  This is a reminder for the developers to always check how to use a framework properly.  Most of the library operations that we are not thread safe.
  For example, did you know that if you use the same Entity Framework context object is thread-safe when access by multiple threads?
• Choose to use a Thread-Safe Class
  For example, pick ConcurrentDictionary<TKey, TValue> over the standard Dictionary<TKey, TValue>.
• Follow Coding Standard and Best Practice
  For example, in the project that I worked on, developers are required to a custom Data Access Manager that can detect possible deadlock scenario if multiple threads access the same database transaction object, or a thread has more than one opened database transactions. It would then throw an exception in debug mode.
  
• Test, Analysis, Fix, and Test Again
  There are many ways to hit a deadlock. So make sure you test your code before putting it to production.

Lastly, remember there is always a trade off when you pick one design over the other design. Always consider about what are the priority of your software characteristics before you pick the design.