The Data Binding, the Invoke, and the Deadlock

When it comes to Windows Application programming, there are a number of things to be considered.

1. Data Binding – the ability for the windows control to automatically refresh its UI when the underlying data changes. This feature saves developer’s time from writing code to update the UI when the underlying data change.
2. UI Thread Invoke – When an UI needs to be updated, the updating code has be execute by original thread that create the UI (i.e. UI Thread, or usually the Main Thread). For example:

private void OnDataRowChanged(DataRow oDataRow)
{
    if (this.InvokeRequired)
    {
      this.Invoke(new OnDRChangedDelegate(OnDataRowChanged), oDataRow);
      return;
    }

    //Do Your UI Update ...
}

1. Deadlock – In a multiple threaded application, when thread A waiting to acquire a resource and never able to get it because thread B is holding this resource, while it is waiting for thread A to release its resource. You will end up having two threads waiting and your program hangs. This can happens for more than two threads locking each other resources.
   
   
   

 

The Problem

In a multi-threaded application, there are many threads interacting with the application data. In my case, it is a DataSet. A DataSet is an ADO.NET (old) technology that acts like a in memory database. It contains tables, columns, constraints, relationships and etc. To ensure the program is thread safe, we have to lock the DataSet before modifying the data using the lock keyword in C#.

This all sounds good until we start doing data blinding. In my case, I have a number of grids (Infragistic UltraGrid) blinded to the DataSet. In the beginning, everything look great. Data are loaded by thread to the data model and grids automatically updated.

Here is the model that causes problem:

Each time when our server updates a piece of data to a client DateSet, it creates a .NET remoting thread (represent by the green), and update the DataSet. To ensure we are thread safe when updating the DataSet, we have a lock to prevent more than one thread updating (writing) the DataSet.

Unfortunately during QA, we occasionally see the UltraGrid display a big red X. The entire gird basically crashes and it no longer able to display data.  The exceptions we got includes NullReferenceException, IndexOutOfRangeException and etc.  Initially, we thought that was a Infragistic bug in the UltraGrid, but in the end, we realized it is another type of Cross Thread operation. For example: InvalidOperationException was unhandled (Cross-thread operation not valid: Control ‘XXX’ accessed from a thread other than the thread it was created on.)

To prevent cross thread operations, we decided to ensure we called the ISynchronizeInvoke.IsInvokeRequired when we updates the DataSet.

Revised model (Deadlock on Invoke and Lock DataSet):

For the first couple of runs, the application seems to be executing correctly using the revised model. However, we got the application to hang.

From the diagram, it is probably obvious  that there is a deadlock from the pictures. However, it isn’t obvious when the order of locking and invoking calls are in different modules.  Because we didn’t standardize how we should do Invoke call and lock the data set first, we end up having random deadlocks on the Invoke call, and at the lock statement block.

Where is the dead lock?

In case the diagram is not obvious to some of the readers, here is what happening. Assuming you have an UI Controls created by only one thread, called the UI Thread (Main Thread), when Thread 1 called ISynchronizeInvoke.Invoke(), the UI Thread is locked to execute the Thread 1 request. At the same time, Thread 2 has locked the DataSet which prevents other threads updating it. When the Main Thread try to lock DataSet, it has to wait for Thread 2 to exit the lock block. At the same time, Thread 2 is waiting for the UI Thread to complete so that it can do the Invoke(). Since 2 threads are waiting for each other resources, we have a dead lock.

Solving The Problem

This model may not be the best solution, but it is simple to understand and it works:

In this model, all I did is to ensure all threads call Invoke() before locking the DataSet.

This design ensures we only have one thread access to the Lock. In our case it is the UI Thread. Whenever called the Invoke() on a Windows Form, your execution is passed to the UI Thread.

Now, you may wonder, why do I still need to lock the DataSet when there the Invoke there there.  From this simplistic diagram, we don’t.

But, you may have N number of Threads that didn’t call the Invoke for whatever the reason. An application can be complicated. To be safe, always lock the DataSet before applying changes.

 

 

 

 

 

How to Find a Deadlock?

You probably don’t want to ship a software that hangs in front of your customs, so you want to find these deadlocks before your customers find them for you.  Well, there is no easy way to determine where are the deadlocks in your application. Here is a couple of suggestions that may help:

• Peer Code Review – have someone double check your check before check in the code. Check for possible deadlock by examining where lock, monitor class, invoke class are used.
• Use a Deadlock Monitor – using a custom locking mechanism to determine if there is a possible dead lock.
• Have a lot of tests – this can be unit tests, integration tests and your manually user tests. Whenever you see the application stop responding, report it and try to solve it.

Find Your Dead Lock using VS.

If you do find your application stops responding, what can you do? First, determine if your application is actually “working” by checking its CPU usage. If it is around 100% for one of your core, your application may be working hard or got simply running an infinite loop. This is NOT a deadlock!

In this case, your best tool would be using Visual Studio! Assuming you have all the debug files (.pdb) and source code for your program:

1. Debug->Attach To Process, and attach to your non-responding application.
2. Click on the pause button or Debug->Break All to stop the application
3. Debug->Windows->Threads

A deadlock usually happen between two threads that hold on two resources, so you need to check where your executing threads has stopped. In the following example, I have a number of threads running, but I am only concerning my owns. These are the last two items in the Threads grid below.

1. Double click on the Main Thread and the Worker Thread to see their executing locations.

For example:

Main Thread executing location

1. Notice the “green” highlight represent were is your thread executing.
2. If you hit F5 (run) to un-pause the thread, and hit pause again, your thread execute would most likely remind at the same place.

Worker Thread executing location

So, at least we know these two threads are trying to get some resources. However, how can you tell if they are holding each other requesting resources?

In this example, we know that the Main Thread waiting, so the Worker Thread have to wait at the “this.Invoke()” line. However, how do you know if the Worker Thread lock the m_oDataModel?

The only way to find out is to trace the executing code through the call stack. (If someone have a better way to find a dead without going though the code, please let me know!!!)

1. Double click on the Work Thread
2. Debug->Windows-> Call Stack

By tracing through the stack, I soon discover that the one of the call stack got a lock on m_oDataModel. Luckily in my case, I only have 2 visible stacks for this thread. If you have more, you will have to walk though them.

Ah, from the above, I see that I called lock(m_oDataModel), follow by calling the AddDataWorker() method.

Once you find out there is a dead lock, you will have to resolve it. Either by fixing the order of the locks or provide a better architecture that prevent developers run into this situation.

Sample Deadlock Code

Sorry that I don’t have a web space to store code file. I can only post the source code as it is. In this example, I have a form that has one button. If you click on the button, a dead lock will occur.

public partial class Form1 : Form
{
    private DataSet m_oDataModel = new DataSet();
    public Form1()
    {
        InitializeComponent();
        m_oDataModel.Tables.Add("MyTable");
        m_oDataModel.Tables["MyTable"].Columns.Add("SomeKey");
        m_oDataModel.Tables["MyTable"].Columns.Add("ItemName");
    }

    private void button1_Click(object sender, EventArgs e)
    {
        ThreadPool.QueueUserWorkItem(new WaitCallback(AddData));

        //Similate doing something
        Thread.Sleep(1000);

        lock(m_oDataModel) //Wait for lock forever (deadlock)
        {
            m_oDataModel.Tables["MyTable"].
            Rows.Add(Guid.NewGuid(), "Some Data");
        }
    }

    private void AddData(object oState)
    {
        lock(m_oDataModel)
        {
            AddDataWorker();
        }
    }

    private void AddDataWorker()
    {
        if (this.InvokeRequired)
        {
            //Wait for Main Thread to be free forever (deadlock)
            this.Invoke(new Action(AddDataWorker));
            return;
        }

        //Update a textbox
        this.textBox1.Text = m_oDataModel.Tables["MyTable"].
        Rows[0]["ItemName"].ToString();
    }
}

Conclusion

As you get more experience and understand the structure of your application, the easier it will be for you to identify a deadlock. Deadlock isn’t just cause by the “lock” keyword in C#. It can be an Invoke() call, a read/write lock on a file, an database transaction lock and etc.  For those who read though the article to this end, thanks for reading and I hope this post helps you solving your deadlock issue.