How to evolve to Sprint Planning in Scrum (Agile Development)

Approx. 2 years ago, my development team began to switch from the traditional Waterfall model to the agile development. In our case, we choice Scrum as our ideal development model.  Switching to agile isn’t an one night process. It took a good year to get everyone comfortable with iterative development.

Since there are already many resources about agile development, I am not going to write in details on what agile development is. Instead, I would like to write about how our development planning process evolves from the waterfall model to the agile sprint planning model.

The following software planning process are listed in chronologic order.

1. Team Lead Prepares the specifications
In this model, developers (incl. myself) were simply given a feature specification document, in return developers provided rough estimate on how long the task may take to complete the projects. Of course, the estimated time had to be somewhat reasonable.  The planning meeting usually involves just the team lead and the assigned developer(s).  In the planning, the general strategy of how resolve the problem is discussed. The project time estimate could range from 2 weeks to a month. The actual length of time to accomplish a task can vary from a couple of day off from the estimate to more than a month.

2. Large tasks are Broken Down
Instead of having a high level specifications overview, developers were given smaller tasks that needs to be accomplished. Often developers may not see the “big picture” of the reason of completing the assignments. The team lead provides his estimated time on how long it should take the developer to accomplish the tasks.

3. Effort Points
The development team learns about Effort Points or Story Points. This was the most challenging part of going into agile development. How do you estimate the effort points. The thing about effort point and time estimate is that given a task and its description, developers should agree upon the same effort point regardless of how fast they code.  An experienced developer may need one day of work to accomplish a task with one effort point. A newbie may need two days of work to accomplish the same task.

Notice my team choice to talk about task, instead of a user story. The team lead estimate the  story point by himself.

4. A Standard Task for 1 Effort Point
It was hard to estimated effort point per task when there are no standard. We needed to come up with a standard of unit. In our product, a typical feature involves creating a new database table definition, add new server side code, add new client code, update the data model, and create a new UI. This set of sub task was used to represent 1 effort point.

5. Small Team instead of Large Team
The development team were broken down into 3 teams. The original team originally has 12 developers. It was broken down such that each team is about 3 to 4 developers focusing on different aspects of the enterprise product suite.

The original team lead now becomes the master team lead of 3 sub team leads.

6. Effort Point and Sprint Planning
In our sprint planning meeting, all developers in the team have to come up with an estimated effort points for a given tasks. Everyone had to agree on the same amount of effort points. To do this, we use planning poker.  Whenever there is a discrepancy of the estimated effort points among the developers, the developers who gave the highest and the lowest effort points would need to explain to the team why they chose the selected card. Doing this has three advantages:

• It allows the developer to express their opinion, or at least force them to speak up.
• It allows exchange of knowledge within the team
• It allows everyone to see other developers’ expectations

After the two developers explain their concerns, everyone get to revolute the efforts and come up with a new estimate. Hopefully everyone will come to the same effort point.

It is actually fun to play planning porker.

Remember, there are no shame disagree other developer estimates. If you truly believe your estimate is correct, you can keep putting a different number than the other does. However, usually majority rules at the end because you are only given a limit amount of time to do the estimate.

7. Two Weeks Sprint + Demo
Instead of doing planning irregularly, the teams had  two week sprints. A sprint’s duration is two weeks. The first day is the sprint planning, and the last day is the sprint demo. That’s right, our developers need to demo what they had done to an audience on what they did during the sprint. The good news for us is that we only demo to the internal people and not customers from outside the company. If the demo crashes, it is not the end of the world. The demo has 3 advantages:

• Ensure developers show us what he/she promise to deliver
• Developers need to test their code well enough so that it works in the demo
• It allows developers to showcase what they have accomplished.

8. Making The Sprint More Efficient
Instead of having all three teams running on the same sprint cycles, it was broken down so that each time do the planning and do the demo on different day of the week. This way the attendees do not have to watch all three demos on the same day.

9. Need Better Tools
Instead of having everything on paper, we start using TFS and its planning tool. The tool keep track on user story, the tasks, the estimated time and the remaining team. This is how the leads keep track if the developers are on track on completing the task.
At this point developers no longer do effort point estimate, instead, we give our estimated time.

10. More Tool
TFS sucks in many usability aspect, as a result we got Urban Turtle to help us.

11. We Need Feedback
With the introduction of Burn Down chart, we now have constantly feedback on how close are we completing our task in time.

12. Dealing with Unknown Factors
I was involved in a project that has many unknowns. In the first 8 to 10 sprints the team under estimated the tasks. Many issues where discovered during the development and were not foreseen in the sprint planning. These new issues either needs to be fix in order to unblock the developers, or they can be push to the next sprint so that they wouldn’t disrupt the current sprint.
After the first  8 to 10 sprints, the  team learned to be pessimistic on their time estimate. It is actually better to over-estimate, so that you are able to complete your committed tasks.

Conclusion

My company’s development methodology continues to evolve. There are many advantages when using agile development, while there are pitfalls we need to be careful of. Here are positive things and pitfalls that I saw in our development model.

Positives:

• Everyone is committed to accomplish the assigned task by the deadline
• Any issues that were discovered during development can properly be addressed in the sprint planning meeting, instead of allowing the developers tackle the problem without meeting with others.
• It is flexible, because every two weeks we have a chance revaluate our task priorities.
• If anyone need to take a sick day, vacation, or leaves the company, it has less impact to the entire projects because the user story is broken down into tasks. A task has to be accomplish at minimum in 2 weeks. If the task takes more two weeks, it needs to be broken down further.

Pitfalls:

• It is temping to under estimate because of peer pressure or their subconscious competitiveness against other developers.
• It is easy to schedule customer’s requested user stories, and forgot developers may want to do their own projects such as writing more unit tests, develop new tools, optimize or improve existing feature quality, preparing training session, which helps improve the development process.

To avoid under estimate the project time, the burn down chart and the retro respective helps the team to recalibrate their thought on the estimate, so that they can do better estimate for the next sprint.

The team can add personal project as a task so that developers get the chance to do their own stuffs.

In LINQ, is Take(n) really the same as Select Top n?

Today I was doing some bug fixes and I ended up wanting to do something like “Select top n From ..” in Linq.  I realized that there isn’t really a Top keyword. Ideally I would like something like this:

var result = from i in itemList
where SomeCheck(i)
select top 10 of i;

Of course,there is no TOP in LINQ. I asked my colleague what should I do, she said use Take(n). It’s true, Take(n) returns you n number of item from an IEnumerable. For example, you can do something like

var result = (from i in itemList
where SomeCheck(i)
orderby i.Rank
select i).Take(10);

However in my case, it is a bit more complicated then that. This method SomeCheck() maintains a state. Doing the above would call this method based on the count of itemList.  What I really want is I want to stop calling this SomeCheck method once I get 10 items.

Take() != Top()

Now then you think about now Top works in SQL Server, I would have guess that a query such as “ Select Top 10 * From [someTable] Where [SomeSlowEvaulation] ” would only evaluates “SomeSlowEvaulation” up to the point we get all 10 items. This cannot be achieve by using Take(n).

My Solution to My Own Problem

To achieve what I want, I ended with something like this:

var count = 0;
var result = from i in itemList
where count <= 10
where SomeCheck(i)
where count++ > -1
select i;

You may wonder what the heck am I doing. What the above code does is that it stop evaluate SomeCheck(i) once I get 10 or less items in the result. The code is very straight forward. Once count <= 10 failed, LINQ stop evaluating the rest of the where statement. If SomeCheck(i) passes, then it has to run count ++ > –1.  In the above case, you can join all these statements together, but it is easier to read the code in this format.

So, why would I want to do that. Of course there is a business logic why I can’t call SomeCheck(i). This method does two things, it checks if I want the result, and it caches the data at the same time. You can see that the flaw in this method is that it does two things at the same times. However, back then it appears to be a good idea.

The only one thing that the above solution can’t minic the top is that it doesn’t stop evaluating until it finished the list. In SQL, if you select top 10 out of 10 million records, you would expect SQL Server does not loop though the entire list.  In my case unfortunately it still does.

Conclusion

If you Google Top for Linq, you will find that most of the answer are using Take(). In most cases it Take() will work. However, just like anything, just because there is an answer to a problem you search, it doesn’t it is the right solution for you. You should always dig a bit deeper to the problem to ensure the answer applies to your problem.

.NET Remoting Causes Security Exception After Upgrade to .NET 4.0

This may well be my last blog post on .NET Remoting. I am not sure how many applications out there still using .NET Remoting, but the current trend is to migrate from .NET Remoting to WCF. Eventually, we will, but in the meantime, the product I am working on will still relay on .NET Remoting.

Yesterday, I updated all the VS project target framework from .NET 3.5 to .NET 4.0.  Usually migrating from one version of .NET framework to a newer one should be fairly simple.  There are some difference between .NET 3.5 to .NET 4.0, for example MS removed the CAS and promote the use of Security Transparent Code. It also introduces new classes like Tuple, which is useful for represent data from a data row.

After changing all the target frameworks and rebuild the solution, everything should works just like before. Started my server and client app, boom…. System.TypeInitializationException!!!

Exception

Exception: "The type initializer for 'ME.APTS.ServiceInterface.DataAccess.DataAccessBase`1' threw an exception."

Inner Exception: System.Security.SecurityException {"Request failed."}

Stack Trace:

Server stack trace: 

at System.Runtime.Serialization.FormatterServices.nativeGetSafeUninitializedObject(RuntimeType type)

at System.Runtime.Serialization.FormatterServices.GetSafeUninitializedObject(Type type)

at System.Runtime.Serialization.Formatters.Binary.ObjectReader.ParseObject(ParseRecord pr)

at System.Runtime.Serialization.Formatters.Binary.ObjectReader.Parse(ParseRecord pr)

at System.Runtime.Serialization.Formatters.Binary.__BinaryParser.ReadObjectWithMapTyped(BinaryObjectWithMapTyped record)

at System.Runtime.Serialization.Formatters.Binary.__BinaryParser.ReadObjectWithMapTyped(BinaryHeaderEnum binaryHeaderEnum)

at System.Runtime.Serialization.Formatters.Binary.__BinaryParser.Run()

at System.Runtime.Serialization.Formatters.Binary.ObjectReader.Deserialize(HeaderHandler handler, __BinaryParser serParser, Boolean fCheck, Boolean isCrossAppDomain, IMethodCallMessage methodCallMessage)

at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Deserialize(Stream serializationStream, HeaderHandler handler, Boolean fCheck, Boolean isCrossAppDomain, IMethodCallMessage methodCallMessage)

at System.Runtime.Remoting.Channels.CoreChannel.DeserializeBinaryRequestMessage(String objectUri, Stream inputStream, Boolean bStrictBinding, TypeFilterLevel securityLevel)

at System.Runtime.Remoting.Channels.BinaryServerFormatterSink.ProcessMessage(IServerChannelSinkStack sinkStack, IMessage requestMsg, ITransportHeaders requestHeaders, Stream requestStream, IMessage& responseMsg, ITransportHeaders& responseHeaders, Stream& responseStream)

Exception rethrown at [0]: 

at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)

at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)

at ME.APTS.ServiceInterface.IBroadcastListener.Process(ArrayList broadcastQueueArrayList)

at ME.APTS.Service.Provider.Remoting.BroadcastQueueRemoting.Process(ArrayList listOfObjectsToProcess) in C:\dev\Streets\Main\APTS\ME.APTS.Service.Provider\Remoting\BroadcastQueueRemoting.cs:line 119

This exception is being throw by my client app, when my server app tries to do a remoting call on the client application. The server when to send a piece of data to the client, and it needs to create an object, but apparently the server has no permission (that’s what I assumed) to create object on the client side.

The Confusion

After spending hours debugging the problem, it became even more confusing when some client application works and some aren’t. These client applications run the same piece of code on the same machine, and yet only some of them work.

Trying to confuse myself more, I tried to look for security attributes setting that we may have used. Nothing turns up. Since we didn’t change anything in the code and not even the application configuration files, it must be changes in the .NET framework.

Debugging

After hours of frustration, my brain was too tired to think. I asked my colleague to look into this issue together. After a couple of testing,  we realized we forget to check the client configuration file. We don’t use App.config, but our setting file is in a very similar format. After doing some configuration files comparison, BINGO!

Solution

It all comes down to a missing .NET Remoting setting, which is highlighted below.

<configuration>
  <system.runtime.remoting>
    <customErrors mode="off" />
    <debug loadTypes="true" />
    <application >
      <service>
        <wellknown mode="SingleCall" objectUri="BroadcastListener.rem" type="ME.APTS.Client.Model.BroadcastListener, ME.APTS.Client.Model" />
      </service>
      <channels>
        <channel ref="tcp" port="9999">
          <serverProviders>
            <formatter ref="binary" typeFilterLevel="Full" />
            <provider type="MyCompany.SecurityServerChannelSinkProvider, MyCompany.ServiceInterface" securityPackage="NTLM" impersonationLevel="impersonate" authenticationLevel="call" logging="true"/>
          </serverProviders>
        </channel>
      </channels>
    </application>
  </system.runtime.remoting>
  <appSettings>

After specifying the typeFilterLevel attribute to Full, the app works again. Here is the strange part, why all of the other working application has a specified typeFilterLevel, and not the one that I was debugging. I checked the existing release build, which runs on .NET 3.5, and it doesn’t have this setting either.

Conclusion

The only conclusion I can draw from this is that a change in .NET 4.0 causing this issue. By default, the typeFilterLevel is set to low. The application that I was testing must have been using the low setting. From the typeFilterLevel documentation, it isn’t clear when you should use typeFilterLevel.Low or typeFilterLevel.Full.

Low: The low deserialization level for .NET Framework remoting. It supports types associated with basic remoting functionality.

Full: The full deserialization level for .NET Framework remoting. It supports all types that remoting supports in all situations.

The changes in .NET Remoting security cause my application to use typeFilterLevel to full. It maybe related to the changes of the CAS. It would be interesting to see what exactly has changed, but there is no time for it. Soon our application will migrate to WCF and .NET Remoting will become the past.