Sunday, January 30, 2011

Infrastructure Knowledge for Developers

As technology is constantly evolving and with the recent market progress in cloud computing the line between hardware are software is getting more obscure every year.

This changes some basic questions. Are you a developer? Are you in Infrastructure? Today, I believe it is unreasonable to expect to be just in one or the other.

Take Microsoft's Azure platform.  Is it hardware?  Is it software?  Well, sort of a little of both.  

If you only know .NET development for instance, how would you accomplish Federated Authentication or even something like OpenID without at least an understanding of Internet Based Authentication?

Authentication has typically been the role of Infrastructure.  Here lies the rub.  A developer is not accustomed to having to decide, "Will I setup to be a "cross country" domain, "worldwide" domain.  These are typically infrastructure questions but need to be answered immediately upon creation of these types of services. 


An example might be a content a Content Delivery Network (CDN).  A CDN can have some fairly complex business rules. Use of a CDN affects marketing, management, system reliability, performance and customer satisfaction. A CDN can also greatly cut down the number of hits to your web site.  There can also be significant costs.

Here is where it gets complicated.  When a development team writes an application and is asked to start moving files to the "Cloud", there is no reason for them to ask about the benefits (or financial implications) of a CDN.  Why would they?  This is historically someone else's consideration.

If a team does all the work without knowing that some day you will be delivering the data over the CDN portion of the storage provider, you may find it almost impossible to do.

The best choice would have been to have development in on the "Infrastructure" part of the discussion.

You may find your team unable to re-write the code to handle the new details in the application.  For instance, some files may be delivered through the CDN and some not.

Most developers are comfortable with the ideas of private and public files, but many will not be familiar with the idea of some public files going through CDN and some not.

Do yourself a favor.  Teach them now about the technology.  Just knowing a tiny bit about it could save you huge expense in the future.

I can think of many such instances where cross-training should be considered and as far as I'm concerned absolutely necessary for the future growth of an internet based company.

The hardware folks need to start sitting in on some developer meetings and developers need to learn about some infrastructure.  This will ultimately help everyone work together and definitely help the teams understand each other's needs.

Sound crazy to you?

Consider a few examples from past experience.

 - .NET developers who are tasked with creating a system of web sites that can intercommunicate, but none of the developers understanding the mechanisms of Public Key Infrastructure, Certificate Authorities, OpenID or any of these authentication technologies typically reserved for Infrastructure People to deal with.  

Clearly this is a large development job and the hardware infrastructure has to support what the development team can do and there is no point of the development team for instance making something requiring VPN's where there is no actual hardware in place (just cloud based servers).

- An SQL server DBA trying to figure out a way to fix a server with so many connections open that it can't even function.  The solution is in a Web Server that is not configured properly in the IIS Connection Pooling settings. 

The SQL DBA doesn't know about the IIS connection pooling because developers handle that and developers don't know about it either because "the hardware people" handle the web servers.

- Developers using the Cache control of a Web Server with no expiry of Cached objects and basically forcing the server to run itself so low on memory that it starts purging memory out of desperation to stay running.

This is clearly a situation where a developer who sat in a Cache discussion about how .NET (in this example) manages the Cache and the fact that a Windows Server does not have unlimited memory resources like a mainframe (just kidding for you mainframe folks out there...I know from past experience that your problems can be worse).

- Consider developers who do not truly understand the impact of making DNS queries and their impact on IP Infrastructure when they make queries to 10,000 servers for DNS records from a large volume email application without any consideration for Caching of DNS requests. 

Any developer who actually saw what happened at the CISCO switches and the upstream DNS servers would cringe at the traffic they created.  Of course, if they only knew.

- The same argument works the other way around.  I have seen a DBA change the password on a SQL server automatically on a rotating schedule without realizing the impact to a Windows Web Service sitting in another building.

It seemed that every week the off-site Web Service crashed and no one could figure out why. It took weeks to determine the cause. 

I have discovered that having developers sit in with the Infrastructure folks occasionally and vice-versa provides tremendous benefits in co-operation between the IT staff.

It's amazing to see a development team wonder if there is a hardware solution to their problem before committing to a month of work.  It's just as amazing to see a hardware person ask if they could just get a developer to make a small tweak to fix a problem and get a positive response just because of a bit of cross-training.

If you're trying to improve productivity, allow the different teams to sit in with each other once in a while.

I have started to bring our developers up to speed with some training on  Caching, PKI (Public Key Infrastructure), Certificate Authorities, CDN's (Content Delivery Networks), different types of Source Code Repositories and basically anything they are interested in reviewing.


Anything the team can learn about together can only benefit everyone.

A Source Code Repository discussion can be equally beneficial for developers (who have to use them) as well as Infrastructure folks who have to plan to make sure they work. Why not let the teams learn this type of technology at the same time in the same location?

Code Repositories are typically a developer item but your Infrastructure folks will greatly appreciate and even help the development team out if they really know why it's so important to development, and maybe even find better ways to  store the data for the team. 

After all, even the best written software still needs proper infrastructure to run it.

I like to schedule the first 1/2 hour of our regular development meeting for members from other teams to join in.  They can see what the team does and how things work.

They are more than welcome to sit in the technical stuff, including table interactions, UI issues, etc. later on. It gives other departments an appreciation for how complicated of a job it can be and definitely helps boost morale.

The point here: Give it a try.  Do a bit of cross-training during your Developer and / or Infrastructure Meetings.  You will be amazed with the results!


Mike Caspar

Friday, January 14, 2011

Team Work Common Sense

Let’s talk about working in Teams.  There are many options for how to work in teams and a variety of implementations.  However, let’s just talk about some business reasons for doing things in teams and why they are an essential part of software development.

Actually, I can think of a variety of other parts of business that could benefit from this approach, but let’s just stick with software development for now.

When working in teams, I try to spread the work amongst the developers.  This shares system knowledge and experience.  As each week passes, each team member becomes knowledgeable about a different part of the system.

At first this approach to the outside observer seems slow and cumbersome. As time goes by, and the team gets used to each other and the application, development gets faster as any team member can effectively work on any part of the system. 

This is the tricky part. An executive who has not seen Agile work before will think that things are slowing down. Yikes.  Of course this is not the case.

Try to focus on the difference in the bug rate, increased value to the customer and to the company with things such as lower churn (loss of customers) rates, a regular and predictable release cycle, customer satisfaction, and the happiness of the support desk staff as the number of support calls goes down. Consider also, what marketing can do with the information about the upcoming release, or even the excitement of the customers experiencing a new, zero bug release every iteration.

Granted, these items seem less tangible to a financial person, but the turnover of unhappy support employees, unhappy developers and churn of unhappy customers can easily be quantified, to name just a few.  In your environment, there are likely many more.

Depending on the size of your application and team, you will soon be confident that any team member can work on any part of the system and make the development team extremely productive and respected.  As a minimum, there will be product knowledge by all developers.  This is key when you are planning on adding new functionality.  It may surprise you what a new novice developer has just learned in a course that you or your team members may not know about!

Don’t get me wrong, each ‘mini project’ or feature in the program still needs a developer who really knows that piece.  I can’t really see getting around this without having a very shallow application. 

I’ll talk about the whole developer managing a mini project thing and its’ advantages another time. For now, though, I’ll just stay on topic.

Once Agile is in full swing (no matter what your discipline or disciplines you use), you will find that you will soon be in a position to be wondering what work you can do next.  This is where Agile truly shines.  The work you are doing has a definable schedule and because of the regular estimates and re-estimates you have a realistic idea of how you are doing.  After a few iterations you will even have an idea of how much productive work gets done by the team in a week, and you know how much work you can already schedule ahead for.

Others will not see it coming and will likely be surprised you are asking.

Don’t be shy! This is the time to let others know how well Agile is working for your company.

Each time a small feature or part of the system is being worked on, I purposely give the piece of code to the same ‘base” developer to do but assign a different “second” developer to work on that piece.  This forces information exchange and empowers the team to know all parts of the system.  It is also a great way to pass on development experience to new developers and sometimes from new developers to existing ones.  The exchange of information is a dramatic thing to witness once a team is used to it.

There is nothing more enjoyable than to publish a new release and to have nothing dramatic happen from the development perspective. 

Where I currently am, I remember after about the 4th iteration (we are on weekly cycles), having a developer meeting 2 days after and the whole team joking about the fact that not only did the support department receive no extra calls, but the call volume actually went down because of some of the previous defects we fixed.

What a feeling of accomplishment and teamwork!

As well as the obvious benefits of removing risk from the company, it provides the added benefits of true teamwork spirit, mutual respect and fellowship.  The developers soon know each others’ strengths and weaknesses and will automatically help each other out.

Techniques are passed on and knowledge that otherwise would have stayed only with one developer is shared.  The transformation from individuals to a team is remarkable and very enjoyable to see and for me at least, the true inspiration for managing as part of an Agile Team.

Agile helps the developers to be happy and fosters an environment where everybody feels like part of the finished product. 

For those of you out there that are regularly trying to figure out how to keep developers motivated and feeling like part of the company, consider the positive effects of true teamwork and each person feeling like they are part of the finished product (which of course they easily could be).