IT's Dilemma

This morning, I received an invitation to Microsoft’s Interact 2008 conference that is taking place from April 8-10 in San Diego, California.  Those of you who have been monitoring this blog for the past month know that I’ve been on the very-fast track to learning about Microsoft Unified Communications.  Although we primarily are developing against Speech Server and Office Communications Server at this point, we envision writing some code against Exchange Web Services to bring e-mails into our CRM also, truly unifying communications.  What gets me fired up is the fact that we truly have the ability to do whatever it is that we want to do.  I’ve said it before and I’ll say it again: what I like most about Microsoft is that they give you a solid foundation, a blank slate, and then they step back and  get out of the way.  Our biggest problem is convincing the business people in our company that even though the sky is the limit, we need to contain things at a more manageable level.

If you’re headed to Interact 2008 and you work in the contact center space or are interested in call distributors, I’d love to take some time to meet up with you and chat one-on-one.  If there is enough interest, we may even be able to form a small community of our own to address contact center concerns and OCS.  It seems that we could all benefit by coming together and building a solid community.  I will continue to blog about our experiences in developing a call distributor, blended predictive dialer, and other software, but we need to get more voices out there.  Feel free to ping me on my blog through comments or by e-mailing me at definitejunkmail (yes, definitejunkmail) on GMail.  I’ll reply back with my work e-mail so that we can stay in touch.

Looking forward to meeting some of you!

One of the first things I really struggled with when ramping up on Microsoft Office Communications Server 2007 was understanding roles, their responsibilities, and how roles overlap.  What roles do I need?  How many of them do I need?  Can an edge role be installed on a mediation server?  What’s more, Microsoft tosses out diagrams like the following with very little explanation:

The information that is available out there about OCS server roles is scattered across many different documents, Web sites, and blog posts.  Hopefully, this post will distill information on the roles available in OCS, their coexistence constraints and hardware/software requirements.  That said, let’s get a little bit of vocabulary out of the way first.

• Office Communications Server: A Microsoft product designed to facilitate communications both inside and outside the office.
• Presence: A metric that takes into account both your availability (available, idle, away) and your willingness (available, busy, on a call) to communicate.
• Endpoint: Any device (SIP phone) or software package that registers itself with Office Communications Server as belonging to a user, meaning that the user can be contacted through the device or software package.
• Enterprise Voice: Probably the most noteworthy addition to the product since 2005; allows calls to enter and exit Office Communications Server.  This means that from any endpoint, users can make or receive calls to traditional phone numbers.
• Public Switched Telephone Network (PSTN): The traditional telephone network that delivers telephone service over dedicated copper cables.

Pool Server Roles

In enterprise-class deployments, multiple servers are typically assigned the traditional front-end server roles.  Although this is sometimes referred to as a cluster, the term that has been assigned in this case is “pool”.  As an aside, I would recommend that any business who is seriously considering OCS for Enterprise Voice use an enterprise-class deployment even if scalability is not a concern.  Resiliency is also something you gain when you deploy multiple servers playing the same role.

Front-End Servers

The front-end server handles a lot of the plumbing for OCS.  It authenticates users against Active Directory, routes invitations to appropriate endpoints, and generally coordinates all other features of OCS.  It is important to note that once a front-end server has assisted in routing the invitation to appropriate endpoints, it gracefully steps back and allows the endpoints to communicate directly.  This means that a front-end server will not be overwhelmed by handling thousands of simultaneous video chats.  It merely directs the invitations and coordinates the set-up of communications, then allows the endpoints to communicate however they may choose.  (The exception to this rule is instant messaging sessions, which always travel through the front-end server for archiving purposes.)  It is also important to note that a front-end server may handle merely these responsibilities or more if the deployment team decides to consolidate multiple roles onto the front-end server.

Specifically, the front-end server:

• Authenticates users against Active Directory
• Aggregates and disseminates presence, contact, and other similar information
• Routes invitations to appropriate endpoints (or gateways) and cancels other invitations once an endpoint has accepted
• Tracks status of sessions (even though the session communicates from endpoint to endpoint) and delivers status update messages like typing, ringing, accepted, etc.

Requirements:

• Dual processor, dual core 2.6GHz+ processor
• 2 x 18GB HDD
• 2GB RAM
• Gigabit NIC
• Windows Server 2003 SP1+*

Directors

Directors have two primary responsibilities: user authentication and direction.  User authentication comes in especially handy when dealing with remote users.  Because remote users cannot be redirected to their home server/pool, the director role proxies their connection through to the correct server/pool.  However, user authentication by a director is also important when an enterprise hosts multiple standard servers or enterprise pools.  Users are “homed” to a server/pool, meaning that that server/pool is the one that stores the users account information.  A director will get the user to the right pool on the first try rather than repeatedly polling every server/pool to see if the user’s account is stored there.  In summary, directors are always recommended when you have remote users and/or multiple standard servers or enterprise pools.

Conferencing Servers

Conferencing servers, in contrast to front-end servers, are designed to maintain a central hub for communications when more than two parties are involved.  This minimizes bandwidth requirements for the conference by maintaining one open channel per endpoint.  Conferencing servers also register and update “focus” on front-end servers.  “Focus” means security and state management for conferences.  Conferencing servers come in two major flavors: A/V conferencing servers, which facilitate audio and/or video conferences, and Web conferencing servers, which facilitate Live Meeting 2007 conferences.

The IM conferencing server:

• Controls focus for three or more IM participants
• Focus includes:
  • Participant list tracking
  • Leader determination
  • Security and management controls
• Is installed by default on all front-end servers and cannot be installed separately

The A/V conferencing server:

• Enables three or more parties to have audio and/or video chats (two parties use a point-to-point connection)
• Reduces bandwidth requirements by mixing audio from other participants into one channel instead of n channels

The Web conferencing server:

• Allows application and file sharing
• May leverage an A/V conferencing server to distribute audio/video in a Live Meeting

The Telephony conferencing server:

• Provides functionality for exposing an audio conference to the PSTN
• Allows the organizer to
  • Mute everyone except the presenter
  • Mute themselves
  • Remove parties
• Is installed by default on all front-end servers and cannot be installed separately

Requirements**:

• Dual processor, dual core 3.0GHz+ processor
• 2 x 18GB HDD
• 4GB+ RAM
• Gigabit NIC
• Windows Server 2003 SP1+*

Archiving and CDR Server

The archiving and CDR server is designed to provide a first-shot at archiving instant messaging sessions and call detail records for compliance purposes.  Although this is Microsoft’s stated intent for the role, it’s hard for a layman to imagine how an out-of-the-box solution could really meet the compliance needs of any reasonable organization.  I have not been able to locate any customizable fields, which means that we have a great track record of who called whom at what time, but that’s pretty much it.

Requirements:

• Dual processor, dual core 2.6GHz+ processor
• 2 x 18GB HDD
• 4GB+ (16GB+ if archiving is enabled) RAM
• Gigabit NIC
• Windows Server 2003 SP1+*

* I was not able get the OCS primary installer to run successfully on Windows Server 2008 RTM.  It may be that the individual installers would run successfully, but I have not confirmed this.  The only role I have successfully installed on Windows Server 2008 is Speech Server 2007.

** The work of mixing audio channels is intense; A/V conferencing servers will benefit from more robust hardware.

My recent experience with discovering, evaluating, and evangelizing Office Communications Server at Extend Health has been great in many senses.  Many projects fail or are axed in their infancy stages.  Some projects probably deserve to be axed at that stage – one of the primary principles of brainstorming is that you entertain all ideas, even the bad ones.  But what about the projects that shouldn’t fail at that stage?  What about the project that are simply struggling to get off the ground and deserve a credible pilot?

I would contest that many projects fail in the idea/skunk works phase because of management failures.  I’m sure many managers of skunk works projects have the best intentions and are qualified managers, but managing a skunk works project is different than managing a sanctioned project.  I scoured the Web (after the project had moved into pilot phase) to see whether there were any documented best practices for managing this type of project.  I didn’t find any distilled, coherent best practices repositories, but I did find some good material that should be referenced here.

Martin Tate had the most to contribute as far as I’m concerned.  He posted a presentation called Implementing Best Practice – A Different Approach, Using ‘Skunkworks’.  In this presentation, he details what skunk works projects are and some of the classic examples (including 3M’s Post-it Notes, Audi’s Quattro, DuPont’s Kevlar, Ford’s Mustang, IBM’s first PC, and the original, Lockheed Martin’s SR-71).  [Note: there weren't any citations, so I'd encourage you to do your own research on those examples.]  He also included several bullet points that just happened to be true of the project I just completed, which was probably dumb luck on my part.  He says that skunk works projects should:

• Like the customer
• Start small and build up
• Maintain low visibility
• Find senior sponsor
• Manage the approvals process
• Create team from the right people

Judy Artunian also had some excellent input in her article for ComputerWorld entitled Skunk Works: The Sweet Smell of Success.  She adds these recommendations:

• Handful of employees with knack for taking a fresh look
• Keep the operation under wraps
• Demonstrate alignment with business objectives
• Keep tabs, but don’t impose a schedule on the project

Finally, Ken Smith blogged about skunk works projects and said:

• Don’t allow skunk works projects to delay normal projects
• Check with product & project/product management first
• Be ready for disappointment

I think the two things that really went right for this project were the fact that I had the CTO as my corporate sponsor, and the fact that I released information at a controlled pace.  I firmly believe that if I had disseminated the information as rapidly as I gathered it, there would have been an overwhelming and sometimes inaccurate flow of information out of the project.  Therefore, if I were to list my own bullet point best practices for skunk works project management (including rehashed versions of points above), this would be it in order of decreasing priority*:

1. Think like the customer (even if the customer is internal)
2. Demonstrate alignment with business objectives
3. Don’t volunteer to manage a project you don’t believe in
4. Find a corporate sponsor who believes in the project (or at least understands its importance and believes in you)
5. Define a series of informal milestones to serve as gating factors for going forward
6. Maintain low visibility
7. Disseminate information at a controlled pace
8. Keep in mind that only a very small percentage of skunk works projects succeed; balance cost-benefit ratio constantly

* If your project doesn’t take into account the first four objectives, don’t bother.  Scrap it and move onto something with more business value.

Overall Rating: 2/5
Response Time: 3.5/5
Overall Knowledge: 1.5/5
Personalities: 4.5/5
Levels to SME*: ?
Familiarity with “Greenfield“: No
Primary Contact: Erwin Gunnells

*Number of contacts (people) I talked to before reaching a subject matter expert.

Shortly before we ruled out Nortel as a potential partner for implementing and deploying Office Communications Server, we contacted Dell to talk about providing the same services.  Where our conversation with Nortel left me uncomfortable, the conversation with Dell ended within two days.  As with Nortel, the people that we spoke with were very pleasant, but I just didn’t feel that they were ready to execute an enterprise deployment of a complicated product.

I might have been able to predict the result of this conversation simply by looking at Dell’s UC site.  There really isn’t very much there at all.  They don’t talk about OCS (at least that I could find), their material seems to be marketing alone, and there aren’t any real case studies.  Aside from the fact that Dell uses OCS internally, there was nothing to make us turn our heads.

The one really positive experience we had with Dell was talking about software concerns.  Casey Spear and a number of Microsoft representatives (Janet Gresco, Steve Todd) have been fantastic in addressing our many questions and even suggesting appropriate alternatives that we didn’t know existed.  If you’re in the Pac/West region, I highly recommend that you contact Casey if you have any questions about Microsoft licensing.

Overall Rating: 3/5
Response Time: 2.5/5
Overall Knowledge: 3/5
Personalities: 4/5
Levels to SME*: 2
Familiarity with “Greenfield“: No
Primary Contact: Matthew Christopher (Western Region ICA Overlay Team)

*Number of contacts (people) I talked to before reaching a subject matter expert.

Nortel was the first partner I contacted, based upon their strategic alliance with Microsoft.  My initial account contact was handled by Efrem Anderson, who did a great job of finding someone for me to talk to.  I was pleased to find that Efrem at least knew the term Office Communications Server, but beyond that there didn’t appear to be a solid understanding of what I needed.  This is completely acceptable at the account manager level; knowing whom to talk to is sufficient.

Efrem put me in contact with Matthew Christopher, who truly seemed to be aware of a lot of the high-level aspects of OCS.  I don’t think I’d have felt comfortable passing all of my needs to him, but as a primary point of contact, Matthew was awesome.  I should also mention that Matthew was very receptive to my questions and seemed to have good knowledge of connectivity to the PSTN, which is one of the areas where I am weakest.  Matthew initially suggested that we pick up a small Nortel CS1000 PBX to help run the system, but when I pressed him for why we needed it and explained our situation, he agreed and even offered that we might want to connect directly to the PSTN through media gateways.  He never offered the word “Greenfield” as the name that typically references this type of deployment, so I got the feeling that he wasn’t completely familiar with how it worked.

In spite of that, if all Nortel employees had been as responsive and knowledgeable as Matthew, we might have selected them as a partner.  Unfortunately, the team Matthew was working with dropped the ball and failed to get me a scope of work.  That, in combination with the fact that I didn’t feel particularly comfortable with the other SMEs on the team, helped us to decide that Nortel wasn’t the partner for us to work with.

I should note that if your business needs a PBX (to hang non-OCS phones, faxes, etc off of), Nortel might make an excellent partner.  My primary problems with them were responsiveness and lack of understanding of “Greenfield” deployments.

In a previous post, I highlighted the discovery of Microsoft Office Communications Server 2007 as a unified communications solution.  I also speculated somewhat about the viability of the solution for a company whose core business revolves around enterprise voice.  Enterprise voice comes part and parcel with a series of challenges.  Routing inbound phone calls to appropriate personnel based on what the caller is seeking (skills-based routing), using presence indicators, and integration with CRM software are only a few of the problems contact centers face.

Typically, inbound call routing is handled by a software package known as an automatic call distributor (ACD).  ACDs can range in intelligence from the most basic level, where the ACD simply hands the call to the next available person, to advanced, where the ACD makes more involved decisions.  ACDs are critical to companies like Extend Health, Inc.  We have employees in different roles, licensure and HIPAA compliance are significant concerns, and we need to control costs across our various internal and outsourced employees.  I was more than a little concerned that we would have trouble achieving this level of functionality in OCS as there were no visible references to how to handle that.  (Not to mention, our Cisco guys are putting pressure on us by saying that it isn’t possible.)

This concern was almost completely resolved today when I stumbled across a blog post by a guy named Michael Dunn.  Michael worked for Magenic at one point in time, but has recently jumped ship to Microsoft.  Michael’s post (aptly called “Creating a WCF ACD (Automatic Call Distributor) Server Part I“) covers in great detail how to build an ACD, from scratch, using .NET 3.0’s Windows Communication Foundation.  After a frantic flurry of communications from me, Michael politely answered the burning question.  He told me that Speech Server 2007, which is part of OCS, ships with the ability to program against it using Windows Workflow Foundation.

Microsoft has essentially handed us a blank slate and told us to go at it.  It has been my experience in the past that this is where Microsoft shines the most: providing that foundation to build your own building on.  According to what I learned today, I should be able to write an ACD whose intelligence is perfectly matched to what we need.

I took the results from my conversation with Michael and went with it.  The results were nothing less than amazing.  In literally one hour, I went from having zero knowledge of programming Speech Server 2007 to having a fully developed application that examined the incoming number, routed it to a licensed agent (determined by the area code), and if the number had called previously, routed it to the same agent unless they were busy, in which case it selected a new agent that was licensed.  For sixty minutes worth of work and no experience, that’s substantial!

This changes my view of OCS from being a potential solution to being a probable solution.  The price delta mentioned earlier, in combination with the ability to program an ACD in an hour, means that if continue exploring OCS and can program our other features as easily as the ACD, we will likely select OCS as our contact center communications infrastructure.

The next step is to detail what we (Extend Health, Inc.)will be looking at to decide for certain whether OCS is an acceptable solution.  I’ll lay out some initial parameters here, so that readers know what we’re examining, but will cover each of these line items in much more detail as we get to it.

Automatic Call Distributor
The ACD I wrote in an hour is impressive, but we need a much more sophisticated ACD.  The ACD we will write will involve five levels of decision making:

1. What type of employee does the person need to speak to?  Enroller?  Agent?  CSR?  Narrow the list to people in that role.
2. Has the person spoken with someone in that role before?  If so, narrow our list to those employees the person has spoken with.
3. If licensure is an issue, narrow the list to licensed employees.
4. Narrow the list to available employees.
5. Narrow the list to the employees with the lowest variable cost.  If an agent that we’re paying per hour is available, lets route the call there before we route it to an agent that gets paid per call.

We don’t even know if we can write an ACD of that level of sophistication in our current Cisco solution.  If we can, we’d have to outsource it and it would likely take several weeks and a lot of money to complete.  If we want to tweak the ACD, we run into more time and money issues.  My guess is that we can write this ACD in a couple of hours internally and tweak in a matter of minutes.

Predictive Dialer
This is something we haven’t tried yet, but we need to be able to take a list of one thousand people we need more information from, assign a pool of ten agents, and keep those ten agents busy by having the system dial every time an agent completes a call.  The system should also screen answering machines since we need to talk to a person.  Speech Server 2007 is capable of making outbound calls as well as handling inbound calls, so this shouldn’t be an issue.

Offline Queuing
This is also something we haven’t tried yet.  There are some Web sites that allow you to click a button and have someone call you when they are available.  We want to combine this with the ability to give the option to those callers who will be on hold for more than five minutes to be called back.  In short, we want to develop a call queue that is agnostic about call directionality: it doesn’t care whether the call is inbound or outbound.

Voice Signature
In the insurance world, signatures are critical.  Voice signatures (like digital signatures) have recently been deemed legally binding.  Last year, we paid a lot of money and spent a lot of time having a voice signature process developed for our Cisco UCCX solution.  Unfortunately, it never worked.  We believe we can develop a solid voice signature solution for a fraction of the cost, in a fraction of the time.

In summary, we are looking to build out a lot of features that are specific to a contact center.  I’m excited about this and will expose as much information as my NDA allows me to on this blog, so stay tuned!

The company I work for, Extend Health, is currently experiencing the best type of problem: rapid growth.  As many are aware, unchecked or overly rapid growth can sound a death knell for companies.  One potential reason for this correlation is the speed at which decision makers must make decisions.  More decisions are part and parcel of a larger company, but the additional staff necessary to make additional decisions generally lags behind the decisions themselves.  This leaves additional decisions to be made to existing decision-makers, which ultimately leads to a shorter decision lifecycle.

This past year, our company grew 10x in year-over-year numbers.  We did more business in a few days at the start of Medicare season than we did in the entire year previous.  We hope to grow another 4x this year.  This type of growth is putting a significant strain on our existing system, a Cisco UCCX with a few integration pieces installed.  For the past month, we have been evaluating two alternative solution: a Cisco UCCE with more integration servers, and an Avaya solution.  Both solutions are flagship contact center solutions, but they carry a hefty price tag: a low estimate puts the solution around $2.5 million, with a final figure more likely to come in above $3 million.

Part of my job is to review solutions and report on their overall ability to integrate into our environment.  Part of my nature is to add commentary on how viable I feel the solution to be.  As I was researching the Cisco solution, I stumbled across a newer offering: Microsoft Office Communications Server (OCS) 2007.

Microsoft Office Communications Server 2007 was launched not six months ago on 16 October 2007.  The launch was traditional Microsoft strategy: conquest through partnership.  Microsoft’s current claim is that it isn’t there to replace the PBX – yet.  As such, it has strategically allied with a number of hardware vendors such as Nortel, Polycom, Ericsson, Mitel, Dialogic, Audiocodes, and more.  The full list of sanctioned hardware vendors can be found at http://www.microsoft.com/uc/partners_hardware.mspx.  In short, Microsoft partners with hardware vendors for one of two reasons:

1. Connections to the public switched telephone network (PSTN).  No one in their right mind would claim that Microsoft is a telephony expert, especially when it comes to the PSTN or PBX spaces.  Microsoft has aligned itself with hardware vendors that handle nailing down calls and releasing calls when finished.  Microsoft has also worked diligently to leverage the feature set on PBX systems to supplement a weak feature set of an initial offering.  While there are plans to implement features like music on hold, call park, etc, Microsoft simply integrates with existing PBX systems at this point to achieve features it lacks.
2. Handsets/headsets.  The telephony paradigm has not changed significantly in 130 years.  People have used hardware to hear, speak, and initiate/terminate calls.  While the paradigm is changing with the advent of softphones, some people prefer to have the comfort of a traditional telephone set.  Microsoft has partnered with a few vendors (Polycom, Nortel, Jabra) to deliver a telephone experience more in line with traditional telephony.  Mind you, these aren’t your average telephone sets: they have fingerprint recognition, colorful presence indicators, and more!

Microsoft’s forte is providing a solid, well-integrated foundational software layer.  Establishing partnerships for hardware is a must at this point in time.  With the partnerships, Microsoft is able to offer OCS as a truly profound communications platform.

What Microsoft already had a strong background in was collaboration, e-mail and instant messaging, presence, and pervasive integration.  Microsoft Word, for instance, has tight connections with Sharepoint, which can be consumed as an RSS feed in Outlook.  Microsoft, with the delivery of OCS, integrates all the same familiar desktop software with the OCS system, essentially exposing presence and the ability for instant communication from Word, Outlook, or Sharepoint.  Exchange gets the ability to store voicemail and archived IM sessions, meaning you have your entire communications history in one place.  You can initiate a call to the author of a Word document, from the document.  You can start a video chat with someone in a discussion thread on Sharepoint.  The depth and thoughtfulness of the integration alone are sufficient to make a solutions-oriented person take pause.

What Microsoft didn’t have was enterprise voice.  In this context, enterprise voice means that we have lots of inbound and outbound calls.  There was voice chat and even video chat in earlier versions of Microsoft products, but those sessions were limited to internal conversations.  OCS leverages the hardware partnerships mentioned above to add the ability to deal with inbound and outbound calls as well.

I’m not certain that OCS is viable for us.  Our business is critically dependant on enterprise voice: we expect to hit a peak of 10,000 calls in an hour this fall.  However, I have taken significant note of where OCS is at and will continue to investigate the solution as rapidly as possible.  One thing is certain: with an estimated implementation-complete price tag of $500,000 for a jaw-dropping solution that we can build on top of, we’re taking this very seriously.

Game theory is a branch of applied mathematics employed in making high-level strategic decisions.  There are a number of well-known applications of game theory, one of which is the zero-sum game.  A zero-sum game is one in which the sum of all participants gains and losses is zero: that is, each participants gains or losses are exactly offset by other participants’ gains or losses.  I have had the privilege of working for a number of noteworthy companies, each of which had their own unique perspective on IT-business alignment.  It’s unfortunate that many of these companies treated alignment as if it were a zero-sum game.  At the other end of the spectrum are the “synergy” companies.  These positive thinkers portray the IT-business relationship as centered in a radiant beam of beautiful energy shining down directly from heaven.  Doves flutter, angels sing, and IT and business just work harmoniously.

The truth is that aligning business and IT is a lot of work.  Coming back to game theory, there is a better paradigm for this relationship: the prisoner’s dilemma.  The prisoner’s dilemma is classically stated as follows: Two suspects, A and B, are arrested by the police. The police have insufficient evidence for a conviction, and, having separated both prisoners, visit each of them to offer the same deal: if one testifies for the prosecution against the other and the other remains silent, the betrayer goes free and the silent accomplice receives the full 10-year sentence. If both remain silent, both prisoners are sentenced to only six months in jail for a minor charge. If each betrays the other, each receives a five-year sentence. Each prisoner must make the choice of whether to betray the other or to remain silent. However, neither prisoner knows for sure what choice the other prisoner will make. So this dilemma poses the question: How should the prisoners act?

Prisoner A realizes that the best case scenario is for them to both stay silent and spend only six months in prison.  The thought comes to him, however, that prisoner B may betray him.  If the B betrays him and he stays silent, he’ll spend 10 years in jail.  If he betrays prisoner B, he’ll either go free or only spend five years in jail.  This results in a Nash equilibrium, which is a fancy way of saying that more often than not, both prisoners will betray the other.

It frequently seems as if business and IT find themselves in the same scenario.  As an IT guy, I have harbored more than my share of grudges at what sales people make (both promises and salary).  The thought surfaces, “They wouldn’t have anything to sell if it wasn’t for me!”  I’m sure the equivalent thought surfaces on the business side.  But despite the analogy to the prisoner’s dilemma, there is a fundamental difference.  There can be communication between business and IT.  Although it is arguably as difficult as facilitating communication between prisoners in different cells, communication between business and IT can be facilitated and can create a synergistic relationship.

The intent of this blog is to chronicle one IT guy’s attempt to understand and communicate across the divide to the business side.  As such, there should be very few technical posts here that lack a statement of a business case.  And maybe, just maybe, someone reading this blog might someday hear angels singing.