We’d like to invite you to participate in the process of designing an indicator to measure the effect that a training program for small business entrepreneurs has on their sales.
ATTENTION: don’t let the math scare you out of it.
In the last 10 years, ZIGLA has designed more than 30 Monitoring, Evaluation & Learning Systems (MELS). If we have learned something, it is that indicators are the cornerstones of any MELS.
In spite of that, we’ve also learned that most organizations underestimate –or even are unaware of— the importance of going through a comprehensive process to design program indicators. In most cases, we find an Indicator Matrix or a Logical Framework, and a list of indicators that in the best scenario includes only imprecise definitions, which are never enough to fully understand how they will be effectively calculated, their time reference, or the precise thing that they are measuring.
What are the consequences of this? Leaving an indicator open to a variety of interpretations leads to having measurement instruments which do not correspond to its original measurement objectives.
But enough talking. This is no bluffing! Let’s look at a concrete example of the kind of situations we’ve encountered while reviewing or designing MELS.
The scene: imagine a program that is looking to promote small businesses (mostly informal) in a Latin American country. The curriculum consists of intensive training in weekly sessions, alternating between a total of 20 lectures and 20 individual workshops throughout 10 months.
There’s little doubt that one indicator that the organization should use to monitor, evaluate and be accountable for the program should be the evolution of the sales or earnings of the businesses it’s supporting. For the sake of simplicity, let’s focus on sales instead of earnings, particularly on how they evolve.
The program coordinator (PC) meets with the Monitoring & Evaluation Specialist (MES). Topic: indicator design, session 1 of 4.
PC: That’s it! I have it! the indicator I need is “percent variation in sales of the businesses participating in the program.”
MES: Not so fast. It’s not as simple as it might seem. Remember a variation always implies a time reference in which something is changing.
PC thinks to herself “Oh wow, she can use big words to state the obvious…”
PC: You are right, the indicator should be “percent variation in sales of the businesses participating in the program, between the baseline and endline”. Now let’s move on to the next indicator.
MES: Hold on! What are the specific stages of the program in which your baseline and endline are actually taking place?
PC: It’s pretty obvious don’t you think? The endline will be right before the program is ending, and the baseline right before the program starts, using the application form.
MES: Mmm… maybe. But I still have a question. Those sales… are they weekly, monthly or annual sales?
PC: You really want me to give you all the detail huh? Fine, they are annual sales. So, the indicator should be “percent variation in annual sales for the businesses participating in the program, between the baseline (application process) and the endline (last class of the program)”.
MES: Don’t the majority of the participants own small and informal businesses?
PC: Yes… so, what?
MES: It makes me doubt that they will even have records of their sales.
PC: That’s true. In fact, the first sessions of the program are meant to promote book keeping so the microentrepreneurs can eventually record their income and expenses and are able to make a cost analysis. I have an idea, we could take monthly sales, then compare the sales of month 9 (second to last month in the program) with the sales of month 2 (once the microentrepreneurs have received the classes on managing a cashflow). I’m sure we would have information for most participants that way.
MES: Mmm… but isn’t comparing sales in month 9 (September) with sales in month 2 (February) a little bit like comparing apples and oranges?
PC: You are right… In fact, February is not a very active month because in the cities where we are most people are on vacation by that time.
MES: Besides that, wouldn’t you say that the effect the program has on the microentrepreneurs takes a little time to influence economic performance?
PC: I’d say it does take time… so, what do we do? We could gather information in month 2 of the first year, and then month 2 of the second year. That way we would compare sales of the same period, backed by the entrepreneurs’ records.
MES: Yes, that sounds like a good direction. Still, I think it would be better to use the endline in month 9, one year after the program. By that time, most participants will be keeping records which they could provide.
PC: Good idea. In that case, we could also evaluate the survival rate of the businesses one year after the program and have a better idea of its impact… but wait!
PC: We would be comparing the sales from month 2 to 9 one year after the program, with the sales of month 2 to 9 of the first year. But the businesses start showing effects in their sales already in months 3 or 4 during the first year. If we do that we could be underestimating the change in sales!
MES: That’s right, and honestly, I can’t think of an ultimate solution. We could calculate two indicators: One taking sales from month 2 to 9 for both years, and another indicator taking sales from the first few months only, months 2 to 4 for example.
PC: Then, even if it’s not perfect, we have our indicator: “Percent variation in sales of businesses participating in the program, between the baseline and endline”, considering two different time periods for the baseline and endline:
Months 2-4 of the first year (t=0) and of the following year (t=1)
Months 2-9 of the first year (t=0) and of the following year (t=1)
PC: And that’s it! We have it!
MES: Actually, we are still missing a very important piece.
MES: We need to write down the formulas for these indicators. Every time you do this, new questions might come up. You will see. Besides, sooner or later someone is going to have to do it, either when preparing a report or when setting up the software platform for the M&E System. It’s better if it’s us rather than them.
PC: You are right. Could it be something like this?
PC: What do you think?
MES: Almost perfect. What would happen with businesses that take a few months to start keeping records of their sales? According to that formula, they are either not considered (because they have at least one month with no data), or it’s comparing an endline and a baseline which do not necessarily have the same number of months of data (which overestimates the change in sales). I think we can improve the indicator like this: “Percent annual variation in average monthly sales of businesses participating in the program, between the baseline and the endline”.
PC: OK now tell me this is done!
MES: Almost. The formula we just sketched refers to each individual business, now we just need to add up all those changes in sales and divide them by the number of businesses that we have, that way we get the final indicator. And now we are finally done with this one! There are only 31 other indicators we need to take care of before we finish the process.
PC: And I thought you were just bluffing when you said designing indicators took a long time…