Evaluating the NBO model performance

The marketing director said, they measured the results by doing the same mar-keting action at the same time for different target groups, the NBO based target group and some other target group, and then compared the results.

“We test and then we look at the results. We check what is the end result with the chosen metrics, such as conversions, and whether the NBO model works better than other tar-geting methods. If we see that it works better, we extend the use and start using it as a default targeting method. Then we don’t test it anymore.” Marketing director

He gave an example of how they had tested the NBO model in display advertis-ing.

“We see it clearly in display advertising, when have taken NBO model to the testing palette. We have had 3, or 5 different targeting methods. NBO, some other older target group based on the company’s own data, some paid external target group, such as Google’s target audience, and general non-targeted advertisement. Then we have consist-ently compared them and noticed, that NBO performs better than the others.” Market-ing director

The marketing director explained, that they mostly used conversions, impres-sions, sales, clicks and traffic as metrics when measuring marketing performance.

“If we could reliably measure the impressions of ads and then the realized sales from those who have seen the ad, it would of course be great. At least we can measure the click and the sales through the clicks. In direct mails measuring the results is a bit easier, because we are able to isolate the effect better in there.” Marketing director

However, the marketing director stated, that the attribution was a persistent problem for them when measuring the performance of marketing activities. The challenge was to decide what weight should be given to each channel and adver-tisement when analyzing the realized sales – should it be from the last click and how much weight should be given to all impressions and direct advertising be-fore the sales.

“That must be the biggest challenge, not only in NBO but measuring marketing perfor-mance in general. If I wanted to understand what has been NBO’s actual effectiveness for example for ROMI, I couldn’t calculate that. But what I can calculate is, that how much it improves the conversions in certain actions.” Marketing director

The marketing director pointed out, that they were not able to measure every-thing also partially because of their capabilities and partially because of the in-dustry-specific regulations. However, he thought that click-based measurement and conversions were good enough metrics to approximate the performance of the NBO model. He emphasized, that it is important that there is enough data points and volume in advertising to get reliable results and to be able to compare the effectiveness of different ads and target groups.

When the different target groups were compared in advertising and the results were analyzed, they continued doing the actions and targeting methods that generated the best results based on the chosen metrics.

“We stop the worst performing and continue and scale bigger the best performing ones.

For example, all of our direct mail is now NBO-based in two product groups. Then we have expanded the use to other channels, such as search engine advertising.” Marketing director

The marketing director said, that an important learning for him had been the sig-nificance of testing and the importance of planning. It is important to carefully plan the pilots and make sure there is always an adequate control group for the NBO target group to fully understand the impact of the recommendation model.

“The target groups have to be originally built so that afterwards it can be reliably ana-lyzed, that the same product was advertised in same circumstances for same kind of au-dience, and with NBO targeting it performed this well with this metric, and for the con-trol group this well with the same metric. The conclusion should be clear.” Marketing director

The digital sales manager said, that in digital marketing the overlap effect brought additional challenges in measuring the NBO performance.

“In digital channels the NBO messages are not the only ones that are shown, there are many other targeting methods and channels, too. It is difficult to measure whether the conversion has happened as a result of one targeting method or as a combination of all of those that are running. Of course, we can evaluate, that in NBO targeting the conversion

is this good, and in the other targeting method this good, but it doesn’t tell how much there is overlap.” Digital sales manager

The CRM director told, that one important metric they also followed was the ac-ceptance rate of loan and credit applications.

“It is not nice for the customer if we first advertise loan for him and then he won’t get the loan. It is very important, that we maintain the level that as many people get the loan, if we advertise it. So, we use the results to optimize this kind of things.” CRM director

The customer service director stated, that they measured conversions and sales from the NBO recommendations also in customer service. In addition, they meas-ured the number of recommendations made by the CSRs. However, the customer service director mentioned that they could not use that reliably as the only metric, as the CSRs were able to tick the box even though they did not really make the NBO recommendation. Thus, they also measured on customer service level, how many CSRs actually made the recommendations, which was the most important metric for them.

“We are thinking about taking the NBO recommendations to our CSRs performance re-ward system. That is the way to get the focus to the NBO, when it is integrated to the performance measurement and reward system.” Customer service director

The analyst stated that her role in measuring the NBO model’s performance was to make sure that the predictions of the model held true and that the model in itself performed as it should perform. Her role was to use the results to optimize the model’s predictions.

“If we see from the reports, that something is not working as it should be, we look for a reason and see if there have been changes in some processes that affect to the predictions.”

Analyst

Further, both the marketing director and CRM director said they measured cus-tomer lifetime value (CLV), but they were not systematically following the met-rics, nor analyzed the value of the NBO for CLV. However, they had made some

estimates about how different actions affect to CLV. The marketing director thought that calculating the effect of different actions to CLV could be useful in-formation but stated, that they either had not had time or capabilities to calculate those yet.

“Probably there would be a lot to do in how the NBO could be measured. For example, if a customer makes a certain action, such as sends a loan application or makes a fund in-vestment, how much the customer’s CLV grows in 3 years for example. Or how much a recommendation made in customer service affects to CLV. At least I haven’t seen this kind of numbers.” Marketing director

However, the data scientist stated, that regarding the NBO the results from single campaigns and activities were measured, but the overall image was not achieved with the prevailing measuring and analyzing methods and resources in the case company.

“To optimize the model in digital marketing channels and marketing activities, the meas-uring and analyzing should be initially put in order. After this, modifying the limits and optimizing marketing actions could be performed. In digital channels, there are possibly many possibilities to optimization. Without full transparency and fact-based results, op-timizing can’t take place. This is the most significant challenge in digital marketing chan-nels regarding the use of the NBO model.” Data scientist

In addition, the data scientist said that long-term profitability impact should be calculated for each product and service which should be used as the base rate for prioritizing the NBO model recommendations. He stated, that the NBO model’s prioritizations were based on gut-feeling and short-term profitability calcula-tions, which ignored the long-term profitability of the NBO recommendations.