Big data

First there is a lot of hype on big data. The definition is still unclear but two main component are :

• to big to handle it with current methods, i.e. regular DBMS
• mostly unstructured

For the first issue, which is size, let me point out that eBay have two data warehouse with many petabytes running Teradata. Obviously, Teradata is far from cutting edge new stuff. I didn’t heard of a Hadoop clusters above the single digit petabyte range and the number of instances over the petabyte range seems equals for Hadoop and Teradata (between 15-20).  Therefore, at least we can say that old technology can handle big data. To be fair, we have to say that it’s all about a trade off decision. Hadhoop is good for scale, i.e. it cost a small capital at the begining and can grow accordingly with the business. That come at the cost of effectiveness. Computational effectiveness but human effectiveness as well.

For the unstructured issue, I would say that it’s just not an issue. The first time I saw unstructured data was in an Oracle instance using XML fields (Oracle 9 from a decade ago can handle XML). What was interresting is the reason of that choice of adding an unstructured XML field to a old relational table. The business guys were having ideas at a velocity the development team couldn’t handle. One can say it’s agility at the cost of usability. To me it’s just increasing the technical debt. Structuring data is boring because we have to think on how such data should be structured to benefit the business. Letting it unstructured is pushing the boring part for later, that’s not a good idea.

Related posts:

1. The cost of reducing costs
2. Book review : Marketing calculator
3. What is the value of your work?

Measuring and managing return on marketing investment, that’s the promise of the book from Guy R. Powell. A famous quote in marketing is : Half of my advertising is wasted; I just don’t know which half (John Wannamacher). Indeed, how many marketing initiative are rigorously evaluated? No doubt price cuts increase volumes, but how often does it to improve the top line, not even talking about the bottom line. This book is about putting some analytics in marketing and knowing the truth.

The five step of marketing effectiveness

The book start with a presentation of their marketing effectiveness framework which is a very good one indeed. It’s based around 4P (Price, Product, Place, Promotion), 3C (Competition, Consumers, Channel) and  1E (Exogenous factor like holidays, weather, regulation, …) which are  all stuff which can impact your KPI whether you have control on it (like the P elements) or not. After that presentation, the book goes through 5 levels of marketive effectiveness.

Activity tracker

The first step seems easy, you just have to track your marketing activities and how much was spent for each one. The book propose a test, you should be able to provide in half an hour a spreadsheet of all activities in the last 3 years and all expected activities in the next year. Many companies fail the test.

Campaign measurer

This is what we can expect from a campaign analysis. Each sale which was driven by the marketing campaign is considered as additional revenue thank to the campaign. This is usually a last touch attribution method (if one week, I use a coupon to buy my weekly pack of beer it will be attributed to the campaign even if I don’t care about the campaign). Then, you  have revenue generated by the campaign on one side, the cost on the other and you can easily compute some ROI.

Mix modeler

As marketing campaign are more and more using multiple channels, analyzing one is more and more tricky. The mix modeler approach uses the 4P3C1E as input and output the sales volumes. Learning is usually done by regression analysis. Now you are able to know what channel is the most relevant, what is expected if a competitor launch a marketing campaign. You can also optimize expenditure for each channel to achieve greater ROI.

Consumer analyser

While the mix modeler approach is very campaign focused, the consumer analyser approach focus on the customer by modelizing it as an agent. I think this is very valuable when you already have some customer segments which behave differently. With such systems you can generate any What-If scenario and see what would happen. It’s a bit like the psychohistory science from Asimov which could predict the future of mankind by mathematic modelization. Definitively the holy grail for any business analyst.

Brand optimizer

This chapter focus on managing a portfolio of brand to maximize shareholder value.

My analysis

This book is about marketing meeting analytics and will give you a lot of ideas. Thanks to the 14 case studies you can already find out where to start. Sadly, the book stop here. Both mix modeler and consumer analyzer are really valuable approaches but the journey is long to obtain them even if it seems easy in the book. Case studies are not really detailed so it’s hard to know which shortcuts they take. Nevertheless, the underlying idea of the book is very strong. Considering the amount of money spent in marketing and that much of it is wasted or misallocated, putting at least a bit of analytics in the process could only help.

Related posts:

1. Book review : Actionable Web Analytics
2. Book review : Competing on analytics
3. Book review : Programming Collective Intelligence

Generalities

The aim of an ETL are numerous. Kimball define 34 subsystems. To be more general I primarily focus on

• gain in speed of development : ETL generally use some visual interface with components in order to fasten the process;
• ease the maintenance/evolution : plain SQL request and programming tend to be quite complex very quickly;
• functionalities : if one of you data base contains XML fields or you want to parse web pages you need to hope that your ETL allows it. If not … very dirty things will happens.

SAP Business Object Data Integrator

It’s the only one with not open source I’ve used (version XI r2). An ETL is composed of Work flow and Data flow. Work flows are the structure as they can contains work flows, data flows and orchestration elements (e.g. if then else block). Data flow only work on data manipulation objects : table, join, switch, … Data Integrator is not designed to let you write complex SQL (it’s possible but not recommended), a table element read a table and that’s all. What is great is that Data Integrator will merge table and join elements to create the complex SQL query if everything can be done in one database (let say in a TEL way), . You quickly end up with many elements in your data flows but no big chunk of SQL. Data Integrator is environment aware, you can define a data source and at run time decide which database it represent (production, dev, …). For the downsides, don’t expect copy paste to work, neither expect to do tricky things.

Talend

Talend is an open source product. While Data Integrator has a few very generic components to build, Talend has hundred of components which cover everything you could want. For instance, you have one table input component for each database (even many table input component for one database). No separation between work flow and data flow, only job (but you can have sub jobs). On the downside, it’s amazingly slow to use. I’m also not a big fan that it generate a Java program (it could be very useful in some case but not for a serious process). A lot of bugs in it too (that’s the cost of containing so much).

Kettle

Kettle is also an open source product integrated in the Pentaho suite. To describe it, let’s say that if Talend is shiny, marketing enhanced, Kettle is … functional and unpractical. There is again jobs (work flow) and transformations (data flow), while the first one is executed sequentially, the second one is multi-threaded, expect fun to understand.  There is many program like Spoon (the designer which could execute both jobs and transformations), Kitchen (execute jobs), Pan (execute transformations). Nevertheless, it’s currently my favorite. When you understand the underlying logic everything is easier and behave as expected. Not as complete as Talend but enough for 99% of what I want and you can always use some Java code for the rest.

How I used ETL in the everyday work

Every times I should extract a big chunk of data I use and ETL (usually a simple input query and an output in a CSV or Excel file). It just takes almost no time and you keep a file with all the process in case you need it later. And of course I use ETL to load the data warehouse. Here is some tips :

• Split stuff : Divide your ETL by subjects (accounting, CRM, …), the higher level vision should be simple.
• Keep it ETL : I always dump all data needed in the staging area, make most transformations in SQL (as everything is in the same database, so one table input and one table output to the data warehouse but no “create table as” statement).
• Make it environment friendly : you should have at least a dev environment and a production environment. Therefore you can test your jobs in the dev environment before putting it in production, the job will not change just the runtime configuration.

Related posts:

1. Data Manipulation Part 1 : SQL
2. Using MySQL as a Data Warehouse
3. Data mining tools

Big SQL Queries

Usually to extract data from a database, you create an SQL query and export the result set to a CSV (comma separated values) or Excel file. Depending on your knowledge of Excel and SQL, you do some post-treatment in Excel or not. I think it is more convenient to do everything in the query in order to be able to reproduct the dataset at any time. I found some SQL tips very useful for big analysis.

Common Table Expressions (CTE)

Using subqueries (and sub sub sub … sub queries) makes a query just impossible to understand (at least for me). The WITH keyword makes things simple in a more functional or procedural fashion (without any performance hit, we still use ensemble theory).

with customers as (
/* Do some cleanup in clients */
    select CLIE_CLIENT_ID as id,
        lower(CLIE_NAME) as name,
        decode(CLIE_TYPE, 'I', 'Individual',
            'P', 'Professional', 'Unknown') as category,
        CLIE_DISTRICT as district,
        case when CLIE_DT_BEGIN > sysdate - 365 then 'New'
            else 'Old' end as age
    from CLIENT
    where CLIE_DISABLED = 'F'
),
location as (
/* Find some geographical attributes */
    select DIST_DISTRICT_ID as district,
        DIST_POPULATION as district_pop,
        COUN_POPULATION as country_pop
    from DISTRICT join COUNTRY on DIST_COUNTRY_ID = COUN_COUNTRY_ID
)
/* Merge all */
select id, name, category, age, district_pop, country_pop
from customers natural join location;

It is easy to understand when thing are break in little separated parts which is what the WITH does.

Analytical aggregations

Analytical aggregations are included in SQL since SQL2003 (i.e. support may vary depending on your DBMS).  It allows you to extends each line of a table with aggregated values. Please find a good description here. For time series you can add the moving average at each point (which is sometime more predictive than the point value). For each customer transaction, you can add the sum of the amount for all transaction of this customer.

Analytical join

Analytical queries aren’t the same as transactionnal ones. Usually we need to join many rows together. When joining two tables, RDBMS generally read each rows of one table, let say A,  (or the subset using the where part of the query) and lookup to the second table, let say B, using an index. It requires a lot of random reads and is an iterative operation which is something we want to avoid in a database.  The complexity is a.log(b) with a the number of row which need to be read in A, and b the total number of rows in B. Oracle could do MERGE join and HASH join, I will briefly explain how they work.

MERGE join takes the subset of A (with the WHERE clause) and the subset of B. It order them by the join key, then make the join. If the join is an equijoin (using just ‘=’) it take only one read of both subset to make the join. The sort operation is what is costly here. So you need to restrict the subset of A and B to the minimum or allow the use of an ordered index.

HASH join is like the LOOKUP but it create first a hashmap with the B subset, thus access is very fast (complexity of 1). The construction of the hasmap is b’.log(b’) (not exactly sure) where b’ is the number of rows in the subset of B. If b’ is a lot smaller then b then it’s very efficient.

You could read here for queries optimization in Oracle. I don’t discuss here for bitmap indexes which are of great help but requires DBA action. The analyse table operation is of course very very useful too, it allows Oracle to find the best join strategy for you.

Related posts:

1. Using MySQL as a Data Warehouse
2. Data Manipulation Part 2 : ETL
3. Data mining tools

Preventive reporting

My last model makes five weeks ahead prediction. The result could be changed within one week. After that, it’s just too late.  As these predictions could have dramatic impacts, it is good for me to be sure it wouldn’t cause any mess (learning is done each time on an updated dataset). As there was thousands of predictions each time it would be impossible for me to check everything (beside the fact that it’s tricky sometimes).

I predict not only 5 weeks ahead but also 6, 7 and 8 weeks ahead. Thus I can watch the evolution of the prediction. If results are greatly changing, it’s good to go deeper. It’s also interesting to see if the results are quite the same 10 weeks ahead or 5 weeks ahead. If they are always the same, you could take opportunity to publish results 5 weeks sooner which could unlock a business opportunity.

I also consider an comparison with past data as my prediction are comparable years after years (but of course not exactly the same). This can avoid big mistakes on a particular prediction.

Corrective reporting

Corrective occurs when an evaluation about the prediction can be made. On the top of such report I have one or more dial chart which gives an aggregated indicator, i.e. is the error  acceptable or not. If it seems fine, there is nothing more to do. If not, I have many statistics to find where the error is. Using only aggregated error indicators like root mean squared error don’t give any hint on why the error is so big. As I’m making hundred numeric predictions each time, I compute the mean deviance from the real values. Usually this value should be around 0%, deviations errors compensate themselves. Nevertheless sometimes I see 5%, i.e. on average each prediction is 5% bigger than the value which should be predicted. This reveal an obvious problem (learning set too old or with mistakes, a change in the process, …). You could also compute RMSE on different subspace of the evaluation set, but it seems quite complicated to obtain useful insights.

As conclusion, I found that prediction is not the end of the data miner job if you want some quality. How do you follow your models?

Related posts:

1. Machine learning vs simulation
2. How to : What to do when your model fails?
3. Data mining tools

I think there is a continuum and it depends on the amount of domain knowledge you add versus the amount of learning. Pure machine learning doesn’t take anything else than data. In theory everything could be learnt from it. On the other side, pure simulation considers that everything is known and that no noise is present.

Of course, in real life, it’s a bit more complex than that. You have to inject some domain knowledge in your models to help them : post-treatment rules, adding some nodes ans leafs in a decision tree.  For simulation, often you can’t modelise everything or there is always a parameters you need to estimate on historical data (it’s more statistics than machine learning, but you still need data).

Machine learning never gives perfect results because of the learning approximation. Nevertheless,  the same apply to simulation and it’s sometimes worse. If you give wrong input data to a simulator it will produce crap. Machine learning doesn’t suffer a lot if the error is consistant (by example all your numericals inputs divided by 2). Morever, machine learning can change it’s model faster than using simulation or to much domain knowledge, underlying process can change.  I’m sure you already made a model with some domain knowledge and where removing the knowledge allows better reults. As always there is a tradeoff to make.

These two tools can be both useful and, to some extends, merged together.

Related posts:

1. About evaluation
2. How to : What to do when your model fails?
3. What is the value of your work?

Sequences

Each patient is represented by a sequence (previous visits and the current one). For a given patient we have many lines in file. The sequence is not length fixed so we can’t just put everything on one line with concatenation.

Ensemble attributes

There is also some ensemble attributes (an attribute which the value is an ensemble). In the data file it is represented by Other-Dx-Code-1, Other-Dx-Code-2, … with Other-Dx-Code-9 often missing. There is also Principal-Dx-Code and Admit-Dx-Code which I see part of the ensemble.

Hierarchical attributes

Some attributes are hierarchy. For instance, Hospital-ID and Region-ID are two levels of a geographical hierarchy. I don’t know how hierarchy can be used in data mining (well in a clever way than standard attributes). I could be interesting for generalization purposes and reducing overfitting.

It’s relational

These three problems have in common their relational nature. I think it’s madness to use it directly as a single table, I think that we need to better formalize the problem  first. Then we could construct a single table using to feature of relational data mining, selection graphs and aggregation (either manually or automatically). Notice that the last link is a paper from the contest organiser Claudia Perlich thus I think I couldn’t be so wrong. I don’t know if it’s the better way, but if I do something it will be clearly in this direction.

Related posts:

1. Data mining tools
2. Mining Twitter data
3. Data Manipulation Part 1 : SQL

This problem can be extended to other areas where human presence is important (call center, postal services, …) or when dealing with supplies. Having too much item of a product on the shelve is costly, but having too few is costly too as missed opportunities, but hardly measurable. If you have this value, the whole problem is just an optimisation one : find the quality of service which maximize earning while minimize costs.

I have not so much clues right now (happy to hear yours if you have one).  I read that a Telco in New Zealand use 0.001 as the probability for a customer to switch to competitor if can’t have an operator on a call. However finding this magic number is not easy (I don’t know how they achieve it).

An idea is to get data on what happened when the quality of service was disturbed. But you can’t collect so much data. If you can, your business have a big issue. Moreover, such data will be quite complex to handle. For instance, a customer come in your shop and need to wait 15 minutes before having a sales representative. If this customer never shows again, is it because of the quality of service last time he came? Or because of an external factor you can’t catch (like a new ad campaign by your competitor)? Small dataset, huge complexity and external factors, only a thin chance to get something statistically relevant.

Feel free to comment if you have a hint on this problem.

Related posts:

1. What is the value of your work?
2. Big data and mobile BI : New hype but same old issue
3. Book review : Marketing calculator

Programming Collective Intelligence is a great book. It covers most of the existing data mining algorithms and presents many applications for them.  It covers clustering (k-means, hierarchical), supervised classification (k-nearest neighbours, Naïve Bayes, decision trees, SVM), data analysis (non negative matrix factorization), optimisation (hill climbing, simulated annealing and genetic algorithms) and end with genetic programming. Along the way, it present application like spam detection, pricing, recommendation, … If you want to start in data mining this is a very good way. 0

Example are given in Python, a language I never used. Nevertheless, it is quite easy to follow. Python has a very concise syntax which avoid to have hundred lines of code in the book. Many third party library are used, especially to connect to third party services (facebook, ebay, …) to produce the datasets.

In comparison to Collective Intelligence in Action, this book is more focused on data mining, there is for instance no discussion on how scale to big datasets.  Nevertheless, it contains a lot more information, so I would recomment this book instead of the other.

Related posts:

1. Book review : Collective Intelligence in Action
2. Book review : Competing on analytics
3. Book review : Actionable Web Analytics

Step 1 : What is my model?

If your model is a neural network, it’s quite hard to get any insight of how it works by looking at the weights or neural functions. How could you improve something you don’t understand?

Step 1.1  : Is there an human version please?

There is two models which are easy to understand : decision trees (for classification) and linear regression (for regression). It cost nothing to use them. If their give results close to the initial model, they could be a good estimation and catch most of the inner model logic.

If results of such a simple model are too far from your model, you could consider using ensembles of models learned on different subsets of the learning set.

Step 1.2  : Does it use the attributes I provide?

When you got this simpler models, you get insight on what attributes are used and how. In a linear regression you could look a p-values to know whose attributes have an impact. Be aware that p-value could be misleading when having colinear attributes.

Step 2 : Where and why is my model failing?

Maybe your model works fine half of the time, but really fails on some cases. Try to find where the model makes bigger mistakes.

Using the simpler version in step 1, you should be able to process manually the model on these failing cases and see why it doesn’t work.

Step 3 : What can I do?

At the time of writing I figure two main action that could be used to improve the model : adding more attributes and segmenting the problem. There is another which doesn’t directly improve the model, but improve the results, I call it cheating (in a machine learning perspective).

A new idea of attributes can arise when finding where the model fails. But in general, you can’t get this attribute. For instance, if you need the oil price three month ahead it would be quite challenging to find. And finally, if you are able to predict it, you will get rich enough to forget the initial problem and drink Mojito on a beach all day long.

Segmenting the problem is the idea of making a separate model for subset of your population which behave differently. It’s like making a decision tree to choose which model to use, if the underlying models are decision trees, it would be generally useless to do it manually.

Cheating can be used when you know something about your problem but your model can’t express it. Maybe it’s a multiplicative factor. Of course a linear regression can’t do the trick. Thus, you could use a filter approach (pre-processing and post-processing) to use it.

Step 4 : What if it does not work?

When you are here, i see only one solution, use genetic programming, give it access to all the data you have, every possible mathematical, logical or whatever functions and wait. You could look at  GenIQ pages for a description. 99.9% of the time I think it will not work. But if you have an unused computer you can run it and come back some days, weeks, months after. In the same time you could do something else. It’s computer at work.

Another idea?

This only present the process I’m actually using. If you have another idea, please feel free to leave a comment.

Related posts:

1. Machine learning vs simulation
2. About evaluation
3. Data mining tools