Big Data Projects

Investment firms increasingly build models on large and unstructured data, from transaction records to news and earnings-call transcripts. This module walks through the stages of a big data project for structured and text data: cleansing and wrangling raw data, exploring and engineering features, and training, evaluating and tuning models with confusion-matrix metrics, ROC curves and RMSE. It closes with an end-to-end text-based forecasting case of the kind Level II item sets use.

Start this lesson free Practice questions

What you will learn

1. Outline the stages of a big data project for structured and unstructured (text) data.

Big data's four Vs: volume, variety, velocity and veracity; veracity (source reliability) is critical in investing.
Structured projects: conceptualisation, collection, preparation and wrangling, exploration, model training.
Text projects: problem formulation, curation, preparation and wrangling, exploration, model training.
Text curation includes web scraping and obtaining reliable labels for supervised learning.
The process is iterative, and most effort goes into collecting and preparing data.

2. Describe how raw data are cleansed, wrangled and explored, including feature selection and feature engineering for text.

Cleansing errors: incompleteness, invalidity, inaccuracy, inconsistency, non-uniformity and duplication.
Trimming removes outliers; winsorisation replaces them with cutoff values.
Normalisation (X - min)/(max - min) maps to [0, 1]; standardisation (X - mean)/sd is less sensitive to outliers.
Text normalisation: lowercase, remove stop words, stem or lemmatise, tokenise; build a document term matrix with n-grams where useful.
Text feature selection uses document frequency, chi-square and mutual information; feature engineering adds n-grams, named entities and parts of speech.

3. Explain how a model is trained and tuned, and evaluate its performance with confusion-matrix metrics, ROC curves and RMSE.

Training = method selection, performance evaluation and tuning; address class imbalance by under- or oversampling.
Precision = TP/(TP + FP); recall = TP/(TP + FN); accuracy = (TP + TN)/N; F1 = 2PR/(P + R).
ROC curve: true positive rate against false positive rate, FP/(FP + TN); higher AUC means better discrimination.
Use RMSE for continuous targets.
Tune hyperparameters by grid search with cross-validation; diagnose bias versus variance error from training and validation errors.

4. Apply the full workflow to a text-based financial forecasting problem.

Formulate: define the text unit, the classes, the probability output and its use.
Curate: collect text and reliable, representative labels.
Prepare: cleanse, keep meaningful negations, normalise, tokenise and build a document term matrix.
Explore: drop extreme-frequency tokens, select by chi-square or mutual information, engineer bigrams and named entities.
Train and tune: evaluate with confusion matrix, F1 and ROC/AUC; a higher threshold raises precision and lowers recall.

Module at a glance

Big data has volume, variety, velocity and veracity; veracity is the investment practitioner's main worry.
Structured projects: conceptualise, collect, prepare and wrangle, explore, train; text projects: formulate, curate, prepare, explore, train.
Cleansing fixes incomplete, invalid, inaccurate, inconsistent, non-uniform and duplicate data; trimming deletes and winsorisation caps outliers.
Normalise with (X - min)/(max - min); standardise with (X - mean)/sd.
Text: lowercase, handle stop words and negations, stem or lemmatise, tokenise, build a document term matrix, select by document frequency, chi-square or mutual information.
Precision = TP/(TP + FP), recall = TP/(TP + FN), F1 = harmonic mean; use ROC/AUC and F1 rather than accuracy when classes are imbalanced.
A higher classification threshold raises precision and lowers recall and the false positive rate.

This free module has 4 lessons, 15 practice questions with full explanations.

Start the lesson