• we have to apply same set of transformation and preprocessing for both training and test set
• So we have to use pipelining for both
• sklearn provides pipeline apis in built
• some typical problems with data -
  • missing values in features
  • numerical attributes are not at the same scale
  • categorical attributes need to be represented with sensible nnumerical representation
  • Often too many features - we have to reduce them
  • sometimes we have to extract features from non numeric data
• Sklearn provides a library of transformer fro data preprocessing
  • Data cleaning ( sklearn.preprocessing) - standarization, missing value imputation
  • Feature extraction( sklearn.feature_extraction ) for extraction of features
  • Feature reduction ( sklearn.decomposition.pca )
  • Feature Expansion ( sklearn.kernel_approximation)
• Transformer methods -
  • fit - method learns model parameter from a traininng set
  • transform - method applies the learnt transformer to the new data
  • fit_transform - perform function of both fit and transform

sklearn.feature_extraction →

• Dictvectorizer → convert dictionary into matrix
• FeatureHasher

Feature extraction from images and text →

• sklearn.feature_extraction.image.*
• sklearn.feature_extraction.text.*

Handling Missing Values

• use sklearn.impute

  • functionality to fill missing values

  • KNNImputer -

    • uses K nearest neighbours of the missing value to fill the missing value
    • based on eucledian distance
    • n_neighbours is a parameter

    from sklearn.impute import KNNImputer
    knn = KNNImputer(n_neighbors=5, weights= 'uniform')
    knn.fit_transform(x)
    

  • SimpleImputer

    • strategies → ‘mean’, ‘median’, ‘most_frequent’, ‘constant’

    from sklearn.impute import SimpleImputer
    si = SimpleImputer(strategy='mean')
    si.fit_transform(x)
    

Numeric Transformers

• Feature Scaling
• Polynomial Transformation
• Discretization

Feature Scaling