MarkovML provides integrations with the following machine-learning frameworks to track experiments automatically. Please review the details below.

• Keras
• Pytorch-lightning
• XGboost
• Scikit-learn

Keras

import markov
import keras

MODEL_NAME = "My Test Keras Model"

markov.keras.auto_record(
    name=MODEL_NAME,
    notes="Testing Keras Auto Record with MarkovML",
    project_id="some_project_id"
)

# Continue creating keras model and training

When auto_record is used for Keras models, the following information is recorded:

1. Hyper-parameters:
   1. Information provided by keras.model.optimizer.get_config()
2. Metrics:
   1. Epoch-time vs. epoch
   2. Loss vs. epoch
   3. Accuracy vs. epoch
   4. Learning rate vs. epoch (if using adaptive learning rate using LearningRateScheduler callback)

Pytorch

import markov
import pytorch_lightning

MODEL_NAME = "My Test PL Model"

markov.pytorchlightning.auto_record(
    name=MODEL_NAME,
    notes="Testing PL Auto Record with MarkovML",
    project_id="some_project_id",
    model_class=markov.ModelClass.CLASSIFICATION
)

# Continue creating Pytorch lightning model and training

When auto_recordis used for Pytorch-Lightning models, the following information is recorded:

1. Hyper-parameters:
   1. Information provided in trainer.optimizer.defaults.
2. Metrics
   1. Epoch-time vs. epoch
   2. Average running loss vs. epoch
   3. Learning rate vs. epoch
   4. Custom metrics logged by the user using self.log in the PyTorch-lightning model.

XGBoost

import markov
import xgboos

MODEL_NAME = "My Test XGBoost Model"

markov.xgboost.auto_record(
    name=MODEL_NAME,
    notes="Testing XGBoost Auto Record with MarkovML",
    project_id="some_project_id",
    model_class=markov.ModelClass.CLASSIFICATION
)

# Continue creating the XGBoost model and training

When auto_record is used for XGBoost models, the following information is recorded:

1. Hyper-parameters:
   1. Parameters returned by xgboost.Booster.save_config(). Refer XGBoost documentation for more info.
2. Metrics
   1. Epoch-time vs. epoch
   2. Custom metrics that the user logs. If the user does not specify any metrics, XGBoost, by default, logs logloss on the validation sets specified.

Scikit-learn

import markov

MODEL_NAME = "My Test Sklearn Model"

markov.sklearn.auto_record(
    name=MODEL_NAME,
    notes="Testing SKlearn Auto Record with MarkovML",
    project_id="some_project_id",
    model_class=markov.ModelClass.CLASSIFICATION
)

# Continue creation Sklearn model and training

When auto_record is used for Sklearn models, the following information is recorded:

1. Hyper-parameters:
   1. Parameters returned bysklearn.base.BaseEstimator.get_params().
2. Metrics are not recorded for sklearn

Refer scikit-learn documentation for more info.