Send Additional Metadata with Each Record

You can also send additional metadata with each record for future analysis. For example, you might want to send all the probabilities generated by the soft-max classifier for different classes. The supported meta_types are here.

How does it work?

The below sample code demonstrates the process of creating and registering a recorder to capture evaluation data in MarkovML. Within the validation loop, predictions and corresponding scores are generated for each input value by the model. These predictions, along with actual labels and scores, are encapsulated within SingleTagInferenceRecord objects.

Additionally, metadata in JSON format is included for each record, providing additional context or information related to the predictions. This metadata, such as class probabilities, is added to the records using the add_meta_data_instance method.

Once all records have been processed, the recording is closed by calling finish, signaling the completion of the evaluation data collection process in MarkovML. This enables comprehensive tracking and analysis of model predictions, including relevant metadata, to facilitate informed decision-making and performance evaluation.

Sample Code

import json

# create recorder code 
recorder = ...
recorder.register()

while ...# Your validation loop
   # This is the prediction from your model and score
   predicted_label, score = model.predict(input_value)
   urid= recorder.gen_urid(input_value)
   record = SingleTagInferenceRecord(urid=urid,
                                    inferred=predicted_label, 
                                    actual="YOUR_ACTUAL_LABEL",
                                    score=score)   
   
   ## New Lines to send additional metadata to MarkovML for each record  
   ## This will store the JSON as a string            
   prob_json = json.dumps({"class_a":0.75,"class_b":0.20,"class_c":0.05})
   record.add_meta_data_instance(key="probabilities",
                                 value=prob_json,
                                 meta_type=RecordMetaType.Text)
   ## To visualize the prob_json as a histogram, save it as RecordMetaType.HISTOGRAM
   ## This is helpful when you are interested in visualizing relative value
   prob_json = {"class_a":0.75,"class_b":0.20,"class_c":0.05}
   record.add_meta_data_instance(key="probabilities",
                                 value=prob_json,
                                 meta_type=RecordMetaType.Histogram)
   ##New Lines to send additional metadata to MarkovML for each record                                               
   recorder.add_record(record)

evaluation_recorder.finish()