Custom Code

If you have a lack functionality on the Datrics platform you can make your own custom brick.

There are two ways to make your own brick:

Datrics CLI can be installed on your computer. Go here to know how to use it.

Datrics web platform.

Datrics web platform

Firstly click on the + button near the Customization folder at the left sidebar and then upload tar archive with brick configuration.

Uploading tar archive with brick configuration

Tar archive can be created by Datrics CLI or by console command.

Datrics CLI

datrics create

datrics push

Console command

tar cvf brick.tar manifest.yml brick_custom_code.py

Upload a tar archive (up to 500 Kbytes) which contains two files: brick_custom_code.py and manifest.yml

Example of brick_custom_code.py:

class BrickCustomCode(Brick):

		def validate_args(self, args):
        return True
    
    def validate_inputs(self):
        if check_number_of_inputs(self, 2):
            if not self.get_input_by_index(0).hasValue():
                self.messages.append('input 0 is empty')
            elif not self.get_input_by_index(1).hasValue():
                self.messages.append('input 1 is empty')

        return len(self.messages) == 0

    def perform_execution(self, ctx: ExecutionContext):
        data1 = self.get_input_by_index(0).value
        data2 = self.get_input_by_index(1).value
        res = pd.concat([data1, data2])
        self.fill_outputs_by_type_with_value('data', res)
        self.status = BrickStatus.SUCCESS

    def perform_light_run(self, ctx: ExecutionContext):
        data1 = self.get_input_by_index(0).value
        data2 = self.get_input_by_index(1).value
        res = pd.concat([data1, data2])
        self.fill_outputs_by_type_with_value('data', res)

Class for all bricks must have the same name - BrickCustomCode and inherit Brick.

Methods validate_args, validate_inputs, perform_execution, perform_light_run are obligatory to implement.

validate_args method is implemented to check the correctness of arguments passed. For example, you can validate if arguments are in a JSON format, if all the needed arguments are present or if they have certain data types and values. To perform such validations you can use the Argument handling functions featured below.

validate_inputs method is used to check the brick inputs, for instance, you can check the number of inputs, whether the inputs are empty or not, or anything else you need for the right model execution. The list of all the useful Input handling functions is here.

perform_execution method sets the commands that are executed during the brick run. You can receive the input by index using self.get_input_by_index(index).value or by type with self.get_input_by_type(type).value. Then you perform some operations and at the end of it, you should fill the brick outputs with the corresponding values using self.fill_outputs_by_type_with_value(type, value) or self.fill_outputs_by_name_with_value(name, value) and set the brick status to SUCCESS or, in case of exception, FAILED.

perform_light_run is implemented to inform the pipeline of the brick output, like data types, columns, etc before its regular run.

Outputs of type data should be a pandas DataFrame.

There is a possibility to pass arguments to the brick in a JSON format. To access the argument value use the following syntax:

column = self.args.get("Column")

manifest.yml helps to make a brick description.

Example:

version: 1.0
title: Custom Union Data
inputs:
  - name: data
    type: data
  - name: data
    type: data
outputs:
  - name: data
    type: data

Now we have only one type of inputs and outputs - data. But you can change the name of that input or output.

Import packages. Custom code bricks do not support imports because of security reasons. But we provided a list of globals that would be helpful for making bricks:

# Brick
DatricsData
Brick
BrickStatus
BrickOutput
BrickInput
ExecutionContext

# Input handling functions
check_number_of_inputs
check_input_with_type_has_value
check_input_model_has_optional_target_variable
check_input_with_index_has_value
check_column_in_input_of_type

# Argument handling functions
check_argument_exists
check_argument_not_none
check_argument_not_empty
check_argument_mathces_type
check_argument_in_list
check_argument_list_in_list
check_argument_in_boundaries

# Calculate metrics functions
transform_types
calculate_metric_r2
calculate_metric_rmse
calculate_metric_mape
calculate_metric_accuracy
calculate_precision_recall_fscore_support

# ML libs
pd or pandas (can use short name and full)
np or numpy (can use short name and full)
ast
scipy
sklearn

# Additional globals
datetime
math
string
re