generator.Process module¶

class generator.Process.Process(ptype, template, process_id=None)[source]¶

Bases: object

Main interface for basic process functionality

The Process class is intended to be inherited by specific process classes (e.g., IntegrityCoverage) and provides the basic functionality to build the channels and links between processes.

Child classes are expected to inherit the __init__ execution, which basically means that at least, the child must be defined as:

class ChildProcess(Process):
    def__init__(self, **kwargs):
        super().__init__(**kwargs)

This ensures that when the ChildProcess class is instantiated, it automatically sets the attributes of the parent class.

This also means that child processes must be instantiated providing information on the process type and jinja2 template with the nextflow code.

Parameters:

ptype : str

Process type. See Process.accepted_types.

template : str

Name of the jinja2 template with the nextflow code for that process. Templates are stored in generator/templates.

Attributes

template_str Class property that returns a populated template string

Methods

`render`(template, context)	Wrapper to the jinja2 render method from a template file
`set_channels`(**kwargs)	General purpose method that sets the main channels
`set_secondary_channel`(source, channel_list)	General purpose method for setting a secondary channel

accepted_types = None¶: list: Accepted process types

pid = None¶: int: Process ID number that represents the order and position in the generated pipeline

process_id = None¶: int or str: optional Process ID that has no effect on the setup of the pipeline channels. It’s used for the POST requests of each main process and is mapped to the process IDs of the innuendo/oneida platform

ptype = None¶: str: Process type. See accepted_types.

template = None¶: str: Template name for the current process. This string will be used to fetch the file containing the corresponding jinja2 template in the _set_template() method

_template_path = None¶: str: Path to the file containing the jinja2 template file. It’s set in _set_template().

input_type = None¶: str: Type of expected input data. Used to verify the connection between two processes is viable.

output_type = None¶: str: Type of output data. Used to verify the connection between two processes is viable.

ignore_type = None¶: boolean: If True, this process will ignore the input/output type requirements. This attribute is set to True for terminal singleton forks in the pipeline.

ignore_pid = None¶: boolean: If True, this process will not make the pid advance. This is used for terminal forks before the end of the pipeline.

dependencies = None¶: list: Contains the dependencies of the current process in the form of the Process.template attribute (e.g., [fastqc])

_main_in_str = None¶: str: String used to specify the prefix of main input channel.

_main_out_str = None¶: str: String used to specify the prefix of main output channel.

_input_channel = None¶: str: Place holder of the main input channel for the current process. This attribute can change dynamically depending on the forks and secondary channels in the final pipeline.

_output_channel = None¶: str: Place holder of the main output channel for the current process. This attribute can change dynamically depending on the forks and secondary channels in the final pipeline.

link_start = None¶: list: List of strings with the starting points for secondary channels. When building the pipeline, these strings will be matched with equal strings in the link_end attribute of other Processes.

link_end = None¶: list: List of dictionaries containing the a string of the ending point for a secondary channel. Each dictionary should contain at least two key/vals: {"link": <link string>, "alias":<string for template>}

status_channels = None¶: list: Name of the status channels produced by the process. By default, it sets a single status channel. If more than one status channels are required for the process, list each one in this attribute (e.g., FastQC.status_channels)

status_strs = None¶: str: Name of the status channel for the current process. These strings will be provided to the StatusCompiler process to collect and compile status reports

forks = None¶: list: List of strings with the literal definition of the forks for the current process, ready to be added to the template string.

_context = None¶: dict: Dictionary with the keyword placeholders for the string template of the current process.

_set_template(template)[source]¶

Sets the path to the appropriate jinja template file

When a Process instance is initialized, this method will fetch the location of the appropriate template file, based on the template argument. It will raise an exception is the template file is not found. Otherwise, it will set the Process.template_path attribute.

_set_main_channel_name(ptype)[source]¶

Sets the prefix for the main channel depending on the process type

Pre-assembly types are set to MAIN_fq, while post-assembly are set to MAIN_assembly. This distinction is important to allow the forking of the last main channel with FastQ files or with assembly files.

static render(template, context)[source]¶

Wrapper to the jinja2 render method from a template file

Parameters:

template : str

Path to template file.

context : dict

Dictionary with kwargs context to populate the template

template_str¶

Class property that returns a populated template string

This property allows the template of a particular process to be dynamically generated and returned when doing Process.template_str.

Returns:

x : str

String with the complete and populated process template

set_channels(**kwargs)[source]¶

General purpose method that sets the main channels

This method will take a variable number of keyword arguments to set the Process._context attribute with the information on the main channels for the process. This is done by appending the process ID (Process.pid) attribute to the input, output and status channel prefix strings. In the output channel, the process ID is incremented by 1 to allow the connection with the channel in the next process.

The **kwargs system for setting the Process._context attribute also provides additional flexibility. In this way, individual processes can provide additional information not covered in this method, without changing it.

Parameters:

kwargs : dict

Dictionary with the keyword arguments for setting up the template context

set_secondary_channel(source, channel_list)[source]¶

General purpose method for setting a secondary channel

This method allows a given source channel to be forked into one or more channels and sets those forks in the Process.forks attribute. Both the source and the channels in the channel_list argument must be the final channel strings, which means that this method should be called only after setting the main channels.

If the source is not a main channel, this will simply create a fork or set for every channel in the channel_list argument list:

SOURCE_CHANNEL_1.into{SINK_1;SINK_2}

If the source is a main channel, this will apply some changes to the output channel of the process, to avoid overlapping main output channels. For instance, forking the main output channel for process 2 would create a MAIN_2.into{...}. The issue here is that the MAIN_2 channel is expected as the input of the next process, but now is being used to create the fork. To solve this issue, the output channel is modified into _MAIN_2, and the fork is set to the channels provided channels plus the MAIN_2 channel:

_MAIN_2.into{MAIN_2;MAIN_5;...}

Parameters:

source : str

String with the name of the source channel

channel_list : list

List of channels that will receive a fork of the secondary channel

class generator.Process.Status(**kwargs)[source]¶