pickling module ¶

Utilities for pickling.

rec_info_registry dict ¶

Registry with instances of RecInfo keyed by RecInfo.id_.

Populate with the required information if any instance cannot be unpickled.

compress function ¶

compress(
    bytes_,
    compression=None,
    file_name=None,
    **compress_kwargs
)

Compress bytes.

For compression options, see extensions.compression under pickling.

Keyword arguments compress_kwargs are passed to the compress method of the compressing package.

decompress function ¶

decompress(
    bytes_,
    compression=None,
    file_name=None,
    **decompress_kwargs
)

Decompress bytes.

For compression options, see extensions.compression under pickling.

Keyword arguments decompress_kwargs are passed to the decompress method of the decompressing package.

dumps function ¶

dumps(
    obj,
    compression=None,
    compress_kwargs=None,
    **kwargs
)

Pickle an object to a byte stream.

Uses dill when available.

Compression is done with compress.

Other keyword arguments are passed to the dumps method of the pickling package.

get_class_from_id function ¶

get_class_from_id(
    class_id
)

Get the class from a class id.

get_compression_extensions function ¶

get_compression_extensions(
    cls_name=None
)

Get all supported compression extensions from vectorbtpro._settings.

get_id_from_class function ¶

get_id_from_class(
    obj
)

Get the class id from a class.

If the object is an instance or a subclass of Pickleable and Pickleable._rec_id is not None, uses the reconstruction id. Otherwise, returns the path to the class definition with find_class.

get_serialization_extensions function ¶

get_serialization_extensions(
    cls_name=None
)

Get all supported serialization extensions from vectorbtpro._settings.

load function ¶

load(
    path,
    compression=None,
    decompress_kwargs=None,
    **kwargs
)

Read a byte stream from a file and unpickle.

Uses load_bytes for reading and decompression and loads for deserialization.

load_bytes function ¶

load_bytes(
    path,
    compression=None,
    decompress_kwargs=None
)

Read a byte stream from a file.

Can recognize the compression algorithm based on the extension.

loads function ¶

loads(
    bytes_,
    compression=None,
    decompress_kwargs=None,
    **kwargs
)

Unpickle an object from a byte stream.

Uses dill when available.

Decompression is done with decompress.

Other keyword arguments are passed to the loads method of the pickling package.

reconstruct function ¶

reconstruct(
    cls,
    rec_state
)

Reconstruct an instance using a class and a reconstruction state.

save function ¶

save(
    obj,
    path=None,
    mkdir_kwargs=None,
    compression=None,
    compress_kwargs=None,
    **kwargs
)

Pickle an object to a byte stream and write to a file.

Uses dumps for serialization and save_bytes for compression and writing.

save_bytes function ¶

save_bytes(
    bytes_,
    path,
    mkdir_kwargs=None,
    compression=None,
    compress_kwargs=None
)

Write a byte stream to a file.

Can recognize the compression algorithm based on the extension.

suggest_compression function ¶

suggest_compression(
    file_name
)

Suggest compression based on the file name.

Pickleable class ¶

Pickleable()

Superclass that defines abstract properties and methods for pickle-able classes.

If any subclass cannot be pickled, override the Pickleable.rec_state property to return an instance of RecState to be used in reconstruction. If the class definition cannot be pickled (e.g., created dynamically), override its _rec_id with an arbitrary id string, dump/save the class, and before loading, map this id to the class in rec_id_map. This will use the mapped class to construct a new instance.

Superclasses

Base

Inherited members

Subclasses

decode_config class method ¶

Pickleable.decode_config(
    str_,
    parse_literals=True,
    run_code=True,
    pack_objects=True,
    use_refs=True,
    use_class_ids=True,
    code_context=None,
    parser_kwargs=None,
    check_type=True,
    **kwargs
)

Decode an instance from a config string.

Can parse configs without sections. Sections can also become sub-dicts if their names use the dot notation. For example, section a.b will become a sub-dict of the section a and section a.b.c will become a sub-dict of the section a.b. You don't have to define the section a explicitly, it will automatically become the outermost key.

If a section contains only one pair _ = _, it will become an empty dict.

If parse_literals is True, will detect any Python literals and containers such as True and []. Will also understand np.nan, np.inf, and -np.inf.

If run_code is True, will run any Python code prepended with !. Will use the context code_context together with already defined np (NumPy), pd (Pandas), and vbt (vectorbtpro).

Warning

Unpickling byte streams and running code has important security implications. Don't attempt to parse configs coming from untrusted sources as those can contain malicious code!

If pack_objects is True, will look for class paths prepended with @ in section names, construct an instance of RecState (any other keyword arguments will be included to init_kwargs), and finally use reconstruct to reconstruct the unpacked object.

If use_refs is True, will substitute references prepended with & for actual objects. Constructs a DAG using graphlib.

If use_class_ids is True, will substitute any class ids prepended with @ with the corresponding class.

Other keyword arguments are forwarded to Pickleable.decode_config_node.

Usage

File types.ini:

string = 'hello world'
boolean = False
int = 123
float = 123.45
exp_float = 1e-10
nan = np.nan
inf = np.inf
numpy = !np.array([1, 2, 3])
pandas = !pd.Series([1, 2, 3])
expression = !dict(sub_dict2=dict(some="value"))
mult_expression = !import math; math.floor(1.5)

>>> from vectorbtpro import *

>>> vbt.pprint(vbt.pdict.load("types.ini"))
pdict(
    string='hello world',
    boolean=False,
    int=123,
    float=123.45,
    exp_float=1e-10,
    nan=np.nan,
    inf=np.inf,
    numpy=<numpy.ndarray object at 0x7fe1bf84f690 of shape (3,)>,
    pandas=<pandas.core.series.Series object at 0x7fe1c9a997f0 of shape (3,)>,
    expression=dict(
        sub_dict2=dict(
            some='value'
        )
    ),
    mult_expression=1
)

File refs.ini:

[top]
sr = &top.sr

[top.sr @pandas.Series]
data = [10756.12, 10876.76, 11764.33]
index = &top.sr.index
name = 'Open time'

[top.sr.index @pandas.DatetimeIndex]
data = ["2023-01-01", "2023-01-02", "2023-01-03"]

>>> vbt.pdict.load("refs.ini")["sr"]
2023-01-01    10756.12
2023-01-02    10876.76
2023-01-03    11764.33
Name: Open time, dtype: float64

decode_config_node class method ¶

Pickleable.decode_config_node(
    key,
    value,
    **kwargs
)

Decode a config node.

dumps method ¶

Pickleable.dumps(
    rec_state_only=False,
    **kwargs
)

Pickle the instance to a byte stream.

Optionally, you can set rec_state_only to True if the instance will be later unpickled directly by the class.

encode_config method ¶

Pickleable.encode_config(
    top_name=None,
    unpack_objects=True,
    compress_unpacked=True,
    use_refs=True,
    use_class_ids=True,
    nested=True,
    to_dict=False,
    parser_kwargs=None,
    **kwargs
)

Encode the instance to a config string.

Based on Pickleable.rec_state. Raises an error if None.

Encodes to a format that is guaranteed to be parsed using Pickleable.decode_config. Otherwise, an error will be thrown. If any object cannot be represented using a string, uses dumps to convert it to a byte stream.

When unpack_objects is True and an object is an instance of Pickleable, saves its reconstruction state to a separate section rather than the byte stream. Appends @ and class name to the section name. If compress_unpacked is True, will hide keys in RecState that have empty values. Keys in RecState will be appended with ~ to avoid collision with user-defined keys having the same name.

If use_refs is True, out of unhashable objects sharing the same id, only the first one will be defined while others will store the reference (& + key path) to the first one.

Note

The initial order of keys can be preserved only by using references.

If use_class_ids is True, substitutes any class defined as a value by its id instead of pickling its definition. If get_id_from_class returns None, will pickle the definition.

If the instance is nested, set nested to True to represent each sub-dict as a section.

Other keyword arguments are forwarded to Pickleable.encode_config_node.

encode_config_node method ¶

Pickleable.encode_config_node(
    key,
    value,
    **kwargs
)

Encode a config node.

file_exists class method ¶

Pickleable.file_exists(
    *args,
    **kwargs
)

Return whether a file already exists.

Resolves the file path using Pickleable.resolve_file_path.

getsize method ¶

Pickleable.getsize(
    readable=True,
    **kwargs
)

Get size of this object.

load class method ¶

Pickleable.load(
    path=None,
    file_format=None,
    compression=None,
    **kwargs
)

Unpickle/decode the instance from a file.

Resolves the file path using Pickleable.resolve_file_path.

loads class method ¶

Pickleable.loads(
    bytes_,
    check_type=True,
    **kwargs
)

Unpickle an instance from a byte stream.

modify_state class method ¶

Pickleable.modify_state(
    rec_state
)

Modify the reconstruction state before reconstruction.

rec_state class property ¶

Reconstruction state of the type RecState.

resolve_file_path class method ¶

Pickleable.resolve_file_path(
    path=None,
    file_format=None,
    compression=None,
    for_save=False
)

Resolve a file path.

A file must have either one or two extensions: file format (required) and compression (optional). For file format options, see pickle_extensions and config_extensions. For compression options, see compression_extensions. Each can be provided either via a suffix in path, or via the argument file_format and compression respectively.

When saving, uses file_format and compression from pickling as default options. When loading, searches for matching files in the current directory.

Path can be also None or directory, in such a case the file name will be set to the class name.

save method ¶

Pickleable.save(
    path=None,
    file_format=None,
    compression=None,
    mkdir_kwargs=None,
    **kwargs
)

Pickle/encode the instance and save to a file.

Resolves the file path using Pickleable.resolve_file_path.

RecInfo class ¶

RecInfo(
    *args,
    **kwargs
)

Class that represents information needed to reconstruct an instance.

Superclasses

Inherited members

cls field ¶

Class.

id_ field ¶

Identifier.

modify_state field ¶

Callback to modify the reconstruction state.

register method ¶

RecInfo.register()

Register self in rec_info_registry.

RecState class ¶

RecState(
    *args,
    **kwargs
)

Class that represents a state used to reconstruct an instance.

Superclasses

Inherited members

attr_dct field ¶

Dictionary with names and values of writeable attributes.

init_args field ¶

Positional arguments used in initialization.

init_kwargs field ¶

Keyword arguments used in initialization.

pdict class ¶

pdict(
    *args,
    **kwargs
)

Pickleable dict.

Superclasses

Base
Comparable
Pickleable
Prettified
builtins.dict

Inherited members

Subclasses

equals method ¶

pdict.equals(
    other,
    check_types=True,
    **kwargs
)

Check two objects for equality.

load_update method ¶

pdict.load_update(
    path=None,
    clear=False,
    **kwargs
)

Load dumps from a file and update this instance in-place.

pickling module¶

rec_info_registry dict¶

compress function¶

decompress function¶

dumps function¶

get_class_from_id function¶

get_compression_extensions function¶

get_id_from_class function¶

get_serialization_extensions function¶

load function¶

load_bytes function¶

loads function¶

reconstruct function¶

save function¶

save_bytes function¶

suggest_compression function¶

Pickleable class¶

decode_config class method¶

decode_config_node class method¶

dumps method¶

encode_config method¶

encode_config_node method¶

file_exists class method¶

getsize method¶

load class method¶

loads class method¶

modify_state class method¶

rec_state class property¶

resolve_file_path class method¶

save method¶

RecInfo class¶

cls field¶

id_ field¶

modify_state field¶

register method¶

RecState class¶

attr_dct field¶

init_args field¶

init_kwargs field¶

pdict class¶

equals method¶

load_update method¶

pickling module ¶

rec_info_registry dict ¶

compress function ¶

decompress function ¶

dumps function ¶

get_class_from_id function ¶

get_compression_extensions function ¶

get_id_from_class function ¶

get_serialization_extensions function ¶

load function ¶

load_bytes function ¶

loads function ¶

reconstruct function ¶

save function ¶

save_bytes function ¶

suggest_compression function ¶

Pickleable class ¶

decode_config class method ¶

decode_config_node class method ¶

dumps method ¶

encode_config method ¶

encode_config_node method ¶

file_exists class method ¶

getsize method ¶

load class method ¶

loads class method ¶

modify_state class method ¶

rec_state class property ¶

resolve_file_path class method ¶

save method ¶

RecInfo class ¶

cls field ¶

id_ field ¶

modify_state field ¶

register method ¶

RecState class ¶

attr_dct field ¶

init_args field ¶

init_kwargs field ¶

pdict class ¶

equals method ¶

load_update method ¶