module heapyc

Methods

HeapView(root: Any+, heapdefs: HeapDefs+) -> HeapView

interpreter(command: command+ [locals: dict+]) -> thread_id

NodeGraph( [edges: iterable+ [is_mapping: boolean+]]) -> NodeGraph

set_async_exc(thread_id: thread_id+, exception: Exception+)

xmemstats()

Attribute

RootState:  anything

Returns a new HeapView object.

Arguments

root: Any+

The initial value of the root member.

heapdefs: HeapDefs+

Definitions of specially treated extension types.

Create a new interpreter structure with a new thread.

Arguments

command: command+

A command that will be exec'd in the new environment.

locals: dict+

Local variables passed to the command when exec'd.

Returns the thread identity number.

The new interpreter and thread is started in a new environment. This environment consists of a new '__main__' module, with the optional locals dict as local variables.

The site-specific initializations are not automatically made. To do that, 'import site' could be used as the first statement in the command string.

The interpreter() function will return after the new thread structure has been created. The command will execute sooner or later. The thread will terminate, and the interpreter structure be deallocated, when the command has been executed, and dependent threads have terminated.

Construct a new NodeGraph object.

Arguments

edges: iterable+

The edges that will be used to initialize the new nodegraph. It should yield a sequence of pairs being edges of the form (source, target).

Default: ()

is_mapping: boolean+

If True, will cause the nodegraph to be treated like a 'mapping'. It will then, for the purpose of indexing, be expected to contain a single target for each source node.

Default: False

Set an exception to be raised asynchronously in a thread.

Print system-dependent memory statistics.

What is printed depends on the system configuration.

The single instance of RootStateType.

HeapView

Constructor

module heapyc.HeapView(root: Any+, heapdefs: HeapDefs+) -> HeapView

For any object HV of kind HeapView:

Methods

HV.cli_class() -> ObjectClassifier

HV.cli_clodo(owners: NodeGraph+, memo: dict+) -> ObjectClassifier

HV.cli_indisize(memo: dict+) -> ObjectClassifier

HV.cli_none() -> ObjectClassifier

HV.cli_rcs(referrers: NodeGraph+, classifier: ObjectClassifier+, memo: dict+) -> ObjectClassifier

HV.cli_type() -> ObjectClassifier

HV.heap() -> NodeSet

HV.indisize_sum(S: iterable+) -> int

HV.numedges(src: Any+, tgt: Any+) -> int

HV.reachable(X: NodeSet+, Y: NodeSet+) -> NodeSet

HV.reachable_x(X: NodeSet+, Y: NodeSet+) -> NodeSet

HV.register__hiding_tag__type(type_with_hiding_tag+)

HV.register_hidden_exact_type(type+)

HV.relate(src: Any+, tgt: Any+) -> RelationStructure

HV.relimg(S: iterable+) -> NodeSet

HV.shpathstep(G: NodeGraph+, U: NodeSet+, S: NodeSet+ [AvoidEdges: NodeGraph+ [find_one: boolean+]]) -> NodeSet

HV.update_dictowners(owners: NodeGraph+)

HV.update_referrers(X: NodeGraph+, Y: NodeSet+)

HV.update_referrers_completely(X: NodeGraph+)

Attributes

HV._hiding_tag_:  anything

HV.delete_extra_type:  anything

HV.is_hiding_calling_interpreter: boolean

HV.is_using_traversing_owner_update: boolean

HV.limitframe:  (

^either:[None ^or frame]

)

A HeapView object provides methods to get memory related information about the system heap and about individual objects.

It implements much of the low-level functionality for the Heapy system. It is intended to provide what can not be done at all or would be much slower if programmed directly in Python. It is not intended to be used directly by a user, but to be wrapped in higher level objects.

Some terms that are referred to in the method descriptions:

Visible objects.

The HeapView object attempts to restrict its view of the heap to only the 'visible objects'. This is to make it possible to analyse the heap via a Python library that inevitably itself is continually allocating and deallocating objects. These should be hidden from the heap view presented. This is primarily done via a special tag attribute, see _hiding_tag_ and register__hiding_tag__type . Frames can be hidden with another mechanism, see limitframe. For hiding all objects of a special type, register_hidden_exact_type may be used. It is also possible to use a separate interpreter and hide its root objects, see is_hiding_calling_interpreter .

Classifiers.

Individual size.

The individual size of an object is its individually allocated memory size.

It includes:

• The basic object size, as can be found out in a standard way.
• The extra memory for variable size objects.
• For GC collected objects, the size of the GC information.
• An alignment to the next highest multiple of a pointer size.
• The size of any other memory allocated that belongs to the object.

Some types of objects have extra memory allocated that can not be accounted for in the standard way. This memory should nevertheless be included in the individual size. To determine the size of these objects, special functions are needed. These are defined for standard builtin types, such as lists and dicts. Other types should be defined via the heapdefs argument to the HeapView constructor.

The individual size does not include:

• Subobjects that are accounted for separately.
• Overhead for the memory allocation system. This varies depending on the kind of memory allocator, the requested size, etc.

Returns a classifier that classifies by "class".

If an object is of instance type, its classification will be its class, as given in its C-level member 'in_class', otherwise it will be its type, as given in 'ob_type'.

Note

This is like classifying by the Python-level '__class__' attribute in newer versions of Python, except that even though the '__class__' attribute may be redefined via the class definition, the classification as made by cli_class isn't affected.

Returns a classifier, that classifies by "Class Or Dict Owner".

The classification of an object is its class (as by cli_class), unless the object is a dict object that is 'owned' by some owner. If the object is such an owned dict, the classification will be a 1-tuple containing the class (as by cli_class) of its owner.

Arguments

owners: NodeGraph+

Used to map each dict object to its owner, or to None if it has no owner. The graph will be automatically updated, from heap information defined by HV, whenever an attempt is made to classify a dict that maps to nothing.

memo: dict+

Used to memoize the 1-tuples generated when classifying owned dicts.

Returns a classifier that classifies by "individual size".

The classification of each object is an int, containing the object's individual memory size.

Argument

memo: dict+

Used to memoize the classification objects.

Returns a classifier that classifies all objects the same.

The classification of each object is None.

Returns a classifier that classifies by Referrer Classification Set.

The classification of an object is the classifications of its referrers, collected in an immutable NodeSet object.

Arguments

referrers: NodeGraph+

Used to map each object to its referrers.

classifier: ObjectClassifier+

Used to classify each referrer.

memo: dict+

Used to memoize the classification sets.

Returns a classifier that classifies by type.

The classification of each object is the type, as given by its C-level member 'ob_type'. (This is the same as the type returned by the Python-level builtin 'type'.)

Returns a set containing all 'visible objects' in the heap view defined by HV.

See also

Visible objects

Returns the sum of the 'individual size' of the objects in S.

See also

Individual size

Returns the number of edges from src to tgt.

Returns the set of objects reached via a path in the visible heap as defined by HV, from some object in X, avoiding any object in Y.

Returns the set of objects reached via a path in the visible heap as defined by HV, from some object in X, avoiding any object in Y except at the end of the path.

Register a type of objects that may be hidden from the heap view defined by HV. The type must have a slot named _hiding_tag_. An object that is an instance of the type, or of a subtype, is hidden when its _hiding_tag_ is HV._hiding_tag_.

Register a type of objects that should be hidden from the heap view defined by HV. Objects of the exact type registered -- not including subtypes -- will be hidden.

Returns a description of the relation between src and tgt.

This is used for descriptions of edges in paths.

[The result is in a special format that I choose to not define here since it is for special low-level use and subject to change.]

Returns the 'relational image of HV wrt S'.

That is, the set of nodes that are directly referred to from the nodes in S via the visible heap reachability relation as defined by HV.

This method implements one step of a shortest path algorithm.

Arguments

G: NodeGraph+

Updated by the method, with the edges from nodes in the source set to the new nodes visited.

U: NodeSet+

The source set for this step.

S: NodeSet+

The set of already visited nodes.

AvoidEdges: NodeGraph+

Edges to avoid.

find_one: boolean+

If True, at most one edge will be found from each node in the source set. Normally, all edges will be found.

Returns the new nodes visited.

This may be used for the U argument the next time the method is called.

See also

shpgraph_algorithm in Path.py.

Update owners with ownership edges.

The dict owners graph will be updated with an edge from each dict object in the heap, to either its owner or to None.

Update referrer graph X for Y.

The visible heap defined by HV will be traversed from the root of HV so that the edges of every path from the root to nodes in Y will be represented, inverted, in X.

Update referrer graph X 'completely'.

[Experimental algorithm that updates X with the referrers to all objects in the heap (of visible nodes as defined in HV). It is not normally used.]

The hiding tag defining what objects are hidden from the view defined by HV. Objects that contain a _hiding_tag_ object which is identical to HV._hiding_tag_, will be hidden from view, in the following cases:

• The object is of a type that has been registered for hiding via register__hiding_tag__type, or is of a subtype of such a type.
• The object is of instance type. Such an object will be checked for a _hiding_tag_ item in its __dict__.

For Internal Use

If True, the data of the interpreter using the HV will be hidden from the heap view as seen from RootState.

This is used when multiple Python interpreters are used. One interpreter will be monitoring the operation of the other interpreter(s). It would set is_hiding_calling_interpreter to True in the HV it is using. Its own data will then be hidden from view, making memory leak detection more practical.

True if update_dictowners is using a recursive traversal algorithm to find the dicts in the heap. When False, the normal case, it will use the dicts found in the gc collection structure, by gc.get_objects(). This was found to be much faster in usual cases, but the old version is available by setting this flag. -- It may be removed in a later release!

^either:[None ^or frame]

)

The traversal limiting frame.

If limitframe is set to a frame object, the frames that are more recently entered than limitframe will be hidden when traversing the heap from the root RootState. It will start traversing from limitframe rather than from the most recent frame as it would otherwise do.

NodeGraph

Constructor

module heapyc.NodeGraph( [edges: iterable+ [is_mapping: boolean+]]) -> NodeGraph

For any object NG of kind NodeGraph:

Methods

NG.add_edge(source: Any+, target: Any+)

NG.add_edges_n1(srcs: iterable+, tgt: Any+)

NG.as_flat_list() -> list

NG.clear()

NG.copy() -> NodeGraph

NG.domain_covers(X: iterable+) -> boolean

NG.domain_restricted(X: iterable+) -> NodeGraph

NG.get_domain() -> NodeSet

NG.get_range() -> NodeSet

NG.invert()

NG.inverted() -> NodeGraph

NG.relimg(X: iterable+) -> NodeSet

NG.update(X: iterable+)

NG.updated(X: iterable+) -> NodeGraph

Operators

len(NG) -> int

# NG[source: Any+] -> Any

# NG[source: Any+] = target_spec: TargetSpec+

iter(NG) -> iterator

Attributes

NG._hiding_tag_:  anything

NG.is_mapping: boolean

NG.is_sorted: boolean

NodeGraph objects are used internally in the Heapy system, for example to record dict ownership and shortest-path graphs.

They may be used generally for mapping and dict-like purposes, but differ from dicts in the following:

• The mapping is based on object identity - no equality or hashing is assumed, so any object can be used as a key. Only the address is used. To distinguish this usage from that of ordinary dicts and sets, such objects are called 'nodes'.
• There may be any number of targets associated with each source.
• Performance characteristics differ from dicts, in somewhat subtle ways.

Add to NG, an edge from source to target.

Add to NG, for each src in srcs, an edge from src to tgt.

Returns the edges of NG in the form [src0, tgt0, src1, tgt1 ...].

Remove all items from NG.

Returns a copy of NG.

Returns True if each node in X is the source of some edge in NG, False otherwise.

Returns a new NodeGraph, containing those edges in NG that have the source in X.

Returns the set of nodes that are the source of some edge in NG.

Returns the set of nodes that are the target of some edge in NG.

Invert the edges of NG.

Returns a copy of NG with the edges inverted.

Returns the relational image of NG wrt X.

That is, the set of nodes that are the target of some edge that have its source in X.

Update NG with the edges from X, specified as pairs of the form (source, target).

Returns a copy of NG updated with the edges from X, specified as pairs of the form (source, target).

Returns the number of edges in NG.

Returns the target(s) of all edges with a particular source. The value depends on if NG was initialized to be a 'mapping' or not:

NG.is_mapping == False

Return a tuple of all targets of edges from the source. The tuple will be empty if there are no such edges.

NG.is_mapping == True

Return the target of the edge from the source. If there is no such edge, KeyError will be raised. If there is more than one edge, ValueError will be raised.

NG.is_mapping == False

The target_spec argument is a tuple of targets. There must already be the same number of edges in NG from the given source as the number of objects in target_spec.

NG.is_mapping == True

The target_spec argument is the target itself. There must already be exactly one edge in NG from the given source.

See also

add_edge, add_edges_n1

Returns an iterator yielding a pair (source, target) for each edge in NG.

The hiding tag: if it is the the same object as HeapView._hiding_tag_ of a HeapView object, the nodegraph will be hidden from the corresponding heap view .

Read only. True if NG is a 'mapping'. Then, only one edge is allowed for each source; indexing returns the actual target object instead of a tuple of targets.

Read only. True if NG is sorted.

It will become unsorted after any update. It will need to be sorted to make it possible to find edges (implementation uses binary search). Any indexing operation will automatically sort it if it was not already sorted. The flag is currently used from Python to see if the nodegraph has been used at least once after update, so that it will not be cleared too early.