Differences

This shows you the differences between two versions of the page.

--- calculate_python_expression [2026/07/19 22:13]
hermann
+++ calculate_python_expression [2026/07/22 18:15] (current)
hermann
@@ Line 1: / Line 1: @@
-====== Calculate R Expression ======
+====== Calculate Python Expression ======
 ===== Description =====
-This is a **[[ego_script#container_functors|container functor]]** that calls R externally and processes the script's outputs as if they were part of Dinamica EGO itself. Like the other calculator functors, data is connected through hook functors placed inside its ''<nowiki>{{ … }}</nowiki>'' block.
+This is a **[[ego_script#container_functors|container functor]]** that runs a Python instance with the user-defined expression. Like the other calculator functors, data is connected through hook functors placed inside its ''<nowiki>{{ … }}</nowiki>'' block. See also [[calculate_r_expression|Calculate R Expression]] for the equivalent functor using R.
 ===== Inputs =====
 ^ Name ^ Type ^ Description ^
-| Expression | [[ego_script#constants|Code]] | The expression that will run on R. ''Code'' values cannot be written as plain text constants in EGO Script — see [[#writing_the_expression_in_ego_script|Writing the expression in EGO Script]] below. |
+| expression | [[ego_script#constants|Code]] | The expression that will run on Python. ''Code'' values cannot be written as plain text constants in EGO Script — see [[#writing_the_expression_in_ego_script|Writing the expression in EGO Script]] below. |
-| Treat Warning As Errors | [[Boolean Value Type]] | Warnings raised by the R script will be treated as errors. |
 ===== Optional Inputs =====
-None.
+^ Name ^ Type ^ Description ^
+| packages //(advanced)// | String | Required packages to be installed by PIP (one per line). Each package can be identified either by its name or by specifying a filename or URL pointing to the corresponding wheel file. Packages that are already installed will be ignored. |
 ===== Outputs =====
@@ Line 24: / Line 24: @@
 ==== Expression inputs ====
-Data is passed into the expression through **hook** functors placed inside the container's ''<nowiki>{{ … }}</nowiki>'' block. Hooks can be added using the //Create a hook// button on the functor bar, or by dragging them in individually.
+Data is passed into the expression through **hook** functors placed inside the container's ''<nowiki>{{ … }}</nowiki>'' block — the same [[ego_script#verbose_form|verbose form]] hook mechanism used by the calculator functors:
-  * Tables and lookup tables → [[Number Table]] → available in R as ''t1'', ''t2'', …, ''t100''
+  * Tables and lookup tables → [[Number Table]] → available in the expression as ''dinamica.inputs["t1"]'', ''dinamica.inputs["t2"]'', …, ''dinamica.inputs["t100"]''
-  * Scalar values → [[Number Value]] → available as ''v1'', ''v2'', …, ''v100''
+  * Scalar values → [[Number Value]] → available as ''dinamica.inputs["v1"]'', ''dinamica.inputs["v2"]'', …, ''dinamica.inputs["v100"]''
-  * Strings → [[Number String]] → available as ''s1'', ''s2'', …, ''s100''
+  * Strings → [[Number String]] → available as ''dinamica.inputs["s1"]'', ''dinamica.inputs["s2"]'', …, ''dinamica.inputs["s100"]''
-Maps cannot be connected — this functor has no cell context. There is no shorthand notation; see [[calculate_functors|Calculate Functors — Complete Operator Documentation]] for the general hook mechanism and syntax.
+Maps cannot be connected — this functor has no cell context. There is no shorthand notation. See [[calculate_functors|Calculate Functors — Complete Operator Documentation]] for the general hook mechanism and syntax.
-=== Tables and lookup tables ===
+Every table or lookup table arriving through ''dinamica.inputs'' is represented in Python as a **list of lists**: the first inner list contains the column names (the header row) and every subsequent inner list is a row of data. The header name of each key column has an asterisk (''*'') appended to it.
-Lookup tables and tables require extra care, since each is transferred to R using a different representation.
+==== Expression outputs ====
-A **lookup table** is transferred as a list with two columns, ''Key'' and ''Value''. Each column is accessed with the ''$'' operator:
+Values are returned to Dinamica by assigning them into ''dinamica.outputs'', keyed by the desired output name. Every assigned value becomes an entry in the output ''result'' struct:
-<code rsplus>
+<code>
-# Access the third key and its corresponding value in lookup table t1
+// Scalar values are assigned directly
-patchId <- t1$Key[3];
+dinamica.outputs["patchCount"] = 42
-patchArea <- t1$Value[3];
+dinamica.outputs["totalArea"] = 1530.5
 </code>
-A **table** is transferred as a [[https://en.wikibooks.org/wiki/R_Programming/Working_with_data_frames|DataFrame]], with each column likewise accessed using ''$''. Two conventions apply to tables in either direction:
+Tables and lookup tables cannot be assigned directly — they must first be converted using the [[#utilities|utilities]] described below. If a table arriving from an input already carries ''*'' markers on its key column names, it can be assigned directly to an output without conversion.
-  * **Key columns** are marked by an asterisk (''*'') appended to their column name — this is the same convention used throughout Dinamica EGO's table representation (see [[calculate_functors#connecting_data_inputs|Connecting Data Inputs]]).
+==== Retrieving outputs ====
-  * R automatically converts string columns to **Factors** inside a ''data.frame'', but Dinamica requires plain **Character Vectors**. Always build tables with ''stringsAsFactors = FALSE'' to prevent this conversion.
-For further detail on the underlying table representation, see [[external_communication#table|External Communication]].
+''CalculatePythonExpression'' returns a single [[struct_type|Struct]] value (via its ''result'' output port) containing every entry assigned to ''dinamica.outputs''. To retrieve individual values from that struct, use the following functors from the //Integration// group:
-==== Expression outputs ====
+^ Functor ^ Retrieves ^
+| [[Extract Struct Number]] | A numeric value (''int'' or ''float'' assigned to ''dinamica.outputs'') |
+| [[Extract Struct String]] | A string value |
+| [[Extract Struct Table]] | A table produced by ''dinamica.prepareTable()'' |
+| [[Extract Struct Lookup Table]] | A lookup table produced by ''dinamica.prepareLookupTable()'' |
+| [[Extract Struct Tuple]] | A tuple value |
-Values are returned to Dinamica by calling one of the following functions from the R script. Every call requires an identifier as its first parameter — the name Dinamica uses to place the value into the output struct:
+Each functor takes two inputs: the ''Struct'' returned by ''CalculatePythonExpression'', and the name of the entry to extract as a string constant. For example, to retrieve a numeric output named ''patchCount'' and a table output named ''filteredPatches'' — the ''String $"(...)"'' wrapper around the Python code is explained in [[#writing_the_expression_in_ego_script|Writing the expression in EGO Script]] below:
-^ Function ^ Output type ^ Example ^
+<code>
-| ''outputDouble()'' | Real | ''outputDouble("myDouble", 3.14)'' |
+result := CalculatePythonExpression (String $"(
-| ''outputNumberVector()'' | Tuple | ''outputNumberVector("myTuple", c(1:10))'' |
+myTable = [["PatchId*", "Area"], [1, 12.4], [2, 8.7]]
-| ''outputString()'' | String | ''outputString("myString", "This is a string")'' |
+dinamica.outputs['patchCount'] = 42
-| ''outputLookupTable()'' | Lookup Table | ''outputLookupTable("myLUT", c(1:10), c(1:10) * 10)'' |
+dinamica.outputs['filteredPatches'] = dinamica.prepareTable(myTable, 1)
-| ''outputTable()'' | Table | ''outputTable("myTable", data.frame(State = c("Massachusetts", "Massachusetts"), City = c("Boston", "Chelsea"), Population = c(667137, 39398), stringsAsFactors = FALSE), 2)'' |
+)") {{ }};
+patchCount      := ExtractStructNumber result "patchCount";
+filteredPatches := ExtractStructTable  result "filteredPatches";
+</code>
-Values passed to these functions can be constructed inline, as in the examples above, or supplied as variables.
+==== Utilities ====
-  * Any numeric value can be passed to ''outputDouble()''.
+=== Installing packages ===
-  * Collections of numbers are valid number vectors for ''outputNumberVector()''.
-  * A [[Lookup Table Type|lookup table]] requires two number vectors of equal length — one for the keys, one for the values.
-  * ''outputTable()'' requires a table built with the [[https://www.r-tutor.com/r-introduction/data-frame|data.frame]] function, using ''stringsAsFactors = FALSE'' as described above. Its optional second parameter (default ''1'') sets how many leading columns, from the left, are key columns.
-==== Retrieving outputs ====
+Packages can be installed in two ways: calling ''dinamica.package(...)'' from within the expression, or listing package names on the //packages// input port, one per line. The two differ in timing: ''dinamica.package(...)'' runs **during** the expression, so it can be called conditionally; the //packages// port always runs **before** the expression executes. Because ''dinamica.package(...)'' runs together with the script, it may fail if a package is already loaded in an incompatible version.
-''CalculateRExpression'' returns a single [[struct_type|Struct]] value (via its ''result'' output port) containing every value passed to an ''output*()'' function. To retrieve individual values from that struct, use the corresponding functor from the //Integration// group:
+''dinamica.package(packageName, installPath=None, loadPath=None)'' installs (if needed via pip) and imports the requested module.
-^ Functor ^ Retrieves ^
+^ Parameter ^ Type ^ Default ^ Description ^
-| [[Extract Struct Number]] | A value passed to ''outputDouble()'' |
+| ''packageName'' | str | — | Identifies the package. Used as the default for both ''installPath'' and ''loadPath'' when those are omitted. |
-| [[Extract Struct Tuple]] | A value passed to ''outputNumberVector()'' |
+| ''installPath'' | str | ''packageName'' | What is passed to ''pip install''. Can be a plain name, a version-pinned requirement, a wheel filename or URL, a ''git+'' URL, or a name followed by extra pip flags. |
-| [[Extract Struct String]] | A value passed to ''outputString()'' |
+| ''loadPath'' | str | ''packageName'' | The name used to ''import'' the module in Python. |
-| [[Extract Struct Lookup Table]] | A value passed to ''outputLookupTable()'' |
-| [[Extract Struct Table]] | A value passed to ''outputTable()'' |
-Each functor takes two inputs: the ''Struct'' returned by ''CalculateRExpression'', and the name of the entry to extract as a string constant.
+Simply install and load ''numpy'':
+<code>
+dinamica.package("numpy")
+</code>
-==== Installing packages ====
+Specify a version:
+<code>
+dinamica.package("numpy", "numpy==1.19.5")
+</code>
-Packages are installed by calling ''dinamicaPackage("packageName")'' from within the expression — one call per package. Unlike [[calculate_python_expression|Calculate Python Expression]], there is no separate input port for listing packages; ''dinamicaPackage()'' is the only mechanism available.
+Install a package whose importable name differs from its pip name:
+<code>
+dinamica.package("segment_anything_py", "segment_anything_py", "segment_anything")
+</code>
-''dinamicaPackage("packageName")'' also acts as R's ''library()'' call: when the package name matches the name of the module to load, calling it both installs the package (if not already present) and loads it, in a single call.
+Use arbitrary pip parameters, such as installing from a remote wheel with a custom index:
+<code>
+dinamica.package("segment_anything_py", "https://files.pythonhosted.org/packages/43/2f/dabe75d90a7eb54a0a609a0fc5c36d1933256319beaea5d6b2f176e213a2/segment_anything_py-1.0-py3-none-any.whl --index-url https://download.pytorch.org/whl/cu118", "segment_anything")
+</code>
-<code rsplus>
+Chain several installs:
-dinamicaPackage("moments");
+<code>
+dinamica.package("cython")
+dinamica.package("numpy")
+dinamica.package("pycocotools", "git+https://github.com/philferriere/cocoapi.git#egg=pycocotools&subdirectory=PythonAPI")
 </code>
-==== Setup ====
+The //packages// port takes one package identifier per line, in any form accepted by ''pip''. It only installs — unlike ''dinamica.package()'', it does not also import the module, and it does not support specifying a separate install name and import name. Every listed package is installed before the expression runs, but the expression must still import each one itself, using its actual importable name (which may differ from the name given to ''pip''):
-There are two ways to run R scripts from ''CalculateRExpression'' — though only the local installation is available on Linux.
+The //packages// port:
+<code>
+numpy
+requests==2.31.0
+torchvision==0.19.1 --index-url https://download.pytorch.org/whl/cu121
+</code>
-=== Dinamica EGO Enhancement Plugin ===
+The expression:
+<code>
+import numpy
+import requests
+import torchvision
+</code>
-Windows only. Download and install the [[plugins_4|Dinamica EGO Enhancement Plugin]]. It contains everything needed to run R scripts inside Dinamica EGO, with no further configuration.
+=== dinamica.prepareTable() ===
-=== Local R installation ===
+Converts a list of lists into a table ready to be assigned to an output. The first inner list must be the header row. As with any table (see [[ego_script#constants|Constants]]), each column's type is inferred from its own values: a column of ''int''/''float'' values produces a Real column, a column of ''str'' values produces a String column.
-On Linux, this is the only option — Dinamica EGO always uses the R installation already present on the system. On Windows, it can be used as an alternative to the plugin. Either way, it requires:
+^ Parameter ^ Type ^ Default ^ Description ^
+| ''inputTable'' | list of lists | — | The table data. First inner list is the header row; subsequent inner lists are data rows. |
+| ''numKeys'' | int | — | Number of key columns. Key column names will have ''*'' appended in the output. |
-  * R installed on the machine, with the ''Rscript'' executable (''Rscript.exe'' on Windows) present in its ''bin'' sub-folder.
+=== dinamica.prepareLookupTable() ===
-  * The [[external_communication|Dinamica package]] for R installed and at its latest version.
-On Windows, this alternative installation is selected in the Dinamica EGO GUI by going to //Tools// → //Options// → //Integration// tab and enabling //Use alternative R installation for Calculate R Expression//.
+Converts a list of lists into a lookup table ready to be assigned to an output. The first inner list must be the header row. Lookup tables are always Real-typed on both key and value sides — there is no String option — so every value in ''lut'' must be numeric.
+^ Parameter ^ Type ^ Default ^ Description ^
+| ''lut'' | list of lists | — | The lookup table data. First inner list is the header row; subsequent inner lists are data rows. |
+=== dinamica.toTable() ===
+Converts several Python data shapes into a valid Dinamica table for output. Column types are inferred the same way as ''dinamica.prepareTable()'' — except for a flat list, which always produces a Real-typed lookup table with sequential keys, matching the Real-only rule for lookup tables.
+^ Parameter ^ Type ^ Default ^ Description ^
+| ''inputTable'' | list of lists; dict of lists; list of tuples; flat list; ''pandas.DataFrame''; ''numpy.array'' | — | The data to convert. A flat list produces a lookup table with sequential keys. A ''numpy.array'' must have its header as the first row. |
+| ''numKeys'' | int | — | Number of key columns. Key column names will have ''*'' appended in the output. Ignored for a flat list, which always produces a lookup table with sequential keys. |
 ==== Examples ====
@@ Line 113: / Line 152: @@
 The following examples use a consistent set of inputs:
-  * ''t1'' — a lookup table of land cover patches, mapping ''Key'' (patch identifier) to ''Value'' (patch area)
+  * ''t1'' — a table of land cover patches, with columns ''PatchId*'', ''Area'', and ''CategoryId''
+  * ''t2'' — a lookup table mapping category identifiers to category names
   * ''v1'' — a scalar minimum area threshold
-  * ''s1'' — a string giving the name to use for the threshold flag column
-Compute the mean and total area across all patches, and report progress to the Message Log:
+Install and import ''numpy'', then inspect all inputs passed in by Dinamica:
-<code rsplus>
+<code>
-patchMean <- mean(t1$Value);
+dinamica.package("numpy")
-patchTotal <- sum(t1$Value);
+print(dinamica.inputs)
-print(paste("Processed", length(t1$Value), "patches"));
-outputDouble("meanArea", patchMean);
-outputDouble("totalArea", patchTotal);
 </code>
-> **Note:** The ''print()'' call is visible in Dinamica EGO's Message Log, shown as a Result-level message. Log levels are ordered Unconditional, Error, Warning, Result, Info, Info2, Debug, Debug2 — messages printed from R are only shown when the Message Log level is set to Result or a more verbose level; at Unconditional, Error, or Warning they are suppressed.
+> **Note:** The ''print()'' call is visible in Dinamica EGO's Message Log, shown as a Result-level message. Log levels are ordered Unconditional, Error, Warning, Result, Info, Info2, Debug, Debug2 — messages printed from Python are only shown when the Message Log level is set to Result or a more verbose level; at Unconditional, Error, or Warning they are suppressed.
-Filter the lookup table down to patches whose area meets the threshold, and return the filtered lookup table:
+Print the rows of both connected tables to verify their contents:
-<code rsplus>
+<code>
-aboveThreshold <- t1$Value >= v1;
+for row in dinamica.inputs["t1"]:
-filteredKeys <- t1$Key[aboveThreshold];
+    print(row)
-filteredValues <- t1$Value[aboveThreshold];
-outputLookupTable("filteredPatches", filteredKeys, filteredValues);
+for row in dinamica.inputs["t2"]:
+    print(row)
 </code>
-Build and return a table with one row per patch, including a column that flags whether each patch meets the threshold. The column's name is taken from the passed string ''s1'' rather than being hard-coded:
+Count the patches whose area meets the minimum threshold and sum their total area:
-<code rsplus>
+<code>
-patchFlags <- data.frame(
+patchCount = 0
-    PatchId = t1$Key,
+totalArea = 0.0
-    Area = t1$Value,
+for row in dinamica.inputs["t1"][1:]:
-    MeetsThreshold = t1$Value >= v1,
+    if row[1] >= dinamica.inputs["v1"]:
-    stringsAsFactors = FALSE
+        patchCount += 1
-);
+        totalArea += row[1]
-names(patchFlags)[3] <- s1;
-outputTable("patchSummary", patchFlags, 1);
+dinamica.outputs["patchCount"] = patchCount
+dinamica.outputs["totalArea"] = totalArea
 </code>
-Install the ''moments'' package and use it to compute the skewness of the patch area distribution — a statistic not available in base R — then flag patches whose area is a statistical outlier:
+Return a filtered table containing only the patches above the threshold:
-<code rsplus>
+<code>
-dinamicaPackage("moments");
+filtered = [dinamica.inputs["t1"][0]]
+for row in dinamica.inputs["t1"][1:]:
+    if row[1] >= dinamica.inputs["v1"]:
+        filtered.append(row)
-patchMean <- mean(t1$Value);
+dinamica.outputs["filteredPatches"] = dinamica.prepareTable(filtered, 1)
-patchStdDev <- sd(t1$Value);
+</code>
-patchSkewness <- skewness(t1$Value);
-isOutlier <- abs(t1$Value - patchMean) > 2 * patchStdDev;
+Compute the total area per category and return it as a lookup table:
-outlierPatches <- data.frame(
-    PatchId = t1$Key[isOutlier],
-    Area = t1$Value[isOutlier],
-    stringsAsFactors = FALSE
-);
-outputDouble("meanArea", patchMean);
+<code>
-outputDouble("stdDevArea", patchStdDev);
+areaSums = {}
-outputDouble("skewnessArea", patchSkewness);
+for row in dinamica.inputs["t1"][1:]:
-outputTable("outlierPatches", outlierPatches, 1);
+    categoryId = row[2]
+    areaSums[categoryId] = areaSums.get(categoryId, 0.0) + row[1]
+lut = [["CategoryId*", "TotalArea"]]
+for catId, total in areaSums.items():
+    lut.append([catId, total])
+dinamica.outputs["areaByCategory"] = dinamica.prepareLookupTable(lut)
 </code>
 ==== Writing the expression in EGO Script ====
-Like [[calculate_python_expression|Calculate Python Expression]], the ''Expression'' input is type [[ego_script#constants|Code]], which is represented in the underlying script using base64 encoding — impractical to write or edit directly as a text constant. Instead, connect a ''String'' carrier functor containing the R expression text to the ''Expression'' port; its output is accepted wherever a ''Code'' value is expected. Since only this one output is needed, the carrier can be [[ego_script#inline_syntax|inlined]] directly into the call.
+When writing a script by hand, the ''expression'' input cannot be filled in directly as a text constant: the [[ego_script#constants|Code]] type is represented in the underlying script using base64 encoding, which is impractical to write or edit directly. Instead, connect a ''String'' carrier functor containing the expression text to the ''expression'' port — its output is accepted wherever a ''Code'' value is expected. Since only this one output is needed, the carrier can be [[ego_script#inline_syntax|inlined]] directly into the call.
+This limitation is specific to hand-written EGO Script. In the Dinamica EGO GUI, the ''expression'' port has a dedicated code editor that edits the ''Code'' value directly — the ''String'' carrier workaround is only necessary when writing or editing the ''.ego'' file as text.
-This limitation is specific to hand-written EGO Script. In the Dinamica EGO GUI, the ''Expression'' port has a dedicated code editor that edits the ''Code'' value directly — the ''String'' carrier workaround is only necessary when writing or editing the ''.ego'' file as text.
+The following counts the patches in a land cover areas table whose area meets a minimum threshold:
 <code>
-result := CalculateRExpression (String $"(
+result := CalculatePythonExpression (String $"(
-aboveThreshold <- t1$Value >= v1;
+total = 0
-outputLookupTable("filteredPatches", t1$Key[aboveThreshold], t1$Value[aboveThreshold]);
+for row in dinamica.inputs['t1'][1:]:
-)") .no {{
+    if row[1] >= dinamica.inputs['v1']:
+        total += 1
+dinamica.outputs['patchCount'] = total
+)") {{
     NumberTable landCoverAreas 1;
     NumberValue minimumArea    1;
 }};
-filteredPatches := ExtractStructLookupTable result "filteredPatches";
+patchCount := ExtractStructNumber result "patchCount";
 </code>
-===== Group =====
+''CalculatePythonExpression'' returns a ''Struct'' containing all values assigned to ''dinamica.outputs''. The ''ExtractStructNumber'' functor then pulls the ''patchCount'' entry out of that struct by name. See [[#retrieving_outputs|Retrieving outputs]] for the full list of extraction functors.
-[[Functor List#Integration|Integration]]
+A more involved example: installing ''numpy'' to compute area statistics for a land cover patches table and flag outlier patches, then retrieving both the scalar statistics and the resulting table:
-===== Internal Name =====
+<code>
+result := CalculatePythonExpression (String $"(
+dinamica.package('numpy')
-CalculateRExpression
+areas = numpy.array([row[1] for row in dinamica.inputs['t1'][1:]])
-===== See Also =====
+meanArea = float(numpy.mean(areas))
+stdArea = float(numpy.std(areas))
-  * [[calculate_python_expression|Calculate Python Expression]]
+isOutlier = numpy.abs(areas - meanArea) > 2 * stdArea
-  * [[External Communication]]
-  * [[ego_script#container_functors|EGO Script — Container Functors]]
-  * [[ego_script#verbose_form|EGO Script — Verbose Form (Hooks)]]
-  * [[calculate_functors|Calculate Functors — Complete Operator Documentation]]
+dinamica.outputs['meanArea'] = meanArea
+dinamica.outputs['stdArea'] = stdArea
+header = dinamica.inputs['t1'][0]
+outlierRows = [row for row, flagged in zip(dinamica.inputs['t1'][1:], isOutlier) if flagged]
+outlierTable = [header] + outlierRows
+dinamica.outputs['outlierPatches'] = dinamica.prepareTable(outlierTable, 1)
+)") {{
+    NumberTable landCoverPatches 1;
+}};
+meanArea       := ExtractStructNumber result "meanArea";
+stdArea        := ExtractStructNumber result "stdArea";
+outlierPatches := ExtractStructTable  result "outlierPatches";
+</code>
+Here ''landCoverPatches'' is bound to ''t1'' through a single [[Number Table]] hook.
+===== Group =====
+[[Functor List#Integration|Integration]]
+===== Internal Name =====
+CalculatePythonExpression