feat: publishing infernet-container-starter v0.2.0

This commit is contained in:
ritual-all
2024-03-29 10:50:13 -04:00
parent 41aaa152e6
commit 4545223364
155 changed files with 6086 additions and 257 deletions

View File

@ -0,0 +1,25 @@
FROM python:3.11-slim as builder
WORKDIR /app
ENV PYTHONUNBUFFERED 1
ENV PYTHONDONTWRITEBYTECODE 1
ENV PIP_NO_CACHE_DIR 1
ENV RUNTIME docker
ENV PYTHONPATH src
RUN apt-get update
RUN apt-get install -y git curl
# install uv
ADD --chmod=755 https://astral.sh/uv/install.sh /install.sh
RUN /install.sh && rm /install.sh
COPY src/requirements.txt .
RUN /root/.cargo/bin/uv pip install --system --no-cache -r requirements.txt
COPY src src
ENTRYPOINT ["hypercorn", "app:create_app()"]
CMD ["-b", "0.0.0.0:3000"]

View File

@ -0,0 +1,17 @@
DOCKER_ORG := ritualnetwork
EXAMPLE_NAME := onnx-iris
TAG := $(DOCKER_ORG)/example-$(EXAMPLE_NAME)-infernet:latest
.phony: build run build-multiplatform
build:
@docker build -t $(TAG) .
run:
docker run -p 3000:3000 $(TAG)
# You may need to set up a docker builder, to do so run:
# docker buildx create --name mybuilder --bootstrap --use
# refer to https://docs.docker.com/build/building/multi-platform/#building-multi-platform-images for more info
build-multiplatform:
docker buildx build --platform linux/amd64,linux/arm64 -t $(TAG) --push .

View File

@ -0,0 +1,96 @@
# Iris Classification via ONNX Runtime
This example uses a pre-trained model to classify iris flowers. The code for the model
is located at
our [simple-ml-models](https://github.com/ritual-net/simple-ml-models/tree/main/iris_classification)
repository.
## Overview
We're making use of
the [ONNXInferenceWorkflow](https://github.com/ritual-net/infernet-ml-internal/blob/main/src/ml/workflows/inference/onnx_inference_workflow.py)
class to run the model. This is one of many workflows that we currently support in our
[infernet-ml](https://github.com/ritual-net/infernet-ml-internal). Consult the library's
documentation for more info on workflows that
are supported.
## Building & Running the Container in Isolation
Note that this container is meant to be started by the infernet-node. For development &
Testing purposes, you can run the container in isolation using the following commands.
### Building the Container
Simply run the following command to build the container.
```bash
make build
```
Consult the [Makefile](./Makefile) for the build command.
### Running the Container
To run the container, you can use the following command:
```bash
make run
```
## Testing the Container
Run the following command to run an inference:
```bash
curl -X POST http://127.0.0.1:3000/service_output \
-H "Content-Type: application/json" \
-d '{"source":1, "data": {"input": [[1.0380048, 0.5586108, 1.1037828, 1.712096]]}}'
```
#### Note Regarding the Input
The inputs provided above correspond to an iris flower with the following
characteristics. Refer to the
1. Sepal Length: `5.5cm`
2. Sepal Width: `2.4cm`
3. Petal Length: `3.8cm`
4. Petal Width: `1.1cm`
Putting this input into a vector and scaling it, we get the following scaled input:
```python
[1.0380048, 0.5586108, 1.1037828, 1.712096]
```
Refer
to [this function in the model's repository](https://github.com/ritual-net/simple-ml-models/blob/03ebc6fb15d33efe20b7782505b1a65ce3975222/iris_classification/iris_inference_pytorch.py#L13)
for more information on how the input is scaled.
For more context on the Iris dataset, refer to
the [UCI Machine Learning Repository](https://archive.ics.uci.edu/ml/datasets/iris).
### Output
By running the above command, you should get a response similar to the following:
```json
[
[
[
0.0010151526657864451,
0.014391022734344006,
0.9845937490463257
]
]
]
```
The response corresponds to the model's prediction for each of the classes:
```python
['setosa', 'versicolor', 'virginica']
```
In this case, the model predicts that the input corresponds to the class `virginica`with
a probability of `0.9845937490463257`(~98.5%).

View File

@ -0,0 +1,50 @@
{
"log_path": "infernet_node.log",
"server": {
"port": 4000
},
"chain": {
"enabled": true,
"trail_head_blocks": 0,
"rpc_url": "http://host.docker.internal:8545",
"coordinator_address": "0x5FbDB2315678afecb367f032d93F642f64180aa3",
"wallet": {
"max_gas_limit": 4000000,
"private_key": "0x59c6995e998f97a5a0044966f0945389dc9e86dae88c7a8412f4603b6b78690d"
}
},
"startup_wait": 1.0,
"docker": {
"username": "your-username",
"password": ""
},
"redis": {
"host": "redis",
"port": 6379
},
"forward_stats": true,
"containers": [
{
"id": "onnx-iris",
"image": "ritualnetwork/example-onnx-iris-infernet:latest",
"external": true,
"port": "3000",
"allowed_delegate_addresses": [],
"allowed_addresses": [],
"allowed_ips": [],
"command": "--bind=0.0.0.0:3000 --workers=2",
"env": {}
},
{
"id": "anvil-node",
"image": "ritualnetwork/infernet-anvil:0.0.0",
"external": true,
"port": "8545",
"allowed_delegate_addresses": [],
"allowed_addresses": [],
"allowed_ips": [],
"command": "",
"env": {}
}
]
}

View File

@ -0,0 +1,52 @@
import asyncio
import aiohttp
from eth_abi import encode, decode # type: ignore
async def ping(session: aiohttp.ClientSession) -> None:
async with session.get("http://127.0.0.1:3000/") as response:
print(await response.text())
async def post_directly_web2(session: aiohttp.ClientSession) -> None:
async with session.post(
"http://127.0.0.1:3000/service_output",
json={
"source": 1,
"data": {"input": [[1.0380048, 0.5586108, 1.1037828, 1.712096]]},
},
) as response:
print(await response.json())
async def post_directly_web3(session: aiohttp.ClientSession) -> None:
async with session.post(
"http://127.0.0.1:3000/service_output",
json={
"source": 0,
"data": encode(
["uint256[]"], [[1_038_004, 558_610, 1_103_782, 1_712_096]]
).hex(),
},
) as response:
print(await response.text())
result = await response.json()
output = result["raw_output"]
result = decode(["uint256[]"], bytes.fromhex(output))[0]
print(f"result: {result}")
# async maine
async def main(session: aiohttp.ClientSession) -> None:
await post_directly_web3(session)
if __name__ == "__main__":
# run main async
async def provide_session() -> None:
async with aiohttp.ClientSession() as session:
await main(session)
asyncio.run(provide_session())

View File

@ -0,0 +1,107 @@
import logging
from typing import Any, cast, List
import numpy as np
from eth_abi import decode, encode # type: ignore
from infernet_ml.utils.model_loader import ModelSource
from infernet_ml.utils.service_models import InfernetInput, InfernetInputSource
from infernet_ml.workflows.inference.onnx_inference_workflow import (
ONNXInferenceWorkflow,
)
from quart import Quart, request
from quart.json.provider import DefaultJSONProvider
log = logging.getLogger(__name__)
class NumpyJsonEncodingProvider(DefaultJSONProvider):
@staticmethod
def default(obj: Any) -> Any:
if isinstance(obj, np.ndarray):
# Convert NumPy arrays to list
return obj.tolist()
# fallback to default JSON encoding
return DefaultJSONProvider.default(obj)
def create_app() -> Quart:
Quart.json_provider_class = NumpyJsonEncodingProvider
app = Quart(__name__)
# we are downloading the model from the hub.
# model repo is located at: https://huggingface.co/Ritual-Net/iris-dataset
model_source = ModelSource.HUGGINGFACE_HUB
model_args = {"repo_id": "Ritual-Net/iris-dataset", "filename": "iris.onnx"}
workflow = ONNXInferenceWorkflow(model_source=model_source, model_args=model_args)
workflow.setup()
@app.route("/")
def index() -> str:
"""
Utility endpoint to check if the service is running.
"""
return "ONNX Iris Classifier Example Program"
@app.route("/service_output", methods=["POST"])
async def inference() -> dict[str, Any]:
req_data = await request.get_json()
"""
InfernetInput has the format:
source: (0 on-chain, 1 off-chain)
data: dict[str, Any]
"""
infernet_input: InfernetInput = InfernetInput(**req_data)
if infernet_input.source == InfernetInputSource.OFFCHAIN:
web2_input = cast(dict[str, Any], infernet_input.data)
values = cast(List[List[float]], web2_input["input"])
else:
# On-chain requests are sent as a generalized hex-string which we will
# decode to the appropriate format.
web3_input: List[int] = decode(
["uint256[]"], bytes.fromhex(cast(str, infernet_input.data))
)[0]
values = [[float(v) / 1e6 for v in web3_input]]
"""
The input to the onnx inference workflow needs to conform to ONNX runtime's
input_feed format. For more information refer to:
https://docs.ritual.net/ml-workflows/inference-workflows/onnx_inference_workflow
"""
result: dict[str, Any] = workflow.inference({"input": values})
if infernet_input.source == InfernetInputSource.OFFCHAIN:
"""
In case of an off-chain request, the result is returned as is.
"""
return result
else:
"""
In case of an on-chain request, the result is returned in the format:
{
"raw_input": str,
"processed_input": str,
"raw_output": str,
"processed_output": str,
"proof": str,
}
refer to: https://docs.ritual.net/infernet/node/containers for more info.
"""
predictions = cast(List[List[List[float]]], result)
predictions_normalized = [int(p * 1e6) for p in predictions[0][0]]
return {
"raw_input": "",
"processed_input": "",
"raw_output": encode(["uint256[]"], [predictions_normalized]).hex(),
"processed_output": "",
"proof": "",
}
return app
if __name__ == "__main__":
"""
Utility to run the app locally. For development purposes only.
"""
create_app().run(port=3000)

View File

@ -0,0 +1,7 @@
quart==0.19.4
infernet_ml==0.1.0
PyArweave @ git+https://github.com/ritual-net/pyarweave.git
web3==6.15.0
onnx==1.15.0
onnxruntime==1.16.3
torch==2.1.2

14
projects/onnx-iris/contracts/.gitignore vendored Normal file
View File

@ -0,0 +1,14 @@
# Compiler files
cache/
out/
# Ignores development broadcast logs
!/broadcast
/broadcast/*/31337/
/broadcast/**/dry-run/
# Docs
docs/
# Dotenv file
.env

View File

@ -0,0 +1,14 @@
# phony targets are targets that don't actually create a file
.phony: deploy
# anvil's third default address
sender := 0x5de4111afa1a4b94908f83103eb1f1706367c2e68ca870fc3fb9a804cdab365a
RPC_URL := http://localhost:8545
# deploying the contract
deploy:
@PRIVATE_KEY=$(sender) forge script script/Deploy.s.sol:Deploy --broadcast --rpc-url $(RPC_URL)
# calling sayGM()
call-contract:
@PRIVATE_KEY=$(sender) forge script script/CallContract.s.sol:CallContract --broadcast --rpc-url $(RPC_URL)

View File

@ -0,0 +1,41 @@
# `ONNX` Consumer Contracts
This is a [foundry](https://book.getfoundry.sh/) project that implements a simple Consumer
contract, [`IrisClassifier`](./src/IrisClassifier.sol).
This readme explains how to compile and deploy the contract to the Infernet Anvil Testnet network.
> [!IMPORTANT]
> Ensure that you are running the following scripts with the Infernet Anvil Testnet network.
> The [tutorial](../../hello-world/README.mdADME.md) at the root of this repository explains how to
> bring up an infernet node.
### Installing the libraries
```bash
forge install
```
### Compiling the contracts
```bash
forge compile
```
### Deploying the contracts
The deploy script at `script/Deploy.s.sol` deploys the `IrisClassifier` contract to the Infernet Anvil Testnet network.
We have the [following make target](./Makefile#L9) to deploy the contract. Refer to the Makefile
for more understanding around the deploy scripts.
```bash
make deploy
```
### Requesting a job
We also have a script called `CallContract.s.sol` that requests a job to the `IrisClassifier` contract.
Refer to the [script](./script/CallContract.s.sol) for more details. Similar to deployment,
you can run that script using the following convenience make target.
```bash
make call-contract
```
Refer to the [Makefile](./Makefile#L14) for more details.

View File

@ -0,0 +1,7 @@
[profile.default]
src = "src"
out = "out"
libs = ["lib"]
via_ir = true
# See more config options https://github.com/foundry-rs/foundry/blob/master/crates/config/README.md#all-options

View File

@ -0,0 +1,2 @@
forge-std/=lib/forge-std/src
infernet-sdk/=lib/infernet-sdk/src

View File

@ -0,0 +1,19 @@
// SPDX-License-Identifier: BSD-3-Clause-Clear
pragma solidity ^0.8.0;
import {Script, console2} from "forge-std/Script.sol";
import {IrisClassifier} from "../src/IrisClassifier.sol";
contract CallContract is Script {
function run() public {
// Setup wallet
uint256 deployerPrivateKey = vm.envUint("PRIVATE_KEY");
vm.startBroadcast(deployerPrivateKey);
IrisClassifier irisClassifier = IrisClassifier(0x663F3ad617193148711d28f5334eE4Ed07016602);
irisClassifier.classifyIris();
vm.stopBroadcast();
}
}

View File

@ -0,0 +1,26 @@
// SPDX-License-Identifier: BSD-3-Clause-Clear
pragma solidity ^0.8.13;
import {Script, console2} from "forge-std/Script.sol";
import {IrisClassifier} from "../src/IrisClassifier.sol";
contract Deploy is Script {
function run() public {
// Setup wallet
uint256 deployerPrivateKey = vm.envUint("PRIVATE_KEY");
vm.startBroadcast(deployerPrivateKey);
// Log address
address deployerAddress = vm.addr(deployerPrivateKey);
console2.log("Loaded deployer: ", deployerAddress);
address coordinator = 0x5FbDB2315678afecb367f032d93F642f64180aa3;
// Create consumer
IrisClassifier classifier = new IrisClassifier(coordinator);
console2.log("Deployed IrisClassifier: ", address(classifier));
// Execute
vm.stopBroadcast();
vm.broadcast();
}
}

View File

@ -0,0 +1,67 @@
// SPDX-License-Identifier: BSD-3-Clause-Clear
pragma solidity ^0.8.13;
import {console2} from "forge-std/console2.sol";
import {CallbackConsumer} from "infernet-sdk/consumer/Callback.sol";
contract IrisClassifier is CallbackConsumer {
string private EXTREMELY_COOL_BANNER = "\n\n"
"_____ _____ _______ _ _ _\n"
"| __ \\|_ _|__ __| | | | /\\ | |\n"
"| |__) | | | | | | | | | / \\ | |\n"
"| _ / | | | | | | | |/ /\\ \\ | |\n"
"| | \\ \\ _| |_ | | | |__| / ____ \\| |____\n"
"|_| \\_\\_____| |_| \\____/_/ \\_\\______|\n\n";
constructor(address coordinator) CallbackConsumer(coordinator) {}
function classifyIris() public {
/// @dev Iris data is in the following format:
/// @dev [sepal_length, sepal_width, petal_length, petal_width]
/// @dev the following vector corresponds to the following properties:
/// "sepal_length": 5.5cm
/// "sepal_width": 2.4cm
/// "petal_length": 3.8cm
/// "petal_width": 1.1cm
/// @dev The data is normalized & scaled.
/// refer to [this function in the model's repository](https://github.com/ritual-net/simple-ml-models/blob/03ebc6fb15d33efe20b7782505b1a65ce3975222/iris_classification/iris_inference_pytorch.py#L13)
/// for more info on normalization.
/// @dev The data is adjusted by 6 decimals
uint256[] memory iris_data = new uint256[](4);
iris_data[0] = 1_038_004;
iris_data[1] = 558_610;
iris_data[2] = 1_103_782;
iris_data[3] = 1_712_096;
_requestCompute(
"onnx-iris",
abi.encode(iris_data),
20 gwei,
1_000_000,
1
);
}
function _receiveCompute(
uint32 subscriptionId,
uint32 interval,
uint16 redundancy,
address node,
bytes calldata input,
bytes calldata output,
bytes calldata proof
) internal override {
console2.log(EXTREMELY_COOL_BANNER);
(bytes memory raw_output, bytes memory processed_output) = abi.decode(output, (bytes, bytes));
(uint256[] memory classes) = abi.decode(raw_output, (uint256[]));
uint256 setosa = classes[0];
uint256 versicolor = classes[1];
uint256 virginica = classes[2];
console2.log("predictions: (adjusted by 6 decimals, 1_000_000 = 100%, 1_000 = 0.1%)");
console2.log("Setosa: ", setosa);
console2.log("Versicolor: ", versicolor);
console2.log("Virginica: ", virginica);
}
}

View File

@ -0,0 +1,271 @@
# Running an ONNX Model on Infernet
Welcome to this comprehensive guide where we'll explore how to run an ONNX model on Infernet, using our [infernet-container-starter](https://github.com/ritual-net/infernet-container-starter/)
examples repository. This tutorial is designed to give you and end-to-end understanding of how you can run your own
custom pre-trained models, and interact with them on-chain and off-chain.
**Model:** This example uses a pre-trained model to classify iris flowers. The code for the model
is located at our [`simple-ml-models`](https://github.com/ritual-net/simple-ml-models/tree/main/iris_classification) repository.
## Pre-requisites
For this tutorial you'll need to have the following installed.
1. [Docker](https://docs.docker.com/engine/install/)
2. [Foundry](https://book.getfoundry.sh/getting-started/installation)
### Ensure `docker` & `foundry` exist
To check for `docker`, run the following command in your terminal:
```bash copy
docker --version
# Docker version 25.0.2, build 29cf629 (example output)
```
You'll also need to ensure that docker-compose exists in your terminal:
```bash copy
which docker-compose
# /usr/local/bin/docker-compose (example output)
```
To check for `foundry`, run the following command in your terminal:
```bash copy
forge --version
# forge 0.2.0 (551bcb5 2024-02-28T07:40:42.782478000Z) (example output)
```
### Clone the starter repository
If you haven't already, clone the infernet-container-starter repository. All of the code for this tutorial is located
under the `projects/onnx-iris` directory.
```bash copy
# Clone locally
git clone --recurse-submodules https://github.com/ritual-net/infernet-container-starter
# Navigate to the repository
cd infernet-container-starter
```
## Making Inference Requests via Node API (a la Web2 request)
### Build the `onnx-iris` container
From the top-level directory of this repository, simply run the following command to build the `onnx-iris` container:
```bash copy
make build-container project=onnx-iris
```
After the container is built, you can deploy an infernet-node that utilizes that
container by running the following command:
```bash copy
make deploy-container project=onnx-iris
```
Now, you can make inference requests to the infernet-node. In a new tab, run:
```bash copy
curl -X POST "http://127.0.0.1:4000/api/jobs" \
-H "Content-Type: application/json" \
-d '{"containers":["onnx-iris"], "data": {"input": [[1.0380048, 0.5586108, 1.1037828, 1.712096]]}}'
```
You should get an output similar to the following:
```json
{
"id": "074b9e98-f1f6-463c-b185-651878f3b4f6"
}
```
Now, you can check the status of the job by running (Make sure job id matches the one
you got from the previous request):
```bash
curl -X GET "http://127.0.0.1:4000/api/jobs?id=074b9e98-f1f6-463c-b185-651878f3b4f6"
```
Should return:
```json
[
{
"id": "074b9e98-f1f6-463c-b185-651878f3b4f6",
"result": {
"container": "onnx-iris",
"output": [
[
[
0.0010151526657864451,
0.014391022734344006,
0.9845937490463257
]
]
]
},
"status": "success"
}
]
```
The `output` corresponds to the model's prediction for each of the classes:
```python
['setosa', 'versicolor', 'virginica']
```
In this case, the model predicts that the input corresponds to the class `virginica`with
a probability of `0.9845937490463257`(~98.5%).
#### Note Regarding the Input
The inputs provided above correspond to an iris flower with the following
characteristics. Refer to the
1. Sepal Length: `5.5cm`
2. Sepal Width: `2.4cm`
3. Petal Length: `3.8cm`
4. Petal Width: `1.1cm`
Putting this input into a vector and scaling it, we get the following scaled input:
```python
[1.0380048, 0.5586108, 1.1037828, 1.712096]
```
Refer
to [this function in the model's repository](https://github.com/ritual-net/simple-ml-models/blob/03ebc6fb15d33efe20b7782505b1a65ce3975222/iris_classification/iris_inference_pytorch.py#L13)
for more information on how the input is scaled.
For more context on the Iris dataset, refer to
the [UCI Machine Learning Repository](https://archive.ics.uci.edu/ml/datasets/iris).
## Making Inference Requests via Contracts (a la Web3 request)
The [contracts](contracts) directory contains a simple forge
project that can be used to interact with the Infernet Node.
Here, we have a very simple
contract, [IrisClassifier](contracts/src/IrisClassifier.sol),
that requests a compute job from the Infernet Node and then retrieves the result.
We are going to make the same request as above, but this time using a smart contract.
Since floats are not supported in Solidity, we convert all floats to `uint256` by
multiplying the input vector entries by `1e6`:
```Solidity
uint256[] memory iris_data = new uint256[](4);
iris_data[0] = 1_038_004;
iris_data[1] = 558_610;
iris_data[2] = 1_103_782;
iris_data[3] = 1_712_096;
```
We have multiplied the input by 1e6 to have enough accuracy. This can be seen
[here](contracts/src/IrisClassifier.sol#19) in the contract's
code.
### Monitoring the EVM Logs
The infernet node configuration for this project includes
an [infernet anvil node](projects/hello-world/README.mdllo-world/README.md#77) with pre-deployed contracts. You can view the
logs of the anvil node to see what's going on. In a new terminal, run:
```bash
docker logs -f anvil-node
```
As you deploy the contract and make requests, you should see logs indicating the
requests and responses.
### Deploying the Contract
Simply run the following command to deploy the contract:
```bash
project=onnx-iris make deploy-contracts
```
In your anvil logs you should see the following:
```bash
eth_getTransactionReceipt
Transaction: 0xeed605eacdace39a48635f6d14215b386523766f80a113b4484f542d862889a4
Contract created: 0x663f3ad617193148711d28f5334ee4ed07016602
Gas used: 714269
Block Number: 1
Block Hash: 0x4e6333f91e86a0a0be357b63fba9eb5f5ba287805ac35aaa7698fd05445730f5
Block Time: "Mon, 19 Feb 2024 20:31:17 +0000"
eth_blockNumber
```
beautiful, we can see that a new contract has been created
at `0x663f3ad617193148711d28f5334ee4ed07016602`. That's the address of
the `IrisClassifier` contract. We are now going to call this contract. To do so,
we are using
the [CallContract.s.sol](contracts/script/CallContract.s.sol)
script. Note that the address of the
contract [is hardcoded in the script](contracts/script/CallContract.s.sol#L13),
and should match the address we see above. Since this is a test environment and we're
using a test deployer address, this address is quite deterministic and shouldn't change.
Otherwise, change the address in the script to match the address of the contract you
just deployed.
### Calling the Contract
To call the contract, run the following command:
```bash
project=onnx-iris make call-contract
```
In the anvil logs, you should see the following:
```bash
eth_sendRawTransaction
_____ _____ _______ _ _ _
| __ \|_ _|__ __| | | | /\ | |
| |__) | | | | | | | | | / \ | |
| _ / | | | | | | | |/ /\ \ | |
| | \ \ _| |_ | | | |__| / ____ \| |____
|_| \_\_____| |_| \____/_/ \_\______|
predictions: (adjusted by 6 decimals, 1_000_000 = 100%, 1_000 = 0.1%)
Setosa: 1015
Versicolor: 14391
Virginica: 984593
Transaction: 0x77c7ff26ed20ffb1a32baf467a3cead6ed81fe5ae7d2e419491ca92b4ac826f0
Gas used: 111091
Block Number: 3
Block Hash: 0x78f98f4d54ebdca2a8aa46c3b9b7e7ae36348373dbeb83c91a4600dd6aba2c55
Block Time: "Mon, 19 Feb 2024 20:33:00 +0000"
eth_blockNumber
eth_newFilter
eth_getFilterLogs
```
Beautiful! We can see that the same result has been posted to the contract.
### Next Steps
From here, you can bring your own pre-trained ONNX model, and with minimal changes, you can make it both work with an
infernet-node as well as a smart contract.
### More Information
1. Check out our [other examples](../../readme.md) if you haven't already
2. [Infernet Callback Consumer Tutorial](https://docs.ritual.net/infernet/sdk/consumers/Callback)
3. [Infernet Nodes Docoumentation](https://docs.ritual.net/infernet/node/introduction)
4. [Infernet-Compatible Containers](https://docs.ritual.net/infernet/node/containers)