Describe the bug I am attempting to run the example of FL as given in manual and getting this error on the aggregator.

Gramine is starting. Parsing TOML manifest file, this may take some time...
Detected a huge manifest, preallocating 64MB of internal memory.
-----------------------------------------------------------------------------------------------------------------------
Gramine detected the following insecure configurations:

  - loader.insecure__use_cmdline_argv = true   (forwarding command-line args from untrusted host to the app)
  - sgx.allowed_files = [ ... ]                (some files are passed through from untrusted host without verification)

Gramine will continue application execution, but this configuration must not be used in production!
-----------------------------------------------------------------------------------------------------------------------

error: PAL failed at ../../Pal/src/db_main.c:pal_main:513 (exitcode = 4, reason=No 'loader.entrypoint' is specified in the manifest)

Also, step 7 in the manual is presented in a bit vague manner for a first-time user. I used the setup given here as a workspace and template. But using this gave above error when I am trying to start federation on aggregator machine.

Describe the bug Attemptin to run the openfl-gramine example. I am getting a error at point 4 "Build the Experiment Docker image"

To Reproduce following the steps mentioned in the openfl-gramine/MANUAL. Also attaching the terminal logs for more reference. output.zip

Can you please help?

In this PR, the problem of image anomaly detection and segmentation is addressed. Anomaly detection involves making a binary decision as to whether an input image contains an anomaly, and anomaly segmentation aims to locate the anomaly on the pixel level. The deep learning variant of support vector data description (SVDD) algorithm is extended to the patch-based method using self-supervised learning. This extension enables anomaly segmentation and improves detection performance. Industrial dataset is used for this tutorial - MVTec AD dataset.

Describe the bug Error while running the kvasir interactive API example

To Reproduce Steps to reproduce the behavior:

1. Use openfl 1.2 and Pytorch_kvasir interactive API tutorial
2. Start Director
3. Start Envoy
4. After dataset is download, there is an error

Expected behavior Federation should start

Screenshots

Note:

using Ubuntu 20.04. The ports 50500 are open and the same was verified through netcat command

In some machines, when I run a federation (even if it both the collaborators and the aggregator are on the same machine) they fail to establish a connection. I have started facing this problem specifically in the interactive API, I have been unsuccessful in running a federation as collaborator is unable to connect to the port where aggregator has started the gRPC server.

Reproducing the error: I did not do anything differently than what is already mentioned in the tutorial. I created a fresh conda environment, installed the openfl library and finally tried to replicate the experiment.

We tried to debug this error and in the process we found out that the gRPC server from the aggregator runs exclusively on IPv6. Whereas, collaborator tries to listen to IPv4. We even tried to hardcode the server and the port numbers but we were unable to make it work. We suspect that the error has something to do with the way the gRPC server are started in https://github.com/intel/openfl/blob/c3c0886aefeb09f426fc3726be0f65de2b344e22/openfl/transport/grpc/server.py and https://github.com/intel/openfl/blob/c3c0886aefeb09f426fc3726be0f65de2b344e22/openfl/transport/grpc/client.py

I think this error can pose a potentially big problem in the future. Therefore, please look into it.

Thanks

ISV product ID and ISV SVN fields are defined and signed in SIGSTRUCT. In Gramine, they are defined in the manifest and then used on enclave signing in enclave build flow. Add the option to configure them in OpenFL manifest with new flags to the makefile: SGX_ISVPRODID and SGX_ISVSVN. Their default values are 0, to maintain backwards compatibility.

Currently ISV SVN is configurable in build time, but this is not necessarily the desired behavior. Required behavior is open for discussion and should be updated once a decision is made.

Related issue: #501

During training, after several epochs (55 in this example, but this number is random) this errors happens.

ERROR    Collaborator failed with error: <_InactiveRpcError of RPC that terminated with:                                                  envoy.py:75
                            status = StatusCode.UNKNOWN
                            details = "Exception calling application: Aggregator does not have an aggregated tensor for
                    TensorKey(tensor_name='block_1.1.running_mean', origin='aggregator_plan.yaml_4ebcd576', round_number=55, report=False,
                    tags=('model',))"
                            debug_error_string = "{"created":"@1646124781.553286945","description":"Error received from peer
                    ipv4:172.20.8.151:59526","file":"src/core/lib/surface/call.cc","file_line":1062,"grpc_message":"Exception calling application:
                    Aggregator does not have an aggregated tensor for TensorKey(tensor_name='block_1.1.running_mean',
                    origin='aggregator_plan.yaml_4ebcd576', round_number=55, report=False, tags=('model',))","grpc_status":2}"
                    >:
                    Traceback (most recent call last):
                      File "/home/lmancuso/bruno/lib/python3.7/site-packages/openfl/component/envoy/envoy.py", line 73, in run
                        self._run_collaborator()
                      File "/home/lmancuso/bruno/lib/python3.7/site-packages/openfl/component/envoy/envoy.py", line 141, in _run_collaborator
                        col.run()
                      File "/home/lmancuso/bruno/lib/python3.7/site-packages/openfl/component/collaborator/collaborator.py", line 145, in run
                        self.do_task(task, round_number)
                      File "/home/lmancuso/bruno/lib/python3.7/site-packages/openfl/component/collaborator/collaborator.py", line 215, in do_task
                        required_tensorkeys
                      File "/home/lmancuso/bruno/lib/python3.7/site-packages/openfl/component/collaborator/collaborator.py", line 257, in
                    get_numpy_dict_for_tensorkeys
                        return {k.tensor_name: self.get_data_for_tensorkey(k) for k in tensor_keys}
                      File "/home/lmancuso/bruno/lib/python3.7/site-packages/openfl/component/collaborator/collaborator.py", line 257, in <dictcomp>
                        return {k.tensor_name: self.get_data_for_tensorkey(k) for k in tensor_keys}
                      File "/home/lmancuso/bruno/lib/python3.7/site-packages/openfl/component/collaborator/collaborator.py", line 331, in
                    get_data_for_tensorkey
                        require_lossless=True
                      File "/home/lmancuso/bruno/lib/python3.7/site-packages/openfl/component/collaborator/collaborator.py", line 367, in
                    get_aggregated_tensor_from_aggregator
                        self.collaborator_name, tensor_name, round_number, report, tags, require_lossless)
                      File "/home/lmancuso/bruno/lib/python3.7/site-packages/openfl/transport/grpc/client.py", line 84, in wrapper
                        response = func(self, *args, **kwargs)
                      File "/home/lmancuso/bruno/lib/python3.7/site-packages/openfl/transport/grpc/client.py", line 285, in get_aggregated_tensor
                        response = self.stub.GetAggregatedTensor(request)
                      File "/home/lmancuso/bruno/lib/python3.7/site-packages/grpc/_interceptor.py", line 221, in __call__
                        compression=compression)
                      File "/home/lmancuso/bruno/lib/python3.7/site-packages/grpc/_interceptor.py", line 257, in _with_call
                        return call.result(), call
                      File "/home/lmancuso/bruno/lib/python3.7/site-packages/grpc/_channel.py", line 333, in result
                        raise self
                      File "/home/lmancuso/bruno/lib/python3.7/site-packages/grpc/_interceptor.py", line 247, in continuation
                        compression=new_compression)
                      File "/home/lmancuso/bruno/lib/python3.7/site-packages/grpc/_channel.py", line 934, in with_call
                        return _end_unary_response_blocking(state, call, True, None)
                      File "/home/lmancuso/bruno/lib/python3.7/site-packages/grpc/_channel.py", line 826, in _end_unary_response_blocking
                        raise _InactiveRpcError(state)
                    grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
                            status = StatusCode.UNKNOWN
                            details = "Exception calling application: Aggregator does not have an aggregated tensor for
                    TensorKey(tensor_name='block_1.1.running_mean', origin='aggregator_plan.yaml_4ebcd576', round_number=55, report=False,
                    tags=('model',))"
                            debug_error_string = "{"created":"@1646124781.553286945","description":"Error received from peer
                    ipv4:172.20.8.151:59526","file":"src/core/lib/surface/call.cc","file_line":1062,"grpc_message":"Exception calling application:
                    Aggregator does not have an aggregated tensor for TensorKey(tensor_name='block_1.1.running_mean',
                    origin='aggregator_plan.yaml_4ebcd576', round_number=55, report=False, tags=('model',))","grpc_status":2}"

I was doing image classification on MNIST using 1 director and 10 envoys with a non-iid data distribution.

This closes #422.

Serialization of tf.keras models is natively supported as of https://github.com/keras-team/keras/pull/14748. This PR removes hotfix function to allow serialization on TF2.7 and greater.

This PR enables IP-based connection in addition to connecting based on registered FQDNs. It also corrects a bug which was leading to the use of incorrect default plan .yaml files.

This adds a workspace for OpenFL to train a TensorFlow 3D U-Net model on BraTS 2020.

• I show how to use a TensorFlow dataloader rather than loading everything into memory. It should make for a nice example to showcase how to use custom data loaders

• I fixed a bug in the KerasRunner on the validation function. It was using x_validate rather than the get_valid_data() function. This change makes it possible to use an arbitrary dataloader.

• I think there's a bug with db_rounds to keep. When I set it to 1 round to keep, the federation crashes on round 2 because it doesn't have the graph variables. It looks like this should be at least >=2 rounds to keep. I'd suggest doing a max(db_rounds, 2) so that the user doesn't inadvertently put in something like 0 (or -999).

• The model trains, but no guarantees on a good Dice score. It's just a template.

Hope you all are well. спасибо друзья!

Core timeout functionality is in place. To apply timeouts to any function/method in openfl codebase, import and decorate the function with @fedtiming.

from openfl.utilities.fed_timer import fedtiming

@fedtiming(timeout=10)    #10 seconds
def some_function(*args, **kwargs):
    pass

Update: 20th Oct

The @fedtiming can now be decorated on normal python functions or async co-routine likewise. Usage remains the same.

This PR introduces additional Director-Admin service to the Director component. New API allows modifying Director component settings and interacting with Envoy and Experiment registries.

Signed-off-by: igor-davidyuk [email protected]

New

• Ubuntu test sequence
• Windows test sequence
• 2 badges in the README.md

The test sequence for both operating systems is following:

• Linter
• PyTest
• Dockerization
• Interactive API (UNet Kvasir example)
• CLI (Keras CNN MNIST template)

Describe the bug fx command fails with the following after installing OpenFL from repo:

$ fx
Traceback (most recent call last):
  File "/home/vmuser/venv_test/bin/fx", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/home/vmuser/venv_test/lib/python3.8/site-packages/pkg_resources/__init__.py", line 3252, in <module>
    def _initialize_master_working_set():
  File "/home/vmuser/venv_test/lib/python3.8/site-packages/pkg_resources/__init__.py", line 3235, in _call_aside
    f(*args, **kwargs)
  File "/home/vmuser/venv_test/lib/python3.8/site-packages/pkg_resources/__init__.py", line 3264, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/home/vmuser/venv_test/lib/python3.8/site-packages/pkg_resources/__init__.py", line 583, in _build_master
    ws.require(__requires__)
  File "/home/vmuser/venv_test/lib/python3.8/site-packages/pkg_resources/__init__.py", line 900, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/home/vmuser/venv_test/lib/python3.8/site-packages/pkg_resources/__init__.py", line 786, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'jsonpointer>1.13; extra == "format-nongpl"' distribution was not found and is required by jsonschema

A second installation on the same environment fixes this issue, but it's not very elegant to demand invoking pip install twice

To Reproduce

python -m venv venv_test
. venv_test/bin/activate
git clone https://github.com/intel/openfl.git
pip install ./openfl
fx

Expected behavior fx runs without dependency errors

Bumps tensorflow from 2.8.3 to 2.9.3.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

For manual PKI exchange, certificates are stored under 'cert' directory inside workspace by default. This PR decouples the location of certificates from workspace in that the user can now specify certificate path to store certs which can reside outside the workspace.

This PR was automatically created by Snyk using the credentials of a real user.

Snyk has created this PR to fix one or more vulnerable packages in the `pip` dependencies of this project.

Changes included in this PR

• Changes to the following files to upgrade the vulnerable dependencies to a fixed version:
  • openfl-workspace/torch_cnn_mnist_straggler_check/requirements.txt

tensorboard 2.1.0 requires numpy, which is not installed.
tensorboard 2.1.0 requires protobuf, which is not installed.
google-auth 1.35.0 requires rsa, which is not installed.

Vulnerabilities that will be fixed

By pinning:

Severity | Priority Score (*) | Issue | Upgrade | Breaking Change | Exploit Maturity :-------------------------:|-------------------------|:-------------------------|:-------------------------|:-------------------------|:------------------------- | 551/1000
Why? Recently disclosed, Has a fix available, CVSS 5.3 | Regular Expression Denial of Service (ReDoS)
SNYK-PYTHON-WHEEL-3180413 | wheel:
0.30.0 -> 0.38.0
| No | No Known Exploit

(*) Note that the real score may have changed since the PR was raised.

Some vulnerabilities couldn't be fully fixed and so Snyk will still find them when the project is tested again. This may be because the vulnerability existed within more than one direct dependency, but not all of the affected dependencies could be upgraded.

Check the changes in this PR to ensure they won't cause issues with your project.

Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs.

For more information: 🧐 View latest project report

🛠 Adjust project settings

📚 Read more about Snyk's upgrade and patch logic

Learn how to fix vulnerabilities with free interactive lessons:

🦉 Regular Expression Denial of Service (ReDoS)