feat(settings): add admin mapping section EE-971

- fixed merge conflicts

.github/ISSUE_TEMPLATE.md

@@ -28,15 +28,17 @@ Briefly describe the problem you are having in a few paragraphs.
 
 **Steps to reproduce the issue:**
 
-1. 2. 3.
+1.
+2.
+3.
 
 Any other info e.g. Why do you consider this to be a bug? What did you expect to happen instead?
 
 **Technical details:**
 
-- Portainer version:
-- Target Docker version (the host/cluster you manage):
-- Platform (windows/linux):
-- Command used to start Portainer (`docker run -p 9443:9443 portainer/portainer`):
-- Target Swarm version (if applicable):
-- Browser:
+* Portainer version:
+* Target Docker version (the host/cluster you manage):
+* Platform (windows/linux):
+* Command used to start Portainer (`docker run -p 9000:9000 portainer/portainer`):
+* Target Swarm version (if applicable):
+* Browser:

.github/ISSUE_TEMPLATE/Bug_report.md

@@ -1,9 +1,6 @@
 ---
 name: Bug report
 about: Create a bug report
-title: ''
-labels: bug/need-confirmation, kind/bug
-assignees: ''
 ---
 
 <!--
@@ -12,7 +9,7 @@ Thanks for reporting a bug for Portainer !
 
 You can find more information about Portainer support framework policy here: https://www.portainer.io/2019/04/portainer-support-policy/
 
-Do you need help or have a question? Come chat with us on Slack http://portainer.slack.com/
+Do you need help or have a question? Come chat with us on Slack http://portainer.io/slack/.
 
 Before opening a new issue, make sure that we do not have any duplicates
 already open. You can ensure this by searching the issue list for this
@@ -43,12 +40,9 @@ You can see how [here](https://documentation.portainer.io/archive/1.23.2/faq/#ho
 
 - Portainer version:
 - Docker version (managed by Portainer):
-- Kubernetes version (managed by Portainer):
 - Platform (windows/linux):
-- Command used to start Portainer (`docker run -p 9443:9443 portainer/portainer`):
+- Command used to start Portainer (`docker run -p 9000:9000 portainer/portainer`):
 - Browser:
-- Use Case (delete as appropriate): Using Portainer at Home, Using Portainer in a Commerical setup.
-- Have you reviewed our technical documentation and knowledge base? Yes/No
 
 **Additional context**
 Add any other context about the problem here.

.github/ISSUE_TEMPLATE/Custom.md

@@ -1,25 +1,17 @@
----
-name: Question
-about: Ask us a question about Portainer usage or deployment
-title: ''
-labels: ''
-assignees: ''
-
----
-Before you start, we need a little bit more information from you:
-
-Use Case (delete as appropriate): Using Portainer at Home, Using Portainer in a Commerical setup.
-
-Have you reviewed our technical documentation and knowledge base? Yes/No
-
-<!--
-
-You can find more information about Portainer support framework policy here: https://old.portainer.io/2019/04/portainer-support-policy/
-
-Do you need help or have a question? Come chat with us on Slack http://portainer.slack.com/
-
-Also, be sure to check our FAQ and documentation first: https://documentation.portainer.io/
--->
-
-**Question**:
-How can I deploy Portainer on... ?
+---
+name: Question
+about: Ask us a question about Portainer usage or deployment
+
+---
+
+<!--
+
+You can find more information about Portainer support framework policy here: https://www.portainer.io/2019/04/portainer-support-policy/
+
+Do you need help or have a question? Come chat with us on Slack http://portainer.io/slack/
+
+Also, be sure to check our FAQ and documentation first: https://portainer.readthedocs.io
+-->
+
+**Question**:
+How can I deploy Portainer on... ?

.github/ISSUE_TEMPLATE/Feature_request.md

@@ -1,34 +1,31 @@
----
-name: Feature request
-about: Suggest a feature/enhancement that should be added in Portainer
-title: ''
-labels: ''
-assignees: ''
-
----
-
-<!--
-
-Thanks for opening a feature request for Portainer !
-
-Do you need help or have a question? Come chat with us on Slack http://portainer.slack.com/
-
-Before opening a new issue, make sure that we do not have any duplicates
-already open. You can ensure this by searching the issue list for this
-repository. If there is a duplicate, please close your issue and add a comment
-to the existing issue instead.
-
-Also, be sure to check our FAQ and documentation first: https://documentation.portainer.io/
--->
-
-**Is your feature request related to a problem? Please describe.**
-A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
-
-**Describe the solution you'd like**
-A clear and concise description of what you want to happen.
-
-**Describe alternatives you've considered**
-A clear and concise description of any alternative solutions or features you've considered.
-
-**Additional context**
-Add any other context or screenshots about the feature request here.
+---
+name: Feature request
+about: Suggest a feature/enhancement that should be added in Portainer
+
+---
+
+<!--
+
+Thanks for opening a feature request for Portainer !
+
+Do you need help or have a question? Come chat with us on Slack http://portainer.io/slack/
+
+Before opening a new issue, make sure that we do not have any duplicates
+already open. You can ensure this by searching the issue list for this
+repository. If there is a duplicate, please close your issue and add a comment
+to the existing issue instead.
+
+Also, be sure to check our FAQ and documentation first: https://portainer.readthedocs.io
+-->
+
+**Is your feature request related to a problem? Please describe.**
+A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+
+**Describe the solution you'd like**
+A clear and concise description of what you want to happen.
+
+**Describe alternatives you've considered**
+A clear and concise description of any alternative solutions or features you've considered.
+
+**Additional context**
+Add any other context or screenshots about the feature request here.

.github/stale.yml

@@ -0,0 +1,53 @@
+# Config for Stalebot, limited to only `issues`
+only: issues
+
+# Issues config
+issues:
+  daysUntilStale: 60
+  daysUntilClose: 7
+
+  # Limit the number of actions per hour, from 1-30. Default is 30
+  limitPerRun: 30
+
+  # Issues with these labels will never be considered stale
+  exemptLabels:
+    - kind/enhancement
+    - kind/question
+    - kind/style
+    - kind/workaround
+    - bug/need-confirmation
+    - bug/confirmed
+    - status/discuss
+
+  # Only issues with all of these labels are checked if stale. Defaults to `[]` (disabled)
+  onlyLabels: []
+
+  # Set to true to ignore issues in a project (defaults to false)
+  exemptProjects: true
+  # Set to true to ignore issues in a milestone (defaults to false)
+  exemptMilestones: true
+  # Set to true to ignore issues with an assignee (defaults to false)
+  exemptAssignees: true
+
+  # Label to use when marking an issue as stale
+  staleLabel: status/stale
+
+  # Comment to post when marking an issue as stale. Set to `false` to disable
+  markComment: >
+    This issue has been marked as stale as it has not had recent activity,
+    it will be closed if no further activity occurs in the next 7 days.
+    If you believe that it has been incorrectly labelled as stale,
+    leave a comment and the label will be removed.
+
+  # Comment to post when removing the stale label.
+  # unmarkComment: >
+  #   Your comment here.
+
+  # Comment to post when closing a stale issue. Set to `false` to disable
+  closeComment: >
+    Since no further activity has appeared on this issue it will be closed.
+    If you believe that it has been incorrectly closed, leave a comment
+    mentioning `ametdoohan`, `balasu` or `keverv` and one of our staff will then review the issue.
+
+    Note - If it is an old bug report, make sure that it is reproduceable in the
+    latest version of Portainer as it may have already been fixed.

.github/workflows/label-conflcts.yaml

@@ -1,15 +0,0 @@
-on:
-  push:
-    branches:
-      - develop
-      - 'release/**'
-jobs:
-  triage:
-    runs-on: ubuntu-latest
-    steps:
-      - uses: mschilde/auto-label-merge-conflicts@master
-        with:
-          CONFLICT_LABEL_NAME: 'has conflicts'
-          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-          MAX_RETRIES: 5
-          WAIT_MS: 5000

.github/workflows/rebase.yml

@@ -1,19 +0,0 @@
-name: Automatic Rebase
-on:
-  issue_comment:
-    types: [created]
-jobs:
-  rebase:
-    name: Rebase
-    if: github.event.issue.pull_request != '' && contains(github.event.comment.body, '/rebase')
-    runs-on: ubuntu-latest
-    steps:
-      - name: Checkout the latest code
-        uses: actions/checkout@v2
-        with:
-          token: ${{ secrets.GITHUB_TOKEN }}
-          fetch-depth: 0 # otherwise, you will fail to push refs to dest repo
-      - name: Automatic Rebase
-        uses: cirrus-actions/rebase@1.4
-        env:
-          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

.github/workflows/stale.yml

@@ -1,27 +0,0 @@
-name: Close Stale Issues
-on:
-  schedule:
-  - cron: '0 12 * * *'
-jobs:
-  stale:
-    runs-on: ubuntu-latest
-    permissions:
-      issues: write
-
-    steps:
-    - uses: actions/stale@v4.0.0
-      with:
-        repo-token: ${{ secrets.GITHUB_TOKEN }}
-
-        # Issue Config
-        days-before-issue-stale: 60
-        days-before-issue-close: 7
-        stale-issue-label: 'status/stale'
-        exempt-all-issue-milestones: true # Do not stale issues in a milestone
-        exempt-issue-labels: kind/enhancement, kind/style, kind/workaround, kind/refactor, bug/need-confirmation, bug/confirmed, status/discuss
-        stale-issue-message: 'This issue has been marked as stale as it has not had recent activity, it will be closed if no further activity occurs in the next 7 days. If you believe that it has been incorrectly labelled as stale, leave a comment and the label will be removed.'
-        close-issue-message: 'Since no further activity has appeared on this issue it will be closed. If you believe that it has been incorrectly closed, leave a comment mentioning `portainer/support` and one of our staff will then review the issue. Note - If it is an old bug report, make sure that it is reproduceable in the latest version of Portainer as it may have already been fixed.'
-        
-        # Pull Request Config
-        days-before-pr-stale: -1 # Do not stale pull request
-        days-before-pr-close: -1 # Do not close pull request

.gitignore

@@ -8,8 +8,10 @@ api/cmd/portainer/portainer*
 **/.vscode/tasks.json
 
 .eslintcache
-__debug_bin
-
-api/docs
 .idea
+test/e2e/cypress/screenshots
+*.db
+*.log
+__debug_bin
+api/docs
 .env

.vscode.example/portainer.code-snippets

@@ -163,19 +163,5 @@
       "// @failure 500 \"Server error\"",
       "// @router /{id} [get]"
     ]
-  },
-  "analytics": {
-    "prefix": "nlt",
-    "body": ["analytics-on", "analytics-category=\"$1\"", "analytics-event=\"$2\""],
-    "description": "analytics"
-  },
-  "analytics-if": {
-    "prefix": "nltf",
-    "body": ["analytics-if=\"$1\""],
-    "description": "analytics"
-  },
-  "analytics-metadata": {
-    "prefix": "nltm",
-    "body": "analytics-properties=\"{ metadata: { $1 } }\""
   }
 }

.vscode.example/settings.json

@@ -1,4 +0,0 @@
-{
-  "go.lintTool": "golangci-lint",
-  "go.lintFlags": ["--fast", "-E", "exportloopref"]
-}

ATTRIBUTIONS.md

@@ -1,32 +0,0 @@
-# Open Source License Attribution
-
-This application uses Open Source components. You can find the source
-code of their open source projects along with license information below.
-We acknowledge and are grateful to these developers for their contributions
-to open source.
-
-### [angular-json-tree](https://github.com/awendland/angular-json-tree)
-
-by [Alex Wendland](https://github.com/awendland) is licensed under [CC BY 4.0 License](https://creativecommons.org/licenses/by/4.0/)
-
-### [caniuse-db](https://github.com/Fyrd/caniuse)
-
-by [caniuse.com](caniuse.com) is licensed under [CC BY 4.0 License](https://creativecommons.org/licenses/by/4.0/)
-
-### [caniuse-lite](https://github.com/ben-eb/caniuse-lite)
-
-by [caniuse.com](caniuse.com) is licensed under [CC BY 4.0 License](https://creativecommons.org/licenses/by/4.0/)
-
-### [spdx-exceptions](https://github.com/jslicense/spdx-exceptions.json)
-
-by Kyle Mitchell using [SPDX](https://spdx.dev/) from Linux Foundation licensed under [CC BY 3.0 License](https://creativecommons.org/licenses/by/3.0/)
-
-### [fontawesome-free](https://github.com/FortAwesome/Font-Awesome) Icons
-
-by [Fort Awesome](https://fortawesome.com/) is licensed under [CC BY 4.0 License](https://creativecommons.org/licenses/by/4.0/)
-
-Portainer also contains the following code, which is licensed under the [MIT license](https://opensource.org/licenses/MIT):
-
-UI For Docker: Copyright (c) 2013-2016 Michael Crosby (crosbymichael.com), Kevan Ahlquist (kevanahlquist.com), Anthony Lapenna (portainer.io)
-
-rdash-angular: Copyright (c) [2014][elliot hesp]

CONTRIBUTING.md

@@ -91,7 +91,7 @@ Then build and run the project:
 $ yarn start
 ```
 
-Portainer can now be accessed at <https://localhost:9443>.
+Portainer can now be accessed at <http://localhost:9000>.
 
 Find more detailed steps at <https://documentation.portainer.io/contributing/instructions/>.
 
@@ -127,9 +127,3 @@ When adding a new route to an existing handler use the following as a template (
 ```
 
 explanation about each line can be found (here)[https://github.com/swaggo/swag#api-operation]
-
-## Licensing
-
-See the [LICENSE](https://github.com/portainer/portainer/blob/develop/LICENSE) file for our project's licensing. We will ask you to confirm the licensing of your contribution.
-
-We may ask you to sign a [Contributor License Agreement (CLA)](http://en.wikipedia.org/wiki/Contributor_License_Agreement) for larger changes.

README.md

@@ -1,14 +1,16 @@
 <p align="center">
-  <img title="portainer" src='https://github.com/portainer/portainer/blob/develop/app/assets/images/portainer-github-banner.png?raw=true' />
+  <img title="portainer" src='https://github.com/portainer/portainer/blob/develop/app/assets/images/logo_alt.png?raw=true' />
 </p>
 
-**Portainer CE** is a lightweight ‘universal’ management GUI that can be used to **easily** manage Docker, Swarm, Kubernetes and ACI environments. It is designed to be as **simple** to deploy as it is to use.
+[![Docker Pulls](https://img.shields.io/docker/pulls/portainer/portainer.svg)](https://hub.docker.com/r/portainer/portainer/)
+[![Microbadger](https://images.microbadger.com/badges/image/portainer/portainer.svg)](http://microbadger.com/images/portainer/portainer 'Image size')
+[![Build Status](https://portainer.visualstudio.com/Portainer%20CI/_apis/build/status/Portainer%20CI?branchName=develop)](https://portainer.visualstudio.com/Portainer%20CI/_build/latest?definitionId=3&branchName=develop)
+[![Code Climate](https://codeclimate.com/github/portainer/portainer/badges/gpa.svg)](https://codeclimate.com/github/portainer/portainer)
+[![Donate](https://img.shields.io/badge/Donate-PayPal-green.svg)](https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=YHXZJQNJQ36H6)
 
-Portainer consists of a single container that can run on any cluster. It can be deployed as a Linux container or a Windows native container.
-
-**Portainer** allows you to manage all your orchestrator resources (containers, images, volumes, networks and more) through a super-simple graphical interface.
-
-A fully supported version of Portainer is available for business use. Visit http://www.portainer.io to learn more
+**_Portainer_** is a lightweight management UI which allows you to **easily** manage your different Docker environments (Docker hosts or Swarm clusters).
+**_Portainer_** is meant to be as **simple** to deploy as it is to use. It consists of a single container that can run on any Docker engine (can be deployed as Linux container or a Windows native container, supports other platforms too).
+**_Portainer_** allows you to manage all your Docker resources (containers, images, volumes, networks and more!) It is compatible with the _standalone Docker_ engine and with _Docker Swarm mode_.
 
 ## Demo
 
@@ -16,51 +18,38 @@ You can try out the public demo instance: http://demo.portainer.io/ (login with
 
 Please note that the public demo cluster is **reset every 15min**.
 
-## Latest Version
+Alternatively, you can deploy a copy of the demo stack inside a [play-with-docker (PWD)](https://labs.play-with-docker.com) playground:
 
-Portainer CE is updated regularly. We aim to do an update release every couple of months.
+- Browse [PWD/?stack=portainer-demo/play-with-docker/docker-stack.yml](http://play-with-docker.com/?stack=https://raw.githubusercontent.com/portainer/portainer-demo/master/play-with-docker/docker-stack.yml)
+- Sign in with your [Docker ID](https://docs.docker.com/docker-id)
+- Follow [these](https://github.com/portainer/portainer-demo/blob/master/play-with-docker/docker-stack.yml#L5-L8) steps.
 
-**The latest version of Portainer is 2.6.x** And you can find the release notes [here.](https://www.portainer.io/blog/new-portainer-ce-2.6.0-release)
-Portainer is on version 2, the second number denotes the month of release.
+Unlike the public demo, the playground sessions are deleted after 4 hours. Apart from that, all the settings are the same, including default credentials.
 
 ## Getting started
 
-- [Deploy Portainer](https://documentation.portainer.io/quickstart/)
+- [Deploy Portainer](https://www.portainer.io/installation/)
 - [Documentation](https://documentation.portainer.io)
-- [Contribute to the project](https://documentation.portainer.io/contributing/instructions/)
-
-## Features & Functions
-
-View [this](https://www.portainer.io/products) table to see all of the Portainer CE functionality and compare to Portainer Business.
-
-- [Portainer CE for Docker / Docker Swarm](https://www.portainer.io/solutions/docker)
-- [Portainer CE for Kubernetes](https://www.portainer.io/solutions/kubernetes-ui)
-- [Portainer CE for Azure ACI](https://www.portainer.io/solutions/serverless-containers)
 
 ## Getting help
 
-Portainer CE is an open source project and is supported by the community. You can buy a supported version of Portainer at portainer.io
+For FORMAL Support, please purchase a support subscription from here: https://www.portainer.io/products-services/portainer-business-support/
 
-Learn more about Portainers community support channels [here.](https://www.portainer.io/help_about)
+For community support: You can find more information about Portainer's community support framework policy here: https://www.portainer.io/2019/04/portainer-support-policy/
 
 - Issues: https://github.com/portainer/portainer/issues
+- FAQ: https://documentation.portainer.io
 - Slack (chat): https://portainer.io/slack/
 
-You can join the Portainer Community by visiting community.portainer.io. This will give you advance notice of events, content and other related Portainer content.
-
 ## Reporting bugs and contributing
 
 - Want to report a bug or request a feature? Please open [an issue](https://github.com/portainer/portainer/issues/new).
-- Want to help us build **_portainer_**? Follow our [contribution guidelines](https://documentation.portainer.io/contributing/instructions/) to build it locally and make a pull request. We need all the help we can get!
+- Want to help us build **_portainer_**? Follow our [contribution guidelines](https://www.portainer.io/documentation/how-to-contribute/) to build it locally and make a pull request. We need all the help we can get!
 
 ## Security
 
 - Here at Portainer, we believe in [responsible disclosure](https://en.wikipedia.org/wiki/Responsible_disclosure) of security issues. If you have found a security issue, please report it to <security@portainer.io>.
 
-## WORK FOR US
-
-If you are a developer, and our code in this repo makes sense to you, we would love to hear from you. We are always on the hunt for awesome devs, either freelance or employed. Drop us a line to info@portainer.io with your details and we will be in touch. 
-
 ## Privacy
 
 **To make sure we focus our development effort in the right places we need to know which features get used most often. To give us this information we use [Matomo Analytics](https://matomo.org/), which is hosted in Germany and is fully GDPR compliant.**
@@ -75,4 +64,8 @@ Portainer supports "Current - 2 docker versions only. Prior versions may operate
 
 Portainer is licensed under the zlib license. See [LICENSE](./LICENSE) for reference.
 
-Portainer also contains code from open source projects. See [ATTRIBUTIONS.md](./ATTRIBUTIONS.md) for a list.
+Portainer also contains the following code, which is licensed under the [MIT license](https://opensource.org/licenses/MIT):
+
+UI For Docker: Copyright (c) 2013-2016 Michael Crosby (crosbymichael.com), Kevan Ahlquist (kevanahlquist.com), Anthony Lapenna (portainer.io)
+
+rdash-angular: Copyright (c) [2014][elliot hesp]

api/adminmonitor/admin_monitor.go

@@ -37,7 +37,7 @@ func (m *Monitor) Start() {
 		case <-time.After(m.timeout):
 			initialized, err := m.WasInitialized()
 			if err != nil {
-				logFatalf("%s", err)
+				logFatalf("failed getting admin user: %s", err)
 			}
 			if !initialized {
 				logFatalf("[FATAL] [internal,init] No administrator account was created in %f mins. Shutting down the Portainer instance for security reasons", m.timeout.Minutes())

api/api-description.md

@@ -1,10 +1,10 @@
 Portainer API is an HTTP API served by Portainer. It is used by the Portainer UI and everything you can do with the UI can be done using the HTTP API.
-Examples are available at https://documentation.portainer.io/api/api-examples/
+Examples are available at https://gist.github.com/deviantony/77026d402366b4b43fa5918d41bc42f8
 You can find out more about Portainer at [http://portainer.io](http://portainer.io) and get some support on [Slack](http://portainer.io/slack/).
 
 # Authentication
 
-Most of the API environments(endpoints) require to be authenticated as well as some level of authorization to be used.
+Most of the API endpoints require to be authenticated as well as some level of authorization to be used.
 Portainer API uses JSON Web Token to manage authentication and thus requires you to provide a token in the **Authorization** header of each request
 with the **Bearer** authentication mechanism.
 
@@ -16,7 +16,7 @@ Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpZCI6MSwidXNlcm5hbWUiOiJhZG1pbiIs
 
 # Security
 
-Each API environment(endpoint) has an associated access policy, it is documented in the description of each environment(endpoint).
+Each API endpoint has an associated access policy, it is documented in the description of each endpoint.
 
 Different access policies are available:
 
@@ -27,27 +27,27 @@ Different access policies are available:
 
 ### Public access
 
-No authentication is required to access the environments(endpoints) with this access policy.
+No authentication is required to access the endpoints with this access policy.
 
 ### Authenticated access
 
-Authentication is required to access the environments(endpoints) with this access policy.
+Authentication is required to access the endpoints with this access policy.
 
 ### Restricted access
 
-Authentication is required to access the environments(endpoints) with this access policy.
+Authentication is required to access the endpoints with this access policy.
 Extra-checks might be added to ensure access to the resource is granted. Returned data might also be filtered.
 
 ### Administrator access
 
-Authentication as well as an administrator role are required to access the environments(endpoints) with this access policy.
+Authentication as well as an administrator role are required to access the endpoints with this access policy.
 
 # Execute Docker requests
 
-Portainer **DO NOT** expose specific environments(endpoints) to manage your Docker resources (create a container, remove a volume, etc...).
+Portainer **DO NOT** expose specific endpoints to manage your Docker resources (create a container, remove a volume, etc...).
 
 Instead, it acts as a reverse-proxy to the Docker HTTP API. This means that you can execute Docker requests **via** the Portainer HTTP API.
 
-To do so, you can use the `/endpoints/{id}/docker` Portainer API environment(endpoint) (which is not documented below due to Swagger limitations). This environment(endpoint) has a restricted access policy so you still need to be authenticated to be able to query this environment(endpoint). Any query on this environment(endpoint) will be proxied to the Docker API of the associated environment(endpoint) (requests and responses objects are the same as documented in the Docker API).
+To do so, you can use the `/endpoints/{id}/docker` Portainer API endpoint (which is not documented below due to Swagger limitations). This endpoint has a restricted access policy so you still need to be authenticated to be able to query this endpoint. Any query on this endpoint will be proxied to the Docker API of the associated endpoint (requests and responses objects are the same as documented in the Docker API).
 
-**NOTE**: You can find more information on how to query the Docker API in the [Docker official documentation](https://docs.docker.com/engine/api/v1.30/) as well as in [this Portainer example](https://documentation.portainer.io/api/api-examples/).
+**NOTE**: You can find more information on how to query the Docker API in the [Docker official documentation](https://docs.docker.com/engine/api/v1.30/) as well as in [this Portainer example](https://gist.github.com/deviantony/77026d402366b4b43fa5918d41bc42f8).

api/archive/targz_test.go

@@ -9,7 +9,6 @@ import (
 	"path/filepath"
 	"testing"
 
-	"github.com/docker/docker/pkg/ioutils"
 	"github.com/stretchr/testify/assert"
 )
 
@@ -27,7 +26,7 @@ func listFiles(dir string) []string {
 }
 
 func Test_shouldCreateArhive(t *testing.T) {
-	tmpdir, _ := ioutils.TempDir("", "backup")
+	tmpdir, _ := ioutil.TempDir("", "backup")
 	defer os.RemoveAll(tmpdir)
 
 	content := []byte("content")
@@ -40,7 +39,7 @@ func Test_shouldCreateArhive(t *testing.T) {
 	assert.Nil(t, err)
 	assert.Equal(t, filepath.Join(tmpdir, fmt.Sprintf("%s.tar.gz", filepath.Base(tmpdir))), gzPath)
 
-	extractionDir, _ := ioutils.TempDir("", "extract")
+	extractionDir, _ := ioutil.TempDir("", "extract")
 	defer os.RemoveAll(extractionDir)
 
 	cmd := exec.Command("tar", "-xzf", gzPath, "-C", extractionDir)
@@ -63,7 +62,7 @@ func Test_shouldCreateArhive(t *testing.T) {
 }
 
 func Test_shouldCreateArhiveXXXXX(t *testing.T) {
-	tmpdir, _ := ioutils.TempDir("", "backup")
+	tmpdir, _ := ioutil.TempDir("", "backup")
 	defer os.RemoveAll(tmpdir)
 
 	content := []byte("content")
@@ -76,7 +75,7 @@ func Test_shouldCreateArhiveXXXXX(t *testing.T) {
 	assert.Nil(t, err)
 	assert.Equal(t, filepath.Join(tmpdir, fmt.Sprintf("%s.tar.gz", filepath.Base(tmpdir))), gzPath)
 
-	extractionDir, _ := ioutils.TempDir("", "extract")
+	extractionDir, _ := ioutil.TempDir("", "extract")
 	defer os.RemoveAll(extractionDir)
 
 	r, _ := os.Open(gzPath)

api/archive/zip_test.go

@@ -16,8 +16,8 @@ func TestUnzipFile(t *testing.T) {
 		Archive structure.
 		├── 0
 		│	├── 1
-		│	│	└── 2.txt
-		│	└── 1.txt
+		│ 	│   └── 2.txt
+		│   └── 1.txt
 		└── 0.txt
 	*/
 

api/backup/backup.go

@@ -2,6 +2,7 @@ package backup
 
 import (
 	"fmt"
+	"log"
 	"os"
 	"path/filepath"
 	"time"
@@ -10,23 +11,39 @@ import (
 	portainer "github.com/portainer/portainer/api"
 	"github.com/portainer/portainer/api/archive"
 	"github.com/portainer/portainer/api/crypto"
-	"github.com/portainer/portainer/api/filesystem"
 	"github.com/portainer/portainer/api/http/offlinegate"
+	"github.com/portainer/portainer/api/s3"
 )
 
 const rwxr__r__ os.FileMode = 0744
 
-var filesToBackup = []string{
-	"certs",
-	"compose",
-	"config.json",
-	"custom_templates",
-	"edge_jobs",
-	"edge_stacks",
-	"extensions",
-	"portainer.key",
-	"portainer.pub",
-	"tls",
+var filesToBackup = []string{"compose", "config.json", "custom_templates", "edge_jobs", "edge_stacks", "extensions", "portainer.key", "portainer.pub", "tls"}
+
+func BackupToS3(settings portainer.S3BackupSettings, gate *offlinegate.OfflineGate, datastore portainer.DataStore, filestorePath string) error {
+	archivePath, err := CreateBackupArchive(settings.Password, gate, datastore, filestorePath)
+	if err != nil {
+		log.Printf("[ERROR] failed to backup: %s \n", err)
+		return err
+	}
+	archiveReader, err := os.Open(archivePath)
+	if err != nil {
+		log.Println("[ERROR] failed to open backup file")
+		return err
+	}
+	defer os.RemoveAll(filepath.Dir(archivePath))
+
+	archiveName := fmt.Sprintf("portainer-backup_%s", filepath.Base(archivePath))
+
+	s3session, err := s3.NewSession(settings.Region, settings.AccessKeyID, settings.SecretAccessKey)
+	if err != nil {
+		log.Printf("[ERROR] %s \n", err)
+		return err
+	}
+	if err := s3.Upload(s3session, archiveReader, settings.BucketName, archiveName); err != nil {
+		log.Printf("[ERROR] failed to upload backup to S3: %s \n", err)
+		return err
+	}
+	return nil
 }
 
 // Creates a tar.gz system archive and encrypts it if password is not empty. Returns a path to the archive file.
@@ -44,7 +61,7 @@ func CreateBackupArchive(password string, gate *offlinegate.OfflineGate, datasto
 	}
 
 	for _, filename := range filesToBackup {
-		err := filesystem.CopyPath(filepath.Join(filestorePath, filename), backupDirPath)
+		err := copyPath(filepath.Join(filestorePath, filename), backupDirPath)
 		if err != nil {
 			return "", errors.Wrap(err, "Failed to create backup file")
 		}

api/backup/backup_scheduler.go

@@ -0,0 +1,118 @@
+package backup
+
+import (
+	"context"
+	"log"
+	"time"
+
+	"github.com/pkg/errors"
+	portainer "github.com/portainer/portainer/api"
+	"github.com/portainer/portainer/api/http/offlinegate"
+	"github.com/robfig/cron/v3"
+)
+
+// BackupScheduler orchestrates S3 settings and active backup cron jobs
+type BackupScheduler struct {
+	cronmanager     *cron.Cron
+	s3backupService portainer.S3BackupService
+	gate            *offlinegate.OfflineGate
+	datastore       portainer.DataStore
+	filestorePath   string
+}
+
+func NewBackupScheduler(offlineGate *offlinegate.OfflineGate, datastore portainer.DataStore, filestorePath string) *BackupScheduler {
+	crontab := cron.New(cron.WithChain(cron.Recover(cron.DefaultLogger)))
+	s3backupService := datastore.S3Backup()
+
+	return &BackupScheduler{
+		cronmanager:     crontab,
+		s3backupService: s3backupService,
+		gate:            offlineGate,
+		datastore:       datastore,
+		filestorePath:   filestorePath,
+	}
+}
+
+// Start fetches latest backup settings and starts cron job if configured
+func (s *BackupScheduler) Start() error {
+	s.cronmanager.Start()
+
+	settings, err := s.s3backupService.GetSettings()
+	if err != nil {
+		return errors.Wrap(err, "failed to fetch settings")
+	}
+
+	if canBeScheduled(settings) {
+		return s.startJob(settings)
+	}
+
+	return nil
+}
+
+// Stop stops the scheduler if it is running; otherwise it does nothing.
+// A context is returned so the caller can wait for running jobs to complete.
+func (s *BackupScheduler) Stop() context.Context {
+	if s.cronmanager != nil {
+		log.Println("[DEBUG] Stopping backup scheduler")
+		return s.cronmanager.Stop()
+	}
+
+	return nil
+}
+
+// Update updates stored S3 backup settings and orchestrates cron jobs.
+// When scheduler has an active cron job, then it shuts it down.
+// When a provided settings has a cron, then starts a new cron job.
+// When ever current cron is being shut down, last cron error going to be dropped.
+func (s *BackupScheduler) Update(settings portainer.S3BackupSettings) error {
+
+	if err := s.s3backupService.UpdateSettings(settings); err != nil {
+		return errors.Wrap(err, "failed to update settings")
+	}
+
+	if err := s.stopJobs(); err != nil {
+		return errors.Wrap(err, "failed to stop current cronjob")
+	}
+
+	if canBeScheduled(settings) {
+		return s.startJob(settings)
+	}
+
+	return nil
+}
+
+// stops current backup cron job and drops last cron error if any
+func (s *BackupScheduler) stopJobs() error {
+	// stopping all cron jobs as there should be only one (c)
+	for _, job := range s.cronmanager.Entries() {
+		s.cronmanager.Remove(job.ID)
+	}
+
+	return s.s3backupService.DropStatus()
+}
+
+func (s *BackupScheduler) startJob(settings portainer.S3BackupSettings) error {
+	_, err := s.cronmanager.AddFunc(settings.CronRule, s.backup(settings))
+	if err != nil {
+		return errors.Wrap(err, "failed to start a new backup cron job")
+	}
+
+	return nil
+}
+
+func canBeScheduled(s portainer.S3BackupSettings) bool {
+	return s.AccessKeyID != "" && s.SecretAccessKey != "" && s.Region != "" && s.BucketName != "" && s.CronRule != ""
+}
+
+func (s *BackupScheduler) backup(settings portainer.S3BackupSettings) func() {
+	return func() {
+		err := BackupToS3(settings, s.gate, s.datastore, s.filestorePath)
+		status := portainer.S3BackupStatus{
+			Failed:    err != nil,
+			Timestamp: time.Now(),
+		}
+		if err = s.s3backupService.UpdateStatus(status); err != nil {
+			log.Printf("[ERROR] failed to update status of last scheduled backup. Status: %+v . Err: %s \n", status, err)
+		}
+	}
+}

api/backup/backup_scheduler_test.go

@@ -0,0 +1,112 @@
+package backup
+
+import (
+	"testing"
+	"time"
+
+	portainer "github.com/portainer/portainer/api"
+	i "github.com/portainer/portainer/api/internal/testhelpers"
+	"github.com/stretchr/testify/assert"
+)
+
+func newScheduler(status *portainer.S3BackupStatus, settings *portainer.S3BackupSettings) *BackupScheduler {
+	scheduler := NewBackupScheduler(nil, i.NewDatastore(i.WithS3BackupService(status, settings)), "")
+	scheduler.Start()
+
+	return scheduler
+}
+
+func settings(cronRule string,
+	accessKeyID string,
+	secretAccessKey string,
+	region string,
+	bucketName string) *portainer.S3BackupSettings {
+	return &portainer.S3BackupSettings{
+		CronRule:        cronRule,
+		AccessKeyID:     accessKeyID,
+		SecretAccessKey: secretAccessKey,
+		Region:          region,
+		BucketName:      bucketName,
+	}
+}
+
+func Test_startWithoutCron_shouldNotStartAJob(t *testing.T) {
+	scheduler := newScheduler(&portainer.S3BackupStatus{}, &portainer.S3BackupSettings{})
+	defer scheduler.Stop()
+
+	jobs := scheduler.cronmanager.Entries()
+	assert.Len(t, jobs, 0, "should have empty job list")
+}
+
+func Test_startWitACron_shouldAlsoStartAJob(t *testing.T) {
+	scheduler := newScheduler(&portainer.S3BackupStatus{}, settings("*/10 * * * *", "id", "key", "region", "bucket"))
+	defer scheduler.Stop()
+
+	jobs := scheduler.cronmanager.Entries()
+	assert.Len(t, jobs, 1, "should have 1 active job")
+}
+
+func Test_update_shouldDropStatus(t *testing.T) {
+	storedStatus := &portainer.S3BackupStatus{Failed: true, Timestamp: time.Now().Add(-time.Hour)}
+	scheduler := newScheduler(storedStatus, &portainer.S3BackupSettings{})
+	defer scheduler.Stop()
+
+	scheduler.Update(*settings("*/10 * * * *", "id", "key", "region", "bucket"))
+	assert.Equal(t, portainer.S3BackupStatus{}, *storedStatus, "stasus should be dropped")
+}
+
+func Test_update_shouldUpdateSettings(t *testing.T) {
+	storedSettings := &portainer.S3BackupSettings{}
+	scheduler := newScheduler(&portainer.S3BackupStatus{}, storedSettings)
+	defer scheduler.Stop()
+
+	newSettings := settings("", "id2", "key2", "region2", "bucket2")
+	scheduler.Update(*newSettings)
+
+	assert.EqualValues(t, *storedSettings, *newSettings, "updated settings should match stored settings")
+}
+
+func Test_updateWithCron_shouldStartAJob(t *testing.T) {
+	scheduler := newScheduler(&portainer.S3BackupStatus{}, &portainer.S3BackupSettings{})
+	defer scheduler.Stop()
+
+	jobs := scheduler.cronmanager.Entries()
+	assert.Len(t, jobs, 0, "should have empty job list upon startup")
+
+	scheduler.Update(*settings("*/10 * * * *", "id", "key", "region", "bucket"))
+
+	jobs = scheduler.cronmanager.Entries()
+	assert.Len(t, jobs, 1, "should have 1 active job")
+}
+
+func Test_updateWithoutCron_shouldStopActiveJob(t *testing.T) {
+	scheduler := newScheduler(&portainer.S3BackupStatus{}, &portainer.S3BackupSettings{})
+	defer scheduler.Stop()
+
+	scheduler.Update(*settings("*/10 * * * *", "id", "key", "region", "bucket"))
+
+	jobs := scheduler.cronmanager.Entries()
+	assert.Len(t, jobs, 1, "should have 1 active job")
+
+	scheduler.Update(*settings("", "id2", "key2", "region2", "bucket2"))
+
+	jobs = scheduler.cronmanager.Entries()
+	assert.Len(t, jobs, 0, "should have no active jobs")
+}
+
+func Test_updateWithACron_shouldStopActiveJob_andStartNewJob(t *testing.T) {
+	scheduler := newScheduler(&portainer.S3BackupStatus{}, &portainer.S3BackupSettings{})
+	defer scheduler.Stop()
+
+	scheduler.Update(*settings("*/10 * * * *", "id", "key", "region", "bucket"))
+
+	jobs := scheduler.cronmanager.Entries()
+	assert.Len(t, jobs, 1, "should have 1 active job")
+	initJobId := jobs[0].ID
+
+	scheduler.Update(*settings("*/10 * * * *", "id", "key", "region", "bucket"))
+
+	jobs = scheduler.cronmanager.Entries()
+	assert.Len(t, jobs, 1, "should have 1 active job")
+	assert.NotEqual(t, initJobId, jobs[0].ID, "new job should have a diffent id")
+}

api/backup/copy.go

@@ -1,4 +1,4 @@
-package filesystem
+package backup
 
 import (
 	"errors"
@@ -8,8 +8,7 @@ import (
 	"strings"
 )
 
-// CopyPath copies file or directory defined by the path to the toDir path
-func CopyPath(path string, toDir string) error {
+func copyPath(path string, toDir string) error {
 	info, err := os.Stat(path)
 	if err != nil && errors.Is(err, os.ErrNotExist) {
 		// skip copy if file does not exist
@@ -21,30 +20,17 @@ func CopyPath(path string, toDir string) error {
 		return copyFile(path, destination)
 	}
 
-	return CopyDir(path, toDir, true)
+	return copyDir(path, toDir)
 }
 
-// CopyDir copies contents of fromDir to toDir.
-// When keepParent is true, contents will be copied with their immediate parent dir,
-// i.e. given /from/dirA and /to/dirB with keepParent == true, result will be /to/dirB/dirA/<children>
-func CopyDir(fromDir, toDir string, keepParent bool) error {
+func copyDir(fromDir, toDir string) error {
 	cleanedSourcePath := filepath.Clean(fromDir)
 	parentDirectory := filepath.Dir(cleanedSourcePath)
 	err := filepath.Walk(cleanedSourcePath, func(path string, info os.FileInfo, err error) error {
 		if err != nil {
 			return err
 		}
-		var destination string
-		if keepParent {
-			destination = filepath.Join(toDir, strings.TrimPrefix(path, parentDirectory))
-		} else {
-			destination = filepath.Join(toDir, strings.TrimPrefix(path, cleanedSourcePath))
-		}
-
-		if destination == "" {
-			return nil
-		}
-
+		destination := filepath.Join(toDir, strings.TrimPrefix(path, parentDirectory))
 		if info.IsDir() {
 			return nil // skip directory creations
 		}

api/backup/copy_test.go

@@ -0,0 +1,104 @@
+package backup
+
+import (
+	"io/ioutil"
+	"os"
+	"path"
+	"path/filepath"
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+)
+
+func listFiles(dir string) []string {
+	items := make([]string, 0)
+	filepath.Walk(dir, func(path string, info os.FileInfo, err error) error {
+		if path == dir {
+			return nil
+		}
+		items = append(items, path)
+		return nil
+	})
+
+	return items
+}
+
+func contains(t *testing.T, list []string, path string) {
+	assert.Contains(t, list, path)
+	copyContent, _ := ioutil.ReadFile(path)
+	assert.Equal(t, "content\n", string(copyContent))
+}
+
+func Test_copyFile_returnsError_whenSourceDoesNotExist(t *testing.T) {
+	tmpdir, _ := ioutil.TempDir("", "backup")
+	defer os.RemoveAll(tmpdir)
+
+	err := copyFile("does-not-exist", tmpdir)
+	assert.NotNil(t, err)
+}
+
+func Test_copyFile_shouldMakeAbackup(t *testing.T) {
+	tmpdir, _ := ioutil.TempDir("", "backup")
+	defer os.RemoveAll(tmpdir)
+
+	content := []byte("content")
+	ioutil.WriteFile(path.Join(tmpdir, "origin"), content, 0600)
+
+	err := copyFile(path.Join(tmpdir, "origin"), path.Join(tmpdir, "copy"))
+	assert.Nil(t, err)
+
+	copyContent, _ := ioutil.ReadFile(path.Join(tmpdir, "copy"))
+	assert.Equal(t, content, copyContent)
+}
+
+func Test_copyDir_shouldCopyAllFilesAndDirectories(t *testing.T) {
+	destination, _ := ioutil.TempDir("", "destination")
+	defer os.RemoveAll(destination)
+	err := copyDir("./test_assets/copy_test", destination)
+	assert.Nil(t, err)
+
+	createdFiles := listFiles(destination)
+
+	contains(t, createdFiles, filepath.Join(destination, "copy_test", "outer"))
+	contains(t, createdFiles, filepath.Join(destination, "copy_test", "dir", ".dotfile"))
+	contains(t, createdFiles, filepath.Join(destination, "copy_test", "dir", "inner"))
+}
+
+func Test_backupPath_shouldSkipWhenNotExist(t *testing.T) {
+	tmpdir, _ := ioutil.TempDir("", "backup")
+	defer os.RemoveAll(tmpdir)
+
+	err := copyPath("does-not-exists", tmpdir)
+	assert.Nil(t, err)
+
+	assert.Empty(t, listFiles(tmpdir))
+}
+
+func Test_backupPath_shouldCopyFile(t *testing.T) {
+	tmpdir, _ := ioutil.TempDir("", "backup")
+	defer os.RemoveAll(tmpdir)
+
+	content := []byte("content")
+	ioutil.WriteFile(path.Join(tmpdir, "file"), content, 0600)
+
+	os.MkdirAll(path.Join(tmpdir, "backup"), 0700)
+	err := copyPath(path.Join(tmpdir, "file"), path.Join(tmpdir, "backup"))
+	assert.Nil(t, err)
+
+	copyContent, err := ioutil.ReadFile(path.Join(tmpdir, "backup", "file"))
+	assert.Nil(t, err)
+	assert.Equal(t, content, copyContent)
+}
+
+func Test_backupPath_shouldCopyDir(t *testing.T) {
+	destination, _ := ioutil.TempDir("", "destination")
+	defer os.RemoveAll(destination)
+	err := copyPath("./test_assets/copy_test", destination)
+	assert.Nil(t, err)
+
+	createdFiles := listFiles(destination)
+
+	contains(t, createdFiles, filepath.Join(destination, "copy_test", "outer"))
+	contains(t, createdFiles, filepath.Join(destination, "copy_test", "dir", ".dotfile"))
+	contains(t, createdFiles, filepath.Join(destination, "copy_test", "dir", "inner"))
+}

api/backup/restore.go

@@ -11,7 +11,6 @@ import (
 	portainer "github.com/portainer/portainer/api"
 	"github.com/portainer/portainer/api/archive"
 	"github.com/portainer/portainer/api/crypto"
-	"github.com/portainer/portainer/api/filesystem"
 	"github.com/portainer/portainer/api/http/offlinegate"
 )
 
@@ -60,7 +59,7 @@ func extractArchive(r io.Reader, destinationDirPath string) error {
 
 func restoreFiles(srcDir string, destinationDir string) error {
 	for _, filename := range filesToRestore {
-		err := filesystem.CopyPath(filepath.Join(srcDir, filename), destinationDir)
+		err := copyPath(filepath.Join(srcDir, filename), destinationDir)
 		if err != nil {
 			return err
 		}

api/bolt/backup.go

@@ -0,0 +1,181 @@
+package bolt
+
+import (
+	"fmt"
+	"io/ioutil"
+	"os"
+	"path"
+	"time"
+
+	portainer "github.com/portainer/portainer/api"
+	plog "github.com/portainer/portainer/api/bolt/log"
+)
+
+var backupDefaults = struct {
+	backupDir        string
+	editions         []string
+	databaseFileName string
+}{
+	"backups",
+	[]string{"CE", "BE", "EE"},
+	databaseFileName,
+}
+
+var backupLog = plog.NewScopedLog("bolt, backup")
+
+//
+// Backup Helpers
+//
+
+// createBackupFolders create initial folders for backups
+func (store *Store) createBackupFolders() {
+	for _, e := range backupDefaults.editions {
+
+		p := path.Join(store.path, backupDefaults.backupDir, e)
+
+		if exists, _ := store.fileService.FileExists(p); !exists {
+			err := os.MkdirAll(p, 0700)
+			if err != nil {
+				backupLog.Error("Error while creating backup folders", err)
+			}
+		}
+	}
+}
+
+func (store *Store) databasePath() string {
+	return path.Join(store.path, databaseFileName)
+}
+
+func (store *Store) editionBackupDir(edition portainer.SoftwareEdition) string {
+	return path.Join(store.path, backupDefaults.backupDir, edition.GetEditionLabel())
+}
+
+func (store *Store) copyDBFile(from string, to string) error {
+	backupLog.Info(fmt.Sprintf("Copying db file from %s to %s", from, to))
+	err := store.fileService.Copy(from, to, true)
+	if err != nil {
+		backupLog.Error("Failed", err)
+	}
+	return err
+}
+
+// BackupOptions provide a helper to inject backup options
+type BackupOptions struct {
+	Edition        portainer.SoftwareEdition
+	Version        int
+	BackupDir      string
+	BackupFileName string
+	BackupPath     string
+}
+
+func (store *Store) setupOptions(options *BackupOptions) *BackupOptions {
+	if options == nil {
+		options = &BackupOptions{}
+	}
+	if options.Edition == 0 {
+		options.Edition = store.edition()
+	}
+	if options.Version == 0 {
+		options.Version, _ = store.version()
+	}
+	if options.BackupDir == "" {
+		options.BackupDir = store.editionBackupDir(options.Edition)
+	}
+	if options.BackupFileName == "" {
+		options.BackupFileName = fmt.Sprintf("%s.%s.%s", backupDefaults.databaseFileName, fmt.Sprintf("%03d", options.Version), time.Now().Format("20060102150405"))
+	}
+	if options.BackupPath == "" {
+		options.BackupPath = path.Join(options.BackupDir, options.BackupFileName)
+	}
+	return options
+}
+
+func (store *Store) listEditionBackups(edition portainer.SoftwareEdition) ([]string, error) {
+	var fileNames = []string{}
+
+	files, err := ioutil.ReadDir(store.editionBackupDir(edition))
+
+	if err != nil {
+		backupLog.Error("Error while retrieving backup files", err)
+		return fileNames, err
+	}
+
+	for _, f := range files {
+		fileNames = append(fileNames, f.Name())
+	}
+
+	return fileNames, nil
+}
+
+func (store *Store) lastestEditionBackup() (string, error) {
+	edition := store.edition()
+
+	files, err := store.listEditionBackups(edition)
+	if err != nil {
+		backupLog.Error("Error while retrieving backup files", err)
+		return "", err
+	}
+
+	if len(files) == 0 {
+		return "", nil
+	}
+
+	return files[len(files)-1], nil
+}
+
+// BackupWithOptions backup current database with options
+func (store *Store) BackupWithOptions(options *BackupOptions) (string, error) {
+	backupLog.Info("creating db backup")
+	store.createBackupFolders()
+
+	options = store.setupOptions(options)
+
+	return options.BackupPath, store.copyDBFile(store.databasePath(), options.BackupPath)
+}
+
+// Backup current database with default options
+func (store *Store) Backup() (string, error) {
+	return store.BackupWithOptions(nil)
+}
+
+// RestoreWithOptions previously saved backup for the current Edition  with options
+// Restore strategies:
+// - default: restore latest from current edition
+// - restore a specific
+func (store *Store) RestoreWithOptions(options *BackupOptions) error {
+
+	// Check if backup file exist before restoring
+
+	options = store.setupOptions(options)
+
+	_, err := os.Stat(options.BackupPath)
+
+	if os.IsNotExist(err) {
+		backupLog.Error(fmt.Sprintf("Backup file to restore does not exist %s", options.BackupPath), err)
+		return err
+	}
+
+	err = store.Close()
+	if err != nil {
+		backupLog.Error("Error while closing store before restore", err)
+		return err
+	}
+	backupLog.Info("Restoring db backup")
+	err = store.copyDBFile(options.BackupPath, store.databasePath())
+	if err != nil {
+		return err
+	}
+	return store.Open()
+}
+
+// Restore previously saved backup for the current Edition  with default options
+func (store *Store) Restore() error {
+	var options = &BackupOptions{}
+	var err error
+	options.BackupFileName, err = store.lastestEditionBackup()
+	if err != nil {
+		return err
+	}
+	return store.RestoreWithOptions(options)
+
+}

api/bolt/backup_test.go

@@ -0,0 +1,118 @@
+package bolt
+
+import (
+	"fmt"
+	"log"
+	"testing"
+
+	portainer "github.com/portainer/portainer/api"
+)
+
+func TestCreateBackupFolders(t *testing.T) {
+	store := NewTestStore(portainer.PortainerEE, portainer.DBVersionEE, false)
+	if exists, _ := store.fileService.FileExists("tmp/backups"); exists {
+		t.Error("Expect backups folder to not exist")
+	}
+	store.createBackupFolders()
+	if exists, _ := store.fileService.FileExists("tmp/backups"); !exists {
+		t.Error("Expect backups folder to exist")
+	}
+	store.createBackupFolders()
+	store.Close()
+	teardown()
+}
+
+func TestStoreCreation(t *testing.T) {
+	store := NewTestStore(portainer.PortainerEE, portainer.DBVersionEE, false)
+	if store == nil {
+		t.Error("Expect to create a store")
+	}
+
+	if store.edition() != portainer.PortainerEE {
+		t.Error("Expect to get EE Edition")
+	}
+
+	version, err := store.version()
+	if err != nil {
+		log.Fatal(err)
+	}
+
+	if version != portainer.DBVersionEE {
+		t.Error("Expect to get EE DBVersion")
+	}
+
+	store.Close()
+	teardown()
+}
+
+func TestBackup(t *testing.T) {
+
+	tests := []struct {
+		edition portainer.SoftwareEdition
+		version int
+	}{
+		{edition: portainer.PortainerCE, version: portainer.DBVersion},
+		{edition: portainer.PortainerEE, version: portainer.DBVersionEE},
+	}
+
+	for _, tc := range tests {
+		backupFileName := fmt.Sprintf("tmp/backups/%s/portainer.db.%03d.*", tc.edition.GetEditionLabel(), tc.version)
+		t.Run(fmt.Sprintf("Backup should create %s", backupFileName), func(t *testing.T) {
+			store := NewTestStore(tc.edition, tc.version, false)
+			store.Backup()
+
+			if !isFileExist(backupFileName) {
+				t.Errorf("Expect backup file to be created %s", backupFileName)
+			}
+			store.Close()
+		})
+	}
+	t.Run("BackupWithOption should create a name specific backup", func(t *testing.T) {
+		edition := portainer.PortainerCE
+		version := portainer.DBVersion
+		store := NewTestStore(edition, version, false)
+		store.BackupWithOptions(&BackupOptions{
+			BackupFileName: beforePortainerUpgradeToEEBackup,
+			Edition:        portainer.PortainerCE,
+		})
+		backupFileName := fmt.Sprintf("tmp/backups/%s/%s", edition.GetEditionLabel(), beforePortainerUpgradeToEEBackup)
+		if !isFileExist(backupFileName) {
+			t.Errorf("Expect backup file to be created %s", backupFileName)
+		}
+		store.Close()
+	})
+
+	teardown()
+}
+
+// TODO restore / backup failed test cases
+func TestRestore(t *testing.T) {
+
+	editions := []portainer.SoftwareEdition{portainer.PortainerCE, portainer.PortainerEE}
+	var currentVersion = 0
+
+	for i, e := range editions {
+		editionLabel := e.GetEditionLabel()
+		currentVersion = 10 ^ i + 1
+		store := NewTestStore(e, currentVersion, false)
+		t.Run(fmt.Sprintf("Basic Restore for %s", editionLabel), func(t *testing.T) {
+			store.Backup()
+			updateVersion(store, currentVersion+1)
+			testVersion(store, currentVersion+1, t)
+			store.Restore()
+			testVersion(store, currentVersion, t)
+		})
+		t.Run(fmt.Sprintf("Basic Restore After Multiple Backup for %s", editionLabel), func(t *testing.T) {
+			currentVersion = currentVersion + 5
+			updateVersion(store, currentVersion)
+			store.Backup()
+			updateVersion(store, currentVersion+2)
+			testVersion(store, currentVersion+2, t)
+			store.Restore()
+			testVersion(store, currentVersion, t)
+		})
+		store.Close()
+	}
+
+	teardown()
+}

api/bolt/datastore.go

@@ -2,11 +2,13 @@ package bolt
 
 import (
 	"io"
-	"log"
 	"path"
 	"time"
 
-	"github.com/portainer/portainer/api/bolt/helmuserrepository"
+	"github.com/portainer/portainer/api/bolt/errors"
+	"github.com/portainer/portainer/api/bolt/internal"
+	"github.com/portainer/portainer/api/bolt/license"
+	"github.com/portainer/portainer/api/bolt/s3backup"
 
 	"github.com/boltdb/bolt"
 	portainer "github.com/portainer/portainer/api"
@@ -18,16 +20,12 @@ import (
 	"github.com/portainer/portainer/api/bolt/endpoint"
 	"github.com/portainer/portainer/api/bolt/endpointgroup"
 	"github.com/portainer/portainer/api/bolt/endpointrelation"
-	"github.com/portainer/portainer/api/bolt/errors"
 	"github.com/portainer/portainer/api/bolt/extension"
-	"github.com/portainer/portainer/api/bolt/internal"
-	"github.com/portainer/portainer/api/bolt/migrator"
 	"github.com/portainer/portainer/api/bolt/registry"
 	"github.com/portainer/portainer/api/bolt/resourcecontrol"
 	"github.com/portainer/portainer/api/bolt/role"
 	"github.com/portainer/portainer/api/bolt/schedule"
 	"github.com/portainer/portainer/api/bolt/settings"
-	"github.com/portainer/portainer/api/bolt/ssl"
 	"github.com/portainer/portainer/api/bolt/stack"
 	"github.com/portainer/portainer/api/bolt/tag"
 	"github.com/portainer/portainer/api/bolt/team"
@@ -36,44 +34,51 @@ import (
 	"github.com/portainer/portainer/api/bolt/user"
 	"github.com/portainer/portainer/api/bolt/version"
 	"github.com/portainer/portainer/api/bolt/webhook"
-	"github.com/portainer/portainer/api/internal/authorization"
 )
 
-const (
+var (
 	databaseFileName = "portainer.db"
 )
 
 // Store defines the implementation of portainer.DataStore using
 // BoltDB as the storage system.
 type Store struct {
-	path                      string
-	connection                *internal.DbConnection
-	isNew                     bool
-	fileService               portainer.FileService
-	CustomTemplateService     *customtemplate.Service
-	DockerHubService          *dockerhub.Service
-	EdgeGroupService          *edgegroup.Service
-	EdgeJobService            *edgejob.Service
-	EdgeStackService          *edgestack.Service
-	EndpointGroupService      *endpointgroup.Service
-	EndpointService           *endpoint.Service
-	EndpointRelationService   *endpointrelation.Service
-	ExtensionService          *extension.Service
-	HelmUserRepositoryService *helmuserrepository.Service
-	RegistryService           *registry.Service
-	ResourceControlService    *resourcecontrol.Service
-	RoleService               *role.Service
-	ScheduleService           *schedule.Service
-	SettingsService           *settings.Service
-	SSLSettingsService        *ssl.Service
-	StackService              *stack.Service
-	TagService                *tag.Service
-	TeamMembershipService     *teammembership.Service
-	TeamService               *team.Service
-	TunnelServerService       *tunnelserver.Service
-	UserService               *user.Service
-	VersionService            *version.Service
-	WebhookService            *webhook.Service
+	path                    string
+	connection              *internal.DbConnection
+	isNew                   bool
+	fileService             portainer.FileService
+	CustomTemplateService   *customtemplate.Service
+	DockerHubService        *dockerhub.Service
+	EdgeGroupService        *edgegroup.Service
+	EdgeJobService          *edgejob.Service
+	EdgeStackService        *edgestack.Service
+	EndpointGroupService    *endpointgroup.Service
+	EndpointService         *endpoint.Service
+	EndpointRelationService *endpointrelation.Service
+	ExtensionService        *extension.Service
+	LicenseService          *license.Service
+	RegistryService         *registry.Service
+	ResourceControlService  *resourcecontrol.Service
+	RoleService             *role.Service
+	S3BackupService         *s3backup.Service
+	ScheduleService         *schedule.Service
+	SettingsService         *settings.Service
+	StackService            *stack.Service
+	TagService              *tag.Service
+	TeamMembershipService   *teammembership.Service
+	TeamService             *team.Service
+	TunnelServerService     *tunnelserver.Service
+	UserService             *user.Service
+	VersionService          *version.Service
+	WebhookService          *webhook.Service
+}
+
+func (store *Store) version() (int, error) {
+	version, err := store.VersionService.DBVersion()
+	if err == errors.ErrObjectNotFound {
+		version = 0
+	}
+	return version, err
 }
 
 func (store *Store) edition() portainer.SoftwareEdition {
@@ -119,7 +124,6 @@ func (store *Store) Open() error {
 }
 
 // Close closes the BoltDB database.
-// Safe to being called multiple times.
 func (store *Store) Close() error {
 	if store.connection.DB != nil {
 		return store.connection.Close()
@@ -133,64 +137,6 @@ func (store *Store) IsNew() bool {
 	return store.isNew
 }
 
-// CheckCurrentEdition checks if current edition is community edition
-func (store *Store) CheckCurrentEdition() error {
-	if store.edition() != portainer.PortainerCE {
-		return errors.ErrWrongDBEdition
-	}
-	return nil
-}
-
-// MigrateData automatically migrate the data based on the DBVersion.
-// This process is only triggered on an existing database, not if the database was just created.
-// if force is true, then migrate regardless.
-func (store *Store) MigrateData(force bool) error {
-	if store.isNew && !force {
-		return store.VersionService.StoreDBVersion(portainer.DBVersion)
-	}
-
-	version, err := store.VersionService.DBVersion()
-	if err == errors.ErrObjectNotFound {
-		version = 0
-	} else if err != nil {
-		return err
-	}
-
-	if version < portainer.DBVersion {
-		migratorParams := &migrator.Parameters{
-			DB:                      store.connection.DB,
-			DatabaseVersion:         version,
-			EndpointGroupService:    store.EndpointGroupService,
-			EndpointService:         store.EndpointService,
-			EndpointRelationService: store.EndpointRelationService,
-			ExtensionService:        store.ExtensionService,
-			RegistryService:         store.RegistryService,
-			ResourceControlService:  store.ResourceControlService,
-			RoleService:             store.RoleService,
-			ScheduleService:         store.ScheduleService,
-			SettingsService:         store.SettingsService,
-			StackService:            store.StackService,
-			TagService:              store.TagService,
-			TeamMembershipService:   store.TeamMembershipService,
-			UserService:             store.UserService,
-			VersionService:          store.VersionService,
-			FileService:             store.fileService,
-			DockerhubService:        store.DockerHubService,
-			AuthorizationService:    authorization.NewService(store),
-		}
-		migrator := migrator.NewMigrator(migratorParams)
-
-		log.Printf("Migrating database from version %v to %v.\n", version, portainer.DBVersion)
-		err = migrator.Migrate()
-		if err != nil {
-			log.Printf("An error occurred during database migration: %s\n", err)
-			return err
-		}
-	}
-
-	return nil
-}
-
 // BackupTo backs up db to a provided writer.
 // It does hot backup and doesn't block other database reads and writes
 func (store *Store) BackupTo(w io.Writer) error {

api/bolt/edgejob/edgejob.go

@@ -95,7 +95,7 @@ func (service *Service) DeleteEdgeJob(ID portainer.EdgeJobID) error {
 	return internal.DeleteObject(service.connection, BucketName, identifier)
 }
 
-// GetNextIdentifier returns the next identifier for an environment(endpoint).
+// GetNextIdentifier returns the next identifier for an endpoint.
 func (service *Service) GetNextIdentifier() int {
 	return internal.GetNextIdentifier(service.connection, BucketName)
 }

api/bolt/edgestack/edgestack.go

@@ -95,7 +95,7 @@ func (service *Service) DeleteEdgeStack(ID portainer.EdgeStackID) error {
 	return internal.DeleteObject(service.connection, BucketName, identifier)
 }
 
-// GetNextIdentifier returns the next identifier for an environment(endpoint).
+// GetNextIdentifier returns the next identifier for an endpoint.
 func (service *Service) GetNextIdentifier() int {
 	return internal.GetNextIdentifier(service.connection, BucketName)
 }

api/bolt/endpoint/endpoint.go

@@ -11,7 +11,7 @@ const (
 	BucketName = "endpoints"
 )
 
-// Service represents a service for managing environment(endpoint) data.
+// Service represents a service for managing endpoint data.
 type Service struct {
 	connection *internal.DbConnection
 }
@@ -28,7 +28,7 @@ func NewService(connection *internal.DbConnection) (*Service, error) {
 	}, nil
 }
 
-// Endpoint returns an environment(endpoint) by ID.
+// Endpoint returns an endpoint by ID.
 func (service *Service) Endpoint(ID portainer.EndpointID) (*portainer.Endpoint, error) {
 	var endpoint portainer.Endpoint
 	identifier := internal.Itob(int(ID))
@@ -41,19 +41,19 @@ func (service *Service) Endpoint(ID portainer.EndpointID) (*portainer.Endpoint,
 	return &endpoint, nil
 }
 
-// UpdateEndpoint updates an environment(endpoint).
+// UpdateEndpoint updates an endpoint.
 func (service *Service) UpdateEndpoint(ID portainer.EndpointID, endpoint *portainer.Endpoint) error {
 	identifier := internal.Itob(int(ID))
 	return internal.UpdateObject(service.connection, BucketName, identifier, endpoint)
 }
 
-// DeleteEndpoint deletes an environment(endpoint).
+// DeleteEndpoint deletes an endpoint.
 func (service *Service) DeleteEndpoint(ID portainer.EndpointID) error {
 	identifier := internal.Itob(int(ID))
 	return internal.DeleteObject(service.connection, BucketName, identifier)
 }
 
-// Endpoints return an array containing all the environments(endpoints).
+// Endpoints return an array containing all the endpoints.
 func (service *Service) Endpoints() ([]portainer.Endpoint, error) {
 	var endpoints = make([]portainer.Endpoint, 0)
 
@@ -76,12 +76,12 @@ func (service *Service) Endpoints() ([]portainer.Endpoint, error) {
 	return endpoints, err
 }
 
-// CreateEndpoint assign an ID to a new environment(endpoint) and saves it.
+// CreateEndpoint assign an ID to a new endpoint and saves it.
 func (service *Service) CreateEndpoint(endpoint *portainer.Endpoint) error {
 	return service.connection.Update(func(tx *bolt.Tx) error {
 		bucket := tx.Bucket([]byte(BucketName))
 
-		// We manually manage sequences for environments(endpoints)
+		// We manually manage sequences for endpoints
 		err := bucket.SetSequence(uint64(endpoint.ID))
 		if err != nil {
 			return err
@@ -96,12 +96,12 @@ func (service *Service) CreateEndpoint(endpoint *portainer.Endpoint) error {
 	})
 }
 
-// GetNextIdentifier returns the next identifier for an environment(endpoint).
+// GetNextIdentifier returns the next identifier for an endpoint.
 func (service *Service) GetNextIdentifier() int {
 	return internal.GetNextIdentifier(service.connection, BucketName)
 }
 
-// Synchronize creates, updates and deletes environments(endpoints) inside a single transaction.
+// Synchronize creates, updates and deletes endpoints inside a single transaction.
 func (service *Service) Synchronize(toCreate, toUpdate, toDelete []*portainer.Endpoint) error {
 	return service.connection.Update(func(tx *bolt.Tx) error {
 		bucket := tx.Bucket([]byte(BucketName))

api/bolt/endpointgroup/endpointgroup.go

@@ -12,7 +12,7 @@ const (
 	BucketName = "endpoint_groups"
 )
 
-// Service represents a service for managing environment(endpoint) data.
+// Service represents a service for managing endpoint data.
 type Service struct {
 	connection *internal.DbConnection
 }
@@ -29,7 +29,7 @@ func NewService(connection *internal.DbConnection) (*Service, error) {
 	}, nil
 }
 
-// EndpointGroup returns an environment(endpoint) group by ID.
+// EndpointGroup returns an endpoint group by ID.
 func (service *Service) EndpointGroup(ID portainer.EndpointGroupID) (*portainer.EndpointGroup, error) {
 	var endpointGroup portainer.EndpointGroup
 	identifier := internal.Itob(int(ID))
@@ -42,19 +42,19 @@ func (service *Service) EndpointGroup(ID portainer.EndpointGroupID) (*portainer.
 	return &endpointGroup, nil
 }
 
-// UpdateEndpointGroup updates an environment(endpoint) group.
+// UpdateEndpointGroup updates an endpoint group.
 func (service *Service) UpdateEndpointGroup(ID portainer.EndpointGroupID, endpointGroup *portainer.EndpointGroup) error {
 	identifier := internal.Itob(int(ID))
 	return internal.UpdateObject(service.connection, BucketName, identifier, endpointGroup)
 }
 
-// DeleteEndpointGroup deletes an environment(endpoint) group.
+// DeleteEndpointGroup deletes an endpoint group.
 func (service *Service) DeleteEndpointGroup(ID portainer.EndpointGroupID) error {
 	identifier := internal.Itob(int(ID))
 	return internal.DeleteObject(service.connection, BucketName, identifier)
 }
 
-// EndpointGroups return an array containing all the environment(endpoint) groups.
+// EndpointGroups return an array containing all the endpoint groups.
 func (service *Service) EndpointGroups() ([]portainer.EndpointGroup, error) {
 	var endpointGroups = make([]portainer.EndpointGroup, 0)
 
@@ -77,7 +77,7 @@ func (service *Service) EndpointGroups() ([]portainer.EndpointGroup, error) {
 	return endpointGroups, err
 }
 
-// CreateEndpointGroup assign an ID to a new environment(endpoint) group and saves it.
+// CreateEndpointGroup assign an ID to a new endpoint group and saves it.
 func (service *Service) CreateEndpointGroup(endpointGroup *portainer.EndpointGroup) error {
 	return service.connection.Update(func(tx *bolt.Tx) error {
 		bucket := tx.Bucket([]byte(BucketName))

api/bolt/endpointrelation/endpointrelation.go

@@ -11,7 +11,7 @@ const (
 	BucketName = "endpoint_relations"
 )
 
-// Service represents a service for managing environment(endpoint) relation data.
+// Service represents a service for managing endpoint relation data.
 type Service struct {
 	connection *internal.DbConnection
 }
@@ -28,7 +28,7 @@ func NewService(connection *internal.DbConnection) (*Service, error) {
 	}, nil
 }
 
-// EndpointRelation returns a Environment(Endpoint) relation object by EndpointID
+// EndpointRelation returns a Endpoint relation object by EndpointID
 func (service *Service) EndpointRelation(endpointID portainer.EndpointID) (*portainer.EndpointRelation, error) {
 	var endpointRelation portainer.EndpointRelation
 	identifier := internal.Itob(int(endpointID))
@@ -55,13 +55,13 @@ func (service *Service) CreateEndpointRelation(endpointRelation *portainer.Endpo
 	})
 }
 
-// UpdateEndpointRelation updates an Environment(Endpoint) relation object
+// UpdateEndpointRelation updates an Endpoint relation object
 func (service *Service) UpdateEndpointRelation(EndpointID portainer.EndpointID, endpointRelation *portainer.EndpointRelation) error {
 	identifier := internal.Itob(int(EndpointID))
 	return internal.UpdateObject(service.connection, BucketName, identifier, endpointRelation)
 }
 
-// DeleteEndpointRelation deletes an Environment(Endpoint) relation object
+// DeleteEndpointRelation deletes an Endpoint relation object
 func (service *Service) DeleteEndpointRelation(EndpointID portainer.EndpointID) error {
 	identifier := internal.Itob(int(EndpointID))
 	return internal.DeleteObject(service.connection, BucketName, identifier)

api/bolt/errors/errors.go

@@ -4,5 +4,5 @@ import "errors"
 
 var (
 	ErrObjectNotFound = errors.New("Object not found inside the database")
-	ErrWrongDBEdition = errors.New("The Portainer database is set for Portainer Business Edition, please follow the instructions in our documentation to downgrade it: https://documentation.portainer.io/v2.0-be/downgrade/be-to-ce/")
+	ErrMigrationToCE  = errors.New("DB is already on CE edition")
 )

api/bolt/extension/extension.go

@@ -12,7 +12,7 @@ const (
 	BucketName = "extension"
 )
 
-// Service represents a service for managing environment(endpoint) data.
+// Service represents a service for managing endpoint data.
 type Service struct {
 	connection *internal.DbConnection
 }

api/bolt/helmuserrepository/helmuserrepository.go

@@ -1,73 +0,0 @@
-package helmuserrepository
-
-import (
-	portainer "github.com/portainer/portainer/api"
-	"github.com/portainer/portainer/api/bolt/internal"
-
-	"github.com/boltdb/bolt"
-)
-
-const (
-	// BucketName represents the name of the bucket where this service stores data.
-	BucketName = "helm_user_repository"
-)
-
-// Service represents a service for managing environment(endpoint) data.
-type Service struct {
-	connection *internal.DbConnection
-}
-
-// NewService creates a new instance of a service.
-func NewService(connection *internal.DbConnection) (*Service, error) {
-	err := internal.CreateBucket(connection, BucketName)
-	if err != nil {
-		return nil, err
-	}
-
-	return &Service{
-		connection: connection,
-	}, nil
-}
-
-// HelmUserRepositoryByUserID return an array containing all the HelmUserRepository objects where the specified userID is present.
-func (service *Service) HelmUserRepositoryByUserID(userID portainer.UserID) ([]portainer.HelmUserRepository, error) {
-	var result = make([]portainer.HelmUserRepository, 0)
-
-	err := service.connection.View(func(tx *bolt.Tx) error {
-		bucket := tx.Bucket([]byte(BucketName))
-
-		cursor := bucket.Cursor()
-		for k, v := cursor.First(); k != nil; k, v = cursor.Next() {
-			var record portainer.HelmUserRepository
-			err := internal.UnmarshalObject(v, &record)
-			if err != nil {
-				return err
-			}
-
-			if record.UserID == userID {
-				result = append(result, record)
-			}
-		}
-
-		return nil
-	})
-
-	return result, err
-}
-
-// CreateHelmUserRepository creates a new HelmUserRepository object.
-func (service *Service) CreateHelmUserRepository(record *portainer.HelmUserRepository) error {
-	return service.connection.Update(func(tx *bolt.Tx) error {
-		bucket := tx.Bucket([]byte(BucketName))
-
-		id, _ := bucket.NextSequence()
-		record.ID = portainer.HelmUserRepositoryID(id)
-
-		data, err := internal.MarshalObject(record)
-		if err != nil {
-			return err
-		}
-
-		return bucket.Put(internal.Itob(int(record.ID)), data)
-	})
-}

api/bolt/helpers_test.go

@@ -0,0 +1,115 @@
+package bolt
+
+import (
+	"fmt"
+	"log"
+	"math/rand"
+	"os"
+	"path"
+	"path/filepath"
+	"testing"
+	"time"
+
+	portainer "github.com/portainer/portainer/api"
+	"github.com/portainer/portainer/api/filesystem"
+)
+
+var (
+	dataStorePath  string
+	testBackupPath string
+)
+
+func init() {
+	rand.Seed(time.Now().UnixNano())
+	databaseFileName = fmt.Sprintf("portainer-%08d.db", rand.Intn(100000000))
+
+	pwd, err := os.Getwd()
+	if err != nil {
+		log.Println(err)
+	}
+
+	dataStorePath = path.Join(pwd, "tmp")
+	testBackupPath = path.Join(dataStorePath, "backups")
+
+	teardown()
+}
+
+func NewTestStore(edition portainer.SoftwareEdition, version int, init bool) *Store {
+	fileService, err := filesystem.NewService(dataStorePath, "")
+	if err != nil {
+		log.Fatal(err)
+	}
+
+	store, err := NewStore(dataStorePath, fileService)
+	if err != nil {
+		log.Fatal(err)
+	}
+
+	err = store.Open()
+	if err != nil {
+		log.Fatal(err)
+	}
+
+	if init {
+		err = store.Init()
+		if err != nil {
+			log.Fatal(err)
+		}
+	}
+
+	err = store.VersionService.StoreEdition(edition)
+	if err != nil {
+		log.Fatal(err)
+	}
+
+	err = store.VersionService.StoreDBVersion(version)
+	if err != nil {
+		log.Fatal(err)
+	}
+
+	return store
+}
+
+func teardown() {
+	err := os.RemoveAll(testBackupPath)
+	if err != nil {
+		log.Fatalln(err)
+	}
+
+	files, err := filepath.Glob(path.Join(dataStorePath, "portainer-*.*"))
+	if err != nil {
+		log.Fatalln(err)
+	}
+	for _, f := range files {
+		if err := os.Remove(f); err != nil {
+			log.Fatalln(err)
+		}
+	}
+}
+
+func isFileExist(path string) bool {
+	matches, err := filepath.Glob(path)
+	if err != nil {
+		return false
+	}
+	return len(matches) > 0
+}
+
+func updateVersion(store *Store, v int) {
+	err := store.VersionService.StoreDBVersion(v)
+	if err != nil {
+		log.Fatal(err)
+	}
+}
+
+func testVersion(store *Store, versionWant int, t *testing.T) {
+	if v, _ := store.version(); v != versionWant {
+		t.Errorf("Expect store version to be %d but was %d", versionWant, v)
+	}
+}
+
+func testEdition(store *Store, editionWant portainer.SoftwareEdition, t *testing.T) {
+	if e := store.edition(); e != editionWant {
+		t.Errorf("Expect store edition to be %s but was %s", editionWant.GetEditionLabel(), e.GetEditionLabel())
+	}
+}

api/bolt/init.go

@@ -33,6 +33,7 @@ func (store *Store) Init() error {
 				AnonymousMode:   true,
 				AutoCreateUsers: true,
 				TLSConfig:       portainer.TLSConfiguration{},
+				URLs:            []string{},
 				SearchSettings: []portainer.LDAPSearchSettings{
 					portainer.LDAPSearchSettings{},
 				},
@@ -40,13 +41,14 @@ func (store *Store) Init() error {
 					portainer.LDAPGroupSearchSettings{},
 				},
 			},
-			OAuthSettings: portainer.OAuthSettings{},
-
+			OAuthSettings: portainer.OAuthSettings{
+				TeamMemberships: portainer.TeamMemberships{
+					OAuthClaimMappings: make([]portainer.OAuthClaimMappings, 0),
+				},
+			},
 			EdgeAgentCheckinInterval: portainer.DefaultEdgeAgentCheckinIntervalInSeconds,
 			TemplatesURL:             portainer.DefaultTemplatesURL,
-			HelmRepositoryURL:        portainer.DefaultHelmRepositoryURL,
 			UserSessionTimeout:       portainer.DefaultUserSessionTimeout,
-			KubeconfigExpiry:         portainer.DefaultKubeconfigExpiry,
 		}
 
 		err = store.SettingsService.UpdateSettings(defaultSettings)
@@ -57,20 +59,20 @@ func (store *Store) Init() error {
 		return err
 	}
 
-	_, err = store.SSLSettings().Settings()
-	if err != nil {
-		if err != errors.ErrObjectNotFound {
-			return err
+	_, err = store.DockerHubService.DockerHub()
+	if err == errors.ErrObjectNotFound {
+		defaultDockerHub := &portainer.DockerHub{
+			Authentication: false,
+			Username:       "",
+			Password:       "",
 		}
 
-		defaultSSLSettings := &portainer.SSLSettings{
-			HTTPEnabled: true,
-		}
-
-		err = store.SSLSettings().UpdateSettings(defaultSSLSettings)
+		err := store.DockerHubService.UpdateDockerHub(defaultDockerHub)
 		if err != nil {
 			return err
 		}
+	} else if err != nil {
+		return err
 	}
 
 	groups, err := store.EndpointGroupService.EndpointGroups()
@@ -81,7 +83,7 @@ func (store *Store) Init() error {
 	if len(groups) == 0 {
 		unassignedGroup := &portainer.EndpointGroup{
 			Name:               "Unassigned",
-			Description:        "Unassigned environments",
+			Description:        "Unassigned endpoints",
 			Labels:             []portainer.Pair{},
 			UserAccessPolicies: portainer.UserAccessPolicies{},
 			TeamAccessPolicies: portainer.TeamAccessPolicies{},
@@ -94,5 +96,17 @@ func (store *Store) Init() error {
 		}
 	}
 
+	roles, err := store.RoleService.Roles()
+	if err != nil {
+		return err
+	}
+
+	if len(roles) == 0 {
+		err := store.RoleService.CreateOrUpdatePredefinedRoles()
+		if err != nil {
+			return err
+		}
+	}
+
 	return nil
 }

api/bolt/internal/json.go

@@ -17,7 +17,7 @@ func UnmarshalObject(data []byte, object interface{}) error {
 }
 
 // UnmarshalObjectWithJsoniter decodes an object from binary data
-// using the jsoniter library. It is mainly used to accelerate environment(endpoint)
+// using the jsoniter library. It is mainly used to accelerate endpoint
 // decoding at the moment.
 func UnmarshalObjectWithJsoniter(data []byte, object interface{}) error {
 	var jsoni = jsoniter.ConfigCompatibleWithStandardLibrary

api/bolt/license/license.go

@@ -0,0 +1,92 @@
+package license
+
+import (
+	"github.com/portainer/liblicense"
+	"github.com/portainer/portainer/api/bolt/internal"
+
+	"github.com/boltdb/bolt"
+)
+
+const (
+	// BucketName represents the name of the bucket where this service stores data.
+	BucketName = "license"
+)
+
+// Service represents a service for managing endpoint data.
+type Service struct {
+	connection *internal.DbConnection
+}
+
+// NewService creates a new instance of a service.
+func NewService(connection *internal.DbConnection) (*Service, error) {
+	err := internal.CreateBucket(connection, BucketName)
+	if err != nil {
+		return nil, err
+	}
+
+	return &Service{
+		connection: connection,
+	}, nil
+}
+
+// License returns a license by licenseKey
+func (service *Service) License(licenseKey string) (*liblicense.PortainerLicense, error) {
+	var license liblicense.PortainerLicense
+	identifier := []byte(licenseKey)
+
+	err := internal.GetObject(service.connection, BucketName, identifier, &license)
+	if err != nil {
+		return nil, err
+	}
+
+	return &license, nil
+}
+
+// Licenses return an array containing all the licenses.
+func (service *Service) Licenses() ([]liblicense.PortainerLicense, error) {
+	var licenses = make([]liblicense.PortainerLicense, 0)
+
+	err := service.connection.View(func(tx *bolt.Tx) error {
+		bucket := tx.Bucket([]byte(BucketName))
+
+		cursor := bucket.Cursor()
+		for k, v := cursor.First(); k != nil; k, v = cursor.Next() {
+			var license liblicense.PortainerLicense
+			err := internal.UnmarshalObject(v, &license)
+			if err != nil {
+				return err
+			}
+			licenses = append(licenses, license)
+		}
+
+		return nil
+	})
+
+	return licenses, err
+}
+
+// AddLicense persists a license inside the database.
+func (service *Service) AddLicense(licenseKey string, license *liblicense.PortainerLicense) error {
+	return service.connection.Update(func(tx *bolt.Tx) error {
+		bucket := tx.Bucket([]byte(BucketName))
+
+		data, err := internal.MarshalObject(license)
+		if err != nil {
+			return err
+		}
+
+		return bucket.Put([]byte(licenseKey), data)
+	})
+}
+
+// UpdateLicense updates a license.
+func (service *Service) UpdateLicense(licenseKey string, license *liblicense.PortainerLicense) error {
+	identifier := []byte(licenseKey)
+	return internal.UpdateObject(service.connection, BucketName, identifier, license)
+}
+
+// DeleteLicense deletes a License.
+func (service *Service) DeleteLicense(licenseKey string) error {
+	identifier := []byte(licenseKey)
+	return internal.DeleteObject(service.connection, BucketName, identifier)
+}

api/bolt/migrate_data.go

@@ -0,0 +1,207 @@
+package bolt
+
+import (
+	"fmt"
+
+	portainer "github.com/portainer/portainer/api"
+	errors "github.com/portainer/portainer/api/bolt/errors"
+	plog "github.com/portainer/portainer/api/bolt/log"
+	"github.com/portainer/portainer/api/bolt/migrator"
+	"github.com/portainer/portainer/api/cli"
+	"github.com/portainer/portainer/api/internal/authorization"
+)
+
+const beforePortainerUpgradeToEEBackup = "portainer.db.before-EE-upgrade"
+
+var migrateLog = plog.NewScopedLog("bolt, migrate")
+
+// FailSafeMigrate backup and restore DB if migration fail
+func (store *Store) FailSafeMigrate(migrator *migrator.Migrator, version int) error {
+	defer func() {
+		if err := recover(); err != nil {
+			migrateLog.Info(fmt.Sprintf("Error during migration, recovering [%v]", err))
+			store.Restore()
+		}
+	}()
+	return migrator.Migrate(version)
+}
+
+// MigrateData automatically migrate the data based on the DBVersion.
+// This process is only triggered on an existing database, not if the database was just created.
+// if force is true, then migrate regardless.
+func (store *Store) MigrateData(force bool) error {
+	// 0 – if DB is new then we don't need to migrate any data and just set version and edition to latest EE
+	if store.isNew && !force {
+		err := store.VersionService.StoreDBVersion(portainer.DBVersionEE)
+		if err != nil {
+			return err
+		}
+
+		err = store.VersionService.StoreEdition(portainer.PortainerEE)
+		if err != nil {
+			return err
+		}
+
+		return nil
+	}
+
+	migrator, err := store.newMigrator()
+	if err != nil {
+		return err
+	}
+
+	if migrator.Edition() == portainer.PortainerCE {
+		// backup before migrating
+		store.BackupWithOptions(&BackupOptions{
+			BackupFileName: beforePortainerUpgradeToEEBackup,
+			Edition:        portainer.PortainerCE,
+		})
+
+		store.VersionService.StorePreviousDBVersion(migrator.Version())
+
+		// 1 – We need to migrate DB to latest CE version
+
+		if migrator.Version() < portainer.DBVersion {
+			store.Backup()
+			err = store.FailSafeMigrate(migrator, portainer.DBVersion)
+			if err != nil {
+				store.Restore()
+				migrateLog.Error("An error occurred while migrating CE database to latest version", err)
+				return err
+			}
+		}
+
+	}
+
+	if portainer.Edition == portainer.PortainerEE {
+		// 2 – if DB is CE Edition we need to upgrade settings to EE
+		if migrator.Edition() < portainer.PortainerEE {
+			err = migrator.UpgradeToEE()
+			if err != nil {
+				migrateLog.Error("An error occurred while upgrading database to EE", err)
+				store.RollbackFailedUpgradeToEE()
+				return err
+			}
+
+		}
+
+		// 3 – if DB is EE Edition we need to migrate to latest version of EE
+		if migrator.Edition() == portainer.PortainerEE && migrator.Version() < portainer.DBVersionEE {
+			store.Backup()
+			err = store.FailSafeMigrate(migrator, portainer.DBVersionEE)
+			if err != nil {
+				migrateLog.Error("An error occurred while migrating EE database to latest version", err)
+				store.Restore()
+				return err
+			}
+		}
+	}
+
+	return nil
+}
+
+// RollbackFailedUpgradeToEE down migrate to previous version
+func (store *Store) RollbackFailedUpgradeToEE() error {
+	return store.RestoreWithOptions(&BackupOptions{
+		BackupFileName: beforePortainerUpgradeToEEBackup,
+		Edition:        portainer.PortainerCE,
+	})
+}
+
+// RollbackToCE rollbacks the store to the current ce version
+func (store *Store) RollbackToCE() error {
+
+	migrator, err := store.newMigrator()
+	if err != nil {
+		return err
+	}
+
+	migrateLog.Info(fmt.Sprintf("Current Software Edition: %s", migrator.Edition().GetEditionLabel()))
+	migrateLog.Info(fmt.Sprintf("Current DB Version: %d", migrator.Version()))
+
+	if migrator.Edition() == portainer.PortainerCE {
+		return errors.ErrMigrationToCE
+	}
+
+	previousVersion, err := store.VersionService.PreviousDBVersion()
+	if err != nil {
+		migrateLog.Error("An Error occurred with retrieving previous DB version", err)
+		return err
+	}
+
+	confirmed, err := cli.Confirm(fmt.Sprintf("Are you sure you want to rollback your database to %d?", previousVersion))
+	if err != nil || !confirmed {
+		return err
+	}
+
+	if previousVersion < 25 {
+		migrator.DowngradeSettingsFrom25()
+	}
+
+	err = store.VersionService.StoreDBVersion(previousVersion)
+	if err != nil {
+		migrateLog.Error(fmt.Sprintf("An Error occurred with rolling back to CE Edition, DB Version %d", previousVersion), err)
+		return err
+	}
+
+	err = store.VersionService.StoreEdition(portainer.PortainerCE)
+	if err != nil {
+		migrateLog.Error(fmt.Sprintf("An Error occurred with rolling back to CE Edition, DB Version %d", previousVersion), err)
+		return err
+	}
+
+	migrateLog.Info(fmt.Sprintf("Rolled back to CE Edition, DB Version %d", previousVersion))
+	return nil
+}
+
+func (store *Store) newMigrator() (*migrator.Migrator, error) {
+
+	version, err := store.version()
+	if err != nil {
+		return nil, err
+	}
+
+	edition := store.edition()
+
+	params := &migrator.Parameters{
+		DB:              store.connection.DB,
+		DatabaseVersion: version,
+		CurrentEdition:  edition,
+
+		EndpointGroupService:    store.EndpointGroupService,
+		EndpointService:         store.EndpointService,
+		EndpointRelationService: store.EndpointRelationService,
+		ExtensionService:        store.ExtensionService,
+		RegistryService:         store.RegistryService,
+		ResourceControlService:  store.ResourceControlService,
+		RoleService:             store.RoleService,
+		ScheduleService:         store.ScheduleService,
+		SettingsService:         store.SettingsService,
+		StackService:            store.StackService,
+		TagService:              store.TagService,
+		TeamMembershipService:   store.TeamMembershipService,
+		UserService:             store.UserService,
+		VersionService:          store.VersionService,
+		FileService:             store.fileService,
+		AuthorizationService:    authorization.NewService(store),
+	}
+
+	return migrator.NewMigrator(params), nil
+}
+
+// RollbackVersion down migrate to previous version
+func (store *Store) RollbackVersion(version int) error {
+	// TODO
+	backupLog.NotImplemented("RollbackVersion")
+	return nil
+}
+
+// RollbackEdition downgrade to previous edition
+func (store *Store) RollbackEdition(edition portainer.SoftwareEdition) error {
+	// TODO
+	backupLog.NotImplemented("RollbackEdition")
+	// Change Edition
+	// Migrate Services
+	// Restore Latest
+	return nil
+}

api/bolt/migrate_data_test.go

@@ -0,0 +1,99 @@
+package bolt
+
+import (
+	"fmt"
+	"log"
+	"testing"
+
+	"github.com/boltdb/bolt"
+	portainer "github.com/portainer/portainer/api"
+	"github.com/portainer/portainer/api/filesystem"
+)
+
+// New Database should be EE and DBVersion
+//
+
+func TestMigrateData(t *testing.T) {
+	var store *Store
+
+	t.Run("MigrateData for New Store", func(t *testing.T) {
+		fileService, err := filesystem.NewService(dataStorePath, "")
+		if err != nil {
+			log.Fatal(err)
+		}
+
+		store, err := NewStore(dataStorePath, fileService)
+		if err != nil {
+			log.Fatal(err)
+		}
+
+		err = store.Open()
+		if err != nil {
+			log.Fatal(err)
+		}
+
+		err = store.Init()
+		if err != nil {
+			log.Fatal(err)
+		}
+
+		store.MigrateData(false)
+
+		testVersion(store, portainer.DBVersionEE, t)
+		testEdition(store, portainer.PortainerEE, t)
+
+		store.Close()
+	})
+
+	tests := []struct {
+		edition         portainer.SoftwareEdition
+		version         int
+		expectedVersion int
+	}{
+		{edition: portainer.PortainerCE, version: 5, expectedVersion: portainer.DBVersionEE},
+		{edition: portainer.PortainerCE, version: 21, expectedVersion: portainer.DBVersionEE},
+	}
+
+	for _, tc := range tests {
+		store = NewTestStore(tc.edition, tc.version, true)
+		t.Run(fmt.Sprintf("MigrateData for %s version %d", tc.edition.GetEditionLabel(), tc.version), func(t *testing.T) {
+			store.MigrateData(false)
+			testVersion(store, tc.expectedVersion, t)
+			testEdition(store, portainer.PortainerEE, t)
+		})
+
+		t.Run(fmt.Sprintf("Restoring DB after migrateData for %s version %d", tc.edition.GetEditionLabel(), tc.version), func(t *testing.T) {
+			store.RollbackToCE()
+			testVersion(store, tc.version, t)
+			testEdition(store, tc.edition, t)
+		})
+
+		store.Close()
+	}
+
+	t.Run("Error in MigrateData should restore backup before MigrateData", func(t *testing.T) {
+		version := 21
+		store = NewTestStore(portainer.PortainerCE, version, true)
+
+		deleteBucket(store.connection.DB, "settings")
+		store.MigrateData(false)
+
+		testVersion(store, version, t)
+		testEdition(store, portainer.PortainerCE, t)
+
+		store.Close()
+	})
+
+	teardown()
+}
+
+func deleteBucket(db *bolt.DB, bucketName string) {
+	db.Update(func(tx *bolt.Tx) error {
+		log.Printf("Delete bucket %s\n", bucketName)
+		err := tx.DeleteBucket([]byte(bucketName))
+		if err != nil {
+			log.Println(err)
+		}
+		return err
+	})
+}

api/bolt/migrator/downgrade_dbversion25.go

@@ -0,0 +1,15 @@
+package migrator
+
+// DowngradeSettingsFrom25 downgrade template settings for portainer v1.2
+func (migrator *Migrator) DowngradeSettingsFrom25() error {
+	legacySettings, err := migrator.settingsService.Settings()
+	if err != nil {
+		return err
+	}
+
+	legacySettings.TemplatesURL = "https://raw.githubusercontent.com/portainer/templates/master/templates-1.20.0.json"
+
+	err = migrator.settingsService.UpdateSettings(legacySettings)
+
+	return err
+}

api/bolt/migrator/migrate_ce.go

@@ -0,0 +1,308 @@
+package migrator
+
+import (
+	"log"
+
+	portainer "github.com/portainer/portainer/api"
+)
+
+// MigrateCE checks the database version and migrate the existing data to the most recent data model.
+func (m *Migrator) MigrateCE() error {
+	// Portainer < 1.12
+	if m.currentDBVersion < 1 {
+		err := m.updateAdminUserToDBVersion1()
+		if err != nil {
+			return err
+		}
+	}
+
+	// Portainer 1.12.x
+	if m.currentDBVersion < 2 {
+		err := m.updateResourceControlsToDBVersion2()
+		if err != nil {
+			return err
+		}
+		err = m.updateEndpointsToDBVersion2()
+		if err != nil {
+			return err
+		}
+	}
+
+	// Portainer 1.13.x
+	if m.currentDBVersion < 3 {
+		err := m.updateSettingsToDBVersion3()
+		if err != nil {
+			return err
+		}
+	}
+
+	// Portainer 1.14.0
+	if m.currentDBVersion < 4 {
+		err := m.updateEndpointsToDBVersion4()
+		if err != nil {
+			return err
+		}
+	}
+
+	// https://github.com/portainer/portainer/issues/1235
+	if m.currentDBVersion < 5 {
+		err := m.updateSettingsToVersion5()
+		if err != nil {
+			return err
+		}
+	}
+
+	// https://github.com/portainer/portainer/issues/1236
+	if m.currentDBVersion < 6 {
+		err := m.updateSettingsToVersion6()
+		if err != nil {
+			return err
+		}
+	}
+
+	// https://github.com/portainer/portainer/issues/1449
+	if m.currentDBVersion < 7 {
+		err := m.updateSettingsToVersion7()
+		if err != nil {
+			return err
+		}
+	}
+
+	if m.currentDBVersion < 8 {
+		err := m.updateEndpointsToVersion8()
+		if err != nil {
+			return err
+		}
+	}
+
+	// https: //github.com/portainer/portainer/issues/1396
+	if m.currentDBVersion < 9 {
+		err := m.updateEndpointsToVersion9()
+		if err != nil {
+			return err
+		}
+	}
+
+	// https://github.com/portainer/portainer/issues/461
+	if m.currentDBVersion < 10 {
+		err := m.updateEndpointsToVersion10()
+		if err != nil {
+			return err
+		}
+	}
+
+	// https://github.com/portainer/portainer/issues/1906
+	if m.currentDBVersion < 11 {
+		err := m.updateEndpointsToVersion11()
+		if err != nil {
+			return err
+		}
+	}
+
+	// Portainer 1.18.0
+	if m.currentDBVersion < 12 {
+		err := m.updateEndpointsToVersion12()
+		if err != nil {
+			return err
+		}
+
+		err = m.updateEndpointGroupsToVersion12()
+		if err != nil {
+			return err
+		}
+
+		err = m.updateStacksToVersion12()
+		if err != nil {
+			return err
+		}
+	}
+
+	// Portainer 1.19.0
+	if m.currentDBVersion < 13 {
+		err := m.updateSettingsToVersion13()
+		if err != nil {
+			return err
+		}
+	}
+
+	// Portainer 1.19.2
+	if m.currentDBVersion < 14 {
+		err := m.updateResourceControlsToDBVersion14()
+		if err != nil {
+			return err
+		}
+	}
+
+	// Portainer 1.20.0
+	if m.currentDBVersion < 15 {
+		err := m.updateSettingsToDBVersion15()
+		if err != nil {
+			return err
+		}
+
+		err = m.updateTemplatesToVersion15()
+		if err != nil {
+			return err
+		}
+	}
+
+	if m.currentDBVersion < 16 {
+		err := m.updateSettingsToDBVersion16()
+		if err != nil {
+			return err
+		}
+	}
+
+	// Portainer 1.20.1
+	if m.currentDBVersion < 17 {
+		err := m.updateExtensionsToDBVersion17()
+		if err != nil {
+			return err
+		}
+	}
+
+	// Portainer 1.21.0
+	if m.currentDBVersion < 18 {
+		err := m.updateUsersToDBVersion18()
+		if err != nil {
+			return err
+		}
+
+		err = m.updateEndpointsToDBVersion18()
+		if err != nil {
+			return err
+		}
+
+		err = m.updateEndpointGroupsToDBVersion18()
+		if err != nil {
+			return err
+		}
+
+		err = m.updateRegistriesToDBVersion18()
+		if err != nil {
+			return err
+		}
+	}
+
+	// Portainer 1.22.0
+	if m.currentDBVersion < 19 {
+		err := m.updateSettingsToDBVersion19()
+		if err != nil {
+			return err
+		}
+	}
+
+	// Portainer 1.22.1
+	if m.currentDBVersion < 20 {
+		err := m.updateUsersToDBVersion20()
+		if err != nil {
+			return err
+		}
+
+		err = m.updateSettingsToDBVersion20()
+		if err != nil {
+			return err
+		}
+
+		err = m.updateSchedulesToDBVersion20()
+		if err != nil {
+			return err
+		}
+	}
+
+	// Portainer 1.23.0
+	// DBVersion 21 is missing as it was shipped as via hotfix 1.22.2
+	if m.currentDBVersion < 22 {
+		err := m.updateResourceControlsToDBVersion22()
+		if err != nil {
+			return err
+		}
+
+		err = m.updateUsersAndRolesToDBVersion22()
+		if err != nil {
+			return err
+		}
+	}
+
+	// Portainer 1.24.0
+	if m.currentDBVersion < 23 {
+		err := m.updateTagsToDBVersion23()
+		if err != nil {
+			return err
+		}
+
+		err = m.updateEndpointsAndEndpointGroupsToDBVersion23()
+		if err != nil {
+			return err
+		}
+	}
+
+	// Portainer 1.24.1
+	if m.currentDBVersion < 24 {
+		err := m.updateSettingsToDB24()
+		if err != nil {
+			return err
+		}
+	}
+
+	// Portainer 2.0.0
+	if m.currentDBVersion < 25 {
+		err := m.updateSettingsToDB25()
+		if err != nil {
+			return err
+		}
+
+		err = m.updateStacksToDB24()
+		if err != nil {
+			return err
+		}
+	}
+
+	// Portainer 2.1.0
+	if m.currentDBVersion < 26 {
+		err := m.updateEndpointSettingsToDB26()
+		if err != nil {
+			return err
+		}
+
+		err = m.updateRbacRolesToDB26()
+		if err != nil {
+			return err
+		}
+	}
+
+	// Portainer 2.2.0
+	if m.currentDBVersion < 27 {
+		err := m.updateStackResourceControlToDB27()
+		if err != nil {
+			return err
+		}
+	}
+
+	// Portainer EE-2.4.0
+	if m.currentDBVersion < 28 {
+		err := m.updateUsersAndRolesToDBVersion28()
+		if err != nil {
+			return err
+		}
+	}
+
+	// Portainer EE-2.4.0
+	if m.currentDBVersion < 29 {
+		err := m.updateRbacRolesToDB29()
+		if err != nil {
+			return err
+		}
+	}
+
+	// Portainer EE-2.7.0
+	if m.currentDBVersion < 31 {
+		err := m.updateSettingsToDB31()
+		if err != nil {
+			return err
+		}
+	}
+
+	log.Println("Update DB version to ", portainer.DBVersion)
+	return m.versionService.StoreDBVersion(portainer.DBVersion)
+}

api/bolt/migrator/migrate_dbversion20.go

@@ -29,11 +29,6 @@ func (m *Migrator) updateUsersAndRolesToDBVersion22() error {
 		return err
 	}
 
-	settings, err := m.settingsService.Settings()
-	if err != nil {
-		return err
-	}
-
 	for _, user := range legacyUsers {
 		user.PortainerAuthorizations = authorization.DefaultPortainerAuthorizations()
 		err = m.userService.UpdateUser(user.ID, &user)
@@ -56,7 +51,7 @@ func (m *Migrator) updateUsersAndRolesToDBVersion22() error {
 		return err
 	}
 	helpDeskRole.Priority = 2
-	helpDeskRole.Authorizations = authorization.DefaultEndpointAuthorizationsForHelpDeskRole(settings.AllowVolumeBrowserForRegularUsers)
+	helpDeskRole.Authorizations = authorization.DefaultEndpointAuthorizationsForHelpDeskRole()
 
 	err = m.roleService.UpdateRole(helpDeskRole.ID, helpDeskRole)
 
@@ -65,7 +60,7 @@ func (m *Migrator) updateUsersAndRolesToDBVersion22() error {
 		return err
 	}
 	standardUserRole.Priority = 3
-	standardUserRole.Authorizations = authorization.DefaultEndpointAuthorizationsForStandardUserRole(settings.AllowVolumeBrowserForRegularUsers)
+	standardUserRole.Authorizations = authorization.DefaultEndpointAuthorizationsForStandardUserRole()
 
 	err = m.roleService.UpdateRole(standardUserRole.ID, standardUserRole)
 
@@ -74,7 +69,7 @@ func (m *Migrator) updateUsersAndRolesToDBVersion22() error {
 		return err
 	}
 	readOnlyUserRole.Priority = 4
-	readOnlyUserRole.Authorizations = authorization.DefaultEndpointAuthorizationsForReadOnlyUserRole(settings.AllowVolumeBrowserForRegularUsers)
+	readOnlyUserRole.Authorizations = authorization.DefaultEndpointAuthorizationsForReadOnlyUserRole()
 
 	err = m.roleService.UpdateRole(readOnlyUserRole.ID, readOnlyUserRole)
 	if err != nil {

api/bolt/migrator/migrate_dbversion25.go

@@ -2,9 +2,10 @@ package migrator
 
 import (
 	portainer "github.com/portainer/portainer/api"
+	"github.com/portainer/portainer/api/internal/authorization"
 )
 
-func (m *Migrator) updateEndpointSettingsToDB25() error {
+func (m *Migrator) updateEndpointSettingsToDB26() error {
 	settings, err := m.settingsService.Settings()
 	if err != nil {
 		return err
@@ -49,3 +50,27 @@ func (m *Migrator) updateEndpointSettingsToDB25() error {
 
 	return nil
 }
+
+func (m *Migrator) updateRbacRolesToDB26() error {
+	defaultAuthorizationsOfRoles := map[portainer.RoleID]portainer.Authorizations{
+		portainer.RoleIDEndpointAdmin: authorization.DefaultEndpointAuthorizationsForEndpointAdministratorRole(),
+		portainer.RoleIDHelpdesk:      authorization.DefaultEndpointAuthorizationsForHelpDeskRole(),
+		portainer.RoleIDStandardUser:  authorization.DefaultEndpointAuthorizationsForStandardUserRole(),
+		portainer.RoleIDReadonly:      authorization.DefaultEndpointAuthorizationsForReadOnlyUserRole(),
+	}
+
+	for roleID, defaultAuthorizations := range defaultAuthorizationsOfRoles {
+		role, err := m.roleService.Role(roleID)
+		if err != nil {
+			return err
+		}
+		role.Authorizations = defaultAuthorizations
+
+		err = m.roleService.UpdateRole(role.ID, role)
+		if err != nil {
+			return err
+		}
+	}
+
+	return m.authorizationService.UpdateUsersAuthorizations()
+}

api/bolt/migrator/migrate_dbversion26.go

@@ -1,9 +1,9 @@
 package migrator
 
 import (
+	"fmt"
 	portainer "github.com/portainer/portainer/api"
 	"github.com/portainer/portainer/api/bolt/errors"
-	"github.com/portainer/portainer/api/internal/stackutils"
 )
 
 func (m *Migrator) updateStackResourceControlToDB27() error {
@@ -18,6 +18,9 @@ func (m *Migrator) updateStackResourceControlToDB27() error {
 		}
 
 		stackName := resource.ResourceID
+		if err != nil {
+			return err
+		}
 
 		stack, err := m.stackService.StackByName(stackName)
 		if err != nil {
@@ -28,7 +31,7 @@ func (m *Migrator) updateStackResourceControlToDB27() error {
 			return err
 		}
 
-		resource.ResourceID = stackutils.ResourceControlID(stack.EndpointID, stack.Name)
+		resource.ResourceID = fmt.Sprintf("%d_%s", stack.EndpointID, stack.Name)
 
 		err = m.resourceControlService.UpdateResourceControl(resource.ID, &resource)
 		if err != nil {

api/bolt/migrator/migrate_dbversion27.go

@@ -0,0 +1,10 @@
+package migrator
+
+func (m *Migrator) updateUsersAndRolesToDBVersion28() error {
+	err := m.roleService.CreateOrUpdatePredefinedRoles()
+	if err != nil {
+		return err
+	}
+
+	return m.authorizationService.UpdateUsersAuthorizations()
+}

api/bolt/migrator/migrate_dbversion28.go

@@ -0,0 +1,31 @@
+package migrator
+
+import (
+	portainer "github.com/portainer/portainer/api"
+	"github.com/portainer/portainer/api/internal/authorization"
+)
+
+func (m *Migrator) updateRbacRolesToDB29() error {
+	defaultAuthorizationsOfRoles := map[portainer.RoleID]portainer.Authorizations{
+		portainer.RoleIDEndpointAdmin: authorization.DefaultEndpointAuthorizationsForEndpointAdministratorRole(),
+		portainer.RoleIDHelpdesk:      authorization.DefaultEndpointAuthorizationsForHelpDeskRole(),
+		portainer.RoleIDOperator:      authorization.DefaultEndpointAuthorizationsForOperatorRole(),
+		portainer.RoleIDStandardUser:  authorization.DefaultEndpointAuthorizationsForStandardUserRole(),
+		portainer.RoleIDReadonly:      authorization.DefaultEndpointAuthorizationsForReadOnlyUserRole(),
+	}
+
+	for roleID, defaultAuthorizations := range defaultAuthorizationsOfRoles {
+		role, err := m.roleService.Role(roleID)
+		if err != nil {
+			return err
+		}
+		role.Authorizations = defaultAuthorizations
+
+		err = m.roleService.UpdateRole(role.ID, role)
+		if err != nil {
+			return err
+		}
+	}
+
+	return m.authorizationService.UpdateUsersAuthorizations()
+}

api/bolt/migrator/migrate_dbversion30.go

@@ -1,19 +1,12 @@
 package migrator
 
-func (m *Migrator) migrateDBVersionToDB30() error {
-	if err := m.migrateSettingsToDB30(); err != nil {
-		return err
-	}
-
-	return nil
-}
-
-func (m *Migrator) migrateSettingsToDB30() error {
+func (m *Migrator) updateSettingsToDB31() error {
 	legacySettings, err := m.settingsService.Settings()
 	if err != nil {
 		return err
 	}
 	legacySettings.OAuthSettings.SSO = false
+	legacySettings.OAuthSettings.HideInternalAuth = false
 	legacySettings.OAuthSettings.LogoutURI = ""
 	return m.settingsService.UpdateSettings(legacySettings)
 }

api/bolt/migrator/migrate_dbversion30_test.go

@@ -2,12 +2,9 @@ package migrator
 
 import (
 	"os"
-	"path"
 	"testing"
-	"time"
 
 	"github.com/boltdb/bolt"
-	"github.com/portainer/portainer/api/bolt/internal"
 	"github.com/portainer/portainer/api/bolt/settings"
 )
 
@@ -19,34 +16,6 @@ var (
 	settingsService    *settings.Service
 )
 
-// initTestingDBConn creates a raw bolt DB connection
-// for unit testing usage only since using NewStore will cause cycle import inside migrator pkg
-func initTestingDBConn(storePath, fileName string) (*bolt.DB, error) {
-	databasePath := path.Join(storePath, fileName)
-	dbConn, err := bolt.Open(databasePath, 0600, &bolt.Options{Timeout: 1 * time.Second})
-	if err != nil {
-		return nil, err
-	}
-	return dbConn, nil
-}
-
-// initTestingDBConn creates a settings service with raw bolt DB connection
-// for unit testing usage only since using NewStore will cause cycle import inside migrator pkg
-func initTestingSettingsService(dbConn *bolt.DB, preSetObj map[string]interface{}) (*settings.Service, error) {
-	internalDBConn := &internal.DbConnection{
-		DB: dbConn,
-	}
-	settingsService, err := settings.NewService(internalDBConn)
-	if err != nil {
-		return nil, err
-	}
-	//insert a obj
-	if err := internal.UpdateObject(internalDBConn, "settings", []byte("SETTINGS"), preSetObj); err != nil {
-		return nil, err
-	}
-	return settingsService, nil
-}
-
 func setup() error {
 	testingDBStorePath, _ = os.Getwd()
 	testingDBFileName = "portainer-ee-mig-30.db"
@@ -66,7 +35,7 @@ func setup() error {
 	return nil
 }
 
-func TestMigrateSettings(t *testing.T) {
+func TestUpdateSettingsToDB31(t *testing.T) {
 	if err := setup(); err != nil {
 		t.Errorf("failed to complete testing setups, err: %v", err)
 	}
@@ -76,7 +45,7 @@ func TestMigrateSettings(t *testing.T) {
 		db:              dbConn,
 		settingsService: settingsService,
 	}
-	if err := m.migrateSettingsToDB30(); err != nil {
+	if err := m.updateSettingsToDB31(); err != nil {
 		t.Errorf("failed to update settings: %v", err)
 	}
 	updatedSettings, err := m.settingsService.Settings()
@@ -89,6 +58,9 @@ func TestMigrateSettings(t *testing.T) {
 	if updatedSettings.OAuthSettings.SSO != false {
 		t.Errorf("unexpected default OAuth SSO setting, want: false, got: %t", updatedSettings.OAuthSettings.SSO)
 	}
+	if updatedSettings.OAuthSettings.HideInternalAuth != false {
+		t.Errorf("unexpected default OAuth HideInternalAuth setting, want: false, got: %t", updatedSettings.OAuthSettings.HideInternalAuth)
+	}
 	if updatedSettings.OAuthSettings.LogoutURI != "" {
 		t.Errorf("unexpected default OAuth HideInternalAuth setting, want:, got: %s", updatedSettings.OAuthSettings.LogoutURI)
 	}

api/bolt/migrator/migrate_dbversion31.go

@@ -1,239 +0,0 @@
-package migrator
-
-import (
-	"fmt"
-
-	portainer "github.com/portainer/portainer/api"
-	"github.com/portainer/portainer/api/bolt/errors"
-	"github.com/portainer/portainer/api/internal/endpointutils"
-	snapshotutils "github.com/portainer/portainer/api/internal/snapshot"
-)
-
-func (m *Migrator) migrateDBVersionToDB32() error {
-	err := m.updateRegistriesToDB32()
-	if err != nil {
-		return err
-	}
-
-	err = m.updateDockerhubToDB32()
-	if err != nil {
-		return err
-	}
-
-	if err := m.updateVolumeResourceControlToDB32(); err != nil {
-		return err
-	}
-
-	if err := m.kubeconfigExpiryToDB32(); err != nil {
-		return err
-	}
-
-	if err := m.helmRepositoryURLToDB32(); err != nil {
-		return err
-	}
-
-	return nil
-}
-
-func (m *Migrator) updateRegistriesToDB32() error {
-	registries, err := m.registryService.Registries()
-	if err != nil {
-		return err
-	}
-
-	endpoints, err := m.endpointService.Endpoints()
-	if err != nil {
-		return err
-	}
-
-	for _, registry := range registries {
-
-		registry.RegistryAccesses = portainer.RegistryAccesses{}
-
-		for _, endpoint := range endpoints {
-
-			filteredUserAccessPolicies := portainer.UserAccessPolicies{}
-			for userId, registryPolicy := range registry.UserAccessPolicies {
-				if _, found := endpoint.UserAccessPolicies[userId]; found {
-					filteredUserAccessPolicies[userId] = registryPolicy
-				}
-			}
-
-			filteredTeamAccessPolicies := portainer.TeamAccessPolicies{}
-			for teamId, registryPolicy := range registry.TeamAccessPolicies {
-				if _, found := endpoint.TeamAccessPolicies[teamId]; found {
-					filteredTeamAccessPolicies[teamId] = registryPolicy
-				}
-			}
-
-			registry.RegistryAccesses[endpoint.ID] = portainer.RegistryAccessPolicies{
-				UserAccessPolicies: filteredUserAccessPolicies,
-				TeamAccessPolicies: filteredTeamAccessPolicies,
-				Namespaces:         []string{},
-			}
-		}
-		m.registryService.UpdateRegistry(registry.ID, &registry)
-	}
-	return nil
-}
-
-func (m *Migrator) updateDockerhubToDB32() error {
-	dockerhub, err := m.dockerhubService.DockerHub()
-	if err == errors.ErrObjectNotFound {
-		return nil
-	} else if err != nil {
-		return err
-	}
-
-	if !dockerhub.Authentication {
-		return nil
-	}
-
-	registry := &portainer.Registry{
-		Type:             portainer.DockerHubRegistry,
-		Name:             "Dockerhub (authenticated - migrated)",
-		URL:              "docker.io",
-		Authentication:   true,
-		Username:         dockerhub.Username,
-		Password:         dockerhub.Password,
-		RegistryAccesses: portainer.RegistryAccesses{},
-	}
-
-	endpoints, err := m.endpointService.Endpoints()
-	if err != nil {
-		return err
-	}
-
-	for _, endpoint := range endpoints {
-
-		if endpoint.Type != portainer.KubernetesLocalEnvironment &&
-			endpoint.Type != portainer.AgentOnKubernetesEnvironment &&
-			endpoint.Type != portainer.EdgeAgentOnKubernetesEnvironment {
-
-			userAccessPolicies := portainer.UserAccessPolicies{}
-			for userId := range endpoint.UserAccessPolicies {
-				if _, found := endpoint.UserAccessPolicies[userId]; found {
-					userAccessPolicies[userId] = portainer.AccessPolicy{
-						RoleID: 0,
-					}
-				}
-			}
-
-			teamAccessPolicies := portainer.TeamAccessPolicies{}
-			for teamId := range endpoint.TeamAccessPolicies {
-				if _, found := endpoint.TeamAccessPolicies[teamId]; found {
-					teamAccessPolicies[teamId] = portainer.AccessPolicy{
-						RoleID: 0,
-					}
-				}
-			}
-
-			registry.RegistryAccesses[endpoint.ID] = portainer.RegistryAccessPolicies{
-				UserAccessPolicies: userAccessPolicies,
-				TeamAccessPolicies: teamAccessPolicies,
-				Namespaces:         []string{},
-			}
-		}
-	}
-
-	return m.registryService.CreateRegistry(registry)
-}
-
-func (m *Migrator) updateVolumeResourceControlToDB32() error {
-	endpoints, err := m.endpointService.Endpoints()
-	if err != nil {
-		return fmt.Errorf("failed fetching environments: %w", err)
-	}
-
-	resourceControls, err := m.resourceControlService.ResourceControls()
-	if err != nil {
-		return fmt.Errorf("failed fetching resource controls: %w", err)
-	}
-
-	toUpdate := map[portainer.ResourceControlID]string{}
-	volumeResourceControls := map[string]*portainer.ResourceControl{}
-
-	for i := range resourceControls {
-		resourceControl := resourceControls[i]
-		if resourceControl.Type == portainer.VolumeResourceControl {
-			volumeResourceControls[resourceControl.ResourceID] = &resourceControl
-		}
-	}
-
-	for _, endpoint := range endpoints {
-		if !endpointutils.IsDockerEndpoint(&endpoint) {
-			continue
-		}
-
-		totalSnapshots := len(endpoint.Snapshots)
-		if totalSnapshots == 0 {
-			continue
-		}
-
-		snapshot := endpoint.Snapshots[totalSnapshots-1]
-
-		endpointDockerID, err := snapshotutils.FetchDockerID(snapshot)
-		if err != nil {
-			return fmt.Errorf("failed fetching environment docker id: %w", err)
-		}
-
-		if volumesData, done := snapshot.SnapshotRaw.Volumes.(map[string]interface{}); done {
-			if volumesData["Volumes"] == nil {
-				continue
-			}
-
-			findResourcesToUpdateForDB32(endpointDockerID, volumesData, toUpdate, volumeResourceControls)
-		}
-	}
-
-	for _, resourceControl := range volumeResourceControls {
-		if newResourceID, ok := toUpdate[resourceControl.ID]; ok {
-			resourceControl.ResourceID = newResourceID
-			err := m.resourceControlService.UpdateResourceControl(resourceControl.ID, resourceControl)
-			if err != nil {
-				return fmt.Errorf("failed updating resource control %d: %w", resourceControl.ID, err)
-			}
-
-		} else {
-			err := m.resourceControlService.DeleteResourceControl(resourceControl.ID)
-			if err != nil {
-				return fmt.Errorf("failed deleting resource control %d: %w", resourceControl.ID, err)
-			}
-
-		}
-	}
-
-	return nil
-}
-
-func findResourcesToUpdateForDB32(dockerID string, volumesData map[string]interface{}, toUpdate map[portainer.ResourceControlID]string, volumeResourceControls map[string]*portainer.ResourceControl) {
-	volumes := volumesData["Volumes"].([]interface{})
-	for _, volumeMeta := range volumes {
-		volume := volumeMeta.(map[string]interface{})
-		volumeName := volume["Name"].(string)
-		oldResourceID := fmt.Sprintf("%s%s", volumeName, volume["CreatedAt"].(string))
-		resourceControl, ok := volumeResourceControls[oldResourceID]
-
-		if ok {
-			toUpdate[resourceControl.ID] = fmt.Sprintf("%s_%s", volumeName, dockerID)
-		}
-	}
-}
-
-func (m *Migrator) kubeconfigExpiryToDB32() error {
-	settings, err := m.settingsService.Settings()
-	if err != nil {
-		return err
-	}
-	settings.KubeconfigExpiry = portainer.DefaultKubeconfigExpiry
-	return m.settingsService.UpdateSettings(settings)
-}
-
-func (m *Migrator) helmRepositoryURLToDB32() error {
-	settings, err := m.settingsService.Settings()
-	if err != nil {
-		return err
-	}
-	settings.HelmRepositoryURL = portainer.DefaultHelmRepositoryURL
-	return m.settingsService.UpdateSettings(settings)
-}

api/bolt/migrator/migrate_dbversion33.go

@@ -1,32 +0,0 @@
-package migrator
-
-import (
-	portainer "github.com/portainer/portainer/api"
-)
-
-func (m *Migrator) migrateDBVersionTo33() error {
-	err := migrateStackEntryPoint(m.stackService)
-	if err != nil {
-		return err
-	}
-
-	return nil
-}
-
-func migrateStackEntryPoint(stackService portainer.StackService) error {
-	stacks, err := stackService.Stacks()
-	if err != nil {
-		return err
-	}
-	for i := range stacks {
-		stack := &stacks[i]
-		if stack.GitConfig == nil {
-			continue
-		}
-		stack.GitConfig.ConfigFilePath = stack.EntryPoint
-		if err := stackService.UpdateStack(stack.ID, stack); err != nil {
-			return err
-		}
-	}
-	return nil
-}

api/bolt/migrator/migrate_dbversion33_test.go

@@ -1,51 +0,0 @@
-package migrator
-
-import (
-	"path"
-	"testing"
-	"time"
-
-	"github.com/boltdb/bolt"
-	portainer "github.com/portainer/portainer/api"
-	"github.com/portainer/portainer/api/bolt/internal"
-	"github.com/portainer/portainer/api/bolt/stack"
-	gittypes "github.com/portainer/portainer/api/git/types"
-	"github.com/stretchr/testify/assert"
-)
-
-func TestMigrateStackEntryPoint(t *testing.T) {
-	dbConn, err := bolt.Open(path.Join(t.TempDir(), "portainer-ee-mig-33.db"), 0600, &bolt.Options{Timeout: 1 * time.Second})
-	assert.NoError(t, err, "failed to init testing DB connection")
-	defer dbConn.Close()
-
-	stackService, err := stack.NewService(&internal.DbConnection{DB: dbConn})
-	assert.NoError(t, err, "failed to init testing Stack service")
-
-	stacks := []*portainer.Stack{
-		{
-			ID:         1,
-			EntryPoint: "dir/sub/compose.yml",
-		},
-		{
-			ID:         2,
-			EntryPoint: "dir/sub/compose.yml",
-			GitConfig:  &gittypes.RepoConfig{},
-		},
-	}
-
-	for _, s := range stacks {
-		err := stackService.CreateStack(s)
-		assert.NoError(t, err, "failed to create stack")
-	}
-
-	err = migrateStackEntryPoint(stackService)
-	assert.NoError(t, err, "failed to migrate entry point to Git ConfigFilePath")
-
-	s, err := stackService.Stack(1)
-	assert.NoError(t, err)
-	assert.Nil(t, s.GitConfig, "first stack should not have git config")
-
-	s, err = stackService.Stack(2)
-	assert.NoError(t, err)
-	assert.Equal(t, "dir/sub/compose.yml", s.GitConfig.ConfigFilePath, "second stack should have config file path migrated")
-}

api/bolt/migrator/migrate_test_helper.go

@@ -0,0 +1,38 @@
+package migrator
+
+import (
+	"path"
+	"time"
+
+	"github.com/boltdb/bolt"
+	"github.com/portainer/portainer/api/bolt/internal"
+	"github.com/portainer/portainer/api/bolt/settings"
+)
+
+// initTestingDBConn creates a raw bolt DB connection
+// for unit testing usage only since using NewStore will cause cycle import inside migrator pkg
+func initTestingDBConn(storePath, fileName string) (*bolt.DB, error) {
+	databasePath := path.Join(storePath, fileName)
+	dbConn, err := bolt.Open(databasePath, 0600, &bolt.Options{Timeout: 1 * time.Second})
+	if err != nil {
+		return nil, err
+	}
+	return dbConn, nil
+}
+
+// initTestingDBConn creates a settings service with raw bolt DB connection
+// for unit testing usage only since using NewStore will cause cycle import inside migrator pkg
+func initTestingSettingsService(dbConn *bolt.DB, preSetObj map[string]interface{}) (*settings.Service, error) {
+	internalDBConn := &internal.DbConnection{
+		DB: dbConn,
+	}
+	settingsService, err := settings.NewService(internalDBConn)
+	if err != nil {
+		return nil, err
+	}
+	//insert a obj
+	if err := internal.UpdateObject(internalDBConn, "settings", []byte("SETTINGS"), preSetObj); err != nil {
+		return nil, err
+	}
+	return settingsService, nil
+}

api/bolt/migrator/migrator.go

@@ -1,9 +1,10 @@
 package migrator
 
 import (
+	"fmt"
+
 	"github.com/boltdb/bolt"
 	portainer "github.com/portainer/portainer/api"
-	"github.com/portainer/portainer/api/bolt/dockerhub"
 	"github.com/portainer/portainer/api/bolt/endpoint"
 	"github.com/portainer/portainer/api/bolt/endpointgroup"
 	"github.com/portainer/portainer/api/bolt/endpointrelation"
@@ -27,8 +28,10 @@ var migrateLog = plog.NewScopedLog("bolt, migrate")
 type (
 	// Migrator defines a service to migrate data after a Portainer version update.
 	Migrator struct {
-		currentDBVersion        int
-		db                      *bolt.DB
+		db               *bolt.DB
+		currentDBVersion int
+		currentEdition   portainer.SoftwareEdition
+
 		endpointGroupService    *endpointgroup.Service
 		endpointService         *endpoint.Service
 		endpointRelationService *endpointrelation.Service
@@ -45,13 +48,14 @@ type (
 		versionService          *version.Service
 		fileService             portainer.FileService
 		authorizationService    *authorization.Service
-		dockerhubService        *dockerhub.Service
 	}
 
 	// Parameters represents the required parameters to create a new Migrator instance.
 	Parameters struct {
-		DB                      *bolt.DB
-		DatabaseVersion         int
+		DB              *bolt.DB
+		DatabaseVersion int
+		CurrentEdition  portainer.SoftwareEdition
+
 		EndpointGroupService    *endpointgroup.Service
 		EndpointService         *endpoint.Service
 		EndpointRelationService *endpointrelation.Service
@@ -68,7 +72,6 @@ type (
 		VersionService          *version.Service
 		FileService             portainer.FileService
 		AuthorizationService    *authorization.Service
-		DockerhubService        *dockerhub.Service
 	}
 )
 
@@ -77,6 +80,7 @@ func NewMigrator(parameters *Parameters) *Migrator {
 	return &Migrator{
 		db:                      parameters.DB,
 		currentDBVersion:        parameters.DatabaseVersion,
+		currentEdition:          parameters.CurrentEdition,
 		endpointGroupService:    parameters.EndpointGroupService,
 		endpointService:         parameters.EndpointService,
 		endpointRelationService: parameters.EndpointRelationService,
@@ -93,299 +97,46 @@ func NewMigrator(parameters *Parameters) *Migrator {
 		versionService:          parameters.VersionService,
 		fileService:             parameters.FileService,
 		authorizationService:    parameters.AuthorizationService,
-		dockerhubService:        parameters.DockerhubService,
 	}
 }
 
-// Migrate checks the database version and migrate the existing data to the most recent data model.
-func (m *Migrator) Migrate() error {
-	// Portainer < 1.12
-	if m.currentDBVersion < 1 {
-		err := m.updateAdminUserToDBVersion1()
-		if err != nil {
-			return err
-		}
-	}
-
-	// Portainer 1.12.x
-	if m.currentDBVersion < 2 {
-		err := m.updateResourceControlsToDBVersion2()
-		if err != nil {
-			return err
-		}
-		err = m.updateEndpointsToDBVersion2()
-		if err != nil {
-			return err
-		}
-	}
-
-	// Portainer 1.13.x
-	if m.currentDBVersion < 3 {
-		err := m.updateSettingsToDBVersion3()
-		if err != nil {
-			return err
-		}
-	}
-
-	// Portainer 1.14.0
-	if m.currentDBVersion < 4 {
-		err := m.updateEndpointsToDBVersion4()
-		if err != nil {
-			return err
-		}
-	}
-
-	// https://github.com/portainer/portainer/issues/1235
-	if m.currentDBVersion < 5 {
-		err := m.updateSettingsToVersion5()
-		if err != nil {
-			return err
-		}
-	}
-
-	// https://github.com/portainer/portainer/issues/1236
-	if m.currentDBVersion < 6 {
-		err := m.updateSettingsToVersion6()
-		if err != nil {
-			return err
-		}
-	}
-
-	// https://github.com/portainer/portainer/issues/1449
-	if m.currentDBVersion < 7 {
-		err := m.updateSettingsToVersion7()
-		if err != nil {
-			return err
-		}
-	}
-
-	if m.currentDBVersion < 8 {
-		err := m.updateEndpointsToVersion8()
-		if err != nil {
-			return err
-		}
-	}
-
-	// https: //github.com/portainer/portainer/issues/1396
-	if m.currentDBVersion < 9 {
-		err := m.updateEndpointsToVersion9()
-		if err != nil {
-			return err
-		}
-	}
-
-	// https://github.com/portainer/portainer/issues/461
-	if m.currentDBVersion < 10 {
-		err := m.updateEndpointsToVersion10()
-		if err != nil {
-			return err
-		}
-	}
-
-	// https://github.com/portainer/portainer/issues/1906
-	if m.currentDBVersion < 11 {
-		err := m.updateEndpointsToVersion11()
-		if err != nil {
-			return err
-		}
-	}
-
-	// Portainer 1.18.0
-	if m.currentDBVersion < 12 {
-		err := m.updateEndpointsToVersion12()
-		if err != nil {
-			return err
-		}
-
-		err = m.updateEndpointGroupsToVersion12()
-		if err != nil {
-			return err
-		}
-
-		err = m.updateStacksToVersion12()
-		if err != nil {
-			return err
-		}
-	}
-
-	// Portainer 1.19.0
-	if m.currentDBVersion < 13 {
-		err := m.updateSettingsToVersion13()
-		if err != nil {
-			return err
-		}
-	}
-
-	// Portainer 1.19.2
-	if m.currentDBVersion < 14 {
-		err := m.updateResourceControlsToDBVersion14()
-		if err != nil {
-			return err
-		}
-	}
-
-	// Portainer 1.20.0
-	if m.currentDBVersion < 15 {
-		err := m.updateSettingsToDBVersion15()
-		if err != nil {
-			return err
-		}
-
-		err = m.updateTemplatesToVersion15()
-		if err != nil {
-			return err
-		}
-	}
-
-	if m.currentDBVersion < 16 {
-		err := m.updateSettingsToDBVersion16()
-		if err != nil {
-			return err
-		}
-	}
-
-	// Portainer 1.20.1
-	if m.currentDBVersion < 17 {
-		err := m.updateExtensionsToDBVersion17()
-		if err != nil {
-			return err
-		}
-	}
-
-	// Portainer 1.21.0
-	if m.currentDBVersion < 18 {
-		err := m.updateUsersToDBVersion18()
-		if err != nil {
-			return err
-		}
-
-		err = m.updateEndpointsToDBVersion18()
-		if err != nil {
-			return err
-		}
-
-		err = m.updateEndpointGroupsToDBVersion18()
-		if err != nil {
-			return err
-		}
-
-		err = m.updateRegistriesToDBVersion18()
-		if err != nil {
-			return err
-		}
-	}
-
-	// Portainer 1.22.0
-	if m.currentDBVersion < 19 {
-		err := m.updateSettingsToDBVersion19()
-		if err != nil {
-			return err
-		}
-	}
-
-	// Portainer 1.22.1
-	if m.currentDBVersion < 20 {
-		err := m.updateUsersToDBVersion20()
-		if err != nil {
-			return err
-		}
-
-		err = m.updateSettingsToDBVersion20()
-		if err != nil {
-			return err
-		}
-
-		err = m.updateSchedulesToDBVersion20()
-		if err != nil {
-			return err
-		}
-	}
-
-	// Portainer 1.23.0
-	// DBVersion 21 is missing as it was shipped as via hotfix 1.22.2
-	if m.currentDBVersion < 22 {
-		err := m.updateResourceControlsToDBVersion22()
-		if err != nil {
-			return err
-		}
-
-		err = m.updateUsersAndRolesToDBVersion22()
-		if err != nil {
-			return err
-		}
-	}
-
-	// Portainer 1.24.0
-	if m.currentDBVersion < 23 {
-		err := m.updateTagsToDBVersion23()
-		if err != nil {
-			return err
-		}
-
-		err = m.updateEndpointsAndEndpointGroupsToDBVersion23()
-		if err != nil {
-			return err
-		}
-	}
-
-	// Portainer 1.24.1
-	if m.currentDBVersion < 24 {
-		err := m.updateSettingsToDB24()
-		if err != nil {
-			return err
-		}
-	}
-
-	// Portainer 2.0.0
-	if m.currentDBVersion < 25 {
-		err := m.updateSettingsToDB25()
-		if err != nil {
-			return err
-		}
-
-		err = m.updateStacksToDB24()
-		if err != nil {
-			return err
-		}
-	}
-
-	// Portainer 2.1.0
-	if m.currentDBVersion < 26 {
-		err := m.updateEndpointSettingsToDB25()
-		if err != nil {
-			return err
-		}
-	}
-
-	// Portainer 2.2.0
-	if m.currentDBVersion < 27 {
-		err := m.updateStackResourceControlToDB27()
-		if err != nil {
-			return err
-		}
-	}
-
-	// Portainer 2.6.0
-	if m.currentDBVersion < 30 {
-		err := m.migrateDBVersionToDB30()
-		if err != nil {
-			return err
-		}
-	}
-
-	// Portainer 2.9.0
-	if m.currentDBVersion < 32 {
-		err := m.migrateDBVersionToDB32()
-		if err != nil {
-			return err
-		}
-	}
-
-	if m.currentDBVersion < 33 {
-		if err := m.migrateDBVersionTo33(); err != nil {
-			return err
-		}
-	}
-
-	return m.versionService.StoreDBVersion(portainer.DBVersion)
+// Version exposes version of database
+func (migrator *Migrator) Version() int {
+	return migrator.currentDBVersion
+}
+
+// Edition exposes edition of portainer
+func (migrator *Migrator) Edition() portainer.SoftwareEdition {
+	return migrator.currentEdition
+}
+
+// Migrate helper to upgrade DB
+func (migrator *Migrator) Migrate(version int) error {
+
+	migrateLog.Info(fmt.Sprintf("Migrating %s database from version %d to %d.", migrator.Edition().GetEditionLabel(), migrator.currentDBVersion, version))
+	// TODO : run backup before migration and restore if failed
+	err := migrator.MigrateCE() //CE
+	if err != nil {
+		migrateLog.Error("An error occurred during database migration", err)
+		return err
+	}
+
+	migrator.versionService.StoreDBVersion(version)
+	migrator.currentDBVersion = version
+
+	return nil
+}
+
+// RollbackVersion rolls back the db to version
+func (migrator *Migrator) RollbackVersion(version int) error {
+
+	err := migrator.versionService.StoreDBVersion(version) // portainer.DBVersion
+	return err
+}
+
+// RollbackEdition rolls back the db to portainer CE
+func (migrator *Migrator) RollbackEdition(edition portainer.SoftwareEdition) error {
+
+	err := migrator.versionService.StoreEdition(portainer.PortainerCE)
+	return err
 }

api/bolt/migrator/migrator_test.go

@@ -0,0 +1,4 @@
+package migrator
+
+// test CE version is always upgraded to latest version of CE
+// test EE version is always upgraded to latest version of EE

api/bolt/migrator/upgrade_ee.go

@@ -0,0 +1,228 @@
+package migrator
+
+import (
+	"fmt"
+
+	portainer "github.com/portainer/portainer/api"
+
+	"github.com/portainer/portainer/api/internal/authorization"
+)
+
+// UpgradeToEE will migrate the db from latest ce version to latest ee version
+// Latest version is v25 on 06/11/2020
+func (m *Migrator) UpgradeToEE() error {
+
+	migrateLog.Info(fmt.Sprintf("Migrating CE database version %d to EE database version %d.", m.Version(), portainer.DBVersion))
+
+	migrateLog.Info("Updating LDAP settings to EE")
+	err := m.updateSettingsToEE()
+	if err != nil {
+		return err
+	}
+
+	migrateLog.Info("Updating user roles to EE")
+	err = m.updateUserRolesToEE()
+	if err != nil {
+		return err
+	}
+	migrateLog.Info("Updating role authorizations to EE")
+	err = m.updateRoleAuthorizationsToEE()
+	if err != nil {
+		return err
+	}
+	migrateLog.Info("Updating user authorizations")
+	err = m.authorizationService.UpdateUsersAuthorizations()
+	if err != nil {
+		return err
+	}
+
+	migrateLog.Info(fmt.Sprintf("Setting db version to %d", portainer.DBVersionEE))
+	err = m.versionService.StoreDBVersion(portainer.DBVersionEE)
+	if err != nil {
+		return err
+	}
+
+	migrateLog.Info(fmt.Sprintf("Setting edition to %s", portainer.PortainerEE.GetEditionLabel()))
+	err = m.versionService.StoreEdition(portainer.PortainerEE)
+	if err != nil {
+		return err
+	}
+
+	m.currentDBVersion = portainer.DBVersionEE
+	m.currentEdition = portainer.PortainerEE
+
+	return nil
+}
+
+func (m *Migrator) updateSettingsToEE() error {
+	legacySettings, err := m.settingsService.Settings()
+	if err != nil {
+		return err
+	}
+
+	legacySettings.LDAPSettings.URLs = []string{}
+	url := legacySettings.LDAPSettings.URL
+	if url != "" {
+		legacySettings.LDAPSettings.URLs = append(legacySettings.LDAPSettings.URLs, url)
+	}
+
+	legacySettings.LDAPSettings.ServerType = portainer.LDAPServerCustom
+
+	return m.settingsService.UpdateSettings(legacySettings)
+}
+
+// Updating role authorizations because of the new policies in Kube RBAC
+func (m *Migrator) updateRoleAuthorizationsToEE() error {
+	migrateLog.Debug("Retriving settings")
+
+	migrateLog.Debug("Updating Endpoint Admin Role")
+	endpointAdministratorRole, err := m.roleService.Role(portainer.RoleID(1))
+	if err != nil {
+		return err
+	}
+	endpointAdministratorRole.Priority = 1
+	endpointAdministratorRole.Authorizations = authorization.DefaultEndpointAuthorizationsForEndpointAdministratorRole()
+
+	err = m.roleService.UpdateRole(endpointAdministratorRole.ID, endpointAdministratorRole)
+
+	migrateLog.Debug("Updating Help Desk Role")
+	helpDeskRole, err := m.roleService.Role(portainer.RoleID(2))
+	if err != nil {
+		return err
+	}
+	helpDeskRole.Priority = 2
+	helpDeskRole.Authorizations = authorization.DefaultEndpointAuthorizationsForHelpDeskRole()
+
+	err = m.roleService.UpdateRole(helpDeskRole.ID, helpDeskRole)
+
+	migrateLog.Debug("Updating Standard User Role")
+	standardUserRole, err := m.roleService.Role(portainer.RoleID(3))
+	if err != nil {
+		return err
+	}
+	standardUserRole.Priority = 3
+	standardUserRole.Authorizations = authorization.DefaultEndpointAuthorizationsForStandardUserRole()
+
+	err = m.roleService.UpdateRole(standardUserRole.ID, standardUserRole)
+
+	migrateLog.Debug("Updating Read Only User Role")
+	readOnlyUserRole, err := m.roleService.Role(portainer.RoleID(4))
+	if err != nil {
+		return err
+	}
+	readOnlyUserRole.Priority = 4
+	readOnlyUserRole.Authorizations = authorization.DefaultEndpointAuthorizationsForReadOnlyUserRole()
+
+	err = m.roleService.UpdateRole(readOnlyUserRole.ID, readOnlyUserRole)
+	if err != nil {
+		return err
+	}
+
+	return nil
+}
+
+// If RBAC extension wasn't installed before, update all users in endpoints and
+// endpoint groups to have read only access.
+func (m *Migrator) updateUserRolesToEE() error {
+	err := m.updateUserAuthorizationToEE()
+	if err != nil {
+		return err
+	}
+
+	migrateLog.Debug("Retriving extension info")
+	extensions, err := m.extensionService.Extensions()
+	for _, extension := range extensions {
+		if extension.ID == 3 && extension.Enabled {
+			migrateLog.Info("RBAC extensions were enabled before; Skip updating User Roles")
+			return nil
+		}
+	}
+
+	migrateLog.Debug("Retriving endpoint groups")
+	endpointGroups, err := m.endpointGroupService.EndpointGroups()
+	if err != nil {
+		return err
+	}
+
+	for _, endpointGroup := range endpointGroups {
+		migrateLog.Debug(fmt.Sprintf("Updating user policies for endpoint group %v", endpointGroup.ID))
+		for key := range endpointGroup.UserAccessPolicies {
+			updateUserAccessPolicyToReadOnlyRole(endpointGroup.UserAccessPolicies, key)
+		}
+
+		for key := range endpointGroup.TeamAccessPolicies {
+			updateTeamAccessPolicyToReadOnlyRole(endpointGroup.TeamAccessPolicies, key)
+		}
+
+		err := m.endpointGroupService.UpdateEndpointGroup(endpointGroup.ID, &endpointGroup)
+		if err != nil {
+			return err
+		}
+	}
+
+	migrateLog.Debug("Retriving endpoints")
+	endpoints, err := m.endpointService.Endpoints()
+	if err != nil {
+		return err
+	}
+
+	for _, endpoint := range endpoints {
+		migrateLog.Debug(fmt.Sprintf("Updating user policies for endpoint %v", endpoint.ID))
+		for key := range endpoint.UserAccessPolicies {
+			updateUserAccessPolicyToReadOnlyRole(endpoint.UserAccessPolicies, key)
+		}
+
+		for key := range endpoint.TeamAccessPolicies {
+			updateTeamAccessPolicyToReadOnlyRole(endpoint.TeamAccessPolicies, key)
+		}
+
+		err := m.endpointService.UpdateEndpoint(endpoint.ID, &endpoint)
+		if err != nil {
+			return err
+		}
+	}
+
+	return nil
+}
+
+func (m *Migrator) updateUserAuthorizationToEE() error {
+	legacyUsers, err := m.userService.Users()
+	if err != nil {
+		return err
+	}
+
+	for _, user := range legacyUsers {
+		user.PortainerAuthorizations = authorization.DefaultPortainerAuthorizations()
+
+		err = m.userService.UpdateUser(user.ID, &user)
+		if err != nil {
+			return err
+		}
+	}
+
+	return nil
+}
+
+func updateUserAccessPolicyToNoRole(policies portainer.UserAccessPolicies, key portainer.UserID) {
+	tmp := policies[key]
+	tmp.RoleID = 0
+	policies[key] = tmp
+}
+
+func updateTeamAccessPolicyToNoRole(policies portainer.TeamAccessPolicies, key portainer.TeamID) {
+	tmp := policies[key]
+	tmp.RoleID = 0
+	policies[key] = tmp
+}
+
+func updateUserAccessPolicyToReadOnlyRole(policies portainer.UserAccessPolicies, key portainer.UserID) {
+	tmp := policies[key]
+	tmp.RoleID = 4
+	policies[key] = tmp
+}
+
+func updateTeamAccessPolicyToReadOnlyRole(policies portainer.TeamAccessPolicies, key portainer.TeamID) {
+	tmp := policies[key]
+	tmp.RoleID = 4
+	policies[key] = tmp
+}

api/bolt/registry/registry.go

@@ -12,7 +12,7 @@ const (
 	BucketName = "registries"
 )
 
-// Service represents a service for managing environment(endpoint) data.
+// Service represents a service for managing endpoint data.
 type Service struct {
 	connection *internal.DbConnection
 }

api/bolt/resourcecontrol/resourcecontrol.go

@@ -12,7 +12,7 @@ const (
 	BucketName = "resource_control"
 )
 
-// Service represents a service for managing environment(endpoint) data.
+// Service represents a service for managing endpoint data.
 type Service struct {
 	connection *internal.DbConnection
 }

api/bolt/role/predefined_roles.go

@@ -0,0 +1,68 @@
+package role
+
+import (
+	portainer "github.com/portainer/portainer/api"
+	"github.com/portainer/portainer/api/bolt/errors"
+	"github.com/portainer/portainer/api/internal/authorization"
+)
+
+// CreateOrUpdatePredefinedRoles update the predefined roles. Create one if it does not exist yet.
+func (service *Service) CreateOrUpdatePredefinedRoles() error {
+	predefinedRoles := map[portainer.RoleID]*portainer.Role{
+		portainer.RoleIDEndpointAdmin: &portainer.Role{
+			Name:           "Endpoint administrator",
+			Description:    "Full control of all resources in an endpoint",
+			ID:             portainer.RoleIDEndpointAdmin,
+			Priority:       1,
+			Authorizations: authorization.DefaultEndpointAuthorizationsForEndpointAdministratorRole(),
+		},
+		portainer.RoleIDOperator: &portainer.Role{
+			Name:           "Operator",
+			Description:    "Operational control of all existing resources in an endpoint",
+			ID:             portainer.RoleIDOperator,
+			Priority:       2,
+			Authorizations: authorization.DefaultEndpointAuthorizationsForOperatorRole(),
+		},
+		portainer.RoleIDHelpdesk: &portainer.Role{
+			Name:           "Helpdesk",
+			Description:    "Read-only access of all resources in an endpoint",
+			ID:             portainer.RoleIDHelpdesk,
+			Priority:       3,
+			Authorizations: authorization.DefaultEndpointAuthorizationsForHelpDeskRole(),
+		},
+		portainer.RoleIDStandardUser: &portainer.Role{
+			Name:           "Standard user",
+			Description:    "Full control of assigned resources in an endpoint",
+			ID:             portainer.RoleIDStandardUser,
+			Priority:       4,
+			Authorizations: authorization.DefaultEndpointAuthorizationsForStandardUserRole(),
+		},
+		portainer.RoleIDReadonly: &portainer.Role{
+			Name:           "Read-only user",
+			Description:    "Read-only access of assigned resources in an endpoint",
+			ID:             portainer.RoleIDReadonly,
+			Priority:       5,
+			Authorizations: authorization.DefaultEndpointAuthorizationsForReadOnlyUserRole(),
+		},
+	}
+
+	for roleID, predefinedRole := range predefinedRoles {
+		_, err := service.Role(roleID)
+
+		if err == errors.ErrObjectNotFound {
+			err := service.CreateRole(predefinedRole)
+			if err != nil {
+				return err
+			}
+		} else if err != nil {
+			return err
+		} else {
+			err = service.UpdateRole(predefinedRole.ID, predefinedRole)
+			if err != nil {
+				return err
+			}
+		}
+	}
+
+	return nil
+}

api/bolt/role/role.go

@@ -12,7 +12,7 @@ const (
 	BucketName = "roles"
 )
 
-// Service represents a service for managing environment(endpoint) data.
+// Service represents a service for managing endpoint data.
 type Service struct {
 	connection *internal.DbConnection
 }
@@ -71,7 +71,9 @@ func (service *Service) CreateRole(role *portainer.Role) error {
 		bucket := tx.Bucket([]byte(BucketName))
 
 		id, _ := bucket.NextSequence()
-		role.ID = portainer.RoleID(id)
+		if role.ID == 0 {
+			role.ID = portainer.RoleID(id)
+		}
 
 		data, err := internal.MarshalObject(role)
 		if err != nil {

api/bolt/s3backup/s3backup.go

@@ -0,0 +1,66 @@
+package s3backup
+
+import (
+	portainer "github.com/portainer/portainer/api"
+	"github.com/portainer/portainer/api/bolt/errors"
+	"github.com/portainer/portainer/api/bolt/internal"
+)
+
+const (
+	bucketName  = "s3backup"
+	statusKey   = "lastRunStatus"
+	settingsKey = "settings"
+)
+
+type Service struct {
+	connection *internal.DbConnection
+}
+
+// NewService creates a new service and ensures corresponding bucket exist
+func NewService(connection *internal.DbConnection) (*Service, error) {
+	err := internal.CreateBucket(connection, bucketName)
+	if err != nil {
+		return nil, err
+	}
+
+	return &Service{
+		connection: connection,
+	}, nil
+}
+
+// GetStatus returns the status of the last scheduled backup run
+func (s *Service) GetStatus() (portainer.S3BackupStatus, error) {
+	var status portainer.S3BackupStatus
+	err := internal.GetObject(s.connection, bucketName, []byte(statusKey), &status)
+	if err == errors.ErrObjectNotFound {
+		return status, nil
+	}
+
+	return status, err
+}
+
+// DropStatus deletes the status of the last sheduled backup run
+func (s *Service) DropStatus() error {
+	return internal.DeleteObject(s.connection, bucketName, []byte(statusKey))
+}
+
+// UpdateStatus upserts a status of the last scheduled backup run
+func (s *Service) UpdateStatus(status portainer.S3BackupStatus) error {
+	return internal.UpdateObject(s.connection, bucketName, []byte(statusKey), status)
+}
+
+// UpdateSettings updates stored s3 backup settings
+func (s *Service) UpdateSettings(settings portainer.S3BackupSettings) error {
+	return internal.UpdateObject(s.connection, bucketName, []byte(settingsKey), settings)
+}
+
+// GetSettings returns stored s3 backup settings
+func (s *Service) GetSettings() (portainer.S3BackupSettings, error) {
+	var settings portainer.S3BackupSettings
+	err := internal.GetObject(s.connection, bucketName, []byte(settingsKey), &settings)
+	if err == errors.ErrObjectNotFound {
+		return settings, nil
+	}
+
+	return settings, err
+}

api/bolt/services.go

@@ -11,13 +11,13 @@ import (
 	"github.com/portainer/portainer/api/bolt/endpointgroup"
 	"github.com/portainer/portainer/api/bolt/endpointrelation"
 	"github.com/portainer/portainer/api/bolt/extension"
-	"github.com/portainer/portainer/api/bolt/helmuserrepository"
+	"github.com/portainer/portainer/api/bolt/license"
 	"github.com/portainer/portainer/api/bolt/registry"
 	"github.com/portainer/portainer/api/bolt/resourcecontrol"
 	"github.com/portainer/portainer/api/bolt/role"
+	"github.com/portainer/portainer/api/bolt/s3backup"
 	"github.com/portainer/portainer/api/bolt/schedule"
 	"github.com/portainer/portainer/api/bolt/settings"
-	"github.com/portainer/portainer/api/bolt/ssl"
 	"github.com/portainer/portainer/api/bolt/stack"
 	"github.com/portainer/portainer/api/bolt/tag"
 	"github.com/portainer/portainer/api/bolt/team"
@@ -89,11 +89,11 @@ func (store *Store) initServices() error {
 	}
 	store.ExtensionService = extensionService
 
-	helmUserRepositoryService, err := helmuserrepository.NewService(store.connection)
+	licenseService, err := license.NewService(store.connection)
 	if err != nil {
 		return err
 	}
-	store.HelmUserRepositoryService = helmUserRepositoryService
+	store.LicenseService = licenseService
 
 	registryService, err := registry.NewService(store.connection)
 	if err != nil {
@@ -107,18 +107,18 @@ func (store *Store) initServices() error {
 	}
 	store.ResourceControlService = resourcecontrolService
 
+	s3backupService, err := s3backup.NewService(store.connection)
+	if err != nil {
+		return nil
+	}
+	store.S3BackupService = s3backupService
+
 	settingsService, err := settings.NewService(store.connection)
 	if err != nil {
 		return err
 	}
 	store.SettingsService = settingsService
 
-	sslSettingsService, err := ssl.NewService(store.connection)
-	if err != nil {
-		return err
-	}
-	store.SSLSettingsService = sslSettingsService
-
 	stackService, err := stack.NewService(store.connection)
 	if err != nil {
 		return err
@@ -181,6 +181,11 @@ func (store *Store) CustomTemplate() portainer.CustomTemplateService {
 	return store.CustomTemplateService
 }
 
+// DockerHub gives access to the DockerHub data management layer
+func (store *Store) DockerHub() portainer.DockerHubService {
+	return store.DockerHubService
+}
+
 // EdgeGroup gives access to the EdgeGroup data management layer
 func (store *Store) EdgeGroup() portainer.EdgeGroupService {
 	return store.EdgeGroupService
@@ -196,7 +201,7 @@ func (store *Store) EdgeStack() portainer.EdgeStackService {
 	return store.EdgeStackService
 }
 
-// Environment(Endpoint) gives access to the Environment(Endpoint) data management layer
+// Endpoint gives access to the Endpoint data management layer
 func (store *Store) Endpoint() portainer.EndpointService {
 	return store.EndpointService
 }
@@ -211,9 +216,9 @@ func (store *Store) EndpointRelation() portainer.EndpointRelationService {
 	return store.EndpointRelationService
 }
 
-// HelmUserRepository access the helm user repository settings
-func (store *Store) HelmUserRepository() portainer.HelmUserRepositoryService {
-	return store.HelmUserRepositoryService
+// License provides access to the License data management layer
+func (store *Store) License() portainer.LicenseRepository {
+	return store.LicenseService
 }
 
 // Registry gives access to the Registry data management layer
@@ -231,16 +236,16 @@ func (store *Store) Role() portainer.RoleService {
 	return store.RoleService
 }
 
+// S3Backup gives access to S3 backup settings and status
+func (store *Store) S3Backup() portainer.S3BackupService {
+	return store.S3BackupService
+}
+
 // Settings gives access to the Settings data management layer
 func (store *Store) Settings() portainer.SettingsService {
 	return store.SettingsService
 }
 
-// SSLSettings gives access to the SSL Settings data management layer
-func (store *Store) SSLSettings() portainer.SSLSettingsService {
-	return store.SSLSettingsService
-}
-
 // Stack gives access to the Stack data management layer
 func (store *Store) Stack() portainer.StackService {
 	return store.StackService

api/bolt/settings/settings.go

@@ -11,7 +11,7 @@ const (
 	settingsKey = "SETTINGS"
 )
 
-// Service represents a service for managing environment(endpoint) data.
+// Service represents a service for managing endpoint data.
 type Service struct {
 	connection *internal.DbConnection
 }

api/bolt/ssl/ssl.go

@@ -1,46 +0,0 @@
-package ssl
-
-import (
-	portainer "github.com/portainer/portainer/api"
-	"github.com/portainer/portainer/api/bolt/internal"
-)
-
-const (
-	// BucketName represents the name of the bucket where this service stores data.
-	BucketName = "ssl"
-	key        = "SSL"
-)
-
-// Service represents a service for managing ssl data.
-type Service struct {
-	connection *internal.DbConnection
-}
-
-// NewService creates a new instance of a service.
-func NewService(connection *internal.DbConnection) (*Service, error) {
-	err := internal.CreateBucket(connection, BucketName)
-	if err != nil {
-		return nil, err
-	}
-
-	return &Service{
-		connection: connection,
-	}, nil
-}
-
-// Settings retrieve the ssl settings object.
-func (service *Service) Settings() (*portainer.SSLSettings, error) {
-	var settings portainer.SSLSettings
-
-	err := internal.GetObject(service.connection, BucketName, []byte(key), &settings)
-	if err != nil {
-		return nil, err
-	}
-
-	return &settings, nil
-}
-
-// UpdateSettings persists a SSLSettings object.
-func (service *Service) UpdateSettings(settings *portainer.SSLSettings) error {
-	return internal.UpdateObject(service.connection, BucketName, []byte(key), settings)
-}

api/bolt/stack/stack.go

@@ -1,14 +1,11 @@
 package stack
 
 import (
-	"strings"
-
 	portainer "github.com/portainer/portainer/api"
 	"github.com/portainer/portainer/api/bolt/errors"
 	"github.com/portainer/portainer/api/bolt/internal"
 
 	"github.com/boltdb/bolt"
-	pkgerrors "github.com/pkg/errors"
 )
 
 const (
@@ -16,7 +13,7 @@ const (
 	BucketName = "stacks"
 )
 
-// Service represents a service for managing environment(endpoint) data.
+// Service represents a service for managing endpoint data.
 type Service struct {
 	connection *internal.DbConnection
 }
@@ -136,76 +133,3 @@ func (service *Service) DeleteStack(ID portainer.StackID) error {
 	identifier := internal.Itob(int(ID))
 	return internal.DeleteObject(service.connection, BucketName, identifier)
 }
-
-// StackByWebhookID returns a pointer to a stack object by webhook ID.
-// It returns nil, errors.ErrObjectNotFound if there's no stack associated with the webhook ID.
-func (service *Service) StackByWebhookID(id string) (*portainer.Stack, error) {
-	if id == "" {
-		return nil, pkgerrors.New("webhook ID can't be empty string")
-	}
-	var stack portainer.Stack
-	found := false
-
-	err := service.connection.View(func(tx *bolt.Tx) error {
-		bucket := tx.Bucket([]byte(BucketName))
-		cursor := bucket.Cursor()
-
-		for k, v := cursor.First(); k != nil; k, v = cursor.Next() {
-			var t struct {
-				AutoUpdate *struct {
-					WebhookID string `json:"Webhook"`
-				} `json:"AutoUpdate"`
-			}
-
-			err := internal.UnmarshalObject(v, &t)
-			if err != nil {
-				return err
-			}
-
-			if t.AutoUpdate != nil && strings.EqualFold(t.AutoUpdate.WebhookID, id) {
-				found = true
-				err := internal.UnmarshalObject(v, &stack)
-				if err != nil {
-					return err
-				}
-				break
-			}
-		}
-
-		return nil
-	})
-
-	if err != nil {
-		return nil, err
-	}
-	if !found {
-		return nil, errors.ErrObjectNotFound
-	}
-
-	return &stack, nil
-}
-
-// RefreshableStacks returns stacks that are configured for a periodic update
-func (service *Service) RefreshableStacks() ([]portainer.Stack, error) {
-	stacks := make([]portainer.Stack, 0)
-	err := service.connection.View(func(tx *bolt.Tx) error {
-		bucket := tx.Bucket([]byte(BucketName))
-		cursor := bucket.Cursor()
-
-		var stack portainer.Stack
-		for k, v := cursor.First(); k != nil; k, v = cursor.Next() {
-			err := internal.UnmarshalObject(v, &stack)
-			if err != nil {
-				return err
-			}
-
-			if stack.AutoUpdate != nil && stack.AutoUpdate.Interval != "" {
-				stacks = append(stacks, stack)
-			}
-		}
-
-		return nil
-	})
-
-	return stacks, err
-}

api/bolt/stack/tests/stack_test.go

@@ -1,111 +0,0 @@
-package tests
-
-import (
-	"testing"
-	"time"
-
-	"github.com/portainer/portainer/api/bolt"
-
-	bolterrors "github.com/portainer/portainer/api/bolt/errors"
-
-	"github.com/portainer/portainer/api/bolt/bolttest"
-
-	"github.com/gofrs/uuid"
-
-	"github.com/stretchr/testify/assert"
-
-	portainer "github.com/portainer/portainer/api"
-	"github.com/portainer/portainer/api/filesystem"
-)
-
-func newGuidString(t *testing.T) string {
-	uuid, err := uuid.NewV4()
-	assert.NoError(t, err)
-
-	return uuid.String()
-}
-
-type stackBuilder struct {
-	t     *testing.T
-	count int
-	store *bolt.Store
-}
-
-func TestService_StackByWebhookID(t *testing.T) {
-	if testing.Short() {
-		t.Skip("skipping test in short mode. Normally takes ~1s to run.")
-	}
-	store, teardown := bolttest.MustNewTestStore(true)
-	defer teardown()
-
-	b := stackBuilder{t: t, store: store}
-	b.createNewStack(newGuidString(t))
-	for i := 0; i < 10; i++ {
-		b.createNewStack("")
-	}
-	webhookID := newGuidString(t)
-	stack := b.createNewStack(webhookID)
-
-	// can find a stack by webhook ID
-	got, err := store.StackService.StackByWebhookID(webhookID)
-	assert.NoError(t, err)
-	assert.Equal(t, stack, *got)
-
-	// returns nil and object not found error if there's no stack associated with the webhook
-	got, err = store.StackService.StackByWebhookID(newGuidString(t))
-	assert.Nil(t, got)
-	assert.ErrorIs(t, err, bolterrors.ErrObjectNotFound)
-}
-
-func (b *stackBuilder) createNewStack(webhookID string) portainer.Stack {
-	b.count++
-	stack := portainer.Stack{
-		ID:           portainer.StackID(b.count),
-		Name:         "Name",
-		Type:         portainer.DockerComposeStack,
-		EndpointID:   2,
-		EntryPoint:   filesystem.ComposeFileDefaultName,
-		Env:          []portainer.Pair{{"Name1", "Value1"}},
-		Status:       portainer.StackStatusActive,
-		CreationDate: time.Now().Unix(),
-		ProjectPath:  "/tmp/project",
-		CreatedBy:    "test",
-	}
-
-	if webhookID == "" {
-		if b.count%2 == 0 {
-			stack.AutoUpdate = &portainer.StackAutoUpdate{
-				Interval: "",
-				Webhook:  "",
-			}
-		} // else keep AutoUpdate nil
-	} else {
-		stack.AutoUpdate = &portainer.StackAutoUpdate{Webhook: webhookID}
-	}
-
-	err := b.store.StackService.CreateStack(&stack)
-	assert.NoError(b.t, err)
-
-	return stack
-}
-
-func Test_RefreshableStacks(t *testing.T) {
-	if testing.Short() {
-		t.Skip("skipping test in short mode. Normally takes ~1s to run.")
-	}
-	store, teardown := bolttest.MustNewTestStore(true)
-	defer teardown()
-
-	staticStack := portainer.Stack{ID: 1}
-	stackWithWebhook := portainer.Stack{ID: 2, AutoUpdate: &portainer.StackAutoUpdate{Webhook: "webhook"}}
-	refreshableStack := portainer.Stack{ID: 3, AutoUpdate: &portainer.StackAutoUpdate{Interval: "1m"}}
-
-	for _, stack := range []*portainer.Stack{&staticStack, &stackWithWebhook, &refreshableStack} {
-		err := store.Stack().CreateStack(stack)
-		assert.NoError(t, err)
-	}
-
-	stacks, err := store.Stack().RefreshableStacks()
-	assert.NoError(t, err)
-	assert.ElementsMatch(t, []portainer.Stack{refreshableStack}, stacks)
-}

api/bolt/tag/tag.go

@@ -12,7 +12,7 @@ const (
 	BucketName = "tags"
 )
 
-// Service represents a service for managing environment(endpoint) data.
+// Service represents a service for managing endpoint data.
 type Service struct {
 	connection *internal.DbConnection
 }

api/bolt/team/team.go

@@ -1,13 +1,11 @@
 package team
 
 import (
-	"strings"
-
+	"github.com/boltdb/bolt"
 	portainer "github.com/portainer/portainer/api"
 	"github.com/portainer/portainer/api/bolt/errors"
 	"github.com/portainer/portainer/api/bolt/internal"
-
-	"github.com/boltdb/bolt"
+	"strings"
 )
 
 const (
@@ -15,7 +13,7 @@ const (
 	BucketName = "teams"
 )
 
-// Service represents a service for managing environment(endpoint) data.
+// Service represents a service for managing endpoint data.
 type Service struct {
 	connection *internal.DbConnection
 }

api/bolt/teammembership/teammembership.go

@@ -12,7 +12,7 @@ const (
 	BucketName = "team_membership"
 )
 
-// Service represents a service for managing environment(endpoint) data.
+// Service represents a service for managing endpoint data.
 type Service struct {
 	connection *internal.DbConnection
 }

api/bolt/tunnelserver/tunnelserver.go

@@ -11,7 +11,7 @@ const (
 	infoKey    = "INFO"
 )
 
-// Service represents a service for managing environment(endpoint) data.
+// Service represents a service for managing endpoint data.
 type Service struct {
 	connection *internal.DbConnection
 }

api/bolt/user/user.go

@@ -1,13 +1,11 @@
 package user
 
 import (
-	"strings"
-
+	"github.com/boltdb/bolt"
 	portainer "github.com/portainer/portainer/api"
 	"github.com/portainer/portainer/api/bolt/errors"
 	"github.com/portainer/portainer/api/bolt/internal"
-
-	"github.com/boltdb/bolt"
+	"strings"
 )
 
 const (
@@ -15,7 +13,7 @@ const (
 	BucketName = "users"
 )
 
-// Service represents a service for managing environment(endpoint) data.
+// Service represents a service for managing endpoint data.
 type Service struct {
 	connection *internal.DbConnection
 }

api/bolt/version/version.go

@@ -11,10 +11,11 @@ import (
 
 const (
 	// BucketName represents the name of the bucket where this service stores data.
-	BucketName  = "version"
-	versionKey  = "DB_VERSION"
-	instanceKey = "INSTANCE_ID"
-	editionKey  = "EDITION"
+	BucketName         = "version"
+	versionKey         = "DB_VERSION"
+	previousVersionKey = "PREVIOUS_DB_VERSION"
+	instanceKey        = "INSTANCE_ID"
+	editionKey         = "EDITION"
 )
 
 // Service represents a service to manage stored versions.
@@ -34,30 +35,6 @@ func NewService(connection *internal.DbConnection) (*Service, error) {
 	}, nil
 }
 
-// DBVersion retrieves the stored database version.
-func (service *Service) DBVersion() (int, error) {
-	var data []byte
-
-	err := service.connection.View(func(tx *bolt.Tx) error {
-		bucket := tx.Bucket([]byte(BucketName))
-
-		value := bucket.Get([]byte(versionKey))
-		if value == nil {
-			return errors.ErrObjectNotFound
-		}
-
-		data = make([]byte, len(value))
-		copy(data, value)
-
-		return nil
-	})
-	if err != nil {
-		return 0, err
-	}
-
-	return strconv.Atoi(string(data))
-}
-
 // Edition retrieves the stored portainer edition.
 func (service *Service) Edition() (portainer.SoftwareEdition, error) {
 	editionData, err := service.getKey(editionKey)
@@ -73,48 +50,54 @@ func (service *Service) Edition() (portainer.SoftwareEdition, error) {
 	return portainer.SoftwareEdition(edition), nil
 }
 
+// StoreEdition store the portainer edition.
+func (service *Service) StoreEdition(edition portainer.SoftwareEdition) error {
+	return service.setKey(editionKey, strconv.Itoa(int(edition)))
+}
+
+// PreviousDBVersion retrieves the stored database version.
+func (service *Service) PreviousDBVersion() (int, error) {
+	version, err := service.getKey(previousVersionKey)
+	if err != nil {
+		return 0, err
+	}
+
+	return strconv.Atoi(string(version))
+}
+
+// DBVersion retrieves the stored database version.
+func (service *Service) DBVersion() (int, error) {
+	version, err := service.getKey(versionKey)
+	if err != nil {
+		return 0, err
+	}
+
+	return strconv.Atoi(string(version))
+}
+
+// StorePreviousDBVersion store the database version.
+func (service *Service) StorePreviousDBVersion(version int) error {
+	return service.setKey(previousVersionKey, strconv.Itoa(version))
+}
+
 // StoreDBVersion store the database version.
 func (service *Service) StoreDBVersion(version int) error {
-	return service.connection.Update(func(tx *bolt.Tx) error {
-		bucket := tx.Bucket([]byte(BucketName))
-
-		data := []byte(strconv.Itoa(version))
-		return bucket.Put([]byte(versionKey), data)
-	})
+	return service.setKey(versionKey, strconv.Itoa(version))
 }
 
 // InstanceID retrieves the stored instance ID.
 func (service *Service) InstanceID() (string, error) {
-	var data []byte
-
-	err := service.connection.View(func(tx *bolt.Tx) error {
-		bucket := tx.Bucket([]byte(BucketName))
-
-		value := bucket.Get([]byte(instanceKey))
-		if value == nil {
-			return errors.ErrObjectNotFound
-		}
-
-		data = make([]byte, len(value))
-		copy(data, value)
-
-		return nil
-	})
+	instanceID, err := service.getKey(instanceKey)
 	if err != nil {
 		return "", err
 	}
 
-	return string(data), nil
+	return string(instanceID), nil
 }
 
 // StoreInstanceID store the instance ID.
 func (service *Service) StoreInstanceID(ID string) error {
-	return service.connection.Update(func(tx *bolt.Tx) error {
-		bucket := tx.Bucket([]byte(BucketName))
-
-		data := []byte(ID)
-		return bucket.Put([]byte(instanceKey), data)
-	})
+	return service.setKey(instanceKey, ID)
 }
 
 func (service *Service) getKey(key string) ([]byte, error) {

api/chisel/schedules.go

@@ -6,7 +6,7 @@ import (
 	portainer "github.com/portainer/portainer/api"
 )
 
-// AddEdgeJob register an EdgeJob inside the tunnel details associated to an environment(endpoint).
+// AddEdgeJob register an EdgeJob inside the tunnel details associated to an endpoint.
 func (service *Service) AddEdgeJob(endpointID portainer.EndpointID, edgeJob *portainer.EdgeJob) {
 	tunnel := service.GetTunnelDetails(endpointID)
 

api/chisel/service.go

@@ -141,7 +141,7 @@ func (service *Service) checkTunnels() {
 		}
 
 		elapsed := time.Since(tunnel.LastActivity)
-		log.Printf("[DEBUG] [chisel,monitoring] [endpoint_id: %s] [status: %s] [status_time_seconds: %f] [message: environment tunnel monitoring]", item.Key, tunnel.Status, elapsed.Seconds())
+		log.Printf("[DEBUG] [chisel,monitoring] [endpoint_id: %s] [status: %s] [status_time_seconds: %f] [message: endpoint tunnel monitoring]", item.Key, tunnel.Status, elapsed.Seconds())
 
 		if tunnel.Status == portainer.EdgeAgentManagementRequired && elapsed.Seconds() < requiredTimeout.Seconds() {
 			continue
@@ -156,19 +156,19 @@ func (service *Service) checkTunnels() {
 
 			endpointID, err := strconv.Atoi(item.Key)
 			if err != nil {
-				log.Printf("[ERROR] [chisel,snapshot,conversion] Invalid environment identifier (id: %s): %s", item.Key, err)
+				log.Printf("[ERROR] [chisel,snapshot,conversion] Invalid endpoint identifier (id: %s): %s", item.Key, err)
 			}
 
 			err = service.snapshotEnvironment(portainer.EndpointID(endpointID), tunnel.Port)
 			if err != nil {
-				log.Printf("[ERROR] [snapshot] Unable to snapshot Edge environment (id: %s): %s", item.Key, err)
+				log.Printf("[ERROR] [snapshot] Unable to snapshot Edge endpoint (id: %s): %s", item.Key, err)
 			}
 		}
 
 		if len(tunnel.Jobs) > 0 {
 			endpointID, err := strconv.Atoi(item.Key)
 			if err != nil {
-				log.Printf("[ERROR] [chisel,conversion] Invalid environment identifier (id: %s): %s", item.Key, err)
+				log.Printf("[ERROR] [chisel,conversion] Invalid endpoint identifier (id: %s): %s", item.Key, err)
 				continue
 			}
 

api/chisel/tunnel.go

@@ -38,7 +38,7 @@ func randomInt(min, max int) int {
 	return min + rand.Intn(max-min)
 }
 
-// GetTunnelDetails returns information about the tunnel associated to an environment(endpoint).
+// GetTunnelDetails returns information about the tunnel associated to an endpoint.
 func (service *Service) GetTunnelDetails(endpointID portainer.EndpointID) *portainer.TunnelDetails {
 	key := strconv.Itoa(int(endpointID))
 
@@ -56,7 +56,7 @@ func (service *Service) GetTunnelDetails(endpointID portainer.EndpointID) *porta
 	}
 }
 
-// SetTunnelStatusToActive update the status of the tunnel associated to the specified environment(endpoint).
+// SetTunnelStatusToActive update the status of the tunnel associated to the specified endpoint.
 // It sets the status to ACTIVE.
 func (service *Service) SetTunnelStatusToActive(endpointID portainer.EndpointID) {
 	tunnel := service.GetTunnelDetails(endpointID)
@@ -68,7 +68,7 @@ func (service *Service) SetTunnelStatusToActive(endpointID portainer.EndpointID)
 	service.tunnelDetailsMap.Set(key, tunnel)
 }
 
-// SetTunnelStatusToIdle update the status of the tunnel associated to the specified environment(endpoint).
+// SetTunnelStatusToIdle update the status of the tunnel associated to the specified endpoint.
 // It sets the status to IDLE.
 // It removes any existing credentials associated to the tunnel.
 func (service *Service) SetTunnelStatusToIdle(endpointID portainer.EndpointID) {
@@ -88,11 +88,11 @@ func (service *Service) SetTunnelStatusToIdle(endpointID portainer.EndpointID) {
 	service.tunnelDetailsMap.Set(key, tunnel)
 }
 
-// SetTunnelStatusToRequired update the status of the tunnel associated to the specified environment(endpoint).
+// SetTunnelStatusToRequired update the status of the tunnel associated to the specified endpoint.
 // It sets the status to REQUIRED.
 // If no port is currently associated to the tunnel, it will associate a random unused port to the tunnel
 // and generate temporary credentials that can be used to establish a reverse tunnel on that port.
-// Credentials are encrypted using the Edge ID associated to the environment(endpoint).
+// Credentials are encrypted using the Edge ID associated to the endpoint.
 func (service *Service) SetTunnelStatusToRequired(endpointID portainer.EndpointID) error {
 	tunnel := service.GetTunnelDetails(endpointID)
 

api/cli/cli.go

@@ -18,7 +18,7 @@ import (
 type Service struct{}
 
 var (
-	errInvalidEndpointProtocol       = errors.New("Invalid environment protocol: Portainer only supports unix://, npipe:// or tcp://")
+	errInvalidEndpointProtocol       = errors.New("Invalid endpoint protocol: Portainer only supports unix://, npipe:// or tcp://")
 	errSocketOrNamedPipeNotFound     = errors.New("Unable to locate Unix socket or named pipe")
 	errInvalidSnapshotInterval       = errors.New("Invalid snapshot interval")
 	errAdminPassExcludeAdminPassFile = errors.New("Cannot use --admin-password with --admin-password-file")
@@ -30,12 +30,11 @@ func (*Service) ParseFlags(version string) (*portainer.CLIFlags, error) {
 
 	flags := &portainer.CLIFlags{
 		Addr:                      kingpin.Flag("bind", "Address and port to serve Portainer").Default(defaultBindAddress).Short('p').String(),
-		AddrHTTPS:                 kingpin.Flag("bind-https", "Address and port to serve Portainer via https").Default(defaultHTTPSBindAddress).String(),
 		TunnelAddr:                kingpin.Flag("tunnel-addr", "Address to serve the tunnel server").Default(defaultTunnelServerAddress).String(),
 		TunnelPort:                kingpin.Flag("tunnel-port", "Port to serve the tunnel server").Default(defaultTunnelServerPort).String(),
 		Assets:                    kingpin.Flag("assets", "Path to the assets").Default(defaultAssetsDirectory).Short('a').String(),
 		Data:                      kingpin.Flag("data", "Path to the folder where the data is stored").Default(defaultDataDirectory).Short('d').String(),
-		EndpointURL:               kingpin.Flag("host", "Environment URL").Short('H').String(),
+		EndpointURL:               kingpin.Flag("host", "Endpoint URL").Short('H').String(),
 		EnableEdgeComputeFeatures: kingpin.Flag("edge-compute", "Enable Edge Compute features").Bool(),
 		NoAnalytics:               kingpin.Flag("no-analytics", "Disable Analytics in app (deprecated)").Bool(),
 		TLS:                       kingpin.Flag("tlsverify", "TLS support").Default(defaultTLS).Bool(),
@@ -43,11 +42,11 @@ func (*Service) ParseFlags(version string) (*portainer.CLIFlags, error) {
 		TLSCacert:                 kingpin.Flag("tlscacert", "Path to the CA").Default(defaultTLSCACertPath).String(),
 		TLSCert:                   kingpin.Flag("tlscert", "Path to the TLS certificate file").Default(defaultTLSCertPath).String(),
 		TLSKey:                    kingpin.Flag("tlskey", "Path to the TLS key").Default(defaultTLSKeyPath).String(),
-		HTTPDisabled:              kingpin.Flag("http-disabled", "Serve portainer only on https").Default(defaultHTTPDisabled).Bool(),
-		SSL:                       kingpin.Flag("ssl", "Secure Portainer instance using SSL (deprecated)").Default(defaultSSL).Bool(),
-		SSLCert:                   kingpin.Flag("sslcert", "Path to the SSL certificate used to secure the Portainer instance").String(),
-		SSLKey:                    kingpin.Flag("sslkey", "Path to the SSL key used to secure the Portainer instance").String(),
-		SnapshotInterval:          kingpin.Flag("snapshot-interval", "Duration between each environment snapshot job").Default(defaultSnapshotInterval).String(),
+		RollbackToCE:              kingpin.Flag("rollback-to-ce", "Rollback the database store to CE").Bool(),
+		SSL:                       kingpin.Flag("ssl", "Secure Portainer instance using SSL").Default(defaultSSL).Bool(),
+		SSLCert:                   kingpin.Flag("sslcert", "Path to the SSL certificate used to secure the Portainer instance").Default(defaultSSLCertPath).String(),
+		SSLKey:                    kingpin.Flag("sslkey", "Path to the SSL key used to secure the Portainer instance").Default(defaultSSLKeyPath).String(),
+		SnapshotInterval:          kingpin.Flag("snapshot-interval", "Duration between each endpoint snapshot job").Default(defaultSnapshotInterval).String(),
 		AdminPassword:             kingpin.Flag("admin-password", "Hashed admin password").String(),
 		AdminPasswordFile:         kingpin.Flag("admin-password-file", "Path to the file containing the password for the admin user").String(),
 		Labels:                    pairs(kingpin.Flag("hide-label", "Hide containers with a specific label in the UI").Short('l')),
@@ -94,10 +93,6 @@ func displayDeprecationWarnings(flags *portainer.CLIFlags) {
 	if *flags.NoAnalytics {
 		log.Println("Warning: The --no-analytics flag has been kept to allow migration of instances running a previous version of Portainer with this flag enabled, to version 2.0 where enabling this flag will have no effect.")
 	}
-
-	if *flags.SSL {
-		log.Println("Warning: SSL is enabled by default and there is no need for the --ssl flag. It has been kept to allow migration of instances running a previous version of Portainer with this flag enabled")
-	}
 }
 
 func validateEndpointURL(endpointURL string) error {

api/cli/confirm.go

@@ -0,0 +1,24 @@
+package cli
+
+import (
+	"bufio"
+	"log"
+	"os"
+	"strings"
+)
+
+// Confirm starts a rollback db cli application
+func Confirm(message string) (bool, error) {
+	log.Printf("%s [y/N]", message)
+
+	reader := bufio.NewReader(os.Stdin)
+	answer, err := reader.ReadString('\n')
+	if err != nil {
+		return false, err
+	}
+	answer = strings.Replace(answer, "\n", "", -1)
+	answer = strings.ToLower(answer)
+
+	return answer == "y" || answer == "yes", nil
+
+}

api/cli/defaults.go

@@ -4,7 +4,6 @@ package cli
 
 const (
 	defaultBindAddress         = ":9000"
-	defaultHTTPSBindAddress    = ":9443"
 	defaultTunnelServerAddress = "0.0.0.0"
 	defaultTunnelServerPort    = "8000"
 	defaultDataDirectory       = "/data"
@@ -14,7 +13,6 @@ const (
 	defaultTLSCACertPath       = "/certs/ca.pem"
 	defaultTLSCertPath         = "/certs/cert.pem"
 	defaultTLSKeyPath          = "/certs/key.pem"
-	defaultHTTPDisabled        = "false"
 	defaultSSL                 = "false"
 	defaultSSLCertPath         = "/certs/portainer.crt"
 	defaultSSLKeyPath          = "/certs/portainer.key"

api/cli/defaults_windows.go

@@ -2,7 +2,6 @@ package cli
 
 const (
 	defaultBindAddress         = ":9000"
-	defaultHTTPSBindAddress    = ":9443"
 	defaultTunnelServerAddress = "0.0.0.0"
 	defaultTunnelServerPort    = "8000"
 	defaultDataDirectory       = "C:\\data"
@@ -12,7 +11,6 @@ const (
 	defaultTLSCACertPath       = "C:\\certs\\ca.pem"
 	defaultTLSCertPath         = "C:\\certs\\cert.pem"
 	defaultTLSKeyPath          = "C:\\certs\\key.pem"
-	defaultHTTPDisabled        = "false"
 	defaultSSL                 = "false"
 	defaultSSLCertPath         = "C:\\certs\\portainer.crt"
 	defaultSSLKeyPath          = "C:\\certs\\portainer.key"

api/cmd/portainer/log.go

@@ -1,19 +0,0 @@
-package main
-
-import (
-	"log"
-
-	"github.com/sirupsen/logrus"
-)
-
-func configureLogger() {
-	logger := logrus.New() // logger is to implicitly substitute stdlib's log
-	log.SetOutput(logger.Writer())
-
-	formatter := &logrus.TextFormatter{DisableTimestamp: true, DisableLevelTruncation: true}
-	logger.SetFormatter(formatter)
-	logrus.SetFormatter(formatter)
-
-	logger.SetLevel(logrus.DebugLevel)
-	logrus.SetLevel(logrus.DebugLevel)
-}

api/cmd/portainer/main.go

@@ -13,7 +13,6 @@ import (
 	"github.com/portainer/portainer/api/crypto"
 	"github.com/portainer/portainer/api/docker"
 
-	"github.com/portainer/libhelm"
 	"github.com/portainer/portainer/api/exec"
 	"github.com/portainer/portainer/api/filesystem"
 	"github.com/portainer/portainer/api/git"
@@ -24,87 +23,97 @@ import (
 	"github.com/portainer/portainer/api/internal/authorization"
 	"github.com/portainer/portainer/api/internal/edge"
 	"github.com/portainer/portainer/api/internal/snapshot"
-	"github.com/portainer/portainer/api/internal/ssl"
 	"github.com/portainer/portainer/api/jwt"
 	"github.com/portainer/portainer/api/kubernetes"
 	kubecli "github.com/portainer/portainer/api/kubernetes/cli"
 	"github.com/portainer/portainer/api/ldap"
+	"github.com/portainer/portainer/api/libcompose"
+	"github.com/portainer/portainer/api/license"
 	"github.com/portainer/portainer/api/oauth"
-	"github.com/portainer/portainer/api/scheduler"
-	"github.com/portainer/portainer/api/stacks"
+	"github.com/portainer/portainer/api/useractivity"
 )
 
 func initCLI() *portainer.CLIFlags {
 	var cliService portainer.CLIService = &cli.Service{}
 	flags, err := cliService.ParseFlags(portainer.APIVersion)
 	if err != nil {
-		log.Fatalf("failed parsing flags: %v", err)
+		log.Fatalf("failed parsing flags: %s", err)
 	}
 
 	err = cliService.ValidateFlags(flags)
 	if err != nil {
-		log.Fatalf("failed validating flags:%v", err)
+		log.Fatalf("failed validating flags:%s", err)
 	}
 	return flags
 }
 
+func initUserActivityStore(dataStorePath string) portainer.UserActivityStore {
+	store, err := useractivity.NewUserActivityStore(dataStorePath)
+	if err != nil {
+		log.Fatalf("Failed initalizing user activity store: %s", err)
+	}
+
+	return store
+}
+
 func initFileService(dataStorePath string) portainer.FileService {
 	fileService, err := filesystem.NewService(dataStorePath, "")
 	if err != nil {
-		log.Fatalf("failed creating file service: %v", err)
+		log.Fatalf("failed creating file service: %s", err)
 	}
 	return fileService
 }
 
-func initDataStore(dataStorePath string, fileService portainer.FileService, shutdownCtx context.Context) portainer.DataStore {
+func initDataStore(dataStorePath string, rollback bool, fileService portainer.FileService) portainer.DataStore {
 	store, err := bolt.NewStore(dataStorePath, fileService)
 	if err != nil {
-		log.Fatalf("failed creating data store: %v", err)
+		log.Fatalf("failed creating data store: %s", err)
 	}
 
 	err = store.Open()
 	if err != nil {
-		log.Fatalf("failed opening store: %v", err)
+		log.Fatalf("failed opening store: %s", err)
 	}
 
 	err = store.Init()
 	if err != nil {
-		log.Fatalf("failed initializing data store: %v", err)
+		log.Fatalf("failed initializing data store: %s", err)
+	}
+
+	if rollback {
+		err := store.RollbackToCE()
+		if err != nil {
+			log.Fatalf("failed rolling back to CE: %s", err)
+		}
+
+		log.Println("Exiting rollback")
+		os.Exit(0)
+		return nil
 	}
 
 	err = store.MigrateData(false)
 	if err != nil {
-		log.Fatalf("failed migration: %v", err)
+		log.Fatalf("failed migration: %s", err)
 	}
-
-	go shutdownDatastore(shutdownCtx, store)
 	return store
 }
 
-func shutdownDatastore(shutdownCtx context.Context, datastore portainer.DataStore) {
-	<-shutdownCtx.Done()
-	datastore.Close()
-}
-
-func initComposeStackManager(assetsPath string, configPath string, reverseTunnelService portainer.ReverseTunnelService, proxyManager *proxy.Manager) portainer.ComposeStackManager {
-	composeWrapper, err := exec.NewComposeStackManager(assetsPath, configPath, proxyManager)
+func initComposeStackManager(assetsPath string, dataStorePath string, reverseTunnelService portainer.ReverseTunnelService, proxyManager *proxy.Manager) portainer.ComposeStackManager {
+	composeWrapper, err := exec.NewComposeStackManager(assetsPath, dataStorePath, proxyManager)
 	if err != nil {
-		log.Fatalf("failed creating compose manager: %s", err)
+		log.Printf("[INFO] [main,compose] [message: falling-back to libcompose] [error: %s]", err)
+		return libcompose.NewComposeStackManager(dataStorePath, reverseTunnelService)
 	}
 
 	return composeWrapper
 }
 
-func initSwarmStackManager(assetsPath string, configPath string, signatureService portainer.DigitalSignatureService, fileService portainer.FileService, reverseTunnelService portainer.ReverseTunnelService) (portainer.SwarmStackManager, error) {
-	return exec.NewSwarmStackManager(assetsPath, configPath, signatureService, fileService, reverseTunnelService)
+func initSwarmStackManager(assetsPath string, dataStorePath string, signatureService portainer.DigitalSignatureService, fileService portainer.FileService, reverseTunnelService portainer.ReverseTunnelService) (portainer.SwarmStackManager, error) {
+	return exec.NewSwarmStackManager(assetsPath, dataStorePath, signatureService, fileService, reverseTunnelService)
 }
 
-func initKubernetesDeployer(kubernetesTokenCacheManager *kubeproxy.TokenCacheManager, kubernetesClientFactory *kubecli.ClientFactory, dataStore portainer.DataStore, reverseTunnelService portainer.ReverseTunnelService, signatureService portainer.DigitalSignatureService, assetsPath string) portainer.KubernetesDeployer {
-	return exec.NewKubernetesDeployer(kubernetesTokenCacheManager, kubernetesClientFactory, dataStore, reverseTunnelService, signatureService, assetsPath)
-}
-
-func initHelmPackageManager(assetsPath string) (libhelm.HelmPackageManager, error) {
-	return libhelm.NewHelmPackageManager(libhelm.HelmConfig{BinaryPath: assetsPath})
+func initKubernetesDeployer(dataStore portainer.DataStore, reverseTunnelService portainer.ReverseTunnelService, signatureService portainer.DigitalSignatureService, assetsPath string) portainer.KubernetesDeployer {
+	return exec.NewKubernetesDeployer(dataStore, reverseTunnelService, signatureService, assetsPath)
 }
 
 func initJWTService(dataStore portainer.DataStore) (portainer.JWTService, error) {
@@ -113,11 +122,11 @@ func initJWTService(dataStore portainer.DataStore) (portainer.JWTService, error)
 		return nil, err
 	}
 
-	if settings.UserSessionTimeout == "" {
-		settings.UserSessionTimeout = portainer.DefaultUserSessionTimeout
-		dataStore.Settings().UpdateSettings(settings)
+	userSessionTimeout := settings.UserSessionTimeout
+	if userSessionTimeout == "" {
+		userSessionTimeout = portainer.DefaultUserSessionTimeout
 	}
-	jwtService, err := jwt.NewService(settings.UserSessionTimeout, dataStore)
+	jwtService, err := jwt.NewService(userSessionTimeout)
 	if err != nil {
 		return nil, err
 	}
@@ -144,29 +153,12 @@ func initGitService() portainer.GitService {
 	return git.NewService()
 }
 
-func initSSLService(addr, dataPath, certPath, keyPath string, fileService portainer.FileService, dataStore portainer.DataStore, shutdownTrigger context.CancelFunc) (*ssl.Service, error) {
-	slices := strings.Split(addr, ":")
-	host := slices[0]
-	if host == "" {
-		host = "0.0.0.0"
-	}
-
-	sslService := ssl.NewService(fileService, dataStore, shutdownTrigger)
-
-	err := sslService.Init(host, certPath, keyPath)
-	if err != nil {
-		return nil, err
-	}
-
-	return sslService, nil
-}
-
 func initDockerClientFactory(signatureService portainer.DigitalSignatureService, reverseTunnelService portainer.ReverseTunnelService) *docker.ClientFactory {
 	return docker.NewClientFactory(signatureService, reverseTunnelService)
 }
 
-func initKubernetesClientFactory(signatureService portainer.DigitalSignatureService, reverseTunnelService portainer.ReverseTunnelService, instanceID string, dataStore portainer.DataStore) *kubecli.ClientFactory {
-	return kubecli.NewClientFactory(signatureService, reverseTunnelService, instanceID, dataStore)
+func initKubernetesClientFactory(signatureService portainer.DigitalSignatureService, reverseTunnelService portainer.ReverseTunnelService, instanceID string) *kubecli.ClientFactory {
+	return kubecli.NewClientFactory(signatureService, reverseTunnelService, instanceID)
 }
 
 func initSnapshotService(snapshotInterval string, dataStore portainer.DataStore, dockerClientFactory *docker.ClientFactory, kubernetesClientFactory *kubecli.ClientFactory, shutdownCtx context.Context) (portainer.SnapshotService, error) {
@@ -181,10 +173,9 @@ func initSnapshotService(snapshotInterval string, dataStore portainer.DataStore,
 	return snapshotService, nil
 }
 
-func initStatus(instanceID string) *portainer.Status {
+func initStatus(flags *portainer.CLIFlags) *portainer.Status {
 	return &portainer.Status{
-		Version:    portainer.APIVersion,
-		InstanceID: instanceID,
+		Version: portainer.APIVersion,
 	}
 }
 
@@ -199,6 +190,7 @@ func updateSettingsFromFlags(dataStore portainer.DataStore, flags *portainer.CLI
 	settings.EnableEdgeComputeFeatures = *flags.EnableEdgeComputeFeatures
 	settings.EnableTelemetry = true
 	settings.OAuthSettings.SSO = true
+	settings.OAuthSettings.HideInternalAuth = true
 
 	if *flags.Templates != "" {
 		settings.TemplatesURL = *flags.Templates
@@ -208,26 +200,7 @@ func updateSettingsFromFlags(dataStore portainer.DataStore, flags *portainer.CLI
 		settings.BlackListedLabels = *flags.Labels
 	}
 
-	err = dataStore.Settings().UpdateSettings(settings)
-	if err != nil {
-		return err
-	}
-
-	httpEnabled := !*flags.HTTPDisabled
-
-	sslSettings, err := dataStore.SSLSettings().Settings()
-	if err != nil {
-		return err
-	}
-
-	sslSettings.HTTPEnabled = httpEnabled
-
-	err = dataStore.SSLSettings().UpdateSettings(sslSettings)
-	if err != nil {
-		return err
-	}
-
-	return nil
+	return dataStore.Settings().UpdateSettings(settings)
 }
 
 func loadAndParseKeyPair(fileService portainer.FileService, signatureService portainer.DigitalSignatureService) error {
@@ -250,7 +223,7 @@ func generateAndStoreKeyPair(fileService portainer.FileService, signatureService
 func initKeyPair(fileService portainer.FileService, signatureService portainer.DigitalSignatureService) error {
 	existingKeyPair, err := fileService.KeyPairFilesExist()
 	if err != nil {
-		log.Fatalf("failed checking for existing key pair: %v", err)
+		log.Fatalf("failed checking for existing key pair: %s", err)
 	}
 
 	if existingKeyPair {
@@ -321,7 +294,7 @@ func createTLSSecuredEndpoint(flags *portainer.CLIFlags, dataStore portainer.Dat
 
 	err := snapshotService.SnapshotEndpoint(endpoint)
 	if err != nil {
-		log.Printf("http error: environment snapshot error (environment=%s, URL=%s) (err=%s)\n", endpoint.Name, endpoint.URL, err)
+		log.Printf("http error: endpoint snapshot error (endpoint=%s, URL=%s) (err=%s)\n", endpoint.Name, endpoint.URL, err)
 	}
 
 	return dataStore.Endpoint().CreateEndpoint(endpoint)
@@ -367,7 +340,7 @@ func createUnsecuredEndpoint(endpointURL string, dataStore portainer.DataStore,
 
 	err := snapshotService.SnapshotEndpoint(endpoint)
 	if err != nil {
-		log.Printf("http error: environment snapshot error (environment=%s, URL=%s) (err=%s)\n", endpoint.Name, endpoint.URL, err)
+		log.Printf("http error: endpoint snapshot error (endpoint=%s, URL=%s) (err=%s)\n", endpoint.Name, endpoint.URL, err)
 	}
 
 	return dataStore.Endpoint().CreateEndpoint(endpoint)
@@ -384,7 +357,7 @@ func initEndpoint(flags *portainer.CLIFlags, dataStore portainer.DataStore, snap
 	}
 
 	if len(endpoints) > 0 {
-		log.Println("Instance already has defined environments. Skipping the environment defined via CLI.")
+		log.Println("Instance already has defined endpoints. Skipping the endpoint defined via CLI.")
 		return nil
 	}
 
@@ -399,15 +372,16 @@ func buildServer(flags *portainer.CLIFlags) portainer.Server {
 
 	fileService := initFileService(*flags.Data)
 
-	dataStore := initDataStore(*flags.Data, fileService, shutdownCtx)
-
-	if err := dataStore.CheckCurrentEdition(); err != nil {
-		log.Fatal(err)
-	}
+	dataStore := initDataStore(*flags.Data, *flags.RollbackToCE, fileService)
 
 	jwtService, err := initJWTService(dataStore)
 	if err != nil {
-		log.Fatalf("failed initializing JWT service: %v", err)
+		log.Fatalf("failed initializing JWT service: %s", err)
+	}
+
+	licenseService := license.NewService(dataStore.License(), shutdownCtx)
+	if err = licenseService.Init(); err != nil {
+		log.Fatalf("failed initializing license service: %s", err)
 	}
 
 	ldapService := initLDAPService()
@@ -420,90 +394,72 @@ func buildServer(flags *portainer.CLIFlags) portainer.Server {
 
 	digitalSignatureService := initDigitalSignatureService()
 
-	sslService, err := initSSLService(*flags.AddrHTTPS, *flags.Data, *flags.SSLCert, *flags.SSLKey, fileService, dataStore, shutdownTrigger)
-	if err != nil {
-		log.Fatal(err)
-	}
-
-	sslSettings, err := sslService.GetSSLSettings()
-	if err != nil {
-		log.Fatalf("failed to get ssl settings: %s", err)
-	}
-
 	err = initKeyPair(fileService, digitalSignatureService)
 	if err != nil {
-		log.Fatalf("failed initializing key pai: %v", err)
+		log.Fatalf("failed initializing key pair: %s", err)
 	}
 
 	reverseTunnelService := chisel.NewService(dataStore, shutdownCtx)
 
 	instanceID, err := dataStore.Version().InstanceID()
 	if err != nil {
-		log.Fatalf("failed getting instance id: %v", err)
+		log.Fatalf("failed to get datastore version: %s", err)
 	}
 
 	dockerClientFactory := initDockerClientFactory(digitalSignatureService, reverseTunnelService)
-	kubernetesClientFactory := initKubernetesClientFactory(digitalSignatureService, reverseTunnelService, instanceID, dataStore)
+	kubernetesClientFactory := initKubernetesClientFactory(digitalSignatureService, reverseTunnelService, instanceID)
 
 	snapshotService, err := initSnapshotService(*flags.SnapshotInterval, dataStore, dockerClientFactory, kubernetesClientFactory, shutdownCtx)
 	if err != nil {
-		log.Fatalf("failed initializing snapshot service: %v", err)
+		log.Fatalf("failed initializing snapshot service: %s", err)
 	}
 	snapshotService.Start()
 
 	authorizationService := authorization.NewService(dataStore)
 	authorizationService.K8sClientFactory = kubernetesClientFactory
 
-	kubernetesTokenCacheManager := kubeproxy.NewTokenCacheManager()
-
-	kubeConfigService := kubernetes.NewKubeConfigCAService(*flags.AddrHTTPS, sslSettings.CertPath)
-
-	proxyManager := proxy.NewManager(dataStore, digitalSignatureService, reverseTunnelService, dockerClientFactory, kubernetesClientFactory, kubernetesTokenCacheManager)
-
-	dockerConfigPath := fileService.GetDockerConfigPath()
-
-	composeStackManager := initComposeStackManager(*flags.Assets, dockerConfigPath, reverseTunnelService, proxyManager)
-
-	swarmStackManager, err := initSwarmStackManager(*flags.Assets, dockerConfigPath, digitalSignatureService, fileService, reverseTunnelService)
+	swarmStackManager, err := initSwarmStackManager(*flags.Assets, *flags.Data, digitalSignatureService, fileService, reverseTunnelService)
 	if err != nil {
 		log.Fatalf("failed initializing swarm stack manager: %s", err)
 	}
+	kubernetesTokenCacheManager := kubeproxy.NewTokenCacheManager()
 
-	kubernetesDeployer := initKubernetesDeployer(kubernetesTokenCacheManager, kubernetesClientFactory, dataStore, reverseTunnelService, digitalSignatureService, *flags.Assets)
+	userActivityStore := initUserActivityStore(*flags.Data)
 
-	helmPackageManager, err := initHelmPackageManager(*flags.Assets)
-	if err != nil {
-		log.Fatalf("failed initializing helm package manager: %s", err)
-	}
+	proxyManager := proxy.NewManager(dataStore, digitalSignatureService, reverseTunnelService, dockerClientFactory, kubernetesClientFactory, kubernetesTokenCacheManager, authorizationService, userActivityStore)
+
+	composeStackManager := initComposeStackManager(*flags.Assets, *flags.Data, reverseTunnelService, proxyManager)
+
+	kubernetesDeployer := initKubernetesDeployer(dataStore, reverseTunnelService, digitalSignatureService, *flags.Assets)
 
 	if dataStore.IsNew() {
 		err = updateSettingsFromFlags(dataStore, flags)
 		if err != nil {
-			log.Fatalf("failed updating settings from flags: %v", err)
+			log.Fatalf("failed updating settings from flags: %s", err)
 		}
 	}
 
 	err = edge.LoadEdgeJobs(dataStore, reverseTunnelService)
 	if err != nil {
-		log.Fatalf("failed loading edge jobs from database: %v", err)
+		log.Fatalf("failed loading edge jobs from database: %s", err)
 	}
 
-	applicationStatus := initStatus(instanceID)
+	applicationStatus := initStatus(flags)
 
 	err = initEndpoint(flags, dataStore, snapshotService)
 	if err != nil {
-		log.Fatalf("failed initializing environment: %v", err)
+		log.Fatalf("failed initializing endpoint: %s", err)
 	}
 
 	adminPasswordHash := ""
 	if *flags.AdminPasswordFile != "" {
 		content, err := fileService.GetFileContent(*flags.AdminPasswordFile)
 		if err != nil {
-			log.Fatalf("failed getting admin password file: %v", err)
+			log.Fatalf("failed getting admin password file: %s", err)
 		}
 		adminPasswordHash, err = cryptoService.Hash(strings.TrimSuffix(string(content), "\n"))
 		if err != nil {
-			log.Fatalf("failed hashing admin password: %v", err)
+			log.Fatalf("failed hashing admin password: %s", err)
 		}
 	} else if *flags.AdminPassword != "" {
 		adminPasswordHash = *flags.AdminPassword
@@ -512,19 +468,20 @@ func buildServer(flags *portainer.CLIFlags) portainer.Server {
 	if adminPasswordHash != "" {
 		users, err := dataStore.User().UsersByRole(portainer.AdministratorRole)
 		if err != nil {
-			log.Fatalf("failed getting admin user: %v", err)
+			log.Fatalf("failed getting admin user: %s", err)
 		}
 
 		if len(users) == 0 {
 			log.Println("Created admin user with the given password.")
 			user := &portainer.User{
-				Username: "admin",
-				Role:     portainer.AdministratorRole,
-				Password: adminPasswordHash,
+				Username:                "admin",
+				Role:                    portainer.AdministratorRole,
+				Password:                adminPasswordHash,
+				PortainerAuthorizations: authorization.DefaultPortainerAuthorizations(),
 			}
 			err := dataStore.User().CreateUser(user)
 			if err != nil {
-				log.Fatalf("failed creating admin user: %v", err)
+				log.Fatalf("failed creating admin user: %s", err)
 			}
 		} else {
 			log.Println("Instance already has an administrator user defined. Skipping admin password related flags.")
@@ -536,28 +493,22 @@ func buildServer(flags *portainer.CLIFlags) portainer.Server {
 		log.Fatalf("failed starting tunnel server: %s", err)
 	}
 
-	sslDBSettings, err := dataStore.SSLSettings().Settings()
+	err = licenseService.Start()
 	if err != nil {
-		log.Fatalf("failed to fetch ssl settings from DB")
+		log.Fatalf("failed starting license service: %s", err)
 	}
 
-	scheduler := scheduler.NewScheduler(shutdownCtx)
-	stackDeployer := stacks.NewStackDeployer(swarmStackManager, composeStackManager)
-	stacks.StartStackSchedules(scheduler, stackDeployer, dataStore, gitService)
-
 	return &http.Server{
 		AuthorizationService:        authorizationService,
 		ReverseTunnelService:        reverseTunnelService,
 		Status:                      applicationStatus,
 		BindAddress:                 *flags.Addr,
-		BindAddressHTTPS:            *flags.AddrHTTPS,
-		HTTPEnabled:                 sslDBSettings.HTTPEnabled,
 		AssetsPath:                  *flags.Assets,
 		DataStore:                   dataStore,
+		LicenseService:              licenseService,
 		SwarmStackManager:           swarmStackManager,
 		ComposeStackManager:         composeStackManager,
 		KubernetesDeployer:          kubernetesDeployer,
-		HelmPackageManager:          helmPackageManager,
 		CryptoService:               cryptoService,
 		JWTService:                  jwtService,
 		FileService:                 fileService,
@@ -566,28 +517,26 @@ func buildServer(flags *portainer.CLIFlags) portainer.Server {
 		GitService:                  gitService,
 		ProxyManager:                proxyManager,
 		KubernetesTokenCacheManager: kubernetesTokenCacheManager,
-		KubeConfigService:           kubeConfigService,
 		SignatureService:            digitalSignatureService,
 		SnapshotService:             snapshotService,
-		SSLService:                  sslService,
+		SSL:                         *flags.SSL,
+		SSLCert:                     *flags.SSLCert,
+		SSLKey:                      *flags.SSLKey,
 		DockerClientFactory:         dockerClientFactory,
+		UserActivityStore:           userActivityStore,
 		KubernetesClientFactory:     kubernetesClientFactory,
-		Scheduler:                   scheduler,
 		ShutdownCtx:                 shutdownCtx,
 		ShutdownTrigger:             shutdownTrigger,
-		StackDeployer:               stackDeployer,
 	}
 }
 
 func main() {
 	flags := initCLI()
 
-	configureLogger()
-
 	for {
 		server := buildServer(flags)
-		log.Printf("[INFO] [cmd,main] Starting Portainer version %s\n", portainer.APIVersion)
+		log.Printf("Starting Portainer %s on %s\n", portainer.APIVersion, *flags.Addr)
 		err := server.Start()
-		log.Printf("[INFO] [cmd,main] Http server exited: %s\n", err)
+		log.Printf("Http server exited: %s\n", err)
 	}
 }

api/crypto/aes_test.go

@@ -7,12 +7,11 @@ import (
 	"path/filepath"
 	"testing"
 
-	"github.com/docker/docker/pkg/ioutils"
 	"github.com/stretchr/testify/assert"
 )
 
 func Test_encryptAndDecrypt_withTheSamePassword(t *testing.T) {
-	tmpdir, _ := ioutils.TempDir("", "encrypt")
+	tmpdir, _ := ioutil.TempDir("", "encrypt")
 	defer os.RemoveAll(tmpdir)
 
 	var (
@@ -52,7 +51,7 @@ func Test_encryptAndDecrypt_withTheSamePassword(t *testing.T) {
 }
 
 func Test_encryptAndDecrypt_withEmptyPassword(t *testing.T) {
-	tmpdir, _ := ioutils.TempDir("", "encrypt")
+	tmpdir, _ := ioutil.TempDir("", "encrypt")
 	defer os.RemoveAll(tmpdir)
 
 	var (
@@ -92,7 +91,7 @@ func Test_encryptAndDecrypt_withEmptyPassword(t *testing.T) {
 }
 
 func Test_decryptWithDifferentPassphrase_shouldProduceWrongResult(t *testing.T) {
-	tmpdir, _ := ioutils.TempDir("", "encrypt")
+	tmpdir, _ := ioutil.TempDir("", "encrypt")
 	defer os.RemoveAll(tmpdir)
 
 	var (

api/crypto/ecdsa.go

@@ -22,7 +22,7 @@ const (
 )
 
 // ECDSAService is a service used to create digital signatures when communicating with
-// an agent based environment(endpoint). It will automatically generates a key pair using ECDSA or
+// an agent based environment. It will automatically generates a key pair using ECDSA or
 // can also reuse an existing ECDSA key pair.
 type ECDSAService struct {
 	privateKey    *ecdsa.PrivateKey

api/dev.sh

@@ -0,0 +1,3 @@
+#! /bin/sh
+
+go run -v -ldflags="-X github.com/portainer/liblicense.LicenseServerBaseURL=http://localhost:8080" cmd/portainer/main.go --data=./tmp/data

api/docker/client.go

@@ -34,15 +34,15 @@ func NewClientFactory(signatureService portainer.DigitalSignatureService, revers
 }
 
 // createClient is a generic function to create a Docker client based on
-// a specific environment(endpoint) configuration. The nodeName parameter can be used
-// with an agent enabled environment(endpoint) to target a specific node in an agent cluster.
+// a specific endpoint configuration. The nodeName parameter can be used
+// with an agent enabled endpoint to target a specific node in an agent cluster.
 func (factory *ClientFactory) CreateClient(endpoint *portainer.Endpoint, nodeName string) (*client.Client, error) {
 	if endpoint.Type == portainer.AzureEnvironment {
 		return nil, errUnsupportedEnvironmentType
 	} else if endpoint.Type == portainer.AgentOnDockerEnvironment {
 		return createAgentClient(endpoint, factory.signatureService, nodeName)
 	} else if endpoint.Type == portainer.EdgeAgentOnDockerEnvironment {
-		return createEdgeClient(endpoint, factory.signatureService, factory.reverseTunnelService, nodeName)
+		return createEdgeClient(endpoint, factory.reverseTunnelService, nodeName)
 	}
 
 	if strings.HasPrefix(endpoint.URL, "unix://") || strings.HasPrefix(endpoint.URL, "npipe://") {
@@ -71,22 +71,13 @@ func createTCPClient(endpoint *portainer.Endpoint) (*client.Client, error) {
 	)
 }
 
-func createEdgeClient(endpoint *portainer.Endpoint, signatureService portainer.DigitalSignatureService, reverseTunnelService portainer.ReverseTunnelService, nodeName string) (*client.Client, error) {
+func createEdgeClient(endpoint *portainer.Endpoint, reverseTunnelService portainer.ReverseTunnelService, nodeName string) (*client.Client, error) {
 	httpCli, err := httpClient(endpoint)
 	if err != nil {
 		return nil, err
 	}
 
-	signature, err := signatureService.CreateSignature(portainer.PortainerAgentSignatureMessage)
-	if err != nil {
-		return nil, err
-	}
-
-	headers := map[string]string{
-		portainer.PortainerAgentPublicKeyHeader: signatureService.EncodedPublicKey(),
-		portainer.PortainerAgentSignatureHeader: signature,
-	}
-
+	headers := map[string]string{}
 	if nodeName != "" {
 		headers[portainer.PortainerAgentTargetHeader] = nodeName
 	}

api/docker/errors.go

@@ -4,5 +4,5 @@ import "errors"
 
 // Docker errors
 var (
-	ErrUnableToPingEndpoint = errors.New("Unable to communicate with the environment")
+	ErrUnableToPingEndpoint = errors.New("Unable to communicate with the endpoint")
 )

api/docker/snapshot.go

@@ -12,7 +12,7 @@ import (
 	"github.com/portainer/portainer/api"
 )
 
-// Snapshotter represents a service used to create environment(endpoint) snapshots
+// Snapshotter represents a service used to create endpoint snapshots
 type Snapshotter struct {
 	clientFactory *ClientFactory
 }
@@ -24,7 +24,7 @@ func NewSnapshotter(clientFactory *ClientFactory) *Snapshotter {
 	}
 }
 
-// CreateSnapshot creates a snapshot of a specific Docker environment(endpoint)
+// CreateSnapshot creates a snapshot of a specific Docker endpoint
 func (snapshotter *Snapshotter) CreateSnapshot(endpoint *portainer.Endpoint) (*portainer.DockerSnapshot, error) {
 	cli, err := snapshotter.clientFactory.CreateClient(endpoint, "")
 	if err != nil {
@@ -47,44 +47,44 @@ func snapshot(cli *client.Client, endpoint *portainer.Endpoint) (*portainer.Dock
 
 	err = snapshotInfo(snapshot, cli)
 	if err != nil {
-		log.Printf("[WARN] [docker,snapshot] [message: unable to snapshot engine information] [environment: %s] [err: %s]", endpoint.Name, err)
+		log.Printf("[WARN] [docker,snapshot] [message: unable to snapshot engine information] [endpoint: %s] [err: %s]", endpoint.Name, err)
 	}
 
 	if snapshot.Swarm {
 		err = snapshotSwarmServices(snapshot, cli)
 		if err != nil {
-			log.Printf("[WARN] [docker,snapshot] [message: unable to snapshot Swarm services] [environment: %s] [err: %s]", endpoint.Name, err)
+			log.Printf("[WARN] [docker,snapshot] [message: unable to snapshot Swarm services] [endpoint: %s] [err: %s]", endpoint.Name, err)
 		}
 
 		err = snapshotNodes(snapshot, cli)
 		if err != nil {
-			log.Printf("[WARN] [docker,snapshot] [message: unable to snapshot Swarm nodes] [environment: %s] [err: %s]", endpoint.Name, err)
+			log.Printf("[WARN] [docker,snapshot] [message: unable to snapshot Swarm nodes] [endpoint: %s] [err: %s]", endpoint.Name, err)
 		}
 	}
 
 	err = snapshotContainers(snapshot, cli)
 	if err != nil {
-		log.Printf("[WARN] [docker,snapshot] [message: unable to snapshot containers] [environment: %s] [err: %s]", endpoint.Name, err)
+		log.Printf("[WARN] [docker,snapshot] [message: unable to snapshot containers] [endpoint: %s] [err: %s]", endpoint.Name, err)
 	}
 
 	err = snapshotImages(snapshot, cli)
 	if err != nil {
-		log.Printf("[WARN] [docker,snapshot] [message: unable to snapshot images] [environment: %s] [err: %s]", endpoint.Name, err)
+		log.Printf("[WARN] [docker,snapshot] [message: unable to snapshot images] [endpoint: %s] [err: %s]", endpoint.Name, err)
 	}
 
 	err = snapshotVolumes(snapshot, cli)
 	if err != nil {
-		log.Printf("[WARN] [docker,snapshot] [message: unable to snapshot volumes] [environment: %s] [err: %s]", endpoint.Name, err)
+		log.Printf("[WARN] [docker,snapshot] [message: unable to snapshot volumes] [endpoint: %s] [err: %s]", endpoint.Name, err)
 	}
 
 	err = snapshotNetworks(snapshot, cli)
 	if err != nil {
-		log.Printf("[WARN] [docker,snapshot] [message: unable to snapshot networks] [environment: %s] [err: %s]", endpoint.Name, err)
+		log.Printf("[WARN] [docker,snapshot] [message: unable to snapshot networks] [endpoint: %s] [err: %s]", endpoint.Name, err)
 	}
 
 	err = snapshotVersion(snapshot, cli)
 	if err != nil {
-		log.Printf("[WARN] [docker,snapshot] [message: unable to snapshot engine version] [environment: %s] [err: %s]", endpoint.Name, err)
+		log.Printf("[WARN] [docker,snapshot] [message: unable to snapshot engine version] [endpoint: %s] [err: %s]", endpoint.Name, err)
 	}
 
 	snapshot.Time = time.Now().Unix()

api/exec/compose_stack.go

@@ -1,18 +1,12 @@
 package exec
 
 import (
-	"context"
 	"fmt"
 	"os"
 	"path"
-	"path/filepath"
-	"regexp"
 	"strings"
 
-	"github.com/pkg/errors"
-
-	libstack "github.com/portainer/docker-compose-wrapper"
-	"github.com/portainer/docker-compose-wrapper/compose"
+	wrapper "github.com/portainer/docker-compose-wrapper"
 
 	portainer "github.com/portainer/portainer/api"
 	"github.com/portainer/portainer/api/http/proxy"
@@ -21,33 +15,35 @@ import (
 
 // ComposeStackManager is a wrapper for docker-compose binary
 type ComposeStackManager struct {
-	deployer     libstack.Deployer
+	wrapper      *wrapper.ComposeWrapper
+	dataPath     string
 	proxyManager *proxy.Manager
 }
 
 // NewComposeStackManager returns a docker-compose wrapper if corresponding binary present, otherwise nil
-func NewComposeStackManager(binaryPath string, configPath string, proxyManager *proxy.Manager) (*ComposeStackManager, error) {
-	deployer, err := compose.NewComposeDeployer(binaryPath, configPath)
+func NewComposeStackManager(binaryPath string, dataPath string, proxyManager *proxy.Manager) (*ComposeStackManager, error) {
+	wrap, err := wrapper.NewComposeWrapper(binaryPath)
 	if err != nil {
 		return nil, err
 	}
 
 	return &ComposeStackManager{
-		deployer:     deployer,
+		wrapper:      wrap,
 		proxyManager: proxyManager,
+		dataPath:     dataPath,
 	}, nil
 }
 
 // ComposeSyntaxMaxVersion returns the maximum supported version of the docker compose syntax
-func (manager *ComposeStackManager) ComposeSyntaxMaxVersion() string {
+func (w *ComposeStackManager) ComposeSyntaxMaxVersion() string {
 	return portainer.ComposeSyntaxMaxVersion
 }
 
 // Up builds, (re)creates and starts containers in the background. Wraps `docker-compose up -d` command
-func (manager *ComposeStackManager) Up(ctx context.Context, stack *portainer.Stack, endpoint *portainer.Endpoint) error {
-	url, proxy, err := manager.fetchEndpointProxy(endpoint)
+func (w *ComposeStackManager) Up(stack *portainer.Stack, endpoint *portainer.Endpoint) error {
+	url, proxy, err := w.fetchEndpointProxy(endpoint)
 	if err != nil {
-		return errors.Wrap(err, "failed to featch environment proxy")
+		return err
 	}
 
 	if proxy != nil {
@@ -56,17 +52,18 @@ func (manager *ComposeStackManager) Up(ctx context.Context, stack *portainer.Sta
 
 	envFilePath, err := createEnvFile(stack)
 	if err != nil {
-		return errors.Wrap(err, "failed to create env file")
+		return err
 	}
 
-	filePaths := getStackFiles(stack)
-	err = manager.deployer.Deploy(ctx, stack.ProjectPath, url, stack.Name, filePaths, envFilePath)
-	return errors.Wrap(err, "failed to deploy a stack")
+	filePath := stackFilePath(stack)
+
+	_, err = w.wrapper.Up(filePath, url, stack.Name, envFilePath, w.dataPath)
+	return err
 }
 
 // Down stops and removes containers, networks, images, and volumes. Wraps `docker-compose down --remove-orphans` command
-func (manager *ComposeStackManager) Down(ctx context.Context, stack *portainer.Stack, endpoint *portainer.Endpoint) error {
-	url, proxy, err := manager.fetchEndpointProxy(endpoint)
+func (w *ComposeStackManager) Down(stack *portainer.Stack, endpoint *portainer.Endpoint) error {
+	url, proxy, err := w.fetchEndpointProxy(endpoint)
 	if err != nil {
 		return err
 	}
@@ -74,28 +71,32 @@ func (manager *ComposeStackManager) Down(ctx context.Context, stack *portainer.S
 		defer proxy.Close()
 	}
 
-	filePaths := getStackFiles(stack)
-	err = manager.deployer.Remove(ctx, stack.ProjectPath, url, stack.Name, filePaths)
-	return errors.Wrap(err, "failed to remove a stack")
+	filePath := stackFilePath(stack)
+
+	_, err = w.wrapper.Down(filePath, url, stack.Name)
+	return err
 }
 
-// NormalizeStackName returns a new stack name with unsupported characters replaced
+// NormalizeStackName returns the passed stack name, for interface implementation only
 func (w *ComposeStackManager) NormalizeStackName(name string) string {
-	r := regexp.MustCompile("[^a-z0-9]+")
-	return r.ReplaceAllString(strings.ToLower(name), "")
+	return name
 }
 
-func (manager *ComposeStackManager) fetchEndpointProxy(endpoint *portainer.Endpoint) (string, *factory.ProxyServer, error) {
+func stackFilePath(stack *portainer.Stack) string {
+	return path.Join(stack.ProjectPath, stack.EntryPoint)
+}
+
+func (w *ComposeStackManager) fetchEndpointProxy(endpoint *portainer.Endpoint) (string, *factory.ProxyServer, error) {
 	if strings.HasPrefix(endpoint.URL, "unix://") || strings.HasPrefix(endpoint.URL, "npipe://") {
 		return "", nil, nil
 	}
 
-	proxy, err := manager.proxyManager.CreateComposeProxyServer(endpoint)
+	proxy, err := w.proxyManager.CreateComposeProxyServer(endpoint)
 	if err != nil {
 		return "", nil, err
 	}
 
-	return fmt.Sprintf("tcp://127.0.0.1:%d", proxy.Port), proxy, nil
+	return fmt.Sprintf("http://127.0.0.1:%d", proxy.Port), proxy, nil
 }
 
 func createEnvFile(stack *portainer.Stack) (string, error) {
@@ -115,29 +116,5 @@ func createEnvFile(stack *portainer.Stack) (string, error) {
 	}
 	envfile.Close()
 
-	return "stack.env", nil
-}
-
-// getStackFiles returns list of stack's confile file paths.
-// items in the list would be sanitized according to following criterias:
-// 1. no empty paths
-// 2. no "../xxx" paths that are trying to escape stack folder
-// 3. no dir paths
-// 4. root paths would be made relative
-func getStackFiles(stack *portainer.Stack) []string {
-	paths := make([]string, 0, len(stack.AdditionalFiles)+1)
-
-	for _, p := range append([]string{stack.EntryPoint}, stack.AdditionalFiles...) {
-		if strings.HasPrefix(p, "/") {
-			p = `.` + p
-		}
-
-		if p == `` || p == `.` || strings.HasPrefix(p, `..`) || strings.HasSuffix(p, string(filepath.Separator)) {
-			continue
-		}
-
-		paths = append(paths, p)
-	}
-
-	return paths
+	return envFilePath, nil
 }

api/exec/compose_stack_integration_test.go

@@ -1,7 +1,7 @@
+// +build integration
 package exec
 
 import (
-	"context"
 	"fmt"
 	"log"
 	"os"
@@ -48,9 +48,7 @@ func Test_UpAndDown(t *testing.T) {
 		t.Fatalf("Failed creating manager: %s", err)
 	}
 
-	ctx := context.TODO()
-
-	err = w.Up(ctx, stack, endpoint)
+	err = w.Up(stack, endpoint)
 	if err != nil {
 		t.Fatalf("Error calling docker-compose up: %s", err)
 	}
@@ -59,7 +57,7 @@ func Test_UpAndDown(t *testing.T) {
 		t.Fatal("container should exist")
 	}
 
-	err = w.Down(ctx, stack, endpoint)
+	err = w.Down(stack, endpoint)
 	if err != nil {
 		t.Fatalf("Error calling docker-compose down: %s", err)
 	}

api/exec/compose_stack_test.go

@@ -10,6 +10,47 @@ import (
 	"github.com/stretchr/testify/assert"
 )
 
+func Test_stackFilePath(t *testing.T) {
+	tests := []struct {
+		name     string
+		stack    *portainer.Stack
+		expected string
+	}{
+		// {
+		// 	name:     "should return empty result if stack is missing",
+		// 	stack:    nil,
+		// 	expected: "",
+		// },
+		// {
+		// 	name:     "should return empty result if stack don't have entrypoint",
+		// 	stack:    &portainer.Stack{},
+		// 	expected: "",
+		// },
+		{
+			name: "should allow file name and dir",
+			stack: &portainer.Stack{
+				ProjectPath: "dir",
+				EntryPoint:  "file",
+			},
+			expected: path.Join("dir", "file"),
+		},
+		{
+			name: "should allow file name only",
+			stack: &portainer.Stack{
+				EntryPoint: "file",
+			},
+			expected: "file",
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			result := stackFilePath(tt.stack)
+			assert.Equal(t, tt.expected, result)
+		})
+	}
+}
+
 func Test_createEnvFile(t *testing.T) {
 	dir := t.TempDir()
 
@@ -19,6 +60,11 @@ func Test_createEnvFile(t *testing.T) {
 		expected     string
 		expectedFile bool
 	}{
+		// {
+		// 	name:     "should not add env file option if stack is missing",
+		// 	stack:    nil,
+		// 	expected: "",
+		// },
 		{
 			name: "should not add env file option if stack doesn't have env variables",
 			stack: &portainer.Stack{
@@ -52,7 +98,7 @@ func Test_createEnvFile(t *testing.T) {
 			result, _ := createEnvFile(tt.stack)
 
 			if tt.expected != "" {
-				assert.Equal(t, "stack.env", result)
+				assert.Equal(t, path.Join(tt.stack.ProjectPath, "stack.env"), result)
 
 				f, _ := os.Open(path.Join(dir, "stack.env"))
 				content, _ := ioutil.ReadAll(f)
@@ -64,21 +110,3 @@ func Test_createEnvFile(t *testing.T) {
 		})
 	}
 }
-
-func Test_getStackFiles(t *testing.T) {
-	stack := &portainer.Stack{
-		EntryPoint: "./file", // picks entry point
-		AdditionalFiles: []string{
-			``,                  // ignores empty string
-			`.`,                 // ignores .
-			`..`,                // ignores ..
-			`./dir/`,            // ignrores paths that end with trailing /
-			`/with-root-prefix`, // replaces "root" based paths with relative
-			`./relative`,        // keeps relative paths
-			`../escape`,         // prevents dir escape
-		},
-	}
-
-	filePaths := getStackFiles(stack)
-	assert.ElementsMatch(t, filePaths, []string{`./file`, `./with-root-prefix`, `./relative`})
-}

api/exec/kubernetes_deploy.go

@@ -14,74 +14,33 @@ import (
 	"strings"
 	"time"
 
-	"github.com/portainer/portainer/api/http/proxy/factory/kubernetes"
-	"github.com/portainer/portainer/api/http/security"
-	"github.com/portainer/portainer/api/kubernetes/cli"
-
 	portainer "github.com/portainer/portainer/api"
 	"github.com/portainer/portainer/api/crypto"
 )
 
-// KubernetesDeployer represents a service to deploy resources inside a Kubernetes environment(endpoint).
+// KubernetesDeployer represents a service to deploy resources inside a Kubernetes environment.
 type KubernetesDeployer struct {
-	binaryPath                  string
-	dataStore                   portainer.DataStore
-	reverseTunnelService        portainer.ReverseTunnelService
-	signatureService            portainer.DigitalSignatureService
-	kubernetesClientFactory     *cli.ClientFactory
-	kubernetesTokenCacheManager *kubernetes.TokenCacheManager
+	binaryPath           string
+	dataStore            portainer.DataStore
+	reverseTunnelService portainer.ReverseTunnelService
+	signatureService     portainer.DigitalSignatureService
 }
 
 // NewKubernetesDeployer initializes a new KubernetesDeployer service.
-func NewKubernetesDeployer(kubernetesTokenCacheManager *kubernetes.TokenCacheManager, kubernetesClientFactory *cli.ClientFactory, datastore portainer.DataStore, reverseTunnelService portainer.ReverseTunnelService, signatureService portainer.DigitalSignatureService, binaryPath string) *KubernetesDeployer {
+func NewKubernetesDeployer(datastore portainer.DataStore, reverseTunnelService portainer.ReverseTunnelService, signatureService portainer.DigitalSignatureService, binaryPath string) *KubernetesDeployer {
 	return &KubernetesDeployer{
-		binaryPath:                  binaryPath,
-		dataStore:                   datastore,
-		reverseTunnelService:        reverseTunnelService,
-		signatureService:            signatureService,
-		kubernetesClientFactory:     kubernetesClientFactory,
-		kubernetesTokenCacheManager: kubernetesTokenCacheManager,
+		binaryPath:           binaryPath,
+		dataStore:            datastore,
+		reverseTunnelService: reverseTunnelService,
+		signatureService:     signatureService,
 	}
 }
 
-func (deployer *KubernetesDeployer) getToken(request *http.Request, endpoint *portainer.Endpoint, setLocalAdminToken bool) (string, error) {
-	tokenData, err := security.RetrieveTokenData(request)
-	if err != nil {
-		return "", err
-	}
-
-	kubecli, err := deployer.kubernetesClientFactory.GetKubeClient(endpoint)
-	if err != nil {
-		return "", err
-	}
-
-	tokenCache := deployer.kubernetesTokenCacheManager.GetOrCreateTokenCache(int(endpoint.ID))
-
-	tokenManager, err := kubernetes.NewTokenManager(kubecli, deployer.dataStore, tokenCache, setLocalAdminToken)
-	if err != nil {
-		return "", err
-	}
-
-	if tokenData.Role == portainer.AdministratorRole {
-		return tokenManager.GetAdminServiceAccountToken(), nil
-	}
-
-	token, err := tokenManager.GetUserServiceAccountToken(int(tokenData.ID), endpoint.ID)
-	if err != nil {
-		return "", err
-	}
-
-	if token == "" {
-		return "", fmt.Errorf("can not get a valid user service account token")
-	}
-	return token, nil
-}
-
-// Deploy will deploy a Kubernetes manifest inside a specific namespace in a Kubernetes environment(endpoint).
+// Deploy will deploy a Kubernetes manifest inside a specific namespace in a Kubernetes endpoint.
 // Otherwise it will use kubectl to deploy the manifest.
-func (deployer *KubernetesDeployer) Deploy(request *http.Request, endpoint *portainer.Endpoint, stackConfig string, namespace string) (string, error) {
+func (deployer *KubernetesDeployer) Deploy(endpoint *portainer.Endpoint, stackConfig string, namespace string) (string, error) {
 	if endpoint.Type == portainer.KubernetesLocalEnvironment {
-		token, err := deployer.getToken(request, endpoint, true)
+		token, err := ioutil.ReadFile("/var/run/secrets/kubernetes.io/serviceaccount/token")
 		if err != nil {
 			return "", err
 		}
@@ -94,7 +53,7 @@ func (deployer *KubernetesDeployer) Deploy(request *http.Request, endpoint *port
 		args := make([]string, 0)
 		args = append(args, "--server", endpoint.URL)
 		args = append(args, "--insecure-skip-tls-verify")
-		args = append(args, "--token", token)
+		args = append(args, "--token", string(token))
 		args = append(args, "--namespace", namespace)
 		args = append(args, "apply", "-f", "-")
 
@@ -180,14 +139,8 @@ func (deployer *KubernetesDeployer) Deploy(request *http.Request, endpoint *port
 		return "", err
 	}
 
-	token, err := deployer.getToken(request, endpoint, false)
-	if err != nil {
-		return "", err
-	}
-
 	req.Header.Set(portainer.PortainerAgentPublicKeyHeader, deployer.signatureService.EncodedPublicKey())
 	req.Header.Set(portainer.PortainerAgentSignatureHeader, signature)
-	req.Header.Set(portainer.PortainerAgentKubernetesSATokenHeader, token)
 
 	resp, err := httpCli.Do(req)
 	if err != nil {
@@ -230,7 +183,7 @@ func (deployer *KubernetesDeployer) Deploy(request *http.Request, endpoint *port
 }
 
 // ConvertCompose leverages the kompose binary to deploy a compose compliant manifest.
-func (deployer *KubernetesDeployer) ConvertCompose(data []byte) ([]byte, error) {
+func (deployer *KubernetesDeployer) ConvertCompose(data string) ([]byte, error) {
 	command := path.Join(deployer.binaryPath, "kompose")
 	if runtime.GOOS == "windows" {
 		command = path.Join(deployer.binaryPath, "kompose.exe")
@@ -242,7 +195,7 @@ func (deployer *KubernetesDeployer) ConvertCompose(data []byte) ([]byte, error)
 	var stderr bytes.Buffer
 	cmd := exec.Command(command, args...)
 	cmd.Stderr = &stderr
-	cmd.Stdin = bytes.NewReader(data)
+	cmd.Stdin = strings.NewReader(data)
 
 	output, err := cmd.Output()
 	if err != nil {

api/exec/swarm_stack.go

@@ -8,18 +8,15 @@ import (
 	"os"
 	"os/exec"
 	"path"
-	"regexp"
 	"runtime"
-	"strings"
 
 	portainer "github.com/portainer/portainer/api"
-	"github.com/portainer/portainer/api/internal/stackutils"
 )
 
 // SwarmStackManager represents a service for managing stacks.
 type SwarmStackManager struct {
 	binaryPath           string
-	configPath           string
+	dataPath             string
 	signatureService     portainer.DigitalSignatureService
 	fileService          portainer.FileService
 	reverseTunnelService portainer.ReverseTunnelService
@@ -27,16 +24,16 @@ type SwarmStackManager struct {
 
 // NewSwarmStackManager initializes a new SwarmStackManager service.
 // It also updates the configuration of the Docker CLI binary.
-func NewSwarmStackManager(binaryPath, configPath string, signatureService portainer.DigitalSignatureService, fileService portainer.FileService, reverseTunnelService portainer.ReverseTunnelService) (*SwarmStackManager, error) {
+func NewSwarmStackManager(binaryPath, dataPath string, signatureService portainer.DigitalSignatureService, fileService portainer.FileService, reverseTunnelService portainer.ReverseTunnelService) (*SwarmStackManager, error) {
 	manager := &SwarmStackManager{
 		binaryPath:           binaryPath,
-		configPath:           configPath,
+		dataPath:             dataPath,
 		signatureService:     signatureService,
 		fileService:          fileService,
 		reverseTunnelService: reverseTunnelService,
 	}
 
-	err := manager.updateDockerCLIConfiguration(manager.configPath)
+	err := manager.updateDockerCLIConfiguration(dataPath)
 	if err != nil {
 		return nil, err
 	}
@@ -45,47 +42,51 @@ func NewSwarmStackManager(binaryPath, configPath string, signatureService portai
 }
 
 // Login executes the docker login command against a list of registries (including DockerHub).
-func (manager *SwarmStackManager) Login(registries []portainer.Registry, endpoint *portainer.Endpoint) {
-	command, args := manager.prepareDockerCommandAndArgs(manager.binaryPath, manager.configPath, endpoint)
+func (manager *SwarmStackManager) Login(dockerhub *portainer.DockerHub, registries []portainer.Registry, endpoint *portainer.Endpoint) {
+	command, args := manager.prepareDockerCommandAndArgs(manager.binaryPath, manager.dataPath, endpoint)
 	for _, registry := range registries {
 		if registry.Authentication {
 			registryArgs := append(args, "login", "--username", registry.Username, "--password", registry.Password, registry.URL)
 			runCommandAndCaptureStdErr(command, registryArgs, nil, "")
 		}
 	}
+
+	if dockerhub.Authentication {
+		dockerhubArgs := append(args, "login", "--username", dockerhub.Username, "--password", dockerhub.Password)
+		runCommandAndCaptureStdErr(command, dockerhubArgs, nil, "")
+	}
 }
 
 // Logout executes the docker logout command.
 func (manager *SwarmStackManager) Logout(endpoint *portainer.Endpoint) error {
-	command, args := manager.prepareDockerCommandAndArgs(manager.binaryPath, manager.configPath, endpoint)
+	command, args := manager.prepareDockerCommandAndArgs(manager.binaryPath, manager.dataPath, endpoint)
 	args = append(args, "logout")
 	return runCommandAndCaptureStdErr(command, args, nil, "")
 }
 
 // Deploy executes the docker stack deploy command.
 func (manager *SwarmStackManager) Deploy(stack *portainer.Stack, prune bool, endpoint *portainer.Endpoint) error {
-	filePaths := stackutils.GetStackFilePaths(stack)
-	command, args := manager.prepareDockerCommandAndArgs(manager.binaryPath, manager.configPath, endpoint)
+	stackFilePath := path.Join(stack.ProjectPath, stack.EntryPoint)
+	command, args := manager.prepareDockerCommandAndArgs(manager.binaryPath, manager.dataPath, endpoint)
 
 	if prune {
-		args = append(args, "stack", "deploy", "--prune", "--with-registry-auth")
+		args = append(args, "stack", "deploy", "--prune", "--with-registry-auth", "--compose-file", stackFilePath, stack.Name)
 	} else {
-		args = append(args, "stack", "deploy", "--with-registry-auth")
+		args = append(args, "stack", "deploy", "--with-registry-auth", "--compose-file", stackFilePath, stack.Name)
 	}
 
-	args = configureFilePaths(args, filePaths)
-	args = append(args, stack.Name)
-
 	env := make([]string, 0)
 	for _, envvar := range stack.Env {
 		env = append(env, envvar.Name+"="+envvar.Value)
 	}
-	return runCommandAndCaptureStdErr(command, args, env, stack.ProjectPath)
+
+	stackFolder := path.Dir(stackFilePath)
+	return runCommandAndCaptureStdErr(command, args, env, stackFolder)
 }
 
 // Remove executes the docker stack rm command.
 func (manager *SwarmStackManager) Remove(stack *portainer.Stack, endpoint *portainer.Endpoint) error {
-	command, args := manager.prepareDockerCommandAndArgs(manager.binaryPath, manager.configPath, endpoint)
+	command, args := manager.prepareDockerCommandAndArgs(manager.binaryPath, manager.dataPath, endpoint)
 	args = append(args, "stack", "rm", stack.Name)
 	return runCommandAndCaptureStdErr(command, args, nil, "")
 }
@@ -109,7 +110,7 @@ func runCommandAndCaptureStdErr(command string, args []string, env []string, wor
 	return nil
 }
 
-func (manager *SwarmStackManager) prepareDockerCommandAndArgs(binaryPath, configPath string, endpoint *portainer.Endpoint) (string, []string) {
+func (manager *SwarmStackManager) prepareDockerCommandAndArgs(binaryPath, dataPath string, endpoint *portainer.Endpoint) (string, []string) {
 	// Assume Linux as a default
 	command := path.Join(binaryPath, "docker")
 
@@ -118,7 +119,7 @@ func (manager *SwarmStackManager) prepareDockerCommandAndArgs(binaryPath, config
 	}
 
 	args := make([]string, 0)
-	args = append(args, "--config", configPath)
+	args = append(args, "--config", dataPath)
 
 	endpointURL := endpoint.URL
 	if endpoint.Type == portainer.EdgeAgentOnDockerEnvironment {
@@ -145,8 +146,8 @@ func (manager *SwarmStackManager) prepareDockerCommandAndArgs(binaryPath, config
 	return command, args
 }
 
-func (manager *SwarmStackManager) updateDockerCLIConfiguration(configPath string) error {
-	configFilePath := path.Join(configPath, "config.json")
+func (manager *SwarmStackManager) updateDockerCLIConfiguration(dataPath string) error {
+	configFilePath := path.Join(dataPath, "config.json")
 	config, err := manager.retrieveConfigurationFromDisk(configFilePath)
 	if err != nil {
 		return err
@@ -188,15 +189,3 @@ func (manager *SwarmStackManager) retrieveConfigurationFromDisk(path string) (ma
 
 	return config, nil
 }
-
-func (manager *SwarmStackManager) NormalizeStackName(name string) string {
-	r := regexp.MustCompile("[^a-z0-9]+")
-	return r.ReplaceAllString(strings.ToLower(name), "")
-}
-
-func configureFilePaths(args []string, filePaths []string) []string {
-	for _, path := range filePaths {
-		args = append(args, "--compose-file", path)
-	}
-	return args
-}

api/exec/swarm_stack_test.go

@@ -1,15 +0,0 @@
-package exec
-
-import (
-	"testing"
-
-	"github.com/stretchr/testify/assert"
-)
-
-func TestConfigFilePaths(t *testing.T) {
-	args := []string{"stack", "deploy", "--with-registry-auth"}
-	filePaths := []string{"dir/file", "dir/file-two", "dir/file-three"}
-	expected := []string{"stack", "deploy", "--with-registry-auth", "--compose-file", "dir/file", "--compose-file", "dir/file-two", "--compose-file", "dir/file-three"}
-	output := configureFilePaths(args, filePaths)
-	assert.ElementsMatch(t, expected, output, "wrong output file paths")
-}

api/filesystem/copy_test.go

@@ -1,92 +0,0 @@
-package filesystem
-
-import (
-	"io/ioutil"
-	"os"
-	"path"
-	"path/filepath"
-	"testing"
-
-	"github.com/stretchr/testify/assert"
-)
-
-func Test_copyFile_returnsError_whenSourceDoesNotExist(t *testing.T) {
-	tmpdir, _ := ioutil.TempDir("", "backup")
-	defer os.RemoveAll(tmpdir)
-
-	err := copyFile("does-not-exist", tmpdir)
-	assert.Error(t, err)
-}
-
-func Test_copyFile_shouldMakeAbackup(t *testing.T) {
-	tmpdir, _ := ioutil.TempDir("", "backup")
-	defer os.RemoveAll(tmpdir)
-
-	content := []byte("content")
-	ioutil.WriteFile(path.Join(tmpdir, "origin"), content, 0600)
-
-	err := copyFile(path.Join(tmpdir, "origin"), path.Join(tmpdir, "copy"))
-	assert.NoError(t, err)
-
-	copyContent, _ := ioutil.ReadFile(path.Join(tmpdir, "copy"))
-	assert.Equal(t, content, copyContent)
-}
-
-func Test_CopyDir_shouldCopyAllFilesAndDirectories(t *testing.T) {
-	destination, _ := ioutil.TempDir("", "destination")
-	defer os.RemoveAll(destination)
-	err := CopyDir("./testdata/copy_test", destination, true)
-	assert.NoError(t, err)
-
-	assert.FileExists(t, filepath.Join(destination, "copy_test", "outer"))
-	assert.FileExists(t, filepath.Join(destination, "copy_test", "dir", ".dotfile"))
-	assert.FileExists(t, filepath.Join(destination, "copy_test", "dir", "inner"))
-}
-
-func Test_CopyDir_shouldCopyOnlyDirContents(t *testing.T) {
-	destination, _ := ioutil.TempDir("", "destination")
-	defer os.RemoveAll(destination)
-	err := CopyDir("./testdata/copy_test", destination, false)
-	assert.NoError(t, err)
-
-	assert.FileExists(t, filepath.Join(destination, "outer"))
-	assert.FileExists(t, filepath.Join(destination, "dir", ".dotfile"))
-	assert.FileExists(t, filepath.Join(destination, "dir", "inner"))
-}
-
-func Test_CopyPath_shouldSkipWhenNotExist(t *testing.T) {
-	tmpdir, _ := ioutil.TempDir("", "backup")
-	defer os.RemoveAll(tmpdir)
-
-	err := CopyPath("does-not-exists", tmpdir)
-	assert.NoError(t, err)
-
-	assert.NoFileExists(t, tmpdir)
-}
-
-func Test_CopyPath_shouldCopyFile(t *testing.T) {
-	tmpdir, _ := ioutil.TempDir("", "backup")
-	defer os.RemoveAll(tmpdir)
-
-	content := []byte("content")
-	ioutil.WriteFile(path.Join(tmpdir, "file"), content, 0600)
-
-	os.MkdirAll(path.Join(tmpdir, "backup"), 0700)
-	err := CopyPath(path.Join(tmpdir, "file"), path.Join(tmpdir, "backup"))
-	assert.NoError(t, err)
-
-	copyContent, err := ioutil.ReadFile(path.Join(tmpdir, "backup", "file"))
-	assert.NoError(t, err)
-	assert.Equal(t, content, copyContent)
-}
-
-func Test_CopyPath_shouldCopyDir(t *testing.T) {
-	destination, _ := ioutil.TempDir("", "destination")
-	defer os.RemoveAll(destination)
-	err := CopyPath("./testdata/copy_test", destination)
-	assert.NoError(t, err)
-
-	assert.FileExists(t, filepath.Join(destination, "copy_test", "outer"))
-	assert.FileExists(t, filepath.Join(destination, "copy_test", "dir", ".dotfile"))
-	assert.FileExists(t, filepath.Join(destination, "copy_test", "dir", "inner"))
-}

api/filesystem/filesystem.go

@@ -31,8 +31,6 @@ const (
 	ComposeStorePath = "compose"
 	// ComposeFileDefaultName represents the default name of a compose file.
 	ComposeFileDefaultName = "docker-compose.yml"
-	// ManifestFileDefaultName represents the default name of a k8s manifest file.
-	ManifestFileDefaultName = "k8s-deployment.yml"
 	// EdgeStackStorePath represents the subfolder where edge stack files are stored in the file store folder.
 	EdgeStackStorePath = "edge_stacks"
 	// PrivateKeyFile represents the name on disk of the file containing the private key.
@@ -43,8 +41,6 @@ const (
 	BinaryStorePath = "bin"
 	// EdgeJobStorePath represents the subfolder where schedule files are stored.
 	EdgeJobStorePath = "edge_jobs"
-	// DockerConfigPath represents the subfolder where docker configuration is stored.
-	DockerConfigPath = "docker_config"
 	// ExtensionRegistryManagementStorePath represents the subfolder where files related to the
 	// registry management extension are stored.
 	ExtensionRegistryManagementStorePath = "extensions"
@@ -52,12 +48,6 @@ const (
 	CustomTemplateStorePath = "custom_templates"
 	// TempPath represent the subfolder where temporary files are saved
 	TempPath = "tmp"
-	// SSLCertPath represents the default ssl certificates path
-	SSLCertPath = "certs"
-	// DefaultSSLCertFilename represents the default ssl certificate file name
-	DefaultSSLCertFilename = "cert.pem"
-	// DefaultSSLKeyFilename represents the default ssl key file name
-	DefaultSSLKeyFilename = "key.pem"
 )
 
 // ErrUndefinedTLSFileType represents an error returned on undefined TLS file type
@@ -82,11 +72,6 @@ func NewService(dataStorePath, fileStorePath string) (*Service, error) {
 		return nil, err
 	}
 
-	err = service.createDirectoryInStore(SSLCertPath)
-	if err != nil {
-		return nil, err
-	}
-
 	err = service.createDirectoryInStore(TLSStorePath)
 	if err != nil {
 		return nil, err
@@ -102,11 +87,6 @@ func NewService(dataStorePath, fileStorePath string) (*Service, error) {
 		return nil, err
 	}
 
-	err = service.createDirectoryInStore(DockerConfigPath)
-	if err != nil {
-		return nil, err
-	}
-
 	return service, nil
 }
 
@@ -115,11 +95,6 @@ func (service *Service) GetBinaryFolder() string {
 	return path.Join(service.fileStorePath, BinaryStorePath)
 }
 
-// GetDockerConfigPath returns the full path to the docker config store on the filesystem
-func (service *Service) GetDockerConfigPath() string {
-	return path.Join(service.fileStorePath, DockerConfigPath)
-}
-
 // RemoveDirectory removes a directory on the filesystem.
 func (service *Service) RemoveDirectory(directoryPath string) error {
 	return os.RemoveAll(directoryPath)
@@ -288,7 +263,7 @@ func (service *Service) StoreTLSFileFromBytes(folder string, fileType portainer.
 	return path.Join(service.fileStorePath, tlsFilePath), nil
 }
 
-// GetPathForTLSFile returns the absolute path to a specific TLS file for an environment(endpoint).
+// GetPathForTLSFile returns the absolute path to a specific TLS file for an endpoint.
 func (service *Service) GetPathForTLSFile(folder string, fileType portainer.TLSFileType) (string, error) {
 	var fileName string
 	switch fileType {
@@ -364,7 +339,13 @@ func (service *Service) WriteJSONToFile(path string, content interface{}) error
 
 // FileExists checks for the existence of the specified file.
 func (service *Service) FileExists(filePath string) (bool, error) {
-	return FileExists(filePath)
+	if _, err := os.Stat(filePath); err != nil {
+		if os.IsNotExist(err) {
+			return false, nil
+		}
+		return false, err
+	}
+	return true, nil
 }
 
 // KeyPairFilesExist checks for the existence of the key files.
@@ -589,83 +570,3 @@ func (service *Service) GetTemporaryPath() (string, error) {
 func (service *Service) GetDatastorePath() string {
 	return service.dataStorePath
 }
-
-func (service *Service) wrapFileStore(filepath string) string {
-	return path.Join(service.fileStorePath, filepath)
-}
-
-func defaultCertPathUnderFileStore() (string, string) {
-	certPath := path.Join(SSLCertPath, DefaultSSLCertFilename)
-	keyPath := path.Join(SSLCertPath, DefaultSSLKeyFilename)
-	return certPath, keyPath
-}
-
-// GetDefaultSSLCertsPath returns the ssl certs path
-func (service *Service) GetDefaultSSLCertsPath() (string, string) {
-	certPath, keyPath := defaultCertPathUnderFileStore()
-	return service.wrapFileStore(certPath), service.wrapFileStore(keyPath)
-}
-
-// StoreSSLCertPair stores a ssl certificate pair
-func (service *Service) StoreSSLCertPair(cert, key []byte) (string, string, error) {
-	certPath, keyPath := defaultCertPathUnderFileStore()
-
-	r := bytes.NewReader(cert)
-	err := service.createFileInStore(certPath, r)
-	if err != nil {
-		return "", "", err
-	}
-
-	r = bytes.NewReader(key)
-	err = service.createFileInStore(keyPath, r)
-	if err != nil {
-		return "", "", err
-	}
-
-	return service.wrapFileStore(certPath), service.wrapFileStore(keyPath), nil
-}
-
-// CopySSLCertPair copies a ssl certificate pair
-func (service *Service) CopySSLCertPair(certPath, keyPath string) (string, string, error) {
-	defCertPath, defKeyPath := service.GetDefaultSSLCertsPath()
-
-	err := service.Copy(certPath, defCertPath, false)
-	if err != nil {
-		return "", "", err
-	}
-
-	err = service.Copy(keyPath, defKeyPath, false)
-	if err != nil {
-		return "", "", err
-	}
-
-	return defCertPath, defKeyPath, nil
-}
-
-// FileExists checks for the existence of the specified file.
-func FileExists(filePath string) (bool, error) {
-	if _, err := os.Stat(filePath); err != nil {
-		if os.IsNotExist(err) {
-			return false, nil
-		}
-		return false, err
-	}
-	return true, nil
-}
-
-func MoveDirectory(originalPath, newPath string) error {
-	if _, err := os.Stat(originalPath); err != nil {
-		return err
-	}
-
-	alreadyExists, err := FileExists(newPath)
-	if err != nil {
-		return err
-	}
-
-	if alreadyExists {
-		return errors.New("Target path already exists")
-	}
-
-	return os.Rename(originalPath, newPath)
-}

api/filesystem/filesystem_fileexists_test.go

@@ -1,55 +0,0 @@
-package filesystem
-
-import (
-	"fmt"
-	"math/rand"
-	"os"
-	"path"
-	"testing"
-
-	"github.com/stretchr/testify/assert"
-)
-
-func Test_fileSystemService_FileExists_whenFileExistsShouldReturnTrue(t *testing.T) {
-	service := createService(t)
-	testHelperFileExists_fileExists(t, service.FileExists)
-}
-
-func Test_fileSystemService_FileExists_whenFileNotExistsShouldReturnFalse(t *testing.T) {
-	service := createService(t)
-	testHelperFileExists_fileNotExists(t, service.FileExists)
-}
-
-func Test_FileExists_whenFileExistsShouldReturnTrue(t *testing.T) {
-	testHelperFileExists_fileExists(t, FileExists)
-}
-
-func Test_FileExists_whenFileNotExistsShouldReturnFalse(t *testing.T) {
-	testHelperFileExists_fileNotExists(t, FileExists)
-}
-
-func testHelperFileExists_fileExists(t *testing.T, checker func(path string) (bool, error)) {
-	file, err := os.CreateTemp("", t.Name())
-	assert.NoError(t, err, "CreateTemp should not fail")
-
-	t.Cleanup(func() {
-		os.RemoveAll(file.Name())
-	})
-
-	exists, err := checker(file.Name())
-	assert.NoError(t, err, "FileExists should not fail")
-
-	assert.True(t, exists)
-}
-
-func testHelperFileExists_fileNotExists(t *testing.T, checker func(path string) (bool, error)) {
-	filePath := path.Join(os.TempDir(), fmt.Sprintf("%s%d", t.Name(), rand.Int()))
-
-	err := os.RemoveAll(filePath)
-	assert.NoError(t, err, "RemoveAll should not fail")
-
-	exists, err := checker(filePath)
-	assert.NoError(t, err, "FileExists should not fail")
-
-	assert.False(t, exists)
-}